.png)
4 months ago
8
Are the LLM's answers perfect? No (ChatGPT o3's statistical analysis had some significant mathematical errors).
Are they helpful? Well, a 35-75% range for the estimated probability of trial success is not very helpful. But the overall analysis, and some of the sources cited, do have value.
How much value these answers provide depends on your role and expertise. If you have access to sophisticated tools for simulating clinical trials, or you are a hedge fund analyst covering immuno-oncology, maybe the LLMs are not that helpful, except as a sanity check.
But if you don't specialize in these areas, need a quick answer, or are doing a first draft of an analysis, these responses are much more informative than pulling the average oncology trial success rates from the literature.
The AI answers are just a starting point. They give you a head start, but there is always room for you to run farther.
These examples represent the floor of what these models are capable of. I did not test different prompts, asked no follow up questions, provided no additional context or information to the models, and did not design any custom tools or workflows to help them accomplish the task more effectively.
Given the right tools and guidance, these LLMs could perform significantly better.
For example, an expert statistician could write custom software for statistical analysis of oncology study results, and give the LLMs access to this software through their "tool calling" functionality.
Or, you could just keep pushing the AI to improve (the same way you would coach an intern). Ask the AI to identify errors in its reasoning, tell it to go deeper on a certain topic, challenge its thinking, etc.
These techniques significantly improve the quality of output: because they leverage human expertise and judgment. Humans + AI > AI alone.
