Can an LLM Be a Black-Box Optimizer?

At Terra AI, I have a great collaborator who describes himself as “optimization-brained,” and always comes up with the best strategies for optimization problems, whether he’s coming up the with best linear algebra way of performing a calculation, or he’s thinking about the best way to apply a set of constraints. This is in contrast to my experience in my PhD, where I mostly applied the classic “graduate student descent” algorithm, manually fiddling with parameters and generally trying and get an intuition for how a system works. In my defence, this was how most of my colleagues did optimization for Earth-system models. Of course, if you’re doing a well-defined problem like regularized inversion of a geophysics problem, you’d use a robust optimizer approach like Gauss-Newton or L-BFGS. But often you’re working something like a complex-system model with many interacting components, and you probably don’t have access to gradients or adjoints. An example of this could be a model of the dynamics of geologic system that couples weathering, erosion, sediment transport, and biological processes, and you’re trying to fit a model to data that has a lot of noise. In Earth sciences, we end up in this situation all the time: you have this huge, complicated simulation code (probably written in MATLAB), and you have to fiddle with some input parameters to get the outputs to match your field data. Oh and also, the simulation takes 2 hours to run or your laptop. Are you really going to wrap this in an optimizer loop? Chris Rackauckas wrote a really nice blog post about this kind of thing in The nonlinear effect of code speed on productivity; Richard Hamming also writes about this back in the 1950s in his book “The Art of Doing Science and Engineering.” Of course you could use some approach like Bayesian optimization, but we’re Earth scientist! We don’t know about sophisticated techniques like that! So instead, people typically end up taking an approach to optimization that is part “grad-student descent,” and part “intuition-guided.”

But this got me thinking: a lot of human intuition is (for better or worse) encoded in written language. What if we could take the intuition that is latent in a large language model (LLM) to guide an optimization process? I’ve recently been getting better at using LLM APIs (I won an award for it!), so this seemed like a fun project to try out! I bet someone’s already done this, but I wanted to see where I could get myself, that’s why it’s called research!

Problem setup

For the optimization setup, I decided to use the classic 2D Rosenbrock function as a first test function. The function is defined as:

where and are the independent two variables. The aim of the optimization is to find the global minimum of the function , which is at with . To keep things simple, I chose to optimize the function in the range .

LLM Optimizer

How do you turn an LLM into an optimizer? Well as I’ve found in previous projects, you can just tell an LLM that it has that persona in its system prompt!

This is combined with a user prompt that specifies the history of past sample points and function values:

Both of these are supplied to the LLM API, and I get back a response like this:

So all I needed to do now is wrap this in a loop, and make a new API call for each step in the evaluation loop! I implemented the LLM optimizer in a stateless mode, which means that the LLM is given the full history of the optimization process at each step, but without any other context. An alternative approach would be to supply each point sequentially, in a chat dialoge fashion. I didn’t end up doing this because I thought the context might get a bit messy by the later evaluation steps. I originally started using o3-mini as my reasoning model of choice, mostly due to its lower cost and higher rate limits. However, after changing my API call scheduler a little bit I was also able to effectively experiment on a bigger model: gpt-5-mini.

Evaluation

Of course, I needed to be able to judge how well this LLM optimizer is doing, so I also implemented two other optimizers: Nelder-Mead, and Bayesian optimization.
Nelder-Mead is a region-based optimization technique that moves the corners of a simplex in the direction of the best point, whereas Bayesian optimization fits a Gaussian process model to the objective function and then samples the next point to evaluate based on the model’s uncertainty.
For the evaluation metric, I’ll be judging the optimizers based on the minimum objective function they find, given a fixed budget of 20 function evaluations.
Before I show the results, what do you think will happen?
Personally, I thought that Bayesian optimization would do the best, since it’s typically pretty sample-efficient, and I thought that the LLM optimizer would do second-best, ahead of gradient descent and Nelder-Mead.

A Slight Mis-Step

Almost immediately, I found a mistake with my problem setup. The LLM would immediately guess that the minimum would either be at the origin, or at $(1, 1)$.

I believe the reason for this is this is where minima are typically located in commonly used optimization test functions.
I fixed this problem with three tweaks: (1) I added a small random translation to the function, (2) set the LLM optimization routine to start with a random initial point, and (3) altered the system prompt to encourage the LLM to explore the domain more.

Results

With that bug cleared up, I was able to run the evaluation suite for all optimization methods! Before looking at all the results, I think it’s cool to look at an example of the LLMs responses.

I found this so cool to see! You can sort of make out a thinking process here, where the LLM first explores wide, and then gets more and more precise.
When running experiments, I’d track the API responses in a terminal window, and I often ended up making similar next guesses to the LLM. I.e., “Ok we’re going along this line, and the value is still decreasing, so let’s take another step in that direction. Oh ok now the value is higher, so let’s take a step back, and move orthogonal instead…” and so on.

Anyway, let’s see how the LLM approach compares to the other optimizers. The figure below shows an example of the optimization paths for each method.

In this example, Nelder-Mead was not able to get close to the global minimum in the fixed budget of 20 function evaluations.
Although Bayesian optimization made a good first start, it didn’t make great progress past the 9th function evaluation.
However, as luck would have it, the LLM optimizer performed better than all the other optimizers, getting fairly close to the global minimum!
I thought this was probably a fluke though, and ran another attempt. This time, I’m visualizing the objective function per iteration step, for each optimization method.

Again, the LLM optimizer ended up doing the best! Still I wan’t convinced. So I ended up running 10 experiments (starting at random initial points) and taking the average of the results. Would the LLM optimizer be statistically better than the other optimizers?

It seems that GPT-5 is actually quite a fine optimizer, in a latency-agnostic, limited function evaluation setting!

Conclusions

So what can I gather from this? Well it’s been making me thing about the different choices one makes when deciding on an optimization (or decision-making) strategy. I think a big deciding factor is robustness. Although approaches like Nelder-Mead etc… don’t have guarrantees to converge to minima, for practical purposes they will “work,” because they have been constructed for that purpose. With a prompt-guided LLM, could really do anything: hallucinate, get caught in a loop, refuse to return formatted outputs, etc. If you have a safety-critical application, one might prefer a more robust method over a “fuzzy” LLM approach. Another big factor is latency. LLM API calls are (usually) slow, and for some settings you just can’t afford that. (On the topic of affordability, LLMs aren’t free! These experiments cost me around $10 in API calls.) Overall this makes me think of another colleague I work with, who works on the Curiosity rover, where decisions are made by committee, and latency is minutes to hours. I don’t see LLMs being used in settings like that, but there are similarities!

Will I actually use this in my work? Probably not in its current form. But if someone made a really robust LLM optimizer, I’d actually consider using it for real problems. I think in cases were you don’t have access to gradients, you have a small budget for function evaluations, and you don’t want to wrap your code in an optimizer loop, this could actually be a valid approach!
I could imaging this working in an interface where I relay inputs and outputs to an LLM that is set up as a kind of hybrid online-offline optimizer, and could see myself using it as a good addition to “intuition-guided” optimization.

I think some interesting future work could be to experiment with different formulations of the prompting, e.g., a stateful dialogue. It would also be good to compare the performance of reasoning vs. non-reasoning LLMs, as well as testing on other optimization problems (even some real settings)! If you’re interested in the code for this project, you can check it out in this github repo!