Modeling Uncertainty for Better Outcomes

By John Mount on August 8, 2025 •

Recently here at Win Vector LLC we have been delivering good client outcomes using the Stan MCMC sampler. It has allowed us to infer deep business factors, instead of being limited surface KPIs (key performance indicators). Modeling uncertainty requires stronger optimizers to solve our problems, but it leads to better anti-fragile business solutions.

A fun part of this is it really improves how visible uncertainty is. Let’s show this in a quick simplified example.

Consider the classic optimization problem that pops up in planning or logistics. We assume we have (from who knows where) a set of (hopefully linear) constraints and a (hopefully linear) objective function (such as expected revenue) to optimize over. We illustrate this as follows.

The cream area is the region of feasible plans or solutions. The blue arrow is the objective to optimize (move as far as possible in this direction). And using the famous simplex method, we end up with a solution at a vertex indicated by the red circle.

We accept the vertex solution as okay as:

• The theory says vertex solutions are inevitable for the popular simplex method solver. There is an old adage that what is inevitable, is considered acceptable.
• The vertex picked is slightly better that any other vertex solution. This is considered desirable, even when there are similarly valued vertices far away from the chosen solution (which should be a red flag).

Now consider the case where instead of being given the objective function from somebody we can hang the blame on, we have to estimate it ourselves from data. This is in fact the project breaking point we want to discuss. Too many quantitative projects get factored into:

• A statistical inference, machine learning, business intelligence, or predictive analytics phase where everything is estimated with no regard for how it will be used.
• A action or optimization phase, where inferences are used with no regard for how they were produced.

We’ve been having great success in combining these phases to improve result quality and reliability. For example, when we use Stan to estimate an objective function from past data: we, by default, get 4000 different estimates. This lets us track consequences between inference and use. The situation now looks more like the following.

Here is the problem: maximizing expected revenue for the original blue objective arrow may not be maximizing it for many of the plausible estimates of the unobserved true market. If this is the case: doing better on the lab workbench picks a strategy that plausibly does worse in the market. We lose the client money while bragging about our great plan.

The collection of different estimates of the objective function helps usefully express the uncertainty in estimation. This is bringing some of the uncertainty one would see in deployment back into the planning stage, where one can do something about it. There are several obvious (not recommended) strategies to optimize in this case.

• Ignore the wealth of information, and just use the first estimate as “the objective.”
• Optimize over a summary of the different objective functions, such as the mean or median.
• Solve the optimization problem for every objective and then average the answer (as depicted in the diagram, notice this degenerated into a non-efficient interior solution).

With the explicit depiction of uncertainty, more powerful methods can be used for additional benefit. The Stan sample of plausible objective functions allows us to use the following solution form.

Define:

• x the encoding our any proposed plan, pricing, offerings, or allocation of resources.
• β the world parameters estimated by Stan.
• f_β(x) the function we are trying to optimize for a given objective function β. The goal is to find a good x that makes f_β(x) large for many β.
• E_β[expression] denotes computing the expected value of expression over all the βs.
• VAR_β[expression] denotes computing the variance of expression over all the βs.
• λ a numeric parameter expressing our level of risk aversion. We either pick this, or try a few values (such as 1/2, 1, 2) and inspect results.

Pick x optimizing E_β[f_β(x)] - λ √(VAR_β[f_β(x)]).

This is no longer a linear program, but within the capabilities of modern solvers (this moves us from chapter 3 of Dantzig Linear Programming and Extensions to chapter 25). This form may seem somewhat arbitrary. However it is the form you get if:

• You try to maximize expected value holding risk constant.
• You try to minimize risk holding expected value constant.
• You treat expected value and risk as two objectives in multi-objective optimization that you can trade-off or price against each other.
• You treat this as a portfolio style problem with a client risk tolerance level.

(That being said, we could still replace the E_β[*] - √(VAR_β[*]) form with a partial moment or even a quantile criteria depending on the application.)

In all cases the solution looks like the following.

The solution is now in the middle of the face, in the direction of more of the objective estimates. This a solution that is high value (a nearly best value for most estimates of the objective function) and low regret (not really bad for the rest of the estimates of the objective function). This isn’t a fragile “well this is best in-lab estimate I bet your farm on.” This is an anti-fragile solution that will perform well when it leaves the lab and is implemented or deployed.

What we have found is: using Stan forces one deal with uncertainty (as it returns multiple estimates) early. Dealing with uncertainty earlier is a bit more complicated and requires more sophisticated optimization ideas and tools. However, dealing with uncertainty early means correcting for it during planning and not being victim to it during deployment. This ends up delivering higher customer value.

Categories: Exciting Techniques Tutorials

Tagged as: operations research Optimization Uncertainty