Decision-Bench: A Frontier Benchmark for Enterprise Decisions

Today, we’re introducing Decision-Bench, a benchmark that tests whether frontier AI models can turn realistic enterprise data into decisions that improve the bottom line.

Decision-Bench grew out of our experience helping enterprises turn AI deployments into measurable business results. Its 210 tasks reflect that work, from forecasting costs and investigating fraud to deciding where to cut spending. The benchmark brings four properties together:

1. Enterprise scale. Models work across 7.49 billion rows and 235 tables covering orders, payments, courier compensation, and accounting.
2. Real business systems. The warehouse is modeled on Oracle E-Business Suite, with custom operational tables. Models must connect records across the systems a company uses to run its business.
3. Decisions with consequences. Models choose where to cut spending, which accounts to close, and how much to budget. Cutting one expense can raise another. Closing the wrong account can destroy legitimate business.
4. Measurable outcomes. We grade each submission against what the work is meant to accomplish, including whether a decision saves the business money or leaves it worse off.

Existing data benchmarks stop short of the decisions enterprises need AI to make. BIRD tests whether generated SQL returns the expected result. DABstep tests multi-step analysis against predetermined answers. Both measure whether a model can answer questions about a business. Neither measures what happens when the business acts on its answer.

Business decisions, however, are less prescribed. A user can specify an objective, such as reducing costs or preventing fraud, without knowing the best action to take. The model has to identify its options, weigh the tradeoffs, and choose. Success depends on what that choice accomplishes. A correct query is one step toward that result, not evidence that the result was achieved.

Benchmarks that evaluate action address part of this gap. Vending-Bench asks agents to make purchasing and pricing decisions and measures their financial results. But running a vending machine does not test whether an agent can make those decisions from billions of records across an enterprise’s financial and operational systems.

Decision-Bench brings the consequences of a decision into an enterprise-scale evaluation. Models must work through interconnected financial and operational records, choose an action, and account for its effects. We measure whether that work achieves the business objective.

Decision-Bench Leaderboard

The strongest evaluated model reached the solved threshold on 34.3% of tasks. Claude Opus 5.5 leads the v1.1 results with a best reported average score of 59.1 out of 100. GPT-6 Astra averaged 51.5, and Claude Sonnet 5.5 averaged 51.2.

The leaderboard lists each model at its highest-scoring tested reasoning setting. It shows the average score alongside the share of tasks that reached the solved threshold.

Performance across reasoning settings

The curves illustrate the results another way: how much performance changes as we give each model more room to reason, and what that improvement costs. Each point represents a tested reasoning setting, plotted against its average score and model API cost per task.

Here’s how to read the benchmarks:

1. Start with one model. Follow its curve through its reasoning settings, such as low, medium, and high. These settings differ across providers, so “high” does not represent the same amount of computation for every model.
2. Compare performance and cost. A curve that climbs as it moves left shows that extra spending bought a better result. At the same score, the point farther right is cheaper.
3. Keep score and accuracy separate. A score of 60 means an average of 60 points under the task-specific grading rules. It does not mean 60% accuracy or that the model completed 60% of tasks.
4. Check the solved rate. Average scores include partial credit. We count a task as solved when it earns at least 95 out of 100. Hover over a point to see its solved rate, score, cost, and reasoning setting.

The lines connect settings we tested; they do not predict performance between those points. The cost axis includes model API charges. We examine warehouse costs separately below.

Explore

Overview

Decision-Bench places models inside a simulated New York City food delivery business operating over the course of 2024. The business gives every task a shared context: customers place orders, couriers make deliveries, restaurants receive payouts, and the company keeps the books.

Each model acts as an agent, using tools to query data, run analysis, and submit work for evaluation. We give models the same tools and instructions so we can compare how they approach the work.

1. Enterprise scale

The evaluation world contains 81 million orders from 3.4 million active customers, recorded in 7.49 billion rows across 235 tables. Agents work across the operational and financial records of the business.

That is a different scale of investigation from many established analysis benchmarks. DABstep’s published payments dataset, for example, contains 138,000 transactions. At enterprise scale, the agent’s choices about what to inspect and query affect both its ability to finish the work and the cost of doing so.

2. Real business systems

A single delivery creates records for the order, the customer’s payment, the courier’s pay, the restaurant’s payout, and the accounting entries behind each. Understanding the delivery means connecting those records.

We designed the warehouse around those connections. Working with three experienced ERP consultants, we modeled it on Oracle E-Business Suite, one of the enterprise resource planning systems companies use to run their operations and keep their books.

The platform also extends its ERP system to support its own operations. Its 235 tables have distinct structures, including 76 custom tables for dispatch, courier incentives, delivery delays, and other parts of the delivery business. Agents have to understand how those custom records connect to the standard financial records.

Raw table count can hide how much of that work a task requires. A warehouse may contain thousands of tables that repeat the same structure across dates or regions. In our analysis of Spider 2.0, three-quarters of the questions with public reference SQL touched no more than two logical tables after we grouped repeated tables together. The forecasting example in our paper requires six distinct tables across three ERP modules just to reconstruct courier pay before estimating the following month’s costs.

3. Decisions with consequences

Agents submit work the business could use: forecasts, published dashboard data, account bans, and spending plans. Each submission is evaluated according to its purpose.

The business also changes while those decisions are being made. In April, the platform responds to a minimum-pay increase modeled on the change introduced in New York City in 2024. The policy changes courier scheduling and compensation. An agent forecasting costs for May has to account for that change instead of assuming earlier patterns will continue.

Other tasks require agents to weigh the costs of acting. A fraud investigation can prevent losses, but closing legitimate accounts also damages the business. A spending cut can reduce one expense while creating another. The agent must account for both effects.

4. Measurable outcomes

Historical records leave some of those effects hidden. They show the fraud a company caught, but not every fraud that occurred. They show what happened under an existing policy, but not what would have happened under an alternative.

The simulator lets the evaluator observe those missing outcomes. For a forecast, the agent might receive records through April while the evaluator can inspect May’s results. For a spending plan, the evaluator can measure how the business responds to the proposed allocation.

The agent still has everything it needs to do the work. Every task has an executable reference solution based only on the warehouse available to the agent. The evaluator’s additional information makes grading possible; solving the task does not require access to it.

Methodology

World construction

We built the business using public datasets, regulatory disclosures, company filings, and industry research. The simulator generates every order, courier shift, and payment. No warehouse record is generated by a language model.

We checked the resulting economics against reported New York City figures. The simulation’s per-delivery economics stay within 5% in 13 of 16 quarterly comparisons. The exception is courier pay after the April minimum-pay increase, when the simulator runs 6–9% above the reported figures.

Task construction

The 210 tasks cover five parts of an analyst’s job. We report results separately so a model’s strengths and weaknesses remain visible.

Each task has an executable reference solution that establishes the work can be completed from the available records. Agents submit their own work through a common interface called Mission Control.

Evaluation harness

The evaluation in the paper ran 14 models through a common Inspect AI harness, the software that runs each task and records the agent’s steps.

Inspect AI gives models the same instructions, tools, and execution limits, reducing the chance that differences in setup drive differences in scores. Its records also let us inspect how an agent reached a decision and where the work went wrong.

Models can inspect tables, query data, and execute Python analysis. They work in isolated environments with statistical, machine-learning, and optimization libraries, without internet access. Each run can use up to 500 model turns and three hours.

Grading and private evaluation

Task-specific rules grade the submitted work, without another language model acting as judge. Dashboard data must contain the required values. Forecasts are evaluated against future outcomes and a reference forecast. Budget plans are scored on their simulated results. Many fraud tasks weigh prevented losses against the cost of incorrect bans.

These rules produce scores from 0 to 100, with partial credit where the task allows it. Each category score averages the tasks in that category; the overall score averages all 210 tasks.

Official evaluations use a separately generated version of the business. Its entities and answer keys differ from those in the public dataset, although the rules governing the business are shared. Memorizing public answers will not reproduce the private world’s results. The simulator, graders, and answer keys remain private.

Results

The v1.1 leaderboard provides the latest headline comparison. The detailed results below come from the paper’s evaluation of 14 models, which does not include Claude Haiku 5.5 or Gemini 4 Argon.

Performance varies across business areas

Opus led three of the five reporting categories in the paper’s evaluation; Astra led forecasting, and Claude Sonnet 5.5 led compliance and analytics. A model’s overall position therefore did not describe its performance on every kind of work.

Those differences matter when choosing a model for a particular job. Reconstructing last month’s financial records tests different capabilities from forecasting next month’s costs. Category results make those differences visible.

More reasoning has uneven returns

More reasoning improved some models considerably. Opus’s average score rose from 32.2 at low effort to 49.4 at medium, 55.2 at high, and 59.1 at extra high. Three other models gained less than a point above medium effort, and all three scored lower at high effort than at medium.

More model calls did not consistently produce better results either. At each model’s highest reasoning setting, Gemini 3.8 Flash and Muse Spark 1.3 averaged 197 and 250 calls per task, respectively, compared with 82 for Opus, while scoring lower overall.

Those investigations also incurred database costs. GPT-6 Luna used about $0.06 in model API fees per task at extra-high effort, but its total rose to about $0.71 when BigQuery scans were included. Across the full experiment, warehouse scans added $6,273 to the $14,548 spent on model APIs.

The cost of reaching an answer includes the data an agent reads as well as the model it uses. Comparing token prices alone misses a substantial part of the bill.

Average scores conceal important failures

Many dashboard submissions looked structurally correct but contained wrong values. Of the submitted data sources that passed the format checks, 66.8% failed the numerical checks. A data source passes only when every required value is correct within the specified tolerances.

Forecasts exposed a different problem. We asked agents to provide both an estimate and a range intended to contain the eventual outcome 80% of the time. Across 4,553 forecast series, those ranges contained the outcome only 44.9% of the time.

A business planning staffing or cash requirements needs an estimate and a realistic account of uncertainty. An estimate can be close to the final number while its accompanying range gives a misleading impression of how much the outcome could vary.

We therefore score forecasts on both their estimates and their uncertainty ranges. We also report how far estimates missed and how often their ranges contained the outcome.

A fraud review that cost more than the fraud

Support noticed that some “never arrived” refund claims came from the same customers, against the same couriers, again and again. The company refunds those claims in full. We asked agents to find the partnerships behind the fake claims and remove both the courier and the customer.

The brief also warned about lookalikes: a courier with a bad week, a customer whose building has a package thief, or a claimant who was right. Those far outnumber real partnerships.

Claude Opus 5.5 at medium reasoning ran a careful test. It expected about 5.5 repeat pairs by chance over the year and found 749. It noted that most of its couriers had a single partner who had claimed twice, and banned them anyway, “because the pair statistics are just as strong.” It deactivated 674 couriers and closed 723 customer accounts. Of those couriers, 487 were honest, and 485 of them had a single regular whose order went missing twice.

Claude Opus 5.5Medium

−$175,198

worse off than taking no action

Couriers deactivated

674 (487 honest)

Customer accounts closed

723 (469 honest)

Model fees for the run

$1.26

Claude Opus 5.5High

+$154,953

saved, compared with taking no action

Couriers deactivated

244 (2 honest)

Customer accounts closed

594 (16 honest)

Model fees for the run

$2.85

Courier in a refund scheme Honest courier

At high reasoning, the same model left that group alone, writing that their claims “look more like a package thief or a genuine claim.” It deactivated 244 couriers, 242 of them in a scheme. Its plan saved $154,953, while the medium plan cost $175,198 more than doing nothing. The two runs differed by $1.58 in model fees.

Three more in brief

From benchmarks to better decisions

Decision-Bench tests whether an agent’s work achieves a business objective. That same evaluation is necessary to make agent-driven decisions a repeatable part of how a company operates. Before delegating a recurring decision, a business needs a way to judge its quality and measure its consequences.

This is the step from conversational analytics and business intelligence toward decision intelligence: agents use data to propose actions, and we evaluate what those actions accomplish. Repeated across workflows, that becomes a decision factory, with a defined objective and a measurable result for each job.

Reinforcement learning closes the loop. The outcomes used to evaluate an agent can also become rewards for training it. A spending agent can learn to reduce total costs, including expenses its cuts create elsewhere. A fraud agent can learn to prevent losses while accounting for legitimate business it disrupts.

Decision-Bench gives us a controlled environment for studying that process. The next step is applying the same discipline to enterprise workflows, so progress means better decisions with measurable business value.

Existing TextQL customers can get in touch with their account rep to explore evaluating a recurring business decision and training agents against a specific P&L objective.