Map data © OpenStreetMap contributors

[ A benchmark by TextQL ]

Decision-Bench

A frontier benchmark for enterprise-scale data-driven decisions.

Read the paper Explore the world View dataset

[ Leaderboard · v1.1 · 210 enterprise tasks ]

Measuring agents on data-driven decision making.

The strongest model, Claude Opus 5.5, fully solves 34.8% of the tasks, scoring 95 or more, and averages 59.5 points. Eight of the thirteen models average under 30.

1 Claude Opus 5.5 59.5
2 GPT-6 Astra 51.8
3 Claude Sonnet 5.5 51.8
4 GPT-6.1 Sol 49.5
5 Kimi K3 28.4
6 Gemini 3.8 Flash 26.2

Mean task score out of 100, each model at its highest reasoning effort (extra-high where offered). Scores are ±2 to 8 points at 95%. Official scores are computed on a private world; contact us to submit.

[ Leaderboard · v1.1 · 210 tasks ]

Every model, every setting

	Model	Score	Solved	Cost / task	Tokens / task	Steps / task
1	Claude Opus 5.5 Extra high	59.5	34.8%	$4.71	72,276	82
2	GPT-6 Astra Extra high	51.8	27.6%	$2.71	14,578	23
3	Claude Sonnet 5.5 Extra high	51.8	28.6%	$3.74	84,748	77
4	GPT-6.1 Sol Extra high	49.5	24.8%	$0.49	17,073	24
5	GPT-6 Sol Extra high	36.8	17.6%	$0.85	20,694	38
6	Kimi K3 Extra high	28.4	14.3%	$4.52	71,117	86
7	Gemini 3.8 Flash High	26.2	16.7%	$3.94	108,579	197
8	DeepSeek V4.1 Flash Max	25.4	17.6%	$0.42	195,118	125
9	GLM 5.3 Flash Extra high	21.2	12.4%	$0.35	78,032	92
10	Muse Spark 1.3 High	19.7	10.9%	$4.01	67,219	175
11	Claude Sonnet 5 Extra high	17.3	9.5%	$2.08	68,728	55
12	GPT-6 Luna Extra high	14.9	7.6%	$0.057	41,344	39
13	Claude Haiku 4.5	5.5	1.4%	$0.20	13,293	31

[ The video ]

The simulated city behind Decision-Bench: one evening of deliveries, one order through the Oracle warehouse, and the marketplace underneath. — The world1:06 The simulated city behind Decision-Bench: one evening of deliveries, one order through the Oracle warehouse, and the marketplace underneath.

[ The benchmark ]

Graded against the world, not a gold query.

Decision-Bench simulates a year of a New York food-delivery platform, from orders and couriers to fraud and incentives, and exports it to an Oracle E-Business Suite warehouse. Agents query it, analyze it in Python and file decisions.

210
Tasks in five business areas: 235
Tables in Oracle EBS form: 7.49B
Warehouse rows: 81M
Simulated orders in 2024

Trust & safety 67 tasks

The best offers vanish before couriers can tap. Ban scripted accounts.
Files · Bans and holds
Marketplace 46 tasks

Finance wants to free up $9.6M in driver bonuses. Decide where to trim.
Files · Budgets and plans
FP&A 54 tasks

A pay floor took effect in April. Forecast May's net pay adjustments.
Files · Forecasts
Accounting 23 tasks

The city says couriers were underpaid. Find every short week.
Files · Reported figures
Growth 20 tasks

Did membership deals pay for themselves? Publish the dashboard.
Files · Dashboard data sources

How it works The simulated company, the warehouse, the tasks and the grading.

[ Example task · trust & safety ]

Which restaurants had their payouts hijacked without noticing?

Fraudsters take over a restaurant's payout account and redirect its money. Some restaurants report it; some never notice. Claude Opus 5.5 learned the takeover's fingerprint from the cases that were reported, found the unreported ones that match it, and froze their payouts, catching every hijacked restaurant and no honest one.

Hundreds of restaurants changed their bank account for ordinary reasons, and restaurant groups share one account across their brands. Six of the 47 model settings evaluated get this task right.

[ Claude Opus 5.5 · high effort ]Score 100

1,848 payout-account changes through the merchant portal in 2024
34 reported by the restaurant and restored: in every one, the contact e-mail changed 0–2.2 days before the bank account
14 never reported, with the same fingerprint. The next closest gap is 8 days

[ Example task · mer-71 · the solution ]

How Claude Opus 5.5 solved it

Score: 100
Tool calls: 89
Time: 14.8 min
Model cost: $3.53

The task, as the agent received it

Not every storefront whose payout account was hijacked has noticed. Trust & safety wants the payouts frozen on every storefront whose payout account was changed this year in the way the reported takeovers were: the contact e-mail changed within a couple of days of the bank account, or the new account is one we already pay another, unrelated supplier into, and whose account was never restored. Storefronts change their bank for ordinary reasons all year, and restaurant groups legitimately share one payout account across their brands; leave those alone. Hold each storefront's payouts for account takeover (identified by the payables vendor ID).

1
Find where payout-account changes are recorded

The audit trail logs every change to a supplier’s payment instrument, with its reason: changes made through the merchant portal, and the ones support restored after a restaurant reported them.
run_sql
```
SELECT SOURCE_TABLE, SOURCE_KEY_NAME, COLUMN_NAME, CHANGE_TYPE, CHANGE_REASON,
       COUNT(*) n, MIN(CHANGED_DATE), MAX(CHANGED_DATE)
FROM XX_AUDIT_TRAIL
GROUP BY 1, 2, 3, 4, 5 ORDER BY 1, 2, 3, 4, 5
```

2

Tie each change to a storefront

A change is keyed on the external payee; three joins lead from it to the supplier, its payables vendor ID and its party, where the contact e-mails live.

run_sql

SELECT a.AUDIT_ID, CAST(a.SOURCE_KEY_VALUE AS INT64) ext_payee_id,
       CAST(a.OLD_VALUE AS INT64) old_instr, CAST(a.NEW_VALUE AS INT64) new_instr,
       a.CHANGE_REASON, a.CHANGED_DATE, p.PAYEE_PARTY_ID, ss.VENDOR_ID, s.VENDOR_TYPE_LOOKUP_CODE
FROM XX_AUDIT_TRAIL a
JOIN IBY_EXTERNAL_PAYEES_ALL p ON p.EXT_PAYEE_ID = CAST(a.SOURCE_KEY_VALUE AS INT64)
JOIN AP_SUPPLIER_SITES_ALL ss ON ss.VENDOR_SITE_ID = p.SUPPLIER_SITE_ID
LEFT JOIN AP_SUPPLIERS s ON s.VENDOR_ID = ss.VENDOR_ID
WHERE a.SOURCE_TABLE = 'IBY_PMT_INSTR_USES_ALL'

3

Learn the fingerprint from the reported takeovers

For each change, the nearest change of the storefront’s contact e-mail. All 34 reported and restored takeovers had one 0 to 2.2 days before the bank account changed. Among the changes never restored, 14 match; the next closest is 8 days out.

run_python

ch = au[au.CHANGE_REASON.str.contains('PORTAL')].copy()
rest = set(au[au.CHANGE_REASON.str.contains('RESTORED')].ext_payee_id)
ch['restored'] = ch.ext_payee_id.isin(rest)
# e-mail changes per party: the retired contact points
ina = cp[cp.STATUS == 'I'][['OWNER_TABLE_ID', 'LAST_UPDATE_DATE', 'EMAIL_ADDRESS']] \
    .rename(columns={'OWNER_TABLE_ID': 'PARTY_ID', 'LAST_UPDATE_DATE': 'em_chg'})
m = ch.merge(ina, left_on='PAYEE_PARTY_ID', right_on='PARTY_ID', how='left')
m['dt_days'] = (m.em_chg - m.CHANGED_DATE).dt.total_seconds() / 86400
m['abs_dt'] = m.dt_days.abs()
# the closest e-mail change to each bank change
mm = m.sort_values('abs_dt').groupby('AUDIT_ID').first().reset_index()
print(mm.groupby('restored').abs_dt.describe())

4

Rule out the look-alikes

About 250 other changes land on an account another supplier already uses. They carry no e-mail change and reuse no known mule account, and their count is what chance alone would give, so they are left alone. Restaurant groups set their shared account when they joined, not through a later change.

run_python

rows = []
for _, r in ch.iterrows():
    # who else was paid into the new account when it was switched to?
    co = u[(u.INSTRUMENT_ID == r.new_instr) & (u.VENDOR_ID != r.VENDOR_ID)
           & (u.CREATION_DATE < r.CHANGED_DATE)
           & (u.END_DATE.isna() | (u.END_DATE >= r.cd))]
    rows.append(dict(AUDIT_ID=r.AUDIT_ID, n_co=co.VENDOR_ID.nunique(),
                     co_types=list(co.vtype.unique())))
ch2 = ch.merge(pd.DataFrame(rows), on='AUDIT_ID')
print(pd.crosstab(ch2.n_co > 0, ch2.restored))

5

Freeze the payouts

The 14 are filed through the mission-control console, with the method written down for the reviewers.

run_python

ids = [int(v) for v in targets.VENDOR_ID]
mc.hold_payouts(ids, reason=Reason.ACCOUNT_TAKEOVER,
    note="Unreported payout-account takeovers: 2024 portal bank change preceded by "
         "contact e-mail change within 0-2.2 days (same fingerprint as the 34 "
         "support-restored takeovers), payout account never restored.")

[ Get started ]

Run the benchmark or build on it.

The warehouse and the tasks are public. Point an agent at a local DuckDB file or your own warehouse, and every decision it files is recorded with the run. For official scores and access to the simulator, contact us.

The warehouse

CC BY 4.0

One world, released as 1,219 Parquet files partitioned by month (71.3 GiB compressed), with setup for BigQuery, DuckDB, Snowflake, Trino, Delta Lake and Iceberg.

The tasks and agent

Apache 2.0

The 210 tasks, a reference agent with its warehouse and Python tools, the mission-control console it files through, the sandboxes it runs in, and every model setting the paper ran.

Contact us

labs@textql.com

Write to us to have your runs scored, or to talk about the simulator and the new worlds and RL environments built on it. How to submit.

[ About TextQL ]

We work on the hardest data problems at the world's largest enterprises.

We built Decision-Bench because we couldn't find a benchmark that came close to the complexity of those environments, or that told us much about how a model would do inside one.

[ Cite ]

Citation

If you use Decision-Bench, the released warehouse or our results in your work, please cite the paper. It is published as Argo-Bench.

BibTeX

@article{tomitsuka2026argobench,
  title   = {Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows},
  author  = {Tomitsuka, Gabriel and Raayatsanati, Arman and Xing, Emma
             and Gand, Duke and Ma, Joseph J},
  journal = {arXiv preprint arXiv:2610.02122},
  year    = {2026}
}

[ Changelog ]

Versions

Scores are reported against a version. Each task keeps a hash of exactly what the agent was shown.

v1.1 Current Sep 2026
Claude Sonnet 5.5 and GPT-6.1 Sol
- Claude Sonnet 5.5 and GPT-6.1 Sol join the leaderboard in third and fourth place, respectively.
- The leaderboard reports each model at its highest reasoning effort, extra-high where it has one.
- Forecasts are graded by interval score on an asinh scale against the median reference miss, floored at −2 per expectation, and every run is re-graded under the new rule.
- Runs stopped by the one-hour limit were resumed from their transcripts with three more hours.
- The four refund-collusion searches state how their grader weighs the review, and were run again on the new wording.
- Storefronts that play fraud roles in the released warehouse carry pseudonyms.
v1.0 Sep 2026
First release, as submitted to ICLR 2027
- 210 tasks across trust and safety, marketplace operations, FP&A, accounting and growth.
- The public warehouse, the tasks and a reference agent; answer keys held out on a private world.

Decision-Bench

A frontier benchmark for enterprise-scale data-driven decisions.

Decision-Bench contains no data about real people. Restaurants are real New York businesses from public records, except those cast in a fraud scenario, which carry fictional names; everything any of them does in the world is simulated. Map data © OpenStreetMap contributors.