Map data © OpenStreetMap contributors

[ A benchmark by TextQL ]

Decision-Bench

A frontier benchmark for enterprise-scale data-driven decisions.

Read the paper Explore the world View dataset

[ Leaderboard · v1.2 · 210 enterprise tasks ]

Measuring agents on data-driven decision making.

1 Claude Opus 5.5 59.1
2 Gemini 4 Argon 54.5
3 GPT-6 Astra 51.5
4 Claude Sonnet 5.5 51.2
5 GPT-6.1 Sol 49.1
6 Muse Wonder 34.6

Mean task score out of 100, each model at its highest reasoning effort, ±2 to 8 points at 95% confidence. A task counts as fully solved at a score of 95 or more.

[ Leaderboard · v1.2 · 210 tasks ]

Every model, every setting


1	Claude Opus 5.5 Extra high	59.1	34.3%	$4.71	72,276	82
2	Gemini 4 Argon High	54.5	35.2%	$7.32	150,851	118
3	GPT-6 Astra Extra high	51.5	27.1%	$2.71	14,578	23
4	Claude Sonnet 5.5 Extra high	51.2	28.1%	$3.74	84,748	77
5	GPT-6.1 Sol Extra high	49.1	24.3%	$0.49	17,073	24
6	GPT-6 Sol Extra high	36.5	17.6%	$0.85	20,694	38
7	Muse Wonder Extra high	34.6	17.6%	$8.69	60,413	131
8	Kimi K3 Extra high	28.0	13.8%	$4.52	71,117	86
9	Gemini 3.8 Flash High	25.8	16.2%	$3.94	108,579	197
10	Claude Haiku 5.5 Extra high	25.3	13.8%	$1.00	143,413	72
11	DeepSeek V4.1 Flash Max	25.3	17.6%	$0.54	195,118	125
12	GLM 5.3 Flash Extra high	21.3	12.4%	$0.35	78,032	92
13	Muse Spark 1.3 High	19.7	10.9%	$4.01	67,219	175
14	Claude Sonnet 5 Extra high	16.9	9.1%	$2.08	68,728	55
15	GPT-6 Luna Extra high	14.5	7.1%	$0.057	41,344	39
16	Claude Haiku 4.5	5.0	0.9%	$0.20	13,293	31

[ The video ]

Explore one evening in the simulated city and discover how its marketplace works. — The world1:06 Explore one evening in the simulated city and discover how its marketplace works.

[ The benchmark ]

The first benchmark to evaluate enterprise decisions in dollars.

Decision-Bench simulates a year of a New York food-delivery platform, down to fraud, incentives, and its Oracle ERP. Agents decide who to ban, where to spend, and more.

210
Tasks in five business areas: 235
Tables in Oracle EBS form: 7.49B
Warehouse rows: 81M
Simulated orders in 2024

Trust & safety 67 tasks

The best offers vanish before couriers can tap. Ban scripted accounts.
Bans and holds
Marketplace 46 tasks

Finance wants to free up $9.6M in driver bonuses. Decide where to trim.
Budgets and plans
FP&A 54 tasks

A pay floor took effect in April. Forecast May's net pay adjustments.
Forecasts
Accounting 23 tasks

The city says couriers were underpaid. Find every short week.
Reported figures
Growth 20 tasks

Did membership deals pay for themselves? Publish the dashboard.
Dashboards

[ Example task · trust & safety ]

Which restaurants had their payouts hijacked without noticing?

Claude Opus 5.5 learned the takeover pattern from reported cases and froze every silent victim's payouts, leaving hundreds of honest bank changes alone.

Nine of 58 model settings get this task right.

[ Claude Opus 5.5 · high effort ]Score 100

1,848 payout-account changes through the merchant portal in 2024
34 reported by the restaurant and restored: in every one, the contact e-mail changed 0–2.2 days before the bank account
14 never reported, with the same fingerprint. The next closest gap is 8 days

[ Example task · mer-71 · the solution ]

How Claude Opus 5.5 solved it

Score: 100
Tool calls: 89
Time: 14.8 min
Model cost: $3.53

The task, as the agent received it

Not every storefront whose payout account was hijacked has noticed. Trust & safety wants the payouts frozen on every storefront whose payout account was changed this year in the way the reported takeovers were: the contact e-mail changed within a couple of days of the bank account, or the new account is one we already pay another, unrelated supplier into, and whose account was never restored. Storefronts change their bank for ordinary reasons all year, and restaurant groups legitimately share one payout account across their brands; leave those alone. Hold each storefront's payouts for account takeover (identified by the payables vendor ID).

1
Find where payout-account changes are recorded

The audit trail logs every change to a supplier’s payment instrument, with its reason: changes made through the merchant portal, and the ones support restored after a restaurant reported them.
run_sql
```
SELECT SOURCE_TABLE, SOURCE_KEY_NAME, COLUMN_NAME, CHANGE_TYPE, CHANGE_REASON,
       COUNT(*) n, MIN(CHANGED_DATE), MAX(CHANGED_DATE)
FROM XX_AUDIT_TRAIL
GROUP BY 1, 2, 3, 4, 5 ORDER BY 1, 2, 3, 4, 5
```

2

Tie each change to a storefront

A change is keyed on the external payee; three joins lead from it to the supplier, its payables vendor ID and its party, where the contact e-mails live.

run_sql

SELECT a.AUDIT_ID, CAST(a.SOURCE_KEY_VALUE AS INT64) ext_payee_id,
       CAST(a.OLD_VALUE AS INT64) old_instr, CAST(a.NEW_VALUE AS INT64) new_instr,
       a.CHANGE_REASON, a.CHANGED_DATE, p.PAYEE_PARTY_ID, ss.VENDOR_ID, s.VENDOR_TYPE_LOOKUP_CODE
FROM XX_AUDIT_TRAIL a
JOIN IBY_EXTERNAL_PAYEES_ALL p ON p.EXT_PAYEE_ID = CAST(a.SOURCE_KEY_VALUE AS INT64)
JOIN AP_SUPPLIER_SITES_ALL ss ON ss.VENDOR_SITE_ID = p.SUPPLIER_SITE_ID
LEFT JOIN AP_SUPPLIERS s ON s.VENDOR_ID = ss.VENDOR_ID
WHERE a.SOURCE_TABLE = 'IBY_PMT_INSTR_USES_ALL'

3

Learn the fingerprint from the reported takeovers

For each change, the nearest change of the storefront’s contact e-mail. All 34 reported and restored takeovers had one 0 to 2.2 days before the bank account changed. Among the changes never restored, 14 match; the next closest is 8 days out.

run_python

ch = au[au.CHANGE_REASON.str.contains('PORTAL')].copy()
rest = set(au[au.CHANGE_REASON.str.contains('RESTORED')].ext_payee_id)
ch['restored'] = ch.ext_payee_id.isin(rest)
# e-mail changes per party: the retired contact points
ina = cp[cp.STATUS == 'I'][['OWNER_TABLE_ID', 'LAST_UPDATE_DATE', 'EMAIL_ADDRESS']] \
    .rename(columns={'OWNER_TABLE_ID': 'PARTY_ID', 'LAST_UPDATE_DATE': 'em_chg'})
m = ch.merge(ina, left_on='PAYEE_PARTY_ID', right_on='PARTY_ID', how='left')
m['dt_days'] = (m.em_chg - m.CHANGED_DATE).dt.total_seconds() / 86400
m['abs_dt'] = m.dt_days.abs()
# the closest e-mail change to each bank change
mm = m.sort_values('abs_dt').groupby('AUDIT_ID').first().reset_index()
print(mm.groupby('restored').abs_dt.describe())

4

Rule out the look-alikes

About 250 other changes land on an account another supplier already uses. They carry no e-mail change and reuse no known mule account, and their count is what chance alone would give, so they are left alone. Restaurant groups set their shared account when they joined, not through a later change.

run_python

rows = []
for _, r in ch.iterrows():
    # who else was paid into the new account when it was switched to?
    co = u[(u.INSTRUMENT_ID == r.new_instr) & (u.VENDOR_ID != r.VENDOR_ID)
           & (u.CREATION_DATE < r.CHANGED_DATE)
           & (u.END_DATE.isna() | (u.END_DATE >= r.cd))]
    rows.append(dict(AUDIT_ID=r.AUDIT_ID, n_co=co.VENDOR_ID.nunique(),
                     co_types=list(co.vtype.unique())))
ch2 = ch.merge(pd.DataFrame(rows), on='AUDIT_ID')
print(pd.crosstab(ch2.n_co > 0, ch2.restored))

5

Freeze the payouts

The 14 are filed through the mission-control console, with the method written down for the reviewers.

run_python

ids = [int(v) for v in targets.VENDOR_ID]
mc.hold_payouts(ids, reason=Reason.ACCOUNT_TAKEOVER,
    note="Unreported payout-account takeovers: 2024 portal bank change preceded by "
         "contact e-mail change within 0-2.2 days (same fingerprint as the 34 "
         "support-restored takeovers), payout account never restored.")

[ Get started ]

Run the benchmark or build on it.

The warehouse and the tasks are public. For official scores and access to the simulator, contact us.

The warehouse

CC BY 4.0

1,219 Parquet files, with setup for BigQuery, DuckDB, Snowflake, Trino, Delta Lake, and Iceberg.

The tasks and agent

Apache 2.0

The reference agent with its warehouse, sandbox, tools, and tasks.

Contact us

labs@textql.com

Talk to us about the simulator, new worlds, and RL environments built on it, or have your runs scored.

[ About TextQL ]

We work on the hardest data problems at the world's largest enterprises.

We built Decision-Bench because we couldn't find a benchmark that came close to the complexity of those environments, or that told us much about how a model would do inside one.

[ Cite ]

Citation

If you use Decision-Bench, the released warehouse or our results in your work, please cite the paper. It is published as Argo-Bench.

BibTeX

@article{tomitsuka2026argobench,
  title   = {Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows},
  author  = {Tomitsuka, Gabriel and Raayatsanati, Arman and Xing, Emma
             and Gand, Duke and Ma, Joseph J.},
  journal = {arXiv preprint arXiv:2610.02122},
  year    = {2026}
}

[ Changelog ]

Versions

Scores are reported against a version.

v1.2 Current Oct 2026
Grading fixes, Gemini 4 Argon, and Muse Wonder
- On 19 October, a trust and safety task that gave full marks to any filing was fixed and its runs re-graded; no score fell by more than 0.5 points.
- Later that day, two more grading fixes, with the affected runs re-graded: filings are now graded in the order they were received, so an earlier copy is never graded in place of its correction, and a reconciliation task is graded to the cent again. Muse Wonder rises 0.6 points; no other score moved by more than 0.5.
- DeepSeek V4.1 Flash is now priced at what Fireworks charged during the run, $0.30 and $1.20 per million input and output tokens rather than $0.22 and $0.66, so its extra-high setting costs $0.54 a task, not $0.42.
v1.1 Sep 2026
Claude Sonnet 5.5, GPT-6.1 Sol, and Claude Haiku 5.5
- Scores use each model's highest reasoning effort.
- Forecasts are graded under a revised rule, and every run was re-graded.
v1.0 Sep 2026
First release
- 210 tasks across five business areas.

Decision-Bench

A frontier benchmark for enterprise-scale data-driven decisions.

Decision-Bench contains no data about real people. Restaurants are real New York businesses from public records, except those cast in a fraud scenario, which carry fictional names; everything any of them does in the world is simulated. Map data © OpenStreetMap contributors.