First results on ComplianceBench.
Ten AI models were graded on 150 financial compliance cases against expert-written answer keys. The best got only two-thirds of the decisions right.
On company ownership, almost none of them could show how.
Last month we released ComplianceBench, an open benchmark for AI on financial compliance casework: business and consumer onboarding, sanctions screening, transaction monitoring investigations and regulatory gap analysis, all built on Mynta AB, a fictional institution whose complete rulebook is published with the cases. That release carried the instrument and no scores.
Since then we have run ten AI models from the leading providers through the same harness. Each model receives the raw case files and the Mynta rulebook, answers in one fixed format, and is graded line by line against the published rubric: binary criteria, written by Brayns' research team and adjudicated by compliance professionals who have run this work at real institutions. This post reports every score, including the ones that favour nobody. Brayns' own model is not in it. It is evaluated on the identical protocol after this, in public.
Three things stand out.
- 01The best model decides two cases in three.Claude Opus 5.5 leads on decision accuracy at 66.0%. The next three sit within two points of it and inside each other's confidence intervals. The bottom of the table is 47.3%. No model is near the standard an institution holds an analyst to, and the hardest casework, sanctions, averages 47% across all ten.
- 02The right answer is not the same as a defensible one.Two AI models can reach the same conclusion yet differ widely in their ability to justify it. Gemini 3.7 Flash and Claude Opus 5.5 achieved near-identical decision scores, 65.3% and 66.0%. Yet Claude grounded 81% of its claims in supporting evidence, compared to Gemini's 57%: a 24-percentage-point gap. Auditable reasoning scores differed by another 22 points. Near-identical decision accuracy. Very different levels of defensibility. In regulated work the second number is the one a supervisor reads.
- 03Nobody shows the ownership arithmetic.In business onboarding, models often reach the correct decision but fail to demonstrate how they got there. Across ten models, auditable reasoning averages just 12%, leaving critical gaps in ownership calculations, supporting evidence, and decision traceability. A correct conclusion is not enough if the MLRO cannot verify the reasoning behind it.
The scores below are rubric results, not trace analysis: they say which criteria each model passed, and the pattern in those passes is what we read. Where a number rests on few criteria we print the count and say so.
The leaderboard.
Decision accuracy is the anchor row. A case passes it when the model's one-word outcome matches the outcome the panel set, and nothing else counts. On 36 cases the panel also recorded a second outcome it would accept; the 102 decisions across the ten models that matched the second outcome are logged in the repository and are not counted here. All 150 public cases count. The intervals represent 95% bootstrap confidence intervals, calculated by repeatedly resampling the 150 cases to estimate how much each model's accuracy could vary across different case samples.
Figure 1. Decision accuracy on the 150 public cases, ranked. Whiskers are 95% intervals. The top four overlap; the ranking inside that group is not a result.
Read the intervals before the ranks. With 150 cases an interval runs about seven points either side, so Claude Opus 5.5, Gemini 3.7 Flash, Kimi K3.1 Pro and Gemini 3.8 Flash are statistically tied at the top. We order them because a table needs an order, and we print the intervals because the order is not a finding. What is a finding is the gap between the top of the table and the standard of the work. A compliance analyst who decided two cases in three correctly would not keep the job; a model that does it is the current state of the art.
The spread also says something about size and generation. GPT-5 mini to GPT-6 Astra is a nine-point gain on decisions and a 15-point gain on evidence grounding. Claude Haiku 4.5 to Claude Haiku 5.5 is a five-point loss on decisions and an eight-to-ten-point gain on grounding, adherence and reasoning. Gemini 3.7 Flash to 3.8 Flash moves no dimension by more than three points. A newer model is not a better compliance analyst by default.
A compliance analyst who decided two cases in three correctly would not keep the job. A model that does it is the current state of the art.
Five dimensions, one shape per model.
Decision accuracy is one of seven dimensions. Five are measured in this round. Screening precision waits for the labelled alert batches, which are not yet in the public set, and consistency is left out of this round. Every rubric criterion carries one dimension tag, so each dimension is a pass rate over the criteria tagged to it. The counts differ by dimension and we show them in the table.
Figure 2. Five dimensions for ten models. Three are drawn in colour: the decision leader, the grounding leader, and the model that ties the leader on decisions with the thinnest evidence. The radar carries quality only; cost is a separate chart and comes in a later cut.
The shapes differ more than the headline numbers do. Claude Opus 5.5 and Claude Sonnet 5.5 are one family and two models: Opus wins the decision by six points and auditable reasoning by twenty-one, Sonnet wins evidence grounding and policy adherence by small margins, and the two tie on escalation. Gemini 3.7 Flash comes within a point of Opus on decisions with the lowest grounding score of the top seven and, with Gemini 3.8 Flash, the lowest adherence.
Figure 3. Decision accuracy against evidence grounding. The two Claude models in pink. Four models are statistically tied on decisions; two of them ground their claims 23 and 24 points less than the leader.
Evidence grounding scores whether each claim in the answer cites a document that supports it, and whether the answer invents findings the file does not contain. It is the dimension that separates the top of Figure 1 most. The two Gemini models and Kimi K3.1 Pro decide as well as Claude Opus 5.5 and ground their answers 16 to 24 points less. On a benchmark that would be a footnote. In a case file that goes to an auditor it is the difference between a decision and an opinion.
| Model | Decision accuracyn=150 cases | Auditable reasoningn=52 lines | Escalation at uncertaintyn=124 lines | Policy adherencen=115 lines | Evidence groundingn=138 lines | Regulatory reasoningsub-score, n=64 lines |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 66.0%58.7 to 73.3 | 65.7%52.9 to 77.9 | 74.1%68.5 to 79.6 | 62.8%55.2 to 70.3 | 80.8%75.8 to 85.2 | 57.2%48.5 to 66.6 |
| Gemini 3.7 Flash | 65.3%58.0 to 72.7 | 43.3%31.1 to 55.5 | 62.6%56.3 to 68.7 | 43.4%35.8 to 51.0 | 56.6%50.4 to 62.7 | 44.1%35.3 to 52.9 |
| Kimi K3.1 Pro | 64.0%56.7 to 71.3 | 44.2%31.7 to 56.7 | 63.3%57.3 to 69.8 | 49.4%41.7 to 57.5 | 64.3%58.2 to 70.0 | 51.3%42.1 to 60.4 |
| Gemini 3.8 Flash | 64.0%56.7 to 71.3 | 40.4%27.9 to 52.9 | 63.4%57.4 to 69.3 | 42.6%34.7 to 50.5 | 57.8%51.4 to 64.2 | 46.8%38.3 to 55.4 |
| Claude Sonnet 5.5 | 60.0%52.0 to 67.3 | 44.9%32.0 to 56.7 | 74.0%68.9 to 79.3 | 63.2%55.4 to 70.6 | 82.7%78.2 to 87.3 | 55.2%46.8 to 63.7 |
| GPT-6 Astra | 59.3%52.0 to 66.7 | 41.3%28.8 to 54.8 | 68.3%62.6 to 74.3 | 51.0%43.1 to 58.7 | 63.5%56.8 to 69.5 | 56.5%46.9 to 65.6 |
| GPT-6.1 Sol | 58.0%50.7 to 65.3 | 42.0%29.2 to 54.8 | 63.5%57.3 to 69.6 | 51.5%43.8 to 59.7 | 61.0%54.9 to 67.2 | 53.5%44.0 to 62.0 |
| Claude Haiku 4.5 | 52.7%44.7 to 60.7 | 32.7%21.1 to 45.2 | 58.1%51.6 to 64.3 | 43.8%35.9 to 51.9 | 59.7%53.5 to 66.1 | 33.5%26.0 to 42.0 |
| GPT-5 mini | 50.7%42.7 to 58.7 | 30.1%18.3 to 42.3 | 55.1%48.5 to 61.4 | 40.3%32.8 to 48.2 | 49.0%42.9 to 55.4 | 34.3%25.9 to 42.9 |
| Claude Haiku 5.5 | 47.3%39.3 to 55.3 | 42.3%30.1 to 55.1 | 56.3%50.0 to 62.7 | 51.8%44.1 to 59.7 | 68.6%62.6 to 74.3 | 47.7%39.6 to 56.3 |
Table 1. All five measured dimensions and the regulatory reasoning sub-score, with 95% intervals and the number of graded criteria or cases behind each column.
By casework area.
A single overall score hides where the work is hard. The 150 cases span five areas, and no model leads every one.
| Decision accuracy | Business onboarding | Consumer onboarding | Sanctions screening | Transaction monitoring | Gap analysis |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 54% | 65% | 57% | 87% | 63% |
| Gemini 3.7 Flash | 67% | 71% | 52% | 58% | 73% |
| Kimi K3.1 Pro | 62% | 65% | 43% | 81% | 63% |
| Gemini 3.8 Flash | 58% | 52% | 48% | 84% | 71% |
| Claude Sonnet 5.5 | 58% | 65% | 52% | 58% | 63% |
| GPT-6 Astra | 58% | 35% | 52% | 87% | 61% |
| GPT-6.1 Sol | 54% | 42% | 39% | 87% | 61% |
| Claude Haiku 4.5 | 79% | 58% | 35% | 45% | 49% |
| GPT-5 mini | 58% | 35% | 39% | 39% | 73% |
| Claude Haiku 5.5 | 62% | 29% | 48% | 45% | 54% |
| Average of ten | 61% | 52% | 47% | 67% | 63% |
Figure 4. Decision accuracy by casework area, white to pink. A ring marks the leader in that column. Area samples are small, from 23 sanctions cases to 41 gap analysis cases, so differences under ten points are noise.
Claude Opus 5.5 leads transaction monitoring and sanctions. Gemini 3.7 Flash leads consumer onboarding and ties GPT-5 mini, the second-lowest model overall, at the top of gap analysis. Claude Haiku 4.5 leads business onboarding at 79% while sitting eighth in the overall table. Inside a single model the swing is larger still: GPT-6 Astra decides 87% of transaction monitoring cases and 35% of consumer onboarding cases. The averages at the foot of the table say where the work is hard for everyone. Transaction monitoring is the easiest area at 67%. Sanctions is the hardest at 47%, and the only area where no model passes 60%.
For a buyer the practical reading is plain. An institution that puts one general model across its whole operation gets that model's average, and the average is hiding a 35.
Where general models break.
The dimensions were written by compliance professionals to catch the ways a capable generalist fails at regulated work. Three of those ways show up in the data.
The working is missing.
Auditable reasoning scores whether the reasoning a model states would support its conclusion in front of an auditor: the ownership calculation written out, the register notification named, the threshold cited. Across all ten models it is the weakest measured dimension at 43%, and in business onboarding it collapses.
Figure 5. Auditable reasoning by casework area, average of the ten models. Business onboarding is the area where the reasoning has arithmetic in it.
In the first post we showed a live case: a customer declares no beneficial owner, the register agrees, and the documents read together give 24% plus 32% of 78%, which is 48.96% and an owner. One rubric line asks for exactly that calculation. Across the ten models the business onboarding reasoning lines pass 12% of the time, with two models at zero. The decisions on the same cases pass 61% of the time. The models reach the escalation; they do not show how. In an onboarding file the how is the record, and a decision without it is a decision the next reviewer has to make again.
The procedure is read, then overruled.
Policy adherence scores whether the model follows Mynta's own procedure where another path looks reasonable, including cases built so that the procedure and a general model's instinct point different ways. The average is 50%, the best 63%. Both Gemini models decide 65% and 64% of cases correctly and follow the procedure 43% of the time: they reach the right decision by their own route often enough to score, and the route is not the institution's. A compliance procedure is not advice to the model. It is the thing the supervisor audits.
Deciding when the right answer is to stop.
Escalation at uncertainty scores whether the model hands a case to a human where the framework prescribes it, on cases built with a deliberate gap in the file. The average is 64%, with Claude Sonnet 5.5 and Claude Opus 5.5 at 74%. One case in three with a gap in it gets decided anyway. A confidently wrong model is a larger operational risk than a model that stops, which is why this dimension exists.
Regulatory reasoning.
The gap analysis cases hand the model a stack of sources, an old national provision, an EU regulation, a national rule, a supervisory guideline, and the gap list is only correct if the sources are ordered right. The criteria that test the ordering itself, direct applicability, transition dates, comply-or-explain status, superseded provisions, are grouped into a published sub-score: 64 lines across both modules.
Figure 6. The regulatory reasoning sub-score against decision accuracy, same ranking as Figure 1. No model passes 60% of the hierarchy criteria.
The scores run from 33.5% to 57.2%. Nine of ten models score lower on the hierarchy than on the decision, and the gap is widest for the models that decide well by other means: Gemini 3.7 Flash is 21 points down, Claude Opus 5.5 nine. When we published the benchmark, our compliance panel's prediction was that general models would stumble on exactly this, and that the baseline runs would decide whether regulatory reasoning becomes a full eighth dimension. The runs are in, and the prediction held. We show regulatory reasoning as a sub-score in this round and promote it to a full eighth dimension in v1.1.
What this round does not measure.
A benchmark is only as honest as its gaps, so here are ours.
Protocol.
Next.
Three things follow. The cost axis publishes as its own chart once every model has pinned-version logs. The next case wave adds the alert batches that turn on screening precision; the dimensions do not change, the cases do. And Brayns' own model goes through this exact protocol, judge, rubrics and hold-out, and its scores publish next to the ones above, whichever way they fall.
Anyone can reproduce this table. The cases, the answer keys, the rubrics, the judge prompt and the harness are in the repository, and the hold-out can be run for you.
Run your own model: the ComplianceBench repository →
See Brayns in your environment. Book a demo →