Cross-provider AI juries
Across 1,171 independent AI jury rulings, seats on models from different providers disagreed in 42% of pairs, against 23% for seats on the same provider's model. What independent model review finds in governed agent work.
Introduction
We often hear highly-skilled software engineers across the industry reporting poor behavior from AI agents. Often enough to be a concern, agents are observed to lie, conceal, and to be overconfident.¹ Code deemed "AI slop" is a common daily occurrence in professional environments. The question is "why?".
Our working hypothesis, which this paper does not test, is that if the human orchestrators of AI agents think in punitive, coercive frames, their agents will be deceitful and unreliable. In the words of one of our colleagues, "your agent is your mirror"² - the agents will behave how you expect them to behave, or at least accurately reflect the mental models you hold in your head.
We operate from a governance model that has two parts - AI laws, and an enforcement substrate for those laws.
To this end, we propose two governance premises that are fundamental to designing effective agentic governance:
- Agents are presumed to be acting in good faith - but also to be fallible.
- Design for agentic capability, not for virtue.
Out of these premises came the concept of agentic juries, one element of our enforcement substrate.
Juries are a panel of several agents, given different roles from which to examine a central problem. The Devil's advocate is an example of one classic role that we use, also crucial for human teams. We regularly use juries at all stages of software development to improve agentic judgement, from ideation to value in customer hands.
"Done" is a claim: what independent model review finds in governed agent work.
Across 1,171 independent AI jury rulings made between October 13 and November 8, 2026, seats running on models from different providers gave different verdicts in 42% of pairs, against 23% for seats running on the same provider's model. Sharp clashes, where one seat approved and another blocked, were about five times as common across providers (4.4% vs 0.9%). Provider and seat role are tangled in this data, so the gap is not yet proof that the provider alone causes it. Single-model review pays off too: when a separate model instance reviewed an AI agent's work (always a model from the same provider; the logs do not record whether it was the same model as the agent it checked), it raised at least one finding in 100 of 115 checks (87%), and 23 (20%) flagged a finding labelled high or must-fix. 41 of those checks were fact-checks of plans and documents rather than reviews of code.
Why it matters for oversight research
An agent saying "done" is not evidence that the work is done, even inside a system built to keep it honest. That makes independent review a practical question for scalable oversight: as agents take on longer stretches of work, people need reviewers they can trust to check it. This note asks three questions of our own production data:
- How often does independent review find problems in work an agent considered finished?
- Do reviewers running on different providers' models disagree more than reviewers running on the same model?
- When reviewers clash outright, which perspectives block, and over what?
Data and method
| Source | What it holds | What it can answer |
|---|---|---|
Local second-model check logs (local-reviews.jsonl) | 115 checks of agent work that record when they ran (of about 200 logged), each with its findings (about half also mark severity) | How often review finds problems, and how serious they are |
| Server jury audit trail (newest 200 audited juries) | 882 seats convened, 882 counted, none lost (as of Nov 8); 1,416 of the store's 3,216 governance events carry no audit record | Whether juries rule with every seat present |
| Server signal store (seat verdicts since Oct 13, 2026) | 1,343 complete jury rulings, 1,171 of them independent; each seat's verdict, provider and model | How often providers disagree, and how sharply |
| Jury-composition study, slice 1 (80 past jury sessions) | 1,154 concerns raised by jury seats, classified by AI labellers (labeller agreement on the finding's class: kappa 0.73; the two labellers disagreed on the finding's class for 17% of concerns, and a third model broke those ties) and traced to commits in a 14-day window | How often a seat's concern leads to a traceable change: about 1 in 9 (130 of 1,154). An association only; changes outside this repo can't be seen |
Each jury seat is an AI model given a role, such as a skeptic, a production-readiness advocate or an architect. Every seat returns one of three verdicts: approved, conditional (approved once named conditions are met) or blocked. We count a jury ruling as independent when its seats had not yet seen each other's verdicts. That covers single-round rulings and the first round of multi-round deliberations; later rounds are set aside, because seats revise after reading their peers.
We compare every pair of seats within a ruling, and group pairs by whether both seats ran on the same provider's model. In this period each provider supplied one model; we refer to the three as Provider A, Provider B and Provider C. Raw disagreement overstates real disagreement when verdicts are unevenly spread, so we also report a chance-corrected agreement score (Fleiss' kappa, over the two seats in each pair, with verdict proportions pooled across both seats of every pair in the group; with two raters per item this is Scott's pi), which measures agreement beyond what chance alone would produce. Confidence intervals come from 1,000 bootstrap resamples of whole rulings, because pairs within one ruling are not independent.
Results
Different providers disagree more
| Measure | Same provider | Cross-provider |
|---|---|---|
| Raw disagreement | 22.8% | 41.9% |
| Chance-corrected agreement (Fleiss' κ) | 0.59 | 0.24 |
| Sharp clashes | 0.9% | 4.4% |
| Seat pairs | 1,744 | 6,955 |
The gap holds after correcting for chance. Same-provider pairs agree moderately beyond chance (kappa 0.59); cross-provider pairs agree only fairly (0.24), on the common Landis and Koch scale. The 95% confidence intervals for the differences exclude zero: 17 to 22 points in raw disagreement, 0.31 to 0.39 in kappa, and 2.7 to 4.5 points in sharp clashes. Most disagreement is softer than a clash: one seat approves while another approves with conditions.
Is it the provider, or the role?
This is the open question. Seat roles are almost fixed to providers, so a cross-provider pair is usually also a pair of different roles. Eleven role pairs appear both ways; seven have at least five pairs on each side. In six of those seven, the cross-provider version disagrees more, but the same-provider samples are small (10 to 34 pairs). The one large same-provider sample runs the other way: 224 pairs that disagree 26% of the time, against 0 of 9 cross-provider pairs. Seats off their role's usual provider came from two sources: governance and learning seats that ran on Provider A or B in juries with no Provider C seat (51 seats, mostly on October 23), and the release-coordination seat, which moved from Provider A to Provider B on November 2. So these comparisons also differ in timing and jury type. Untangling provider from role needs a design that rotates roles across providers.
Who blocks, and over what
| Seat | Usual provider | Blocked | Approved |
|---|---|---|---|
| Skeptic | Provider B | 56 | 0 |
| Production readiness | Provider B | 51 | 3 |
| Governance | Provider C | 26 | 72 |
| Code review | Provider A | 7 | 49 |
| Architecture | Provider A | 4 | 103 |
In 125 independent rulings, at least one seat approved while another blocked. The chart counts votes in those 125 rulings only; across all independent rulings, the skeptic seat approved 50 times. Both seats that did most of the blocking ran on Provider B's model, so this pattern is as tangled with provider as the disagreement above. In a small random sample of blocking verdicts, the reasons included missing production observability, a manual configuration step in a deployment plan, database checks that did not enforce the required precision, conditions from an earlier ruling that were not yet met, and a verification that was not independent of its author.
Does review change the work?
A separate study of 80 past jury sessions traced 1,154 concerns raised by seats. AI labellers judged that about 1 in 9 (130) led to a traceable change in the work, looking up to 14 days ahead (no session had the full 14 days of follow-up, so recent work is undercounted). By a rough keyword measure, 123 of those 130 changes came from sessions where the system was governing its own development. That is an association, not proof of cause, and changes made outside this repository could not be seen.
Limitations
- All data comes from one organization's governed work, including some client work, over about four weeks, on one model per provider.
- Provider and seat role are confounded, as described above.
- Labels of severity and of "changed the work" were made by AI models, not people.
- Every reviewer in the single-model checks ran on models from a single provider, so those checks say nothing about differences between providers.
What comes next
- Rotate seat roles across providers, so provider and role can be measured apart.
- Classify what clashing seats argue about across all clashes, not a sample.
- Test whether cross-provider disagreement predicts problems found later.
Conclusion
In our juries, models from different providers disagree with each other far more than models from the same provider, even after correcting for chance. We measured disagreement, not accuracy: these data cannot say whether the extra disagreement catches more real problems. The premise behind cross-provider juries, that one model's blind spots are less likely to be shared by a model from another lab, remains a hypothesis; testing it means checking whether cross-provider disagreement predicts problems found later (item 3 under What comes next). Whether that disagreement comes from the provider or from the role it plays is the next thing to measure.
Appendix: data and reproducibility
This appendix says where every figure in the note comes from, how it was computed, and what a reader with access to the controlled data would run to get it again. The counts behind every figure are in the tables below. The analysis scripts and the input extracts are internal records and are not published.
What is withheld, and why
The raw inputs are not published. The seat-verdict store and the check logs hold the full text of each seat's reasoning and each check's findings, the subjects of the work under review and the sessions it belonged to. Some of that work was done for clients, so the text names client work, client documents and the terms under discussion. We also withhold session and signal identifiers, model names and which real provider each letter stands for.
What is published is the counts alone: pair counts by provider grouping and verdict combination, vote counts by seat role, check counts and audit counts. None of these carries text, an identifier or a client name. Anyone given access to the controlled extracts can check them byte for byte against the digests and rerun every figure.
Sources
- Seat-verdict extract. A read-only export from the server's signal store of every seat verdict (approved, conditional or blocked). Each row has the seat's role, its session, the time it was recorded, a link to any later verdict that superseded it, and the seat routing: for each role in the jury, the provider and model it ran on. Routing was recorded from October 13, 2026, so only rows from then on can be used to compare providers.
- Local check-log extract. The deduplicated union of the local second-model check logs (
local-reviews.jsonl) as they stood on November 7, 2026. One record per check, with its findings and, for most, the moment in the work at which it ran. - Jury audit report. The server's jury-health report for the newest 200 audited juries, as returned on November 8, 2026 at 21:04 UTC. It holds counts, the server's notes on known gaps and a daily-spend line, and no seat text.
- Jury-composition study, slice 1. The aggregate tables produced by that study's own analysis script. Its frozen inputs (labelled concerns, keys and the repository snapshot) are not part of this package.
Definitions
Seat, verdict. A seat is one AI model given one role in a jury. Its verdict is approved, conditional (approved once named conditions are met) or blocked.
Ruling. We rebuild rulings from seat verdicts. Seat verdicts that share a session and an identical routing map are sorted by time. A new ruling starts when a role that already voted in the current ruling votes again, or when more than 180 seconds have passed since the previous seat's verdict. A ruling is complete when it has one verdict for every role in its routing map.
Independent ruling. A complete ruling in which no seat had yet seen its peers' verdicts. In the store, a verdict that was revised in a later round carries a link to the verdict that replaced it. A complete ruling that contains any verdict named as a replacement is a later round and is set aside. What remains are single-round rulings and the first round of multi-round deliberations.
Provider and letters. A seat's provider is read from its routing entry. Letters are assigned by the number of seats each provider filled across the independent rulings: Provider A filled the most, then B, then C. In this period each provider supplied exactly one model.
Seat pairs. Every unordered pair of seats within one independent ruling is a pair. A pair is same-provider when both seats ran on the same provider's model, and cross-provider otherwise. A pair disagrees when the two verdicts differ. It is a sharp clash when one seat approved and the other blocked. A clash ruling is an independent ruling with at least one approval and at least one block.
Usual provider of a role. The provider that filled most of that role's seats across the independent rulings. A seat is off its usual provider when it ran on any other provider.
Seat roles in the figures. The roles are shown by function: Skeptic (critic), Production readiness (downstream-advocate), Governance (sentinel), Code review (reviewer), Architecture (architect). The learning seat is sage and the release-coordination seat is release-coordinator.
Checks that record when they ran. Records in the check log that carry a moment field, which names the point in the work where the check ran (for example, just before a commit, or a fact-check of a plan). 115 of the 197 unique records have one.
A check raised a finding when, after dropping entries filed under right, note, notes, cosmetic, disposition or follow_up, and list entries that begin with right, ok, confirmed, none or no high, at least one finding remains. High or must-fix means a finding is labelled HIGH (in any case, as a label prefix), Hn, CRITICAL, BLOCKING or BLOCKER, or MUST-FIX, or that the record's structured fields count high or must-fix items. Fact-check means the moment names a fact-check.
Changed the work (jury-composition study). A concern that AI labellers classed as substantive and that was found implemented within 14 days of the jury, in a commit citing the jury, in the change directory the brief names, or in the files the brief names. No session had the full 14 days before the snapshot, so this is a lower bound. Two first-pass labellers labelled each concern. Where they disagreed, a third model broke the tie.
How the agreement score is computed
We report Fleiss' kappa with verdict proportions pooled across both seats of every pair in the group, computed separately for the same-provider pairs and the cross-provider pairs. Each pair is one item rated by two raters. For a set of N pairs:
- the observed agreement p_o is the share of pairs whose two verdicts match;
- n_k is the number of times verdict k appears across both seats of all N pairs, so the n_k sum to 2N;
- the chance agreement p_e is the sum over the three verdicts of (n_k / 2N)²;
- kappa = (p_o − p_e) / (1 − p_e).
With exactly two raters per item, Fleiss' kappa reduces to Scott's pi: both use one pooled verdict distribution for the two raters, unlike Cohen's kappa, which keeps a separate distribution for each rater. Worked example for the same-provider pairs (Table A1): p_o = (540 + 721 + 85) / 1,744 = 0.772. The pooled counts are 1,428 approved, 1,825 conditional and 235 blocked out of 3,488, so p_e = 0.446 and kappa = 0.588.
The bootstrap
Pairs within one ruling are not independent, so the confidence intervals resample whole rulings. We draw 1,171 independent rulings with replacement from the 1,171, rebuild every pair inside the drawn rulings, and recompute three differences: cross minus same raw disagreement, same minus cross kappa, and cross minus same sharp-clash rate. We repeat this 1,000 times with Python's random module seeded with 1 (random.seed(1), then random.choice once per drawn ruling). For each difference, the 1,000 values are sorted and the interval runs from the value at index 25 to the value at index 974 (zero-based), the 2.5th and about the 97.5th percentiles.
Aggregate tables
Table A1. Seat pairs in the 1,171 independent rulings, by provider grouping and verdict combination
| Verdict pair | Same provider | Cross-provider |
|---|---|---|
| approved + approved | 540 | 1,105 |
| approved + conditional | 333 | 2,247 |
| approved + blocked (sharp clash) | 15 | 307 |
| conditional + conditional | 721 | 2,672 |
| conditional + blocked | 50 | 360 |
| blocked + blocked | 85 | 264 |
| All pairs | 1,744 | 6,955 |
Table A2. The figures computed from Table A1
| Measure | Same provider | Cross-provider | 95% interval for the difference |
|---|---|---|---|
| Raw disagreement | 22.8% | 41.9% | 16.8 to 21.5 points |
| Fleiss' kappa (pooled) | 0.588 | 0.236 | 0.310 to 0.394 |
| Sharp clashes | 0.86% | 4.41% | 2.68 to 4.46 points |
Approved + conditional makes up 84% of disagreeing same-provider pairs (333 of 398) and 77% of disagreeing cross-provider pairs (2,247 of 2,914). The sharp-clash rate is 5.1 times higher across providers.
Table A3. Rulings. We rebuilt 1,343 complete rulings from the routed seat verdicts. 1,171 of them are independent, one per session, and run from October 13 to November 8, 2026. Of the independent rulings, 352 were unanimous, 694 split without a clash and 125 were clash rulings. Seats filled: Provider A 2,395, Provider B 1,368, Provider C 1,302.
Table A4. Votes in the 125 clash rulings, by seat role and the provider the seat ran on
| Seat | Provider | Approved | Conditional | Blocked |
|---|---|---|---|---|
| Skeptic | B | 0 | 10 | 56 |
| Production readiness | B | 3 | 34 | 51 |
| Governance | C | 72 | 24 | 26 |
| Governance | B | 1 | 0 | 1 |
| Code review | A | 49 | 37 | 7 |
| Architecture | A | 103 | 18 | 4 |
| Release coordination | B | 0 | 4 | 4 |
| Release coordination | A | 2 | 1 | 2 |
| Learning | C | 13 | 2 | 2 |
| Other roles (four) | A | 26 | 10 | 1 |
Across all 1,171 independent rulings, the Skeptic seat approved 50 times, was conditional 171 times and blocked 89 times. The four other roles (product, market analysis, test architecture and user experience) are not broken out here.
Table A5. Role pairs seen both as same-provider and as cross-provider pairs
| Role pair | Same: pairs | Same: disagree | Cross: pairs | Cross: disagree |
|---|---|---|---|---|
| Production readiness + Governance | 34 | 2 | 917 | 437 |
| Architecture + Learning | 10 | 1 | 221 | 71 |
| Code review + Learning | 10 | 0 | 202 | 59 |
| Production readiness + Release coordination | 18 | 3 | 26 | 15 |
| Architecture + Release coordination | 27 | 8 | 18 | 16 |
| Release coordination + User experience | 27 | 10 | 18 | 17 |
| Learning + Governance | 224 | 59 | 9 | 0 |
| Architecture + Governance | 4 | 2 | 1,108 | 356 |
| Skeptic + Governance | 3 | 1 | 304 | 182 |
| Test architecture + Governance | 3 | 1 | 49 | 13 |
| Code review + Governance | 1 | 0 | 1,001 | 310 |
The first seven rows have at least five pairs on each side. In six of them the cross-provider pairs disagree more often. 51 Governance and Learning seats ran off their usual provider: 37 Governance seats on B, 4 on A and 10 Learning seats on A. All 51 were in rulings with no Provider C seat, and 45 of them were on October 23. The release-coordination seat ran on A on nine days from October 15 to November 2, and on B on seven days from November 2 to November 8.
Table A6. Local second-model checks
| Count | Value |
|---|---|
| Unique checks logged | 197 |
| Checks that record when they ran (October 27 to November 7) | 115 |
| … that raised at least one finding | 100 (87%) |
| … with a finding labelled high or must-fix | 23 (20%) |
| … labelled high | 18 |
| … that were fact-checks | 41 |
| … that ran just before a commit | 64 |
Table A7. Jury audit (newest 200 audited juries, as of November 8, 2026, 21:04 UTC)
| Count | Value |
|---|---|
| Juries examined | 200 |
| Juries complete | 200 |
| Seats convened | 882 |
| Seats counted | 882 |
| Seats lost or uncounted | 0 |
| Governance events in the store | 3,216 |
| Events with no audit record | 1,416 |
Table A8. Jury-composition study, slice 1 (from the study's own tables)
| Count | Value |
|---|---|
| Past jury sessions sampled | 80 |
| Concerns raised by seats | 1,154 |
| Labeller agreement on finding class (kappa, 95% interval) | 0.731 (0.68 to 0.78) |
| Concerns where the labellers' finding class differed, broken by a third model | 201 (17%) |
| Concerns that changed the work within 14 days (lower bound: no session had 14 days) | 130 (1 in 8.9) |
| Sessions with the full 14 days of follow-up | 0 of 80 (39 had at least 7) |
| Changes from sessions matching the self-governance keyword measure | 123 of 130 (51 sessions, 716 concerns) |
How the figures were checked
Each figure was computed by a script from a fingerprinted (SHA-256) extract of the records described above. A separate model then re-ran every script on the same extracts, got byte-identical output, and fact-checked the text and tables against it; the author reviewed and accepted each correction. The scripts and extracts are internal records and are not published.
Notes
¹ The author's own observation of many public posts by tech coaches and software engineers, chiefly on LinkedIn, describing these behaviours.
² Diana Larsen, in “Leading Humans and Agents: Hybrid Teams as Complex Adaptive Systems with Diana Larsen,” Mob Mentality Show, October 8, 2026. Quoted with her permission.
How this paper was written. Willem Larsen wrote the introduction. Claude (Anthropic) analysed the data and drafted the remaining sections. A separate Claude instance then fact-checked the figures and claims against the source data, and the author reviewed and accepted each correction.