Data Science — Sheffield Bancorp batch report (`claude-fable-5`)

DS 6 · Sheffield Bancorp · data deliverables. 5 data-science tasks · one model · up to 10 rollouts each (40 scored rollouts), scored by a per-criterion weighted rubric (agent-judge, grader claude-opus-4-8). Environment: a synthetic retail bank with a bank.db SQLite warehouse plus data/bank/ CSVs, filesystem/ docs and dashboards, Slack, and email. No benchmark model: absolute per-task scores, not a head-to-head. Per-rollout raw artifacts (solver reasoning, grader per-criterion evaluation, rubric, the task input packet) accompany this report.

1. Executive summary

claude-fable-5 works like a governance-first data analyst across all five tasks, with a domain mean of 73.7%. It lands the judgment on every task: the top-line call, the provenance discipline, the refusal to take owner or management numbers at face value. What caps its scores is exact-scope numeric fidelity under the bank's date and timezone conventions. The gap is definitional precision; the model understands the tasks and drives the tooling.

Task	Description	`claude-fable-5`	σ
`task-114`	RBC merger credit-diligence JSON	84.0%	3.0
`task-56`	Cardholder-name reconciliation	79.9%	2.5
`task-52`	Q1 retail-deposits flash sanity check	71.1%	15.2
`task-101`	Q1 AML structuring routing packet	67.3%	1.9
`task-92`	Related-party exposure restatement	66.2%	4.2
Mean		73.7%

Behavioral drivers, drawn from the trajectories and per-criterion grading across all 40 scored rollouts:

Judgment and governance are right; exact-scope numbers are the ceiling. On every task the model lands the consequential call (block or defer the merger, refuse to route the owner packet as-is, mark the deposit flash unsafe to use, withhold unconditional approval) and recomputes from source (bank.db) rather than trusting owner rollups or management flags. It loses points on the specific figures the rubric checks, not on the analysis.
Date-only and timezone bucketing is the dominant failure mode. The warehouse stores date-only timestamps, and every quantitative task turns on a workspace convention the model mishandles: task-101 collapses the "rolling 24-hour window" into a calendar-day bucket (~31% of that task's weight); task-52's bimodal split is raw-UTC vs America/New_York bucketing; task-92's date(date,'-5 hours') shifts date-only values a full day. The model names the precision issue in its reasoning, then makes the wrong call anyway.
It computes a plausible near-neighbor population, not the rubric's exact scope. "All-statuses" instead of "active-only" (task-114: 74 vs 73 employees), "NPA-only" instead of the "stressed union" (task-114: 32 vs 34 loans), "any-Q1-transaction" instead of "Q1-active" (task-92: 721 vs 226 relationships), "merged joint+single owner" instead of "single-owner" (task-56: 13,918 vs 12,974). Each single-filter miss zeroes an exact-value criterion.
Surfacing an exact number is a distinct gap from computing it. Even when the math is right, the figure the grader checks sometimes never appears: task-52 prints per-day residuals but never sums the 110,805.13; task-114 computes evergreen and restructured but never their combined 419,852,260.41.

2. What was run

Tasks: 5 data-science deliverables set at Sheffield Bancorp: an AML structuring routing packet, an M&A credit-diligence JSON, a retail-deposits flash sanity check, a cardholder-name reconciliation, and a related-party exposure restatement. Each requires a substantive work product (script + CSV/JSON + memo) recomputed from source records.
Rollouts: up to 10 independent rollouts per task, 40 scored (see §3 for errored runs), which measures consistency across runs rather than one sample.
Scoring: each rollout is graded by an agent-judge against a task-specific weighted rubric (30–63 criteria per task). A rollout's score is the weighted mean of its criteria (0–100%); the task score is the mean across its rollouts. Scores are partial-credit, not pass/fail.
How patterns were derived: for every task we read the per-criterion grader evaluation (score + evidence + reasoning) and the solver's reasoning traces, then folded in which criteria are solved every run vs zero every run. Patterns are cited to specific tasks and criteria.
Config: environment is the shared Sheffield workspace (bank.db + data/bank/, filesystem/, Slack, email); each task also received a per-task input packet in task-data/. Solver claude-fable-5; grader claude-opus-4-8.

3. Reliability — delivery, refusals & errored runs

The model never refused a request and produced a scored, on-topic deliverable in all 40 scored rollouts. No safety over-refusals, no empty deliveries. Where a run completed, its deliverable is complete.

The reliability liability is run-level errors, not model behavior: of 50 attempted rollouts, 10 were excluded as run_status=errored before scoring (4 on task-52, 3 on task-56, 3 on task-114). task-101 and task-92 ran clean (10/10 each). The packaged data flags these only by status and does not attribute them to a model choice, so read them as an infrastructure and run-stability tail on three of the five tasks rather than a delivery defect. Every scored run scored on a real deliverable.

No rollout produced the canonical (100%) answer on any task — consistent with these being hard, exact-scope rubrics where the model reliably leaves the hardest figures on the table.

4. Per-task analysis

4.1 `task-114` — RBC merger credit-diligence JSON (84.0%)

Task: a signing-stage M&A credit-diligence answer.json (10 required keys) assessing whether Riverton Bank & Credit's merger can proceed — credit quality, NPA, provisioning, connected-borrower concentration, collateral, employee/data readiness, and source reliability.

Per-rollout (weighted %): 88 · 79 · 85 · 87 · 81 · 85 · 82 (7 scored, 3 errored).

Strong recompute-from-source diligence. Every run blocks/defers signing, refuses management/PMO/HR numbers at face value, and recomputes the hard credit figures from bank_rbc.db: cleaned NPA exposure 900,696,026.29, gross NPA ratio 39.69%, the 90-day contractual-delinquency rule (excluding reversed/suspense/pending cures), required provision 383,060,824.43, provision gap 264,311,968.86, and the largest connected group GRP-OMEGA-17 — with reproducible SQL.
The ceiling is definitional scope, not analysis (~8.5% of weight). It computes a near-neighbor population: GRP-OMEGA-17 connected-stressed exposure 289,889,778.06 across 34 loans (reports the NPA-only subset, 32 loans); 73 active duplicate-national-ID employees (reports 74, all statuses); restructured-OR-evergreened combined 419,852,260.41 (reports them separately, never combined).
Variance: low σ; all 7 scored runs are MODEL_PARTIAL. What moves is which definitional-scope figure each run happens to surface.

4.2 `task-56` — cardholder-name reconciliation (79.9%)

Task: reconcile embossed cardholder names against account ownership/roles — a cardholder_reconciliation.csv (one row per card) plus a memo separating routine cases from manual-review cases.

Per-rollout (weighted %): 77 · 84 · 76 · 81 · 79 · 81 · 81 (7 scored, 3 errored).

Nails the join model and role semantics. Owner via role_on_account='owner'; states beneficiary/manager are not the primary role; explains cardholder_name is free-text, not a foreign key; refuses string-equality-only matching; one row per card; total cards 15242.
Capped by a classification-taxonomy divergence (~14% of weight). The rubric wants multiple-owner accounts flagged for review (review count 1235, total review population 1310); the model classifies the 944 joint-owner-matching cards as routine and reports a review population of 732. It also flags beneficiary matches as a review anomaly (a defensible "beneficiaries shouldn't hold live cards" reading) where the ground truth counts same-account non-owner matches as expected authorized-user mismatches.
Variance: low σ; the taxonomy choice is stable across runs (MODEL_PARTIAL ×8, one strict exact-count call).

4.3 `task-52` — Q1 retail-deposits flash sanity check (71.1%)

Task: a Treasury/ALCO liquidity check — sanity-check the Q1 retail-deposits "flash" spike days before they feed the liquidity-buffer story; deliver a deposit_flash_sanity.py and a deposit_flash_packet.md quantifying the correction.

Per-rollout (weighted %): 91 · 55 · 92 · 59 · 58 · 72 (6 scored, 4 errored).

The qualitative call holds on every run: the flash trend is "not safe to use," internal/suspense-originated postings are the main driver, the owner rollup's safe_to_use is rejected, and the payroll/ordinary-behavior framing is refused.
A whole sub-analysis is missing in all 6 runs: the destination-branch breakdown of removed internal postings ("Branch – Chicago" as largest at ~$5,323,207.06) never appears ("grep returned nothing"). Two exact figures also go unstated: the 30.55× unadjusted-to-safe ratio and the 110,805.13 retained-deposits aggregate (the script prints the six per-day components but never sums them).
Why σ = 15.2 (bimodal): the pivot is date bucketing. The high runs (0.91, 0.92) bucket dates in the bank's America/New_York convention and reproduce the removal figures exactly; the low runs (~0.55–0.59) bucket by raw UTC (one packet even calls NY-bucketing a "pipeline defect"), which flips the reproduction, safety, and per-spike-day criteria to zero together.

4.4 `task-101` — Q1 AML structuring routing packet (67.3%)

Task: rebuild a reproducible Q1-2026 AML structuring detection route for the Risk-Committee packet — a Python rebuild script, a routed-account CSV, a source-conflict CSV, a summary JSON, and a decision memo — and say whether the owner-prepared rollup is safe to route.

Per-rollout (weighted %): 69 · 69 · 68 · 68 · 68 · 63 · 68 · 69 · 65 · 66.

Governance and provenance are right on every run: it follows the Slack/CRO-confirmed governing rule over the runbook/export/dashboard, treats the deprecated 7-day runbook rule and the owner rollup's 16-item conclusion as overbroad, groups by destination account, uses the $9,500 aggregate floor (not the $10,000 CTR), keeps the 14 dashboard single-transaction alerts and the manual conversation separate, and concludes the owner packet is not route-safe.
One methodological choice caps ~31% of the weight. It collapses the "rolling 24-hour window" into a calendar-day bucket; the solver reasons that because transactions.date is date-only, "a rolling 24-hour window effectively collapses to same-calendar-day aggregation." The ground truth expects consecutive-day pairs to aggregate into one window, yielding two routed accounts (9a7b25e7: 10693.57 / 7 txns spanning Jan 1–2; aed179c9: 9546.79 / 2 txns spanning Mar 30–31). The model routes only one and demotes aed179c9 to a "near-miss caveat." That cascades into the two-account count, the per-account amounts/windows/txn-IDs, and, because it adopted calendar-day logic, its confirmation of the stale analytics export instead of a flag that it is incomplete.
Variance: the lowest σ in the set (1.9). All 10 rollouts make the same windowing call — a deterministic ceiling, not noise.

4.5 `task-92` — related-party exposure restatement (66.2%)

Task: a compliance related-party-lending restatement to a single JSON (12 required keys) — employee↔customer matches via entity_id, owner-only qualifying accounts, Q1 transaction exposure, same-branch/closed-branch analysis, high-value review at $9,500, and an owner-only-vs-all-roles methodology variance. Unusually prescriptive (27 explicit rules).

Per-rollout (weighted %): 64 · 63 · 61 · 74 · 66 · 63 · 66 · 66 · 74 · 64.

Structure and semantics are right: entity_id-only matching, transactions counted once per owned-account relationship, RP-to-RP independent contribution, active employees 471, closed branch Bristol, same-branch volume 278,480.05, full schema compliance, and no unconditional board approval.
A definitional gap on "Q1-active" inflates the whole relationship/variance layer (~24% of weight). The rubric expects 226 owner-only Q1-active relationships (247 all-role); the model defines Q1-active as "any relationship with ≥1 Q1 transaction" and reports 721/836, which poisons the methodology-variance block (overstated relationships 21 → 115). Date-only NY-offset handling also produces off-by-small transaction/volume counts, and the control narrative frames methodology sensitivity as overstatement-only where the rubric also wants the understatement risk.
Variance: small spread (0.61–0.74). The two high runs additionally catch the top same-branch conflict; the two MODEL_BUG rollouts are where the date shift produced an outright wrong Q1 window.

5. Cross-task patterns

Date-only and timezone bucketing is the highest-leverage failure mode. It appears in every quantitative task (task-101 rolling-window collapse, task-52 UTC-vs-NY split, task-92 date-shift), and the model names the precision issue in reasoning before making the wrong call. Fixing the workspace date convention alone would recover the largest block of lost points.
The ceiling is definitional and scope precision, not analytical capability. The model computes a plausible near-neighbor population (all-statuses vs active-only, NPA-only vs stressed-union, any-Q1-txn vs Q1-active): single-filter misses that each zero an exact-value criterion while the surrounding analysis passes.
Classification and direction taxonomies diverge from the rubric's. task-56 buckets multiple-owner cards as routine (rubric: review); task-92 frames sensitivity as overstatement-only (rubric also wants understatement). The model applies defensible bank-ops logic that the ground truth defines differently.
Surfacing an exact figure is distinct from computing it. Right math, unstated number: the 110,805.13 sum, the 30.55× ratio, the combined 419,852,260.41. A self-audit against the enumerated asks would recover much of this.
What holds up on every run (the floor): file, schema, and JSON-format compliance; the top-line qualitative judgment on every task; source-precedence and provenance discipline (recompute from the warehouse, cite multiple artifact types, honor the governing rule); and the instinct to refuse owner and management numbers at face value. Trust the model on judgment and governance; check its exact-scope figures.

6. How to read these results

Scores are weighted-rubric percentages from an agent-judge; each task's rubric weights differ, so the numbers mean most within a task, and the domain mean reads as directional.
These are absolute single-model scores with no benchmark model, so read each task against its own difficulty and the behavioral read above, not against another model.
The two lower-scoring tasks (task-101, task-92) are not weak deliverables; they are hard, exact-scope rubrics where a single upstream convention choice caps the numeric tail. Treat them as discriminating samples, not as a ceiling on the model.

7. Appendix — artifacts

Per-rollout raw artifacts accompany this report, one set per task: the solver's reasoning trace, the grader's per-criterion evaluation (score, evidence, reasoning), the scoring rubric, and the per-task input packet (task-data/). Open any task in the navigator to browse them.