1 · grader soundness
0.0%
false-flag rate on hand-labeled fixtures, real Ghidra ground truth — target 0.
2 · vs frontier models
0.0%
groundre's own hallucination rate on the shared scoreboard corpus.
3 · confidence calibration
0.0832
Expected Calibration Error over 22 labeled claims (0 = perfect).
reliability diagram — does N% confidence mean right N% of the time?
| confidence bucket | n | mean confidence | actual accuracy | gap |
|---|
| 0.6–0.7 | 1 | 0.70 | 1.00 | 0.30 |
| 0.9–1.0 | 21 | 0.93 | 1.00 | 0.07 |
Methodology. REAL Ghidra ground truth + deterministic verifiers scored against hand-labeled fixtures (fixtures/*.expected.json). Metric: false-flag rate — true claims wrongly marked CONTRADICTED. Target: 0. Frontier LLMs are each asked to reverse-engineer the SAME real binaries; groundre's deterministic verifiers grade every claim against Ghidra ground truth. The model under test is graded — the grader is never a model. Expected Calibration Error (reliability diagram) over the soundness run's per-claim (confidence, correct) pairs — measured, not asserted.
Reproduce it.python -m bench.soundness # grader false-flag rate
python -m bench.run # per-binary + aggregate metrics
python -m bench.publish # versioned, citable scoreboard
Every binary used is pinned by sha256 in this report's corpus.cases.
Corpus: 5 benign sample(s), each pinned by sha256 — full manifest + raw JSON at
/accuracy-report.
bench 2026.1 · schema 1 · generated 2026-07-20T19:02:51.798029+00:00 · live from the backend