// research logBenchmarks & research
Honest, reproducible benchmarks — published as STROT is run on new data over time. Every number ships with a script in the repo, measured on real datasets in clean containers. Wins and losses both make the list.
2026-10-19VERIS/VCDB · CFPB consumer complaints
Risk narratives — cyber & finance
Record-level, unstructured text — the kind cyber and financial triage actually run on. Public, ground-truth datasets; classes balanced so accuracy is interpretable; baselines shown. STROT never sees the test labels.
| Task | Baseline | Naive | Compiled | Notes |
| Cyber — incident note → attack action (6-class) | 16.7% random | 36.8% | 39.5% | 8/114 abstain |
| Finance — complaint → product (6-class) | 26.4% majority | 54.2% | 61.1% | 5/72 abstain |
Both clear their baselines and the compiler lifts the same 144M model (+2.7 / +6.9pp), with low-confidence records abstaining. Honest proof the architecture works on messy real text — not production underwriting accuracy. For that, slot a task-trained specialist into nlu=.
risk_narratives.py ↗2026-10-13AG News · DAIR Emotion
Semantic text classification
Does compilation help when the answer is not a structured field? 300 held-out records each; the compiler never sees any test label.
| Task | Baseline | Naive | Compiled | Notes |
| AG News — 4-class topic | — | 72.7% | 86.7% | +14.0pp |
| Emotion — 6-class tweets | — | 93.3% | 93.3% | measured labels recover a loss |
Compilation wins big when integration is the bottleneck (AG News). It can hurt when the naive interface is already optimal (Emotion) — which is exactly why measured label search keeps whichever scores higher.
semantic_text.py ↗2026-10-11seaborn mpg (398 cars)
Grounded routing
Route each record to the correct workflow. The compiler synthesizes labels + input projection from the registry and schema alone, then is measured against held-out ground truth.
| Task | Baseline | Naive | Compiled | Notes |
| Route by origin (3-class) | — | 74.8% | 92.5% | +18pp, same model |
On the identical model, a good projection vs naive integration is worth +18pp. The model never changed; the integration did.
routing_mpg.py ↗2026-10-15tweet_eval hate (400 tweets)
Where it falls down — hate detection
The honest ceiling of a 144M generalist on an adversarial, implicit task — published because losses matter as much as wins.
| Task | Baseline | Naive | Compiled | Notes |
| Hate speech (binary) | 59.0% majority | 47–53% | ~59–64% | distilled + calibrated |
No amount of compiler cleverness fixes what the embeddings don't encode. The fix is a task-trained specialist in the nlu= slot, not STROT magic — and the calibration told the truth, abstaining when it couldn't promise accuracy.
benchmarks/README ↗More as we run STROT on new datasets. Reproduce any of these from benchmarks/.