STROT is a compile-time decision compiler: it moves LLM reasoning to compile time, producing a frozen, local, calibrated "battery" that runs cheaply and privately at inference. This post walks through the architecture and logs every experiment behind it.
Abstract
Large language models give strong single-record decisions but cost per call, add latency, and send every record off-infrastructure — unacceptable for regulated, PII-heavy, high-volume work. STROT is a compile-time decision compiler: it uses an LLM once, at compile time, to synthesize a small frozen "battery" (a base encoder + optional low-rank adapter + calibration) that then runs locally for every record at ~$0 and ~20–50ms, returning a typed decision with calibrated confidence and honest abstention. On a hard cyber-intent task we show a frozen encoder hits a ~69% ceiling that no information-only method crosses (better labels, definitions, NLI, reranking, retrieval, and context-passing all stall) because the distinction is representation-bound; only light fine-tuning crosses it (~71–77%), cheaply, via cached-lower + top-N adaptation and LoRA weight-soups. We argue the right objective is not blanket accuracy but reliability on a calibrated confident slice (~90–100% on the records it answers), with a drift-aware optimizer widening that slice over time. Most tasks never need the fine-tune tier; a single opt-in flag gates it.
Problem & contributions
Problem. Calling a hosted LLM per record is accurate and simple, but at scale it is expensive, adds latency, and ships every record (and its PII) to a third party. For regulated, PII-heavy, high-volume decisions, "just call the LLM" is off the table.
Thesis. Do the LLM reasoning once at compile time and freeze it into a small, local, calibrated artifact (a battery) that serves decisions on the user's own infrastructure — cheap, fast, private, honest. STROT wins on cost / latency / data-residency / control, not on out-accuracy-ing the LLM.
Contributions.
- The capability-gap vs information-gap distinction, with eight no-fine-tune methods shown to stall on a representation-bound task, plus a cheap diagnostic to detect it.
- An escalation ladder that fine-tunes only when frozen fails, gated by an opt-in flag that is a no-op on easy tasks.
- Cached-lower + top-N fine-tuning that breaks the ceiling on CPU; the top-4 knee; LoRA soups (and why full fine-tuning doesn't soup).
- Calibrated conformal abstention as the product metric — reliability on the confident slice, not blanket accuracy.
- A privacy-preserving optimizer (one-time supervised distillation from the LLM teacher) with PII masking owned at the teacher boundary.
Method
Compile → battery → serve. STROT moves the LLM reasoning to compile time and freezes it into a battery served locally.
- Compile (GPU, once): the LLM provisions data (labels real records, generates synthetic for gaps, masks PII at the teacher boundary); the optimizer measures the frozen ceiling and escalates only as needed; fits calibration; packages the battery.
- Battery (artifact): frozen base pointer + optional LoRA adapter + calibration (temperature + conformal cutoff) + schema/metadata. A few MB, versioned, one per business case.
- Serve (CPU, forever): load the battery, one forward pass per record → typed decision + calibrated confidence + abstain. Local, ~$0, deterministic, PII never leaves.
Escalation ladder (cheapest mechanism that clears the bar). frozen (labels + calibration + model-selection) → cached-lower + top-N adaptation → LoRA → (full FT). Run tier N, measure on real held-out, escalate only if the gap justifies the cost; keep a tier only if it beats the prior on real data. The optimize flag gates the fine-tune tiers (permit, not force).
Fine-tuning tier. Cached-lower + a single top-4 LoRA adapter: lower layers frozen and their activations cached once, only the top-4 adapter trains. Output is a frozen adapter baked into the battery.
Calibration + conformal abstention. Temperature scaling (fix over/under-confidence) + a conformal cutoff (abstain threshold for a target accuracy/risk), both fit on a real held-out split and applied locally at runtime. Low-confidence or OOD records abstain.
Dispatch by default; abstention is opt-in. With no target set, the runtime dispatches every record with its calibrated confidence and never hard-abstains — the caller decides what to trust. Abstention turns on only when a reliability or accuracy target is set, and abstained records are returned for the caller to re-run. STROT does not loop.
Optimizer (compile-time, opt-in, once). The optimize flag, together with a reliability or accuracy target, gates a single fine-tune at compile time — cached-lower + top-N LoRA, with a K=3 diverse-adapter ensemble averaged at inference. STROT never retrains at runtime and never loops; without a target, optimize is a no-op and the frozen battery ships. The LLM teacher is used only as a labeler of real records (distillation), PII-masked at the boundary so no raw record leaves the box.
Experiments
All runs on cyber-incident classification (VERIS/VCDB, 6 classes: hacking / malware / error / misuse / physical / social), single-label records, real human-coded labels, 300–540 records. Frozen encoders run under no_grad (only small heads/adapters train). Each row: setup → result → takeaway.
Frozen methods — none cross the intent wall
| Method | Setup | Result | Takeaway |
|---|---|---|---|
| Linear probe, encoder sweep | frozen mean-pool + logistic reg | Julia 50.5 · roberta-base 59.2 · roberta-large 65.1 · SecureBERT 56.8 · CySecBERT 55.2 · UltraBERT-150M 60.4 | base size+quality drives the ceiling, not domain pre-training |
| Well-tuned probe | ModernBERT, 540 recs, 256 tok | ~69% | frozen ceiling is higher than the early "58%" (data + tuning dependent) |
| Cheap frozen tricks | decomposition / distill / GLiNER / measured-labels | −12 / −10.6 / −4 pp / neutral | no frozen lever moves representation-bound cyber |
| Intent diagnostic | flip accident↔attack, cos-distance | 0.029 vs 0.042 arbitrary (ratio 0.67) | encoder barely responds to intent |
| Projection head / SupCon | frozen + MLP + contrastive head | linear 57.5 · MLP 58.4 · SupCon 57.1 | a head only separates what the encoder already separated |
| NLI cross-encoder | roberta-large-mnli, 6 hypotheses | 41.3% (hacking 16, error 10; social 88) | joint scoring confirms the gap; surface classes OK |
| Reranker | ms-marco MiniLM | 47.0% | same as NLI |
| Context-passing | class defs packed into input, frozen | plain 69.3 → ctx_all 66.4 / cls 46.1 / input 63.8 | information can't fix a capability gap (it hurt) |
Fine-tuning — crosses the wall
| Method | Setup | Result | Takeaway |
|---|---|---|---|
| LoRA (roberta-base) | attn + classifier, r=16 | ~70% (69.1 / 71.7) | fine-tuning breaks the frozen ceiling |
| Cached-lower + top-N (ModernBERT) | freeze+cache lower layers, train top-N, single seed | head 60.5 · top-4 71.1 · top-8 76.3 · top-12 71.7 · top-16 77.6 | top-4 is the knee; depth beyond = noise; caching lets it finish on CPU |
Model soups
| Soup | Individuals (best / mean) | Soup | Takeaway |
|---|---|---|---|
| Full-FT top-4 (3 seeds) | 67.8 / 65.4 | 65.1 (−2.6 vs best) | full-FT does NOT soup (weights drift too far → soup = mean) |
| LoRA top-4 (3 seeds) | 75.0 / 73.5 | 77.0 (+2.0 vs best) | LoRA soups cleanly (low-rank, shared basin) |
| LoRA top-8 (4 seeds) | 75.0 / 69.9 | 77.0 (+2.0 vs best, +7.1 vs mean) | reproducible; depth doesn't help the soup; strong variance reduction |
Decision: ship a single top-4 LoRA adapter (not the soup) — robustness comes from the K=3 prediction ensemble, not weight-averaging.
Calibration & economics
| Item | Result | Takeaway |
|---|---|---|
| Confidence — raw softmax | 63% of records claim ≥90% but that bin is 87.6% accurate | raw softmax is overconfident — unusable as-is |
| Confidence — temp-calibrated (T=2.7) | ≥90%→100% (12%), 70–90%→92% (35%); confident slice ≥70% ≈ 47% @ ~94% | calibration makes confidence honest; abstain gate works |
| LoRA-adapter reuse (HF search) | only VERIS adapters found were 7B-welded (Mistral/Qwen) | find-before-train usually comes up empty for bespoke small-encoder tasks |
| Timing | top-4 LoRA ~80–100s/seed CPU (Docker) + ~50s cache; ~30s on a T4; serve ~20–50ms/record CPU | compile cheap on GPU (pennies), serve cheap on CPU |
Selective prediction: frozen vs fine-tuned (the key reframe)
The risk-coverage frontier — sort test by confidence, answer the most-confident X%, report accuracy on the answered set — exposes fine-tuning's real value, which overall accuracy hides. On a seed where overall accuracy is noise-tied, the fine-tuned model's confident slice is nearly double the frozen one's:
| Max coverage at accuracy target | Frozen (no fine-tune) | Fine-tuned (top-4 LoRA) |
|---|---|---|
| ≥95% | 2% | 32% |
| ≥90% | 37% | 61% |
| ≥85% | 59% | 73% |
| overall (answer all) | 73.0% | 72.3% |
Fine-tuning barely moved overall accuracy here (73.0 vs 72.3 — seed noise) but roughly doubled the confident slice (@90%: 37%→61%; @95%: 2%→32%). It sharpens the confidence ranking — the model knows which predictions to trust — which calibration (monotonic) cannot create. Implication: the escalation trigger should key off the confident-slice target, not overall accuracy — frozen suffices for “~37% @ 90%”, but fine-tuning is required for “60% @ 90%” or anything @ 95%. (Single seed; the confident-slice gap is the robust finding, exact coverages pending a 3-seed sweep.)
Maximizing the confident slice: the ensemble lever
cov@90 — how many records you can answer at ≥90% accuracy — is the product metric. A prediction-ensemble (average the softmax of K diverse top-4 LoRA models) widens it sharply, with essentially no accuracy change:
| overall acc | cov@95 | cov@90 | |
|---|---|---|---|
| single model (avg of 4) | ~77% | 46.5% | 64.4% |
| ensemble (avg softmax of 4) | 78.0% | 63.8% | 83.0% |
| Δ | +1pp | +17.4pp | +18.6pp |
The ensemble beats the best single seed's cov@90 (67.4%) by +15.6pp — not noise-averaging. Overall accuracy barely moves (+1pp) but cov@90 jumps +18.6pp: ensembling is a confidence-ranking tool, and cov@90 is a ranking metric. Cost: K× inference (still local, ~$0, ~120ms/record at K=4; K is tunable). This is the lever for maximizing the confident slice — distinct from the weight-soup, which only moved accuracy. Two levers stack: more teacher data (data growth) grows cov@90 toward the plateau (measured 5%→25%→61% as data grew); the ensemble sharpens the ranking on top.
The two levers stack — cov@90 at each training-data size (K=3 ensemble, simulating data growth):
| train size | single cov@90 | ensemble cov@90 |
|---|---|---|
| 60 | 3% | 1% |
| 150 | 23% | 39% |
| 250 | 42% | 48% |
| 399 | 57% | 79% |
Data grows the slice (3%→57% single); the ensemble adds on top once there's enough data to train on (+5 to +22pp), negative only at the cold start (60 records — too undertrained to ensemble). Best cov@90 = full data + ensemble (79%), higher than either lever alone. Gains are noisy (single seed) but the stacking direction is clear: data growth does the heavy lifting, the ensemble (ranking) sharpens it on top.
Synthetic augmentation hurts (negative result). We tested whether LLM-generated synthetic data — grounded in real examples plus some random, balanced at 26/class (~29% of the training set) — helps the cyber task. It hurt: real-only 76.3% acc / 73.0% cov@90 → real+synthetic 74.3% / 55.9% (−2.0pp acc, −17.1pp cov@90). The mechanism is label leakage: the generator knows the class, so it writes the intent explicitly ("accidentally" for error), amplifying a cue that real, ambiguous incidents do not carry. The model learns the easy cue and its confidence ranking collapses on real data — exactly where cov@90 lives, hence the −17pp. Single-seed, but the direction is strong.
Data-provisioning policy. The teacher LLM labels real records (distillation); it never bulk-generates training data. Class imbalance is handled real-first: pull more real data of the minority class and label it; failing that, oversample or class-weight the real minority; synthetic is a last-resort bootstrap only for a near-empty class. PII masking at the teacher boundary uses GLiNER as the default masker (names, addresses, organizations), with a thin regex pre-pass for format-defined tokens (emails, cards, SSNs).
Key findings
- Capability gap vs information gap (the central result). Information-bound: the signal is in the frozen representation and just needs framing/labels — information methods help. Representation-bound (capability gap): the frozen model cannot perceive the distinction — and no information method helps. Eight tested (labels, definitions, decomposition, GLiNER, NLI, reranker, retrieval, context-passing); all at or below the frozen probe. Only weight changes (fine-tuning) cross it. The 0.029 intent-flip diagnostic quantifies it.
- Cached-lower + top-N fine-tuning is enough. Freeze+cache the lower layers, train only the top N on cached activations. top-4 is the knee; caching makes it finish on CPU and dodges OOM. Depth beyond is noise.
- LoRA soups; full fine-tuning doesn't. Averaging K adapters beats the best single seed by +2pp and erases seed noise — but only for LoRA (low-rank, shared basin). Full FT drifts too far → soup = mean. We ship a single adapter + gate-before-swap.
- Calibrated abstention is the product metric. Raw softmax is overconfident; temperature + conformal calibration makes confidence honest. Target ~90–100% reliability on the confident slice, not 90% on everything.
- Most tasks never need fine-tuning. Routing and tool-selection are information-bound → frozen clears the bar. The
optimizeflag gates the fine-tune tier (permit, not force) — a no-op on easy tasks. - Economics. Compile once (GPU, pennies/seconds) → serve forever (CPU, ~$0, PII-local). The small-encoder choice (150M, not 7B) is what makes CPU serving viable.
- Fine-tuning's value is selective-prediction coverage, not overall accuracy. On a seed where overall accuracy is noise-tied (frozen 73% vs fine-tuned 72%), fine-tuning still roughly doubled the confident slice (@ ≥90%: 37%→61%; @ ≥95%: 2%→32%). It sharpens the confidence ranking, which calibration cannot.
- Ensembling maximizes the confident slice (cov@90), not accuracy. Averaging K diverse LoRA models' predictions widened cov@90 from 64%→83% (+18.6pp) and cov@95 +17.4pp, while overall accuracy moved +1pp — the confidence-ranking lever, matched to the selective-prediction metric. The two cov@90 levers: more teacher data (data growth) grows it toward the plateau; the ensemble sharpens the ranking. Cost: K× inference (tunable).
- At small scale, simple beats clever (three times). Weight-soup (dropped), context-passing (hurt), and smart active-learning acquisition (random won) all failed to beat the simple baseline on small/noisy data — the robust lever is always more data, not cleverer selection/averaging.
Limitations & honest ceiling
- Numbers are cyber-specific, small-data, often single-seed. 540 records, single-label, ~±3pp seed noise. The direction (frozen caps, fine-tuning crosses, LoRA soups, calibration works) is robust; exact cell values are not.
- 90% overall is likely a mirage. Capped by label noise (VERIS is human-coded), irreducible ambiguity (multi-action incidents, uninformative summaries), and possibly human inter-annotator agreement. Chasing it means fitting noise.
- Confident-slice reframe. STROT targets reliability on what it answers, not blanket accuracy. More data (via the optimizer) widens the slice; it does not lift the hard-record ceiling.
- No free-text reasoning at runtime. The battery is a frozen encoder; explanations are a compile-time artifact (the battery card), not a runtime feature.
- Compute caveats. All runs were Docker-CPU (no GPU/MPS on the Mac), which inflated timing and forced the caching trick; small calibration bins make the 100% cell suggestive, not locked. A GPU multi-seed sweep is the next step to lock the numbers.
Reproducibility
- Dataset: VERIS Community Database (VCDB), joined JSON; 6 VERIS action classes; single-action records with summaries > 40 chars; capped per class (45–90) for balance; 300–540 records per run.
- Models:
answerdotai/ModernBERT-base(150M, fine-tune base);FacebookAI/roberta-base(LoRA proxy that finishes on CPU);roberta-large-mnli(NLI);cross-encoder/ms-marco-MiniLM-L-6-v2(reranker); Julia-1, SecureBERT, CySecBERT (frozen comparison). - Fine-tune config: cached-lower + top-N; LoRA r=16, alpha=32, lr 1e-3, 6 epochs, max_len 128; adapter on the top layers only; splits ~60/14/26 train/cal/test; seeds 11/23/37/51.
- Calibration: temperature grid-search on a held-out cal split (minimize NLL); conformal cutoff for a target risk.
- Code:
benchmarks/*.py— cyber_encoders, cyber_frozen_contrast, cyber_crossencoder, cyber_context_passing, cyber_cached_top, cyber_soup_lora, cyber_confidence_dist, cyber_lora. All runs in Docker (strot-julia image, CPU; no GPU/MPS). - Open next: GPU multi-seed sweep to lock the depth/soup numbers; a second task (routing or tool-selection) to confirm the frozen path suffices; richer input than one-line summaries.
components/comments.tsx.