STROT
← Blog// research

Compile-time distillation: turning an LLM into a frozen local decision battery

Amit Rath · Nov 10, 2026

STROT is a compile-time decision compiler: it moves LLM reasoning to compile time, producing a frozen, local, calibrated "battery" that runs cheaply and privately at inference. This post walks through the architecture and logs every experiment behind it.

Abstract

Large language models give strong single-record decisions but cost per call, add latency, and send every record off-infrastructure — unacceptable for regulated, PII-heavy, high-volume work. STROT is a compile-time decision compiler: it uses an LLM once, at compile time, to synthesize a small frozen "battery" (a base encoder + optional low-rank adapter + calibration) that then runs locally for every record at ~$0 and ~20–50ms, returning a typed decision with calibrated confidence and honest abstention. On a hard cyber-intent task we show a frozen encoder hits a ~69% ceiling that no information-only method crosses (better labels, definitions, NLI, reranking, retrieval, and context-passing all stall) because the distinction is representation-bound; only light fine-tuning crosses it (~71–77%), cheaply, via cached-lower + top-N adaptation and LoRA weight-soups. We argue the right objective is not blanket accuracy but reliability on a calibrated confident slice (~90–100% on the records it answers), with a drift-aware optimizer widening that slice over time. Most tasks never need the fine-tune tier; a single opt-in flag gates it.

Problem & contributions

Problem. Calling a hosted LLM per record is accurate and simple, but at scale it is expensive, adds latency, and ships every record (and its PII) to a third party. For regulated, PII-heavy, high-volume decisions, "just call the LLM" is off the table.

Thesis. Do the LLM reasoning once at compile time and freeze it into a small, local, calibrated artifact (a battery) that serves decisions on the user's own infrastructure — cheap, fast, private, honest. STROT wins on cost / latency / data-residency / control, not on out-accuracy-ing the LLM.

Contributions.

  • The capability-gap vs information-gap distinction, with eight no-fine-tune methods shown to stall on a representation-bound task, plus a cheap diagnostic to detect it.
  • An escalation ladder that fine-tunes only when frozen fails, gated by an opt-in flag that is a no-op on easy tasks.
  • Cached-lower + top-N fine-tuning that breaks the ceiling on CPU; the top-4 knee; LoRA soups (and why full fine-tuning doesn't soup).
  • Calibrated conformal abstention as the product metric — reliability on the confident slice, not blanket accuracy.
  • A privacy-preserving optimizer (one-time supervised distillation from the LLM teacher) with PII masking owned at the teacher boundary.

Method

Compile → battery → serve. STROT moves the LLM reasoning to compile time and freezes it into a battery served locally.

  • Compile (GPU, once): the LLM provisions data (labels real records, generates synthetic for gaps, masks PII at the teacher boundary); the optimizer measures the frozen ceiling and escalates only as needed; fits calibration; packages the battery.
  • Battery (artifact): frozen base pointer + optional LoRA adapter + calibration (temperature + conformal cutoff) + schema/metadata. A few MB, versioned, one per business case.
  • Serve (CPU, forever): load the battery, one forward pass per record → typed decision + calibrated confidence + abstain. Local, ~$0, deterministic, PII never leaves.

Escalation ladder (cheapest mechanism that clears the bar). frozen (labels + calibration + model-selection) → cached-lower + top-N adaptation → LoRA → (full FT). Run tier N, measure on real held-out, escalate only if the gap justifies the cost; keep a tier only if it beats the prior on real data. The optimize flag gates the fine-tune tiers (permit, not force).

Fine-tuning tier. Cached-lower + a single top-4 LoRA adapter: lower layers frozen and their activations cached once, only the top-4 adapter trains. Output is a frozen adapter baked into the battery.

Calibration + conformal abstention. Temperature scaling (fix over/under-confidence) + a conformal cutoff (abstain threshold for a target accuracy/risk), both fit on a real held-out split and applied locally at runtime. Low-confidence or OOD records abstain.

Dispatch by default; abstention is opt-in. With no target set, the runtime dispatches every record with its calibrated confidence and never hard-abstains — the caller decides what to trust. Abstention turns on only when a reliability or accuracy target is set, and abstained records are returned for the caller to re-run. STROT does not loop.

Optimizer (compile-time, opt-in, once). The optimize flag, together with a reliability or accuracy target, gates a single fine-tune at compile time — cached-lower + top-N LoRA, with a K=3 diverse-adapter ensemble averaged at inference. STROT never retrains at runtime and never loops; without a target, optimize is a no-op and the frozen battery ships. The LLM teacher is used only as a labeler of real records (distillation), PII-masked at the boundary so no raw record leaves the box.

Experiments

All runs on cyber-incident classification (VERIS/VCDB, 6 classes: hacking / malware / error / misuse / physical / social), single-label records, real human-coded labels, 300–540 records. Frozen encoders run under no_grad (only small heads/adapters train). Each row: setup → result → takeaway.

Frozen methods — none cross the intent wall

MethodSetupResultTakeaway
Linear probe, encoder sweepfrozen mean-pool + logistic regJulia 50.5 · roberta-base 59.2 · roberta-large 65.1 · SecureBERT 56.8 · CySecBERT 55.2 · UltraBERT-150M 60.4base size+quality drives the ceiling, not domain pre-training
Well-tuned probeModernBERT, 540 recs, 256 tok~69%frozen ceiling is higher than the early "58%" (data + tuning dependent)
Cheap frozen tricksdecomposition / distill / GLiNER / measured-labels−12 / −10.6 / −4 pp / neutralno frozen lever moves representation-bound cyber
Intent diagnosticflip accident↔attack, cos-distance0.029 vs 0.042 arbitrary (ratio 0.67)encoder barely responds to intent
Projection head / SupConfrozen + MLP + contrastive headlinear 57.5 · MLP 58.4 · SupCon 57.1a head only separates what the encoder already separated
NLI cross-encoderroberta-large-mnli, 6 hypotheses41.3% (hacking 16, error 10; social 88)joint scoring confirms the gap; surface classes OK
Rerankerms-marco MiniLM47.0%same as NLI
Context-passingclass defs packed into input, frozenplain 69.3 → ctx_all 66.4 / cls 46.1 / input 63.8information can't fix a capability gap (it hurt)

Fine-tuning — crosses the wall

MethodSetupResultTakeaway
LoRA (roberta-base)attn + classifier, r=16~70% (69.1 / 71.7)fine-tuning breaks the frozen ceiling
Cached-lower + top-N (ModernBERT)freeze+cache lower layers, train top-N, single seedhead 60.5 · top-4 71.1 · top-8 76.3 · top-12 71.7 · top-16 77.6top-4 is the knee; depth beyond = noise; caching lets it finish on CPU

Model soups

SoupIndividuals (best / mean)SoupTakeaway
Full-FT top-4 (3 seeds)67.8 / 65.465.1 (−2.6 vs best)full-FT does NOT soup (weights drift too far → soup = mean)
LoRA top-4 (3 seeds)75.0 / 73.577.0 (+2.0 vs best)LoRA soups cleanly (low-rank, shared basin)
LoRA top-8 (4 seeds)75.0 / 69.977.0 (+2.0 vs best, +7.1 vs mean)reproducible; depth doesn't help the soup; strong variance reduction

Decision: ship a single top-4 LoRA adapter (not the soup) — robustness comes from the K=3 prediction ensemble, not weight-averaging.

Calibration & economics

ItemResultTakeaway
Confidence — raw softmax63% of records claim ≥90% but that bin is 87.6% accurateraw softmax is overconfident — unusable as-is
Confidence — temp-calibrated (T=2.7)≥90%→100% (12%), 70–90%→92% (35%); confident slice ≥70% ≈ 47% @ ~94%calibration makes confidence honest; abstain gate works
LoRA-adapter reuse (HF search)only VERIS adapters found were 7B-welded (Mistral/Qwen)find-before-train usually comes up empty for bespoke small-encoder tasks
Timingtop-4 LoRA ~80–100s/seed CPU (Docker) + ~50s cache; ~30s on a T4; serve ~20–50ms/record CPUcompile cheap on GPU (pennies), serve cheap on CPU

Selective prediction: frozen vs fine-tuned (the key reframe)

The risk-coverage frontier — sort test by confidence, answer the most-confident X%, report accuracy on the answered set — exposes fine-tuning's real value, which overall accuracy hides. On a seed where overall accuracy is noise-tied, the fine-tuned model's confident slice is nearly double the frozen one's:

Max coverage at accuracy targetFrozen (no fine-tune)Fine-tuned (top-4 LoRA)
≥95%2%32%
≥90%37%61%
≥85%59%73%
overall (answer all)73.0%72.3%

Fine-tuning barely moved overall accuracy here (73.0 vs 72.3 — seed noise) but roughly doubled the confident slice (@90%: 37%→61%; @95%: 2%→32%). It sharpens the confidence ranking — the model knows which predictions to trust — which calibration (monotonic) cannot create. Implication: the escalation trigger should key off the confident-slice target, not overall accuracy — frozen suffices for “~37% @ 90%”, but fine-tuning is required for “60% @ 90%” or anything @ 95%. (Single seed; the confident-slice gap is the robust finding, exact coverages pending a 3-seed sweep.)

Maximizing the confident slice: the ensemble lever

cov@90 — how many records you can answer at ≥90% accuracy — is the product metric. A prediction-ensemble (average the softmax of K diverse top-4 LoRA models) widens it sharply, with essentially no accuracy change:

overall acccov@95cov@90
single model (avg of 4)~77%46.5%64.4%
ensemble (avg softmax of 4)78.0%63.8%83.0%
Δ+1pp+17.4pp+18.6pp

The ensemble beats the best single seed's cov@90 (67.4%) by +15.6pp — not noise-averaging. Overall accuracy barely moves (+1pp) but cov@90 jumps +18.6pp: ensembling is a confidence-ranking tool, and cov@90 is a ranking metric. Cost: K× inference (still local, ~$0, ~120ms/record at K=4; K is tunable). This is the lever for maximizing the confident slice — distinct from the weight-soup, which only moved accuracy. Two levers stack: more teacher data (data growth) grows cov@90 toward the plateau (measured 5%→25%→61% as data grew); the ensemble sharpens the ranking on top.

The two levers stack — cov@90 at each training-data size (K=3 ensemble, simulating data growth):

train sizesingle cov@90ensemble cov@90
603%1%
15023%39%
25042%48%
39957%79%

Data grows the slice (3%→57% single); the ensemble adds on top once there's enough data to train on (+5 to +22pp), negative only at the cold start (60 records — too undertrained to ensemble). Best cov@90 = full data + ensemble (79%), higher than either lever alone. Gains are noisy (single seed) but the stacking direction is clear: data growth does the heavy lifting, the ensemble (ranking) sharpens it on top.

Synthetic augmentation hurts (negative result). We tested whether LLM-generated synthetic data — grounded in real examples plus some random, balanced at 26/class (~29% of the training set) — helps the cyber task. It hurt: real-only 76.3% acc / 73.0% cov@90 → real+synthetic 74.3% / 55.9% (−2.0pp acc, −17.1pp cov@90). The mechanism is label leakage: the generator knows the class, so it writes the intent explicitly ("accidentally" for error), amplifying a cue that real, ambiguous incidents do not carry. The model learns the easy cue and its confidence ranking collapses on real data — exactly where cov@90 lives, hence the −17pp. Single-seed, but the direction is strong.

Data-provisioning policy. The teacher LLM labels real records (distillation); it never bulk-generates training data. Class imbalance is handled real-first: pull more real data of the minority class and label it; failing that, oversample or class-weight the real minority; synthetic is a last-resort bootstrap only for a near-empty class. PII masking at the teacher boundary uses GLiNER as the default masker (names, addresses, organizations), with a thin regex pre-pass for format-defined tokens (emails, cards, SSNs).

Key findings

  1. Capability gap vs information gap (the central result). Information-bound: the signal is in the frozen representation and just needs framing/labels — information methods help. Representation-bound (capability gap): the frozen model cannot perceive the distinction — and no information method helps. Eight tested (labels, definitions, decomposition, GLiNER, NLI, reranker, retrieval, context-passing); all at or below the frozen probe. Only weight changes (fine-tuning) cross it. The 0.029 intent-flip diagnostic quantifies it.
  2. Cached-lower + top-N fine-tuning is enough. Freeze+cache the lower layers, train only the top N on cached activations. top-4 is the knee; caching makes it finish on CPU and dodges OOM. Depth beyond is noise.
  3. LoRA soups; full fine-tuning doesn't. Averaging K adapters beats the best single seed by +2pp and erases seed noise — but only for LoRA (low-rank, shared basin). Full FT drifts too far → soup = mean. We ship a single adapter + gate-before-swap.
  4. Calibrated abstention is the product metric. Raw softmax is overconfident; temperature + conformal calibration makes confidence honest. Target ~90–100% reliability on the confident slice, not 90% on everything.
  5. Most tasks never need fine-tuning. Routing and tool-selection are information-bound → frozen clears the bar. The optimize flag gates the fine-tune tier (permit, not force) — a no-op on easy tasks.
  6. Economics. Compile once (GPU, pennies/seconds) → serve forever (CPU, ~$0, PII-local). The small-encoder choice (150M, not 7B) is what makes CPU serving viable.
  7. Fine-tuning's value is selective-prediction coverage, not overall accuracy. On a seed where overall accuracy is noise-tied (frozen 73% vs fine-tuned 72%), fine-tuning still roughly doubled the confident slice (@ ≥90%: 37%→61%; @ ≥95%: 2%→32%). It sharpens the confidence ranking, which calibration cannot.
  8. Ensembling maximizes the confident slice (cov@90), not accuracy. Averaging K diverse LoRA models' predictions widened cov@90 from 64%→83% (+18.6pp) and cov@95 +17.4pp, while overall accuracy moved +1pp — the confidence-ranking lever, matched to the selective-prediction metric. The two cov@90 levers: more teacher data (data growth) grows it toward the plateau; the ensemble sharpens the ranking. Cost: K× inference (tunable).
  9. At small scale, simple beats clever (three times). Weight-soup (dropped), context-passing (hurt), and smart active-learning acquisition (random won) all failed to beat the simple baseline on small/noisy data — the robust lever is always more data, not cleverer selection/averaging.

Limitations & honest ceiling

  • Numbers are cyber-specific, small-data, often single-seed. 540 records, single-label, ~±3pp seed noise. The direction (frozen caps, fine-tuning crosses, LoRA soups, calibration works) is robust; exact cell values are not.
  • 90% overall is likely a mirage. Capped by label noise (VERIS is human-coded), irreducible ambiguity (multi-action incidents, uninformative summaries), and possibly human inter-annotator agreement. Chasing it means fitting noise.
  • Confident-slice reframe. STROT targets reliability on what it answers, not blanket accuracy. More data (via the optimizer) widens the slice; it does not lift the hard-record ceiling.
  • No free-text reasoning at runtime. The battery is a frozen encoder; explanations are a compile-time artifact (the battery card), not a runtime feature.
  • Compute caveats. All runs were Docker-CPU (no GPU/MPS on the Mac), which inflated timing and forced the caching trick; small calibration bins make the 100% cell suggestive, not locked. A GPU multi-seed sweep is the next step to lock the numbers.

Reproducibility

  • Dataset: VERIS Community Database (VCDB), joined JSON; 6 VERIS action classes; single-action records with summaries > 40 chars; capped per class (45–90) for balance; 300–540 records per run.
  • Models: answerdotai/ModernBERT-base (150M, fine-tune base); FacebookAI/roberta-base (LoRA proxy that finishes on CPU); roberta-large-mnli (NLI); cross-encoder/ms-marco-MiniLM-L-6-v2 (reranker); Julia-1, SecureBERT, CySecBERT (frozen comparison).
  • Fine-tune config: cached-lower + top-N; LoRA r=16, alpha=32, lr 1e-3, 6 epochs, max_len 128; adapter on the top layers only; splits ~60/14/26 train/cal/test; seeds 11/23/37/51.
  • Calibration: temperature grid-search on a held-out cal split (minimize NLL); conformal cutoff for a target risk.
  • Code: benchmarks/*.py — cyber_encoders, cyber_frozen_contrast, cyber_crossencoder, cyber_context_passing, cyber_cached_top, cyber_soup_lora, cyber_confidence_dist, cyber_lora. All runs in Docker (strot-julia image, CPU; no GPU/MPS).
  • Open next: GPU multi-seed sweep to lock the depth/soup numbers; a second task (routing or tool-selection) to confirm the frozen path suffices; richer input than one-line summaries.
ShareX ↗LinkedIn ↗Hacker News ↗
// comments
Comments load once configured.Giscus (GitHub Discussions) is scaffolded here — enable Discussions on the repo, install the giscus app, and fill in the repo/category IDs in components/comments.tsx.