125.8M parameters. No pretrained checkpoint, no distillation — every weight was initialized fresh and trained on 2.04 billion tokens cleaned, deduplicated, and decontaminated from three public sources. This page is the full paper trail.
This is the raw base checkpoint — one epoch, no instruction-tuning, no RLHF, no safety filtering beyond what the training data itself excluded. It completes text the way a language model from 2019–2020 would; it does not follow instructions or answer questions directly. Numbers and case names it produces are learned style, not retrieved fact — treat everything it writes as fabricated. Served live from a CPU container on Modal; first request after idle may take 10–20s to cold-start.
No architectural novelty here on purpose — a plain decoder-only transformer, so every result below is attributable to the data and training recipe, not a clever architecture.
The two legal sources only contain about 2 billion clean tokens between them — nowhere near enough for a 70% share of a 10B-token corpus. So the recipe takes all of both legal sources and tops up with a capped slice of web text.
| Source | Rows | Field | Est. available | Token cap |
|---|---|---|---|---|
| US case law HFforLegal/case-law | 282,390 | document | ~0.81B | 1.0B |
| SEC filings PleIAs/SEC | 48,543 | text | ~1.16B | 1.3B |
| fineweb-edu sample-10BT | 9,670,000 | text | ~11.67B | 0.5B |
Line-level noise filtering, boilerplate stripping (10-K headers, page numbers, SEC letterhead), 4-gram repetition detection, ASCII/langdetect English gate, and a dictionary-based OCR-garble gate for scanned case law.
BLAKE2 exact-hash dedup, MinHash/LSH near-duplicate removal on case law (threshold 0.8), and a 13-gram contamination strip against the CaseHOLD / LexGLUE eval sets.
A fresh byte-level BPE tokenizer trained on the cleaned corpus, then packed into 1,024-token windows with an <|eos|> between documents, every 100th window routed to validation.
8× fewer GPUs than the initial default — at 125M parameters the model can't keep a larger cluster fed, so the extra GPUs would mostly sit idle at higher cost.
Built with Modal (CPU data pipeline, GPU training, and the CPU endpoint powering the demo above), weights hosted on Hugging Face, this page on Vercel. Tokens seen ≈ 16.2× parameter count — just under the ~20× Chinchilla-optimal ratio for a single epoch over this corpus.
A full fine-tune of the base checkpoint above on 10,000 grounded question-answer pairs — every weight updated, no adapters. Below: the live model, exactly how its training data was built, and what actually happened when it trained (including a real overfitting bug caught mid-run).
This model was fine-tuned closed-book — it never sees the source passage, only the question, at training or inference time. At 125.8M parameters it doesn't have the capacity to reliably memorize facts from a 2.4B-token corpus, so treat specific names, dates, and figures as unverified — the improvement here is in domain style and answer structure, not factual recall. Served live from a second CPU container on Modal; first request after idle may take 10–20s.
Not new data — the same cleaned, deduped corpus from the pretraining pipeline, turned into question-answer pairs by an LLM teacher, then filtered hard.
~300-character passages sampled from the corpus, proportional to the realized pretraining mix (case law 35% / SEC 42% / web 23%), so the QA data reflects the same domain balance.
3 grounded QA pairs requested per passage, run as parallel real-time calls fanned out across Modal workers (a Batch API attempt stalled with zero visible progress for over an hour and was cancelled). 100% success rate, 0 failures across 4,761 chunks.
Word-overlap grounding check against the source passage, exact-match dedup on normalized questions, then embedding-similarity near-dup removal (cosine ≥ 0.92) via local sentence-transformers — no extra API cost.
| Filter step | Removed | Remaining |
|---|---|---|
| Parsed from teacher output | — | 14,283 |
| Not grounded in source passage | −18 | 14,265 |
| Exact-duplicate questions | −454 | 13,811 |
| Near-duplicate (embedding sim ≥ 0.92) | −550 | 13,261 |
| Randomly trimmed to target size | −3,261 | 10,000 |
Full fine-tune, not LoRA — at 125.8M params there's no memory problem adapters need to solve. The interesting part isn't the setup, it's what the validation loss revealed mid-run.
The run was configured for 3 epochs (918 steps). Train loss fell smoothly the whole way, down to ~1.0 — which looked great until the validation curve told a different story: val loss bottomed out at step 300, under one full epoch, then rose for the remaining two-thirds of training as the model started memorizing the training set instead of generalizing. The first run shipped the last-step checkpoint by default and would have deployed the overfit model. The code was changed to track and save whichever checkpoint had the lowest validation loss during training, and the run was repeated — the shipped model above is from step 300, not step 918.
Five held-out validation questions the model never trained on — expected answer (from the Gemini teacher) next to what the fine-tuned model actually generates.
| Question | Expected | Generated |
|---|