This page explains every concept behind slm-125m in plain English — what pretraining actually is, why the dataset looks the way it does, how the tokenizer works, how to read the training curve, and how the whole thing is deployed on Modal and Vercel. No prior ML background assumed.
A language model is, mechanically, a function that takes some text and predicts what word (technically, token — more on that below) comes next. Pretraining is the process of teaching that function to be good at this prediction, by showing it an enormous amount of real text and letting it guess, over and over, billions of times.
Every one of the model's 125,847,552 parameters starts as a small random number. Before training, the model has never seen a single sentence — it knows nothing about English, law, or finance. It guesses randomly, and it's wrong in a very specific, measurable way: it assigns roughly equal probability to all 16,384 possible next tokens, which is exactly the worst-case starting point.
Repeat that loop 3,889 times (this project's entire run), each time over a fresh batch of 524,288 tokens, and the "nudges" accumulate into a model that has implicitly learned grammar, facts, and style — not because anyone told it the rules, but because predicting the next token well requires learning them.
This is a base model — it only ever did the loop above. It was never shown "here's a question, here's a good answer" pairs (that additional stage is called instruction-tuning), and it never got human feedback on which responses are better (RLHF). So it doesn't answer questions — it continues text, the way autocomplete does, just far more fluently. Type "The plaintiff shall" into the live demo and it'll keep writing in that style, not respond to it as an instruction.
Pretraining is show, don't tell at an enormous scale — the model never receives a rule of grammar or a legal definition directly. It only ever gets one signal, repeated billions of times: "here's some text, predict what's next," followed by "here's how wrong you were." Everything it appears to "know" is a side effect of getting extremely good at that one narrow task.
For comparison, GPT-3 has 175 billion parameters — about 1,400× larger. This project deliberately stays small: it trains in under 3 hours on a single GPU for about $12, which makes the entire pipeline (data → clean → dedup → tokenize → train → deploy) something one person can run end to end and afford out of pocket.
Six phases, each one a small, cheap, resumable cloud job — not one long fragile script.
Stream three public datasets, filter out noise, remove near-duplicates, and strip anything that overlaps with the benchmarks we'll later use to judge the model.
Train a custom vocabulary from scratch on the cleaned text, then convert every document into the fixed-length integer sequences the model actually trains on.
Run the pretraining loop on a single GPU, then push the finished weights to Hugging Face and stand up a live inference endpoint.
The specific bet this project makes: instead of a generic model trained on a random scrape of the internet, train one almost entirely on US case law and SEC filings — two dense, high-quality, domain-specific text sources — with just enough general web text mixed in for fluency. See the dataset section for why that mix isn't 70/20/10.
Three public, ungated sources, streamed directly from Hugging Face — never fully downloaded to disk:
HFforLegal/case-law — 282,390 US court opinions. Some are scanned documents with OCR noise, which is why this source alone gets an extra cleaning gate (below).PleIAs/SEC — 48,543 filings (10-Ks and similar), born-digital so far cleaner per document than the scanned case law.HuggingFaceFW/fineweb-edu — general educational web text, used only as a small fluency top-up, not the backbone.
The original idea was 70% case law, 20% SEC, 10% web. That's arithmetically impossible:
measuring actual clean-token yield showed case law tops out around 0.81B
tokens and SEC around 1.16B — together, about 2 billion tokens. You
cannot make case law 70% of a 10-billion-token corpus when the entire source doesn't
contain that much text. fineweb-edu, by contrast, is
effectively bottomless (~11.67B tokens available in the sampled slice alone).
So the actual strategy — take all of both legal sources, and add just enough web text to round things out — produces roughly 40% case law, 40% SEC, 20% web by design, landing at 35/42/23 after cleaning and dedup removed slightly different fractions from each source.
Every document passes through the same deterministic chain before it's allowed into the corpus:
Two more passes happen after cleaning. Exact deduplication hashes every document and drops byte-for-byte repeats. Near-duplicate detection (via MinHash, run on case law specifically) catches documents that are mostly the same text with minor edits — the kind of near-copy exact hashing would miss.
Then, critically: a 13-word overlap check against two legal benchmarks (CaseHOLD and LexGLUE) strips out any training document that shares a long verbatim chunk with those test sets. Without this, the model could effectively "memorize the answer key" and score well on an eval it was never supposed to have seen — a problem called test-set contamination. This run removed about 24,000 case-law documents on that basis alone.
| Stage | Tokens kept |
|---|---|
| Raw, cleaned | 2.68B |
| + deduped + decontaminated | 2.40B |
| + tokenized & packed | 2.04B |
Every drop reason is logged, not silent — this is what lets you audit exactly why a document was excluded rather than trusting a black box.
A neural network doesn't read letters — it reads numbers. Tokenization is the step that converts text into a sequence of integers, and byte-pair encoding (BPE) is the specific algorithm used here to decide what counts as one "unit."
The naive options are both bad: split by character and sequences become enormous (slow, hard to learn long-range patterns); split by whole word and you need a vocabulary of millions of entries plus a fallback for every word you've never seen. BPE finds a middle ground — common enough to be efficient, small enough to handle any input, including words the tokenizer has never encountered.
Start with every input broken into individual bytes. Repeatedly find the most frequent adjacent pair of units in the training corpus and merge it into one new unit. Do this thousands of times. Common sequences — "tion", "the", whole short words — end up as single tokens; rare or unfamiliar text falls back to smaller pieces or individual bytes, so nothing is ever "unrepresentable."
(A simplified illustration — the real algorithm operates over the whole training corpus at once, at byte level, not word-by-word like this.)
The tokenizer isn't a pretrained, off-the-shelf one — it's trained from scratch (Phase 3) on this project's own cleaned corpus, so its vocabulary reflects legal and financial language specifically, not generic internet text. It's byte-level (the starting alphabet is raw bytes, not Unicode characters), which guarantees any input — any language, emoji, or malformed text — can always be encoded, with a vocabulary of exactly 16,384 entries.
>>> tok.encode("The plaintiff shall bear the
burden of proof by a preponderance
of the evidence.")
15 tokens
>>> tok.encode("The Company's net revenues
increased 12% year over year
pursuant to the agreement.")
16 tokens
>>> tok.decode(tok.encode(text)) == text
True # lossless roundtrip, always
Bigger vocabularies mean fewer tokens per sentence (faster, more efficient) but a larger embedding table to train. At 125M total parameters, a huge vocabulary would eat a disproportionate share of the model's capacity. 16,384 is sized for this model, not copied from a larger model's defaults.
Every training step, the model produces a probability distribution over all 16,384 possible next tokens. Loss (specifically, cross-entropy loss) measures how much probability it assigned to the token that actually came next — low probability on the right answer means high loss, high probability means low loss.
Perplexity is the same number, just more intuitive: it's
e^loss, and it roughly means "the model is as confused as if
it were guessing uniformly among this many options." A perplexity of 10 means: about as
uncertain as picking correctly out of 10 equally likely choices.
ln(16384), the mathematical signature of a freshly-initialized model with no learned signal at all. Seeing this number confirms the model started correctly, before a single dollar was spent training further.
The step size the model takes on each update (the learning rate) isn't
constant. It ramps up linearly for the first ~381 steps (warmup — starting at
full speed on random weights tends to destabilize training), peaks at
6e-4, then decays smoothly along a cosine curve down to
6e-5 by the final step — large careful steps early, small
precise ones as the model converges.
One epoch means one full pass through all 2.04B training tokens. This run did exactly one — the corpus is small enough, and Chinchilla scaling research suggests roughly 20 tokens per parameter is compute-optimal for a model this size; this run landed at about 16.2, just under that line.
1× H100 GPU, ~2.85 hours, ~$12–13 for this phase — the original design defaulted to 8 GPUs, which would mostly sit idle at this model size. See the Modal section for why.
Every compute step in this project — the CPU data pipeline, GPU training, and the live inference endpoint powering the demo — runs on Modal.
Modal runs Python functions in the cloud without you managing any servers. You write a normal function, add a decorator specifying what it needs (which packages, how much CPU/GPU/memory, how long it's allowed to run), and calling it from your laptop executes it on Modal's infrastructure instead of yours — output streams back to your terminal like it ran locally.
modal.App("slm-125m"))..pip_install(...) calls — Modal builds and caches it automatically.slm-125m) that every phase reads from and writes to — the cleaned corpus, the tokenizer, the checkpoints all live here.@app.function(...) — runs on Modal's cloud with whatever resources you specified.# app.py import modal app = modal.App("hello") image = modal.Image.debian_slim().pip_install("numpy") @app.function(image=image) def double(x: int) -> int: import numpy as np return int(np.array(x) * 2) @app.local_entrypoint() def main(): print(double.remote(21)) # runs in the cloud
Sign up at modal.com, then pip install modal and run modal token new (opens a browser to authorize) — or create an API token in the dashboard and run modal token set --token-id ak-... --token-secret as-... for a non-interactive setup.
Decorate any Python function with @app.function(image=..., volumes=..., gpu=..., timeout=...). Only what you specify gets provisioned — CPU-only and free-tier-cheap unless you ask for a GPU.
modal run app.py::main executes the local entrypoint, which can call your cloud functions with .remote() (blocks and waits) or fan out many in parallel with .starmap(work).
--detachmodal run --detach app.py::main submits the job and disconnects your terminal from it entirely — essential for multi-hour training jobs, since without it, your laptop losing wifi kills the remote job too. This project's own training run hit that failure twice before switching to this pattern.
modal deploymodal run is ephemeral — it stops when your local command exits. modal deploy app.py creates a persistent app with a stable URL, which is how this project's live inference endpoint (https://...--slm-125m-web.modal.run) stays up indefinitely, scaling to zero cost when idle.
The exact commands used for every phase of this specific project — with expected output at
each step — are documented in the project's docs/PRETRAINING.md
and docs/HUGGINGFACE_DEPLOYMENT.md.
This site (the one you're reading right now) is static HTML/CSS/JS with no build step, deployed to Vercel.
The "Try it" demo on the homepage doesn't route through Vercel
at all — the browser calls the Modal inference endpoint directly from
client-side JavaScript. This works because the Modal endpoint sends
Access-Control-Allow-Origin: * (a CORS header granting
cross-origin browser access), so no proxy server is needed in between. The result: this
entire site is just files, no server code of its own to maintain.
npm install -g vercel
# from inside the site's own directory
vercel link --yes --project my-site
vercel deploy --prod
Vercel dashboard → Settings → Tokens → Create Token. This authenticates the CLI without needing an interactive browser login every time.
vercel link --yes --project <name> — first run creates the project, later runs just reconnect. Run it from the actual site directory, not a parent folder with unrelated files in it.
vercel deploy --prod uploads the files and returns a live URL — plus a stable alias (like slm-125m-phi.vercel.app) that stays constant across every future deploy, so links you've shared keep working.