jcardena.com › Blog › Model Quantization: the Ocean and the Crystal
184 posts
EN ES

Model Quantization: the Ocean and the Crystal

AI

Quantization isn't 'making models smaller', it's cashing in over-parameterization slack to fit a giant mind in a small machine. Why it works, where it silently breaks, why the frontier won't ship it.

Model Quantization: the Ocean and the Crystal
Field note. A deep-read from an enterprise architect, what model quantization actually is, why it works, where it silently breaks, and the one question that cuts through the hype: if it works, why isn't it the standard?

Everyone explains quantization the same way: "it makes the model smaller." That is true the way "a diet makes you lighter" is true, accurate, and almost completely beside the point. After years of putting models into production, I have come to believe the smallness is a side effect. The real act is stranger, and more interesting.

Here is the claim I want to defend: quantization is the act of cashing in a model's over‑parameterization slack. Training produces far more numerical precision than any single task actually uses. Quantization spends that surplus, deliberately, sometimes painfully, in exchange for memory, speed, and cost. It is not compression of intelligence. It is the harvesting of a margin that was sitting there unused.

The poet in the datacenter

Picture the full‑precision model, 32‑bit or 16‑bit floating point, as a master poet. It has a million‑word vocabulary and can express infinitesimally fine shades of meaning. But it lives in a vast, energy‑hungry library (a datacenter), and it takes a long time and a lot of power to form a single sentence.

We need that intelligence out in the world, on phones, in cars, on a single consumer GPU, responding instantly, cheaply. So we send the poet to a brutal linguistic bootcamp and force it to learn a 16‑word field dialect: a handful of allowed values instead of billions. The poet becomes a field engineer. It can still build the bridge. It just can't be as eloquent about it internally.

Why does this matter now? Because somewhere around 2020 the economics flipped. The marginal cost of running a model, tokens per second, watts, dollars, every single time it answers, overtook the cost of training it once. The dream became simple and universal: keep the intelligence, shrink the machinery.

A vast dark space filled with flowing rivers of warm liquid light.
The Ocean of Thought: full precision is fluid, warm, able to take any shape.

Translating thought into integers

Concretely: a neural network is mostly an enormous table of numbers. In FP16, each of those numbers can take about 65,000 possible values, a smooth, near‑continuous range. Quantize to INT4 and each number can take exactly 16. You take the continuous range, draw a coarse grid over it, and snap every value to the nearest grid line.

The whole idea, in one line

smooth range (~65,000 values)  →  coarse grid (16 values)  →  snap each weight to the nearest line

A 70B model: 140 GB in FP16 → 35 GB at INT4 → ~18 GB with aggressive mixed precision. A giant mind, folded into a small machine.

That is the entire trick. The surprise isn't that it saves memory, of course it does. The surprise is that the model still works after you've thrown away that much numerical detail. Why?

Why it survives the crush

A large network tolerates having its numbers brutally rounded for three reasons, and you can hold all three intuitively.

1. Massive redundancy. The model is wildly over‑parameterized. It has learned hundreds of overlapping ways to represent the same concept, call it 500 different words for "blue." Quantization takes away 490 of them. But "blue" is encoded so redundantly across the network's structure that the remaining ten still carry it. The meaning lives in the pattern, not in any one perfect number.

2. Robustness to noise. Quantization adds a little static to every parameter. But a well‑trained model's core logic is like a clear voice shouted across a room, a bit of static doesn't stop you understanding the message. Training itself was noisy (dropout, imperfect data, approximate optimization), so the network already learned to be robust.

3. Relative weights, not exact ones. What drives a model's decisions isn't a parameter's exact value (0.834512) but its relationship to its neighbors, much larger, a little smaller, about the same. Like a sports team: what matters isn't each player's height to the millimeter, but that the center is much taller than the point guard. Quantization preserves that ranking, and the ranking is what makes decisions.

The untranslatable word

Now the honest half. The translation breaks, and it breaks in specific, knowable places.

The precision cliff. Above ~4 bits, you're usually fine. With naive rounding, somewhere below ~3 bits the degradation stops being gradual and falls off a cliff, you run out of words. Imagine a dialect where "sad," "angry," and "afraid" all map to one word: "bad." The grid gets so coarse that fundamentally different concepts fall into the same bucket, and the logic collapses. (The good methods, below, push that cliff lower, and exactly where it sits depends on the task, not just the bit‑width.)

The outlier catastrophe, the untranslatable word. This is the killer. Every language has a few crucial words with no equivalent elsewhere (the German Schadenfreude). In a network, these are outliers: rare activations or weights with enormous magnitude that encode a critical feature. Your 16‑word dialect has no slot for them. The outlier gets rounded to something mundane, and a whole line of reasoning is silently destroyed. This is one of the central causes of catastrophic failure in aggressively quantized models, especially in low‑bit regimes, and it's why the good methods (GPTQ, AWQ, SmoothQuant) are really just clever ways to protect the outliers.

Layer sensitivity, and the soul test. The first and last layers are like a book's opening and conclusion, they need precision; crush them and you've replaced the nuanced ending with "and they lived happily ever after." And even when the benchmarks pass, something subtler can be lost: a quantized model can write fluent, statistically plausible text (low perplexity) while having lost its spark. It passes the grammar test and fails the soul test.

A pulse of light striking a vertex of a crystal lattice and a crack spiderwebbing through it.
The flaw in the crystal: an outlier strikes a vertex the lattice can't hold, and a silent crack spreads.

Two ways to freeze it

For a practitioner, one choice matters more than any other: how you freeze. Post‑training quantization (PTQ) is the quick freeze, take a fully trained model and apply a clever algorithm to crush it, no further training. Fast, cheap, and what almost everyone running a local model uses. The good PTQ methods, GPTQ (which uses second‑order information to compensate each weight as it rounds it) and AWQ (which protects the small fraction of weights tied to the largest activations), are really just smart, data‑aware ways to protect the outliers. Quantization‑aware training (QAT) is the born‑crystalline route: you simulate the precision loss during training, so the model learns to be robust to it from the start. Expensive and slow, what you reach for when the deployment is serious.

In practice you'll meet this as formats and tools: GGUF (llama.cpp) for local, GPTQ/AWQ for served inference, NF4 (bitsandbytes / QLoRA) for cheap fine‑tuning, and FP8, which is mostly a training‑and‑accelerator‑native format, not the same thing as 4‑bit weight‑only inference. One more reality the guides skip: in long‑context apps the real memory hog often isn't the weights but the KV cache, and quantizing that (e.g. FP8 KV) is where a lot of the modern savings actually come from. And a note for the precise: the model we're usually "freezing" today isn't pure FP32, it's already BF16. FP32 is the poetic ideal; BF16 is the ocean we actually start from.

If it works, why isn't it the standard?

This is the question that cuts through everything, and I owe it a straight answer. If you can crush a model to 4 bits and keep the intelligence, why don't the frontier labs just ship quantized models by default? Their behavior is the tell.

Look at the actual split. The frontier API models, the flagships from the big labs, are served at carefully validated BF16 or FP8 (still reduced precision, but a tightly controlled kind), not casual commodity INT4. The 4‑bit world is overwhelmingly local and open models, run on consumer hardware. That divide is not an accident. It is a near‑perfect market signal of the real trade‑off.

The frontier labs sell excellence and reliability. Their whole promise is "the best, most capable model on the planet." For them a 1% quality drop is a business catastrophe, and they serve millions of unknown workloads, they can't tell whether you're writing low‑stakes poetry or doing high‑stakes financial analysis, so they must serve the highest‑fidelity version to everyone. They cut cost with scale and serving infrastructure, never by degrading the core asset.

The local and open‑source world is solving a different problem entirely: access. The goal isn't perfection, it's running a 70‑billion‑parameter model at all on a single 24 GB GPU. The alternative to a quantized model there isn't a full‑precision model. It's no model.

The reveal: quantization is a tool for managing a resource constraint, VRAM, bandwidth, not a free‑lunch efficiency win. The frontier doesn't have that constraint the same way, so it refuses to pay the quality tax. When intelligence is the product, you don't compress the product.

The dark geometry

So here is the architect's view, the failure modes that never show up on a benchmark, because benchmarks (MMLU, GSM8K, perplexity) are short, single‑turn, and over‑represented in training. They survive 4‑bit beautifully. That's exactly why "it works." The damage hides everywhere they don't look.

  • Silent, spiky‑capability loss. Intelligence is emergent from trillions of high‑precision interactions. The first things to blur are the most emergent, "spiky" abilities, the "aha!" leaps, the clever trick. The model becomes more of an aggregator and less of a reasoner, and a single‑turn test will never see it.
  • The "it passed my tests" trap. Your eval suite runs green, you deploy, and two weeks later a customer finds the model can no longer parse addresses containing the word "West," because the neuron cluster for that nuance was damaged by a rounding error. Your tests are almost never comprehensive enough to find these.
  • Cascading reasoning errors. A tiny rounding in step 1 (4.99 becomes 4 instead of 5) compounds into a catastrophic logical failure by step 5. Long chains, long context, and agentic tool‑use loops degrade far faster than single‑shot accuracy suggests. The demo passes; the deployment silently regresses on the tail.
  • It's a triplet, not a property. You can never say "this model is safe to quantize." You can only say "this model, with this method, for this workload, is probably safe." The same 4‑bit model can be fine for chat summaries and a disaster for contract analysis.
  • The real economics. The calculation isn't "GPU savings > 0." It's "GPU savings > probability of error × cost of error." For an enterprise, the cost‑of‑error term, lost customers, lawsuits, brand, can make the GPU savings look like pocket change.
  • Alignment blunting, the one that should scare you. Alignment is a delicate, fine‑tuned property resting on subtle distinctions ("describe violence in a novel" vs. "give instructions for violence"). Quantization can blunt that fine‑tuning, weakening refusals and raising jailbreak susceptibility. You may be saving VRAM at the cost of your safety guardrails.

The MP3 and the WAV

If you remember one thing, remember this. Quantization is exactly like audio compression.

A 128 kbps MP3 is "basically free" and perfectly fine for a podcast while you jog. The convenience is enormous and the quality loss is irrelevant to that task. That is the world of local models and low‑stakes applications. But an audio engineer mastering an album would be fired for using an MP3, their job depends on hearing every imperfection, so they need the full 24‑bit WAV. That is the world of the frontier and high‑stakes enterprise.

Same compression. Opposite verdict. Nothing about the compression changed, only the cost of the detail you lost. Which is why the real question was never "does quantization work?" It is: "can you afford the risk of the part that silently doesn't?"

Trust it for

Personal use, experiments, creative work, when you're the expert in the loop, when the output is a suggestion not a decision, when being slightly wrong costs nearly nothing.

Do NOT trust it for

Unattended production workflows · anything with numbers, code, or precise facts · the final product delivered to a customer · anytime safety, security, or legal/financial liability is on the line.

One more honesty: the ocean isn't sacred

It would be easy to leave you thinking the full‑precision model is a perfect, godlike original and quantization is a fall from grace. That's a trap, and I'd be lying by omission if I didn't name it. The FP32 model is itself a flawed, biased, incomplete approximation, it has its own hallucinations and failures. The goal of engineering was never fidelity to a flawed source. It is the most useful model you can field within a budget. Sometimes the honest move is to quantize and accept the trade. Sometimes it's to refuse. The point is to know which one you're doing, and why.

Everyone wants the free lunch: the full genius, on a phone, instant, free. What you actually get is a negotiated settlement across three axes, fidelity, performance, and reliability, and you cannot maximize all three. Quantization doesn't remove the model's limits. It moves them: from the obvious wall of size to the hidden wall of your specific distribution. Spend the slack where you can afford to. Guard it where you can't.

A human silhouette standing between a flowing ocean of light and a compact cracked crystal.
The architect between the Ocean and the cracked Crystal: a tool, not a perfect replica.

So, is quantization good, or not?

Think of a photo on your phone. For texting a friend or posting online, it's perfect, small, fast, looks great. That's quantization at its best: cheap, quick, and good enough for almost everything you do in a day.

But blow that same photo up to a giant poster, and suddenly you see the blur, the missing detail. The flaw was always there. You just didn't notice until you gave it a job that mattered.

That's the whole story. Quantization is good when "good enough" really is enough, a draft, a summary, a quick answer you'll check yourself. It's not good when a hidden mistake could quietly cost you and no one is there to catch it.

So don't ask whether it works. It works. Ask the one question that counts: can you afford the part that silently doesn't? Shrink the photo to text your friends, just don't use it to print the billboard.

, Juan Cardena

Companion film, The Ocean and the Crystal, and the field guide of the same name.

JC
Juan Cardena
Enterprise AI & Agentic Systems Architect

Enterprise architect with 25 years across web, software, data, and AI. MIT CDAO ’25. Writing on agentic AI in production.