A jcardena.com field guide

The Crystal
and The Ocean

A field guide to model quantization.

Quantization isn't "making models smaller." It's cashing in a model's over-parameterization slack, forcing a poet who thinks in calculus to speak in integers, so the genius can leave the datacenter and run on your phone. This is the honest guide: why it works, where it silently breaks, the methods, the decision rules, and the dark geometry, including why the frontier won't ship it.

Juan Cardena
Enterprise architect · 25 years building data, software & AI systems

Contents

  1. The Poet in the Datacenter
  2. Translating Thought into Integers
  3. The Ghost of Redundancy
  4. The Untranslatable Word
  5. The Quantizer's Toolkit
  6. The Art of the Trade-Off
  7. Dark Geometry
  8. The Road Ahead

Chapter One

The Poet in the Datacenter

Everyone explains quantization as "making the model smaller." That is true the way "a diet makes you lighter" is true, accurate, and almost beside the point. The smallness is a side effect. The real act is the harvesting of a margin that was sitting there unused.

Picture the full-precision model, FP32 or FP16, as a master poet: a million-word vocabulary, capable of infinitesimally fine shades of meaning. But it lives in a vast, energy-hungry library (the datacenter) and takes time and power to form a single sentence. We need that intelligence out in the world: on phones, in cars, on a single consumer GPU, instant and cheap. So we send the poet to a brutal bootcamp and force it to speak a 16-word field dialect. The poet becomes a field engineer. It can still build the bridge; it just can't be as eloquent about it internally.

Around 2020 the economics flipped: the marginal cost of running a model overtook the cost of training it. The dream became universal, keep the intelligence, shrink the machinery.

Chapter Two

Translating Thought into Integers

A neural network is mostly an enormous table of numbers. In FP16, each number can take about 65,000 values, smooth, near-continuous. Quantize to INT4 and each takes exactly 16. You draw a coarse grid over the continuous range and snap every value to the nearest line. That is the entire mechanism.

The prize, concretely
A 70B model: 140 GB (FP16) → 35 GB (INT4) → ~18 GB (aggressive mixed precision). A giant mind, folded into a small machine. Quantization can be weight-only (the safe default, shrink the stored weights, compute in higher precision) or weight-and-activation (more savings, more danger).

The surprise is never that it saves memory. It is that the model still works after you throw away that much numerical detail. The next two chapters are why, and why not.

Chapter Three

The Ghost of Redundancy

A large network tolerates being crushed for three reasons you can hold intuitively.

1 · Massive redundancy

The model is wildly over-parameterized, hundreds of overlapping ways to represent the same concept, 500 words for "blue." Quantization takes 490 away; "blue" is encoded so redundantly that the remaining ten still carry it. The meaning lives in the pattern, not in any one perfect number.

2 · Robustness to noise

Quantization adds static to every parameter. But the model's core logic is a clear voice shouted across a room, a little static doesn't stop the message. Training itself was noisy (dropout, imperfect data), so the network already learned to be robust.

3 · Relative weights, not exact ones

What drives decisions isn't a parameter's exact value but its relationship to its neighbors. Like a team: not each player's height to the millimeter, but that the center is much taller than the point guard. Quantization preserves that ranking, and the ranking is what decides.

The "magic" of quantization is really a property of the original model's scale. You aren't compressing intelligence; you're cashing in the slack that scale created.

Chapter Four

The Untranslatable Word

Now the honest half. The translation breaks in specific, knowable places.

The precision cliff

Above ~4 bits you're usually fine. Below ~3 bits the degradation isn't gradual, it's a cliff. You run out of words: a dialect where "sad," "angry," and "afraid" all map to "bad." The grid gets so coarse that different concepts fall in the same bucket and the logic collapses.

The outlier catastrophe

This is the killer. Every language has a few crucial words with no equivalent (Schadenfreude). In a network these are outliers: rare activations or weights of enormous magnitude that encode a critical feature. Your 16-word dialect has no slot for them; the outlier rounds to something mundane and a whole line of reasoning is silently destroyed. The single most common cause of catastrophic failure. Every good method exists to protect the outliers.

Layer sensitivity, and the soul test

The first and last layers are a book's opening and conclusion, crush them and you've replaced the nuanced ending with "happily ever after." And even when benchmarks pass, the model can lose its spark: fluent, plausible text (low perplexity) with the life gone out of it. It passes the grammar test and fails the soul test.

Chapter Five

The Quantizer's Toolkit

The methods, jargon kept to a minimum.

The two regimes
  • PTQ (post-training quantization): quantize an already-trained model. Fast, cheap, no retraining, but fragile if naive.
  • QAT (quantization-aware training): train (or fine-tune) the model knowing it will be quantized. Expensive and slow, but the most robust quality.

The outlier-aware wave (this is most of the field)

Formats you'll meet

The practitioner's compass
  • Run it locally: llama.cpp + GGUF (the k-quants), Ollama, LM Studio.
  • Quantize it: AutoGPTQ / AutoAWQ, bitsandbytes (NF4/INT8), Hugging Face transformers.
  • Serve it fast: vLLM, TensorRT-LLM, TGI, the runtime decides whether your bit-width is actually accelerated on your GPU (INT4/FP8 need kernels).
  • Don't forget the KV cache. In long-context apps the memory bottleneck is often the KV cache, not the weights, quantizing it (e.g. FP8 KV) is where much of the modern savings comes from. And note: most models you "freeze" today are already BF16, not FP32.

Chapter Six

The Art of the Trade-Off

There is no "best" quantization, only the right one for a (model, method, task) triplet. You are negotiating across three axes you cannot all maximize: fidelity (truth to the original), performance (speed/size/cost), and reliability (failure rate on edge cases).

The decision rules
  • Quantize freely when: the model is huge + over-parameterized (70B+), the task is statistically robust (summary, classification, simple RAG), a human is in the loop, and the cost of a subtle error is low. 8-bit and 6-bit here are nearly free.
  • Be very careful when: a small/specialized model (7B), aggressive bits (2–3), multi-step reasoning, agentic or tool-use loops, or anything with numbers, code, or precise facts.
  • The economics: the test isn't "GPU savings > 0." It's "GPU savings > probability of error × cost of error." In an enterprise, cost-of-error can dwarf the GPU savings.
  • Always validate on YOUR distribution, not just public benchmarks. The same model is safe for one workload and a disaster for another.

Chapter Seven

Dark Geometry

If quantization works, why isn't it the standard? Why won't the frontier labs ship it by default? Their behavior is the tell, and it is the most honest thing in this book.

The split is no accident: frontier API models are served at BF16/FP8; the 4-bit world is overwhelmingly local and open. The frontier sells excellence and reliability, a 1% quality drop is a business catastrophe, and they serve millions of unknown workloads, so they must serve the highest-fidelity version to everyone. The local world is solving for access: the alternative isn't a full-precision model, it's no model on a 24 GB GPU.

The reveal: quantization is a tool for managing a resource constraint, not a free-lunch efficiency win. When intelligence is the product, you don't compress the product.

The silent killers

The MP3-vs-WAV verdict
A 128 kbps MP3 is "basically free" and fine for a podcast on a jog (local, low-stakes). An audio engineer mastering an album would be fired for using an MP3, they need the 24-bit WAV (frontier, high-stakes). Same compression, opposite verdict, because the cost of the lost detail changed, not the compression. Quantization isn't "does it work?" It's "can you afford the risk of the part that silently doesn't?"

And don't romanticize the Ocean. The full-precision model is itself flawed, its own hallucinations, biases, failures. The goal was never fidelity to a perfect source; it's the most useful model you can field within a budget. Sometimes the honest move is to quantize and accept the trade; sometimes it's to refuse. Know which one you're doing.

Chapter Eight

The Road Ahead

Quantization is one lever among several for the same goal, useful intelligence within a budget, and it's increasingly combined with the others:

The one thing to carry out
Quantization doesn't remove a model's limits, it moves them: from the obvious wall of size to the hidden wall of your specific distribution. Spend the slack where you can afford to. Guard it where you can't.
The Crystal and The Ocean · jcardena.com · companion to the film & the essay "Model Quantization: the Ocean and the Crystal"