Quantization and Distillation: How Do You Make a Large Model Small Enough to Serve?

There is a moment in almost every model serving project where the arithmetic stops being abstract. You have a model that works. It scores well, your evaluations are green, the demo is convincing. Then you try to serve it, and one H100 is not enough, or it is enough but the tokens arrive so slowly that nobody wants to use it, or it is fast enough for one user and falls over at sixty.

At that point you have exactly two families of tools. You can make each parameter cheaper, which is quantization. Or you can use fewer parameters, which is distillation. Everything else in the model compression literature is a variation on one of those two ideas.

This article is a walkthrough of both, in the order you would actually apply them. I want to start somewhere slightly unusual though, because the usual framing of quantization is wrong in a way that leads people to the wrong decisions.

1. The bottleneck is the bus, not the math

Most people first meet quantization as a memory saving. Sixteen bit weights become eight bit weights, the checkpoint halves, and you fit on a smaller card. That is true and it is not the interesting part.

Take a seven billion parameter model on one H100 SXM. The card has 80 GB of HBM3 running at 3.35 TB/s, and 989 TFLOP/s of dense FP16 tensor core throughput. Now generate one token.

To produce that single token, the GPU has to read every weight out of memory. All fourteen gigabytes of them, because at sixteen bits per parameter that is what seven billion parameters weigh. And with each weight it does roughly two floating point operations: one multiply, one add.

So the ratio of work to traffic during decode is about two FLOPs per two bytes, which is one FLOP per byte. The hardware needs roughly 295 FLOPs per byte to keep its tensor cores saturated. Decode supplies one. The memory bus runs flat out and the tensor cores are idle over ninety nine percent of the time.

That ratio is the whole story. It means the time to generate a token is set by bytes moved, not by math done:

$$t_{\text{token}} \approx \frac{\text{bytes of weights}}{\text{memory bandwidth}}$$

Fourteen gigabytes over 3.35 TB/s is about 4.2 milliseconds, which is 240 tokens per second for a single stream. Halve the bytes and you halve the time. Quarter them and you quarter it.

This is why I said the framing matters. If you think of quantization as a memory saving, you reach for it only when you are out of memory. If you understand it as a latency knob, you reach for it whenever decode latency matters, which is almost always.

One important caveat before we go further. This argument applies to the decode phase, where you generate one token at a time. The prefill phase, where you process the whole prompt at once, is genuinely compute bound, because you are doing a matrix multiply against hundreds or thousands of tokens at a time and each weight gets reused. Quantization helps prefill much less, and if your workload is dominated by long prompts and short answers, your gains will be smaller than the numbers above.

2. What a number actually is

Before you can reason about what quantization costs, you need to be precise about what a low precision number is. The usual intuition is that fewer bits means blurrier numbers, in the way a low resolution photo is a blurry photo. That intuition is wrong and it causes real mistakes.

Fewer bits does not mean the numbers are fuzzy. It means fewer numbers exist. The set of representable values shrinks, and every value you wanted to store gets rounded to a member of that set.

Take one weight with the value 0.1 and store it six ways.

In FP32 you get one sign bit, eight exponent bits and twenty three mantissa bits, and 0.1 comes back as 0.10000000149011612. That is not exact, because 0.1 is not representable in binary, but the error is in the ninth decimal place.

FP16 keeps ten mantissa bits and gives you 0.0999755859375. BF16 spends its sixteen bits differently, keeping FP32's full eight bit exponent and only seven mantissa bits, so it covers the same enormous range with a coarser step and returns 0.1000976562. This is why BF16 became the training default: overflow, not precision, is what kills training runs.

FP8 in the E4M3 layout has one sign bit, four exponent bits and three mantissa bits, and returns 0.1015625. It is still floating point, so it still has a per-value exponent.

INT8 drops the exponent entirely. There is one shared scale for a whole tensor or group, and 255 evenly spaced levels across it. Store 0.1 and you get 0.1023622.

INT4 leaves fifteen usable levels for the entire tensor. Over the range negative one to one, the step is one seventh. The nearest representable value to 0.1 is 0.142857, which is a 42.86 percent error on that single weight.

The distinction between floating point and integer formats is worth holding on to. A float carries its own exponent, so it has fine resolution near zero and coarse resolution far from zero, which matches how neural network weights are actually distributed. An integer format has uniform resolution everywhere, and pays for its dynamic range with a separate scale factor. That difference is the source of almost every problem in the rest of this article.

3. The scale, the zero point, and where the error comes from

Integer quantization is two numbers and a rounding operation. For symmetric quantization, which is what almost everything uses for weights:

$$\begin{aligned} s &= \frac{\max |W|}{2^{b-1} - 1} \\[4pt] W_q &= \text{round}\left(\frac{W}{s}\right) \\[4pt] \hat{W} &= s \cdot W_q \end{aligned}$$

For eight bits, $2^{b-1} - 1 = 127$. For four bits it is 7. The scale $s$ is stored in FP16 alongside the integers.

Asymmetric quantization adds a zero point, which lets the representable range be offset rather than centred on zero:

$$\begin{aligned} s &= \frac{\max(X) - \min(X)}{2^b - 1} \\[4pt] z &= \text{round}\left(-\frac{\min(X)}{s}\right) \\[4pt] X_q &= \text{round}\left(\frac{X}{s}\right) + z \end{aligned}$$

You use asymmetric for tensors that are not centred on zero, and post-ReLU or post-GELU activations are the obvious case, since they are mostly non-negative. Using a symmetric scheme there throws away half your levels on values that never occur.

The error from all of this has two separable sources, and keeping them separate is what lets you debug a bad quantization run.

Rounding error is the unavoidable part. Every value lands within half a step of where it started, so the error is bounded by $s/2$ and behaves roughly like uniform noise. You reduce it by using more bits or a smaller range.

Clipping error happens when you deliberately choose a range narrower than the true maximum, so values outside it saturate. This error is unbounded for the values that clip, but it buys a smaller step size for everything else.

That tradeoff is the entire design space. A wide range means no clipping and a coarse step. A narrow range means a fine step and some values get destroyed. Every calibration method you will read about is a different answer to where to put that boundary, and the good ones choose it by minimising an actual error metric rather than by taking the maximum.

4. One outlier sets the price for everyone

Now the problem that makes low bit quantization hard.

The weights of a typical linear layer form a tight bell around zero, with a standard deviation somewhere around 0.03. Overlay the fifteen levels that INT4 gives you across that range and it looks fine. The bell is a few tenths of a unit wide, the levels are spaced a few hundredths apart, and every weight has a level near it.

Now one weight shows up at 1.4. Just one, out of half a million.

A per-tensor scale is a single number covering the whole tensor, and it has no way to treat that one weight differently from its neighbours. The scale must stretch to reach 1.4, which means the step size goes from about 0.017 to 0.2. The bell that was comfortably spread over a dozen levels now fits inside three of them. Half a million weights collapse onto three values because one weight was loud.

The fix is to stop sharing. Split the tensor into groups, typically 128 consecutive weights along the input dimension, and give each group its own scale. Three of the four groups in the figure keep a tight scale and quantize cleanly at around one percent mean error. Only the group that actually contains the outlier pays for it.

This is not free. A group size of 128 means one extra FP16 scale per 128 weights, which is 16 bits per 128 weights, or 0.125 bits per weight. If you also store a zero point you pay roughly twice that. So a nominal four bit quantization is really about 4.25 to 4.5 bits per weight in practice. That is the number you should use when you estimate your model size, and it is why a "4 bit" 7B model is 3.9 GB rather than 3.5 GB.

Group size is the real quality knob in four bit quantization. Group 32 is better quality and more overhead, group 128 is the common default, per-channel (which is group size equal to the row length) is cheapest and worst. If a four bit quantization is failing your evaluations, shrinking the group size is usually the first thing to try, before you reach for anything cleverer.

5. Activations are the hard tensor, so move the problem

Everything so far was about weights. Weights are the easy case, because they are fixed. You can look at them, measure them, and choose scales offline with as much compute as you like.

Activations are different. They change with every input, so you have to choose a scale that will work for inputs you have not seen. And in large transformers, activations have a much nastier distribution than weights.

Specifically, a small number of input channels, typically between four and thirty out of several thousand, carry magnitudes ten to a hundred times larger than everything else. They are consistent: the same channel indices are hot across different inputs, which is what makes them fixable. But they are brutal for a per-tensor scale. In the example in the figure, the maximum absolute activation is 58.0 while a typical channel sits near 1.15, a spread of about fifty times. Divide 255 levels by that spread and a typical channel gets five levels. Five.

Meanwhile the weight tensor next to it has a spread of about 1.3, so a typical weight channel gets 194 of its 255 levels. One tensor is nearly impossible to quantize and the other is trivial.

SmoothQuant's insight is that a linear layer does not care how you split the magnitude between the two, because for a diagonal matrix $\text{diag}(\mathbf{s})$:

$$Y = XW = \left(X \, \text{diag}(\mathbf{s})^{-1}\right)\left(\text{diag}(\mathbf{s}) \, W\right)$$

The product is unchanged. So divide the activations by a per-channel factor and multiply the corresponding weight rows by the same factor. You have moved difficulty from the tensor that cannot handle it to the tensor that can. And because the scaling factor is per input channel and the weights are fixed, the multiplication folds into the weights offline at zero runtime cost.

The factor itself is a geometric interpolation controlled by a migration strength $\alpha$:

$$s_j = \frac{\max|X_j|^{\alpha}}{\max|W_j|^{1-\alpha}}$$

At $\alpha = 0$ nothing moves. At $\alpha = 1$ the activations become perfectly flat and the weights absorb the entire problem. At $\alpha = 0.5$, both tensors land at the geometric mean of their original maxima, and in the example both end up at 2.173 with a spread of 7.7 times, which is 33 levels each. Activations went from 5 levels to 33. Weights went from 194 to 33. Both are now comfortably quantizable.

In practice $\alpha = 0.5$ is the default and works for most models. Models with unusually extreme activation outliers want a higher $\alpha$, around 0.75 to 0.85. If you push it too high you simply break the weights instead, and your evaluation scores tell you immediately.

6. Calibrate after, or train through

There are two moments at which you can decide what the quantized model will be.

Post-training quantization, or PTQ, takes a finished model, runs a few hundred sample inputs through it to observe the activation ranges, picks scales, and writes out the quantized weights. It costs minutes to hours on a single GPU. It requires no labels, no optimizer, and no training infrastructure. This is what almost everyone uses.

The calibration set matters more than people expect. A few hundred sequences is usually enough, but they have to look like your production traffic. Calibrating a code model on Wikipedia gives you activation ranges from the wrong distribution, and the failure shows up as a model that is fine on generic text and terrible on the thing you built it for.

Quantization aware training, or QAT, inserts fake quantization into the forward pass during training. The weights are rounded to the quantization grid, the forward pass runs on the rounded values, and the loss is computed on outputs the deployed model would actually produce. The model then learns weights that survive rounding.

There is an obvious problem: the rounding function has a derivative of zero almost everywhere, so gradients cannot flow through it. The standard fix is the straight-through estimator, which computes the forward pass with rounding and then simply pretends during the backward pass that the rounding was the identity function:

$$\frac{\partial \hat{w}}{\partial w} \approx \mathbb{1}_{|w| \le \text{clip}}$$

It is a lie, and it works, and the indicator function matters: gradients are passed through for values inside the clipping range and zeroed for values outside it, because a weight that is already saturated should not keep being pushed further out.

The decision is straightforward. At eight bits, PTQ typically loses well under one percent on standard benchmarks and QAT is not worth the cost. At four bits with good group sizes, PTQ loses one to three percent and is still usually the right call. Below four bits, or at four bits on a model that is already small and has no redundancy to spare, PTQ falls apart and QAT is the only thing that works. The rule of thumb: try PTQ first, always, because it costs you an afternoon. Reach for QAT only when PTQ has demonstrably failed.

7. GPTQ and AWQ, two theories of the same crime

The two dominant four bit PTQ methods start from different premises and end up roughly equally good, which is worth understanding because the choice between them is usually made on tooling rather than quality.

GPTQ treats quantization as an error compensation problem. It processes weights column by column. Each time it rounds a column to the grid, it measures the error that rounding introduced, and then updates all the not-yet-quantized columns to compensate for it. The update uses second order information from the Hessian of the layer's reconstruction loss, approximated from the calibration data as $H \approx 2XX^{T}$.

The intuition is that if you round one weight up and that makes the layer's output slightly too large, you can nudge a later weight down to cancel it out. By the time you reach the last column you have absorbed most of the accumulated damage. The cost is a Hessian inverse per layer, which is why GPTQ takes tens of minutes on a large model rather than seconds.

AWQ, which is activation aware weight quantization, starts from a different observation: not all weights matter equally, and the ones that matter are identifiable from the activations rather than from the weights. A weight that multiplies a large activation contributes more to the output than a weight that multiplies a small one, regardless of the weight's own magnitude.

So AWQ looks at the calibration activations, identifies the roughly one percent of input channels with the largest magnitudes, and scales up the corresponding weight channels before quantizing them. Scaling a channel up before rounding means the rounding error, which is fixed relative to the grid, is smaller relative to that channel's value. The scale is then folded away at inference. No Hessians, no sequential dependency, and it runs in a few minutes.

In published comparisons the two land within about a point of each other on perplexity and standard benchmarks, with AWQ often slightly ahead on instruction tuned models and GPTQ slightly ahead at very low bit widths. AWQ is faster to produce and its kernels tend to be better optimised in the popular serving stacks. If you have no strong reason to prefer one, use whichever your serving framework supports best, and spend the time you saved on evaluation instead.

8. The cache is bigger than the model

Here is the thing that catches most teams by surprise in production.

Run Llama 3 8B on one H100 at FP16. The weights are 16 GB. Framework overhead, CUDA context, activation buffers and fragmentation take another 4 GB or so. You have 60 GB free on an 80 GB card, which feels like plenty.

Now admit users. Every token of every sequence stores a key and a value vector in every layer, and that cache never shrinks for the life of the conversation. For this model the per-token cost is:

$$\begin{aligned} &2 \times 32 \text{ layers} \times 8 \text{ KV heads} \times 128 \text{ dims} \times 2 \text{ bytes} \\[4pt] &= 128 \text{ KB per token} \end{aligned}$$

The factor of two is the key and the value. The eight KV heads rather than thirty two are grouped query attention, which already cut this by a factor of four. Even so, an 8,192 token context costs a full gigabyte per concurrent sequence.

Sixty users and the card is full. Not because the model is big, but because the conversations are. The weights, which everybody worries about, are a fixed 16 GB. The cache, which many people never think about, grows linearly with concurrency and with context length, and it wins.

Storing the cache in eight bits doubles that to 120 concurrent sequences on the same card with the same weights. Keys and values are activations, and they quantize well, though not identically: keys are more sensitive than values, because they feed the attention softmax where small errors get amplified by the exponential. A common configuration is INT8 keys and INT4 values, or per-head scales on the keys and per-tensor on the values.

The general lesson is about where to look. If your bottleneck is single stream latency, quantize the weights. If your bottleneck is concurrency or context length, quantize the cache, and do it before you touch the weights. At 128K context a single user costs more memory than the entire model, and no amount of weight quantization changes that.

9. Where quantization stops paying

Quantization has a floor, and it is worth being clear about where it is.

Each halving of precision gives you roughly a halving of decode latency and memory, but the quality cost is not linear. Going from FP16 to INT8 typically costs you well under one percent on benchmarks and is close to free. INT8 to INT4 costs one to three percent with good group sizes and calibration. INT4 to INT3 or INT2 falls off a cliff, and you end up spending more engineering effort recovering quality than the memory was worth.

There is also a structural limit: quantization cannot change the shape of the computation. A 70B model quantized to four bits still runs 70 billion multiply-accumulate operations per token, still has 80 layers of sequential dependency, and still needs its attention over the full context. You made every parameter cheaper. You did not make the model simpler.

That is the point where you stop asking how to store the weights and start asking whether you need that many weights. Which is distillation.

10. Soft labels carry the teacher's opinion

Knowledge distillation trains a small student model to imitate a large teacher model, and the reason it works better than simply training the small model from scratch is subtler than it first appears.

Train a model on hard labels and each example carries one bit of useful supervision: this is the correct answer, everything else is wrong. The loss is cross entropy against a one-hot target, and the model learns nothing about how the wrong answers relate to each other.

Now ask a good teacher to predict the next token after "the capital of France is". The teacher does not just say Paris. It says Paris with probability 0.89, Lyon with 0.04, Marseille with 0.03, Nice with 0.01, and so on down a long tail. That distribution contains information the hard label does not: it says these other options are French cities, that Lyon is a more plausible mistake than Nice, and that the model should be quite but not absolutely certain.

Hinton called this dark knowledge, and it is why distillation gets more out of each training example than supervised learning does.

The mechanism for exposing it is temperature. Divide the logits by $T$ before the softmax:

$$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$

At $T = 1$ you get the teacher's real distribution, which for a confident model is nearly one-hot and carries almost no extra signal. Raise $T$ and the distribution flattens, and the relative ordering of the low probability options becomes numerically visible to the loss. At $T = 5$ the Paris probability might drop to 0.32 while the tail rises, and the gradients now actually carry the structure.

The training objective combines both signals:

$$\begin{aligned} \mathcal{L} = \;& \alpha T^2 \cdot \text{KL}(p_{\text{teacher}}^{T} \,\|\, p_{\text{student}}^{T}) \\[4pt] &+ (1 - \alpha) \cdot \text{CE}(y, p_{\text{student}}) \end{aligned}$$

Two details matter in practice. The $T^2$ factor is not cosmetic: softening the distributions scales the gradients down by roughly $1/T^2$, so without it the distillation term silently vanishes as you raise the temperature. And $\alpha$ is usually set high, around 0.9, because the soft targets carry more information per example than the hard ones do.

11. Train the student where the student actually goes

Standard distillation has a flaw that only shows up at generation time, and it is the single most important practical idea in the modern distillation literature.

In the usual setup, you take a corpus of text, feed it to both models, and train the student to match the teacher's next-token distribution at each position. Every position the student is trained on is a prefix from the corpus, or equivalently a prefix the teacher would have produced. The student never trains on its own output.

Then you deploy it and ask it to generate. Now the student produces token one, and token one was sampled from the student's distribution, not the teacher's. Token two is conditioned on that. By token fifty the student is standing in a place that never appeared in its training distribution, and it has no idea what to do there. Small errors compound into an entirely different trajectory, and the model degrades in a way that its training loss never predicted.

This is exposure bias, and it is the same problem that scheduled sampling and DAgger address in other parts of machine learning.

On-policy distillation fixes it by inverting who produces the text. The student generates. The teacher scores what the student actually produced, position by position, producing a full distribution at every one of the student's own states. The student then trains to match the teacher at exactly the places it will find itself at inference time.

The cost is real. You now need the teacher available during training, not just once to produce a dataset, and you need a teacher forward pass for every student rollout. That is a serious infrastructure difference: off-policy distillation is a data generation job followed by an ordinary training job, while on-policy distillation is a training loop with a large model in it.

The payoff is that on-policy distillation is what makes small models genuinely good at multi-step generation. If your student is going to produce long outputs, which for any chat or coding model it is, this is worth the complexity. If it is a classifier or a short-answer model, off-policy is fine.

12. The rest of the distillation family

A few variants are worth knowing by name, because they show up in papers and in framework documentation.

Sequence-level distillation skips the distributions entirely. You have the teacher generate complete outputs for a set of prompts, then fine tune the student on those outputs as if they were ordinary training data. It is by far the simplest option, it needs no access to teacher logits, and it works with any teacher you can call over an API. It is also the weakest per example, because you have thrown away everything except the argmax. In practice it is the right first thing to try, and a surprising amount of production distillation is exactly this.

Feature distillation matches intermediate hidden states rather than output distributions, usually with a learned projection to handle the dimension mismatch between teacher and student. It provides much denser supervision, since every layer contributes a loss term rather than just the output. It also requires the student's architecture to correspond to the teacher's in a way you can map, which constrains your design.

Attention transfer matches attention maps specifically, on the theory that what the teacher attends to is the reusable part of what it learned. It works well for encoder models and is less commonly used for large decoder models.

Self-distillation uses a model as its own teacher, either a previous checkpoint or an ensemble of augmented views of the same model. It gives smaller gains and needs no second model, and it mostly shows up as a regularisation technique rather than a compression one.

One constraint applies to all of them: the capacity gap. A student that is more than about ten times smaller than its teacher generally cannot absorb what the teacher knows, and the distillation loss plateaus well above where you want it. The standard workaround is a teaching assistant, distilling 70B into 13B and then 13B into 3B, which usually beats distilling 70B into 3B directly.

13. Distill first, then quantize

The two families compose, and the order matters.

Start with a 70B model at FP16. It is 140 GB, needs four GPUs with tensor parallelism, and serves about 96 tokens per second per stream once you account for the cross-GPU communication on every layer.

Distil it into an 8B student. That is 16 GB at FP16, which fits on one card. Getting off multi-GPU is worth more than the parameter count suggests, because you have eliminated the all-reduce on every layer as well as shrinking the model. Throughput goes to roughly 209 tokens per second, an 8.8 times improvement, for about a three percent quality cost on a well executed distillation.

Now quantize that student to four bits. It becomes 4.5 GB including the group scales, still on one card, and the decode path now moves a quarter of the bytes. Throughput goes to roughly 745 tokens per second, another 3.6 times, for another 0.6 percent.

Roughly thirty one times less memory and thirty one times the throughput, for 3.6 percent of the quality. That is the headline, and the order is not arbitrary.

Distil first, because distillation is a training process and it needs a clean, high precision gradient signal. Distilling into an already quantized student means fighting the quantization grid with every update, and you get a worse student for more compute. Quantize second, because quantization is a post-processing step that works on whatever weights you hand it, and its cost is a fixed small percentage regardless of where the weights came from.

The exception is QAT, which by definition happens during training. If you know you are shipping at four bits and PTQ on your distilled student is not good enough, the right move is to fold quantization awareness into the distillation training itself rather than running them as two separate training jobs.

14. What serving actually looks like

A few things change once you are serving quantized weights, and they surprise people who reason only about the byte counts.

The arithmetic usually still happens in higher precision. A W4A16 configuration, which is what most four bit deployments use, stores weights in four bits and dequantizes them to FP16 inside the kernel, on the fly, right before the matrix multiply. You get the bandwidth saving, which is the part that matters for decode, and you do not get any compute saving, which does not matter because you were never compute bound.

This means the kernel matters enormously. A four bit model running on a naive dequantize-then-GEMM path can easily be slower than the FP16 model it replaced, because you have added a decompression step to a pipeline that was already saturated. The gains in this article assume fused kernels of the kind that vLLM, TensorRT-LLM, llama.cpp and the Marlin kernels provide. Always benchmark your actual stack rather than trusting the byte arithmetic.

True W8A8, with both weights and activations in eight bits and the multiply happening in INT8 on the tensor cores, does give you a compute saving as well. This is worth it for prefill-heavy workloads and for throughput-oriented batch serving, and it is exactly the configuration SmoothQuant was designed to make possible.

Batching interacts with all of this. As batch size grows, each weight gets reused across more sequences, the arithmetic intensity rises, and you drift from memory bound toward compute bound. At batch size one, quantization gives you close to its full theoretical speedup. At batch size 256, it gives you much less on latency, though the memory it frees still buys you a larger batch, which buys throughput. Know which of those two numbers your product is actually measured on.

15. Evaluating a compressed model

This is the section people skip, and it is the one that determines whether the project succeeds.

Perplexity is the standard reported metric and it is close to useless on its own. A model can hold perplexity within a tenth of a point of its FP16 baseline and still have lost a capability you care about, because perplexity averages over an entire corpus and your capability lives in a thin slice of it.

Quantization damage is not uniform. It concentrates in exactly the places you would expect from the mechanics above: long context behaviour, because errors in the KV cache accumulate over the sequence; multi-step reasoning, because small logit perturbations change which branch gets taken and the error compounds; rare tokens and rare languages, because they sit in the tail that the scale was never chosen to protect; and instruction following in the specific format you trained for, because format adherence depends on sharp distributions that quantization flattens.

So evaluate on the axes that match your product. Run your task benchmarks, not just the academic ones. Test at your real context length rather than 2K. If you use structured output, test that the JSON still parses at the same rate, because that is one of the first things to break. Test in every language you serve. And run a generation-level comparison, not just a scoring one: sample a few hundred prompts, generate from both models, and have a judge model or a human compare them, because the failure modes that matter are usually visible in the outputs and invisible in the loss.

Set your quality budget before you start, in the metric your users feel. "Under two percent on our task suite and no regression in JSON validity" is a decision rule. "Perplexity looks fine" is not.

16. A decision procedure

Putting the whole thing together, here is the order I would work in.

Start by identifying the bottleneck. Single stream latency, concurrency, or memory ceiling. These have different answers and treating them as one problem is the most common mistake.

If it is concurrency or long context, quantize the KV cache first. INT8 keys and values is close to free and doubles your batch. This is the highest return per unit of effort in the entire list and it is routinely skipped.

Then quantize weights to eight bits with PTQ. It costs an afternoon, loses under a percent, and halves your decode time. There is essentially no argument against doing this.

If you need more, go to four bits with AWQ or GPTQ, group size 128. Calibrate on data that looks like your production traffic. Evaluate properly. Expect one to three percent and be prepared to shrink the group size to 32 if it is worse than that.

If activations are the problem, apply SmoothQuant before quantizing. You will know because your W8A8 attempt lost far more than your W8A16 attempt did.

If four bits is not enough, stop quantizing and start distilling. Below four bits the returns invert. Distillation is the tool that changes the shape of the computation rather than the cost of each parameter.

Distil with sequence-level first, because it is simple. Move to logit distillation if you need more, and to on-policy distillation if your student has to generate long outputs. Respect the capacity gap and use a teaching assistant if your compression ratio exceeds about ten times.

Then quantize the student. In that order, always, unless you are doing QAT, in which case fold it into the distillation run.

The underlying point is that these are not competing techniques. Quantization makes each parameter cheaper and leaves the computation graph alone. Distillation makes the graph smaller and leaves the parameters at full precision. They attack different terms in the same product, which is why applying both in the right order multiplies rather than adds, and why a 70B model can become something that serves thirty times faster on a single card without becoming a different product.