Arpit Singh Gautam
About Publications Experience Media & Awards Projects Blog Misc
← Blog

Where a Learned Quantizer Spends Its Last Bit

Sep 2026 11 min read Quantization · Efficiency · Benchmarks

A quantization release normally hands you weights and a table. This one hands you the decision.

ISTA-DASLab, Dan Alistarh’s group at the Institute of Science and Technology Austria, put up GGUF quantizations of Qwen3.8-27B at four sizes, from 2.50 to 3.50 bits per weight. Alongside each file sits a 25 kB text file listing the quantization type assigned to every one of the model’s 851 tensors, plus the importance matrix used to produce them. The methods are two of their own papers, GSQ and RCO.

I downloaded all four allocation dumps and parsed them. Where the bits went turned out to be more interesting than the benchmark table, which hinges on a single question out of 198.

Two problems wearing one coat

“Quantize this model to N bits” is really two questions.

The first is the quantizer. Given one weight tensor and a target format, where do you put the grid points and the group scales? Scalar methods like GPTQ and AWQ answer this well down to about 3 or 4 bits and then plateau. Vector and trellis methods like QTIP and AQLM do better below that, at the cost of a format ordinary inference kernels cannot read. GSQ argues the gap is not fundamental: relax the discrete choice of grid point with a Gumbel-Softmax whose cardinality matches the handful of levels you actually have at 2 to 3 bits, learn assignments and scales together, and a scalar quantizer recovers most of the vector frontier while still producing something llama.cpp can execute today.

The second is the allocator. This model has 402 weight tensors worth quantizing and they are not equally fragile. Given K candidate formats, N tensors and a total file size, which format does each tensor get? Model loss does not decompose across tensors, so a greedy per-tensor answer is wrong. Evolutionary search sees the true loss but has no gradient. Penalty methods get a gradient but only approximately satisfy the budget, and you tune the penalty weight per problem.

RCO’s move is geometric. Under a softmax relaxation the budget constraint turns out to describe a smooth Riemannian manifold in logit space whose normal vector has a closed form. So you can run Adam on the actual loss, project onto the tangent space, retract with a binary search, and land exactly on the budget with no constraint hyperparameter at all.

Quantize every tensor at every candidate type, search for the assignment, stitch the winners into one standard GGUF. The result runs unmodified in llama.cpp and Ollama.

Every score in the table is an integer in disguise

The model card leads with the 3.50 bpw file and calls it task-lossless: identical to the BF16 base on AIME25 and LiveCodeBench v6, and 0.51 points behind on GPQA-Diamond, at roughly a fifth of the size.

Those three benchmarks hold 30, 198 and 175 items respectively. So I multiplied every published score by n/100 to see whether it came back to a whole number of problems. All 24 of them did, to within 0.01. That confirms the denominators, and it converts the whole table into something more legible than percentages.

Once you can read it that way:

A benchmark cannot express a difference smaller than one item, which is 3.33 points on AIME25, 0.51 on GPQA-Diamond and 0.57 on LiveCodeBench. Three of the four headline numbers are sitting on that floor.

Every published gap in the GSQ-RCO release converted into a count of benchmark items Six comparisons drawn as horizontal bars whose length is the number of benchmark items separating two models. The three gaps the model card leads with, all at 3.50 bits per weight, are one question, one problem and three problems. The three at 2.50 bits per weight are three problems, eight problems and seventeen GPQA-Diamond questions. A dashed vertical line at one item marks the smallest difference any of these benchmarks can express. Only the seventeen question gap stands clearly above that floor. Every gap in the release, in problems The three reasoning benchmarks hold 30, 198 and 175 items. Each published gap converted back into items. 1 item = the smallest difference the benchmark can express the three gaps the card leads with, at 3.50 bpw GPQA-Diamond IQ3_S vs BF16 1 question (0.51 pts) the card's headline claim: trails by 0.51 points AIME25 IQ3_S vs UD-IQ3_S 1 problem (3.33 pts) leads by 3.33 points LiveCodeBench IQ3_S vs UD-IQ3_S 3 problems (1.71 pts) leads by 1.71 the same comparison at 2.50 bpw, matched file size AIME25 IQ2_XS vs UD-IQ2_S 3 problems (10.00 pts) LiveCodeBench IQ2_XS vs UD-IQ2_S 8 problems (4.57 pts) GPQA-Diamond IQ2_XS vs UD-IQ2_S 17 questions (8.59 pts) the only gap here a benchmark this size can resolve 0 5 10 15 items separating the two models Gaps are the card’s, item counts are mine. Unpaired binomial standard error on a difference at n=198 is about 3 to 4 points, so the top bar sits at 0.17 sigma and the bottom one at 2.17.
Every published gap, converted into items. Gaps are the model card’s; the item counts and the sigma figures are mine, from the harness linked at the end. The dashed line is the resolution floor: no difference smaller than one item can exist.

The standard error makes the same point less charitably. For a difference of two proportions at n=198, the unpaired binomial standard error is

SE=p1(1−p1)n+p2(1−p2)n≈3.1to4.0points

which puts the flagship 0.51-point gap at 0.17 sigma. These evaluations run on identical items, so a paired test would be tighter and fairer, but per-item results are not published, so nobody outside the lab can run one.

AIME25 is the worst offender and it is not really the release’s fault. The base model scores 30 out of 30. A saturated benchmark cannot report a degradation smaller than one problem, and “the quantized model also got 30 out of 30” tells you almost nothing about the quantizer.

The result that survives is at the other end

Run the same conversion at the tightest budget, where the GSQ-RCO file at 8.4 GB meets Unsloth’s dynamic quantization at the same 8.4 GB:

Seventeen questions out of 198 is 2.17 sigma even on the conservative unpaired test. That is a real result, and it is the one the card mentions third.

It is also visible in the lab’s own chart, at the far left, where the two 2.5-bit points sit 7.7 task-average points apart. By 3.5 bits the same gap is 1.5 points and everything has converged onto the base model’s line.

The model card’s own scatter plot of task average against average bit-width. The four GSQ-RCO points rise steeply from 86.0 at 2.5 bits to 91.7 at 3.5 bits and flatten against a dashed line marking the BF16 base at 91.87. Three grey Unsloth dynamic points sit below the curve, the lowest at 78.3 at 2.49 bits.
The lab’s own plot of the same result. Task average is the mean of AIME25, GPQA-Diamond and LiveCodeBench v6. From the model card for ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, read 5 September 2026. The gap worth looking at is on the left of their chart, not the right.

Learned allocation earns its keep exactly where the budget is too tight for a bad assignment to be absorbed. That is also where hand-tuning is hardest, which is presumably why the gap is there. At 3.50 bpw both allocators are close enough to the base model that these benchmarks cannot tell them apart, and reporting a win there is reporting a coin flip.

The cheap metric behaved and the expensive one did not

All three perplexity columns fall monotonically as the budget grows. Wikitext2 goes 7.69, 7.39, 7.20, 7.07. C4 and FineWeb-Edu do the same, without an inversion.

The five-task zero-shot average goes 74.54, 75.70, 74.81, 74.47. The 2.50 bpw model beats the 3.50 bpw model on it. All four quantized models also score above the BF16 base’s 74.34, which is where the recovery figures of 100.3%, 101.8%, 100.6% and 100.2% come from.

Perplexity is monotone in the bit budget and the zero-shot average is not Two line charts sharing an x axis of bit budget at 2.50, 2.75, 3.00 and 3.50 bits per weight. The left shows wikitext2 perplexity falling cleanly from 7.69 to 7.39 to 7.20 to 7.07. The right shows the five-task zero-shot average moving from 74.54 up to 75.70, back down to 74.81 and then to 74.47, crossing above and below a dashed line marking the BF16 base model's 74.34. The smallest model scores higher than the largest on this second metric. One of these two metrics ranks the four models correctly Both columns come from the same results table. Only one is monotone in the bit budget. wikitext2 perplexity, lower is better 7.69 2.50 7.39 2.75 7.20 3.00 7.07 3.50 bit budget (bpw) Strictly monotone. So are c4 and fineweb-edu. five-task zero-shot average, higher is better BF16 base, 74.34 74.54 2.50 75.70 2.75 74.81 3.00 74.47 3.50 bit budget (bpw) Not monotone. The 2.50 bpw model beats the 3.50 by 0.07. The card’s recovery column is computed from this one.
Two columns from the same table. Perplexity ranks the four files in budget order without an inversion. The zero-shot average does not, and every quantized model sits above its own base model on it.

A quantized model does not acquire capability its base model lacked. Four recovery numbers above 100% are four measurements of the noise floor on those five tasks. And the recovery column, which is the card’s fidelity metric, is computed from the column that wanders. Perplexity, the proxy everyone is trained to distrust, is the one that ranks these four files correctly.

What the allocator actually decided

The harness that produces every number below is in the post’s research folder.

One caveat on method. Bits per weight for the IQ types are the figures llama.cpp’s own quantize tool prints. Every mean below is unweighted over tensors, so it describes the pattern of decisions and not the file’s true bit-width, which would need tensor shapes I do not have.

The first thing the dump gives you is the architecture, which the card never mentions. Sixty-four blocks. Blocks 3, 7, 11 and every fourth through 63, sixteen in total, are full attention with separate attn_q, attn_k, attn_v and attn_output. The other 48 carry a fused attn_qkv, an attn_gate and an SSM mixer. Qwen3.8-27B is a 3:1 hybrid and you can read that straight off a list of tensor names. 

# RCO per-tensor GGUF allocation
# Target whole-file bpw: 3.50
# Tensor count: 851
# Quant-type counts: BF16=96, F32=353, IQ1_M=1, IQ2_S=17, IQ2_XS=9, IQ2_XXS=5,
#                    IQ3_S=144, IQ3_XXS=78, IQ4_XS=96, Q2_K=13, Q4_K=39
blk.0.attn_qkv.weight: IQ4_XS
blk.0.ffn_down.weight: IQ2_S
blk.0.ffn_gate.weight: IQ2_XS
blk.0.ffn_up.weight: IQ2_XXS
blk.0.ssm_alpha.weight: BF16
blk.0.ssm_beta.weight: BF16

RCO chooses for 402 of the 851 tensors. Of the rest, 353 stay F32, which is the usual norms and biases, and 96 stay BF16. Those 96 are exactly ssm_alpha.weight and ssm_beta.weight, one pair per SSM block, and the set is identical in all four files including the 2.50 bpw one. The SSM gating parameters are never quantized at any budget. That looks like a constraint someone imposed rather than something the optimizer discovered, and it is defensible: in a recurrent mixer an error in the state transition does not stay local to one token, it rides along the sequence. It is still worth knowing before you quote “2.50 bits per weight” at somebody.

Deeper layers get more bits

Mean assigned bits rise monotonically with depth, in every one of the four files.

RCO assigns more bits to deeper blocks, consistently across all four size budgets Two panels. The left plots mean assigned bits per weight for each of the 64 blocks at the 3.50 bits per weight budget, with red bars marking the four quartile means at 3.10, 3.41, 3.77 and 3.92 bits. Individual blocks scatter widely but the quartile means rise steadily with depth. The right panel plots quartile means for all four budgets as four parallel rising lines, showing the same gradient of roughly 0.8 bits from the first quarter of the network to the last, whether the total budget is 2.50, 2.75, 3.00 or 3.50 bits per weight. Deeper layers get more bits, at every budget Mean bits per weight assigned by RCO across the 402 tensors it chooses for. Unweighted by tensor size, so this is the decision pattern, not the file’s true bit-width. 3.50 bpw budget, all 64 blocks 2.0 2.5 3.0 3.5 4.0 3.10 3.41 3.77 3.92 0 16 32 48 63 block index bits per weight Dots are single blocks. Red bars are the quartile means. Individual blocks are noisy. The trend is not. quartile means, all four budgets 2.50 bpw 2.75 bpw 3.00 bpw 3.50 bpw Q1 Q2 Q3 Q4 blocks 0-15, 16-31, 32-47, 48-63 Four separate optimizations, same slope: Spearman +0.74, +0.75, +0.80, +0.77. Parsed from the four tensor-allocation dumps shipped with the release. Bit-widths per quant type are llama.cpp’s own figures.
Where RCO put the precision. Parsed from the four tensor-allocation/*.rco-allocation.txt files shipped with the release. The left panel is one budget across all 64 blocks; the right is all four budgets summarised by quartile.

The gap from the first quarter of the network to the last is +0.77, +0.82, +0.85 and +0.82 bits at the four budgets. Spearman correlation between block index and mean assigned bits is +0.74, +0.75, +0.80 and +0.77. Four independent optimizations, same slope, and nothing in the setup told it to do this.

The embedding gets sacrificed, the head does not

At 2.50 bpw, token_embd.weight is IQ1_M, 1.75 bits, the most crushed tensor in the file. output.weight is IQ4_XS at 4.25 bits, near the top of the ladder. That ordering holds at all four budgets.

Compare that to llama.cpp’s own quantizer, which hard-codes a floor for these tensors: llama_tensor_get_type forces output.weight up to Q5_K when the rest of the file is on IQ1, IQ2 or IQ3 types, and Q6_K otherwise. An untied token_embd, which is what this model has, gets no such protection.

So llama.cpp protects the head by rule and leaves the embedding exposed, and RCO arrives at the same shape from the loss alone. With two twists. It puts the head at 4.25 bits, below the Q5_K floor llama.cpp would have forced, so it is less conservative than the hand-written rule. And it is far more aggressive on the embedding than any default would dare.

An embedding lookup error is a fixed offset entering a 64-block residual stream that has every opportunity to absorb it. The output projection writes straight into the logits with nothing downstream to correct it.

The allocations are not nested, and that matters

Raising the budget sometimes takes bits away from a tensor.

Between adjacent budgets, 14.7%, 14.2% and 6.7% of the 402 tensors get fewer bits than they had at the smaller budget. Comparing the 2.50 bpw file to the 3.50 bpw file, a 40% larger budget, seven tensors still end up lower. blk.11.attn_k.weight goes from Q4_K down to IQ2_S, losing two full bits while the file around it gains. And blk.13.ffn_gate.weight runs IQ1_S, IQ1_M, IQ4_XS, IQ1_M across the four budgets; it is the only tensor still on a 1-bit type in the 3.50 bpw file.

My reading is that this is what a flat objective looks like. Many allocations reach nearly the same loss, each budget is solved independently, and the search lands somewhere different each time. The aggregate pattern barely moves while the individual assignments shuffle.

So a single tensor’s assignment is not a sensitivity measurement and should not be read as one. If you were about to look up your favourite layer in the dump and conclude something about it, do not. The depth gradient and the embedding result are worth trusting because they survive four separate optimizations. The blk.13 curiosity is exactly the kind of thing that does not.

Three ways to answer the same question

I should declare an interest here. I built RAMP, which solves per-layer bit allocation as a constrained MDP with Soft Actor-Critic, so I have spent a while inside this problem and I am not a neutral party on it.

There are now three families of answers:

Heuristics, including Unsloth’s hand-tuned dynamic quants and llama.cpp’s built-in type rules. Cheap, transparent, and the baseline this release beats by 17 GPQA questions at 2.50 bpw and ties at 3.50.

Learned policies, which is what RAMP does. Expensive to train once, then applied to a new model with no new search.

Constrained continuous optimization, which is RCO. Exact budget, no constraint hyperparameter, optimizes the real loss.

The axis nobody states is amortized against per-instance. RCO is not a competitor to a learned policy so much as a better solver for a single instance, and it pays that cost every time: a new model or a new budget means a new search. The four non-nested allocations in this release are that property showing through. A learned policy trades some per-instance quality for the ability to skip the search entirely. Which one you want depends on whether you are quantizing one model carefully or forty models on a schedule.

What I would take from this

If you are choosing a file: the 2.50 bpw one is where this method is actually doing something for you. At 3.50 bpw you are picking between allocators that these benchmarks cannot distinguish, so pick on file size and move on.

If you are reading a quantization result: divide by the number of items before you believe a gap. Published quantization comparisons report GPQA-Diamond to two decimal places. One question is worth 0.51 points, so everything after the first decimal is decoration.

If you are shipping a quantized model: ship the allocation. It cost this release 25 kB per file and it is the only reason this post has anything in it beyond arithmetic. I would not have found the depth gradient, the untouched SSM gates or the non-nested solves from the weights alone, and neither can anyone else.

The GSQ-RCO release, the results table and the plot above are ISTA-DASLab’s, from the model card, read 5 September 2026. The methods are Dadgarnia, Tabesh, Nikdan, Helcig, Kurtic, Kleinegger and Alistarh (GSQ) and Helcig and Alistarh (RCO). Benchmark sizes: GPQA-Diamond 198 questions, AIME 2025 30 problems, LiveCodeBench v6 175. The item-count conversion, the standard errors, the allocation parse and the three drawn figures are mine, and the harness that reproduces every number is in the post’s research folder.