Before You Serve a Quantised LLM, Run This Short Check

Key takeaways

• Where a file came from does not show whether it works: the failing files matched exact rebuilds from the original weights.
• If a functional test gives a low score, check the chat template and how the file was made before you reject it.
• Test with your production cache settings, the settings for reusing cached prompt prefixes. In one study, that reuse changed some answers, with no detectable net change in accuracy.
• Passing a short functional test is evidence against near-total failure on the task tested. You still need tests of your own workload, against acceptance criteria you set.

Jump to section

• A quantised LLM can run normally and still fail its task
• The check to run before you serve a quantised LLM
• Why a file's origin does not prove it works
• Check the chat template before you blame the file
• Why the quantisation recipe alone cannot predict failure
• What passing a short functional test does not cover
• How cache settings can change answers after a build passes
• Limits of the evidence
• Where we fit
• References

A quantised LLM can run normally and still fail its task

Your health check passes: the model loads, answers and streams text at normal speed. That is not proof that it works.

Developers download quantised models from public registries to run them locally ^[1]. A quantised build trades some precision for a smaller memory footprint, so large models fit on more devices ^[2].

One audit, revised on 29 October 2026, tested one registry: the official Ollama library. It covered all 305 eligible code-capable GGUF tags, plus 22 popular Hugging Face community files. GGUF is a binary file format built for quick loading and saving of models ^[3]. A quick 15-task screen flagged suspects, and these were retested on all 164 HumanEval+ coding tasks. Seven official tags (2.3%) solved at most six of the 164 ^[1]. They did not crash or refuse; they "stream text at normal speed" ^[4].

The audit covers only that registry's code-capable instruct tags of 8 GB or less. Counted by model variant instead, 3 of 35 were affected (8.6%); the author calls this the better summary ^[4]. Neither figure is a failure rate for your models.

The check to run before you serve a quantised LLM

A short functional screen tests whether a model can do a task, not only whether it loads and streams text. A pass is evidence against near-total failure on the task tested; it does not replace testing your own workload.

Test the file you will serve, with the backend and chat template you will serve it with ^[4].

1. Record the file, backend, chat template and cache settings (the settings for reusing cached prompt prefixes). A re-upload under another name is not an independent build: it can be the same broken file.
2. Check that your serving engine supports the file's format on your GPU. For example, vLLM does not support INT8 on Blackwell GPUs (compute capability 10.0 and above) ^[11].
3. Run a short functional screen on that stack. The published screen, quantcheck, tests Python code; it took a median of 30 s of generation per file on one consumer GPU ^[4]. For other work, test your own task.
4. If it fails, look at the template, the recipe and its calibration before you reject the file.
5. If it passes the screen, you have evidence against near-total failure on that task only. Then test your own task, language and input length, against acceptance criteria you set.
6. Send multi-turn requests that share a prefix, more than once, with production cache settings, and compare the answers.

Who runs the checks, and who asks for the results

Builders: add the screen to the pipeline, and record the configuration, results and limits.

Leaders: ask for that record, and for the acceptance criteria, before a build is served. A team that buys a model through an API can ask which file, backend, template and cache settings sit behind its endpoint.

Why a file's origin does not prove it works

Four questions often get mixed up. Where did the file come from? Was it built as described? Does the runtime call it correctly? Does it do the task? Only the last one shows that the setup does its job.

A file can pass the first two and still fail the last. The audit rebuilt five failing files from the original weights, without calibration. Each rebuild matched the registry file tensor for tensor, so none was corrupted ^[1]. None of the registries examined documented functional testing.

The registry did not warn users either. It served five broken files unchanged from August to October 2026, and its pages gave neither the recipe nor whether a tag was tested ^[4]. Copies travel too. The broken phi3.5 file was byte-identical to two Hugging Face uploads under unrelated names; only a tensor-level comparison caught a third copy. Only a functional test shows whether a file does its task.

Signing does not close this gap either. A signature covers the full-precision weights, but users run a quantised file made in ways not fully disclosed. A separate study looked at the 200 most-downloaded Hugging Face GGUF repositories (198 in scope): none recorded its build environment ^[5].

Check the chat template before you blame the file

A failure is not always in the weights. Two deepseek-coder-6.7B files from TheBloke carry no chat template, so the runtime falls back to a generic format, ChatML. It differs from the model's documented instruction format, and the two files solved 5 and 6 of 15 tasks. With the model's own template, the same files solved 13 and 15. A low score alone does not show its cause ^[4].

So a failed screen means: look at the template, and at the recipe (the settings used to make the file), before you reject it.

Why the quantisation recipe alone cannot predict failure

In that registry, all seven failing tags were 2-bit or 3-bit builds (q2_K to q3_K_L) of models with at most four billion parameters ^{[1, 4, 3]}. Whether other models fail with uncalibrated low-bit recipes remains open.

Calibration can help, but the recipe alone does not predict failure. Here, calibration means an importance matrix. It is computed from a model and a text dataset, and it can be used during quantisation to improve quality ^[6].

With an importance matrix, Qwen2.5-Coder-3B went from zero to 76-113 of 164 tasks, at three quantisation levels. Yet the same uncalibrated recipe left the smaller Qwen2.5-Coder-1.5B working ^[1].

For that 3B model, a matrix computed from code solved 6 to 13 more tasks than one computed from WikiText, at each quantisation level ^[4]. So if you calibrate, it may pay to use text like the model's work. Even so, one level, q3_K_S, stayed weak whatever the calibration ^[7].

What passing a short functional test does not cover

A pass is narrow. The screen tests only Python code, so a file can pass and still fail other tasks. It looks for near-total failure, not a gradual loss of quality ^[4]. Among 66 sampled tags that passed, none turned out to be non-functional ^[1].

A 2026 study of tool use tested three models of 7B to 11B parameters, with Polish prompts and English tool schemas. All three collapsed between 3-bit and 2-bit. That threshold held for all three models; the ways they failed did not ^[8].

A 2025 study of long inputs, 64K tokens or more, on five models found that 4-bit methods lost up to 59% ^[9]. On average, 8-bit methods kept accuracy, with a drop of about 0.8%. Losses tend to be worse for input not in English, and they depend heavily on the method, the model and the task.

Its authors call for careful, task-specific evaluation before deploying quantised models, especially for long inputs and languages other than English ^[9].

How cache settings can change answers after a build passes

By default, major open-source serving stacks reuse cached work for a shared prompt prefix, the opening text that requests have in common. One study ran a multi-turn agent workload and held the model, decoding, seed and request order fixed, at batch size one. Without that reuse, repeated runs matched token for token; with it, the cached and recomputed paths diverged. With GGUF weights on one engine, Qwen2.5-7B diverged in 37.4 percent of episodes at 16-bit, and in 82.5 percent at four-bit ^[10]. In a single-turn test, the divergence changed some task outcomes, but with no detectable net change in accuracy.

On that evidence, different answers were not worse answers on average. But a check run with caching off does not test what you serve.

Limits of the evidence

Most of this evidence is preprints. The audit, the screen and the cache study are by one author ^{[1, 10]}; the build-records and Polish studies are by others ^{[5, 8]}. Only the long-context study is published, in the EMNLP 2025 proceedings ^[9].

None of these sources reports a test in Indian languages. The 2025 study reports worse losses outside English, without naming the languages ^[9].

A pass tells you only what the screen tested. Your own tests, with criteria you set, decide the rest.

Where we fit

IndicStack is one OpenAI-compatible API for open-weight models such as DeepSeek, GLM, Qwen and Gemma, billed in rupees. Every model is labelled India-hosted or Global, and one switch keeps your requests on India-hosted models only. The questions in this check apply to any endpoint, ours included.

---

Disclaimer

This article is for information only. It is not legal, procurement or technical advice, and it does not replace your own testing of any model file, serving setup or provider. It was drafted with the help of AI. IndicStack chose the sources, checked every figure and reference against its source, and reviewed the article before publication. Product details are as at the date of publication and may change. Naming an organisation, study or product does not imply endorsement by, or affiliation with, IndicStack.

Copyright © IndicStack Consultancy Services LLP 2026

References

1. Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them. arXiv (A. Patodiya). https://arxiv.org/abs/2609.05881v3 (accessed 3 November 2026).
2. Quantization. vLLM documentation. https://docs.vllm.ai/en/latest/features/quantization/ (accessed 3 November 2026).
3. GGUF. Hugging Face Hub documentation. https://huggingface.co/docs/hub/gguf (accessed 3 November 2026).
4. Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them. arXiv (A. Patodiya), full text. https://arxiv.org/html/2609.05881v3 (accessed 3 November 2026).
5. What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains. arXiv (W.-L. Liu, C.-Y. Tsai). https://arxiv.org/abs/2610.19498v1 (accessed 3 November 2026).
6. llama.cpp/tools/imatrix. llama.cpp (ggml-org), imatrix README. https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/imatrix/README.md (accessed 3 November 2026).
7. aditi-p31/quantcheck. GitHub (quantcheck repository). https://github.com/aditi-p31/quantcheck (accessed 3 November 2026).
8. Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit. arXiv (J. Prejzner). https://arxiv.org/abs/2609.32042v1 (accessed 3 November 2026).
9. Does quantization affect models’ performance on long-context tasks?. ACL Anthology (EMNLP 2025; Mekala et al.). https://aclanthology.org/2025.emnlp-main.479/ (accessed 3 November 2026).
10. Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving. arXiv (A. Patodiya). https://arxiv.org/abs/2609.04748v2 (accessed 3 November 2026).
11. INT8 W8A8. vLLM documentation. https://docs.vllm.ai/en/stable/features/quantization/llm_compressor/int8_w8a8/ (accessed 3 November 2026).