V100 32GB PCIe using llama.cpp locally inferencing Qwen3.8-27B on Windows, averaging nearly 50 tps — sharing the optimal settings and some tips
4402
0
2026-09-03 20:07:53
Reproduction without the author's authorization is prohibited
44
12
64
6
课程总结
收起全部章节
关闭课程总结
暂无课程总结
The V100 32GB PCIe I bought yesterday arrived. Seeing that on Bilibili most people are focused on optimizing multi-card vLLM, I'll talk about the best pre-made meal experience I've found so far.
Just download the latest precompiled CUDA12 package for llama.cpp directly.
The model is a certain author's Qwen3.8-27B-NVFP4-MTP-HIGHEST on Hugging Face. Just search directly and you'll find it. With MTP enabled, other versions from this author are about 10 tps slower than HIGHEST. This is the fastest 3.8-27B model I've used so far. With MTP=3, text runs at 40 tps and code at 55 tps, but the model weights are 21.5GB (HF model card shows 23.2GB). With -b 2048 -ub 512, MTP at 3, and both KV Cache and MTP's KV Cache fully quantized to q8_0, the maximum context is 196608, and the VRAM usage is basically full with only about 300MB left.
The most important setting is KV Cache quantization. It can only be native or q8_0. In the video demo, k q8 v q4 directly slowed down by about 20 tps, and kv dual q4 was even slower. Moreover, when below q8, the CPU load increases significantly.
My current guess is that this card lacks many native computing units, causing this situation. Only when using q8, bf16, or fp16 can the computational load of dequantization be alleviated. Coincidentally, although the HIGHEST quantization is NVFP4, it contains a large amount of q8 and bf16 quantization, which makes it faster than models with much smaller quantization levels.
