amd yes! Using the 7900xtx to push qwen3.8-27b to 97t/s! 256K context with vision!
23.2K
2
2026-08-29 01:05:33
282
88
688
142
课程总结
收起全部章节
关闭课程总结
暂无课程总结
Using a single 7900XTX card, Ubuntu 24.04 + llama.cpp + dflash2 + ngarm-map-k4v + HIP pushes Qwen3.8-27B to a peak of 97 t/s!
The above are all measured data during optimization. The modified llama source code will be uploaded to GitHub later.
Project: https://github.com/QingYis/llama-7900xtx-qwen3.8-27b
Detailed explanation has been updated in the next video. Follow this channel for more useful AI content.
Full startup parameters:
LLAMA_ATTN_ROT_DISABLE=1 ./build-rocmfp4/bin/llama-server \
-m models/Qwen3.8-27B-heretic-ara.ROCmFP4-STRIX.gguf \ # main model (ROCmFP4/STRIX)
-md models/Qwen3.8-27B-DFlash2-Q4_K_M.gguf \ # DFlash2 draft model
--mmproj models/mmproj-Qwen3.8-27B-Q8_0.gguf \ # vision encoder (drop this line to disable)
--spec-type ngram-map-k4v,draft-dflash \ # stacked speculators, applied in order
--spec-ngram-map-k4v-size-n 12 \ # n-gram lookup key length
--spec-ngram-map-k4v-size-m 48 \ # max draft tokens on an n-gram hit
--spec-ngram-map-k4v-min-hits 1 \ # minimum occurrences for a hit
--spec-draft-n-max 5 \ # DFlash2 draft depth (measured optimum)
--spec-draft-p-min 0.4 \ # draft confidence cutoff (measured optimum)
-ctk q4_0 -ctv q4_0 \ # 4-bit KV cache (~14 GiB saved @256K)
-ngl 99 -ngld 99 \ # full offload: target / draft
-c 262144 \ # 256K context
-b 2048 -ub 512 \ # logical/physical batch, bounds prefill buffers
-fa on \ # fused attention kernels
--jinja \ # model's own chat template
--no-kv-unified \ # prevents full-history re-prefill on hybrid archs
--parallel 1 \ # single slot (pairs with the flag above)
--ctx-checkpoints 2 \ # spec-decode rollback snapshots (default 32 ≈ 10 GiB)
--cache-ram 4096 \ # spill prompt cache entries to host RAM
--no-warmup \ # skip warm-up pass (VRAM is nearly full)
--host 0.0.0.0 --port 1234 # listen address and port
