A 27B model at 350 tokens a second on one RTX 5090
We serve a fine-tune of Qwen3.8-27B to our own product from one RTX 5090 (32 GB): eight chats at once, each with a context of 262,144 tokens. One chat decodes about 350 tokens a second on reasoning and code, and a 32K-token prompt gets its first token after 2.5 s. The server has no PyTorch and no CUDA C++ in it. The engine and its GPU kernels are written in Gizlang, and it is one static binary (Language models is its guide).
This page says what we measured, how the engine gets there, what it costs in quality, what did not work, and how to run it yourself with any checkpoint of the Qwen3.8-27B family.
The numbers
One RTX 5090 with nothing else on it, 2026-11-11, the flags we serve with (--slots=8 --context=262144 --kv=nv4).
The model is Swift 1.5, a fine-tune of Qwen3.8-27B, with every
linear layer in NVFP4 (a 17.4 GB file) and the DFlash 2 block
drafter. The files are the ones on Hugging Face,
downloaded into an empty cache.
| Measurement | Result |
|---|---|
| Prefill, 1K-token prompt | 14,184 tokens/s (1,078 tokens in 75 ms) |
| Prefill, 4K / 8K / 16K | 14,460 / 14,595 / 14,146 tokens/s |
| Prefill, 32K | 13,097 tokens/s (first token after 2.5 s) |
| Prefill, 128K | 9,097 tokens/s (131K tokens in 14.4 s) |
| Decode, 1 chat | 349, 352 and 353 tokens/s in three runs; first token after 17 ms |
| Decode, 8 chats at once | 191, 221 and 235 tokens/s for each chat (median of the 8) in three runs |
| From start to ready | 48 s in a new folder (45 s of it compiles and tunes the kernels); 31 to 46 s after that |
| KV positions in the memory left beside the weights | 742,176, one pool for the 8 chats |
- Prefill is
llmrun needle: one run for each length, a 2048-row step, with a fact at 50% of the prompt. We kept the answer line of the 1K, 16K and 128K runs, and the fact was right in each. - Decode is
llmrun loadagainst the server: eight reasoning and coding prompts (prompts8.json), greedy, a limit of 4096 tokens. The answers ended by themselves after 700 to 1,200 tokens. - For 8 chats we give the speed of each answer. The total over the wall time was 750, 388 and 388 tokens/s: the answers have different lengths, and the total counts the tail in which only the longest one still runs.
What you will not see: 350 tokens/s on every answer
The decode speed depends on the text. A step takes about 17 ms whether it yields one token or seven, because the model checks drafts of the next tokens in the same step (below).
| Text | Tokens a step | Tokens/s, 1 chat |
|---|---|---|
| No drafts (measured 2026-11-08) | 1 | about 60 |
| A story from a list of random words (400 tokens) | about 2.5 | 135 to 157 |
| Reasoning, proofs, code (the benchmark above) | 5 to 6 | 350 |
| Copying or editing text that is in the context | up to the whole draft depth | 1,279 in one measurement at 2K tokens |
On one day of production traffic, the median answer of 100 tokens or more ran at 135 to 158 tokens/s: requests overlap, and many are tool calls in long contexts. A request with a JSON schema or other logit shaping makes no drafts.
How it gets there
NVFP4 from end to end
The RTX 50 series multiplies NVFP4 (4-bit E2M1 values with an 8-bit scale for each 16) in its tensor cores at twice the FP8 rate. The engine keeps everything it can in that format:
- Weights. Every linear layer is NVFP4, the attention projections too. Their inputs are rounded to NVFP4 by row (W4A4), as standard NVFP4 serving does.
- Keys and values. The attention cache is NVFP4 as well, and the query-key and probability-value products run as FP4 products on the codes as they are. No kernel decodes a key or a value. This took 31% from attention time and gives four times the positions of 16-bit keys and values.
- One persistent kernel for the matrix products, with tiles tuned for each row count on the card it runs on (the tunings are kept in the model folder).
The matrix products reach 72% of the card's FP4 peak at 2048 rows. Copying the weights from L2 to the cores takes as long as the multiplies do, so we believe this is near what the card allows.
Drafts: several tokens a step
Decode of a 27B model is bound by reading its weights: 17 GB at 1.8 TB/s is 9.5 ms for a step, however small. A step that checks 17 rows costs 1.33 times a step of 1 row (17.25 ms against 13.0 ms). So the lever is tokens a step.
- A block drafter (DFlash 2, 5 layers) drafts a block of 8 tokens in one pass from the model's hidden states. A step of 7 drafts gives about 6.4 tokens on code. The model's own MTP layer gives about 4.6.
- Recall. Where the last 8 tokens came before in the chat (a code edit, a quote), the tokens after that place are the draft.
- The count is measured, not set. The engine scores every draft against the tokens that followed, for each depth, and takes the count with the most expected tokens for the measured step time. It measures again as the load changes.
- The model checks every draft, so the tokens follow the same distribution as without drafts.
Long contexts and many chats in 32 GB
- The weights take about 17 GB. The embeddings (2.5 GB) and the vision tower (0.9 GB) stay in host memory.
- The eight chats share one pool of KV pages. A full pool makes a request wait or moves the latest one to the host; it does not fail.
- A chat's next turn reads only its new messages. The start of an idle chat goes to host memory and comes back when the same start arrives: 92% of the prompt tokens of later turns came from the cache in our load test.
- Qwen3.8's 48 recurrent (delta-rule) layers keep about 81 MB of state for a chat, saved at the end of each prompt.
The engine itself
The products, attention and normalization kernels are Gizlang source, compiled for the device at the first run of a shape. The same source runs on NVIDIA (sm_80 and later), AMD and Apple GPUs and on the CPU, with other weight formats there. The CPU path is the reference that the GPU path is tested against. The server speaks the OpenAI chat API with tools, pictures, reasoning and JSON-schema grammars.
What it costs
The reference is the same NVFP4 weights run exactly (F32 inputs, 16-bit keys and values), on an English text:
| Setting | Tokens | KL | Top 1 the same |
|---|---|---|---|
| NVFP4 inputs to the linears, 16-bit KV | 4K | 0.0319 | 92% |
| The same with NVFP4 keys and values (what we serve) | 4K | 0.0393 | 90% |
| The same | 16K | 0.0406 | 91% |
- Nearly all of the error is the rounding of the inputs to NVFP4, which any W4A4 server has. The NVFP4 keys and values add about 0.007.
- A fact at 10%, 50% and 90% of 32K tokens, at 50% of 100K and at five depths of 200K was answered right.
- We did not measure the fine-tune in NVFP4 against the same fine-tune in BF16 on agent tasks. The numbers above do not prove that the two score the same.
Against other servers
We did not run another server on this card with this model: the card is in production. What we can set side by side:
| Who measured | Setup | 1 request | 8 at once |
|---|---|---|---|
| The DFlash 2 model card | SGLang, one H200, Qwen3.8-27B in BF16, the same drafter, 7 drafts a step | 184 to 236 tokens/s by task | 1,090 to 1,368 tokens/s in total |
| This page | Gizlang, one RTX 5090, a fine-tune in NVFP4, the same drafter, the draft count chosen by the engine | 349 to 353 tokens/s | 191 to 235 tokens/s for each chat |
That is not a like-for-like comparison: the cards, the precision (BF16 against 4-bit NVFP4), the prompts and the sampling differ. It says only that a 32 GB consumer card can be in that range for this model.
Two comparisons are like for like, on smaller models, and their setup is in bench/ai/llm:
| Model, card | Setup | Result |
|---|---|---|
| Qwen3.5-4B BF16, RTX 4080 | vLLM 0.31 and Gizlang, both with 3 MTP drafts, 64-token answers | Gizlang 1.37 times the tokens/s at 1 chat, 1.21 times at 4 and at 8 |
| Gemma 4 12B 4-bit, RTX 4080 | vLLM 0.32 (W4A16) and Gizlang, no drafts | 1 chat: 75 against 66 tokens/s. 16 chats: vLLM is faster, 453 against 412 in total |
Those runs are short, and the outputs of the two servers were not identical. If you run another server on a 5090, the client below gives a number you can set beside ours.
What did not work
| Tried | Result |
|---|---|
| Scoring half of the keys in each pass of long attention | Faster, but a 260K recall quoted the wrong function. We could not prove it lossless and dropped it. |
| A drafter trained again for the fine-tune | The kept share of drafts was already the same as for the base model. |
| A shorter kernel warm-up (32 → 3 tokens for each shape) | 6 s less at the start, 10% less speed at 8 chats. |
| 1024-row prompt steps | First token 1 to 2% later than with 2048. |
| Fusing the gate product with the NVFP4 rounding | No gain (88.9 → 89.3 ms). |
| 1M tokens of context on one card | Does not fit in any format. |
Two mistakes we made in production: a draft count that was measured slow once was never tried again, so four chats at once ran at 62 tokens/s instead of 155 until the engine learned to measure again; and a grammar step sorted the whole vocabulary, 180 ms a step, until it selected by threshold.
Run it
You need Linux x86-64 and an NVIDIA RTX 50 GPU (sm_120) with 32 GB for the NVFP4 path. A 16 GB card
runs the same model in another format. The install needs Node.js 20.12 or later, or no Node.js at all
(curl -fsSL https://gizstack.com/install | sh); the Quickstart has the details.
npm install -g @gizai/cli
giz doctor
giz ai serve gizai/Swift-1.5-Qwen3.8-27B-NVFP4-gizlang --port 8000 --slots=8 --context=262144 --kv=nv4The first run downloads 19 GB and converts nothing. Then:
curl -s http://127.0.0.1:8000/v1/chat/completions -H 'content-type: application/json' \
-d '{"model": "gizai/Swift-1.5-Qwen3.8-27B-NVFP4-gizlang", "messages": [{"role": "user", "content": "Prove that the sum of the first n odd numbers is n squared."}], "stream": true}'--slots=1 --context=32768 leaves most of the card free. NAME=REPO in place of the repository serves it under a
shorter name.
Any checkpoint of the family
The repository above is a convenience. Every Qwen3.8-27B checkpoint converts with the same two commands: the base model, NVIDIA's NVFP4 checkpoint (its values are kept as they are), or a fine-tune.
giz ai convert nvidia/Qwen3.8-27B-NVFP4 /models/q27 --weights nv4 --head nv4 # or Qwen/Qwen3.8-27B, or a fine-tune
giz ai convert incoai/Qwen3.8-27B-DFlash2 /models/q27/drafter # the block drafter beside it
giz ai serve /models/q27 --port 8000 --slots=8 --context=262144 --kv=nv4- The catalog names do both steps on the first use:
giz ai serve qwen3.8-27borgiz ai serve swift-1.5converts the BF16 checkpoint for the card in use (NVFP4 with--head nv4on an RTX 50 GPU, the 16 GB format on others) with the drafter beside it.giz ai modelslists them. - The drafter was trained for the base model. On two fine-tunes it kept the same share of its drafts (56% and 58%),
so we use it for them as it is. Without
drafter/the engine drafts with the model's own MTP layer. - Swift 1.5 is under the Swift Open License: commercial use is licensed below a revenue threshold. The base model, NVIDIA's checkpoint and the drafter are Apache-2.0.
A 16 GB card
Without --weights nv4 an unquantized checkpoint converts to 4-bit codebook weights with a 6-bit output layer, which
fit a 16 GB card and run on any NVIDIA GPU from sm_80. Start from the BF16 repository: an NVFP4 checkpoint keeps its
NVFP4 values in every format (20.7 GB), which a 16 GB card refuses.
giz ai convert Qwen/Qwen3.8-27B /models/q27-4 # the unquantized checkpoint: a 54 GB download
giz ai convert incoai/Qwen3.8-27B-DFlash2 /models/q27-4/drafter
giz ai serve /models/q27-4 --port 8000 --slots=4 --context=16384On an RTX 4070 Ti SUPER (16 GB, sm_89) this format gave 95 tokens/s for one chat with the model's own MTP drafts (2026-10-19, bench/ai/llm).
Measure it
git clone https://github.com/gizai/gizlang && cd gizlang
bin/gizlang build bench/ai/llm/run.giz -o llmrun
./llmrun speed /models/q27 1 8 # prefill of 1K tokens, decode of 1 and 8 sequences
cat docs/guides/*.md docs/reference/*.md > text.txt # the text we used: about 200K tokens of English
./llmrun needle /models/q27 text.txt 32768 50 # a 32K prompt with a fact at 50%
./llmrun quality /models/q27 text.txt 4096 --kv nv4 # KL against the exact engine
LLM_LOAD_PROMPTS="$(cat bench/ai/llm/prompts8.json)" ./llmrun load http://127.0.0.1:8000/v1 1 1 4096
LLM_LOAD_PROMPTS="$(cat bench/ai/llm/prompts8.json)" ./llmrun load http://127.0.0.1:8000/v1 8 8 4096loadworks against any server of the OpenAI chat API, so the same command measures vLLM or llama.cpp.- For 8 chats, compare the median tokens/s of an answer. The total over the wall time includes the tail in which only the longest answer still runs.
- An A/B is one flag:
--drafts 0,--draft mtp,--kv 8,--exact. Language models lists every option.
Limits
- NVFP4 products need sm_120 or sm_121. We measured one card model, the RTX 5090.
- The toolchain runs on Linux x86-64 only.
- The block drafter needs the model on one GPU.
- The speeds are of one fine-tune of Qwen3.8-27B. Prefill depends only on the weights' shapes; decode depends on what the model writes.
- Every number is from one day and one machine, with one to three runs each.
- Greedy output can differ in a rare place from a run without drafts: a step of more rows sums in another order.
- There is no fast mode of higher precision yet (FP8 inputs to NVFP4 weights).
--exactis the reference and is many times slower.