Inferenz Benchmarks
2026-09-21 vLLM mit Qwen 3.8 27b INT8
(config in /opt/vllm-qwen38)
Benchmark-Aufruf auf milla:
uvx llama-benchy --base-url $OPENAI_API_URL --api-key $OPENAI_API_KEY --pp 32000 --no-cache --concurrency 4 --format md --model Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 --served-model-name qwen38-27b --runs 2
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|---|---|
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp32000 (c4) | 1927.26 ± 3.08 | 1006.52 ± 564.21 | 41470.01 ± 18590.88 | 41467.98 ± 18590.88 | 41470.01 ± 18590.88 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c4) | 2.46 ± 0.00 | 20.87 ± 30.50 | 32.00 ± 0.00 | 23.31 ± 30.47 |
llama-benchy (0.4.0) date: 2026-09-21 19:36:48 | latency mode: api
2026-08-30 vLLM mit Qwen 3.8 27b INT8
Benchmark-Aufruf auf milla:
uvx llama-benchy --base-url $OPENAI_API_URL --api-key $OPENAI_API_KEY --pp 2048 32000 64000 128000 \
--no-cache --concurrency 1 2 4 8 --format md --model Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 \
--served-model-name qwen38-27b
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|---|---|
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp2048 (c1) | 2195.92 ± 88.95 | 2195.92 ± 88.95 | 936.52 ± 37.23 | 934.75 ± 37.23 | 936.52 ± 37.23 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c1) | 66.82 ± 4.55 | 66.82 ± 4.55 | 68.97 ± 4.70 | 68.97 ± 4.70 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp2048 (c2) | 2175.03 ± 13.52 | 1620.96 ± 532.74 | 1418.79 ± 465.48 | 1417.02 ± 465.48 | 1418.79 ± 465.48 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c2) | 40.12 ± 0.82 | 36.44 ± 14.22 | 61.00 ± 0.82 | 40.91 ± 11.42 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp2048 (c4) | 2204.24 ± 14.81 | 1153.90 ± 648.27 | 2319.89 ± 1041.14 | 2318.12 ± 1041.14 | 2319.89 ± 1041.14 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c4) | 36.73 ± 0.51 | 25.62 ± 19.11 | 104.67 ± 1.25 | 33.27 ± 15.75 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp2048 (c8) | 2176.39 ± 2.29 | 738.87 ± 592.25 | 4226.90 ± 2154.08 | 4225.13 ± 2154.08 | 4226.90 ± 2154.08 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c8) | 34.38 ± 0.40 | 15.08 ± 14.38 | 132.00 ± 7.87 | 20.90 ± 13.77 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp32000 (c1) | 1990.71 ± 1.41 | 1990.71 ± 1.41 | 16076.91 ± 11.35 | 16075.14 ± 11.35 | 16076.91 ± 11.35 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c1) | 74.29 ± 9.37 | 74.29 ± 9.37 | 76.68 ± 9.67 | 76.68 ± 9.67 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp32000 (c2) | 1977.61 ± 4.66 | 1487.13 ± 498.28 | 24241.74 ± 8121.94 | 24239.97 ± 8121.94 | 24241.74 ± 8121.94 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c2) | 3.71 ± 0.02 | 36.44 ± 34.03 | 32.00 ± 0.00 | 38.99 ± 33.78 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp32000 (c4) | 1956.14 ± 1.19 | 1018.54 ± 568.61 | 40902.14 ± 18286.44 | 40900.37 ± 18286.44 | 40902.14 ± 18286.44 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c4) | 2.47 ± 0.04 | 17.15 ± 23.79 | 32.00 ± 0.00 | 20.06 ± 24.93 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp32000 (c8) | 1731.80 ± 307.50 | 521.02 ± 460.47 | 94488.30 ± 48079.06 | 94486.53 ± 48079.06 | 94488.36 ± 48079.10 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c8) | 2.05 ± 0.06 | 7.90 ± 15.94 | 23.33 ± 12.26 | 10.79 ± 17.22 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp64000 (c1) | 1489.29 ± 52.36 | 1489.29 ± 52.36 | 43028.05 ± 1484.93 | 43026.28 ± 1484.93 | 43029.51 ± 1482.93 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c1) | 49.42 ± 4.98 | 49.42 ± 4.98 | 63.96 ± 11.91 | 63.96 ± 11.91 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp64000 (c2) | 1367.74 ± 336.29 | 966.10 ± 442.01 | 81524.79 ± 35740.88 | 81523.02 ± 35740.88 | 81524.79 ± 35740.88 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c2) | 1.49 ± 0.02 | 27.45 ± 24.44 | 32.00 ± 0.00 | 32.43 ± 28.16 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp64000 (c4) | 1620.36 ± 21.02 | 814.26 ± 432.98 | 100069.83 ± 43246.41 | 100068.06 ± 43246.41 | 100069.83 ± 43246.41 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c4) | 0.99 ± 0.03 | 18.92 ± 27.29 | 32.00 ± 0.00 | 23.43 ± 28.51 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp64000 (c8) | 1691.43 ± 2.78 | 567.24 ± 451.32 | 170771.27 ± 86373.88 | 170769.50 ± 86373.88 | 170771.27 ± 86373.88 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c8) | 0.93 ± 0.00 | 9.75 ± 17.10 | 32.00 ± 0.00 | 13.02 ± 19.26 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp128000 (c1) | 1333.24 ± 16.29 | 1333.24 ± 16.29 | 96023.90 ± 1182.87 | 96022.12 ± 1182.87 | 96023.90 ± 1182.87 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c1) | 62.31 ± 9.03 | 62.31 ± 9.03 | 110.27 ± 47.76 | 110.27 ± 47.76 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp128000 (c2) | 1361.46 ± 3.35 | 1020.37 ± 339.68 | 141079.37 ± 46959.48 | 141077.60 ± 46959.48 | 141079.37 ± 46959.48 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c2) | 0.59 ± 0.01 | 63.79 ± 8.14 | 32.00 ± 0.00 | 73.59 ± 7.25 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp128000 (c4) | 1359.38 ± 1.06 | 707.95 ± 395.29 | 235394.83 ± 105268.48 | 235393.06 ± 105268.48 | 235394.83 ± 105268.48 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c4) | 0.39 ± 0.00 | 65.37 ± 11.10 | 32.00 ± 0.00 | 76.58 ± 13.26 | |||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp128000 (c8) | 1358.85 ± 1.57 | 462.11 ± 373.76 | 423654.57 ± 215887.54 | 423652.80 ± 215887.54 | 423654.57 ± 215887.54 | ||
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | tg32 (c8) | 0.33 ± 0.00 | 58.32 ± 14.31 | 32.00 ± 0.00 | 68.63 ± 14.86 |
llama-benchy (0.4.0) date: 2026-08-30 15:13:03 | latency mode: api
2026-04-18 vLLM mit cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
vLLM optionen:
--model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
--tensor-parallel-size 2
--max-model-len 65536
--gpu-memory-utilization 0.85
--enable-prefix-caching
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--max-num-seqs 32
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
Benchmark mit uvx llama-benchy --base-url http://cogito.buero.ping.de:8000/v1 --depth 2000 32768 63000
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit | pp2048 @ d2000 | 5463.38 ± 111.87 | 748.82 ± 14.93 | 741.48 ± 14.93 | 748.93 ± 14.93 | |
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit | tg32 @ d2000 | 103.13 ± 22.06 | 112.49 ± 24.41 | |||
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit | pp2048 @ d32768 | 5178.25 ± 25.55 | 6731.33 ± 33.06 | 6724.00 ± 33.06 | 6731.41 ± 33.05 | |
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit | tg32 @ d32768 | 25.65 ± 1.43 | 27.93 ± 1.52 | |||
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit | pp2048 @ d63000 | 4534.72 ± 42.10 | 14353.15 ± 133.93 | 14345.82 ± 133.93 | 14353.26 ± 133.94 | |
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit | tg32 @ d63000 | 12.85 ± 3.50 | 14.45 ± 3.21 |
Plan: P2P einschalten, da geht noch mehr...
No comments to display
No comments to display