Skip to main content

Inferenz Benchmarks

2026-08-30 vLLM mit Qwen 3.8 27b INT8

Benchmark-Aufruf auf milla:

uvx llama-benchy --base-url $OPENAI_API_URL --api-key $OPENAI_API_KEY --pp 2048 32000 64000 128000 --no-cache --concurrency 1 2 4 8 --format md --model Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 --served-model-name qwen38-27b
model test t/s (total) t/s (req) peak t/s peak t/s (req) ttfr (ms) est_ppt (ms) e2e_ttft (ms) Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp2048 (c1) 2195.92 ± 88.95 2195.92 ± 88.95 936.52 ± 37.23 934.75 ± 37.23 936.52 ± 37.23 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c1) 66.82 ± 4.55 66.82 ± 4.55 68.97 ± 4.70 68.97 ± 4.70 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp2048 (c2) 2175.03 ± 13.52 1620.96 ± 532.74 1418.79 ± 465.48 1417.02 ± 465.48 1418.79 ± 465.48 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c2) 40.12 ± 0.82 36.44 ± 14.22 61.00 ± 0.82 40.91 ± 11.42 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp2048 (c4) 2204.24 ± 14.81 1153.90 ± 648.27 2319.89 ± 1041.14 2318.12 ± 1041.14 2319.89 ± 1041.14 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c4) 36.73 ± 0.51 25.62 ± 19.11 104.67 ± 1.25 33.27 ± 15.75 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp2048 (c8) 2176.39 ± 2.29 738.87 ± 592.25 4226.90 ± 2154.08 4225.13 ± 2154.08 4226.90 ± 2154.08 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c8) 34.38 ± 0.40 15.08 ± 14.38 132.00 ± 7.87 20.90 ± 13.77 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp32000 (c1) 1990.71 ± 1.41 1990.71 ± 1.41 16076.91 ± 11.35 16075.14 ± 11.35 16076.91 ± 11.35 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c1) 74.29 ± 9.37 74.29 ± 9.37 76.68 ± 9.67 76.68 ± 9.67 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp32000 (c2) 1977.61 ± 4.66 1487.13 ± 498.28 24241.74 ± 8121.94 24239.97 ± 8121.94 24241.74 ± 8121.94 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c2) 3.71 ± 0.02 36.44 ± 34.03 32.00 ± 0.00 38.99 ± 33.78 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp32000 (c4) 1956.14 ± 1.19 1018.54 ± 568.61 40902.14 ± 18286.44 40900.37 ± 18286.44 40902.14 ± 18286.44 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c4) 2.47 ± 0.04 17.15 ± 23.79 32.00 ± 0.00 20.06 ± 24.93 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp32000 (c8) 1731.80 ± 307.50 521.02 ± 460.47 94488.30 ± 48079.06 94486.53 ± 48079.06 94488.36 ± 48079.10 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c8) 2.05 ± 0.06 7.90 ± 15.94 23.33 ± 12.26 10.79 ± 17.22 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp64000 (c1) 1489.29 ± 52.36 1489.29 ± 52.36 43028.05 ± 1484.93 43026.28 ± 1484.93 43029.51 ± 1482.93 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c1) 49.42 ± 4.98 49.42 ± 4.98 63.96 ± 11.91 63.96 ± 11.91 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp64000 (c2) 1367.74 ± 336.29 966.10 ± 442.01 81524.79 ± 35740.88 81523.02 ± 35740.88 81524.79 ± 35740.88 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c2) 1.49 ± 0.02 27.45 ± 24.44 32.00 ± 0.00 32.43 ± 28.16 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp64000 (c4) 1620.36 ± 21.02 814.26 ± 432.98 100069.83 ± 43246.41 100068.06 ± 43246.41 100069.83 ± 43246.41 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c4) 0.99 ± 0.03 18.92 ± 27.29 32.00 ± 0.00 23.43 ± 28.51 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp64000 (c8) 1691.43 ± 2.78 567.24 ± 451.32 170771.27 ± 86373.88 170769.50 ± 86373.88 170771.27 ± 86373.88 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c8) 0.93 ± 0.00 9.75 ± 17.10 32.00 ± 0.00 13.02 ± 19.26 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp128000 (c1) 1333.24 ± 16.29 1333.24 ± 16.29 96023.90 ± 1182.87 96022.12 ± 1182.87 96023.90 ± 1182.87 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c1) 62.31 ± 9.03 62.31 ± 9.03 110.27 ± 47.76 110.27 ± 47.76 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp128000 (c2) 1361.46 ± 3.35 1020.37 ± 339.68 141079.37 ± 46959.48 141077.60 ± 46959.48 141079.37 ± 46959.48 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c2) 0.59 ± 0.01 63.79 ± 8.14 32.00 ± 0.00 73.59 ± 7.25 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp128000 (c4) 1359.38 ± 1.06 707.95 ± 395.29 235394.83 ± 105268.48 235393.06 ± 105268.48 235394.83 ± 105268.48 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c4) 0.39 ± 0.00 65.37 ± 11.10 32.00 ± 0.00 76.58 ± 13.26 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 pp128000 (c8) 1358.85 ± 1.57 462.11 ± 373.76 423654.57 ± 215887.54 423652.80 ± 215887.54 423654.57 ± 215887.54 Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 tg32 (c8) 0.33 ± 0.00 58.32 ± 14.31 32.00 ± 0.00 68.63 ± 14.86

llama-benchy (0.4.0) date: 2026-08-30 15:13:03 | latency mode: api

2026-04-18 vLLM mit cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit

vLLM optionen:

      --model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
      --tensor-parallel-size 2
      --max-model-len 65536
      --gpu-memory-utilization 0.85
      --enable-prefix-caching
      --reasoning-parser qwen3
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --max-num-seqs 32
      --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

Benchmark mit uvx llama-benchy --base-url http://cogito.buero.ping.de:8000/v1 --depth 2000 32768 63000

model                                        test              t/s       peak t/s         ttfr (ms)      est_ppt (ms)     e2e_ttft (ms)
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit  pp2048 @ d2000 5463.38 ± 111.87                    748.82 ± 14.93    741.48 ± 14.93    748.93 ± 14.93
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit    tg32 @ d2000   103.13 ± 22.06 112.49 ± 24.41                                                      
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit pp2048 @ d32768  5178.25 ± 25.55                   6731.33 ± 33.06   6724.00 ± 33.06   6731.41 ± 33.05
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit   tg32 @ d32768     25.65 ± 1.43   27.93 ± 1.52                                                      
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit pp2048 @ d63000  4534.72 ± 42.10                 14353.15 ± 133.93 14345.82 ± 133.93 14353.26 ± 133.94
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit   tg32 @ d63000     12.85 ± 3.50   14.45 ± 3.21                                                      

Plan: P2P einschalten, da geht noch mehr...