# Inferenz Benchmarks

#### 2026-09-21 vLLM mit Qwen 3.8 27b INT8
(config in /opt/vllm-qwen38)

Benchmark-Aufruf auf milla:

```bash
uvx llama-benchy --base-url $OPENAI_API_URL --api-key $OPENAI_API_KEY --pp 32000 --no-cache --concurrency 4 --format md --model Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 --served-model-name qwen38-27b  --runs 2
```

| model                                        |         test |    t/s (total) |        t/s (req) |     peak t/s |   peak t/s (req) |           ttfr (ms) |        est_ppt (ms) |       e2e_ttft (ms) |
|:---------------------------------------------|-------------:|---------------:|-----------------:|-------------:|-----------------:|--------------------:|--------------------:|--------------------:|
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp32000 (c4) | 1927.26 ± 3.08 | 1006.52 ± 564.21 |              |                  | 41470.01 ± 18590.88 | 41467.98 ± 18590.88 | 41470.01 ± 18590.88 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |    tg32 (c4) |    2.46 ± 0.00 |    20.87 ± 30.50 | 32.00 ± 0.00 |    23.31 ± 30.47 |                     |                     |                     |

llama-benchy (0.4.0)
date: 2026-09-21 19:36:48 | latency mode: api

#### 2026-08-30 vLLM mit Qwen 3.8 27b INT8

Benchmark-Aufruf auf milla:
```bash
uvx llama-benchy --base-url $OPENAI_API_URL --api-key $OPENAI_API_KEY --pp 2048 32000 64000 128000 \ 
--no-cache --concurrency 1 2 4 8 --format md --model Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 \ 
--served-model-name qwen38-27b
```

| model                                        |          test |      t/s (total) |        t/s (req) |       peak t/s |   peak t/s (req) |             ttfr (ms) |          est_ppt (ms) |         e2e_ttft (ms) |
|:---------------------------------------------|--------------:|-----------------:|-----------------:|---------------:|-----------------:|----------------------:|----------------------:|----------------------:|
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |   pp2048 (c1) |  2195.92 ± 88.95 |  2195.92 ± 88.95 |                |                  |        936.52 ± 37.23 |        934.75 ± 37.23 |        936.52 ± 37.23 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c1) |     66.82 ± 4.55 |     66.82 ± 4.55 |   68.97 ± 4.70 |     68.97 ± 4.70 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |   pp2048 (c2) |  2175.03 ± 13.52 | 1620.96 ± 532.74 |                |                  |      1418.79 ± 465.48 |      1417.02 ± 465.48 |      1418.79 ± 465.48 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c2) |     40.12 ± 0.82 |    36.44 ± 14.22 |   61.00 ± 0.82 |    40.91 ± 11.42 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |   pp2048 (c4) |  2204.24 ± 14.81 | 1153.90 ± 648.27 |                |                  |     2319.89 ± 1041.14 |     2318.12 ± 1041.14 |     2319.89 ± 1041.14 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c4) |     36.73 ± 0.51 |    25.62 ± 19.11 |  104.67 ± 1.25 |    33.27 ± 15.75 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |   pp2048 (c8) |   2176.39 ± 2.29 |  738.87 ± 592.25 |                |                  |     4226.90 ± 2154.08 |     4225.13 ± 2154.08 |     4226.90 ± 2154.08 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c8) |     34.38 ± 0.40 |    15.08 ± 14.38 |  132.00 ± 7.87 |    20.90 ± 13.77 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |  pp32000 (c1) |   1990.71 ± 1.41 |   1990.71 ± 1.41 |                |                  |      16076.91 ± 11.35 |      16075.14 ± 11.35 |      16076.91 ± 11.35 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c1) |     74.29 ± 9.37 |     74.29 ± 9.37 |   76.68 ± 9.67 |     76.68 ± 9.67 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |  pp32000 (c2) |   1977.61 ± 4.66 | 1487.13 ± 498.28 |                |                  |    24241.74 ± 8121.94 |    24239.97 ± 8121.94 |    24241.74 ± 8121.94 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c2) |      3.71 ± 0.02 |    36.44 ± 34.03 |   32.00 ± 0.00 |    38.99 ± 33.78 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |  pp32000 (c4) |   1956.14 ± 1.19 | 1018.54 ± 568.61 |                |                  |   40902.14 ± 18286.44 |   40900.37 ± 18286.44 |   40902.14 ± 18286.44 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c4) |      2.47 ± 0.04 |    17.15 ± 23.79 |   32.00 ± 0.00 |    20.06 ± 24.93 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |  pp32000 (c8) | 1731.80 ± 307.50 |  521.02 ± 460.47 |                |                  |   94488.30 ± 48079.06 |   94486.53 ± 48079.06 |   94488.36 ± 48079.10 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c8) |      2.05 ± 0.06 |     7.90 ± 15.94 |  23.33 ± 12.26 |    10.79 ± 17.22 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |  pp64000 (c1) |  1489.29 ± 52.36 |  1489.29 ± 52.36 |                |                  |    43028.05 ± 1484.93 |    43026.28 ± 1484.93 |    43029.51 ± 1482.93 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c1) |     49.42 ± 4.98 |     49.42 ± 4.98 |  63.96 ± 11.91 |    63.96 ± 11.91 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |  pp64000 (c2) | 1367.74 ± 336.29 |  966.10 ± 442.01 |                |                  |   81524.79 ± 35740.88 |   81523.02 ± 35740.88 |   81524.79 ± 35740.88 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c2) |      1.49 ± 0.02 |    27.45 ± 24.44 |   32.00 ± 0.00 |    32.43 ± 28.16 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |  pp64000 (c4) |  1620.36 ± 21.02 |  814.26 ± 432.98 |                |                  |  100069.83 ± 43246.41 |  100068.06 ± 43246.41 |  100069.83 ± 43246.41 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c4) |      0.99 ± 0.03 |    18.92 ± 27.29 |   32.00 ± 0.00 |    23.43 ± 28.51 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |  pp64000 (c8) |   1691.43 ± 2.78 |  567.24 ± 451.32 |                |                  |  170771.27 ± 86373.88 |  170769.50 ± 86373.88 |  170771.27 ± 86373.88 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c8) |      0.93 ± 0.00 |     9.75 ± 17.10 |   32.00 ± 0.00 |    13.02 ± 19.26 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp128000 (c1) |  1333.24 ± 16.29 |  1333.24 ± 16.29 |                |                  |    96023.90 ± 1182.87 |    96022.12 ± 1182.87 |    96023.90 ± 1182.87 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c1) |     62.31 ± 9.03 |     62.31 ± 9.03 | 110.27 ± 47.76 |   110.27 ± 47.76 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp128000 (c2) |   1361.46 ± 3.35 | 1020.37 ± 339.68 |                |                  |  141079.37 ± 46959.48 |  141077.60 ± 46959.48 |  141079.37 ± 46959.48 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c2) |      0.59 ± 0.01 |     63.79 ± 8.14 |   32.00 ± 0.00 |     73.59 ± 7.25 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp128000 (c4) |   1359.38 ± 1.06 |  707.95 ± 395.29 |                |                  | 235394.83 ± 105268.48 | 235393.06 ± 105268.48 | 235394.83 ± 105268.48 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c4) |      0.39 ± 0.00 |    65.37 ± 11.10 |   32.00 ± 0.00 |    76.58 ± 13.26 |                       |                       |                       |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 | pp128000 (c8) |   1358.85 ± 1.57 |  462.11 ± 373.76 |                |                  | 423654.57 ± 215887.54 | 423652.80 ± 215887.54 | 423654.57 ± 215887.54 |
| Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 |     tg32 (c8) |      0.33 ± 0.00 |    58.32 ± 14.31 |   32.00 ± 0.00 |    68.63 ± 14.86 |                       |                       |                       |

llama-benchy (0.4.0)
date: 2026-08-30 15:13:03 | latency mode: api

#### 2026-04-18 vLLM mit cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit

vLLM optionen:
```      
      --model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
      --tensor-parallel-size 2
      --max-model-len 65536
      --gpu-memory-utilization 0.85
      --enable-prefix-caching
      --reasoning-parser qwen3
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --max-num-seqs 32
      --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
```

Benchmark mit `uvx llama-benchy --base-url http://cogito.buero.ping.de:8000/v1 --depth 2000 32768 63000`

| model                             |            test |              t/s |       peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:----------------------------------|----------------:|-----------------:|---------------:|------------------:|------------------:|------------------:|
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit |  pp2048 @ d2000 | 5463.38 ± 111.87 |                |    748.82 ± 14.93 |    741.48 ± 14.93 |    748.93 ± 14.93 |
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit |    tg32 @ d2000 |   103.13 ± 22.06 | 112.49 ± 24.41 |                   |                   |                   |
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit | pp2048 @ d32768 |  5178.25 ± 25.55 |                |   6731.33 ± 33.06 |   6724.00 ± 33.06 |   6731.41 ± 33.05 |
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit |   tg32 @ d32768 |     25.65 ± 1.43 |   27.93 ± 1.52 |                   |                   |                   |
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit | pp2048 @ d63000 |  4534.72 ± 42.10 |                | 14353.15 ± 133.93 | 14345.82 ± 133.93 | 14353.26 ± 133.94 |
| cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit |   tg32 @ d63000 |     12.85 ± 3.50 |   14.45 ± 3.21 |                   |                   |                   |

Plan: P2P einschalten, da geht noch mehr...