Source files / docs · raw · rendered

reference-video-recipe.md

# Reference build: 2× P100 on a Dell T5810 (YouTube)

A summary of the video description the owner pasted on 2026-11-01. It is the
target configuration for Tersler. Its numbers are the video author's; they
haven't been reproduced here yet.

## Reported results (2× P100, Qwen3.8 27B Q6_K, speculative decoding on)

| Test | Result |
|---|---|
| Reasoning puzzle, reasoning effort "Low" (single run) | 66.7 tok/s (Q4_K_M: 67) |
| ~4,000-token document prompt (median of 3) | 49.2 tok/s |
| Speculative decoding off | 29.7 tok/s |
| Prompt processing | ~400 tok/s |
| 128K context full (thinking off) | 37 tok/s |
| Gemma 4 26B-A4B IQ4_XS, one card | 79 tok/s |
| RTX 3090, same speculative decoding | 66.4 tok/s |

The author reports a later improvement to 74 tok/s (announced as the next
video).

## Their platform

- Dell Precision T5810, Xeon E5-2690 **v4**, 80 GB RAM.
- A second PSU just for card 2.
- Ducted shroud.

## Recipe

### Cards

- Back up the firmware: `nvflash --save p100_cardN_backup.rom`.
- Graphics mode, because the T5810 BIOS can't map 16 GB BAR1:
  `nvflash -i N --gpumode graphics`.
  - The first reboot may power-cycle three times.
  - Re-enable ECC afterwards: `nvidia-smi -i N -e 1` + reboot.
  - On Tersler, try **Above 4G Decoding** first; see
    [hardware-notes.md](hardware-notes.md).
- Power cap: `nvidia-smi -pm 1`, then `nvidia-smi -i N -pl 150`. More power
  gave < 1% more speed.

### System

- Ubuntu 24.04.
- NVIDIA driver **580** (server branch, the last one with Pascal support).
- CUDA **12.9**.
- Kernel boot parameter **`iommu=pt`**, needed for P2P
  (`GGML_CUDA_P2P=1`). Without it, P2P crashed both cards. Without
  `iommu=pt`, leave P2P off; it's worth only a few percent.

### Software

- Kmic-68's P100 fork: <https://github.com/Kmic-68/llama.cpp>, branch
  `p100-optimizations`, tested at commit `d90ec36a3`.
- Build: `-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DGGML_CUDA_NCCL=OFF
  -DGGML_CUDA_FA_QUANTS=all`. sm_60 only: a second architecture disables the
  Pascal tuning. See the repo's `p100-docs/BUILD.md`.

### Model

Qwen3.8-27B **Q6_K** GGUF **including the MTP head**. The author ran a
community finetune of the same size (`Qwen3.8-27B-Uncensored-Q6_K.gguf`).

### Launch

```bash
GGML_CUDA_P2P=1 GGML_CUDA_GRAPHS_PRE_VOLTA=3 LLAMA_SPEC_SAMPLE_TEMP=1.0 LLAMA_SPEC_DRAFT_TOPK=20 \
llama-server -m Qwen3.8-27B-Uncensored-Q6_K.gguf -ngl 99 -sm tensor -fa 1 -ctk q4_0 -ctv q4_0 \
  -c 131072 -b 32768 -ub 2048 -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.2 -ngld 99 -ubd 64 -ctkd q4_0 -ctvd q4_0 \
  --jinja --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0
```

Their notes on the command:

- Drop `GGML_CUDA_P2P=1` without `iommu=pt`.
- Two drafts vs four: about the same.
- One card at PCIe x8: ~2% slower.

## Related projects (credited in the video)

- Kmic: <https://github.com/Kmic-68/llama.cpp>
- fallentree GP100 fork: <https://github.com/thefallentree/llama.cpp> (URL
  truncated in the description; verify)
- qr0n P100 patches: <https://github.com/key-sh/p100-llama-cpp>
- Upstream: <https://github.com/ggml-org/llama.cpp>