Source files / docs · raw · rendered

keeping-up.md

# Keeping up with upstream

The Pascal/P100 llama.cpp scene moves fast: forks get new branches,
speculative-decoding modes change flags, and models get re-quantized. Hardware
support, on the other hand, is frozen: **driver R580 and CUDA 12.x are the end
of the line for Pascal.** Tersler handles this with three pieces:

1. **Pin everything.** `install/tersler.conf` pins the fork, branch, commit,
   driver package, CUDA toolkit and launch flags. The installer holds the
   driver package (`apt-mark hold`) and pins NVIDIA-repo drivers out, so
   nothing changes behind your back.
2. **Detect changes.** `scripts/check-updates.sh` (installed as
   `tersler-check-updates`) reports:
   - new commits on the fork beyond the pin, with the commit log
   - new branches on the fork
   - changes in the other repos in `install/upstreams.list`
   - new GitHub repos matching P100/Pascal searches
   - driver and CUDA package updates (on the rig)
   - model repo updates on Hugging Face (`MODEL_HF_REPO`)

   It exits with code `10` when something is new, so it works in cron or a
   systemd timer.
3. **Try, measure, then switch.** Builds are versioned side by side under
   `/opt/tersler/builds/<commit>`. `current` points to the active build, and
   `previous` keeps the last one.

## Workflow

```bash
# 1. what's new?
tersler-check-updates -o ~/tersler-updates/$(date +%F).md

# 2. build the candidate next to the current one (doesn't touch the running server)
sudo systemctl stop tersler-llama           # two models don't fit in VRAM
LLAMA_COMMIT=<new-commit> sudo -E ./install/install.sh build
#    the build stage switches `current` to it, so remember the old one:
ls -l /opt/tersler/previous
#    a full build takes ~25 min and ~1.3 GB (fork's BUILD.md)

# 2b. correctness first: the fork ships a gate (decode bench, perplexity, FA op tests).
#     Check its usage in the source tree, it may expect its own build dir:
less /opt/tersler/src/llama.cpp/p100-docs/BUILD.md     # section "Verify"
#     expected per the fork: perplexity ≈ 2.610 (band 2.6209 ± 0.0199); tg256 ≈ 32 t/s cold
#     a change can pass all op tests and still produce NaN in real inference
#     (perplexity catches that), so never skip this step.

# 3. compare: same model, prompts and settings
tersler-bench /opt/tersler/previous
tersler-bench /opt/tersler/current
column -s, -t < /opt/tersler/bench/results.csv | tail -12

# 4a. keep it: pin the new commit in install/tersler.conf, commit to git
# 4b. roll back:
sudo ln -sfn "$(readlink -f /opt/tersler/previous)" /opt/tersler/current

sudo systemctl start tersler-llama
```

- **New launch flags:** read the fork's docs (`p100-docs/` in the fork) and
  diff its README. Try them by appending flags to `tersler-server` before
  changing `LLAMA_ARGS`.
- **New fork or branch:** change `LLAMA_REPO`/`LLAMA_BRANCH` in
  `install/tersler.local.conf`, then build and bench as above.

## Automating the check

On the rig, a weekly user timer:

```bash
mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/tersler-updates.service <<'EOF'
[Service]
Type=oneshot
ExecStart=/bin/sh -c '/opt/tersler/bin/tersler-check-updates -o %h/tersler-updates/latest.md || [ $? -eq 10 ]'
EOF
cat > ~/.config/systemd/user/tersler-updates.timer <<'EOF'
[Timer]
OnCalendar=weekly
Persistent=true
[Install]
WantedBy=timers.target
EOF
systemctl --user enable --now tersler-updates.timer
```

The script can't judge whether a change is *good*. For that, read the commit
log, benchmark, and record the result in [changelog.md](changelog.md).

## Hard limits (don't "upgrade" past these)

| Component | Limit | Why |
|---|---|---|
| NVIDIA driver | R580 branch | the last branch with Pascal support |
| Kernel modules | proprietary, not `-open` | the open modules don't support Pascal |
| CUDA | 12.x | CUDA 13 dropped Pascal |
| llama.cpp build | `CMAKE_CUDA_ARCHITECTURES=60` only | a second architecture disables the fork's Pascal tuning |
| Ubuntu | 24.04 LTS | R580 + CUDA 12.9 are packaged for it; check before moving to a newer release |

## Notes from the fork's BUILD.md (read 2026-11-03)

- The CMake flags match `install/tersler.conf` exactly.
- `GGML_CUDA_FA_QUANTS=all` compiles flash attention for every K/V type pair.
  This makes `libggml-cuda.so` bigger, and the library is loaded into **VRAM
  on each card**. Growing it from 374 to 530 MB once broke a 262144-token
  context. If the 128K configuration stops starting after an update, check the
  library size first. Upstream's default set (`q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16`)
  covers our q4_0 KV cache and builds a smaller library.
- `--version` reports the commit at *configure* time and can lag. Tersler
  writes the real commit to `build-info.json` in every build dir.