Move CI runners off the workstation onto a build server #103

New issue

Closed

opened 2026-09-27 13:01:14 +00:00 by backspace119 · 2 comments

backspace119 commented

2026-09-27 13:01:14 +00:00

Owner

Both release jobs (Linux tarball, Windows zip) run on the cachyos host runner, which is the dev workstation. Move them to a server.

What currently ties CI to this machine

Hard-coded paths in .forgejo/workflows/release.yml: CI_ROOT=/home/bryan/ci/badalchestorm, CI_ROOT_WIN=/home/bryan/ci/badalchestorm-win, and CACHE_SRC pointing at a dev worktree (.claude/worktrees/agent-…) for .venv, .build-variables, .autobuild-installables*. Replace with runner-local cache dirs.
Host toolchain (Linux job): CachyOS GCC plus the portability guarantees that assume it: glibc_compat.h symver pins, fix_isa_note.py, make glibc-check (glibc <= 2.38, x86-64-v2). A different server distro changes the compiler and sysroot, so re-verify glibc-check/isa-check output, or pin a container image that matches.
Windows cross toolchain: clang/lld/llvm (clang-cl, lld-link, llvm-rc/mt/lib/dlltool), 7z, the Vulkan loader + headers (for make_vulkan_lib.sh), ~/.xwin (MSVC CRT + Windows SDK via xwin --accept-license splat; the license must be accepted for the server too), ~/.cache/wincross/crt (from fetch_vcredist.py).
Python venv with autobuild + llsd, and the build-variables clone.
Secrets: RELEASE_TOKEN (repo read/write). Uploads use the LAN API (192.168.96.20:3000) because Cloudflare caps bodies at 100 MB, so the server must reach Forgejo on the LAN.

Plan

Container image (recommended): one image with the Linux toolchain, clang/lld/llvm, 7z, Vulkan headers/loader, a pre-splatted xwin sysroot and the CRT DLLs baked in (or mounted as a volume). Runner label e.g. builder; jobs runs-on: builder with container:. Makes the toolchain reproducible and the machine disposable.
- Alternative: a plain host runner on the server with the same packages installed. Faster to set up, easier to drift.
Persistent volumes for build dirs and caches (incremental builds matter: a cold Linux build is ~1300 TUs; the Windows cross build is similar). Budget ~60 GB disk, plus the 3p caches (~2 GB) and xwin (~0.7 GB).
Parameterise paths via env/vars instead of /home/bryan/...; drop CACHE_SRC.
Run a ci-* dry run on the new runner, compare glibc-check/isa-check output and a Wine smoke of the Windows zip against the current builds, then retire the cachyos label from the workstation.
Optional: a Wine startup smoke test in CI (headless via Xvfb + lavapipe) so a broken Windows build fails before release.

Sizing notes

The workstation has 48 cores (Linux build ~minutes). A smaller server works, but expect 30–60 min cold builds; persistent build dirs keep tagged releases incremental.

Both release jobs (Linux tarball, Windows zip) run on the `cachyos` **host** runner, which is the dev workstation. Move them to a server. ### What currently ties CI to this machine - **Hard-coded paths in `.forgejo/workflows/release.yml`:** `CI_ROOT=/home/bryan/ci/badalchestorm`, `CI_ROOT_WIN=/home/bryan/ci/badalchestorm-win`, and `CACHE_SRC` pointing at a **dev worktree** (`.claude/worktrees/agent-…`) for `.venv`, `.build-variables`, `.autobuild-installables*`. Replace with runner-local cache dirs. - **Host toolchain (Linux job):** CachyOS GCC plus the portability guarantees that assume it: `glibc_compat.h` symver pins, `fix_isa_note.py`, `make glibc-check` (glibc <= 2.38, x86-64-v2). A different server distro changes the compiler and sysroot, so re-verify `glibc-check`/`isa-check` output, or pin a container image that matches. - **Windows cross toolchain:** clang/lld/llvm (clang-cl, lld-link, llvm-rc/mt/lib/dlltool), 7z, the Vulkan loader + headers (for `make_vulkan_lib.sh`), `~/.xwin` (MSVC CRT + Windows SDK via `xwin --accept-license splat`; the license must be accepted for the server too), `~/.cache/wincross/crt` (from `fetch_vcredist.py`). - **Python venv** with autobuild + llsd, and the `build-variables` clone. - **Secrets:** `RELEASE_TOKEN` (repo read/write). Uploads use the LAN API (`192.168.96.20:3000`) because Cloudflare caps bodies at 100 MB, so the server must reach Forgejo on the LAN. ### Plan 1. **Container image** (recommended): one image with the Linux toolchain, clang/lld/llvm, 7z, Vulkan headers/loader, a pre-splatted xwin sysroot and the CRT DLLs baked in (or mounted as a volume). Runner label e.g. `builder`; jobs `runs-on: builder` with `container:`. Makes the toolchain reproducible and the machine disposable. - Alternative: a plain host runner on the server with the same packages installed. Faster to set up, easier to drift. 2. Persistent volumes for build dirs and caches (incremental builds matter: a cold Linux build is ~1300 TUs; the Windows cross build is similar). Budget ~60 GB disk, plus the 3p caches (~2 GB) and xwin (~0.7 GB). 3. Parameterise paths via env/`vars` instead of `/home/bryan/...`; drop `CACHE_SRC`. 4. Run a `ci-*` dry run on the new runner, compare `glibc-check`/`isa-check` output and a Wine smoke of the Windows zip against the current builds, then retire the `cachyos` label from the workstation. 5. Optional: a Wine startup smoke test in CI (headless via Xvfb + lavapipe) so a broken Windows build fails before release. ### Sizing notes The workstation has 48 cores (Linux build ~minutes). A smaller server works, but expect 30–60 min cold builds; persistent build dirs keep tagged releases incremental.

backspace119 added the

platform

label

2026-09-27 13:01:14 +00:00

backspace119 referenced this issue

2026-09-27 20:46:11 +00:00

Move git host to git.slipstreamviewer.org; host slipstreamviewer.org (.com redirect) #127

backspace119 referenced this issue from a commit

2026-09-28 22:51:31 +00:00

CI: run releases on applejack (builder), not the workstation

backspace119 commented

2026-09-28 23:25:10 +00:00

Author

Owner

Done: CI now runs on applejack. It's a dedicated LXC (CT 125 on rainbowcrash) in the DMZ at 192.168.96.24: Ubuntu 24.04, 16 cores, 32 GB RAM, 200 GB disk.

Setup

Networking: DNS 192.168.96.1, search domain dmz.internal, apt forced to IPv4 (the DMZ has no IPv6 route). It reaches Forgejo over VLAN 96 only; there are no inbound ports.
Toolchain:
- GCC 14 as the default cc/c++, CMake 4.4 from Kitware. The stock 3.28 can't find the lib-prefixed Windows 3p libraries.
- clang/LLD 22 from apt.llvm.org, with clang-cl, lld-link, llvm-{rc,mt,lib,dlltool} linked into /usr/local/bin.
- The xwin sysroot and VC CRT are copied from the workstation to ~ci/.xwin and ~ci/.cache/wincross/crt, so both machines use the identical SDK.
Runner: forgejo-runner v13.2.0 as the system service forgejo-runner.service (User=ci), registered as applejack with label builder, capacity 1.
Workflow: both jobs run on runs-on: builder. Persistent trees are /srv/ci/slipstream and /srv/ci/slipstream-win, and each keeps its own venv, build-variables and 3p cache. Nothing refers to a developer machine any more.
Verified: dry run ci-applejack-2 passed both jobs.
Workstation runner: stopped and disabled. The offline cachyos runner registration can be deleted in the Forgejo UI.

Privacy fix (the PII leak)

0.12.0-beta.2's Linux binary contained 766 absolute paths like /home/<user>/ci/..., from __FILE__ in LL and Boost macros.
a5f2ee564f adds -ffile-prefix-map=<repo>=. (GCC and clang-cl) and /pdbaltpath:%_PDB% (Windows).
The applejack build now has no build-machine paths. The only remaining paths are from the 3p prebuilts compiled on GitHub's runners (/home/runner/work/3p-*), and the Windows exe references only slipstream-bin.pdb.

Portability

Linux floor on applejack: GLIBC_2.38, GLIBCXX_3.4.32.
This also resolves #147: the tester's system has 3.4.33.

One follow-up remains: extend make glibc-check to fail on GLIBCXX above a chosen floor, so a toolchain change can't raise it silently again.

**Done: CI now runs on applejack.** It's a dedicated LXC (CT 125 on rainbowcrash) in the DMZ at `192.168.96.24`: Ubuntu 24.04, 16 cores, 32 GB RAM, 200 GB disk. **Setup** - **Networking:** DNS `192.168.96.1`, search domain `dmz.internal`, apt forced to IPv4 (the DMZ has no IPv6 route). It reaches Forgejo over VLAN 96 only; there are no inbound ports. - **Toolchain:** - GCC 14 as the default `cc`/`c++`, CMake 4.4 from Kitware. The stock 3.28 can't find the `lib`-prefixed Windows 3p libraries. - clang/LLD 22 from apt.llvm.org, with `clang-cl`, `lld-link`, `llvm-{rc,mt,lib,dlltool}` linked into `/usr/local/bin`. - The xwin sysroot and VC CRT are copied from the workstation to `~ci/.xwin` and `~ci/.cache/wincross/crt`, so both machines use the identical SDK. - **Runner:** `forgejo-runner` v13.2.0 as the system service `forgejo-runner.service` (User=ci), registered as **applejack** with label **builder**, capacity 1. - **Workflow:** both jobs run on `runs-on: builder`. Persistent trees are `/srv/ci/slipstream` and `/srv/ci/slipstream-win`, and each keeps its own venv, build-variables and 3p cache. Nothing refers to a developer machine any more. - **Verified:** dry run `ci-applejack-2` passed both jobs. - **Workstation runner:** stopped and disabled. The offline `cachyos` runner registration can be deleted in the Forgejo UI. **Privacy fix (the PII leak)** - 0.12.0-beta.2's Linux binary contained **766** absolute paths like `/home/<user>/ci/...`, from `__FILE__` in LL and Boost macros. - a5f2ee564f adds `-ffile-prefix-map=<repo>=.` (GCC and clang-cl) and `/pdbaltpath:%_PDB%` (Windows). - The applejack build now has no build-machine paths. The only remaining paths are from the 3p prebuilts compiled on GitHub's runners (`/home/runner/work/3p-*`), and the Windows exe references only `slipstream-bin.pdb`. **Portability** - Linux floor on applejack: `GLIBC_2.38`, `GLIBCXX_3.4.32`. - This also resolves #147: the tester's system has 3.4.33. One follow-up remains: extend `make glibc-check` to fail on GLIBCXX above a chosen floor, so a toolchain change can't raise it silently again.

backspace119 commented

2026-09-28 23:49:43 +00:00

Author

Owner

Tested and shipped in v0.12.0-beta.3.

backspace119 closed this issue

2026-09-28 23:49:43 +00:00