Sanitizer CI: periodic ASan + UBSan build and smoke run, off the release path #1248

Open
opened 2026-10-28 20:33:40 +00:00 by sky_line · 0 comments
Owner

Part of #1243.

Scope. The tree has no sanitizer configuration (USE_ASAN only adds -Og; nothing passes -fsanitize). Add real ones and run them regularly on a box that is not on the release path.

  1. CMake: SL_SANITIZE option (address, undefined, address;undefined, thread as a separate non-default), with presets linux-asan (clang or GCC, -fno-omit-frame-pointer, -O1, -g) and linux-ubsan, own binary dirs, -fno-sanitize=alignment plus the suppression file (indra/fuzz/ubsan.supp pattern), never combined with LL_FUZZ or windows-cross. Reuse indra/cmake/LLFuzz.cmake flag logic rather than a second copy.
  2. Known requirements to bake in: ASAN_OPTIONS=protect_shadow_gap=0 (NVIDIA driver maps into the shadow gap), detect_leaks=0 initially (LL's lazily grown tables), the existing "ASan VK build" notes in the maintainer's reference; vulkan loader interposition (LD_PRELOAD-free: link ASan runtime first); the shader-binary cache must be deleted per run (memory note: stale .shaderbin).
  3. Runs:
    • Weekly (and on demand): build linux-asan on applejack, run the default unit suite plus the fuzz replay tests under it (cheap, catches memory errors in all unit-tested modules).
    • Weekly smoke on the garbage test box (GTX 1080) through testctl: package the ASan build with TESTCTL=1, run make validate SCENARIO=core (login, teleport, rez, close) with a 3x timeout multiplier, collect the ASan reports as artifacts. A clean core run is the bar; sync is optional.
    • UBSan build: same, unit suite plus the core scenario.
  4. Triage: reports are deduplicated by top frame; new ones open a ticket labelled sanitizer; known benign ones go in the suppression file with a comment and a ticket reference.
  5. Off the release path: not part of release.yml, no tag triggers it, failures never block a beta; the release checklist only asks that no sanitizer-labelled P1 is open.

Acceptance criteria.

  • cmake --preset linux-asan && cmake --build --preset linux-asan produces a working sanitised viewer binary and unit test executables; documented in doc/DEVELOPER_GUIDE.md.
  • The weekly job completes, publishes artifacts, and the first run's findings are triaged (fixed, suppressed with justification, or ticketed) before the job is declared done.
  • A planted use-after-free in a unit-tested module is caught by the weekly unit-suite run.
  • Wall time: ASan unit suite under 10 minutes; smoke run under 45 minutes; RAM use of the sanitised viewer on the 1080 box recorded (expect 2x or more).

Effort. About 2 to 3 days for presets and the CI workflow; open-ended triage of the first findings (budget 2 days, spill into separate tickets). Risk: the viewer's 2.8 GB debug-info binary under ASan is slow to link and large; use -gline-tables-only for the sanitised build.

Part of #1243. **Scope.** The tree has no sanitizer configuration (`USE_ASAN` only adds `-Og`; nothing passes `-fsanitize`). Add real ones and run them regularly on a box that is not on the release path. 1. CMake: `SL_SANITIZE` option (`address`, `undefined`, `address;undefined`, `thread` as a separate non-default), with presets `linux-asan` (clang or GCC, `-fno-omit-frame-pointer`, `-O1`, `-g`) and `linux-ubsan`, own binary dirs, `-fno-sanitize=alignment` plus the suppression file (`indra/fuzz/ubsan.supp` pattern), never combined with `LL_FUZZ` or `windows-cross`. Reuse `indra/cmake/LLFuzz.cmake` flag logic rather than a second copy. 2. Known requirements to bake in: `ASAN_OPTIONS=protect_shadow_gap=0` (NVIDIA driver maps into the shadow gap), `detect_leaks=0` initially (LL's lazily grown tables), the existing "ASan VK build" notes in the maintainer's reference; vulkan loader interposition (`LD_PRELOAD`-free: link ASan runtime first); the shader-binary cache must be deleted per run (memory note: stale `.shaderbin`). 3. Runs: - Weekly (and on demand): build `linux-asan` on applejack, run the default unit suite plus the fuzz replay tests under it (cheap, catches memory errors in all unit-tested modules). - Weekly smoke on the garbage test box (GTX 1080) through testctl: package the ASan build with `TESTCTL=1`, run `make validate SCENARIO=core` (login, teleport, rez, close) with a 3x timeout multiplier, collect the ASan reports as artifacts. A clean `core` run is the bar; `sync` is optional. - UBSan build: same, unit suite plus the `core` scenario. 4. Triage: reports are deduplicated by top frame; new ones open a ticket labelled `sanitizer`; known benign ones go in the suppression file with a comment and a ticket reference. 5. Off the release path: not part of `release.yml`, no tag triggers it, failures never block a beta; the release checklist only asks that no `sanitizer`-labelled P1 is open. **Acceptance criteria.** - `cmake --preset linux-asan && cmake --build --preset linux-asan` produces a working sanitised viewer binary and unit test executables; documented in `doc/DEVELOPER_GUIDE.md`. - The weekly job completes, publishes artifacts, and the first run's findings are triaged (fixed, suppressed with justification, or ticketed) before the job is declared done. - A planted use-after-free in a unit-tested module is caught by the weekly unit-suite run. - Wall time: ASan unit suite under 10 minutes; smoke run under 45 minutes; RAM use of the sanitised viewer on the 1080 box recorded (expect 2x or more). **Effort.** About 2 to 3 days for presets and the CI workflow; open-ended triage of the first findings (budget 2 days, spill into separate tickets). Risk: the viewer's 2.8 GB debug-info binary under ASan is slow to link and large; use `-gline-tables-only` for the sanitised build.
sky_line added this to the Slipstream project 2026-10-28 20:33:40 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sky_line/Slipstream#1248
No description provided.