Fuzzing in CI: nightly runs per target, crash regressions kept as tests #1247

Open
opened 2026-10-28 20:33:40 +00:00 by sky_line · 0 comments
Owner

Part of #1243.

Scope. The harnesses and runner exist on branch fuzzing (indra/fuzz, scripts/fuzz/run.sh, preset linux-fuzz). This ticket makes them continuous.

  1. Nightly Forgejo Actions workflow on applejack (CT 125): configure linux-fuzz, build fuzz_all (cache the build dir; a full clang + ASan build of the libraries is slow, so incremental), run scripts/fuzz/run.sh all 120 (2 minutes per target, about 40 minutes total; make the budget a workflow input), upload fuzz-work/*/crashes/ as artifacts on failure, restore and save the grown corpora between runs (cache keyed per target) and run.sh merge weekly to keep them small.
  2. Failure handling: a crash opens (or comments on) a ticket labelled fuzz with the target, the sanitizer stack and the artifact path; the workflow never auto-closes.
  3. Crash regressions as unit tests. A fix for a fuzz finding must land with its input in indra/fuzz/corpus/<name>/crash-* and a replay test in the default suite: build each harness a second time without libFuzzer (a tiny standalone driver StandaloneFuzzTargetMain.cpp that calls LLVMFuzzerTestOneInput for each file argument) under the normal GCC build as fuzz_replay_<name>, registered with sl_add_unit_test under label fast (the seed corpora are tiny, so seconds). Under GCC without sanitizers a replay will only catch crashes and LL_ERRS; the same replay under the sanitizer CI build (next ticket) catches memory errors.
  4. run.sh check (one pass over seeds with the libFuzzer binaries) as a post-merge job on the sanitizer-capable CI box.
  5. Keep harness coverage honest: a scripts/check module that fails when a parser listed in indra/fuzz/README.md loses its harness, and README "not covered" items (SLBridge, weather tag, mesh header, glTF asset loader, DXT) turn into tickets as their refactors land.

Acceptance criteria.

  • Nightly job green on applejack for 7 consecutive nights with no manual intervention, total wall time under 90 minutes.
  • Planting a known bug (e.g. remove a bounds check in LLSDBinaryParser) is found by the nightly run within its budget and produces the artifact and ticket.
  • Every existing corpus file is replayed by a fast unit test in the default build; the added replay time is under 1 s.
  • indra/fuzz/README.md documents the CI schedule and the "add a regression" flow (already written for the manual case).

Effort. 2 to 3 days for the workflow, caches and replay driver. Prerequisite: the first main-session build of fuzz_all (the harnesses on the branch were written without compiling; expect a round of link fixes, e.g. llprimitive's link closure for fuzz_mesh_* and fuzz_gltf_material).

Part of #1243. **Scope.** The harnesses and runner exist on branch `fuzzing` (`indra/fuzz`, `scripts/fuzz/run.sh`, preset `linux-fuzz`). This ticket makes them continuous. 1. Nightly Forgejo Actions workflow on applejack (CT 125): configure `linux-fuzz`, build `fuzz_all` (cache the build dir; a full clang + ASan build of the libraries is slow, so incremental), run `scripts/fuzz/run.sh all 120` (2 minutes per target, about 40 minutes total; make the budget a workflow input), upload `fuzz-work/*/crashes/` as artifacts on failure, restore and save the grown corpora between runs (cache keyed per target) and `run.sh merge` weekly to keep them small. 2. Failure handling: a crash opens (or comments on) a ticket labelled `fuzz` with the target, the sanitizer stack and the artifact path; the workflow never auto-closes. 3. **Crash regressions as unit tests.** A fix for a fuzz finding must land with its input in `indra/fuzz/corpus/<name>/crash-*` and a replay test in the default suite: build each harness a second time **without** libFuzzer (a tiny standalone driver `StandaloneFuzzTargetMain.cpp` that calls `LLVMFuzzerTestOneInput` for each file argument) under the normal GCC build as `fuzz_replay_<name>`, registered with `sl_add_unit_test` under label `fast` (the seed corpora are tiny, so seconds). Under GCC without sanitizers a replay will only catch crashes and `LL_ERRS`; the same replay under the sanitizer CI build (next ticket) catches memory errors. 4. `run.sh check` (one pass over seeds with the libFuzzer binaries) as a post-merge job on the sanitizer-capable CI box. 5. Keep harness coverage honest: a `scripts/check` module that fails when a parser listed in `indra/fuzz/README.md` loses its harness, and README "not covered" items (SLBridge, weather tag, mesh header, glTF asset loader, DXT) turn into tickets as their refactors land. **Acceptance criteria.** - Nightly job green on applejack for 7 consecutive nights with no manual intervention, total wall time under 90 minutes. - Planting a known bug (e.g. remove a bounds check in `LLSDBinaryParser`) is found by the nightly run within its budget and produces the artifact and ticket. - Every existing corpus file is replayed by a `fast` unit test in the default build; the added replay time is under 1 s. - `indra/fuzz/README.md` documents the CI schedule and the "add a regression" flow (already written for the manual case). **Effort.** 2 to 3 days for the workflow, caches and replay driver. Prerequisite: the first main-session build of `fuzz_all` (the harnesses on the branch were written without compiling; expect a round of link fixes, e.g. llprimitive's link closure for `fuzz_mesh_*` and `fuzz_gltf_material`).
sky_line added this to the Slipstream project 2026-10-28 20:33:40 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sky_line/Slipstream#1247
No description provided.