Epic: unit tests that run at build time and fail the build on regression #1243

Open
opened 2026-10-28 20:33:40 +00:00 by sky_line · 0 comments
Owner

Why. Slipstream has a lot of pure logic that regresses silently (weather maths, run-off, cache / segment store, inventory safety, key-binding migration, SLBridge line parsing, LLSD round trips) and the only automated gates today are make check (source / package lint) and the manual make validate runtime suite on the test box. A bad change is found by a person, in a running viewer, or after a tag. Several bug classes in this repo (Windows macro names, glibc pins, LL_ERRS reachable from input) already have a mechanical guard; logic regressions do not.

State of the inherited LL test framework (verified in the tree, 2026-10-28).

  • It is still in the tree and wired: indra/test (the lltest runner: test.cpp main + lltut.*, 25 *_tut.cpp suites for llmessage / llcommon / llinventory), per-library tests/*_test.cpp files (about 40 in llcommon alone, plus llimage, llmath, llmessage, llinventory, llprimitive, llui, llxml, llfilesystem, lllogin, llcachestore) registered through LL_ADD_PROJECT_UNIT_TESTS / LL_ADD_INTEGRATION_TEST in indra/cmake/LLAddBuildTest.cmake, and Slipstream added newview integration tests (slgputexmigrate, llgputexrecord, slj2cmigrate, slassetmigrate; recent commits touched them, so the sources are maintained).
  • It is off everywhere: LL_TESTS defaults to OFF (indra/cmake/Variables.cmake:28), is not set by linux-releaseos / linux-release and is forced "LL_TESTS": "OFF" in windows-cross (indra/CMakePresets.json). Nothing in .forgejo/, scripts/ci/ or scripts/check/ runs ctest, and the Makefile's validate target is the testctl runtime suite, not ctest. So no CI or local build compiles or runs these tests; whether they still build is unknown and should be assumed bitrotted (the unit-test macros for llcorehttp and newview are already commented out, LL_ADD_INTEGRATION_TEST(llhttpretrypolicy ...) too).
  • It depends on tut (3p prebuilt, Tut.cmake, a 2008 header-only library) plus APR for the runner; LLAddBuildTest.cmake also builds test.cpp, lltut.cpp into every test executable (one executable per source file, so ~100 link steps when enabled).
  • The infrastructure that does work and is worth keeping: CTest registration, per-test timeouts, BUILD_TESTS / BUILD_AND_RUN_TESTS targets, LL_TEST_COMMAND.
  • There is no ASan / UBSan configuration either: USE_ASAN etc. only add -Og in 00-Common.cmake:242; the tree has no -fsanitize flags outside the new LL_FUZZ build (branch fuzzing).

Scope.

  1. Pick the framework (ticket "Test framework choice").
  2. Add a fast default test suite that is built and run as part of a normal build, on Linux local builds and CI, failing the build on a test failure. Mechanism: a CMake option SL_UNIT_TESTS (ON by default for Linux, OFF for windows-cross at first), one aggregate target sl_unit_tests, and a stamp file (unit_tests.stamp, produced by a custom command that runs the suite) that slipstream-bin depends on, so a failing test stops the build before the multi-gigabyte viewer link. make gets it for free; make test / ctest --preset linux-releaseos -L fast runs only the tests.
  3. Heavy suites (GPU, fuzz replay of large corpora, long soak) are opt-in through CTest labels; the default build only runs label fast.
  4. Windows: the same test sources are cross-compiled in make windows / CI (catches clang-cl and Windows-macro breaks at compile time, which is where most Windows regressions arrive); running them is a label wine job on the CI box using a throwaway wine prefix (no menu pollution), not part of the default Windows build.
  5. Developer docs: doc/DEVELOPER_GUIDE.md section on writing and running tests; CODING_RULES.md note that new pure-logic code ships with a suite.
  6. Sub-tickets: framework choice, first suites, GPU / render regression tests, fuzzing in CI, sanitizer CI.

Acceptance criteria.

  • make on a clean Linux checkout builds and runs the default suite; a deliberately broken assertion makes make fail before slipstream-bin links, with the failing test name and file:line in the output.
  • Default suite wall time: at most 5 s on the CI box and a 4-core laptop, measured and printed (ctest --output-on-failure plus a total); a CI check fails if the budget is exceeded by 2x.
  • CI runs the same suite (Linux job) and a Windows cross-build of the test target (compile only).
  • Incremental rebuilds: touching one .cpp under indra/slcore rebuilds and re-runs only the affected test executable (stamp per executable).
  • The old tut harness is either removed or kept behind LL_TESTS with a documented status; there is no third state.

Effort. Epic umbrella: roughly 3 to 4 weeks across the sub-tickets (framework + plumbing 3 days, first suites 1 to 2 weeks, GPU 1 week, fuzz CI 2 days, sanitizer CI 2 days).

**Why.** Slipstream has a lot of pure logic that regresses silently (weather maths, run-off, cache / segment store, inventory safety, key-binding migration, SLBridge line parsing, LLSD round trips) and the only automated gates today are `make check` (source / package lint) and the manual `make validate` runtime suite on the test box. A bad change is found by a person, in a running viewer, or after a tag. Several bug classes in this repo (Windows macro names, glibc pins, LL_ERRS reachable from input) already have a mechanical guard; logic regressions do not. **State of the inherited LL test framework (verified in the tree, 2026-10-28).** - It is still in the tree and wired: `indra/test` (the `lltest` runner: `test.cpp` main + `lltut.*`, 25 `*_tut.cpp` suites for llmessage / llcommon / llinventory), per-library `tests/*_test.cpp` files (about 40 in llcommon alone, plus llimage, llmath, llmessage, llinventory, llprimitive, llui, llxml, llfilesystem, lllogin, llcachestore) registered through `LL_ADD_PROJECT_UNIT_TESTS` / `LL_ADD_INTEGRATION_TEST` in `indra/cmake/LLAddBuildTest.cmake`, and Slipstream added newview integration tests (`slgputexmigrate`, `llgputexrecord`, `slj2cmigrate`, `slassetmigrate`; recent commits touched them, so the sources are maintained). - It is **off everywhere**: `LL_TESTS` defaults to OFF (`indra/cmake/Variables.cmake:28`), is not set by `linux-releaseos` / `linux-release` and is forced `"LL_TESTS": "OFF"` in `windows-cross` (`indra/CMakePresets.json`). Nothing in `.forgejo/`, `scripts/ci/` or `scripts/check/` runs `ctest`, and the Makefile's `validate` target is the testctl runtime suite, not ctest. So no CI or local build compiles or runs these tests; whether they still build is **unknown** and should be assumed bitrotted (the unit-test macros for llcorehttp and newview are already commented out, `LL_ADD_INTEGRATION_TEST(llhttpretrypolicy ...)` too). - It depends on `tut` (3p prebuilt, `Tut.cmake`, a 2008 header-only library) plus APR for the runner; `LLAddBuildTest.cmake` also builds `test.cpp`, `lltut.cpp` into every test executable (one executable per source file, so ~100 link steps when enabled). - The infrastructure that does work and is worth keeping: CTest registration, per-test timeouts, `BUILD_TESTS` / `BUILD_AND_RUN_TESTS` targets, `LL_TEST_COMMAND`. - There is no ASan / UBSan configuration either: `USE_ASAN` etc. only add `-Og` in `00-Common.cmake:242`; the tree has no `-fsanitize` flags outside the new `LL_FUZZ` build (branch `fuzzing`). **Scope.** 1. Pick the framework (ticket "Test framework choice"). 2. Add a fast default test suite that is built and run as part of a normal build, on Linux local builds and CI, failing the build on a test failure. Mechanism: a CMake option `SL_UNIT_TESTS` (ON by default for Linux, OFF for `windows-cross` at first), one aggregate target `sl_unit_tests`, and a stamp file (`unit_tests.stamp`, produced by a custom command that runs the suite) that `slipstream-bin` depends on, so a failing test stops the build **before** the multi-gigabyte viewer link. `make` gets it for free; `make test` / `ctest --preset linux-releaseos -L fast` runs only the tests. 3. Heavy suites (GPU, fuzz replay of large corpora, long soak) are opt-in through CTest labels; the default build only runs label `fast`. 4. Windows: the same test sources are cross-compiled in `make windows` / CI (catches clang-cl and Windows-macro breaks at compile time, which is where most Windows regressions arrive); running them is a label `wine` job on the CI box using a throwaway wine prefix (no menu pollution), not part of the default Windows build. 5. Developer docs: `doc/DEVELOPER_GUIDE.md` section on writing and running tests; CODING_RULES.md note that new pure-logic code ships with a suite. 6. Sub-tickets: framework choice, first suites, GPU / render regression tests, fuzzing in CI, sanitizer CI. **Acceptance criteria.** - `make` on a clean Linux checkout builds and runs the default suite; a deliberately broken assertion makes `make` fail before `slipstream-bin` links, with the failing test name and file:line in the output. - Default suite wall time: **at most 5 s** on the CI box and a 4-core laptop, measured and printed (`ctest --output-on-failure` plus a total); a CI check fails if the budget is exceeded by 2x. - CI runs the same suite (Linux job) and a Windows cross-build of the test target (compile only). - Incremental rebuilds: touching one `.cpp` under `indra/slcore` rebuilds and re-runs only the affected test executable (stamp per executable). - The old tut harness is either removed or kept behind `LL_TESTS` with a documented status; there is no third state. **Effort.** Epic umbrella: roughly 3 to 4 weeks across the sub-tickets (framework + plumbing 3 days, first suites 1 to 2 weeks, GPU 1 week, fuzz CI 2 days, sanitizer CI 2 days).
sky_line added this to the Slipstream project 2026-10-28 20:33:40 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sky_line/Slipstream#1243
No description provided.