Test framework choice: revive tut, or doctest / Catch2 / GoogleTest #1244

Open
opened 2026-10-28 20:33:40 +00:00 by sky_line · 0 comments
Owner

Part of #1243.

Decision to make. Which framework runs the new unit tests; what happens to the existing tut tests.

Recommendation: doctest (single vendored header), with the tut tests left under LL_TESTS until individually ported or deleted.

Criterion tut (LL) doctest Catch2 v3 GoogleTest
Build speed fine (header only) but needs test.cpp/lltut.cpp/APR in every executable fastest of the modern ones; designed for low compile overhead; DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN in one TU slower; v3 is no longer single-header, needs building a library slow; builds gtest + gmock libs, heavier headers
Distribution 3p prebuilt, 2008, unmaintained one MIT header, vendor it (no 3p manifest entry, no download at configure) needs a 3p package or FetchContent needs a 3p package or FetchContent
CTest LL macros only doctest_discover_tests, or one ctest entry per executable plus --list-test-cases; JUnit / xml reporters catch_discover_tests gtest_discover_tests
IDE support none CLion, VS Code (test adapter), VS all best
clang-cl / GCC works with LL's own workarounds works; no exceptions-only design (usable with -fno-exceptions if ever needed) works works
Fit for "a few seconds, run on every build" the runner design (one exe per source) is the slow part one small executable per module, tests register at static-init OK OK but heavier
Fixtures / mocking tut object model, awkward subcases and plain functions; no mock framework (we do not want one: tests should use real pure functions) sections, generators fixtures + gmock (not needed)

Reasons in short: the goal is a gate that adds seconds, not minutes, to every build, so compile and link cost per test matters more than feature count; a single vendored header removes a network dependency from configure (the 3p cache already is a pain point); doctest's TEST_CASE / SUBCASE style is enough for table-driven maths and parser tests; and it can be dropped into any module without the APR-based runner. GoogleTest is the right answer only if we later want death tests or gmock; Catch2 v3 has no advantage over doctest here and costs compile time. Reviving tut means keeping a 2008 library and a runner that is wired to APR and LL singletons, with no IDE story.

Existing tut tests. Do not mass-port. Keep LL_TESTS (tut) buildable as an opt-in -DLL_TESTS=ON build, fix it once so it compiles (spike below), and port a suite to doctest only when it covers something worth running every build (the llcommon llsdserialize, llstring, lluri, llkeybind tests are the likely ones; the 25 indra/test/*_tut.cpp llmessage / http-node suites mostly are not). Anything that stays red and unowned after the spike is deleted, not left.

Scope.

  1. Spike (1 day): configure and build the existing tut suites with -DLL_TESTS=ON on Linux, record which build and which pass; write the result in the ticket. Fix or delete accordingly.
  2. Vendor doctest.h under indra/slcore/test/third_party/doctest/ with its LICENSE; add indra/cmake/SlTest.cmake with sl_add_unit_test(<name> SOURCES ... LIBS ... LABELS fast) (one executable per module, add_test, labels, timeout 10 s per test by default).
  3. Convert two existing tests as the worked example (suggest llstring and lluri or llsdserialize), delete their tut copies.
  4. A timing report: the runner prints per-executable wall time; a CTest fixture fails the fast label if the sum exceeds the budget.

Acceptance criteria. The decision and the table are recorded in doc/DEVELOPER_GUIDE.md; sl_add_unit_test works under GCC (Linux) and clang-cl (the windows-cross preset compiles it); two ported suites pass in under 1 s each; the spike's findings on the tut build are written down (builds / does not build, what was fixed or removed).

Effort. 3 to 4 days including the spike.

Part of #1243. **Decision to make.** Which framework runs the new unit tests; what happens to the existing tut tests. **Recommendation: doctest (single vendored header), with the tut tests left under `LL_TESTS` until individually ported or deleted.** | Criterion | tut (LL) | doctest | Catch2 v3 | GoogleTest | | --- | --- | --- | --- | --- | | Build speed | fine (header only) but needs `test.cpp`/`lltut.cpp`/APR in every executable | fastest of the modern ones; designed for low compile overhead; `DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN` in one TU | slower; v3 is no longer single-header, needs building a library | slow; builds `gtest` + `gmock` libs, heavier headers | | Distribution | 3p prebuilt, 2008, unmaintained | one MIT header, vendor it (no 3p manifest entry, no download at configure) | needs a 3p package or FetchContent | needs a 3p package or FetchContent | | CTest | LL macros only | `doctest_discover_tests`, or one ctest entry per executable plus `--list-test-cases`; JUnit / xml reporters | `catch_discover_tests` | `gtest_discover_tests` | | IDE support | none | CLion, VS Code (test adapter), VS | all | best | | clang-cl / GCC | works with LL's own workarounds | works; no exceptions-only design (usable with `-fno-exceptions` if ever needed) | works | works | | Fit for "a few seconds, run on every build" | the runner design (one exe per source) is the slow part | one small executable per module, tests register at static-init | OK | OK but heavier | | Fixtures / mocking | tut object model, awkward | subcases and plain functions; no mock framework (we do not want one: tests should use real pure functions) | sections, generators | fixtures + gmock (not needed) | Reasons in short: the goal is a gate that adds seconds, not minutes, to every build, so compile and link cost per test matters more than feature count; a single vendored header removes a network dependency from configure (the 3p cache already is a pain point); doctest's `TEST_CASE` / `SUBCASE` style is enough for table-driven maths and parser tests; and it can be dropped into any module without the APR-based runner. GoogleTest is the right answer only if we later want death tests or gmock; Catch2 v3 has no advantage over doctest here and costs compile time. Reviving tut means keeping a 2008 library and a runner that is wired to APR and LL singletons, with no IDE story. **Existing tut tests.** Do not mass-port. Keep `LL_TESTS` (tut) buildable as an opt-in `-DLL_TESTS=ON` build, fix it once so it compiles (spike below), and port a suite to doctest only when it covers something worth running every build (the llcommon `llsdserialize`, `llstring`, `lluri`, `llkeybind` tests are the likely ones; the 25 `indra/test/*_tut.cpp` llmessage / http-node suites mostly are not). Anything that stays red and unowned after the spike is deleted, not left. **Scope.** 1. Spike (1 day): configure and build the existing tut suites with `-DLL_TESTS=ON` on Linux, record which build and which pass; write the result in the ticket. Fix or delete accordingly. 2. Vendor `doctest.h` under `indra/slcore/test/third_party/doctest/` with its LICENSE; add `indra/cmake/SlTest.cmake` with `sl_add_unit_test(<name> SOURCES ... LIBS ... LABELS fast)` (one executable per module, `add_test`, labels, timeout 10 s per test by default). 3. Convert two existing tests as the worked example (suggest `llstring` and `lluri` or `llsdserialize`), delete their tut copies. 4. A timing report: the runner prints per-executable wall time; a CTest fixture fails the `fast` label if the sum exceeds the budget. **Acceptance criteria.** The decision and the table are recorded in `doc/DEVELOPER_GUIDE.md`; `sl_add_unit_test` works under GCC (Linux) and clang-cl (the `windows-cross` preset compiles it); two ported suites pass in under 1 s each; the spike's findings on the tut build are written down (builds / does not build, what was fixed or removed). **Effort.** 3 to 4 days including the spike.
sky_line added this to the Slipstream project 2026-10-28 20:33:40 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sky_line/Slipstream#1244
No description provided.