Cache store: migration from the current caches + inspect/dump tool

sky_line commented

2026-10-12 11:17:23 +00:00

Owner

Part of #715 (cache rewrite epic).

Convert existing J2C / GPU-ready / asset caches into the segment store in the background (or rebuild if conversion is not worth it -- measure), never blocking login.
Remove the old files once converted; respect the disk-full guard during conversion.
A small tool (and/or Develop menu entry) to list, inspect and extract records, and print store stats (segments, live %, bytes written per day) -- needed for debugging once files are no longer one per item.

Part of #715 (cache rewrite epic). - Convert existing J2C / GPU-ready / asset caches into the segment store in the background (or rebuild if conversion is not worth it -- measure), never blocking login. - Remove the old files once converted; respect the disk-full guard during conversion. - A small tool (and/or Develop menu entry) to list, inspect and extract records, and print store stats (segments, live %, bytes written per day) -- needed for debugging once files are no longer one per item.

sky_line added the

triage

feature

labels

2026-10-12 11:17:23 +00:00

sky_line added this to the Slipstream project

2026-10-12 11:17:23 +00:00

sky_line referenced this issue

2026-10-12 11:17:23 +00:00

[Epic] Cache rewrite: segment store, compression, tiered texture caching, disk-aware sizing #715

sky_line commented

2026-10-18 22:52:44 +00:00

Author

Owner

Interface audit (step 1 of #715): what migration reads from each old cache, and the tool's hooks.

Sources (counts from #708 / #705)

Old cache	Files	How to convert
J2C (`texturecache/`, 42.9k textures, 11.0 GB)	`texture.entries` + `texture.cache` + `[0-f]/<uuid>.texture`	For each live entry (lltexturecache.h:68-83): bytes = `texture.cache[idx*600 .. +min(600, size)]` + body file. Meta = `mImageSize`; access time = `mTime` -> `prev_session_access`. Skip entries whose body size disagrees (the old validation, lltexturecache.cpp:1960).
GPU-ready (`gputex/`, 29.6k files, 8.2 GB)	`<uuid>_<w>x<h>_e_l_v[_mask].gtx`	`parse_name` (llgputexcache.cpp:137) gives the key + meta. Verify like `run_read` (:252). Rebuildable, so delete it instead if import is not worth it (measure).
Assets (`cache/`, 265k files)	`sl_cache_<uuid>_0.asset`	Stream each file in as ASSET (type unknown: today's name has none; phase 1 keys by UUID anyway). Access time = file mtime.
Meshes (inside `cache/`)	same names	Telling a mesh apart needs the 12-byte preamble (version 1, header size, flags; llmeshrepository.cpp:353-354, :936). Split into MESH parts by the header's offsets and the `InCache` flags. Anything unparseable is dropped (refetchable).
Cluster LOD (`clusterlod/`), if it joins	`.clod`	Like gputex.

Sounds (.dsf) and the object cache stay files and need no migration (the object cache reads its old files, #713).

Slice of the interface

Importer = Store::beginStaged(kind) -> append -> commit(key, meta, flags) at LOW priority, plus a meta field for the imported access time.
Respect setDiskReserve (#711): import one old file, then delete it, so peak extra disk use is one segment.
Order by old access time, newest first, so the hot set is usable first. Lookups that miss the store during migration fall back to the old file through the adapter, so login is never blocked.
Resumable: a marker per old cache records progress, and a crash mid-import only re-imports the tail (the store drops a half-written record).
Tool (Develop menu + a CLI slcachetool): Store::stats() (segments, live %, bytes written per day), forEach(kind) to list, verifyAll(), extract a record by key to a file, and a hex/meta dump of one record.

Drop-in plan

One importer per store consumer, landing with that consumer: GPU-ready first (or plain delete), J2C with the J2C adapter, assets + mesh split with the asset adapter. The tool lands with the store core because #708's tests and debugging need it.

**Interface audit (step 1 of #715): what migration reads from each old cache, and the tool's hooks.** ### Sources (counts from #708 / #705) | Old cache | Files | How to convert | |---|---|---| | J2C (`texturecache/`, 42.9k textures, 11.0 GB) | `texture.entries` + `texture.cache` + `[0-f]/<uuid>.texture` | For each live entry (lltexturecache.h:68-83): bytes = `texture.cache[idx*600 .. +min(600, size)]` + body file. Meta = `mImageSize`; access time = `mTime` -> `prev_session_access`. Skip entries whose body size disagrees (the old validation, lltexturecache.cpp:1960). | | GPU-ready (`gputex/`, 29.6k files, 8.2 GB) | `<uuid>_<w>x<h>_e_l_v[_mask].gtx` | `parse_name` (llgputexcache.cpp:137) gives the key + meta. Verify like `run_read` (:252). Rebuildable, so delete it instead if import is not worth it (measure). | | Assets (`cache/`, 265k files) | `sl_cache_<uuid>_0.asset` | Stream each file in as ASSET (type unknown: today's name has none; phase 1 keys by UUID anyway). Access time = file mtime. | | Meshes (inside `cache/`) | same names | Telling a mesh apart needs the 12-byte preamble (version 1, header size, flags; llmeshrepository.cpp:353-354, :936). Split into MESH parts by the header's offsets and the `InCache` flags. Anything unparseable is dropped (refetchable). | | Cluster LOD (`clusterlod/`), if it joins | `.clod` | Like gputex. | Sounds (`.dsf`) and the object cache stay files and need no migration (the object cache reads its old files, #713). ### Slice of the interface - Importer = `Store::beginStaged(kind)` -> `append` -> `commit(key, meta, flags)` at LOW priority, plus a meta field for the imported access time. - Respect `setDiskReserve` (#711): import one old file, then delete it, so peak extra disk use is one segment. - Order by old access time, newest first, so the hot set is usable first. Lookups that miss the store during migration fall back to the old file through the adapter, so login is never blocked. - Resumable: a marker per old cache records progress, and a crash mid-import only re-imports the tail (the store drops a half-written record). - Tool (Develop menu + a CLI `slcachetool`): `Store::stats()` (segments, live %, bytes written per day), `forEach(kind)` to list, `verifyAll()`, extract a record by key to a file, and a hex/meta dump of one record. ### Drop-in plan One importer per store consumer, landing with that consumer: GPU-ready first (or plain delete), J2C with the J2C adapter, assets + mesh split with the asset adapter. The tool lands with the store core because #708's tests and debugging need it.

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

llcachestore: no re-entrant budget pass; compaction may use the reserve (#708, #709, #711)

sky_line referenced this issue from a commit

2026-10-19 11:20:27 +00:00

#715 #714: open the viewer's cache store; GPU-ready textures on it; .gtx migration

sky_line referenced this issue from a commit

2026-10-19 11:20:27 +00:00

#714: Develop > Cache > Cache Store entries (stats, verify, list, extract)

sky_line referenced this issue from a commit

2026-10-19 11:20:27 +00:00

#714: migration follows "finer version wins" for keys the store already has