Cache store: append-only segment files (replaces file-per-item for textures and assets)

sky_line commented

2026-10-12 11:17:22 +00:00

Owner

Part of #715 (cache rewrite epic).

Why: every cache is one file per item today (J2C bodies 42.9k files, GPU-ready 29.6k, assets 265k in one folder). Costs: a seek per item on hard disks, slow file opens on Windows (antivirus scans each open), a 4 KB minimum per file (55% of GPU-ready files are <= 16 KB), slow startup scans and purges.

Lessons from LL's old single-file VFS (removed 2020): any unclean exit wiped the whole cache; one mutex serialized all disk I/O; a pre-sized file with its own 1 KB block allocator (fragmentation, corruption handling by deletion); a second viewer instance hit sharing violations; space was never given back. The design below keeps the filesystem's robustness without the per-file costs.

Design

Segments: append-only files of 128-256 MB per store. Self-describing records: header (UUID, kind, size, checksum) + payload. Records are never modified in place.
Index: in memory (UUID -> segment, offset, size, access time, reuse count), saved as periodic snapshots; rebuilt by scanning record headers if a snapshot is missing or stale.
Crash safety: a crash loses at most the unfinished tail of the segment being written (truncated at startup). Checksums reject bad records individually. Never wipe the whole store.
Concurrency: finished segments are immutable; any thread reads with pread (not mmap -- a truncated mapped file is a SIGBUS). One writer thread appends. No global lock.
Eviction: marks records dead; space comes back by deleting whole segments after compaction (policy in the SSD wear ticket).
Multiple viewers: a lock file per store. A second instance opens it read-only and writes to a small private overflow.
Scope: J2C textures, GPU-ready textures and assets in one store under one budget. The object cache (one file per region) and the decoded sound hot cache stay as they are.
Tests: crash at every write step, corrupted / truncated records, disk full, two instances.

Part of #715 (cache rewrite epic). **Why:** every cache is one file per item today (J2C bodies 42.9k files, GPU-ready 29.6k, assets 265k in one folder). Costs: a seek per item on hard disks, slow file opens on Windows (antivirus scans each open), a 4 KB minimum per file (55% of GPU-ready files are <= 16 KB), slow startup scans and purges. **Lessons from LL's old single-file VFS (removed 2020):** any unclean exit wiped the whole cache; one mutex serialized all disk I/O; a pre-sized file with its own 1 KB block allocator (fragmentation, corruption handling by deletion); a second viewer instance hit sharing violations; space was never given back. The design below keeps the filesystem's robustness without the per-file costs. **Design** - **Segments:** append-only files of 128-256 MB per store. Self-describing records: header (UUID, kind, size, checksum) + payload. Records are never modified in place. - **Index:** in memory (UUID -> segment, offset, size, access time, reuse count), saved as periodic snapshots; rebuilt by scanning record headers if a snapshot is missing or stale. - **Crash safety:** a crash loses at most the unfinished tail of the segment being written (truncated at startup). Checksums reject bad records individually. Never wipe the whole store. - **Concurrency:** finished segments are immutable; any thread reads with `pread` (not mmap -- a truncated mapped file is a SIGBUS). One writer thread appends. No global lock. - **Eviction:** marks records dead; space comes back by deleting whole segments after compaction (policy in the SSD wear ticket). - **Multiple viewers:** a lock file per store. A second instance opens it read-only and writes to a small private overflow. - **Scope:** J2C textures, GPU-ready textures and assets in one store under one budget. The object cache (one file per region) and the decoded sound hot cache stay as they are. - **Tests:** crash at every write step, corrupted / truncated records, disk full, two instances.

sky_line added the

triage

perf

feature

labels

2026-10-12 11:17:22 +00:00

sky_line added this to the Slipstream project

2026-10-12 11:17:22 +00:00

sky_line referenced this issue

2026-10-12 11:17:23 +00:00

[Epic] Cache rewrite: segment store, compression, tiered texture caching, disk-aware sizing #715

sky_line referenced this issue

2026-10-18 22:52:43 +00:00

[Epic] Cache rewrite: segment store, compression, tiered texture caching, disk-aware sizing #715

sky_line commented

2026-10-18 22:52:43 +00:00

Author

Owner

Interface audit (step 1 of #715): what the segment store replaces and how it drops in.

Replaces

Code	What goes
llfilesystem/lldiskcache.cpp:92 `purge()`, :235 `clearCache()`, :351 `LLPurgeDiskCacheThread` (every 15 min, stats every file)	removed; the store evicts on write
llfilesystem/llfilesystem.cpp (whole class)	becomes an adapter over the store
lldiskcache.cpp:215 `metaDataToFilepath` (`sl_cache_<uuid>_0.asset`, type ignored)	key (uuid, kind, variant)
newview/lltexturecache.cpp `doRead` :282 / `doWrite` :546, `texture.entries` + `texture.cache` + body files, `purgeTextures*` :1786/:1876, `purgeAllTextures` :1736	J2C records
newview/llgputexcache.cpp `run_scan/run_read/run_write` :221-300, `evict` :182	GPUTEX records
(optional, later) newview/llcleanclustercache.cpp, same pattern	CLUSTER records

These stay files: llaudio/llsoundcache.cpp (the audio back end loads WAVs by path) and newview/llvocache.cpp (gets only the codec, #713).

Store core (this ticket's slice)

<cache>/store/: lock, index.snap (+ .prev), seg-<seq>.dat (128-256 MB, append-only).
Record = 64-byte header {magic, version, kind, codec, flags (tombstone), UUID + variant, kind_version, meta_len, raw_bytes, stored_bytes, time, seq, XXH64 payload, XXH64 header} + up to 32 B meta + payload. A record counts only if both hashes match.
Index: Key -> (segment, offset, sizes, codec, kind_version, last access, previous-session access, reuse count, meta). 64 shards, one small mutex each; lookup() never does I/O. Snapshot saved periodically and at exit; at startup load the snapshot, then scan only the segment tails written after it.
One writer thread. Reads pread finished segments from any thread. Segments are refcounted so compaction deletes only after the last reader lets go (share-delete on Windows).
Read-your-writes: a record still in the writer queue is served from the queue. This is needed because the asset system delivers assets through the cache (fetch writes, callback reads back, llviewerassetstorage.cpp:556).
Per-kind format_version replaces the whole-cache wipes (LocalCacheVersion, DiskCacheVersion, the J2C encoder string, gputex FILE_VERSION).
Lock file. A second instance opens read-only and writes to a small private overflow. The overflow is required, not optional: assets are delivered through the cache, so a purely read-only second instance could not fetch any asset. Its asset / mesh writes go to the overflow; its reads check the overflow first, then the main store.
Clear Cache frees disk at once: a segment at 0% live is deleted immediately whatever the compaction cap, and a user Clear Cache deletes the segments outright. <cache>/store joins the folders deleted on a cache-location change (llappviewer.cpp:9105-9115). Today the second instance is read-only for textures, gputex, sounds, cluster and objects, but not for assets: it writes freely and runs the 15-minute purge over the shared folder (purge thread started outside the read-only check, llappviewer.cpp:9156).

API sketch: lookup / touch / read / readRange / readSync / readRangeSync / write(WriteSource, meta, flags) / beginStaged -> append, commit(final_key) | abort / remove / removeAll(kind) / stats / forEach. Full sketch in the #715 comment.

Drop-in plan

Library indra/llcachestore + tests: crash at each write step, corrupt / truncated records, disk full (fault injection), two instances.
GPU-ready on the store behind LLGpuTexCache's current static API (renderer untouched).
LLTextureCache adapter. One record per texture = the whole J2C prefix (the 600-byte split goes). readRangeSync(offset, size) covers the fetcher's offset reads (lltexturefetch.cpp:1256). Writes use SUPERSEDE_PENDING + SKIP_IF_NOT_LARGER.
LLFileSystem adapter. READ = pread at record offset + position (seek is free, so the Ogg decoder's ov_callbacks keep working). WRITE/APPEND = a per-UUID staging table that outlives the LLFileSystem objects: the UDP transfer and xfer code opens a new APPEND object per packet (lltransfertargetvfile.cpp:141, llxfer_vfile.cpp:263/317) and checks getExists on a half-received temp (llxfer_vfile.cpp:82/337), so READ / exists / size answer from staged bytes; rename(new_id) commits one record (the download temp-UUID flow), WRITE restarts, remove aborts, session end commits. PIN_SESSION so a fresh asset is not evicted before its callback reads it. Phase 1 keeps UUID-only keys with the type in meta. Today the type is ignored, and llfloaterauction.cpp:206/216 relies on that by accident.
Mesh is not a drop-in. READ_WRITE + setSize sparse file + preamble flags (llmeshrepository.cpp:4336-4365, 4414+) move to per-part records (header, LOD0-3, skin, physics convex, physics mesh) in phase 1. About 14 sites in one file; the m*InCache flags become "record exists".

**Interface audit (step 1 of #715): what the segment store replaces and how it drops in.** ### Replaces | Code | What goes | |---|---| | llfilesystem/lldiskcache.cpp:92 `purge()`, :235 `clearCache()`, :351 `LLPurgeDiskCacheThread` (every 15 min, stats every file) | removed; the store evicts on write | | llfilesystem/llfilesystem.cpp (whole class) | becomes an adapter over the store | | lldiskcache.cpp:215 `metaDataToFilepath` (`sl_cache_<uuid>_0.asset`, type ignored) | key (uuid, kind, variant) | | newview/lltexturecache.cpp `doRead` :282 / `doWrite` :546, `texture.entries` + `texture.cache` + body files, `purgeTextures*` :1786/:1876, `purgeAllTextures` :1736 | J2C records | | newview/llgputexcache.cpp `run_scan/run_read/run_write` :221-300, `evict` :182 | GPUTEX records | | (optional, later) newview/llcleanclustercache.cpp, same pattern | CLUSTER records | These stay files: llaudio/llsoundcache.cpp (the audio back end loads WAVs by path) and newview/llvocache.cpp (gets only the codec, #713). ### Store core (this ticket's slice) - `<cache>/store/`: `lock`, `index.snap` (+ `.prev`), `seg-<seq>.dat` (128-256 MB, append-only). - Record = 64-byte header {magic, version, kind, codec, flags (tombstone), UUID + variant, kind_version, meta_len, raw_bytes, stored_bytes, time, seq, XXH64 payload, XXH64 header} + up to 32 B meta + payload. A record counts only if both hashes match. - Index: Key -> (segment, offset, sizes, codec, kind_version, last access, previous-session access, reuse count, meta). 64 shards, one small mutex each; `lookup()` never does I/O. Snapshot saved periodically and at exit; at startup load the snapshot, then scan only the segment tails written after it. - One writer thread. Reads `pread` finished segments from any thread. Segments are refcounted so compaction deletes only after the last reader lets go (share-delete on Windows). - Read-your-writes: a record still in the writer queue is served from the queue. This is needed because the asset system delivers assets through the cache (fetch writes, callback reads back, llviewerassetstorage.cpp:556). - Per-kind `format_version` replaces the whole-cache wipes (`LocalCacheVersion`, `DiskCacheVersion`, the J2C encoder string, gputex `FILE_VERSION`). - Lock file. A second instance opens read-only and writes to a small private overflow. The overflow is required, not optional: assets are delivered through the cache, so a purely read-only second instance could not fetch any asset. Its asset / mesh writes go to the overflow; its reads check the overflow first, then the main store. - Clear Cache frees disk at once: a segment at 0% live is deleted immediately whatever the compaction cap, and a user Clear Cache deletes the segments outright. `<cache>/store` joins the folders deleted on a cache-location change (llappviewer.cpp:9105-9115). Today the second instance is read-only for textures, gputex, sounds, cluster and objects, but **not for assets**: it writes freely and runs the 15-minute purge over the shared folder (purge thread started outside the read-only check, llappviewer.cpp:9156). API sketch: `lookup / touch / read / readRange / readSync / readRangeSync / write(WriteSource, meta, flags) / beginStaged -> append, commit(final_key) | abort / remove / removeAll(kind) / stats / forEach`. Full sketch in the #715 comment. ### Drop-in plan 1. Library `indra/llcachestore` + tests: crash at each write step, corrupt / truncated records, disk full (fault injection), two instances. 2. GPU-ready on the store behind `LLGpuTexCache`'s current static API (renderer untouched). 3. `LLTextureCache` adapter. One record per texture = the whole J2C prefix (the 600-byte split goes). `readRangeSync(offset, size)` covers the fetcher's offset reads (lltexturefetch.cpp:1256). Writes use SUPERSEDE_PENDING + SKIP_IF_NOT_LARGER. 4. `LLFileSystem` adapter. READ = pread at record offset + position (seek is free, so the Ogg decoder's `ov_callbacks` keep working). WRITE/APPEND = a per-UUID staging table that outlives the `LLFileSystem` objects: the UDP transfer and xfer code opens a new APPEND object per packet (lltransfertargetvfile.cpp:141, llxfer_vfile.cpp:263/317) and checks `getExists` on a half-received temp (llxfer_vfile.cpp:82/337), so READ / exists / size answer from staged bytes; `rename(new_id)` commits one record (the download temp-UUID flow), WRITE restarts, `remove` aborts, session end commits. PIN_SESSION so a fresh asset is not evicted before its callback reads it. Phase 1 keeps UUID-only keys with the type in meta. Today the type is ignored, and llfloaterauction.cpp:206/216 relies on that by accident. 5. Mesh is **not** a drop-in. `READ_WRITE` + `setSize` sparse file + preamble flags (llmeshrepository.cpp:4336-4365, 4414+) move to per-part records (header, LOD0-3, skin, physics convex, physics mesh) in phase 1. About 14 sites in one file; the `m*InCache` flags become "record exists".

sky_line referenced this issue

2026-10-18 22:52:43 +00:00

Cache store: zstd compression for GPU-ready textures #710

sky_line referenced this issue

2026-10-18 22:52:43 +00:00

Texture disk tiers: one budget, GPU-ready cache as a reuse-admitted hot tier, caching strategy option, disk-space-based default sizes #705

sky_line referenced this issue

2026-10-18 22:52:43 +00:00

Cache sizes: defaults from free disk space, explicit sizes kept, disk-full guard #711

sky_line referenced this issue

2026-10-18 22:52:44 +00:00

Object cache: zstd compression #713

sky_line referenced this issue

2026-10-18 22:52:44 +00:00

Cache store: migration from the current caches + inspect/dump tool #714

sky_line referenced this issue

2026-10-18 23:31:23 +00:00

[Epic] Cache rewrite: segment store, compression, tiered texture caching, disk-aware sizing #715

sky_line commented

2026-10-18 23:31:23 +00:00

Author

Owner

Placement design (decided 2026-10-18, see #715)

1. Placement hint on every write. It is computed when the bytes arrive, from what is using the asset at that moment:

Requested by	Hint
Textures / meshes on in-world objects all in one region (not attachments)	that region (region UUID; the grid handle is kept for lookup)
Avatars: bakes, attachments' textures / meshes, animations, attachment sounds	`SHARED`
Inventory, UI images, previews, uploads	`SHARED`
Objects in several regions	`SHARED`

A wrong hint only costs locality.

2. Per-region runs, still one writer and one append stream. The writer buffers records per hint (up to ~4 MB or ~2 s per region) and appends each batch as one contiguous run. Segments hold runs from many regions, so there are no extra open files and no half-empty per-region files for regions you only passed through. A crash loses only the unflushed buffer, which is re-downloaded.

segment 0041: [region A 2.1 MB][shared 0.8 MB][region B 3.4 MB][region A 1.2 MB] ...

3. Region manifest (its own record kind, loaded on demand, ~70 KB for a 3,000-texture region):

the region UUID and its grid handle;
last visit and visit count;
the region's own runs (segment, offset, length);
references to the shared records it used (pointers, not copies);
rough first-request order, used for prefetch priority.

Manifests are updated on each visit.

4. Promotion without copying. A record in region A's run that is later used from region B is flagged shared in the index. It stays where it is and is no longer tied to A's lifetime.

Tests to add: a hint that disagrees with later use (it must still be served correctly), losing the buffer in a crash, a manifest that references dead records, and a region UUID whose grid handle changes.

## Placement design (decided 2026-10-18, see #715) **1. Placement hint on every write.** It is computed when the bytes arrive, from what is using the asset at that moment: | Requested by | Hint | |---|---| | Textures / meshes on in-world objects all in one region (not attachments) | that region (region UUID; the grid handle is kept for lookup) | | Avatars: bakes, attachments' textures / meshes, animations, attachment sounds | `SHARED` | | Inventory, UI images, previews, uploads | `SHARED` | | Objects in several regions | `SHARED` | A wrong hint only costs locality. **2. Per-region runs, still one writer and one append stream.** The writer buffers records per hint (up to ~4 MB or ~2 s per region) and appends each batch as one contiguous **run**. Segments hold runs from many regions, so there are no extra open files and no half-empty per-region files for regions you only passed through. A crash loses only the unflushed buffer, which is re-downloaded. ``` segment 0041: [region A 2.1 MB][shared 0.8 MB][region B 3.4 MB][region A 1.2 MB] ... ``` **3. Region manifest** (its own record kind, loaded on demand, ~70 KB for a 3,000-texture region): - the region UUID and its grid handle; - last visit and visit count; - the region's own runs (segment, offset, length); - references to the shared records it used (pointers, not copies); - rough first-request order, used for prefetch priority. Manifests are updated on each visit. **4. Promotion without copying.** A record in region A's run that is later used from region B is flagged shared in the index. It stays where it is and is no longer tied to A's lifetime. **Tests to add:** a hint that disagrees with later use (it must still be served correctly), losing the buffer in a crash, a manifest that references dead records, and a region UUID whose grid handle changes.

sky_line referenced this issue

2026-10-18 23:31:24 +00:00

Cache store: hard-disk locality -- region-ordered writes and hot-set rewrites #712

sky_line referenced this issue

2026-10-18 23:31:24 +00:00

Texture disk tiers: one budget, GPU-ready cache as a reuse-admitted hot tier, caching strategy option, disk-space-based default sizes #705

sky_line referenced this issue

2026-10-19 00:25:41 +00:00

[Epic] Cache rewrite: segment store, compression, tiered texture caching, disk-aware sizing #715

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

Asset disk cache: no periodic purge in a second viewer instance (#715, #708)