sky_line/Slipstream

Fork 1

Cache store: hard-disk locality -- region-ordered writes and hot-set rewrites #712

New issue

Open

opened 2026-10-12 11:17:23 +00:00 by sky_line · 2 comments

sky_line commented

2026-10-12 11:17:23 +00:00

Owner

Part of #715 (cache rewrite epic).

On hard disks a seek (7-12 ms) costs far more than reading the data, so layout beats compression.

Writes go in arrival order, which is roughly per region: a revisit reads a few segments sequentially instead of thousands of scattered files.
On HDD only (drive type from the SSD wear ticket): on a revisit, gather the region's hot set into the current segment when it is spread over many segments, within the daily compaction cap.
Measure: region-arrival texture/asset load time on an HDD before and after.

Part of #715 (cache rewrite epic). On hard disks a seek (7-12 ms) costs far more than reading the data, so layout beats compression. - Writes go in arrival order, which is roughly per region: a revisit reads a few segments sequentially instead of thousands of scattered files. - On HDD only (drive type from the SSD wear ticket): on a revisit, gather the region's hot set into the current segment when it is spread over many segments, within the daily compaction cap. - Measure: region-arrival texture/asset load time on an HDD before and after.

sky_line added the

triage

perf

feature

labels

2026-10-12 11:17:23 +00:00

sky_line added this to the Slipstream project

2026-10-12 11:17:23 +00:00

sky_line referenced this issue

2026-10-12 11:17:23 +00:00

[Epic] Cache rewrite: segment store, compression, tiered texture caching, disk-aware sizing #715

sky_line commented

2026-10-18 22:52:43 +00:00

Author

Owner

Interface audit (step 1 of #715): what the store gives hard disks and this ticket's hooks.

Today

Every cached item is its own file: J2C bodies under texturecache/[0-f]/ (16 folders, lltexturecache.cpp:1121-1126) plus a 600-byte stripe per texture in one shared texture.cache read at idx * 600 (lltexturecache.cpp:405), GPU-ready files in gputex/, and 265k assets in one flat cache/ folder (lldiskcache.cpp:215). A region revisit reads thousands of scattered files, each with a seek plus a directory lookup. The J2C stripe file's slots are reused through a free list, so neighbouring slots do not mean neighbouring textures.

Slice of the interface

Arrival order = locality. One writer thread appends to the active segment in queue order, so textures, assets and meshes that arrive together while loading a region land next to each other. The fetchers carry no region id. Arrival order is all the store gets, and that is enough for this ticket as written. A Priority on writes changes order only within the queue's short window.
WearPolicy.locality_rewrites (HDD only, drive type from #709) + the daily compaction cap: on a revisit, when the records read in a short window are spread over many segments, the store re-appends that hot set to the current segment. It reuses the compaction path (copy live record, kill old) and counts against the same cap.
Reads: pread in key order within a segment. The batch read for a hot set sorts by (segment, offset) before issuing.
Stats: per-read latency by drive type; "records gathered" and gathered bytes per day.

Drop-in plan

Policy on top of the store, after #709's drive detection. Measure the region-arrival load time on an HDD before and after: first with arrival-order writes only (comes free with the store), then with hot-set rewrites.

**Interface audit (step 1 of #715): what the store gives hard disks and this ticket's hooks.** ### Today Every cached item is its own file: J2C bodies under `texturecache/[0-f]/` (16 folders, lltexturecache.cpp:1121-1126) plus a 600-byte stripe per texture in one shared `texture.cache` read at `idx * 600` (lltexturecache.cpp:405), GPU-ready files in `gputex/`, and 265k assets in one flat `cache/` folder (lldiskcache.cpp:215). A region revisit reads thousands of scattered files, each with a seek plus a directory lookup. The J2C stripe file's slots are reused through a free list, so neighbouring slots do not mean neighbouring textures. ### Slice of the interface - **Arrival order = locality.** One writer thread appends to the active segment in queue order, so textures, assets and meshes that arrive together while loading a region land next to each other. The fetchers carry no region id. Arrival order is all the store gets, and that is enough for this ticket as written. A `Priority` on writes changes order only within the queue's short window. - `WearPolicy.locality_rewrites` (HDD only, drive type from #709) + the daily compaction cap: on a revisit, when the records read in a short window are spread over many segments, the store re-appends that hot set to the current segment. It reuses the compaction path (copy live record, kill old) and counts against the same cap. - Reads: `pread` in key order within a segment. The batch read for a hot set sorts by (segment, offset) before issuing. - Stats: per-read latency by drive type; "records gathered" and gathered bytes per day. ### Drop-in plan Policy on top of the store, after #709's drive detection. Measure the region-arrival load time on an HDD before and after: first with arrival-order writes only (comes free with the store), then with hot-set rewrites.

sky_line referenced this issue

2026-10-18 23:31:23 +00:00

[Epic] Cache rewrite: segment store, compression, tiered texture caching, disk-aware sizing #715

sky_line commented

2026-10-18 23:31:24 +00:00

Author

Owner

Prefetch on arrival (decided 2026-10-18, see #715 / #708 placement)

Triggers:

teleport: the destination is known before any object arrives, so prefetch runs during the teleport screen;
region crossing;
a neighbour connecting while the camera or avatar moves towards it.

Mechanism:

load the region's manifest and sort its runs by segment and offset;
read them sequentially in the background, below demand reads, then the shared references in manifest order;
Linux: posix_fadvise(WILLNEED) / readahead on the ranges (the page cache does the rest);
Windows: sequential reads into a bounded read-ahead buffer (e.g. 256 MB), which demand reads consult first.

Guards:

demand reads always win;
cancel when the region is left (a quick pass-through must not drag in a whole region);
cap bytes per prefetch and in flight;
don't prefetch while the disk is busy with demand reads.

HDD-only hot-set rewrite: on a revisit, if the region's records and its shared references are spread over many segments, append copies into a fresh run for that region, within #709's daily write cap. Never on SSD.

Measure: region-arrival load time (first full frame, textures and meshes resident) on an HDD and on an SSD, revisit with prefetch vs without.

## Prefetch on arrival (decided 2026-10-18, see #715 / #708 placement) **Triggers:** - teleport: the destination is known before any object arrives, so prefetch runs during the teleport screen; - region crossing; - a neighbour connecting while the camera or avatar moves towards it. **Mechanism:** - load the region's manifest and sort its runs by segment and offset; - read them sequentially in the background, below demand reads, then the shared references in manifest order; - Linux: `posix_fadvise(WILLNEED)` / `readahead` on the ranges (the page cache does the rest); - Windows: sequential reads into a bounded read-ahead buffer (e.g. 256 MB), which demand reads consult first. **Guards:** - demand reads always win; - cancel when the region is left (a quick pass-through must not drag in a whole region); - cap bytes per prefetch and in flight; - don't prefetch while the disk is busy with demand reads. **HDD-only hot-set rewrite:** on a revisit, if the region's records and its shared references are spread over many segments, append copies into a fresh run for that region, within #709's daily write cap. Never on SSD. **Measure:** region-arrival load time (first full frame, textures and meshes resident) on an HDD and on an SSD, revisit with prefetch vs without.

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

llcachestore: segment store core with SSD wear policy and region placement (#708, #709, #715)

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

sky_line/Slipstream#712

No description provided.

Rows
Columns