sky_line/Slipstream

Fork 1

Cache store: zstd compression for GPU-ready textures #710

New issue

Open

opened 2026-10-12 11:17:23 +00:00 by sky_line · 1 comment

sky_line commented

2026-10-12 11:17:23 +00:00

Owner

Part of #715 (cache rewrite epic).

Measured on one install (sample of the GPU-ready cache, one Zen 4 core):

zstd level	size	compress	decompress
1	54%	680 MB/s	1.75 GB/s
3	51%	330 MB/s	1.6 GB/s
9	48%	92 MB/s	1.65 GB/s

By format at level 3: BC1 29%, BC3 18%, BC5 61%, BC7 42% -- about 45% overall.

Level 3 by default; level 1 on machines with <= 4 cores. Decompress on the cache's worker threads.
The same budget holds ~2x the textures; a GPU-ready hit is sub-millisecond vs a 30-200 ms J2C decode + encode on a miss, so the hit-rate gain matters most on the weakest machines.
Load time per texture: neutral on SATA/NVMe, slightly faster on hard disks (less to read).
Not applied to J2C or assets (already compressed: ~94% at zstd-3).
Later: rate-distortion-optimised BC encoding for better compressibility; GPU decompression.

Part of #715 (cache rewrite epic). Measured on one install (sample of the GPU-ready cache, one Zen 4 core): | zstd level | size | compress | decompress | |---|---|---|---| | 1 | 54% | 680 MB/s | 1.75 GB/s | | 3 | 51% | 330 MB/s | 1.6 GB/s | | 9 | 48% | 92 MB/s | 1.65 GB/s | By format at level 3: BC1 29%, BC3 18%, BC5 61%, BC7 42% -- about 45% overall. - Level 3 by default; level 1 on machines with <= 4 cores. Decompress on the cache's worker threads. - The same budget holds ~2x the textures; a GPU-ready hit is sub-millisecond vs a 30-200 ms J2C decode + encode on a miss, so the hit-rate gain matters most on the weakest machines. - Load time per texture: neutral on SATA/NVMe, slightly faster on hard disks (less to read). - Not applied to J2C or assets (already compressed: ~94% at zstd-3). - Later: rate-distortion-optimised BC encoding for better compressibility; GPU decompression.

sky_line added the

triage

perf

feature

labels

2026-10-12 11:17:23 +00:00

sky_line added this to the Slipstream project

2026-10-12 11:17:23 +00:00

sky_line referenced this issue

2026-10-12 11:17:23 +00:00

[Epic] Cache rewrite: segment store, compression, tiered texture caching, disk-aware sizing #715

sky_line commented

2026-10-18 22:52:43 +00:00

Author

Owner

Interface audit (step 1 of #715): the codec shim and how GPU-ready uses it.

Touches

newview/llgputexcache.cpp run_write (:282) / run_read (:252). The payload is what gets compressed; the 128-byte LLGpuTexHeader (llgputexcache.h:65) becomes record meta + an uncompressed prefix.
Writer source: the renderer passes a mapped Vulkan staging buffer + a released flag (llcleanrenderervk.cpp:12170). Compression runs on the store's writer side after the copy, so the renderer is unchanged.
In-flight bounds stay: reads 24 / 96 MB, write copies 64 MB (llgputexcache.cpp:59-62).

Slice of the interface

struct Codec {                         // bytes in -> bytes out, independent of the backend
  enum Id : U8 { IDENTITY = 0, ZSTD = 1 };
  static bool encode(Id, int level, const U8* src, size_t n, std::vector<U8>& dst); // false -> store IDENTITY
  static bool decode(Id, const U8* src, size_t n, U8* dst, size_t raw_n);
};
struct KindPolicy { Codec::Id codec; int level; U32 format_version; bool ranged_reads; ... };
Store::setKindPolicy(Kind::GPUTEX, {Codec::ZSTD, cores <= 4 ? 1 : 3, ...});

The record header stores the codec id, raw_bytes and stored_bytes; RecordInfo exposes both, so budgets count stored bytes and stats show the ratio.
Decode runs on the store's read threads (the gputex pool today).
A toggle (KindPolicy.codec = IDENTITY) works per kind at runtime. Old records stay readable because each record names its codec.
J2C, assets, mesh and cluster pages stay IDENTITY: J2C and mesh are already compressed (~94% at zstd-3), and they need range reads (pread into the record).
The same Codec is used by the object cache without the store (#713).

Drop-in plan

Add zstd to indra/cmake/3p_manifest.json (scripts/3p/manifest.py add; Linux + Windows cross). It is not a dependency today (zlib-ng only).
Codec in the store library with round-trip and corrupt-input tests.
GPU-ready moves to the store (first consumer, #708) with ZSTD on. Before / after: hit rate at the same budget, bytes per texture, read time.

**Interface audit (step 1 of #715): the codec shim and how GPU-ready uses it.** ### Touches - newview/llgputexcache.cpp `run_write` (:282) / `run_read` (:252). The payload is what gets compressed; the 128-byte `LLGpuTexHeader` (llgputexcache.h:65) becomes record meta + an uncompressed prefix. - Writer source: the renderer passes a mapped Vulkan staging buffer + a `released` flag (llcleanrenderervk.cpp:12170). Compression runs on the store's writer side after the copy, so the renderer is unchanged. - In-flight bounds stay: reads 24 / 96 MB, write copies 64 MB (llgputexcache.cpp:59-62). ### Slice of the interface ```cpp struct Codec { // bytes in -> bytes out, independent of the backend enum Id : U8 { IDENTITY = 0, ZSTD = 1 }; static bool encode(Id, int level, const U8* src, size_t n, std::vector<U8>& dst); // false -> store IDENTITY static bool decode(Id, const U8* src, size_t n, U8* dst, size_t raw_n); }; struct KindPolicy { Codec::Id codec; int level; U32 format_version; bool ranged_reads; ... }; Store::setKindPolicy(Kind::GPUTEX, {Codec::ZSTD, cores <= 4 ? 1 : 3, ...}); ``` - The record header stores the codec id, `raw_bytes` and `stored_bytes`; `RecordInfo` exposes both, so budgets count stored bytes and stats show the ratio. - Decode runs on the store's read threads (the gputex pool today). - A toggle (`KindPolicy.codec = IDENTITY`) works per kind at runtime. Old records stay readable because each record names its codec. - J2C, assets, mesh and cluster pages stay IDENTITY: J2C and mesh are already compressed (~94% at zstd-3), and they need range reads (`pread` into the record). - The same `Codec` is used by the object cache without the store (#713). ### Drop-in plan 0. Add zstd to `indra/cmake/3p_manifest.json` (`scripts/3p/manifest.py add`; Linux + Windows cross). It is not a dependency today (zlib-ng only). 1. `Codec` in the store library with round-trip and corrupt-input tests. 2. GPU-ready moves to the store (first consumer, #708) with ZSTD on. Before / after: hit rate at the same budget, bytes per texture, read time.

sky_line referenced this issue

2026-10-18 22:52:44 +00:00

Object cache: zstd compression #713

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

sky_line/Slipstream#710

No description provided.

Rows
Columns