Our own asset fetcher: pipelined HTTP/1.1 client for texture/mesh GETs to the asset CDN

sky_line commented

2026-10-06 12:18:09 +00:00

Owner

Part of #484.

Why: asset downloads are plain HTTP/1.1 GETs to one known CDN (asset-cdn.glb.agni.lindenlab.com, Akamai: HTTP/1.1 only, no HTTP/2, no usable HTTPS; see #391). They're today's only use of curl's HTTP/1.1 pipelining, which modern curl removed. A purpose-built client lets curl be upgraded freely and gives us control where performance matters.

Scope: GET only; Content-Length-framed (plus chunked if the CDN ever sends it); keep-alive; a fixed pipeline depth per connection; responses matched in order; retry or requeue when a connection drops; timeouts.

Connection count within LL's shipped caps (texture ≤ 12, mesh2 ≤ 32) per the TPV policy (1.a / 2.f). Never more without LL's agreement.
Our scheduling: visible and near assets first; cancel requests that are no longer needed (including in-flight pipelined ones, by dropping their responses); feed the existing texture fetch and mesh repo state machines.
Zero-copy into the decode and GPU upload path where possible.
Proxies: honour the viewer's HTTP/SOCKS proxy settings, or fall back to curl when a proxy is configured.
Hostile-input safety: strict response parser, size caps, fuzz tests, fault injection (drops, partial responses, slow server).
Stats: requests/s, bytes/s, connections, pipeline occupancy, retries, time to first byte.

Gate: start only if the #391 measurement shows pipelining matters. Otherwise plain modern curl within the caps is enough.

Part of #484. **Why:** asset downloads are plain HTTP/1.1 GETs to one known CDN (`asset-cdn.glb.agni.lindenlab.com`, Akamai: HTTP/1.1 only, no HTTP/2, no usable HTTPS; see #391). They're today's only use of curl's HTTP/1.1 pipelining, which modern curl removed. A purpose-built client lets curl be upgraded freely and gives us control where performance matters. **Scope:** GET only; `Content-Length`-framed (plus chunked if the CDN ever sends it); keep-alive; a fixed pipeline depth per connection; responses matched in order; retry or requeue when a connection drops; timeouts. - **Connection count within LL's shipped caps** (texture ≤ 12, mesh2 ≤ 32) per the TPV policy (1.a / 2.f). Never more without LL's agreement. - **Our scheduling:** visible and near assets first; cancel requests that are no longer needed (including in-flight pipelined ones, by dropping their responses); feed the existing texture fetch and mesh repo state machines. - **Zero-copy** into the decode and GPU upload path where possible. - **Proxies:** honour the viewer's HTTP/SOCKS proxy settings, or fall back to curl when a proxy is configured. - **Hostile-input safety:** strict response parser, size caps, fuzz tests, fault injection (drops, partial responses, slow server). - **Stats:** requests/s, bytes/s, connections, pipeline occupancy, retries, time to first byte. **Gate:** start only if the #391 measurement shows pipelining matters. Otherwise plain modern curl within the caps is enough.

sky_line added the

labels

2026-10-06 12:18:09 +00:00

sky_line referenced this issue

2026-10-06 12:18:09 +00:00

Epic: networking -- own the asset fetch path, modern curl + OpenSSL for everything else #484

sky_line referenced this issue

2026-10-06 12:39:15 +00:00

3p: update curl 7.54 -> current 8.x #391

sky_line referenced this issue

2026-10-06 12:39:15 +00:00

Epic: networking -- own the asset fetch path, modern curl + OpenSSL for everything else #484

sky_line commented

2026-10-06 12:39:15 +00:00

Author

Owner

Deprioritised: measured on #391, pipelining on vs off is roughly a wash (±8%, within run-order noise) in both a light and a busy region. Plain curl 8 at the current connection counts is enough. Keep this only for scheduling features (cancellation, visible-first priority), at low priority.

sky_line added this to the Slipstream project

2026-10-06 15:04:14 +00:00

sky_line commented

2026-10-06 21:00:23 +00:00

Author

Owner

Status (2026-10-06): curl 8 + OpenSSL 3 shipped in 1.1.0 Beta 5 to 7 with pipelining removed (8 texture requests in flight over 8 connections, the same setup measured as a wash in #391). A Wireshark pass found no TLS problems; the only real incompatibility (the CDN's bogus Content-Encoding) is handled with raw-body fetches. The Alchemy dev confirms pipelining was the only reason he advised against the curl upgrade.

Decision: keep this ticket parked as the fallback. Watch beta reports and the AssetFetch timeline log (#AssetFetch, settle time after login) for slower texture/mesh loading. If regressions show up, build our own pipelined HTTP/1.1 asset fetcher here.

Status (2026-10-06): curl 8 + OpenSSL 3 shipped in 1.1.0 Beta 5 to 7 with pipelining removed (8 texture requests in flight over 8 connections, the same setup measured as a wash in #391). A Wireshark pass found no TLS problems; the only real incompatibility (the CDN's bogus Content-Encoding) is handled with raw-body fetches. The Alchemy dev confirms pipelining was the only reason he advised against the curl upgrade. **Decision:** keep this ticket parked as the fallback. Watch beta reports and the AssetFetch timeline log (`#AssetFetch`, settle time after login) for slower texture/mesh loading. If regressions show up, build our own pipelined HTTP/1.1 asset fetcher here.

sky_line referenced this issue from a commit

2026-10-07 00:37:07 +00:00

AssetFetch log: new connections vs requests per asset class (keep-alive health)

sky_line commented

2026-10-09 05:23:59 +00:00

Author

Owner

Design direction (owner, 2026-10-09) + evidence from a high-speed mainland flight

Evidence (1.2.0 Beta 6, flying fast across Zindra, 4096 m draw distance, 25-region ring). About 15–30 s into the flight:

all connected sims went silent at the same instant (two runs), and the circuits then timed out;
in the same window the fetch log showed 2,227 new connections for 23,062 texture requests and ~18,000 queued fetches;
RAM went from 5 to 19 GB: 39k raw images, CPU texture copies past their allowance, and the memory governor denying decodes ~230k/s.

The current guess is server-side abuse protection reacting to the request/connection burst (not confirmed; a Firestorm A/B is pending).

Design: the fetcher should rate-limit itself.

Connections: a small fixed pool of long-lived keep-alive connections per CDN host, with HTTP/1.1 pipelining (HTTP/2 if available). No per-request connection churn; maximum connections are a hard cap.
Queue: one priority queue for textures and mesh, re-ranked from live demand. A request is only sent when a pool slot frees, so the queue can grow without the network seeing it.
Cancel before send: requests whose demand expired or whose object left view are dropped from the queue before sending (VK demand already expires at 5 s). At high speed most queued requests age out unsent, which rate-limits naturally. When the user slows down, the queue settles on what is actually nearby and drains.
Bounded memory: queued requests hold no data; decode and CPU copies happen only for what arrives. This also bounds the RAM runaway above.

Possible companion (#568): pace how fast new neighbour (child-agent) circuits open while moving very fast, in case that churn is the trigger rather than HTTP.

## Design direction (owner, 2026-10-09) + evidence from a high-speed mainland flight **Evidence (1.2.0 Beta 6, flying fast across Zindra, 4096 m draw distance, 25-region ring).** About 15–30 s into the flight: - all connected sims went silent at the same instant (two runs), and the circuits then timed out; - in the same window the fetch log showed **2,227 new connections for 23,062 texture requests** and ~18,000 queued fetches; - RAM went from 5 to 19 GB: 39k raw images, CPU texture copies past their allowance, and the memory governor denying decodes ~230k/s. The current guess is server-side abuse protection reacting to the request/connection burst (not confirmed; a Firestorm A/B is pending). **Design: the fetcher should rate-limit itself.** 1. **Connections:** a small fixed pool of long-lived keep-alive connections per CDN host, with HTTP/1.1 pipelining (HTTP/2 if available). No per-request connection churn; maximum connections are a hard cap. 2. **Queue:** one priority queue for textures and mesh, re-ranked from live demand. A request is only *sent* when a pool slot frees, so the queue can grow without the network seeing it. 3. **Cancel before send:** requests whose demand expired or whose object left view are dropped from the queue before sending (VK demand already expires at 5 s). At high speed most queued requests age out unsent, which rate-limits naturally. When the user slows down, the queue settles on what is actually nearby and drains. 4. **Bounded memory:** queued requests hold no data; decode and CPU copies happen only for what arrives. This also bounds the RAM runaway above. **Possible companion (#568):** pace how fast new neighbour (child-agent) circuits open while moving very fast, in case that churn is the trigger rather than HTTP.

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: llcorehttp: fixed keep-alive pool per class, socket open/close gauge, reply notifier

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: llcorehttp: smooth start pacing and per-class busy back-off

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: asset CDN connection plan, non-CDN texture class, pacing/back-off settings

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: texture fetch gate: priority queue, cancel-before-send, event-driven refill, parked decode admission

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: texture fetch: drop unsent requests for textures that left the view

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: mesh fetch: in-flight limit, drop orphaned requests before sending

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: FetchGate stats line; active trim of CPU texture copies over the allowance

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: review fixes: deleted workers skip parked decodes, cheaper deleteOK, larger queue cap, exemptions

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: updateRegions respects a real time budget (no 1 s loosening outside the progress screen)

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: bound texture fetch workers; raw/formatted byte gauges; network-DD texture gate

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: mesh dispatch never starved by delayed retries; pending cap; horizontal network-DD gate; MeshDelayed log

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Refs #485: RAM attribution + fetch-gate log fields; mesh retry delays on a real clock, delayed count from list sizes; bounded worker eviction per sweep

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Fix build: mesh delayed-due llmin mixed F64 with LLUnit seconds (Refs #485)

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

RAM log: mallinfo2 heap stats opt-in (SLMemLogHeapStats, default off) -- it walked the heap under arena locks, 426 ms hitches (Refs #485)

sky_line commented

2026-10-09 08:53:45 +00:00

Author

Owner

Done differently, shipped in 1.2.0 Beta 7. This ticket's gate said to build a custom pipelined client only if pipelining mattered; #391 measured it as a wash, and a curl probe confirmed the asset CDN speaks HTTP/1.1 only (no HTTP/2, no usable HTTPS). So instead of a new client, the existing llcorehttp path was fixed and put under our own scheduling:

long-lived keep-alive pool per class (churn came from curl's shrinking idle cache and mixed hosts in the texture class; non-CDN textures now have their own class); 18 connections total across texture/mesh/large mesh, unchanged from LL's caps;
token-bucket start pacing; busy back-off on 429/5xx/timeouts honouring Retry-After;
priority queue with cancel-before-send, event-driven refill, dropping requests for things no longer wanted;
horizontal network-draw-distance gate for texture and mesh requests;
bounded fetch workers, decoded/encoded bytes and pending mesh requests; FetchGate stats line.
Measured on a 27-crossing dense flight: new connections ~0/s after warm-up, 0 server errors, RAM peak 11 GB (was 58 GB).

Done differently, shipped in 1.2.0 Beta 7. This ticket's gate said to build a custom pipelined client only if pipelining mattered; #391 measured it as a wash, and a curl probe confirmed the asset CDN speaks HTTP/1.1 only (no HTTP/2, no usable HTTPS). So instead of a new client, the existing llcorehttp path was fixed and put under our own scheduling: - long-lived keep-alive pool per class (churn came from curl's shrinking idle cache and mixed hosts in the texture class; non-CDN textures now have their own class); 18 connections total across texture/mesh/large mesh, unchanged from LL's caps; - token-bucket start pacing; busy back-off on 429/5xx/timeouts honouring Retry-After; - priority queue with cancel-before-send, event-driven refill, dropping requests for things no longer wanted; - horizontal network-draw-distance gate for texture and mesh requests; - bounded fetch workers, decoded/encoded bytes and pending mesh requests; FetchGate stats line. Measured on a 27-crossing dense flight: new connections ~0/s after warm-up, 0 server errors, RAM peak 11 GB (was 58 GB).

sky_line closed this issue

2026-10-09 08:53:45 +00:00

sky_line referenced this issue

2026-10-09 08:53:45 +00:00

Epic: networking -- own the asset fetch path, modern curl + OpenSSL for everything else #484

sky_line referenced this issue from a commit

2026-10-12 13:45:01 +00:00

#707: texture CPU copies only while a consumer holds a lease; drop, never thumbnail

Rows
Columns

Our own asset fetcher: pipelined HTTP/1.1 client for texture/mesh GETs to the asset CDN #485

Design direction (owner, 2026-10-09) + evidence from a high-speed mainland flight