3p: update curl 7.54 -> current 8.x

sky_line commented

2026-10-04 14:10:16 +00:00

Owner

Update libcurl to a current 8.x release. No prebuilt package exists from AlchemyViewer or secondlife, so this needs our own 3p build (fork 3p-curl), built together with the OpenSSL update.

HTTP/1.1 pipelining was removed in curl 7.62. CURLMOPT_PIPELINING=1 (llcorehttp/_httplibcurl.cpp:494/497) would be ignored, so texture and mesh fetch fall back to connection limits: measure fetch throughput and tune connection counts.
Deprecated options to update: CURLOPT_PROTOCOLS/REDIR_PROTOCOLS (_httpoprequest.cpp:530-531, httpcommon.cpp:298-300), CURLINFO_SIZE_DOWNLOAD (_httpoprequest.cpp:287).
Enables curl_multi_poll/curl_multi_wakeup, which the HTTP-service busy-poll ticket needs.
Test: login, texture/mesh fetch rate, marketplace and web pages, event queue.

Part of keeping third-party dependencies current; check Linux and Windows packages together, and update autobuild.xml with autobuild edit installable.

Update libcurl to a current 8.x release. No prebuilt package exists from AlchemyViewer or secondlife, so this needs our own 3p build (fork 3p-curl), built together with the OpenSSL update. - HTTP/1.1 pipelining was removed in curl 7.62. `CURLMOPT_PIPELINING=1` (`llcorehttp/_httplibcurl.cpp:494/497`) would be ignored, so texture and mesh fetch fall back to connection limits: measure fetch throughput and tune connection counts. - Deprecated options to update: `CURLOPT_PROTOCOLS`/`REDIR_PROTOCOLS` (`_httpoprequest.cpp:530-531`, `httpcommon.cpp:298-300`), `CURLINFO_SIZE_DOWNLOAD` (`_httpoprequest.cpp:287`). - Enables `curl_multi_poll`/`curl_multi_wakeup`, which the HTTP-service busy-poll ticket needs. - Test: login, texture/mesh fetch rate, marketplace and web pages, event queue. Part of keeping third-party dependencies current; check Linux and Windows packages together, and update `autobuild.xml` with `autobuild edit installable`.

sky_line added the

platform

label

2026-10-04 14:10:16 +00:00

sky_line commented

2026-10-06 11:45:53 +00:00

Author

Owner

Analysis before upgrading (2026-10-06)

The claim that newer curl "drops HTTP/1.1" is a misunderstanding. Current curl still fully supports HTTP/1.1. What it removed (deprecated around 7.62, removed in 7.65, 2019) is HTTP/1.1 pipelining. On 8.x, CURLPIPE_HTTP1 is ignored. LL's llcorehttp pipelines AP_TEXTURE (8 connections by default, max 12, TextureFetchConcurrency) and AP_MESH2 (8, max 32, Mesh2MaxConcurrentRequests) (llappcorehttp.cpp ~64-117; the log prints "HTTP Pipelining enabled!"). After an upgrade, nothing breaks, but per-connection concurrency drops to 1, so asset loading could slow down.

The asset CDN has no HTTP/2 (tested: asset-cdn.glb.agni.lindenlab.com, Akamai):

Test	Result
Plain HTTP/1.1 (current)	200, `image/x-j2c`, keep-alive
h2c, prior knowledge	rejected ("peer does not support HTTP/2")
h2c `Upgrade`	ignored, stays on 1.1
HTTPS	serves Akamai's generic `a248.e.akamai.net` certificate (hostname mismatch). With verification off, ALPN offers `http/1.1` only

So HTTP/2 multiplexing can't replace pipelining for assets.

TPV policy constraint:

1.a: "must not use our protocol in a way that unduly burdens our servers"
2.f: "must not impose an unreasonable or disproportionately large load on our infrastructure"

Request and byte volume don't change without pipelining, only the connection count. Stay within LL's own shipped maxima (texture ≤ 12, mesh2 ≤ 32 connections). Do NOT raise the caps above LL's without LL's agreement.

Plan:

On the current build, measure cold-cache load time in a busy region (time until textures and meshes are sharp, texture console fetch rate) in three runs:
1. baseline;
2. HttpPipelining = false;
3. pipelining off with TextureFetchConcurrency = 12 and Mesh2MaxConcurrentRequests = 32.
If run (iii) is on par with the baseline: upgrade curl to 8.x together with OpenSSL 3 (#392), ship the higher in-cap defaults, and remove the dead pipelining paths from llcorehttp.
If there's a real gap: ask LL through the TPV developer channels (connection allowance, CDN HTTP/2 or HTTPS plans), and/or reduce fetches: visible-first priority, cancelling requests no longer needed, the tier-2 disk cache. Don't exceed LL's caps unilaterally.
Regression checks after the upgrade: login, region capabilities, inventory and AIS, marketplace and web media, voice, update check, chat image previews.

Security note: 7.54.1 is from 2017 and has many published vulnerabilities, and chat image previews now fetch arbitrary web URLs through it. So the upgrade should happen; the question is only how to tune it.

## Analysis before upgrading (2026-10-06) **The claim that newer curl "drops HTTP/1.1" is a misunderstanding.** Current curl still fully supports HTTP/1.1. What it removed (deprecated around 7.62, removed in 7.65, 2019) is **HTTP/1.1 pipelining**. On 8.x, `CURLPIPE_HTTP1` is ignored. LL's `llcorehttp` pipelines **AP_TEXTURE** (8 connections by default, max 12, `TextureFetchConcurrency`) and **AP_MESH2** (8, max 32, `Mesh2MaxConcurrentRequests`) (`llappcorehttp.cpp` ~64-117; the log prints "HTTP Pipelining enabled!"). After an upgrade, nothing breaks, but per-connection concurrency drops to 1, so asset loading could slow down. **The asset CDN has no HTTP/2** (tested: `asset-cdn.glb.agni.lindenlab.com`, Akamai): | Test | Result | |---|---| | Plain HTTP/1.1 (current) | 200, `image/x-j2c`, keep-alive | | h2c, prior knowledge | rejected ("peer does not support HTTP/2") | | h2c `Upgrade` | ignored, stays on 1.1 | | HTTPS | serves Akamai's generic `a248.e.akamai.net` certificate (hostname mismatch). With verification off, ALPN offers `http/1.1` only | So HTTP/2 multiplexing can't replace pipelining for assets. **TPV policy constraint:** - 1.a: "must not use our protocol in a way that unduly burdens our servers" - 2.f: "must not impose an unreasonable or disproportionately large load on our infrastructure" Request and byte volume don't change without pipelining, only the connection count. **Stay within LL's own shipped maxima (texture ≤ 12, mesh2 ≤ 32 connections). Do NOT raise the caps above LL's without LL's agreement.** **Plan:** 1. On the current build, measure cold-cache load time in a busy region (time until textures and meshes are sharp, texture console fetch rate) in three runs: 1. baseline; 2. `HttpPipelining` = false; 3. pipelining off with `TextureFetchConcurrency` = 12 and `Mesh2MaxConcurrentRequests` = 32. 2. **If run (iii) is on par with the baseline:** upgrade curl to 8.x together with OpenSSL 3 (#392), ship the higher in-cap defaults, and remove the dead pipelining paths from `llcorehttp`. 3. **If there's a real gap:** ask LL through the TPV developer channels (connection allowance, CDN HTTP/2 or HTTPS plans), and/or reduce fetches: visible-first priority, cancelling requests no longer needed, the tier-2 disk cache. Don't exceed LL's caps unilaterally. 4. Regression checks after the upgrade: login, region capabilities, inventory and AIS, marketplace and web media, voice, update check, chat image previews. **Security note:** 7.54.1 is from 2017 and has many published vulnerabilities, and chat image previews now fetch arbitrary web URLs through it. So the upgrade should happen; the question is only how to tune it.

sky_line referenced this issue

2026-10-06 12:18:09 +00:00

Our own asset fetcher: pipelined HTTP/1.1 client for texture/mesh GETs to the asset CDN #485

sky_line referenced this issue

2026-10-06 12:18:09 +00:00

Epic: networking -- own the asset fetch path, modern curl + OpenSSL for everything else #484

sky_line commented

2026-10-06 12:39:15 +00:00

Author

Owner

Measured: HttpPipelining on vs off, cold cache (2026-10-06)

Setup: the AssetFetch timeline log (e991adb381), real caches moved aside, default connection counts (texture 8, mesh2 8, well within LL's caps).

Light region (on first, then off):

	Mesh done	Textures drained	Queue < 10	Texture GETs
ON	+12.1 s	+40.6 s	+44.7 s	6,435
OFF	+8.0 s	+36.3 s	+40.4 s	6,420

Busy region (off first, then on; similar camera turn in both):

	Mesh main load	Textures drained	Queue empty	Texture GETs	Mesh
OFF	+26 s	+86.5 s	+90.6 s	19,313	20,174 GETs, 243 MB
ON	+32.5 s	+79.4 s	+83.5 s	19,489	20,070 GETs, 239 MB

Conclusion:

Pipelining's effect is about ±8% and roughly a wash. Whichever run went second was faster in both pairs, which suggests Akamai edge-cache warm-up and is about the same size as the effect itself.
No retries or errors in any run.
Throughput was about 200–245 texture GETs/s either way: limited by our own fetch and decode scheduling, not by pipelining.

Decision:

Upgrading to curl 8.x (no pipelining) is safe for asset loading at the current connection counts. No cap increase is needed, so no TPV-policy concern.
#485 (our own pipelined fetcher) is no longer needed for throughput; it's optional for scheduling only.
The next win is our fetch and decode pipeline.

## Measured: HttpPipelining on vs off, cold cache (2026-10-06) Setup: the AssetFetch timeline log (e991adb381), real caches moved aside, default connection counts (texture 8, mesh2 8, well within LL's caps). **Light region** (on first, then off): | | Mesh done | Textures drained | Queue < 10 | Texture GETs | |---|---|---|---|---| | ON | +12.1 s | +40.6 s | +44.7 s | 6,435 | | OFF | **+8.0 s** | **+36.3 s** | **+40.4 s** | 6,420 | **Busy region** (off first, then on; similar camera turn in both): | | Mesh main load | Textures drained | Queue empty | Texture GETs | Mesh | |---|---|---|---|---|---| | OFF | **+26 s** | +86.5 s | +90.6 s | 19,313 | 20,174 GETs, 243 MB | | ON | +32.5 s | **+79.4 s** | **+83.5 s** | 19,489 | 20,070 GETs, 239 MB | **Conclusion:** - Pipelining's effect is about ±8% and roughly a wash. Whichever run went second was faster in both pairs, which suggests Akamai edge-cache warm-up and is about the same size as the effect itself. - No retries or errors in any run. - Throughput was about 200–245 texture GETs/s either way: limited by our own fetch and decode scheduling, not by pipelining. **Decision:** - Upgrading to curl 8.x (no pipelining) is safe for asset loading at the current connection counts. No cap increase is needed, so no TPV-policy concern. - #485 (our own pipelined fetcher) is no longer needed for throughput; it's optional for scheduling only. - The next win is our fetch and decode pipeline.

sky_line referenced this issue

2026-10-06 12:39:15 +00:00

Our own asset fetcher: pipelined HTTP/1.1 client for texture/mesh GETs to the asset CDN #485

sky_line referenced this issue from a commit

2026-10-06 13:14:08 +00:00

AssetFetch timeline log: texture/mesh fetch progress after login + settle time (#484)

sky_line referenced this issue from a commit

2026-10-06 14:14:05 +00:00

OpenSSL 3.5.9 + curl 8.22.0; llcorehttp adapted to curl 8

sky_line added this to the Slipstream project

2026-10-06 15:04:14 +00:00

sky_line added the