#1536: DLSS Frame Generation with SDR output #1543

Closed
TheSlowly wants to merge 0 commits from TheSlowly/Slipstream:framegen-sdr into main
Collaborator

Closes #1536 (follow-up of #1517): frame generation needed HDR10 output; it now runs with 8-bit SDR output too, on Windows and Linux (X11 included).

While the swapchain is a frame generation chain, SDR renders the world into the present slot's own image (the chain's 8-bit format, exactly what SDR shows; that is the generator's input) and the interface, HUDs and markers into a separate layer. One composite (display shader encoding 5, the SDR swapchain's own blend in 8-bit encoded space) lays name tags, then the interface, over the world. The real frame and every generated frame use it, so the interface is the same on all of them. With frame generation off, SDR draws exactly as before: none of that path changed.

  • The renderer's HDR flag is split into "linear output" and "a UI layer exists", so SDR can have the layer.
  • Snapshots with HUDs or the interface (and high-resolution ones) compose the layer while it runs.
  • scRGB HDR and *_SRGB-only SDR surfaces get their own reasons; Preferences and docs say SDR and HDR10.
  • No add-on change needed: the add-on already takes the format and the HDR flag. Its minimum version stays as it was (1.3, 1.4 for 5x / 6x); the 1.4.1 fixes on main apply to SDR too.
  • New check framegen-sdr, X11 rule in framegen-linux, test plan doc/testplans/1536-frame-generation-sdr.md.

Tested in world on Windows (RTX 5070 Ti, exclusive fullscreen): SDR 2x and 4x, generated-only pictures carry the same interface, live Off / 2x / Off and HDR <-> SDR switches, name tags, typing and dragging, snapshots. Not tested: starting the viewer with frame generation already on in SDR, and Linux.

Known, minor: with Windows in HDR mode and the viewer in SDR, present calls wait in the driver (Windows converts the SDR picture); no fps cost. A swapchain rebuild that fails its frame generation images can freeze the picture for a pipeline compile.

Closes #1536 (follow-up of #1517): frame generation needed HDR10 output; it now runs with 8-bit SDR output too, on Windows and Linux (X11 included). While the swapchain is a frame generation chain, SDR renders the world into the present slot's own image (the chain's 8-bit format, exactly what SDR shows; that is the generator's input) and the interface, HUDs and markers into a separate layer. One composite (display shader encoding 5, the SDR swapchain's own blend in 8-bit encoded space) lays name tags, then the interface, over the world. The real frame and every generated frame use it, so the interface is the same on all of them. With frame generation off, SDR draws exactly as before: none of that path changed. - The renderer's HDR flag is split into "linear output" and "a UI layer exists", so SDR can have the layer. - Snapshots with HUDs or the interface (and high-resolution ones) compose the layer while it runs. - scRGB HDR and `*_SRGB`-only SDR surfaces get their own reasons; Preferences and docs say SDR and HDR10. - No add-on change needed: the add-on already takes the format and the HDR flag. Its minimum version stays as it was (1.3, 1.4 for 5x / 6x); the 1.4.1 fixes on main apply to SDR too. - New check `framegen-sdr`, X11 rule in `framegen-linux`, test plan `doc/testplans/1536-frame-generation-sdr.md`. Tested in world on Windows (RTX 5070 Ti, exclusive fullscreen): SDR 2x and 4x, generated-only pictures carry the same interface, live Off / 2x / Off and HDR <-> SDR switches, name tags, typing and dragging, snapshots. Not tested: starting the viewer with frame generation already on in SDR, and Linux. Known, minor: with Windows in HDR mode and the viewer in SDR, present calls wait in the driver (Windows converts the SDR picture); no fps cost. A swapchain rebuild that fails its frame generation images can freeze the picture for a pipeline compile.
The frame's situation now says which output the chain presents: 8-bit SDR, another SDR format,
HDR10 or scRGB. 8-bit SDR and HDR10 are outputs frame generation runs on; scRGB keeps its block with
a clearer text, and an SDR surface in an *_SRGB format gets its own. sdrLayer() is the one test for
the SDR interface layer: 8-bit SDR on a frame generation chain.

Until the SDR frame path exists, the render loop still builds a frame generation chain for HDR10
only, so nothing changes for SDR yet.
The display shader gets encoding 5: an SDR world (8-bit, sRGB-encoded) with the moving labels and
the interface layer over it, in that encoded space -- the SDR swapchain's own blend, with no
transfer functions and no dither. Every picture of an SDR frame generation chain will use it, the
real one and each generated one, so the interface is the same texels in all of them. A pixel no
label and no interface reaches keeps the world texel exactly.

vkdisplay::composeLayerSdr mirrors it in C++. The new framegen-sdr check tests the mirror (an 8-bit
world unchanged, opaque and additive texels, against drawing straight onto the world: exact for one
8-bit draw, within two steps for two stacked translucent draws, labels before the interface) and
pins the shader to it; the HDR UI-layer contract runs its blend-mode table through it.
VkDisplayEncode gets an SDR-layer mode for an 8-bit SDR target (B8G8R8A8 or R8G8B8A8 UNORM). It
builds no HDR pipelines and never encodes; it draws the real picture (recordLayerComposite) and each
generated one (recordFgComposite) with encoding 5, through one shared draw, so the two cannot
drift apart. fgHudlessFormat() gives the generator's image format for a chain: HDR10's 10-bit
format, or the SDR chain's own. frameGenReady() says whether the encode has what frame generation
needs on its output, and the frame loop asks it.

Nothing prepares the SDR-layer mode yet; HDR is unchanged.
On an 8-bit SDR chain each slot's world image and the generator's scratch image are made in the
chain's format (on HDR10 they stay 10-bit). The slot's world image gets a swapchain image's usage,
since the world will render straight into it. A GPU that cannot write that format as a storage
image is refused before any image is made, with its own reason in the status. A chain that changes
format at the same size gets new images.
The renderer's HDR flag meant two things: the world is linear light, and the markers, HUDs and
interface draw into a separate UI layer. They are now separate. The UI layer exists in HDR, as
before, and can be asked for in SDR (requestDisplayOutput's new uiLayer); frame generation will ask
for it on an SDR chain. Everything that draws over the finished world follows the layer flag: its
format, the HUD pipelines' layer-opaque faces, the late layer, the HUD target and the guards.
Everything that changes the world image (tonemap, glow, final, sphere, probes) still follows linear
output. The pipeline bank's key includes the layer, so a cached bank of the other kind is never
reused.

Nothing asks for an SDR layer yet; HDR and SDR draw as before.
configureDisplayOutput takes the UI layer before init, so a renderer created
while the main swapchain is already an SDR frame generation chain builds its
HUD, marker and interface pipelines for the layer at once instead of waiting
for an output request. The default keeps every other caller as it was.
On an SDR frame generation chain the world is drawn into the present slot's
world image, the interface into the slot's layer, and one composite lays
labels and interface over the world into the backbuffer; the present thread
runs the same composite over each generated world. One local, sdrLayer,
selects all of it: on a normal SDR chain nothing differs from before.

The output transaction carries the UI layer, so the renderer's layer
pipelines are staged before the chain is rebuilt for frame generation and
dropped when it is rebuilt without. The SDR composite is made and released
only with the present thread drained, and a last-line gate withholds an
SDR-layer frame whose present slots, composite or renderer layer do not
match the chain.

The generator's feature takes the chain's own format and, in SDR, the
swapchain's size; colorHdr stays HDR10's. chainGenerated is given
llfg::outputGenerates. Snapshot captures on such a chain still read the
world image alone; that follows separately.

Rules: framegen-direct FRAME / OUTPUT-CHAIN, framegen-sdr LAYER,
framegen-labels.
On an SDR frame generation chain the world and the interface are drawn into separate images, so a
snapshot that should include the HUDs or the interface would have had the world only. At those
capture points the frame now composes the two into the backbuffer with the same composite as the
picture, and the snapshot copies that (the frame's own composite rewrites it at the end). The
capture uses its own descriptor set, so the frame's final composite never rewrites a set the
command buffer already bound. High-resolution (offscreen) snapshots with HUDs get their own
transient layer, as in HDR, composed into an 8-bit capture.

Other readers of the frame, checked:
- the world-only snapshot point and the periodic logout screenshot read the world image, before
  any interface, as in SDR without frame generation;
- instant replay reads the composed backbuffer, so it keeps the interface;
- the interface backdrop is the world image without HUDs, as in HDR;
- the VR mirror never runs on a frame generation chain.

A live snapshot preview taken while frames are being generated shows the world and interface
without the moving name tags (they are in their own atlas at that point).
X11 was left out only because frame generation needed HDR output, which X11 does not have. With
SDR supported, nothing in frame generation looks at the window system: the gates are the same on
X11 and Wayland. The framegen-linux check now keeps it that way. No gate in frame generation's
sources or the frame loop may name X11, Wayland or the video driver, and the availability rule
reads only the output in use.
The Frame Generation tooltip and the RenderVKFrameGen setting's description no longer say it needs
HDR10 output; both name SDR and HDR10. The framegen-sdr check keeps any "needs HDR" wording out of
the tooltip, the setting descriptions, the availability texts and frame generation's refusal
reasons.
The user doc now says frame generation runs with SDR or HDR10 output, and on Linux in SDR on X11
and Wayland. It gives the new reasons for scRGB and for an SDR format it cannot take, and describes
the SDR frame: the world in the slot's own 8-bit image, the interface in its layer, one composite.
The design doc adds why the SDR layer exists only while frame generation runs, the SDR composite
and why it is exact, and the small differences from direct SDR drawing while it runs. A new test
plan covers SDR on Windows and Linux and its snapshots; the #1517 plan and the add-on README follow.
The offscreen snapshot's colour target was made sampleable only with HDR output, but in the SDR
layer the HUD shot's composite samples it too; without that usage the draw is invalid Vulkan and
may fail on some drivers. It is now sampleable whenever the renderer has a UI layer (HDR is
unchanged: it always has one).
If the renderer rejects the pipelines for the SDR interface layer, frame generation now says so in
its status and the viewer stays on a normal chain. Before, it asked for the same pipelines again
every 2 seconds and held back every swapchain rebuild meanwhile, resizes included, until frame
generation was switched off.

The SDR composite's encoder is also released only after a rebuild whose device wait succeeded:
frames recorded before a failed wait may still be using it.
The renderer's "no UI layer for this frame" warnings no longer start with "HDR:", since an SDR
frame generation frame has a UI layer too. A comment that said a live snapshot preview lacks the
moving name tags while frames are generated is corrected: those frames draw the labels into the
interface layer, so the shot has them.
Author
Collaborator

Merged into main (every commit of this branch is in main); closing.

Merged into main (every commit of this branch is in main); closing.
TheSlowly closed this pull request 2026-11-15 23:05:45 +00:00

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sky_line/Slipstream!1543
No description provided.