#1517: DLSS Frame Generation up to 4x (6x in fullscreen), the interface sharp on every frame #1525
No reviewers
Labels
No labels
amd
avatars
bug
builder
cannot-reproduce
crash
epic
feature
from-viewer
gl-removal
known-limitation
monitoring
parity
perf
platform
priority
critical
priority
high
priority
low
priority
medium
provenance
rendering
rt
status
in-progress
status
testing
status
up-next
triage
ui
upstream
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
sky_line/Slipstream!1525
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "TheSlowly/Slipstream:framegen-1517"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Fixes #1517 (part of #1114). DLSS Frame Generation for the Vulkan renderer: 2x to 4x on RTX 50 (2x on RTX 40), and 5x / 6x in fullscreen, where the driver paces the frames.
Builds on #1453 (HDR): this branch sits on top of
hdr-display-1451, so its first commits here are that PR's until it is merged.What it does:
Needs: the DLSS add-on 1.4.0 (
contrib/ngxshim) with NVIDIA's frame generation library, DLSS upscaling, HDR10 output, and on Windows hardware-accelerated GPU scheduling. SDR output is a follow-up.Docs:
doc/frame-generation.md(how it works),doc/frame-generation-design.md(why each low-level decision, prior art: NVIDIA RTX Remix's presenter, no code copied),doc/testplans/1517-frame-generation.md.Tested in world on Windows (RTX 5070 Ti, HDR, 2x to 6x, labels, translucent windows, fullscreen driver pacing). Not yet run: Linux, RTX 40. Source checks:
python3 scripts/check/run.py --stage source(38 pass).An earlier HDR measurement put the world at about half its SDR brightness, and reading the chain from the tonemap to the encode found no such factor. The probe could only see the post-chain input and the encode input, so there was no way to tell which stage between them, if any, changes the light. (With these taps the factor did not reproduce: both stages read the same in SDR and HDR at the same pixel.) RenderVKFSRDebugProbe now also logs one pixel (RenderVKFSRDebugProbeX/Y) of the tonemap's own output ("tonemapOut", before the glow, AA, FX and final passes) and of the finished world image ("finalOut", before the HUD and UI), in both modes. HDR lines give relative linear (1 = paper white); SDR lines decode the display-encoded value with the sRGB curve, so the HDR / SDR ratio of each stage reads straight off the log. A target the probe can sample is read by its existing dispatch; the SDR swapchain, which has no sampled usage, has its one texel copied instead. Debug only: nothing runs while the setting is off. Test-Plan: clang-cl /Zs (Windows flags) on llcleanfsr3vk.cpp and llcleanrenderervk.cpp: no errors, no new warnings clean_fsr_probe.comp: glslc + spirv-val (unchanged) Texel decode helpers (half, BGRA8, A2B10G10R10, sRGB decode) checked in a standalone harness HDR suite (hdr_display_units_checks, hdr_live_output_contract, hdr_sdr_capture_contract, test_hdr_settings_split, hdr_snapshot_source_checks, display_mode_resident_source_checks, display_hdr_checks.cpp): pass In world (after a build): RenderVKFSRDebugProbe on, stand still, read the tonemapOut / finalOut lines in SDR and in HDR at the same pixel and compare relative vs decoded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>On an HDR panel the brightest highlights stopped near 200 nits with 456 available. The headroom shoulder gave a scene luminance L only 1 - exp(-(L - 1) / 12) of the room above paper white with the default white point 6: a highlight four times diffuse white got 23 % of it, and the curve's slope jumped at diffuse white, which can show as a ring in smooth gradients such as the sky. The shoulder now adds room * g(L), g = (1 - exp(-k (L - 1)))^2 with k = 2.97 / max(W - 1, 1): zero with zero slope at diffuse white, so the SDR result hands over without a kink, 0.9 of the headroom at the tone curve's white point W and the rest above it. The per-channel peak bound and the hue-preserving scale stay. vkdisplay::headroomWeight mirrors it for the log line and the checks. Nothing changes at or below diffuse white, and nothing changes in SDR. Test-Plan: hdr_display_units_checks.py (production GLSL with GLM): hdrHeadroomWeight 0 / flat at L = 1, >= 0.9 at W in {2, 6, 16}, monotonic on [1, 64]; displayHeadroom at headroom ratios 1, 1.25, 5.7, 12: SDR base at L <= 1 and at ratio 1, continuous at 1, flat at 1+, >= 0.9 of the room at W = 6, monotonic, never past the peak (watched fail on the old curve) display_hdr_checks.cpp: the same for vkdisplay::headroomWeight / mapScenePeak (watched fail) clean_tonemap.frag normal and straight-alpha: glslc + spirv-val; world pack regenerated, budgets pass clang-cl /Zs: llvulkancontext.cpp, vk_display_encode.cpp (vk_display_hdr.h): no errors or warnings In world (after a build): sun or a bright lamp in HDR, the encode probe's max reaches >= 0.8 x peak; a sky gradient shows no ring at diffuse white Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>With HDR display output on, translucent UI came out lighter than in SDR (the Contacts rows grey instead of navy): every UI fragment was turned into linear nits before the fixed-function blend, so the UI blended in linear light, while the skin's colours are made for blending in sRGB-encoded space. The UI, the world overlays and the in-world text now draw with their SDR pipelines into a transparent RGBA8 UI layer that exists only while HDR output is on (one image, ~15 MB at 2560x1440). The display encode composites it over the world in the same pass, in the sRGB curve continued above white, at paper white: over anything at or below white the result is what SDR shows, and over a highlight a translucent edge stays smooth instead of clipping. The UI's linear-nits specialization and its paper-white push are gone. The layer's coverage is the same src.a + dst.a (1 - src.a) the SDR pipelines always accumulated, so alpha, premultiplied and darkening draws compose exactly; additive draws add colour without hiding the world, and draws with blending off cover it fully. HDR snapshots that include the HUD or the UI (live and one-shot) now read the world with the layer composited, written by the encode into an FP16 capture image made on the first such shot. Until HUD attachments and selection outlines move into the layer too, in-world text draws over HUDs in HDR. Test-Plan: hdr_ui_layer_contract.py (new): UI target and format under HDR, layer allocation / budget / lifetime, the encode's second binding and composite, no linear-nits UI pipeline left, and a blend simulation of the production alpha rule: over, premultiplied, additive, colour-weighted additive, darken and replace, two draws deep over sRGB-encoded world 0..1, layer + composite == SDR (watched fail before; each rule mutated makes it fail) hdr_display_units_checks.py: encodeExt / decodeExt round trip and monotonic on [0, 12], the sRGB curve up to white; displayComposeUi == SDR's UI-over-world within 0.5/255 for world <= white, empty layer passes the world through, continuous at white, monotonic, no clip over highlights Updated contracts: hdr_live_output_contract, hdr_sdr_capture_contract, display_calibration_limit_source_checks clean_display_output.frag, ui.frag, ui_occl.frag, ui_atest.frag: glslc + spirv-val; world and UI packs regenerated clang-cl /Zs: vk_display_frame.cpp, vk_display_encode.cpp, vk_pipeline_ui.cpp, llrendervk.cpp, llvulkancontext.cpp, llcleanrenderervk.cpp: no errors, no warnings in the changed files HDR suite (11 checks): pass In world (after a build, HDR on, against SDR): Contacts rows navy, translucent windows and text as in SDR; a snapshot with "Show interface" includes the UI Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>With HDR display output on, HUD attachments and selection outlines were converted to linear nits in their fragment shaders and blended onto the world image in linear light, so translucent HUD faces and the dimmed hidden edges came out lighter than in SDR, the same problem the UI had. They now draw with their SDR shaders into the HDR UI layer, in today's order (world, selection outlines, overlays and in-world text, HUDs, UI), and the display encode composites them with the rest of the layer. The opaque and masked HUD faces cover the world under them whatever their texture alpha; alpha HUD faces and outlines blend their coverage as the UI does. The wireframe view keeps its cleared world and physics shapes in the world image and draws its edges in the layer. The rez variants of the HUD pipelines follow the same format and specialization. The HUD pipelines' display recipes were registered before their depth and blend state were set, so a live switch between SDR and HDR rebuilt the alpha HUD pipeline without blending and with depth writes; each is now registered with the state it is built with. An offscreen render (snapshot without the UI through the offscreen route) has no UI layer: in HDR it leaves the HUDs and outlines out and logs it once; the default live snapshot keeps them. Test-Plan: hdr_ui_layer_contract.py: no linear-nits spec left on HUD / selection / rez pipelines; their recipes are UI-layer targets (RGBA8 in HDR, at init, in the mode-switch bank and for rez); opaque / mask HUD recipes force coverage 1, the alpha HUD recipe blends, each registered with its built state; blended draws accumulate ONE / ONE_MINUS_SRC_ALPHA coverage; the HUD pass and selection lines target the layer the context hands over after clearing it (watched fail first) clean_draw.frag / clean_select.frag: world pack regenerated (446 + 18 variants), push and shared-memory budgets pass clang-cl /Zs: llcleanrenderervk.cpp, llvulkancontext.cpp, llcleansnapmattevk.cpp, llcleanseloutlinevk.cpp, llfxruntimevk.cpp: no errors, no warnings in the changed files HDR suite (11 checks): pass In world (after a build, HDR on, against SDR): a worn HUD, a selection outline (edit an object) and the wireframe view match SDR; HUDs stay right after switching HDR off and on Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>The first switch to the other output mode in a session took about 50 s before it applied: the switch rebuilds the 17 output pipelines from their recipes, and the three HUD pipelines compiled the full forward world fragment shader (about 670k SPIR-V words), 45 s each on a cold driver cache, with the other 14 cache hits or under 0.2 s. They now use the placeholder-tier module (CLEAN_BASIC, about 50k words). A HUD fragment leaves shade() before the two tiers differ, so HUD output is unchanged. A switch rebuilds from the registered recipes, so a recipe has to hold the state its pipeline is built with. The tonemap's recipe was registered before its view count was set, and its one-view stereo twin had no recipe, so it kept the old mode's format; both are now registered as built. The opaque HUD faces' coverage is set as the last write, after the rez fade, which blending-off SDR ignores too. The world renderer's UI layer view is cleared after each frame's world pass, so a render after a resize or HDR-off never sees a freed layer. Test-Plan: display_recipe_state_checks.py (new): every registerDisplayPipeline recipe is built with the state it was registered with (fragSpec excepted where the switch sets it), and the stereo tonemap twin has a recipe (watched fail on the tonemap's view count) hdr_ui_layer_contract.py: HUD recipes build from kWorldFragBasicSpv; the CLEAN_BASIC sites in clean_draw.frag are the known five and the HUD returns from shade() before them; the HUD coverage is forced after the rez fade; UI batch keys include format and layer; the world's UI layer view is per frame (watched fail first) World pack regenerated, budgets pass; clang-cl /Zs llcleanrenderervk.cpp, llvulkancontext.cpp clean HDR suite (12 checks): pass In world (after a build): switch RenderVKDisplayHDR off and on; the first switch applies within a few seconds (log: "output_pipeline_wall_ms"), HUD and UI look the same before and after Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>With HDR output on, an SDR snapshot was converted from the HDR display image: everything above paper white was clipped to white (the cinema sign lost its detail), where SDR mode rolls highlights off with its tone curve. Offscreen-route shots also lost their HUD attachments and selection outlines, since those now draw into the UI layer. A live frame that an SDR snapshot samples (a live snapshot for an SDR image, or a presented-frame capture) now runs the world through the SDR post chain: tone curve, glow, AA, final effects, dither and FX, as offscreen shots always have. The display encode shows that frame from the SDR world at paper white. The shot's composite with the UI layer is written back SDR-encoded, so every capture point reads exactly what SDR mode saves. FX permutations for such frames compile at shot time, as offscreen shots' do. Shots that also save the HDR file keep the HDR frame. The offscreen route already renders the SDR chain. A shot that includes the HUD now draws HUDs and selection outlines into its own transient UI layer, composited into the shot's source the same way. Test-Plan: hdr_snapshot_source_checks.py: the SDR snapshot frame (renderer flag, context decision before recordWorld and reset after the last capture point, encode shows the SDR world, composite writes it SDR-encoded, live snapshot reads FP16 SDR-encoded, HDR-file shots excluded), FX sync compile, the offscreen HUD layer and its composite (watched fail first) hdr_display_units_checks.py: displaySdrWorldNits round-trips to the SDR value, and the shot composite over it equals SDR's UI-over-world within 0.5/255 at paper white 80, 203, 480 clean_display_output.frag, clean_snap_live.comp: world pack regenerated, budgets pass clang-cl /Zs: llvulkancontext, llcleanrenderervk, llcleansnapshotvk, llfxruntimevk, vk_display_frame, vk_display_encode: clean HDR suite (13 checks) and snapshot_readback_checks.cpp: pass In world (after a build, HDR on): snapshot a bright scene (the cinema sign) to disk in HDR and in SDR; the files match, highlights keep their detail. Repeat with "Show interface" and with "Show HUD", and once with RenderSnapshotOffscreen on and a HUD worn: the HUD is in the file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>The paper-white setting's description still said the HUD and UI sRGB values are linearized before scaling, and the HDR status notes listed the single linear-nits composition as the design. Both now describe the UI layer: drawn as in SDR, composited over the world at paper white. Test-Plan: settings.xml parses; test_hdr_settings_split.py passes HDR suite (13 checks): pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>The final review found three problems with HDR on: - The classic selection style (the default) built its pipelines in the world image's FP16 format but now draws inside the RGBA8 UI layer. That attachment format mismatch is undefined behaviour every frame something is selected in Build mode. - The outline style still drew its edges into the world image, so they came out near-black, and the HUD outline sat under the HUD faces. - After the first HDR snapshot that included the UI or HUD, its FP16 capture image stayed allocated until HDR was turned off. Both selection styles now build in the layer's format and draw into the layer, the HUD outline over the HUD as on the swapchain. The capture image is freed 240 frames after the last shot; acquire() has waited every slot that used it by then. The encoder's descriptor cache compared view handles. An offscreen shot's transient views and the images of a new frame (a resize) die, and their handle values can come back for new views, so both now invalidate the cache and the next record rewrites the sets. Test-Plan: hdr_ui_layer_contract.py: classic selection pipes and the outline edge pass build in uiLayerFormat() and draw into the layer (world and HUD); the encoder cache is invalidated after the offscreen composite and on a new frame; the capture image is released after kCaptureIdleFrames (watched fail first) clang-cl /Zs: llcleanseloutlinevk, llcleanrenderervk, llvulkancontext, vk_display_frame, vk_display_encode: clean HDR suite (13 checks): pass In world (after a build, HDR on): Build mode, select a prim, a mesh, an avatar and a HUD prim in each RenderVKSelectionStyle (0 classic, 1 outline); outlines look as in SDR and the HUD outline draws over the HUD Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>With HDR display output on, Show Physics Shape When Editing drew its cost-coloured shapes into the world image, which holds linear light at display scale in HDR. The flat SDR colours went in unconverted, so the blue to red ramp came out dark and washed out. The shapes now draw with the selection lines into the HDR UI layer, in the same order as in SDR (wireframe edges, physics shapes, selection lines) and against the display depth as before. Their pipelines are built in the layer's format and blend colour over with coverage ONE / ONE_MINUS_SRC_ALPHA, as the outlines do, so the see-through parts and fills composite as in SDR. They rebuild when the target format changes on a live SDR <-> HDR switch and are never bound in the other format meanwhile. In HDR the world scope only clears for the wireframe view. An offscreen render without a layer leaves them out, as it does the outlines, and the existing log line says so. SDR is unchanged. Test-Plan: hdr_ui_layer_contract.py: physics-shape pipelines build in uiLayerFormat() and rebuild when it changes, standard coverage blend, not drawn while built for another format; in HDR recordPhysShapes is recorded in the layer scope between the wireframe edges and the selection lines, and the world scope no longer draws them (watched fail first) HDR suite (12 checks): pass clang-cl /Zs: llcleanphysshapevk.cpp, llcleanrenderervk.cpp: no errors or warnings In world (after a build, HDR on): Build mode, select an object with Show Physics Shape When Editing on; the colours match SDR and the see-through parts look as in SDR Switch HDR off and on live: the shapes stay right in both modes Wireframe view with HDR on: the shapes draw over the cleared world under the edges Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>With HDR display output the world image between the tonemap and the final pass holds relative linear light (1 = paper white), but two passes there still worked as if it were SDR-encoded: - The RLVa @setsphere Blend mode mixed the frame toward its sRGB colour in linear light, so tints came out lighter and less saturated and a fading blindfold edge looked weaker than in SDR. The sphere is now a display recipe with the linear specialization (rebuilt by the live SDR/HDR switch) and a per-frame flag; with HDR it mixes in the sRGB curve continued past white (the space the display encode composites the UI in) and decodes back. Blur, chromatic aberration and pixelate are unchanged. - The Lightbox look effects (vignette, aim-down-sights vignette, CVD compensation, film grain, dither, CVD/false-colour preview) ran on linear values: the vignette darkened less and the grain sat differently across the tones. With HDR they now run on the frame encoded at paper white and decoded after, only while one of them is on. The wipe seam is unchanged. Test-Plan: hdr_display_units_checks.py: new sphereChecks (HDR Blend equals SDR for colours, strengths and values; highlights untouched at strength 0, no brightening toward a darker colour) and finalLookChecks (HDR look equals SDR at 80/203/480 nits white; SDR snapshot frames stay SDR; nothing converted with all effects off). Failed before (sphereBlend / finalLook missing), pass after. HDR suite (scratchpad hdr-1451-suite.sh): 13 passed. glslc --target-env=vulkan1.3 of clean_rlv_sphere.frag and clean_final.frag; sphere push block 112 B (budget 128). clang-cl /Zs with the Windows flags: llcleanrenderervk.cpp. In world, RenderVKDisplayHDR on/off: RLV @setsphere mode 0 with a grey and a red colour over a 0..1 value ramp -- same tint and same gradient band; RenderVignetteAmount 0.5 and RenderFilmGrainAmount 0.5 -- same corner darkening and grain; CVD compensation on -- same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>The ray-tracing debug views (RenderVKDebugView 13/14/15/18/30) drew their SDR false colour into the world image after the final pass, which with HDR holds absolute nits: the view came out nearly black. Their pipeline was also built once, at RT init, in the swapchain format, so after a live SDR-to-HDR switch it was bound against the RGBA16F target in a different format. They now draw into the UI layer, like the selection lines that follow them: the views write alpha 1 everywhere, so the layer hides the world under them exactly as the SDR overwrite did, and markers drawn after them still sit on top. The pipeline follows the layer's format and is rebuilt (the old one retired behind the frames in flight) when a switch changes it. The rule is now written next to the late-pass order in recordWorld and checked by hdr_late_pass_contract.py: every pipeline built in the world format carries an "hdr-late:" annotation (a display recipe with the linear specialization, an SDR twin bound only while the output is not linear, or an allowed exception with its reason), and every use of the world image after the tonemap is a reviewed one. The known HDR differences of display-point FX presets and the classic glow are noted at their call sites. Test-Plan: hdr_late_pass_contract.py (new): failed before on the RT debug pipeline's unannotated world format, the RT debug draw into colorView, the sphere not being a linear recipe and the Lightbox structure; passes after (11 world-format uses classified). HDR suite (scratchpad hdr-1451-suite.sh, now with the new contract): 13 passed. clang-cl /Zs with the Windows flags: llcleanrenderervk.cpp, llcleanrtvk.cpp, llcleansnapmattevk.cpp. In world with ray tracing on, RenderVKDebugView 13, 14 and 18: with RenderVKDisplayHDR on the views look as with it off (not near black); toggle HDR off and on again while a view is shown -- it keeps drawing, no validation error in the log; with a selection, its lines still show over the view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.