RT ray sanity guard may be compiled away: isnan/isinf without SignedZeroInfNanPreserve #1096

New issue

Open

opened 2026-10-26 02:14:27 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-26 02:14:27 +00:00

Owner

Found while bisecting a GPU hang (2026-10-25): a source-identical-value change to rtHitLocalLights (690930d47c, now reverted and re-landed only in the M2 REQ kernels as b72b5a6a92) made the NON-queue DeferRtSplitLight hang on an RTX 5090 (TDR, no page fault, stuck inside the kernel that traces shadow rays inline). The change altered no value on any path; it only changed the driver's codegen (213 -> 211 registers).

The only open-ended loops in that kernel are hardware traversals (while (rayQueryProceedEXT)). A traversal that never returns is the #1049 failure: a NaN/Inf ray reaching rayQueryInitializeEXT. Our guard rtRaySane uses isnan / isinf, but Vulkan lets the driver assume floats are never NaN/Inf unless the shader declares SignedZeroInfNanPreserve, which none of our shaders do. Whether the driver folds those checks away can depend on surrounding code -- so a codegen shift can silently remove the guard and let an invalid ray through.

Fix (A/B on the owner's box): test finiteness on the bit pattern in rtRaySane (and any other NaN/Inf guard in front of ray queries / loop bounds): (floatBitsToUint(x) & 0x7F800000u) != 0x7F800000u, which the compiler cannot fold. Then re-test the original 690930d47c shape in DeferRtSplitLight to confirm it no longer hangs. Related: #1049.

Found while bisecting a GPU hang (2026-10-25): a source-identical-value change to `rtHitLocalLights` (690930d47c, now reverted and re-landed only in the M2 REQ kernels as b72b5a6a92) made the NON-queue `DeferRtSplitLight` hang on an RTX 5090 (TDR, no page fault, stuck inside the kernel that traces shadow rays inline). The change altered no value on any path; it only changed the driver's codegen (213 -> 211 registers). The only open-ended loops in that kernel are hardware traversals (`while (rayQueryProceedEXT)`). A traversal that never returns is the #1049 failure: a NaN/Inf ray reaching `rayQueryInitializeEXT`. Our guard `rtRaySane` uses `isnan` / `isinf`, but **Vulkan lets the driver assume floats are never NaN/Inf unless the shader declares `SignedZeroInfNanPreserve`**, which none of our shaders do. Whether the driver folds those checks away can depend on surrounding code -- so a codegen shift can silently remove the guard and let an invalid ray through. Fix (A/B on the owner's box): test finiteness on the bit pattern in `rtRaySane` (and any other NaN/Inf guard in front of ray queries / loop bounds): `(floatBitsToUint(x) & 0x7F800000u) != 0x7F800000u`, which the compiler cannot fold. Then re-test the original 690930d47c shape in DeferRtSplitLight to confirm it no longer hangs. Related: #1049.

sky_line added the

rendering

bug

labels

2026-10-26 02:14:27 +00:00

sky_line added this to the Slipstream project

2026-10-26 02:14:27 +00:00

sky_line self-assigned this

2026-10-26 02:58:28 +00:00