Combine Ray Tracing Denoising and Super-resolution

Posted on Edited on Views: Word count in article: 4.5k Reading time ≈ 4 mins.

1. Why combine denoising and super-resolution?

Rendering at low resolution with very few samples per pixel and then jointly denoising and upscaling can deliver near high‑resolution path‑traced quality at a fraction of the cost. The design space spans: ordering (denoise→SR vs SR→denoise), temporal reuse, motion/geometry feature usage, and neural vs analytic filters.

2. Publicly available papers and resources

  • Temporally Stable Real-Time Joint Neural Denoising and Supersampling (HPG 2022)
    • Joint network for denoising + supersampling with temporal stability; processes low‑res inputs to high‑res outputs.
    • Link: momentsingraphics.de (HPG 2022)
  • End-to-End Adaptive Monte Carlo Denoising and Super-Resolution (SRD) (2021)
    • Two‑stage deep network with deformable conv; takes low‑res noisy path‑traced inputs and outputs high‑res, denoised images.
    • Link: arXiv:2108.06915
  • Auxiliary Features‑Guided Super‑Resolution for Monte Carlo Rendering (2023)
    • Cross‑modality Transformer uses high‑res G‑buffers to guide SR of low‑res noisy render, improving fine detail.
    • Link: arXiv:2310.13235
  • Neural Supersampling and Denoising for Real‑Time Path Tracing (AMD GPUOpen)
    • Single U‑Net‑style model supports denoise‑only and denoise+upscale, targeting real‑time 1 spp inputs.
    • Link: AMD GPUOpen article
  • Denoising‑Aware Adaptive Sampling for Monte Carlo Ray Tracing
    • Couples deep denoising with adaptive sampling via variance of network outputs to reduce equal‑time error.
    • Link: PDF

3. Practical integration notes

  • Ordering: For path‑traced inputs at sub‑pixel spp, denoise at render resolution first, then upscale temporally; joint models can learn both in one pass.
  • Inputs: Feed temporally reprojected history, motion vectors, per‑pixel hit distance/roughness/normal/albedo; keep them in sync across SR scale.
  • Stability: Use reactive masks/exposure, and clamp history in disocclusions; joint models often outperform TAA+denoise cascades.
  • Training data: Pair low‑res, low‑spp with high‑res, high‑spp references; randomize camera/lighting/materials and include motion.
  • Spatiotemporal Variance‑Guided Filtering (SVGF, 2017) — classic analytic baseline for path tracing denoise.
  • Intel Open Image Denoise (OIDN) — CPU neural denoiser for offline/interactive use.
  • Temporal upscalers (DLSS/XeSS/FSR2+) — industry SR baselines; DLSS 3.5 "Ray Reconstruction" integrates a neural ray‑tracing denoiser with SR in practice.

If you want, I can add summaries/figures per paper and a small experimental checklist (datasets, metrics, and ablations) tailored to your current pipeline.

5. Key points per paper (joint denoising + super‑resolution)

  • Temporally Stable Real‑Time Joint Neural Denoising and Supersampling (HPG 2022)
    • Joint task in a single network: denoise and upscale with temporal feedback for stability.
    • Processes low‑res inputs; outputs high‑res, temporally stable results; avoids TAA ghosting.
    • Shares a low‑precision feature extractor with higher‑precision filtering heads to balance cost/quality.
    • Uses motion vectors, normals, albedo, and history buffers for reprojected guidance.
    • Shows better quality than separate denoiser + TAA/neural SR cascades at similar budgets.
  • End‑to‑End Adaptive Monte Carlo Denoising and Super‑Resolution (SRD, 2021)
    • Two‑stage cascade: denoising and SR with shared components; deformable conv handles differing receptive fields.
    • Trains on low‑res, low‑spp inputs to reconstruct high‑res, clean outputs; improves detail retention.
    • End‑to‑end objective encourages the denoiser to be SR‑aware (reduced oversmoothing before upscaling).
    • Demonstrates superior PSNR/SSIM/LPIPS vs. sequential pipelines on MC rendering datasets.
  • Auxiliary Features‑Guided Super‑Resolution for Monte Carlo Rendering (2023)
    • Treats SR with noisy low‑res color but leverages high‑res auxiliary buffers (normals, depth, albedo) as guidance.
    • Cross‑modality Transformer fuses modalities; residual Swin groups preserve fine geometric/texture detail.
    • Effectively performs denoising during SR by conditioning on noise‑free high‑res guides.
    • Outperforms prior SR and denoising‑then‑SR baselines on MC scenes, especially thin features.
  • Neural Supersampling and Denoising for Real‑Time Path Tracing (AMD GPUOpen)
    • Single U‑Net‑style model supports both denoise‑only and denoise+upscale modes for real‑time 1‑spp inputs.
    • Multi‑branch, multi‑scale design ingests noisy radiance and aliased but noise‑free guides (normals, depth, motion).
    • Trained for temporal stability and detail recovery at display resolution > render resolution (e.g., 4K from 1080p).
    • Practical engineering notes for integration (history usage, reactive masks, disocclusion handling).
  • Denoising‑Aware Adaptive Sampling for Monte Carlo Ray Tracing
    • Not a joint SR+denoise network, but complements them: adaptively allocates samples where the denoiser (and SR) are most uncertain.
    • Variance of network outputs guides per‑iteration sampling; improves equal‑time error vs. uniform sampling.
    • Pairs well with joint pipelines by reducing artifacts in high‑frequency/reflective regions that SR struggles to reconstruct.