Song's Home
Volumetric clouds rendered in real time.

Real-Time Sky and Volumetric Cloud Rendering

For the last few months, I've been working on real-time volumetric clouds and sky atmosphere rendering for my internship at  D5 Render. I was fortunate to spend a relatively long stretch focused on real-time volume rendering — a topic I have always been interested in. To wrap things up, I decided to write a short summary of a systematic framework for real-time cloud and sky atmosphere rendering. This article does not aim to a deep dive into the technical details since there're a lot of papers and talks online, but as a collection of those resources I found very helpful and want to share. Implementation details are omitted; the analysis and summary below is my own.

Real-time volume rendering differs from an offline path tracer, and focuses on different concerns. For an unbiased offline path tracer, one usually resorts to free-flight PDF sampling, but this may not be practical in a real-time renderer. First, the stochastic workload of each path can cause thread divergence: some paths terminate only after many bounces, so the other threads in the warp must wait for them, badly hurting performance. In addition, a large amount of computation may be spent on null-scattering events, which carry no radiance contribution while still costing a density lookup. Finally, coherence is poor, since sampling locations within a warp vary widely, leading to a low cache hit rate.

Ray marching greatly reduces these drawbacks, at the cost of less precision and more bias. It also normally traces only first-order light (a single bounce within the medium), so the result appears darker because of the lost energy, and extra compensation has to be added for multiscattering. Clouds and the sky atmosphere handle this differently. Clouds usually use octaves to add the lost radiance back, while the atmosphere uses lookup tables (LUTs), because it is spatially low-frequency.

A general framework for a real-time cloud and sky atmosphere rendering system

Performance

The clouds together with the sky atmosphere run at roughly 210 to 280 FPS at 1080p on an NVIDIA 4060, depending on the cloud setup. The range comes from sparsity: when the clouds are sparse, most threads exit early and save time. Below is a video showing the frame rate along with the authoring process, implemented in Unity for simplicity.

Screen recording of authoring the clouds on Unity for a clean testing environment

Another interesting thing I found was that domain warping — warping the UV input of the cloud 3D noise with a separate distortion noise texture — hurt performance badly, because the warping produces divergent UVs. This might not be an issue for ordinary rendering algorithms since the texture sampling happens once. But under ray marching, a texture fetch may be executed M × N times for a single ray, where M is the number of marching steps along the view ray and N the number along the shadow ray, so the overhead accumulates quickly. In practice, I found distorting the 3D noise accounts for roughly 73% of the overhead of each texture evaluation, owing to the higher cache miss rate, draging the performance from 7.5ms per frame to 24.4. One workaround might be mip-maps. Another is to pre-bake the distortion into the 3D noise, though that also means the distortion map can no longer move at a different velocity from the cloud noise.

Gallery

Cumulus clouds rendered in daylight.
Cumulus, daylight.
Cumulus clouds in daylight, a second viewpoint.
Cumulus, daylight - a second viewpoint.
Mixed cumulus and stratocumulus layers in daylight.
Mixed cumulus and stratocumulus layers, daylight.
A stratocumulus deck rendered in daylight.
A stratocumulus deck, daylight.
Altocumulus clouds lit at sunset.
Altocumulus at sunset.
High cirrus cloud rendered at sunset.
High cirrus at sunset.

Some notes and thoughts

Real-time volumetric cloud and sky atmosphere rendering is a relatively large, heavy system. Even though it produces an acceptable visual result, managing something built from so many artifacts, approximations and tricks is neither intuitive nor productive. Bringing volume path tracing into game engines is probably an interesting direction. I know some projects are already using it, but the trade-offs and the performance are not clear to me, and few games seem to use path tracing for clouds. I would be very excited to implement volume path tracing together with DLSS 5 in real time and try it out.

Another idea that came to mind was using 4D Gaussian splatting for volumetric clouds and atmosphere. It seems like a good fit for distant views, where the camera never travels through the clouds and the sky and clouds are really just an advanced environment map.

Please email me if I have made a mistake in this article, or if you would like to discuss it. Thanks for reading. Below is a video I took on a flight; it clearly shows different cloud types — cumulus at the bottom left, and altostratus on the right.

Cloud filmed from a plane window.

References