Implementing hybrid ray tracing in a rasterized game engine

For the past month, many of you have been asking on social media or in the comments section of our blog for a video recording of our hybrid rendering demo. Since some of our readers have missed our GDC 2014 sessions or didn’t get a chance to drop by and see our PowerVR Ray Tracing demos running at our booth, I have asked my colleagues in the PowerVR Ray Tracing group to put together short clips that present the work they’ve done to highlight some use cases for ray tracing.

The first example we’ll be analysing today is the hybrid rendering demo. Before we begin, there is a caveat: please keep mind that this demo is running on a previous generation of the PowerVR Ray Tracing technology. The PowerVR GR6500 ray tracing GPU we’ve announced at GDC 2014 is designed to be integrated in a System-on-Chip and therefore can provide much better raw performance (a higher ray budget) and tighter/faster memory coupling while consuming lower power. Therefore the performance, framerate and resolution in the video isn’t representative of the PowerVR GR6500 or any other Wizard GPU.

The hybrid rendering demo running on our PowerVR Ray Tracing IP, as shown at GDC 2014

For this ray tracing demo, we’ve started with an experimental version of the Unity game engine and implemented OpenRL support which enables us to use the G-Buffer from the rasterizer to initialize rays (see my recent Gamasutra article for more details). Then, the results of the ray tracing are used by Unity’s final lighting pass and fed right back into the game engine.

Ray tracing alongside rasterization opens up many possibilities and enables effects that are either difficult or impossible to do without artefacts using raster techniques alone.

Hard and soft shadows

The first scene presents hard shadows. We are using the properties of the G-Buffer (world-space positions, normal and material IDs) to cast shadow rays. Zooming in on the tree in this scene reveals how every leaf is casting and receiving a very accurate shadow.

Hybrid ray tracing in a rasterized game engine - hard shadows (2)Ray tracing can be used to generate very accurate hard shadows

The second example in this scene introduces a similar situation where the complex geometry of the bridge will cause resolution issues when casting shadows in a traditional game engine. However, ray tracing doesn’t have any resolution problems because the rays are cast for each screen pixel.  Developers do not have to worry about selecting the appropriate shadow map resolutions for various positions on the game level, and rigorously testing the game looking for artefacts. Ray traced shadows look realistic from afar (check out the beginning of the scene) and maintain fidelity all the way to the end. We can go as close to the wall as we want and we will never run out of shadow map resolution.

Hybrid ray tracing in a rasterized game engine - hard shadows (3)Ray traced shadows will not have resolution issues

The next scene focuses on soft shadows. While hard shadows simulate sunlight on a clear day at noon, soft shadows appear when the light source has area (e.g. a lamp with a shade) or there is a medium to scatter the light between the light source and the scene (e.g. clouds). In our hybrid rendering example, by simply adding some random noise to the shadow ray directions, we can obtain the appearance of a cloudier day. This technique simulates the effect where the penumbra size is dependent on the distance between the occlude and the object receiving the shadow. In the video example, the head of the lamp has a very soft shadow while the base retains its hard edges, just like in real life. This would be very difficult to achieve using conventional techniques.

Additionally, implementing ray traced shadows in game engines enables you to have many shadows casting lights because sampling decisions are made on a pixel-by-pixel basis. In the scene above, half of the pixels don’t need to sample the headlights of the car because those lights can be rejected by cheap cone-test. There is no need for a separate pass to handle each dynamic shadow casting light.

Reflections, refractions and transparency

Another use case for ray tracing is reflections, which can be handled in a similar fashion to shadows. Instead of directing the ray towards a known light source, we send the ray in the direction calculated by performing the mathematical reflect function between the vector from the viewpoint and the normal vector at the surface. Ray traced reflections have several advantages over raster techniques like reflection maps.  Not only do they reflect dynamic objects without needing separate rendering passes, nearby objects and self-reflections are handled perfectly.

Hybrid ray tracing in a rasterized game engine - reflectionWhen using ray tracing in your game engine, reflections look crisp clear

Additionally, transparency is also handled by casting rays. In our example, the windshield is tinting the rays and continuing them on to provide order-independent transparency. If you look closely at the car, you can notice we have implemented both transparency and reflections techniques for the windshield glass material; this is easy to do and produces a realistic effect where the trees are visible in the glass. The proportion of the blending between the reflection and the transparency takes into account the Fresnel term calculated from the viewing angle, and produces a convincing model of real life.

Another example of ray traced transparency is the fence. It combines transparent objects with shadows to achieve an interesting visual effect. The fence is made of quads with an alpha texture applied to create the wire mesh. Because the rays respect the texture’s mip level, the shadow just looks like a faint tint from a reasonable distance. Once we get closer, the texture is starting to come out and the shadows gradually become more detailed, right up to the resolution limit of the wire mesh texture itself.

Hybrid ray tracing in a rasterized game engine - hard shadowsIn ray tracing, you can combine transparent objects with shadows

A texture can also be used to tint a ray traced shadow for a quick and easy way to get perspective textures. In our example, this effect is noticeable on the pavement as the light cast from inside the diner is combined with the window decal to produce a coloured shadow.

Finally, this wouldn’t be a ray tracing demo if we didn’t show some refraction. The open sign visible through the neck of the bottle or the bottle’s label is visible through the mug. We apply the refraction equation to bend the ray based on the difference of the index of refraction between glass and air, and then apply it again in reverse when the ray exits the material.

Hybrid ray tracing in a rasterized game engine - refractionRefraction and glossy reflections

Glossy reflections also work with this hybrid method. The normal map on this table means that the reflection rays are scattered a little bit. We can faintly see the reflection of the open sign and the beer label in the table, although it is too fuzzy to make it out clearly.

Final words

In conclusion, we’ve seen how adding just a few rays-per-pixel to a rasterized game engine can add a lot of effects that would be difficult or impossible to do with rasterization alone. Ray tracing can do a better job approximating the physical light transport phenomena, and using ray traced effects can avoid the artefacts of complicated raster techniques and often work out to be even be more efficient.

Thank you for watching! I hope this article has come a little way towards quenching your thirst for ray tracing demos. If you have any suggestions or questions, please don’t hesitate to leave a comment in the box below.

Come back to our blog for a second article where we focus on another example where PowerVR Ray Tracing is used inside the Unity 5 lightmap editor and make sure you follow us on Twitter (@ImaginationPR, @PowerVRInsider) for the latest news and announcements on PowerVR and ray tracing.

, , , , , , , , ,

  • Pingback: It won't be long before mobile gaming looks this beautiful | allcom.se

  • Pingback: It won’t be long before mobile gaming looks this beautiful – Birmingham Life, Entertainment, Life & Leisure, News, jobs, Going Out

  • Pingback: It won’t be long before mobile gaming looks this beautiful | mobilemakers.org

  • Pingback: It won’t be long before mobile gaming looks this beautiful | PJ Tec - Online Tech Services

  • Pingback: It won’t be long before mobile gaming looks this beautiful | 247 Bulletin News

  • Pingback: It won’t be long before mobile gaming looks this beautiful | Best News Source in World

  • Pingback: It won’t be long before mobile gaming looks this beautiful | Compare That App

  • Pingback: It won't be long before mobile gaming looks this beautiful | Computer and Technology NewsComputer and Technology News

  • Pingback: It won't be long before mobile gaming looks this beautiful - Get Gaming Now | Get Gaming Now

  • Pingback: It won’t be long before mobile gaming looks this beautiful | My Blog / Website

  • Pingback: It won’t be long before mobile gaming looks this beautiful | JKGC

  • Pingback: Jogos em dispositivos móveis serão incríveis! | Boa Informação

  • Pingback: It won’t be long before mobile gaming looks this beautiful |

  • Pingback: It won't be long before mobile gaming looks this beautiful | Techy News Today

  • TrollinIsBad

    Tracing shadow rays would still need the complete scene as BVH/KD-tree in the video memory. As the g-buffer only contains information about visible objects and not the occluding geometry from light’s perspective. This is one of the limitations of ray-tracing. Although I agree the shadow quality would be very accurate. Modern games already use some kind of ray-tracing for screen space reflections using just the g-buffer.

  • Sean Lumly

    The PowerVR tracer is quite innovative in this regard: It uses a single object that works with OpenGL (similar to a display list, iirc) but that holds the entire scene data (incl. shaders) for the ray tracer. In this way, the rasterizer draws the scene into a GBuffer, and the tracer can then do tracing over each of those framebuffer pixels with complete scene data at hand for better reflections, transparencies, shadows, etc. Submission of objects to this unified data structure makes the marriage of rasterizing and ray tracing quite straight forward.*

    Tracing should also save a lot of bandwidth, not requiring a scene to be rendered from multiple perspectives.

    >Modern games already use some kind of ray-tracing for screen space reflections using just the g-buffer
    An important thing to note is this: this is a mobile platform, and despite relatively constrained bandwidth and compute, the on paper tracing performance is nothing short of amazing. While modern desktop/console games may use tracing to get an approximation of light ray intersection from some buffer, cube/sphere-map, etc, it is at best a hack an in no way comparable to the flexibility, accuracy, and simplicity of real ray tracing.

    * corrections appreciated where applicable

  • TrollinIsBad

    Cool. Good to know. It seems like OpenGL and OpenRL work together pretty well in this regard.

    I agree the SS reflections are hacky and this architecture would provide a more simplified way to perform accurate ray-tracing. I would like to see some accurate voxel cone tracing for diffuse global illumination using this.

  • Sean Lumly

    Me too. I’m sure that a great many things can be explored with this architecture, and who knows? I remember reading that this can handle ~5 rays per pixel, future generations at smaller process-nodes would be capable of much more (including tablet specific SoCs with different power constraints), and we may have something that is directly capable of path tracing.

  • dodo

    250W-PC-GPU version hardware please … :D

  • http://withimagination.imgtec.com/index.php/author/alexvoica Alexandru Voica

    I’m working on an article to explain how this scales to the PC/console space.

  • Sean Lumly

    This is a very exciting prospect. I hope that you are also considering GPUs that scale very far beyond the bounds of mobile GPUs. The next generation of consoles could have Imgtec tracing built-in, and fill-out a 400mm^2 die at 14nm!

    Oh, and coupled with a PoP Hybrid Memory Cube, this GPU could enjoy insane memory bandwidth to augment the insane performance. Low-noise path tracing at interactive framerates would certainly be a realistic proposition with this hardware..

    Let me have my dreams! :)

  • Sean Lumly

    Here’s a neat idea that could use ray-tracing to significantly augment a rasterizer:

    Ray Assisted Rasterization

    PROBLEM:
    As scene complexity grows, fragment workload stays constant (ie. TBDR), but the bandwidth and computation cost associated with triangle grows more-or-less proportionally.

    SOLUTION:
    The idea is simple: use the ray-tracer to discover potential intersecting triangles prior to the rasterization step.

    1) In this case, we ray trace over every pixel in a tile, and build a list of candidate triangles according to the ray tracing acceleration structure (eg. BVH).

    2) Once we have traced all pixels, we then take the list of triangles, stream in the appropriate triangle data from memory, transform them and rasterize them with appropriate per-pixel depth info to account for proper draw order.

    3) If there are pixels in the tile that haven’t been rasterized, we can go back to step 1 for those pixels.

    4) Once all pixels in the tile have been rasterized, they are then fragment shaded.

    5) We repeat this for each tile in the target frame.

    BENEFIT:
    This effectively culls triangles based on overdraw. The important point to note is that similar to eliminating hidden-fragment shading calculations, it also eliminates the very need to do any triangle-related work for triangles that are not seen! In other words if the triangle is hidden, it is not even processed. Similarly triangles that are hidden will not read from memory, leading to potentially large bandwidth saving as well.

    The effect on scene complexity and memory bandwidth could be tremendous. Only triangles that will be used in the scene will be streamed from memory into the tracer, and only triangles that will be seen will be calculated. Scene complexity could grow exponentially and be much more memory-capacity bound than bandwidth/performance bound.

    It would also eliminate the need for sorting logic as is the case with the current PowerVR TBDR solution.

    LIMITATIONS:
    Of course, there are limitations. For one, the scene would have to reside in a structure prior to beginning the render. Additionally, handling of transparency in this scenario would not be straight forward and would have to be solved appropriately (eg. multiple tile buffers). Lastly, for sub-pixel triangle sizes, this could be quite expensive.

    CONCLUSION:
    This scheme of Ray Assisted Rasterization has the benefit of saving bandwidth and computation as scene complexity increases.

    I hope that you consider and share this idea with your colleagues, as it is a potentially powerful alternative to the current TBDR scheme of reducing compute complexity and optimizing bandwidth.

  • http://www.ubergizmo.com/hubertnguyen/ Hubert Nguyen (Ubergizmo)

    Screen space reflections is a very fast way to get reflections going, but it is also quite hacky from an imaging point of view. That’s OK for games since people aren’t there to check if reflections are accurate or not as long as the scene is believable. Shadows need something more orthogonal and accurate.

    That said, if you take into account the light fall-off distance, and take into account that not all lights require soft shadows, it is possible to do RT shadows without having the whole scene on chip, although one needs more geometry than what’s visible on screen.

    Global illumination would also benefit from the ability to sample based on close proximity geometry. it would be much more straightforward than using other methods such as cone-tracing or light propagation volumes. Both are proven to be effective, but generate their fair share of corner-cases etc.

    I’m pretty excited by what Imgtec has done, although we’ll have to see if RT gets adopted by NV and AMD.

  • http://www.ubergizmo.com/hubertnguyen/ Hubert Nguyen (Ubergizmo)

    A console version would be bad-ass. Although the next cycle is pretty far away…

  • http://withimagination.imgtec.com/index.php/author/alexvoica Alexandru Voica

    Hi,

    The proposed game console would have a multiprocessor, multicluster PowerVR Wizard GPU that would support the latest APIs and scale beyond 1.2 TFLOPS. More about this soon.

    Regards,
    Alex.

  • http://withimagination.imgtec.com/index.php/author/alexvoica Alexandru Voica

    Not necessarily. There have been several interesting devices that run on Android but use mobile-derived hardware. By upgrading to a multiprocessor, multicluster PowerVR Wizard GPU, they would be able to scale to PlayStation/Xbox levels.

    Regards,
    Alex.

  • http://withimagination.imgtec.com/index.php/author/alexvoica Alexandru Voica

    For those interested in MIPS, PowerVR and ray tracing, please read this interview with GamingBolt:

    http://gamingbolt.com/imagination-techonlogies-interview-photorealistic-gaming-on-mobiles-and-tablets

    A few points from this article are already present there, but I also talk about next-gen game consoles, off-line rendering and cloud gaming and other topics.

    Regards,
    Alex.