This post details the breakdown of how we achieve this result and the tricks discovered along the way to create particle physics simulations:
Above is a showcase of our custom-made interactive application where I am dragging our sand emitter around the screen with my mouse cursor. Sand is emitted and continue to fall, bouncing off objects. We manage to achieve and render interactive frame rates in almost real-time, rendering and simulating 100,000 individual sand grains colliding dynamically with moving and rotating objects, with physics. Our scene has been highly optimized to render within seconds instead of hours, allowing us to freely edit parameters throughout our iterations.
Warning: This post is quite technical until the end where we share how this was achieved:
In making this, the main issues to tackle were:
1) Creating believable sand physics (friction, bounciness, spread)
2) Sand colliding dynamically with other 3D objects that are in motion
3) The sheer quantity of particles on screen simultaneously, and keeping realistic render times
First thoughts & Research
Early talks about whether we should do sand simulation in the group wasn’t very positive.
None of us had done it before, nor are we even aware of what techniques we can use to achieve believable sand.
Sand is a complex substance – it is not a viscous liquid in which we can use fluid simulations. Yet at the same time it does not act as a single mold-able solid like clay.
Research on how to achieve this led to pretty complicated techniques and sometimes involve software beyond our understanding. Even in Maya, we had not gone deep enough to understand how to use Maya’s nParticles, which are our guess at how we would execute it in Maya.
I narrowed my search to see if it was possible to create this in Unity. As it is not primarily a simulation software, there were many challenges and question marks.
Thousands of individual sand grains have to be simulated. And what about collisions? Would our sand interact with our gear geometry in the scene? If it had to be done in two separate software, then compositing them together would also be a challenge – we could do sand overlay on the gears, non-interactively, but we had no idea how that would look. If it is too obvious it’s an overlay of two distinct videos, it won’t look good.
I knew that it was possible to use CPU-based particles in Unity that use an underlying physics engine (PhysX 3.4).
But as much as I wanted to use Unity (as our gears are rendered there), the sheer quantity of particles destroyed any hope of this being possible, or is it?
Many youtube tutorials on how to do real-time sand physics within Unity did not achieve the fantastic results we were hoping for. The sand particles were always too big, and too few.
The truth is, physics calculations consist of plenty of mathematical operations that are expensive to compute in large quantities.
The best simulations went to 20,000 particles, maybe more. But this was still a far cry from believable sand. In other words, we had little luck with tutorials. Bumping it up would exponentially take a toll on simulation times and make it difficult to iterate creatively.
Achieving interactive frame rates was a challenge. But I wanted to see how far we could push it.
I gave all the important gears a Mesh Collider, and enabled collision on particles so they would be added to the physics engine and collide.
I started with a basic particle system that simulates 1,000 particles.
Seeing sand interact with the gears for the first tine is cool. But aside from cool physics, the visuals look nasty. How about 10,000?
Better but not good. I pushed for 100,000 particles. This was a huge quantity and the simulation slowed down tremendously.
I decided to be crazy to the extreme, and bump it up by a factor of 10 again!
-
1,000,000 Particles! The grain sizes were finally fine enough, but at this point each frame was taking one full second to render. It is not too bad, but still extremely sluggish and difficult to experiment with. It was time to see if we could do some optimization.
We are being frame rate-throttled on the CPU, not the GPU, as the mathematical and physics calculations are done on the CPU. Knowing this, all our optimizations are targeted at the CPU-side.
Optimization Tricks
1) Randomizing per-particle properties creates illusion of more sand
The first secret to get fast simulations is to completely NOT simulate sand particles colliding with each other. Instead, we fake it. To help sell the illusion that the particles collide with each other and pile up, we randomize the per-particle properties, indicated in red arrows below.
Our particle collision module settings Dampen affects how much velocity a particle loses in contact with a surface (aka friction).
Bounce affects how much a sand particle retains its vertical potential energy. High values make each grain act like a bouncy ball.
We randomize them for every sand grain so a sand grain can be heavy, or light, or bouncy, or slidy. This creates variation in our sand behaviour.
This randomization across the particles create higher coverage of sand across the screen, and results in a “spreading” effect lets us use less particle numbers to give illusion of more sand.
2) Using primitive colliders is simpler for the physics engine to calculate
Secondly, we use basic collider shapes.
How collider shape affects physics calculations Gears have a very complex shape, but when broken down, it is basically a sphere (circle in 2D). In some cases, using multiple sphere colliders is less taxxing than a single, complicated shape collider.
Sphere colliders are easier to detect collision with, as the physics engine only needs to check the distance between two points in 3D space. This makes spheres the cheapest colliders. As our gears are circles from the camera perspective, we use spheres for our gears’ main body instead of cylinders.
We can use both – objects that require precise collision use mesh, while small objects not prioritized in our composition fall back to primitive colliders, and we can even toggle between the collider types through code in the midst of rendering if we really want full control!
Basic colliders give us up to 5 times faster render times. Now we’re talking speed!
3) Restricting physics to a single plane
Simulation is restricted to 2D space, faster computations We reduce the particle count needed to be simulated by limiting them to a single plane in our 3D space. This way, it is effectively computed in just 2D while being able to interact with our 3D objects. Withour particles overlapping or hidden in the z-axis, it allows us to be as efficient as possible with the particles we emit.
4) Decrease simulation speed
This last trick can double the render speeds but is case specific. By slowing down our simulation speed to 0.5, we effectively only calculate our physics every 2nd frame instead of every frame. Speed wise, this means the CPU performs half as many calculations over the same period of time. However, this can produce undesired effects, such as slow motion.
There was lots of trial and error involved but bit by bit the simulation got faster and was extremely rewarding to watch.
We went from rendering 1 frame per second, needing 1 million particles, to 10 frames a second, needing only 100,000 particles. That is a 10x speed increase in our simulation and render times!
We went from talking about how many seconds we need to render a frame, to how many frames we can render every second. I was personally amazed that we could simulate 100,000 colliding particles at such speeds.
When it became fast enough to render, we aren’t bogged down by render times, so it was time for the fun stuff!
Out of fun, I made the sand glow by giving it emission, and turned off all our scene lights and made the background black.
This created a very interesting composition:
I then wrote the code to make a button that can toggle between both modes in real-time through a single click, as can be seen by the “FX Mode” white button at the top right of the above GIF. This makes it convenient to test compositions while we experiment.
I doubt anyone is interested in the code I wrote (C#), but for the curious:
Function that is executed when FX mode is changed Additionally, I added a slider that progressively trades off performance for quality for our final render, so we can choose how much sand we want and the quality of the simulation results, based on how much time we are willing to wait for it to render. The longer we are willing to wait for longer render times, the higher the sand particle count. Collision parameters are not changed as this keeps the render consistent regardless of the quality used.
Progressively increasing quality increases sand particle count and render times, while inversely decreasing sand grains’ size This is done by writing a script to set our particle sizes and emission count. Again, for anyone who is interested to look at the script to achieve this:
I also wrote another script that reduces the particles used during edit mode, and pumps up the particle count only during our rendering. This way we can iterate even faster with less lag in the scene view during our iterations, and not have to manually remember to increase the final quality before we render.
Gears 3D Modelling Breakdown
This is part 1, the beginning of the entire development process focusing on Concept 2 of our ideation for the Silk Road theme. For subsequent parts, please see:
One of the core visual inspiration of this project is gears. Thus, we started with asset creation for these elements.
GIF showing current stage of render Below is a high resolution video result of our current 3D gears render, 20 seconds rendered at 3840 x 2160px:
https://vimeo.com/user85218205/review/360882605/0bd928edf5
This post details the breakdown in the asset creation pipeline and how rendering of our gears is achieved. This breakdown showcases the entire pipeline:
- Modelling
- Texturing
- Rendering
- Lighting
- Post Processing
- Animation
- Particle Effects
The composition is not confirmed. Should the video not work, this is a high resolution screenshot from the video render.
Modelling the Geometry
Modelling is our first phase to obtain the 3D geometry we need. Modelling each 3D gear asset individually can be a labourous process. But what if it did not have to be this way?
Using Maya 2018, we discovered that it is possible to procedurally generate gears instead of hand-modelling individual ones, leading to huge time savings and procedural generation meant our assets are non-destructive (able to go back to edit easily its parameters).
Using procedurally generated attributes to modify our gear Here is a screenshot presenting attributes of the gears we can edit when creating our gears through this method. It allows for easy experimentation.
Parameters to allow flexible creation of gear polygons With varying parameters, the aim was to produce interesting and contrasting shapes and proportions.
Various types of gears created in Maya 2018 To give a realistic look, bevels were added to prevent the ‘CGI look’ resulting from edges with 90 degree angles. This will allow gears to catch lighting and reflections more realistically when lighting is added later on.
Without bevels (left) vs With bevels (right, highlighted in orange) To make sure our gears look old enough to be from the 19th Century, much detail has to be considered in its texturing. It is easy to make metal look shiny and glossy. But this clean and shiny look is not what we want. Thus, we move to the next phase.
Adding detail through textures
With our models done, we need to make them look convincing by applying 2D textures and set up our material to emulate real-life materials, such as their physical properties (glossiness, smoothness).
For textures, I looked at various metals:
- Galvanized Metal
- Scratched Steel
To give the old, worn-down look, I use:
- Grunge
- Rust
- Surface imperfections
Grunge Albedo Texture, that was later used as a detail mask To achieve a realistic finish, a technique called physically-based rendering is used. The standard surface shader in Unity supports this. The nature behind this idea is that in real life, no surface is perfectly flat and they have to respond to dynamic lighting easily in our CGI pipeline.
Normal maps lets us add micro-surface detail by simulating scratches, a common surface imperfection in metals. Notice that we also vary reflection in certain areas – a metal surface is not equally reflective across its surface.
Rust was applied using a Detail Mask, specifying three channels (Albedo, Smoothness and Normal).
In Photoshop, I took a rust texture and using a soft brush, desaturated areas of the texture where I did not want rust, as well as the rust layer itself, until I was happy with the percentage coverage of the rust on the gears.
Testing of rust amount on material Below is a summary of the gear at different stages of the material set-up process for comparison of start to end.
Gear Texturing and materials step by step breakdown Notice how in step #2 the reflections and captured lighting falling on the gear are a lot more believable.
The detail masks added in steps #3 and #4 also affect the surface reflection, in addition to the albedo (color), as seen in the specular areas.
These are our two material set ups.
With our meshes and textures, we proceeded to try rendering them to see how they look.
Rendering
In addition to quality results, the key to our workflow is as much about being able to iterate quickly throughout our conceptualization process.
Using the power of Real-time Rendering
I did not use Maya or Cinema4D to texture or render the gears in the meantime, as I wanted to adopt a faster rendering solution, observing that the long render times in these softwares limit what we can change in the final weeks, and it can be creatively-limited. Thus, I tapped onto the power of a game engine to do real-time rendering, and also because I am accustomed to using it.
Note: Above is a screenshot, not a video, as it is too large to upload here We could get a 4K video rendered in 20 seconds in the Unity game engine, complete with lighting, reflections, shadows, particle effects, post processing and color grading. What is amazing is that it outputs a video directly with the option of individual image frames for lossless quality. So we can get a final render look and iterate extremely quickly without going through external video editing software, minimizing additional rendering steps. This lets us focus on the creative aspect instead of worrying about rendering time as it takes seconds to render high resolution 3D content instead of hours.
Visualization in Unity editor. Top shows our scene viewport, bottom shows the immediate render output. This additional creativity time let us experiment a lot with different lighting, camera perspectives and screen compositions as seen in pictures below. Plenty of happy accidents came from the results of this which will be talked about in future posts, which will be exciting to share. These would not have been possible in a pre-rendered solution. We are still open to use cinema4d if needed later on in the project.
Exploring compositions and perspectives on the first night of testing:
Lighting & Post Processing
Lighting
The scene itself uses fully real-time and dynamic lights. These lights can be changed on the fly interactively to achieve results we want.
The scene is made up of a directional light as the key light, and a few spot and point lights for fill and rim lighting, as well as strategically placed lights to show off the surface detail of our gears.
Having fun with lighting adjustments (at the top I am adjusting the light, bottom is output render) Using lighting, it is easy to change the mood and tone of the scene globally as well. Above is one of the first few early videos of me playing with the directional lighting.
Post Processing
Post Processing involves several effects, one of which is tonemapping to get a filmic look. We also slightly bump the contrast and apply subtle color grading where needed.
Other subtle effects include ambient occlusion, which darken edges where objects appear to touch, making them look more grounded in reality, as well as bloom to simulate how intense light appear as bright spots through a camera lens.
Depth of field was also used to blur out gears in the background, just to test the composition and bring focus to the main gears.
These effects are added in the 3D engine so we can preview the gears’ look without going it through AE or other compositing softwares for faster iteration.
Reflections
Reflection Probe lights up darker areas of gears Ambient lighting is also used to set-up optimized reflections, by using something called Reflection Probes, as seen above. The reflections light up the darker areas of the gear parts, revealing details that are previously in shadow. A Reflection Probe is an optimized way to capture a 360 cubemap to cast as reflection onto nearby objects. The effect can be toggled off if we prefer the areas in shadow to remain dark.
This technique is cheap and fast to compute, and is highly optimized compared to Maya and C4D which calculate reflections by simulating light rays, known as ray tracing, which is comparatively fairly expensive computation. Our current technique of using reflection probes works well as we do not need super accurate mirror-like sharp reflections.
Animation
This project does not feature animation heavily, but is still essential to the process. To create intricacy and mesmerizing effects, we will play with:
- Speed of rotation of gears
- Direction of gear rotations
- Animating of lights (future post)
- Layering of multiple rotating gears in z-axis space
The gears are not animated in Maya using keyframes as we want to be able to change individual gears’ rotation speeds easily and flexibly. Thus, the gears are rotated using a script.
I wrote a script that lets me specify values (in degrees per second) to rotate the gear on a chosen axis, by modifying its transform values.
The script can easily be copied and applied to every gear, with each one rotating at selected user-input speeds that can be changed on the go. Below is a screenshot of the script written in Visual Studio (C#), with comments, for those interested.
RotateScript.cs To give illusion that gears are interlocked and rotation in sync with one another, we discovered this engineering trick:
Speed of Rotation is inversely proportional to its radius and number of teeth (sides).
Which means to say, if we halve the size (radius) of a gear, we have to halve the number of teeth and double its rotation speed.
Being a programmer helped discover this as I often use power of two values and this made me realise that as long as we ensure that our gears are modelled with exactly Power of Two number of sides – 8, 16, 32, 64, they are easily compatibility with one another.
Particle Systems
Lastly, a test 3D particle system emits chinese symbols at the bottom right, which are tests for linking to the symbols on a compass we have yet to model.
Particle Effects test The particles’ direction, positions and speeds are also yet to be confirmed. The initial plan is to have them float upward from the surface of the compass, which is different from what our current footage shows.
The particle system uses an emissive shader with a chinese symbol as texture, with its velocity generated by perlin noise to create a randomized movement.
Tutorials & References
Lastly, here are some additional materials referenced that aided in the process.
