Sony Imageworks Animation Layout Workflow with Unreal Engine and OpenUSD

This addresses our journey to building a new rough layout pipeline at Sony Pictures Imageworks. There are two main components of the new pipeline: standing up Unreal Engine as the creative tool to use, and implementing the USD filetype so we can get data from the Windows environment of Unreal Engine to our Linux based production pipeline. We will share solutions to interesting production issues that we faced, and dive deeper into the USD pipeline

One of the test shots from the animated feature The Mitchells vs. the Machines: Left: rough-layout rendered with Maya playblast. Right: rough-layout rendered with Unreal Engine.

we have built in order to expedite our Unreal Engine workflow. Also, we will explore our Unreal development by walking through goals, challenges and interesting findings. Some areas we will touch on include publishing USD shots, building an Unreal pipeline in parallel to production, and the automation of importing USD into Maya and exporting it into the legacy pipeline.

Our journey ranges from determining how to name and construct directories to building sophisticated tools that automate shot structures and LevelSequence organization. Some Unreal tools that we will discuss include: creating sequence structures in Unreal, storing changed Unreal materials and properties for USD instanced prims, speeding up exporting USD from Unreal (in some extreme cases, our export speed became 20x faster), issues we fixed in the Unreal source code, and animating visibilities of USD actors in Unreal LevelSequences. We do part of the development by creating our own Unreal Blueprint libraries, actors and components with C++, as well as Python modules for parsing the final USD output.

We will discuss several successful tools that we built. For saving and loading the Unreal properties and materials applied to USD, we implemented a mechanism to collect and store all changes. This makes our code more portable and easy to use across multiple projects. We developed an elegant solution for animating USD visibility in Sequencers that is performant when scrubbing through frames, playblasting, and rendering. Lastly, we created a tool to export our USD output for use in other DCCs.

The design of our shot-structure in the Unreal Level Sequence

Assets are imported to Unreal on Windows with our USD asset resolver through Linux symlinks using our asset management database. We use different USD layers in Unreal for staging assets, set-dressing, and categories based on the departments and shots. The artists can then use an Unreal Engine sequencer to design their layout sequence. We've recreated many of the tools of our Maya pipeline in Unreal Engine to maximize productivity of artists. We organize assets in Sequencer under the folders by the asset type like cam, char, prop etc. Our publish script processes the USD export and parses it based on the asset types. After that, the layout sequences are chopped according to shot length, and each shot USD file will have sublayers of asset types as well as staged and set-dressed environments. Finally, Maya reads these shot-based USD files and recreates them as SPI animation ready files.

The design of our USD layers

At the moment we still use FBX as the file format to bring characters into Unreal Engine. By fully adopting UsdSkel export we could potentially use the USD file type for this step. Also, it is useful to import caches published by the animation department into Unreal to see the animation in context with lighting and fx created by the layout artists, as well as realtime camera work. Currently we are using Alembic for this purpose, since it is more performant than USD for animated caches in Unreal. If we can achieve better performance for animated USD caches, we could use that instead. This would also involve further USD layer management for the characters.

Although we have achieved remarkable results in creating content from Unreal Engine and delivering it to other DCCs, it is considered that crossing DCCs with multiple types of assets is inefficient. An ideal workflow would be completing the entire process within a minimum number of DCCs as it has a lesser chance to lose data and saves time on converting assets. However, it is inevitable to use multiple DCCs to take advantage of their strengths. From our experience of building the Unreal-to-Maya pipeline, USD plays an important role in resolving various difficulties such as converting asset formats, and provides the ability to layer the edits made by each department while preserving materials and animation. Our next USD pipeline goal would be making Unreal-to-Katana workflow for key lighting, character pipeline for rigging, and post-animation pipeline for final layout.