In addition to addressing the interchange problem, USD also provides powerful mechanisms for large-scale collaboration and context-dependent asset refinement anywhere in a production pipeline. For example:
- layers enable artists in different departments to all work simultaneously on the same "thing", whether that be a singular asset, an aggregate asset, a sequence, or a shot.
- variants enable asset creators to package up an enumerated set of variations, which can be selected and re-selected by downstream artists in consuming contexts. Using variants in combination with the inherits composition operator, a downstream artist can introduce new variations for all instances of an asset, in a particular (set, sequence, shot, etc.) context. Abilities like this can dramatically reduce the number of pipeline-stalling "asset fix requests" needed to support downstream artists.
This set of documents describes the concrete Usd software package, its APIs, and their effective use. In contrast, the forthcoming Composition Compendium centers on USD composition's detailed semantics.
Following is an overview of the architecture, followed by information for getting started with USD, and some background on how the USD project came to be.
The USD repository is divided into four "core" packages, plus 3rd-party application plugins. The four packages are:
- base - Contains application-agnostic, foundation modules, upon which most Pixar software builds, including possible future open-source projects.
- usd - Contains all the modules directly related to authoring, reading, and composing universal scene description. Defines the plugin interfaces by which USD can be extended and adapted.
- imaging - Contains the embeddable Hydra hardware renderer as well as other low-level, imaging-related services.
- usdImaging - Ties usd and hydra together by providing an adapter to Hydra that reads data directly and efficiently from a UsdStage. Defines a plugin interface by which the adapter can be extended, and provides usdview , the standalone graphical USD inspector tool.
- extras - Contains a collection of tutorial material, sample code, and small example USD assets.
To author and consume USD, you need only the base and usd packages. Most of the open-source third-party plugins, however, do make use of usdImaging for fast preview, and as part of the answer to the question of what application XXX should do when it is asked to import a USD prim type of which it has no corresponding native representation. (Answer: it creates a proxy representation that delegates preview to usdImaging, and presents the prim's properties for inspection and overriding.)
Following is a deeper indexing of the four packages.
The "base" package
- Arch: Architecture Dependent module centralizes functionality that must be implemented differently for different operating systems or architectures, so that we can avoid littering the entire code-base with conditionally-compiled code.
- Tf: Tools Foundations module is a catch-all for low-level, commonly used services developed at Pixar, including facilities for memory tracking, error reporting and debugging, string utilities, boost python wrapping aids, threading/synchronization tools, registry and singleton management, smart pointers, and "observer pattern" support, which we call "notification".
- Gf : Graphics Foundations module provides Pixar's equivalent to imath, and provides many of the types enumerated in Vt. It also provides other foundational graphics abstractions like frusta and a camera model. Given unlimited resources we would have replaced our use of the linear algebra components in Gf with imath, but Gf's use in Pixar's code-base is substantial and its API's do not map one-to-one to imath.
- Js: JSON I/O module provides a thin API wrapper on top of the RapidJSON package, which is what our plugin system uses for multi-threaded plugin discovery.
- Tracelite: Performance tracking module provides a stubbed interface for embeddable performance-profiling tagging. USD does not yet ship with an implementation of this interface, but may at some point in the future.
- Vt : Value Types module provides many of the concrete types that Sdf is able to recognize and serialize. It provides a copy-on-write array-type, VtArray, which is used for all array types in Usd, and an efficient type-erasure wrapper class, VtValue, that also provides datatype conversion facilities and support for unboxing python objects from boost python. VtValue is supported in all Usd API for getting and setting values.
- Work : Multi-threaded Dispatch module provides a thin abstraction layer on top of Intel's TBB (Thread Building Blocks), and is leveraged extensively in Usd core and higher-level services for multi-threading.
- Plug: Plugin Framework module provides the organization and access API for all plugins to Pixar open software. It identifies plugins by the existence of a plugInfo.json manifest file, which enumerates the number and kinds of plugins provided by a module.
The "usd" package
The USD package begins with the low-level modules for path resolution, scene description serialization, and composition, upon which the "core" Usd module relies. The USD package is rounded out by a set of schema and utility modules that layer on top of the core. In low-to-high order, the modules are:
- Ar: Asset Resolution module defines the abstract interface for USD's asset resolution plugin, so that clients can author asset references in their USD files that make sense to their asset management systems. It also provides a "fallback resolver" that is active when no site-level plugin has been provided; the fallback resolver provides basic search-path based resolution.
- Kind : Extensible Categorization module provides a simple, site-extensible token-based typing system, which USD uses, for example, to identify and classify types of "models".
- Sdf : Scene Description Foundations module defines the Usd data model, namely: prims, attributes, relationships, meta-data, as well as the concrete types that attributes can possess. Sdf also provides the key abstraction SdfLayer, which represents a file of data subscribing to the Usd data model, and provides low-level data authoring and extraction API's. SdfLayer also has an associated plugin mechanism known as SdfFileFormat that allows any reasonable file format to be dynamically translated into Usd; it is via this mechanism that we added a binary encoding to the pre-existing ASCII format for Usd, as well as how we support referencing and reading of Alembic files.
- Pcp : PrimCache Population (Composition) module implements the composition logic at the heart of USD; Pcp efficiently builds and caches an "index cache" that the Usd scenegraph uses to determine which prims need to be populated, and where to look for values for any particular property in a multi-layer aggregation of assets.
- Usd : Universal Scene Description (Core) module builds the USD scene-graph (UsdStage), and provides the primary API's for authoring and reading composed scene description. The Usd module and the concepts it contains are meant to be fairly generic, allowing many different domain-specific schemas to be built over it, as the USD package does for geometry, shading, etc. We make an exception for two particular concepts/schemas, "Model" and "AssetInfo", which are defined in the core Usd module, because they extremely valuable for organizing scene description, and other core behaviors are built on top of the concept of Model.
- UsdGeom : USD Geometry Schema : The primary graphics-supporting schema module for DCC interchange, UsdGeom provides geometric primitives (meshes, curves, patches, xforms, etc.), as well as a Camera schema and "Primvars" that encode UV's and various user-defined fields over a primitive. The majority of the schema classes are produced entirely via the usdGenSchema code generator included with the core.
- UsdShade : USD Shading Schema Schema module that defines Looks, Shaders, connectible Parameters, with API for building shading networks and user-facing "Look Interfaces" that allow shading networks to be shared (via instancing) by multiple Looks. UsdShade also prescribes how geometry is bound to Looks. With just the objects in UsdShade one can encode completely generic shading; however, it also allows renderer-specific shading schemas to be built on top of it, and a Look can host the network "terminals" for any number of renderer schemas.
- UsdHydra : USD Hydra Schemas module provides a set of schemas for authoring Hydra-targetted shading in the USD shading model.
- UsdRi: USD Renderman Schemas module for data targeted at Pixar's RenderMan specification. Includes shading schemas for both RSL and RIS shading systems, as well as a "catch all"
RiStatements schema for encoding various RenderMan concepts/directives, such as attributes, options, coordinate systems, etc.
- usdUtils_page_front module provides a number of utilities and conventions that have proven useful in a USD-based pipeline.
The "imaging" package
- Garch : The Graphics Architecture module provides abstractions for architecture-specific functionality that may pull in imaging-related dependencies.
- Glf : The Gl Foundations module provides access to textures, ptextures, and glslfx files. It also provides some basic glsl shaders upon which Hd behaviors are built.
- CameraUtil The Camera Utilities module provides a small but important set of common camera-related computations that don't really belong in the core GfCamera class.
- PxOsd : The Pixar OpenSubdiv module provides some higher-level utilities on top of OpenSubdiv that may, one day make it into OpenSubdiv, but are still "cooking". Hydra interacts with OpenSubdiv through pxOsd.
- Hd : The Hydra module provides the Hydra renderer. Hydra uses OpenSubdiv for tessellating subdivision meshes, and supports general programmable glsl shaders, and multiple render passes.
- Hdx : The Hydra Extensions module provides higher-level functionality on top of Hydra, such as organization and packaging for particular kinds of multiple render-pass tasks.
The "usdImaging" package
- UsdImaging : An adapter to Pixar's modern, high-performance GL rendering engine Hydra (which in turn leverages OpenSubdiv) that provides preview imaging (including streaming playback) for any UsdStage encoded with UsdGeom and UsdShade. UsdImaging takes full advantage of scenegraph instancing in USD, and uses multiple threads to extract data from a UsdStage and marshal it into Hd objects.
- Usdviewq : A python package that provides the controller and GUI elements used by usdview. The elements are structured for ease of embedding in other python applications, also.
Python Support, numpy, etc.
Usd includes python wrapping for all core classes and generated schema classes. Each module described in the last section, plus the core Usd module, are independently importable. Each module includes in-python help/documentation extracted from the doxygen docs for the module. We have chosen, initially, to keep the python API as true to the C++ API as possible, with one notable exceptions.
- Some underlying modules support more pythonic API's. Some of the foundational modules shared with Presto, such as Sdf, provide a more pythonic API in which member-variable-like-things become properties in python. We have hesitated to deploy this level of pythonification at higher levels of the system because SdfLayer is the only container that truly owns its data: all the primary Usd-level classes represent restricted views on data that resides in one-or-more SdfLayer s, and don't possess any member variables/properties other than those that serve as temporary caches.
USD's central array type, used to hold all array-valued scene description types, is VtArray. THe python wrapping for VtArray supports python's "buffer protocol", so VtArray 's returned by USD to python can be trivially accessed like and converted to native python containers, as well as numpy arrays. Further, USD methods that expect a VtArray input argument should always successfully convert native python containers and numpy arrays whose underlying element-type is compatible.
Please visit the USD Documentation Hub where you will find tutorials, FAQs, and other supporting material.
USD: What's the Point, and Why Isn't it Alembic ?
The outward-facing goal of Universal Scene Description is to take the next step in DCC application data interchange beyond what is encodable in the ground-breaking Alembic interchange package. The chief component of that step is the ability to encode, interchange, and edit entire scenes with the ability to share variation and repeated/instanced scene data across shots and sequences, by providing robust asset-level (but not restricted to asset-level) file-referencing with sparse override capabilities. Additionally, USD provides a handful of other "composition operators" and concepts that target:
- Encoding asset variation and preserving the ability to switch variants late in the pipeline
- Scale to scenes of unlimited complexity by deferring reading of heavy property data until it is needed (as in Alembic), and also by providing composition semantics that allow deferred (and reversible) loading and composition of arbitrary amounts of scene description without sacrificing the ability to perform robust dependency analysis between prims in a scene. (See the discussion of payloads for more information).
The USD project also developed under a high priority inward-facing mandate to simplify and unify Pixar's binary cache-based geometry package (TidScene) with its ASCII, animation and rigging scene description and composition system (Presto core). This mandate required that the advanced rigging concepts and datatypes of Presto be layerable on top of a reduced-featureset, shared (with USD) core. Given the time and resource constraints, and necessity to not massively disrupt our in-use codebase, we chose to largely retain our existing data model, while looking to Alembic as a guide for many schema decisions along the way. While it is unlikely Pixar will attempt to expand the Alembic API and core to encompass USD composition, asset resolution, and necessary plugin mechanisms, we are committed to providing a USD "file format" plugin that allows the USD core and toolset to consume and author Alembic files as if they were native USD files (obviously writing cannot always produce an equally meaningful Alembic file because composition operators cannot be represented in Alembic).