Asset Resolution (Ar) 2.0
Copyright © 2020, Pixar Animation Studios v 1.0
- Background and Goals
- General Cleanup
- Add Documentation and Examples
- Add Identifier Concept
- Remove Repository and Search Path
- Improve Resolve and Asset Info
- Remove Filesystem-specific Code
- Add Asset Writing Interface
- Add URI Resolvers
- Allow Creation of ArResolverContext From Strings
- Remove ArResolver::ConfigureResolverForAsset
- Rollout and Transition
- Proposed API
Background and Goals
The Ar 2.0 project is an effort to revise the Ar (Asset Resolution) library used by USD to interface with a user's asset management and storage infrastructure.
The initial version of Ar was implemented prior to USD's open source release in 2016. The primary focus at the time was to extract Pixar-specific implementation details from the USD codebase in preparation for USD's public release. Although Ar was intended to be a general-purpose interface, it was not entirely successful in that regard. For example, Ar was geared towards assets stored on a traditional filesystem, and parts of USD still assume that asset paths are filesystem paths that can be manipulated as such. Also, some Pixar-specific concepts like repository paths and search paths are still present in the interface.
Ar 2.0 aims to address these issues, as well as incorporate feedback and add support for new use cases that have developed over the last few years.
The ArResolver interface will be reimplemented using the non-virtual interface (NVI) pattern (as described here) to provide more flexibility and potentially allow future changes without breaking the public interface.
In the sections below, we will be presenting the proposed new public API for ArResolver. See the proposed headers for details on the virtual functions used to override behavior.
The API on ArResolver will all be marked const to aid with writing const-correct code and as an indicator that these functions may be called concurrently and must be thread-safe (following the semantics used by the Standard Library, see here for more details)
Add Documentation and Examples
More extensive documentation and example resolver implementations will be provided as part of this work.
Add Identifier Concept
The current ArResolver interface contains a number of functions that are used primarily in Sdf to compute identifiers for SdfLayer objects given an asset path or an (anchoring asset path, relative asset path) pair, e.g., for calls to SdfLayer::FindOrOpen or CreateNew. These functions lack clear documentation and introduce confusing concepts, some of which are remnants of Pixar-specific implementation details. To simplify the interface, these functions will be removed:
and replaced with:
These functions are responsible for taking an asset path and computing an ArIdentifier that refers to a logical asset. An ArIdentifier is a simple wrapper around a std::string to allow clients to more clearly indicate intent, as opposed to using bare std::string objects everywhere.
This identifier has consequences for operations like SdfLayer::Reload, which re-resolve the identifier to see if an update is necessary. In the simplest cases, the identifier may just be the same as the input asset path, but for more complex resolution behaviors determining the identifier may be more involved.
For example, the ArDefaultResolver implementation included with USD supports normal filesystem paths, like "/Dir/File.usd" or "./Dir/File.usd". In these cases, the corresponding identifier would simply be the normalized, absolutized (anchored either to the current working directory or the supplied anchor path) forms of these paths. So, calling SdfLayer::FindOrOpen with these paths would return SdfLayer objects with identifiers "/Dir/File.usd" and "/current/working/dir/Dir/File.usd".
ArDefaultResolver also supports "search paths" that look like "Dir/File.usd". For these paths, ArDefaultResolver searches an ordered list of directories specified by the ArDefaultResolverContext by prepending those directories to the path and returning the first one that exists on disk. For these search paths, the corresponding identifier is the search path itself. So, calling SdfLayer::FindOrOpen with the above path would return an SdfLayer with identifier "Dir/File.usd". Calling SdfLayer::Reload on this layer would cause the identifier to be resolved again, but since the identifier was kept in search path form ArDefaultResolver knows to re-run the search to see if the asset had been installed in a new directory in the search order.
The CreateIdentifierForNewAsset functions are separated from CreateIdentifier to support existing functionality where calling SdfLayer::CreateNew with a particular asset path may return an SdfLayer with a different identifier than calling SdfLayer::FindOrOpen with the same asset path. This distinction will show up again in the Resolve functions described in the next section.
For example, when using ArDefaultResolver calling SdfLayer::CreateNew will always anchor the given path to the current working directory. Calling SdfLayer::CreateNew with a search path like "Dir/File.usd" will always result in an SdfLayer with identifier "/current/working/dir/Dir/File.usd" and create the new layer at that path. Calling SdfLayer::FindOrOpen with that same search path would instead return an SdfLayer with identifier "Dir/File.usd" as described above.
Remove Repository and Search Path
The addition of the identifier concept above allows us to remove the repository path and search path concepts from Ar. Aside from the functions mentioned above, the following will also be removed from Ar and related libraries:
Repository paths are currently used as an additional index in the SdfLayer registry. This will also be removed as part of this work.
Improve Resolve and Asset Info
The Resolve functions on ArResolver will be modified to:
Similar to ArIdentifier, an ArResolvedPath is a simple wrapper around a std::string that makes it clear to clients when they are interacting with a resolved vs. unresolved path. This will help resolve confusion around whether other ArResolver APIs are expected to handle resolved or unresolved paths -- the APIs that require a resolved path will be updated to take these objects as parameters instead of a bare std::string.
We considered the idea of allowing resolvers to store arbitrary blind data in an ArResolvedPath. This would allow resolvers to store additional information during asset resolution that could be used later when calling other ArResolver functions, like OpenAsset. However, there were concerns that clients would fill the blind data with information *required* to consume the resolved path in these APIs. This would potentially break the ability to replace an authored asset path in a USD layer with its resolved path to 'bake in' the asset resolution for workflows like asset isolation/freezing. These concerns led to this idea being dropped, at least for now.
Like CreateIdentifierForNewAsset, ResolveForNewAsset is required primarily for SdfLayer::CreateNew to resolve the given asset path to a location where a new layer may be written. Resolve cannot be used for this purpose, since it is expected to return an empty resolved path if no asset exists at a given location. ResolveForNewAsset is a replacement for ArResolver::ComputeLocalPath and will allow us to remove a number of other related functions for layer creation as well:
Both Resolve and ResolveForNewAsset will accept an optional ArAssetInfo to populate during resolution. In the old ArResolver interface, this was done via a separate ResolveWithAssetInfo function to avoid adding a default parameter to a public virtual function. Using the public NVI pattern avoids this issue, allowing us to remove ResolveWithAssetInfo and reduce the size of the C++ API. (Note, however, that ResolveWithAssetInfo and ResolveForNewAssetWithAssetInfo will still exist in the Python API since there is no concept of an optional output parameter in Python.) These functions will be the only ones expected to populate an ArAssetInfo; the UpdateAssetInfo function will be removed.
Remove Filesystem-specific Code
Remnants of Ar's filesystem-centric origins still exist in the ArResolver interface and various parts of the USD codebase. A large part of the Ar 2.0 project will be finding and replacing these remnants with Ar-based abstractions.
The ArResolver::FetchToLocalResolvedPath function was an early attempt at supporting asset systems that did not store their data directly on disk. The idea was that USD would call this function to save assets from remote data stores onto the local filesystem. This was necessary because, at the time, USD only supported reading data from files on disk. Since then, much of the codebase has been updated to use the ArAsset API, which provides a more general interface for reading data from any backing store and does not require assets to be saved locally. So, these functions will be removed:
The primary downside of this removal is that it potentially makes interacting with 3rd-party libraries that require a file on disk more difficult. For example, the Alembic library requires a file path to open an Alembic archive. To support non-filesystem asset stores, the usdAbc plugin could have used FetchToLocalResolvedPath to save a .abc file to the local disk and passed that file path to the Alembic API. This would give the ArResolver implementation control over how and where the .abc file would be saved. Without this function, an alternate approach would be needed. One possibility is that the usdAbc plugin could use the ArAsset API to open the .abc file from its data store and then save the file to a temporary location. But, that would move control of the localization process to the usdAbc plugin itself.
Add Asset Writing Interface
The ArAsset API allows USD to read data from non-filesystem locations and most of the USD codebase that reads data from assets has been updated to take advantage of this. However, parts of USD that write data out (for example, the .usd/.usda/.usdc file formats) still use filesystem-specific API to do so. To fully separate USD from filesystems, an API for writing data will be added to ArResolver:
These functions will return instances of a new ArWritableAsset interface that provide an interface for writing bytes to some destination. This interface has been kept as minimal as possible to make it easier to implement. However, resolvers are not required to implement these functions and may return a NULL pointer if they do not support writes.
The USD codebase (particularly the file formats that ship with USD) will be modified to use this new API for writing data. This will allow us to remove ArResolver::CreatePathForLayer.
Add URI Resolvers
Ar 2.0 will add native support for multiple resolver implementations based on URI scheme. This will allow users to easily plug in support for multiple asset systems and forms of asset paths without needing to modify the existing resolver implementation. Ar will still require a "primary" resolver that will handle asset paths without an associated URI or an unhandled URI.
To implement a URI resolver, users would implement an ArResolver subclass and associate it with a given URI scheme or schemes in that subclass' entry in the corresponding plugInfo.json file. At runtime, Ar would parse given asset paths to determine their URI scheme and dispatch it to the associated URI resolver. If no URI scheme is present in the asset path or no URI resolver for the scheme is found, Ar will dispatch the asset path to the primary resolver that is used today.
For example, a user might implement an HTTPResolver that handles "http:" and "https:" asset paths like:
At runtime, the HTTPResolver would be registered alongside the default ArDefaultResolver. Ar would handle dispatching to the appropriate resolver automatically, so:
ArResolverContext is a container used to supply additional data to resolve implementations for use during asset resolution. Support for multiple resolvers will be added so that context objects for different resolvers can be held in a single ArResolverContext object and passed through to UsdStage and other consumers as they are today. For example, ArDefaultResolver has an associated ArDefaultResolverContext that allows clients to specify directories to search when resolving a search path. The HTTPResolver might have a context object to allow users to specify credentials for retrieving assets from a web server:
A client could create a UsdStage using these context objects like:
Note that binding an ArResolverContext via ArResolverContextBinder will override any ArResolverContext that had previously been bound. This behavior ensures consistency, for example:
Allow Creation of ArResolverContext From Strings
The ArResolver interface currently allows implementations to create a default resolver context or a resolver context for a given asset via CreateDefaultContext and CreateDefaultContextForAsset. However, this level of control is insufficient for DCCs and plugins that want to allow users to specify other resolver contexts that are specific to their resolver implementation. To support this, a new function will be added to allow ArResolverContext objects to be created from strings:
These functions will call protected virtual functions on ArResolver to allow resolvers to parse the given string and return an appropriate ArResolverContext object. Using the first overload would pass the given string to the primary resolver to parse and return an ArResolverContext. An example usage might look like:
The second form is used to create an ArResolverContext for multiple resolvers. An example usage might look like:
In this case, ArResolverContext will use the URI resolver associated with the "http" scheme to create a context using the "username=..." string, use the primary resolver to create a context using the "/search/..." string, then combine them into the returned ArResolverContext object.
Note that the public API is defined on ArResolverContext. This made more sense for organization and avoids some ambiguities that would arise (particularly for multiple resolver support) if this API was put on ArResolver instead. For consistency, the existing functions for creating default contexts will also be moved to ArResolverContext.
This function allows resolvers the opportunity to configure themselves in applications that only load a single asset. This seems like an odd and unnecessarily specific case to call out and does not cover any use cases where multiple assets may be loaded. Animal Logic's Maya USD plugin actually hijacks this function and allows users to pass in a general configuration string instead of an asset path; this will hopefully no longer be necessary with the new ability to create ArResolverContext objects from strings.
To simplify the ArResolver API, this function will be removed.
Rollout and Transition
Ar 2.0 will substantially modify the ArResolver interface and other related classes. To maintain backwards compatibility with existing ArResolver implementations and client code sites, users will be able to specify whether they want to enable Ar 2.0 at build time. If enabled, the AR_VERSION flag will be defined and to 2 and the Ar 2.0 changes will be built and installed, causing old code to break if not updated to accommodate those changes. If disabled, AR_VERSION will be set to 1 and Ar will revert back to the code that exists today, ensuring no change in behavior or build breakages.
For its initial release, Ar 2.0 will be disabled by default – the current Ar implementation will remain the default for backwards compatibility, but will be deprecated. Users will be able to enable Ar 2.0 using the build flag to begin testing and transitioning their resolver implementations and client code to the new interface. Ar 2.0 will be enabled by default in the subsequent release, and the old Ar implementation will be removed at that time.
The following are rough sketches of the proposed new interface on ArResolver and related classes. These are not final and subject to change.