Dynamic File Formats

Overview

A dynamic file format is an SdfFileFormat that allows the contents of its layers to be generated dynamically, when included as a payload, within the context of the prim in which it is included. This feature relies on the fact that all layer file paths can optionally have file format arguments that are appended to the layer asset path and are passed to the file format when the layer is opened and read (see SdfLayer::GetFileFormatArguments, SdfLayer::CreateIdentifier, SdfLayer::SplitIdentifier). Any file format can use these arguments to guide how it will translate the contents of the referenced data file into a valid USD layer. What makes a dynamic file format unique is that it is able to compose values of prim fields to generate the file format arguments within the context where the layer will be included. And when the value of any of the composed fields changes, the prim automatically regenerates the file format arguments and creates new layer contents.

Creating a Dynamic File Format

To create a dynamic file format, we first create a plugin library that implements a new derived subclass of SdfFileFormat, just like we would for any other custom file format, and add the type to the library's pluginInfo.json.

For an example in this document, we'll start with a new file format, MyDynamicFileFormat:

class MyDynamicFileFormat : public SdfFileFormat
{
public:
    // Required SdfFileFormat overrides.
    bool CanRead(const std::string &file) const override;
    bool Read(SdfLayer *layer,
              const std::string& resolvedPath,
              bool metadataOnly) const override;

protected:
    SDF_FILE_FORMAT_FACTORY_ACCESS;

    virtual ~MyDynamicFileFormat();
    MyDynamicFileFormat() : 
        SdfFileFormat(
            TfToken("MyDynamicFileFormat"), // formatId
            TfToken("1.0"), // versionString
            TfToken("usd"), // target
            "mydynamicfile") // extension
    {}
}

### plugInfo.json

{
    "Plugins": [
        {
            "Info": {
                "Types": {
                    "MyDynamicFileFormat": {
                        "bases": [
                            "SdfFileFormat"
                        ],
                        "displayName": "Dynamic File Format",
                        "extensions": [
                            "mydynamicfile"
                        ],
                        "formatId": "MyDynamicFileFormat",
                        "primary": true,
                        "target": "usd"
                    }
                }
            },
            "LibraryPath": "@PLUG_INFO_LIBRARY_PATH@",
            "Name": "myDynamicFileFormat",
            "ResourcePath": "@PLUG_INFO_RESOURCE_PATH@",
            "Root": "@PLUG_INFO_ROOT@",
            "Type": "library"
        }
    ]
}

An essential piece of implementing a dynamic file format, that will not be covered in depth here, is using file format arguments that can be appended to the layer's file path to generate or alter some portion of the contents of the layer when the file is read. This can be done by using the file format arguments in the implementation of the Read function or, in the case of a fully procedural layer, creating a custom subclass of SdfAbstractData that utilizes the arguments. The file format arguments are also part of the identity of the layer, meaning layers opened with the same asset path but different arguments are opened as separate layers. Refer to the included examples for detailed methods of creating dynamic content.

Assume that for this example we wrote the implementation of MyDynamicFileFormat::Read to use SdfLayer::GetFileFormatArguments to get the file format arguments from the asset path of the layer it is reading (these would be the same args that are passed to SdfLayer::FindOrOpen when the layer is opened). Also assume our Read function uses the values of the arguments dynamicName and isPositive to alter the contents of the layer.

Now to make this file format dynamic, we must also have this class derive from PcpDynamicFileFormatInterface and implement its two abstract functions.

class MyDynamicFileFormat : 
    public SdfFileFormat, public PcpDynamicFileFormatInterface
{
    ...

public:
    // Required PcpDynamicFileFormatInterface overrides
    void ComposeFieldsForFileFormatArguments(
        const std::string& assetPath, 
        const PcpDynamicFileFormatContext& context,
        FileFormatArguments* args,
        VtValue *dependencyContextData) const override;
    bool CanFieldChangeAffectFileFormatArguments(
        const TfToken& field,
        const VtValue& oldValue,
        const VtValue& newValue,
        const VtValue &dependencyContextData) const override;

   ...
}

Now, because this file format implements PcpDynamicFileFormatInterface, ComposeFieldsForFileFormatArguments will be called while composing the prim whenever a prim spec includes a payload to a file with the extension ".mydynamicfile". This is called before the file is opened to generate additional file format arguments that will be added to the file asset path. The function can use the given PcpDynamicFileFormatContext to compose the value of the strongest opinion of a field on the prim being indexed at the point in which the payload is being included. Currently the only fields that are allowed to be evaluated by the context are custom defined plugin metadata fields, so we'll have to define the fields we plan to use in our plugInfo.json. This restriction may be lifted in the future to include builtin fields like variantSelection but for now those fields cannot be used.

So, in our example plugInfo.json, we'll also define some new SdfMetadata fields, dynamicName and dynamicNumber that we can use to compute file format arguments:

### plugInfo.json

{
    "Plugins": [
        {
            "Info": {
                "SdfMetadata": {
                    "dynamicName": {
                        "type": "string",
                        "default": "",
                        "displayGroup": "Core",
                        "appliesTo": ["prims"],
                        "documentation:": "Example custom string metadata."
                    },
                    "dynamicNumber": {
                        "type": "int",
                        "default": 1,
                        "displayGroup": "Core",
                        "appliesTo": ["prims"],
                        "documentation:": "Example custom number metadata"
                    }
                },

                ...
            },

            ...
        }
    ]
}

In our ComposeFieldsForFileFormatArguments implementation we'll use the context to compose the strongest value of the dynamicName and dynamicNumber fields to generate file format arguments to add to args:

void MyDynamicFileFormat::ComposeFieldsForFileFormatArguments(
    const std::string& assetPath, 
    const PcpDynamicFileFormatContext& context,
    FileFormatArguments* args,
    VtValue *dependencyContextData) const
{
    static const TfToken dynamicNameToken("dynamicName");
    VtValue dynamicNameValue;
    if (context.Compose(dynamicNameToken, &dynamicNameValue)) {
        (*args)[dynamicNameToken] = TfStringify(dynamicNameValue);
    }

    static const TfToken dynamicNumberToken("dynamicNumber");
    static const TfToken isPositiveToken("isPositive");
    VtValue dynamicNumberValue;
    if (context.Compose(dynamicNumberToken, &dynamicNumberValue)) {
        if (dynamicNumberValue.IsHolding<int>() &&
            dynamicNumberValue.UncheckedGet<int>() > 0) {
            (*args)[isPositiveToken] = "true";
        } else {
            (*args)[isPositiveToken] = "false";
        }
    }
}

For dynamicName, we add its computed string value into the args with the key dynamicName. However it's not necessary to always directly transpose field values into args using the field name as the key. For dynamicNumber we compute its composed value to check if it is a positive integer and write either "true" or "false" into args with the key isPositive instead.

CanFieldChangeAffectFileFormatArguments must be implemented so for the moment we'll have it always return true which is a good starting default. See Dependencies and Change Management for the explanation of how this function is used and can be implemented.

bool MyDynamicFileFormat::CanFieldChangeAffectFileFormatArguments(
    const TfToken& field,
    const VtValue& oldValue,
    const VtValue& newValue,
    const VtValue &dependencyContextData) const
{
    return true;
}

With the plugin complete, here's how the dynamic file format would work in practice. Let's say we have the following usd file:

### root.usd

#usda 1.0

def "Root" (
    references = </Params>
    payload = @./dynamic.mydynamicfile@
)
{
}

def "Params" (
    dynamicName = "Foo"
    dynamicNumber = 8
)
{
}

The prim Root has a reference to the Params prim which has value opinions for the plugin fields dynamicName and dynamicNumber. Root also has a payload to a file with the ".mydynamicfile" extension. When the prim index is computed for Root, and the indexer gets to composing the payload, it will see that file format is MyDynamicFileFormat and it will call the format's ComposeFieldsForFileFormatArguments function to produce the file format arguments. At this point in composition, the context includes the reference to Params and will get its values for dynamicName and dynamicNumber as those fields' strongest opinions to produce the fileformat arguments:

• dynamicName = "Foo"
• isPositive = "true"

These args are added to the asset path of the payload layer that will be read giving the resolved layer path:
dynamic.mydynamicfile:SDF_FORMAT_ARGS:dynamicName=Foo:isPositive=true

As mentioned above MyDynamicFileFormat's Read function uses these arguments to generate the identity and contents of the layer. Now say we update root.usd and add the dynamicName field to Root with the value "Bar":

### root.usd

#usda 1.0

def "Root" (
    dynamicName = "Bar"
    references = </Params>
    payload = @./dynamic.mydynamicfile@
)
{
}

def "Params" (
    dynamicName = "Foo"
    dynamicNumber = 8
)
{
}

When Root is prim composed again, the stongest opinion for dynamicName, in the context where payload is composed, will come from Root giving us the file format arguments:

• dynamicName = "Bar"
• isPositive = "true"

Note that the strongest opinion for dynamicNumber still comes from Params. The resolved payload layer path is now:
dynamic.mydynamicfile:SDF_FORMAT_ARGS:dynamicName=Bar:isPositive=true

We have new layer with a different identity and contents from the same payload field without changing the payload declaration itself.

Advanced Examples

We include two examples of dynamic file format plugins in pxr/extras/usd/examples. One ot the major differences between these examples that's worth highlighting is how the scene description is represented. SdfAbstractData is the base class for all scene description represented by a layer and we have a choice when writing a file format as to whether we want use the default SdfData class for our scene description or if we want to write our own custom data representation.

• usdRecursivePayloadsExample - This example uses file format arguments to recursively generate prims with payloads targeting the same file but with a different set of arguments. It uses the default SdfData representation provided by SdfFileFormat::InitData, just like the text based sdf and usda file formats, and creates prim specs in its Read function through the standard SdfPrimSpec API. The generated scene description is pretty simple and minimal so it doesn't warrant the complexity of a custom SdfAbstractData type.
• usdDancingCubesExample - This example generates a cube made up of animated cubes backed by a completely procedural scene description representation. It implements its own SdfAbstractData subclass that is returned by overriding SdfFileFormat::InitData. The file format generates a small set of parameters from the file format arguments and provides them to the data implementation of the layer. The SdfAbstractData subclass uses these parameters to cache some information about the scene and provides the API that generates spec data on the fly when requested. This example greatly benefits from a customized SdfAbstractData implementation as it avoids having to precompute every time sample for every prim when the layer is opened.

Dynamic Payloads

As mentioned above, the composition of prim fields into file format arguments only occurs when a dynamic file is included as a payload. We refer to such a payload as a dynamic payload. This behavior is intentionally exclusive to payloads, as opposed to references, for a couple of reasons:

• Payloads are the weakest composition arcs that read in layer files. The effect of this is that when prim indexing encounters a dynamic payload, the context used for composing fields will have access to all local or referenced opinions on those fields, giving the most complete context with which to process the dynamic file's arguments.
• Payloads can be loaded and unloaded providing a convenient way to recompute dynamic layers whose contents depend on factors other than just file format arguments alone.

A prim index can have multiple payload arcs with any number of them being dynamic. Opinions from stronger payloads are included in the context for weaker dynamic payloads when computing file format arguments.

Dependencies and Change Management

When the PcpDynamicFileFormatContext is used to compute a field value in ComposeFieldsForFileFormatArguments using ComputeValue (or ComputeValueStack) during prim indexing, a dependency is automatically registered for that payload arc on that field value. This means that change management in Pcp knows which fields were used to generate file format arguments for the payload's layer and therefore may need to invalidate the prim index that includes the payload if any of those fields change. It is recommended when writing an implementation of ComposeFieldsForFileFormatArguments to only call ComputeValue on fields as needed if the use of any fields are conditional as it prevents unnecessary change dependencies on unused fields.

Since prim indexes that include dynamic payloads automatically have a dependency on changes to the computed fields, the other interface function CanFieldChangeAffectFileFormatArguments exists to filter out field changes that we know will not alter the file format arguments. Looking at MyDynamicFileFormat still, the dynamicNumber field holds an integer value that is used to populate the boolean isPositive argument. There are multiple values of dynamicNumber that produce the same arguments so we can write CanFieldChangeAffectFileFormatArguments to take advantage of this:

bool MyDynamicFileFormat::CanFieldChangeAffectFileFormatArguments(
    const TfToken& field,
    const VtValue& oldValue,
    const VtValue& newValue,
    const VtValue &dependencyContextData) const
{
    static const TfToken dynamicNumberToken("dynamicNumber");
    if (field == dynamicNumberToken) {
        if (oldValue.IsEmpty() != newValue.IsEmpty()) {
            return true;
        }
        const bool oldIsPositive = (oldValue.IsHolding<int>() && 
                                    oldValue.UncheckedGet<int>() > 0);
        const bool newIsPositive = (newValue.IsHolding<int>() && 
                                    newValue.UncheckedGet<int>() > 0);
        return oldIsPositive != newIsPositive;
    }

    return true;
}

Here if the field is dynamicNumber we check if the old and new values would produce the same isPositive argument, and return false if they would, thus telling Pcp change management that we don't need to invalidate the prim index that includes the payload.

There is one more parameter dependencyContextData that exists in both ComposeFieldsForFileFormatArguments and CanFieldChangeAffectFileFormatArguments. This is an arbitrary typed VtValue that can be populated in ComposeFieldsForFileFormatArguments if there's specific information that would be helpful in determining if a field value change is relevant. The dependencyContextData is stored and passed back to CanFieldChangeAffectFileFormatArguments when processing a field change within the same prim index context. See usdRecursivePayloadsExample for a very basic example of how this can be used.