glTF MaterialX USD Interop

The following are notes for discussion which involves glTF, MaterialX and USD

Background

1. USD / glTF / MaterialX interactions require a clear understanding of workflows and the roles each component plays within those workflows.

2. glTF is a "last-mile" format so what a glTF -> MaterialX, or MaterialX -> glTF workflow while compelling to have, we should carefully consider what this entails in terms of data representation and data "loss" (for lack of a better word).

3. The glTF -> MaterialX (and USD) workflow can be mostly non-lossy with the current MaterialX glTF pbr node definition (which when imported can be a USDShade node).

   1. This allows representation of glTF pbr plus directly connected upstream images+placement to be accessible in various authoring / lookdev / rendering worklows.

   2. Also code generation for consistent conversion to target shading languages now possible. ( glTF direct implementations does not inherently guarantee any kind of render implementation consistency. )

4. The MaterialX -> glTF workflow requires "distillation" and so far current prototype implies that this can always be can be lossy (Autodesk).

   1. One vector for loss is that different shading models must be translated to glTF PBR.

   2. A second vector is arbitrary shading graphs can be created in MaterialX which must be "baked" somehow.

   3. All meta-data is lost. 

5. You can run into this with USD where it can be "worse" via custom schemas.

Points of Discussion 

1. MaterialX as an extension.

   1. Similar to the MDL proposal

   2. The key difference is the proposal would not to keep 2 representations of the same material but to directly support a MaterialX atomic "node".

   3. The MaterialX schema defines the inputs and outputs in the data model versus this being part of code logic.

   4. Implementation toggles are not part of the definition.

2. There are concerns that this "breaks" the mandate of the glTF format as MaterialX becomes an external dependency which requires more "complex" handling.

   1. This is already required but is provided by multiple different libraries (e.g. there is no "one" parser / evaluator for the glTF format).

   2. There is the idea to just have MaterialX in JSON format but this misses the requirement for graph value resolution (which is similar but less extensive in MaterialX than what is required for USD resolution).
      Usage of many "hand-made" JSON parsers can be appealing but can lead to inconsistency quickly.

   3. Pre-evaluation is possible here. (*)

3. Code generation available but there are concerns about shader performance and duplication of resources. These are generally valid concerns.

   1. There is inherent complexity in the shaders as fidelity is the initial goal. Refinements can made over time.

      1. Fidelity options are being discussed on the MaterialX side with optimizations for IBL sampling code already proposed.

   2. Performance metrics would be worthwhile to obtain in general (not just for Web deployment).

   3. Again pre-evaluation is possible here. (*)

   4. For resources, MaterialX does no manage resources. However If multiple formats or non-performant resource formats are explicitly being used then this should be looked for alternatives. This can be part of the shader translation, utility or core definition libraries. e.g Support for cubemap lookups for IBLs might be worthwhile as a new core definition.

4. (*) Pre-evaluation:

   1. A small subset of "essential" MaterialX shader nodes / graphs can be considered.

      1. This is interest to IKEA and Autodesk, but could also be to others.

      2. The better the match between runtime and authoring definitions – the better the interop.

   2. It seems worthwhile to discuss what this set is.

   3. The converse of adding encapsulations of glTF upstream mappings as new definitions is already being considered. e.g. for Occlusion/Roughnes/Metal channel extraction / combining which helps the distillation process but could be a generally useful utility for image data packing/unpacking – perhaps even for output packing for things like AOVs.

   4. There is the "node system" in Three.js as well which can be looked at.

      1. Work on this can be found here: https://github.com/mrdoob/three.js/issues/20541

      2. The idea of JSON bindings, and approach to pull pull out code for MaterialX "equivalent" nodes can be found in this discussion.

   5. USD definitions consume MaterialX definitions so definition / interface synchronization is already considered. This is not currently the case even with the single glTF pbr node.

   6. From a shader point of view there has been discussion about pre-generating code blocks which can be glued together for reuse. This may or may not be something which requires formal definitions but could be done on the fly.