Image Plane Schema Proposal (draft)

Introduction

This proposal aims to add native support for Image Planes to USD. An Image Plane is a camera constrained image that can be seen in "screen space" when looking through that camera.

The majority of work in VFX is centered around augmenting existing footage, so it is important to view animation in the context of that footage (it doesn't matter if the animation looks good if it doesn't look integrated). Without this ability, the utility of tools like usdview is quite restricted and requires compositing before work can be evaluated in context.

Applications outside of VFX:

TODO:

High Level Anatomy of an Image Plane

Conceptually, an Image plane must provide the following information:

associated camera
image (can be animated per frame)
fit to filmback (how image is positioned in relation to cameras filmback)
visibility (should be hide-able for different workflows)

And should probably also provide:

depth (how far away from pinhole camera this plane should exist in space)
alpha gain or other "image attributes"
more attributes to fine tune fit

UsdImagePlane API Schema

The UsdImagePlane schema defines basic properties for rendering an image plane. The UsdImagePlane schema derives from UsdSchemaBase. TODO: Verify best base class.

TODO: Continue fleshing out

Properties

asset image = @@
- Path to image file, can be time sampled per frame
double depth = 0
- distance from the pinhole to the point on the optical axis where the image plane sits
  - l_o in CG Camera diagram 2.1 in https://cookeoptics.com/i-technology/ > "Cooke Camera Lens Definitions for VFX"
  - alternative name: "distance" (Autodesk Maya used "depth")
  - A nice default could be -1 or "infinite" so that it would always be behind CG. That might be hard to coexist with a physically placed in scene camera depth that exists in front of and behind cg elements.
uniform token fit = "vertical"
- How the image plane is fit to the filmback. Possible values:
  - "horizontal" - fit image width to filmback and keep image aspect ratio
  - "vertical"- fit image height to filmback and keep image aspect ratio
  - "best" - from Maya, maybe too ambiguos
  - "fill" - stretched to filmback
  - "to size" - constant size, centered on filmback, and requiring more data to define "image size"
float[2] offset
- in same units as camera filmback - tenth of scene unit
token visibility
- Control image plane visibility. Possible values:
  - "inherited"
  - "visible"
  - "invisible"

Open Questions

Should you have the ability to define multiple Image planes per camera? YES

Should the image plane be a separate prim or part of the camera prim? CAMERA

Should the "image" be implemented as a UsdTexture? Or any other part of Image Plane defined as its component part?

Implementation Strategies

Proposed: As a Multi-Apply Schema

If we implement a Multi-Apply API Schema for Image Planes we can:

author attributes directly to camera
author multiple image planes to camera
- This could be used to view CG elements in context with foreground or background plates.

A Minimum example (with sample API):

camera = stage.GetPrimAtPath('/world/cam')
imagePlaneApi = UsdImagePlaneAPI(camera)
imagePlaneApi.SetImagePlane("imagePlane1")
framesToImages = [(f, fileSequence.frameAt(i) for f in fileSequence.frames()]
imagePlaneApi.SetImagePlaneImageAttr(framesToImages, "imagePlane1")

Which would generate this .usda:

def Xform "world" {
    def Camera "cam" {
        ...
        string[] imagePlanes = ["imagePlane1"]
        asset imagePlane1:image = {
            1001: @/path/to/image.1001.exr@, 
            1002:...
            }
        float imagePlane1:depth = 1000.0
    }
}

Alternative Strategies

As a concrete prim type

Here is a working implementation that was done before Multi-Apply schemas were added to USD. Its a fully featured solution that is based around Autodesk Maya's "underworld" image plane nodes where many planes can live under a camera. This specific implementation is just a quick attempt to move maya image planes across packages and the hydra implementation is just a workaround that generates an HdMesh rprim (instead of creating a new image plane prim type).

Reference PR from Luma: https://github.com/LumaPictures/USD/pull/1

Pros:

[visibility, purpose] can be leveraged to provide intuitive functionality

Shortfalls:

Requires relationships between camera + plane prims
Would have to traverse below camera to find whether has any image planes

Using existing USD types (Plane + material + camera + expressions)

It is possible to create a UsdPlane with a material and accomplish the same thing as an image plane.

TODO: Get example

Shortfalls:

Requires users to implement complex authoring code.
No explicit identity as an "Image Plane". This will make it hard to successfully roundtrip from DCCs to USD.

Example Use Cases

Hydra / Usdview

It would be helpful to be able to view CG elements against a backdrop of the production plate when checking exported usd caches.

DCC Interchange

Having a "UsdLux" style standard for Image Planes would be very useful when implementing importers and exporters for DCCs like Maya, Blender, Nuke, etc.