2022 ASWF Open Source Days OpenEXR Project Update Notes

Owned by Cary Phillips
Last updated: Aug 11, 2022
4 min read

  • OpenEXR v3.2 preview:

  •  

    • Release in September, before the deadline for VFX Reference Platform 2022

    • Swapping out the internals of the C++ layer to use the new C core.

    • An API compatible switch

    • The old API will continue to work, but will add new access patterns

    • The new core is natively multithreaded (can read 100 tiles at once if you want)

  • OpenEXR v4.0 topics

  •  

    • Extend the format, primarily to support new compression schemes:

    •  

      • Kimball: We hear, “Why don’t you use zstd, etc?”

      • Cary: Is the juice worth the squeeze? It’s been so long since the format has changed that we’re not accustomed to dealing with exr files that can’t be read by older software. Are the benefits of new compression worth the disruption?

      • Rod Bogart: Yes, please, we should do that. 

      • Eric Enderton: current compression is zip/deflate.  NVIDIA has done gdeflate, 4x times faster, friendlier to the GPU. It would be great to standardize.  An interim path: Steve Parker has written code to decode existing exr on the GPU. Can annotate the exr file with a table that says where each line starts. 

      • Steve Parker:  All good.  gdeflate is in nvCOMP, so we’ve turned it into a product. 25 Gb/sec. Note that gdeflate is lossless compression.

      •  Steve: It was my pandemic project. There are three options:

      •  

        1. Decompress existing exr format. Can get around 12Gb/sec. Pixel format swizzling is on top of that.

        2. Existing exr format plus small amount of additional data. Can get 25-30 Gb/sec. 

        3. New format, incompatible file, can get 50-60 Gb/sec

      • Kevin (?): GPU decode is great for review. “Just for playback”.

      • Rod: Files don’t have to be for interchange. Existing workflows use files that become something for one particular purpose, i.e. off-ramp files, do something bespoke.  Epic wrote something that was exr-like but wasn’t actually exr.  It would be much better to make it real exr.

      • Kimball: Do we change the way the header is written?

      • Larry: Question for Steve/NVIDIA regarding option 3, a new format: Will it be implemented on CPU, would the performance gain/lose/stay neutral on CPU?

      • Steve: it’s literally just a reshuffling of bitstream. Deflate’s been tuned on CPU’s for decades. Gdeflate hasn’t gotten that yet.

      • Piotr Stanczyk: A window of time when old applications can’t read certain new files. But the alternative of no action could mean the format becomes only archival, doesn’t ever change. We can all live through the pain.

      • Steve: 3 bottlenecks

      •  

        1. Getting all the metadata on the CPU, never have to touch the data on CPU, only GPU.

        2. Decompression. Was the most insurmountable, but now there are reasonable options, not mutually exclusive. Could have an implementation that it supports all of them. 

        3. For some specific types of textures (BC) some other conversion bottlenecks.

      • Larry: add metadata to a file without having to read/write entire data.

      • Exr file: a lot of its job is just to sit there with data. SMPTE standards. Keeping pace with technology will introduce volatility.

      • Ffmpeg has their own implementation that doesn’t read exr2 file.

  • Unaffiliated, unsupported implementations/bindings for exr files: what can be done? 

  •  

    • Rod: ASWF should be reaching out to the other projects. Should be part of what the board does. OpenEXR project shouldn’t have to do this.

    • Nick Porcino: There's a lot of adoption of the little projects. But moving forward they’ll stall official adoption.   I would really love it if the burden isn’t on the project.

    • Eric Reinecke: It seems to me a lot of these have come into existence because the first party “unboxing experience” hasn't been as great as users would like. Perhaps if some resources could be found to address the first party unboxing experience, a lot of those other things might fall off?

    • Nick: I definitely appreciate that sentiment, but the unboxing experience is largely dictated by third parties with only limited interest, such as cmake, pypi, vcpkg, and much of the unboxing experience is not solvable by a "kinder" cmake script or whatever..... On a more positive note, we do have the Rez project, perhaps that provides a venue to better wrangle an unboxing.

    • Eric: Right, I was thinking mostly about things like pypi, macports, and brew - someone did it to address the “I want to pip install openexr” use case

  • Improve/change the symbol versioning scheme?

  •  

    • Kimball: ABI at the C level. Can we change the namespacing policy so the version is not in the namespace?

    • What’s the use case for when you don’t want the namespace? Kimball: when you link against a new OS distribution.

    • Mark Boorer: OCIO we do the namespace version thing, too. If we change the library, we expect them to recompile.

    • Larry: It goes beyond OS distributions. We have the same thing with DCC’s, stuck on an older version. Or, walk away from older versions, but it’s trivial to update. If we can pull the old applications forward without even knowing it, that would pay off.

    • Nick: Maybe part of the outreach Rod suggested for the board, might be to reach out to the brew organization, and the others to let them know there's a venue for cooperation