2022-07-11

Owned by Michael Dolan
Last updated: Jul 12, 2022 by Remi Achard
6 min read

July 11, 2022

Host: Doug Walker

Secretary: Rémi Achard

Attendees:

Rémi Achard (TSC) - DNEG
Mark Boorer (TSC) - Industrial Light & Magic
Mei Chu (TSC) - Sony Pictures Imageworks
Sean Cooper (TSC ACES TAC Rep) - ARRI
Michael Dolan (TSC) - Epic Games
Patrick Hodoul (TSC) - Autodesk
John Mertic - Academy Software Foundation / Linux Foundation
Mark Titchener (TSC) - Foundry
Carl Rand (TSC) - Weta Digital
Doug Walker (TSC Chief Architect) - Autodesk
Kevin Wheatley (TSC) - Framestore
Zach Lewis - Method
Cuneyt Ozdas - Autodesk

Apologies:

  • Carol Payne (TSC Chair) - Netflix

OCIO TSC Meeting Notes

  • GitHub Wiki:

    • Doug: Created GitHub wiki page to discuss about cache and cacheID usage in OCIO. It's a confusing area that could use better documentation, please have a look, looking forward for comments.

    • Doug: Plan is to eventually migrate to the official documentation (RTD). 

  • Update processor caching, issue #1645, PR #1656:

    • Doug: Wanted to discuss latest comment on the PR. Unexpected results from ocioperf, was expecting for OCIO_CACHE_FALLBACK to reduce first run's processor creation time.

    • Remi: Tried to show how disabling the cache fallback impacts performance in two different scenarios, one is never using the cache fallback while the other is always using it. Both involve creating a single processor encompassing a full display transform chain from a config making heavy use of large 3DLUTs.

    • Remi: First iteration to create processor really long due to parsing 3DLUT, the files are never in the cache on first run.

    • Remi: Interestingly, GPU Processor creation time greatly benefit from the a faster hashing algorithm, not just the base processor creation.

    • Kevin: These are the worst case scenarios, maybe not reflecting the reality of production. We typically don't merge all transforms into a single processor, so might not benefit as much from the PR as is shown here.

    • Kevin: For a typical pipeline we do a neutral grade step in one ColorSpaceTransform, a creative grade in another ColorSpaceTransform, etc, maybe less case for this PR. Hopefully common pipeline step like display rendering LUTs will be found in cache before even reaching the fallback step.

    • Kevin: How we handle grading at various step in the pipeline doesn't generally fit well in OCIO.

    • Zach: Thinking about usage of NamedTransform in this context, looking to work on a NamedTransform OFX plugin.

    • Group: Would be a great contribution!

    • Kevin: Wondering about the performance gain expected in more typical production pipeline where reasonable level of cache fallback hit can be expected. Need to weight the benefit for most users against the build complexity added.

    • License: ASWF should be consulted for the inclusion of a 3rd party library. xxHash uses a BSD-2-Clause license.

    • Doug: Group tend to agree the PR is useful, only objection is the added build complexity and potential build issues.

    • Doug: Mark proposed a simple hash algorithm but only 32 bits hash, this might have collision issues. In OCIO collision should be avoided at all cost, as it would be a serious bug.

    • Kevin: There is a trade off between including a known and good library like xxHash and dealing with potentially more build maintenance, versus spending lot of time and effort upfront picking a minimal implementation (few lines of codes) suiting our needs with hopefully less maintenance.

    • Zach: Compatibility and match of generated hash cross-platform is really useful, and already provided by MD5.

    • Cuneyt: Standard library ships with std::hash starting from C++11.

    • Doug: We already use it but only for small hash, large hash are currently handled with MD5.

    • Kevin: Discussing about issue #1666, for the use case of config fingerprinting, we need a resilient hash algorithm if we are going to use it as proof of configs matching.

    • Zach: Same need for processor for caching transforms.

    • Kevin: Want to know if this colour space definition is the same as the other config. Need a global config hash not dependent on the text formatting.

    • Doug: Change to xxHash still good for this usage. Need to loop over every possible processor in the config, compute the hash and accumulate.

    • Doug: Adding the config hash output to ociocheck, need to be cross-platform as already mentioned.

    • Zach: Investigating assigning hash to transform to improve perf, reduce duplication. What level of optimisation to use in this case?

    • Doug: Assume using the default level of optimisation would be good here.

    • Kevin: If we want to go further, can even compare configs with different reference space, using known role like aces_interchange. Can check whether two colour spaces in separate configs match even if defined relative to different references.

    • Remi: Interesting thought but maybe goes beyond the current scope and RFC.

    • Kevin: Agreed. If we compute hash based on processor apply, there are also other things to consider for cross-platform compatibility like fast fp-math compiler optimisations, etc.

    • Remi: Current PR with xxHash already maintain compatibility of hash cross-platform.

    • Kevin: But need to be careful what exact bit pattern is fed to the hashing algorithm and make sure this does not rely on any kind of optimisation.

    • Doug: Currently there is no optimisation like fast fp-math factored in the cache ID hash.

    • Kevin: If we were to rely on the processor apply for hashing in the future, we will need to investigate that more carefully.

    • Doug: xxHash gives the same results cross-platform. But if calculating hash on the result from a processor apply, might have some differences yes.

    • Doug: Everyone seems to be in favour for the PR, what about symbol hiding issues? Are there other issues?

    • Zach: Lot of what Kevin said about more robust transform hash interesting. 

    • Doug: Yes all agree. Going back to give Remi feedback.

    • Kevin: Key one would be to make sure there is no symbol leaking. We can push for a good implementation now, no need to solve all use cases at once. In favor if symbols are properly encapsulated.

    • Michael: Need final discussion and meeting with quorum.

    • Remi: Mark and Larry might have more ideas and opinions about simple hash algorithms fitting our use cases (128 bits, good performance and low collision probability).

    • Michael: Interested to investigate what OIIO uses for ImageBuf caching.

    • Doug: Hoping to merge for upcoming 2.2 release.

  • Archive feature:

    • Doug: Autodesk starting to investigate that.

    • Doug: Looking for 3rd party lib to do zip and archive compression, Zlib probably good candidate because it's widely used (eg. OpenEXR) and multi platform. Looking at minizip fork, handle zip archive manipulation, and it's modern fork minizip-ng.

    • Doug: Going to write follow-up on GitHub issue #1627, any thoughts?

    • Michael: Zlib pretty standard, don't think there will be objections using it.

    • Kevin: There are other candidates providing better performance and hardware specific optimisations.

    • Kevin: For OCIO use case and config archiving, probably don't need this level of performance, the archiving is not going to be used a lot. Unless it take 3 weeks to compress, we don't care, test case could be archiving ACES 1.2 config, if it takes less than 1 minute it's probably fine.

    • Doug: We are working on it.

  • GPU optimisation flags and UX:

    • Michael: There are a lot of flags for CPU and GPU optimisation, flags grouping, etc in the API.

    • Michael: Curious what people think on how / if these should be exposed in GUI. For example, if you have a renderer implementation in maya, would you considerer exposing optimisation flags to the user?

    • Doug: Are we talking about GPU only? Most of the flags are not specific to the GPU path.

    • Michael: Optimisation flags impacting ops created for processor that later becomes shader code. Ops optimisations could have significant impact on GPU performance.

    • Doug: This could indeed result in significant gains, but apply equally to CPU and GPU. 

    • Michael: Discussion less about differences between CPU and GPU, more about whether the potential gains would be large enough to be worth exposing flags to the users.

    • Doug: Don't think it's worth exposing to users because flags impact is not easy to appreciate. Need advanced user to make those decision, at that point maybe best to stick with environment variable than exposing in UI.

    • Kevin: This is the kind of thing people spend a lot of time to benchmark and test, shouldn't be exposed to normal users but only controlled by advanced users. Better use environment variables.

    • Doug: Some flags should be used with care. There is a flag to concatenate all 3DLUT together (OPTIMIZATION_COMP_LUT3D), users could inadvertently introduces precision issues downstream.

    • Michael: Good part of the OCIO API is about optimisation flags (environment variables, parameters in functions). More for test and debugging than general use?

    • Doug: Yes, mostly. There was a case recently where turning off an optimisation flag provided a temporary workaround for an OCIO bug found in nuke. Note this is now fixed in latest releases, PR #1614.

    • Remi: Is the point about simplifying the API and removing extra function parameters so that environment variables are the main entry point for optimisation specification?

    • Michael: Not necessarily, mostly wondering about the various use cases to adjust the optimisation flags. Using it for debugging make sense.

  • USD / Material X

    • Doug: Recent discussions on MaterialX and USD groups, about colour management and interactions between the projects and OCIO.

    • Doug: Doesn't really involve OCIO but these projects need colour management. There are debated points, for example in USD, if every prim define it's own colour space, potential to introduce a lot of conversions and hurt performances.

    • Doug: Little progress on this issue, from 3 or 4 years ago, no implementation yet for OCIO integration. Any thoughts on this?

    • Remi: Have been looking at USD recently, colour management is limited. In UsdPreviewSurface, input colour management uses a sourceColorspace token which only defines the transfer function (sRGB-piecewise or linear) to decode textures. Implementation like maya-usd then try to convert to the rendering space by deriving a Rec.709 to rendering space matrix with string matching. OCIO is not currently integrated. USDview supports OCIO for display rendering, recent PR by Apple to implement OCIO 2 GPU path for Metal and OpenGL which is great. Uses a DisplayViewTransform internally, input colour space is set to the default role if not provided which seem unusual.

    • Doug: Apple worked on Metal related PRs recently, have been providing some help related to dynamic properties implementation.