Page Comparison

FFmpeg is frequently used by different studios for encoding their media, however the documentation for ffmpeg is often poor, or cryptic so its often harder than it should be to come up with a good starting point. We are aiming to come up with recommendations for different scenarios as well as document what the different flags are doing with the aim to make this easier to get to a good baseline.

Overview

We are looking for recommendations for the following:

• Best color preservation for output to:
  • Web, OSX, IOS and Windows.
  • Common applications: e.g. RV, Nuke.
  • Rec709 and sRGB displays to start with, but eventually, P3, rec2020 and HDR displays.
  • Web browser - Firefox reviewing mp4. - use firefox plugin.
  • RV
• Codec recommendations for:
  • Proxy H264 playback (e.g. web streaming), should be setup for web streaming.
  • Animation/Modelling/Layout movie playback. - somewhat lower quality playback, but should always provide smooth motion.
  • Lookdev/lighting/compositing movie playback - should have excellent color fidelity and minimal encoding artifacts
    • Should any filmlook be baked in, or should we assume that is always applied during viewing.
    • How much should we be able to adjust color and have the image hold up? (Or rely on exr's for that?).
  • Export to editorial.
  • High-resolution or frame rate - e.g. 4k, 8k, 60fps, 120fps.
  • Stereo or VR.
• Q: Which container should we be considering: mov, mp4, mxf.

Where ffmpeg arguments, it would be great to document why we are using them, rather than ending up with a recipe. 

• , mp4, mxf.

Where ffmpeg arguments, it would be great to document why we are using them, rather than ending up with a recipe. 

Overall workflow

For this paper, we are assuming that we are encoding from a file-sequence of frames into a movie (rather than re-encoding a movie), but we are also assuming almost all of the colorspace work would be done outside of ffmpeg, with tools using the OCIO library. Examples could include nuke and oiio. Once we get to ffmpeg, the goal being that the pixel data we get in, should be as close as possible to the data we get out of ffmpeg. However, there are still quite a few areas that a ffmpeg user could go wrong, which we break down below. 

TODO: Find examples of overall workflow. 

Codec Selection.

x264

x264rgb

Prores

DnxHD

Color Preservation

Testing Methodology

Converting SMPTE color bars to the compressed movie, using ffmpeg to expand and then compare with OIIO. NOTE, for compression schemes that are not 444 we may need to mask the transitions.

...

Q: Currently focusing just on color matching in vs. out, but should also do EXR ACEScg in to resulting movie. Feels like we should also bless full pipeline, e.g.: Reference "Dailies script" https://github.com/jedypod/generate-dailies

Test Sources

SMPTE test chart: https://commons.wikimedia.org/wiki/File:SMPTE_Color_Bars_16x9.svg

...

Explore netflix: https://opencontent.netflix.com/

Test Results:

taurich.org/encodingTests/results.html

Links

• An excellent starting point for this is: https://trac.ffmpeg.org/wiki/colorspace
• https://github.com/RxLaboratory/DuME/blob/master/src/FFmpeg_COLORS.md
• https://medium.com/invideo-io/talking-about-colorspaces-and-ffmpeg-f6d0b037cc2f
• https://docs.nvidia.com/video-technologies/video-codec-sdk/ffmpeg-with-nvidia-gpu/
• https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.1886-0-201103-I!!PDF-E.pdf - the BT1886 spec, esentially gamma 2.4 rec709 primaries.
• https://github.com/bbc/qtff-parameter-editor - A BBC open source app, for setting quicktime NCLC attributes.widget
• urlhttps://vimeo.com/349868875height300 - Video from baselight, reviewing setting the right NCLC tags for apple colorsync.

Notes

The big issue here is that by default if you start converting images to another format, and ffmpeg cannot determine the colorspace  it will default to bt601. So many of the flags below are to:

...

C: Do as clean a conversion from RGB to YUV as possible.

Example Usages

Need to specify the build of ffmpeg. – ? - and specify build flags. -loglevel trace

-pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colormatrix=bt470bg:bt709" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1

-pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 

Compression quality

Testing Methodology

VMAF

I did explore using VMAF - Video Multi-Method Assessment Fusion as a way to quantify the compression, the notes for setting this up are below, however I think we are going with a fairly high compression factor , so I think this is probably not really going to help us much.

...

https://ottverse.com/vmaf-ffmpeg-ubuntu-compilation-installation-usage-guide/ - building VMAF on ubuntu.

ffmpeg flags

Codec

x264

x264rgb

Prores

DnxHD

RGB/YCrVb Colorspace Conversion  

As a rule of thumb, we would like ffmpeg to do as little as possible in terms of color space conversion. i.e. what comes in goes out. The problem is that most of the codecs are doing some sort of RGB to YUV conversion (technically YCrCb). The notable exception is x264rgb (see above). For more information, see: https://trac.ffmpeg.org/wiki/colorspace and https://ffmpeg.org/ffmpeg-filters.html#colorspace

...

TODO, CONFIRM - THIS IS TO GET THE CONVERSION TO TREAT THE RESULTING DATA AS BT709 rather than BT601? Why does fast=1 work? Surely I do want it to do all the flags?

Color Metadata

The above gets the underlying data stored correctly, but there are additional metadata flags that can be set that are interpreted by some players, these are the NCLC color tags for color primaries, transfer function and conversion matrix.  This is defined as a ISO spec here (https://www.iso.org/standard/57794.html which sadly is paywalled). The numbers below are part of the definition.

...

• https://vimeo.com/349868875 
•  https://developer.apple.com/documentation/avfoundation/media_assets_and_metadata/sample-level_reading_and_writing/tagging_media_with_video_color_information 
• https://www.iso.org/standard/73412.html - Note this has a link to the download of the earlier version of the doc, the latest and paywalled version is here: https://www.iso.org/standard/57794.html

Web Browser Deliverables

How should we be encoding content for a web browser.

...

Gamma 2.4

There is not a color_trc flag for gamma 2.4, the only option that exists for OSX is a cheat 

...

ffmpeg -y -i chip-chart-1080.png -c:v libx264 -pix_fmt yuv444p -qscale:v 1 -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 2 -movflags write_colr+write_gama -mov_gamma 2.4 test2-h264-ffmpeg-yuv444p-gamma24.mp4

Full range vs. legal range

Typically x264 (and other codecs) are following the video standard that lumance is scaled to the range 16-235. This has a history from early signaling where 236-255 were used for signaling and 0-15 to avoid any noise in the low end (some of the logic was derived from analog video) 

...

• https://news.ycombinator.com/item?id=20036710

Other links

https://github.com/bbc/qtff-parameter-editor