Page Comparison

FFmpeg is frequently used by different studios for encoding their media, however the documentation for ffmpeg is often poor, or cryptic so its often harder than it should be to come up with a good starting point. We are aiming to come up with recommendations for different scenarios as well as document what the different flags are doing with the aim to make this easier to get to a good baseline.

Overview

We are looking for recommendations for the following:

...

Where ffmpeg arguments, it would be great to document why we are using them, rather than ending up with a recipe. 

Presentation Links

Overall workflow

For this paper, we are assuming that we are encoding from a file-sequence of frames into a movie (rather than re-encoding a movie), but we are also assuming almost all of the colorspace work would be done outside of ffmpeg, with tools using the OCIO library. Examples could include nuke and oiio. Once we get to ffmpeg, the goal being that the pixel data we get in, should be as close as possible to the data we get out of ffmpeg. However, there are still quite a few areas that a ffmpeg user could go wrong, which we break down below. 

TODO: Find examples of overall workflow. 

Codec Selection.

h264

Key flags (see https://trac.ffmpeg.org/wiki/Encode/H.264 ) 

...

• Suggestions for max-bitrate?
• Suggestions for preset - ? slow
• Suggestions for tune

h265/HVEC

Support: https://caniuse.com/hevc (or https://www.chromium.org/audio-video ) currently no support for h265 on chrome or chromium based browsers.

...

• https://codecalamity.com/encoding-uhd-4k-hdr10-videos-with-ffmpeg/
• https://codecalamity.com/encoding-settings-for-hdr-4k-videos-using-10-bit-x265/
• https://support.frame.io/en/articles/4305241-creating-hdr-files-for-frame-io

VMAF

I did explore using VMAF - Video Multi-Method Assessment Fusion as a way to quantify the compression, the notes for setting this up are below, however I think we are going with a fairly high compression factor , so I think this is probably not really going to help us much.

...

https://ottverse.com/vmaf-ffmpeg-ubuntu-compilation-installation-usage-guide/ - building VMAF on ubuntu.

Color Preservation

Testing Methodology

Converting SMPTE color bars to the compressed movie, using ffmpeg to expand and then compare with OIIO. NOTE, for compression schemes that are not 444 we may need to mask the transitions.

...

Q: Currently focusing just on color matching in vs. out, but should also do EXR ACEScg in to resulting movie. Feels like we should also bless full pipeline, e.g.: Reference "Dailies script" https://github.com/jedypod/generate-dailies

Test Sources

SMPTE test chart: https://commons.wikimedia.org/wiki/File:SMPTE_Color_Bars_16x9.svg 

...

C: Do as clean a conversion from RGB to YUV as possible.

RGB/YCrCb Colorspace Conversion  

As a rule of thumb, we would like ffmpeg to do as little as possible in terms of color space conversion. i.e. what comes in goes out. The problem is that most of the codecs are doing some sort of RGB to YUV conversion (technically YCrCb). The notable exception is x264rgb (see above). For more information, see: https://trac.ffmpeg.org/wiki/colorspace 

For examples comparing these see: https://richardssam.github.io/ffmpeg-tests/tests/chip-chart-yuvconvert/compare.html

colormatrix filter

ffmpeg -y -i ../sourceimages/chip-chart-1080-noicc.png -sws_flags spline+accurate_rnd+full_chroma_int -vf "colormatrix=bt470bg:bt709" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -qscale:v 1 -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 ./chip-chart-yuvconvert/spline444colormatrix2.mp4

This is the most basic colorspace filtering. bt470bg is essentially part of the bt601 spec.  See: https://www.ffmpeg.org/ffmpeg-filters.html#colormatrix

colorspace filter

ffmpeg -y -i ../sourceimages/chip-chart-1080-noicc.png -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -qscale:v 1 -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 ./chip-chart-yuvconvert/spline444colorspace.mp4

Using colorspace filter, better quality filter, SIMD so faster too, can support 10-bit too.  The second part -vf "colorspace=bt709:iall=bt601-6-625:fast=1" encodes for the output being bt709, rather than the default bt601 matrix. iall=bt601-6-625 says to treat all the input (colorspace, primaries and transfer function) with the bt601-6-625 label). fast=1 skips gamma/primary conversion in a mathematically correct way.  See:  https://ffmpeg.org/ffmpeg-filters.html#colorspace

libswscale filter

ffmpeg -y -i ../sourceimages/chip-chart-1080-noicc.png -sws_flags spline+accurate_rnd+full_chroma_int+full_chroma_inp -vf "scale=in_range=full:in_color_matrix=bt709:out_range=tv:out_color_matrix=bt709" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -qscale:v 1 -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 ./chip-chart-yuvconvert/spline444out_color_matrix.mp4

Using the libswscale library. Seems similar to colorspace, but with image resizing, and levels built in.  https://www.ffmpeg.org/ffmpeg-filters.html#scale-1

This is the recommended filter.

Color Metadata NCLC

The above gets the underlying data stored correctly, but there are additional metadata flags that can be set that are interpreted by some players, these are the NCLC color tags for color primaries, transfer function and conversion matrix.  This is defined as a ISO spec here (see https://www.iso.org/standard/73412.html). The numbers below are part of the definition.

...

The docs are pretty sparse for this, some of the better info is FFmpeg/pixfmt.h at master · FFmpeg/FFmpeg · GitHub

links:

Color Range

Uses the flag -color_range e.g. -color_range 1 or -color_range tv

Color Space

This defines the YUV colorspace type, as defined by ISO/IEC 23091-2_2019 subclause 8.3

...

Color Primaries

Chromaticity coordinates of the source primaries. These values match the ones defined by ISO/IEC 23091-2_2019 subclause 8.1 and ITU-T H.273.

...

Color Transfer Characteristic aka color_trc

These values match the ones defined by ISO/IEC 23091-2_2019 subclause 8.2.

...