Page Comparison

FFmpeg is frequently used by different studios for encoding their media, however the documentation for ffmpeg is often poor, or cryptic so its often harder than it should be to come up with a good starting point. We are aiming to come up with recommendations for different scenarios as well as document what the different flags are doing with the aim to make this easier to get to a good baseline.

Overview

We are looking for recommendations for the following:

• Best color preservation for output to:
  • Web, OSX, IOS and Windows.
  • Common applications: e.g. RV, Nuke.
  • Rec709 and sRGB displays to start with, but eventually, P3, rec2020 and HDR displays.
  • Web browser - Firefox reviewing mp4. - use firefox plugin.
  • RV
• Codec recommendations for:
  • Proxy H264 playback (e.g. web streaming), should be setup for web streaming.
  • Animation/Modelling/Layout movie playback. - somewhat lower quality playback, but should always provide smooth motion.
  • Lookdev/lighting/compositing movie playback - should have excellent color fidelity and minimal encoding artifacts
    • Should any filmlook be baked in, or should we assume that is always applied during viewing.
    • How much should we be able to adjust color and have the image hold up? (Or rely on exr's for that?).
  • Export to editorial.
  • High-resolution or frame rate - e.g. 4k, 8k, 60fps, 120fps.
  • Stereo or VR.
• Q: Which container should we be considering: mov, mp4, mxf.

Where ffmpeg arguments, it would be great to document why we are using them, rather than ending up with a recipe. 

Presentation Links

Overall workflow

For this paper, we are assuming that we are encoding from a file-sequence of frames into a movie (rather than re-encoding a movie), but we are also assuming almost all of the colorspace work would be done outside of ffmpeg, with tools using the OCIO library. Examples could include nuke and oiio. Once we get to ffmpeg, the goal being that the pixel data we get in, should be as close as possible to the data we get out of ffmpeg. However, there are still quite a few areas that a ffmpeg user could go wrong, which we break down below. 

TODO: Find examples of overall workflow. 

Codec Selection.

...

x264rgb

...

DnxHD

...

-pix_fmt yuv422p10le -c:v prores_ks -profile:v 3 -vendor ap10 -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1

...

Note the -vendor ap10 part below is only needed if working with Final Cut, but it does no harm otherwise.

-profile:v 3 is equivalent to -profile:v hq

x264

Key flags (see FFmpeg is frequently used by different studios for encoding their media, however the documentation for ffmpeg is often poor, or cryptic so its often harder than it should be to come up with a good starting point. We are aiming to come up with recommendations for different scenarios as well as document what the different flags are doing with the aim to make this easier to get to a good baseline.

Overview

We are looking for recommendations for the following:

• Best color preservation for output to:
  • Web, OSX, IOS and Windows.
  • Common applications: e.g. RV, Nuke.
  • Rec709 and sRGB displays to start with, but eventually, P3, rec2020 and HDR displays.
  • Web browser - Firefox reviewing mp4. - use firefox plugin.
  • RV
• Codec recommendations for:
  • Proxy H264 playback (e.g. web streaming), should be setup for web streaming.
  • Animation/Modelling/Layout movie playback. - somewhat lower quality playback, but should always provide smooth motion.
  • Lookdev/lighting/compositing movie playback - should have excellent color fidelity and minimal encoding artifacts
    • Should any filmlook be baked in, or should we assume that is always applied during viewing.
    • How much should we be able to adjust color and have the image hold up? (Or rely on exr's for that?).
  • Export to editorial.
  • High-resolution or frame rate - e.g. 4k, 8k, 60fps, 120fps.
  • Stereo or VR.
• Q: Which container should we be considering: mov, mp4, mxf.

Where ffmpeg arguments, it would be great to document why we are using them, rather than ending up with a recipe. 

Presentation Links

Overall workflow

For this paper, we are assuming that we are encoding from a file-sequence of frames into a movie (rather than re-encoding a movie), but we are also assuming almost all of the colorspace work would be done outside of ffmpeg, with tools using the OCIO library. Examples could include nuke and oiio. Once we get to ffmpeg, the goal being that the pixel data we get in, should be as close as possible to the data we get out of ffmpeg. However, there are still quite a few areas that a ffmpeg user could go wrong, which we break down below. 

TODO: Find examples of overall workflow. 

Codec Selection.

h264

Key flags (see https://trac.ffmpeg.org/wiki/Encode/H.264 ) 

• -crf 23 - This is the constant rate factor, controlling the default quality (see: https://slhck.info/video/2017/02/24/crf-guide.html ) where -crf 0 is uncompressed. By default this is set to 23, which is probably good enough for our needs.
• -qp 23 - Quantization Parameter - it is recommended that you do not use this, in preference to -crf above (see: https://slhck.info/video/2017/03/01/rate-control.html )
• -preset slower - https://trac.ffmpeg.org/wiki/Encode/H.264#FAQ
• -tune film - Optionally use the tune option to change settings based on specific inputs - https://trac.ffmpeg.org/wiki/Encode/H.264#FAQ - see also: https://superuser.com/questions/564402/explanation-of-x264-tune I suspect that we would want to use one of:
  1. -tune film    good for live action content.
  2. -tune animation good for animated content with areas of flat colors.
  3. -tune grain   good for live action content where you want to preserve the grain as much as possible.
• -qscale:v 1 - Generic quality scale flag: https://www.ffmpeg.org/ffmpeg.html#toc-Main-options - not sure if its needed?

-preset slow -crf 18   level 4 profile high.

RV - 10bit YUV444.

Better to stick with yuv – 

FFMPEG.

TODO:

• Suggestions for max-bitrate?
• Suggestions for preset - ? slow
• Suggestions for tune

h265/HVEC

Support: https://caniuse.com/hevc (or https://www.chromium.org/audio-video ) currently no support for h265 on chrome or chromium based browsers.

links:

• https://codecalamity.com/encoding-uhd-4k-hdr10-videos-with-ffmpeg/
• https://codecalamity.com/encoding-settings-for-hdr-4k-videos-using-10-bit-x265/
• https://support.frame.io/en/articles/4305241-creating-hdr-files-for-frame-io
• https://stackoverflow.com/questions/69251960/how-can-i-encode-rgb-images-into-hdr10-videos-in-ffmpeg-command-line
• https://brandur.org/fragments/ffmpeg-h265

HDR

• https://developer.apple.com/av-foundation/High-Dynamic-Range-Metadata-for-Apple-Devices.pdf
• https://www.avsforum.com/threads/open-source-video-testing-calibration-patterns.2944378/
• https://github.com/test-full-band/tfb-video/releases
• https://trev16.hatenablog.com/entry/2021/07/23/145725 – good site for HDR test sites.

ProRes

There are four Prores encoders, Prores, Prores_ks, Prores_aw and now with ffmpeg 5 VideoToolBox Prores, which is a hardware based OSX M1 encoder/decoder. 

From https://trac.ffmpeg.org/wiki/Encode/H.264 ) 

• -crf 23 - This is the constant rate factor, controlling the default quality (see: https://slhck.info/video/2017/02/24/crf-guide.html ) where -crf 0 is uncompressed. By default this is set to 23, which is probably good enough for our needs.
• -qp 23 - Quantization Parameter - it is recommended that you do not use this, in preference to -crf above (see: https://slhck.info/video/2017/03/01/rate-control.html )
• -preset slower - https://trac.ffmpeg.org/wiki/Encode/H.264#FAQ
• -tune film - Optionally use the tune option to change settings based on specific inputs - https://trac.ffmpeg.org/wiki/Encode/H.264#FAQ
• -qscale:v 1 - Generic quality scale flag: https://www.ffmpeg.org/ffmpeg.html#toc-Main-options - not sure if its needed?

-preset slow -crf 18   level 4 profile high.

RV - 10bit YUV444.

Better to stick with yuv – 

FFMPEG.

TODO:

...

VFX the recommendation is to use Prores_ks with -profile:v 3 and the qscale of 11

Options that can be used include:

• -profile:v values can be one of.
  • proxy (0)
  • lt (1)
  • standard (2)
  • hq (3)
  • 4444 (4)
  • 4444xq (5)
• -qscale:v between values of 9 - 13 give a good result, 0 being best.
• -vendor apl0 - tricks the codec into believing its from an Apple codec.

Using this with the usual color space flags, seems to work well with the exception of ffmpeg itself, which needs the flags:-vf scale=in_color_matrix=bt709:out_color_matrix=bt709 added to the command to ensure the right input colorspace is recognised, e.g.:

ffmpeg.exe -i INPUTFILE.mov -compression_level 10 -pred mixed -pix_fmt rgba64be -sws_flags spline+accurate_rnd+full_chroma_int -vframes 1 -vf scale=in_color_matrix=bt709:out_color_matrix=bt709 OUTPUTFILE.png

However, other encoders seem to be recognised correctly, so there is clearly some metadata missing. I did try using the prores_metadata filter to try adding some additional parameters, but it didnt seem to help.

ffmpeg.exe -i ./chip-chart-yuvconvert\basicnclc.mov -c copy -bsf:v prores_metadata=color_primaries=bt709:color_trc=bt709:colorspace=bt709 chip-chart-yuvconvert\basicnclcmetadata.mov

TODO:

• Figure out the missing metadata.
• Wedge qscale values
• Do some colorspace tests with different qscale values to see where color breaks down.

VMAF

I did explore using VMAF - Video Multi-Method Assessment Fusion as a way to quantify the compression, the notes for setting this up are below, however I think we are going with a fairly high compression factor , so I think this is probably not really going to help us much.

...

Q: Currently focusing just on color matching in vs. out, but should also do EXR ACEScg in to resulting movie. Feels like we should also bless full pipeline, e.g.: Reference  Reference "Dailies script" https://github.com/jedypod/generate-dailies

Test Sources

SMPTE test chart: https://commons.wikimedia.org/wiki/File:SMPTE_Color_Bars_16x9.svg 

...

Explore netflix: https://opencontent.netflix.com/

Test Results:

taurich.org/encodingTests/results.html

Links

• An excellent starting point for this is: https://trac.ffmpeg.org/wiki/colorspace
• https://github.com/RxLaboratory/DuME/blob/master/src/FFmpeg_COLORS.md
• https://medium.com/invideo-io/talking-about-colorspaces-and-ffmpeg-f6d0b037cc2f
• https://docs.nvidia.com/video-technologies/video-codec-sdk/ffmpeg-with-nvidia-gpu/
• https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.1886-0-201103-I!!PDF-E.pdf - the BT1886 spec, esentially gamma 2.4 rec709 primaries.

Notes

The big issue here is that by default if you start converting images to another format, and  and ffmpeg cannot determine the colorspace  colorspace  it will default to bt601. So many of the flags below are to:

...

C: Do as clean a conversion from RGB to YUV as possible.

RGB/YCrCb Colorspace Conversion  

As a rule of thumb, we would like ffmpeg to do as little as possible in terms of color space conversion. i.e. what comes in goes out. The problem is that most of the codecs are doing some sort of RGB to YUV conversion (technically YCrCb). The notable exception is x264rgb (see above). For more information, see: https://trac.ffmpeg.org/wiki/colorspace 

For examples comparing these see: https://richardssam.github.io/ffmpeg-tests/tests/chip-chart-yuvconvert/compare.html

colormatrix filter

ffmpeg -y -i ../sourceimages/chip-chart-1080-noicc.png -sws_flags spline+accurate_rnd+full_chroma_int -vf "colormatrix=bt470bg:bt709" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -qscale:v 1 -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 ./chip-chart-yuvconvert/spline444colormatrix2.mp4

This is the most basic colorspace filtering. bt470bg is essentially part of the bt601 spec.  See: https://www.ffmpeg.org/ffmpeg-filters.html#colormatrix

colorspace filter

ffmpeg -y -i ../sourceimages/chip-chart-1080-noicc.png -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -qscale:v 1 -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 ./chip-chart-yuvconvert/spline444colorspace.mp4

Using colorspace filter, better quality filter, SIMD so faster too, can support 10-bit too.  The second part -vf "colorspace=bt709:iall=bt601-6-625:fast=1" encodes for the output being bt709, rather than the default bt601 matrix. iall=bt601-6-625 says to treat all the input (colorspace, primaries and transfer function) with the bt601-6-625 label). fast=1 skips gamma/primary conversion in a mathematically correct way.  See:  https://ffmpeg.org/ffmpeg-filters.html#colorspace

libswscale filter

ffmpeg -y -i ../sourceimages/chip-chart-1080-noicc.png -sws_flags spline+accurate_rnd+full_chroma_int+full_chroma_inp -vf "scale=in_range=full:in_color_matrix=bt709:out_range=tv:out_color_matrix=bt709" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -qscale:v 1 -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 ./chip-chart-yuvconvert/spline444out_color_matrix.mp4

Using the libswscale library. Seems similar to colorspace, but with image resizing, and levels built in.  resizing, and levels built in.  https://www.ffmpeg.org/ffmpeg-filters.html#scale-1

This is the recommended filter.

Color Metadata NCLC/NCLX

The above gets the underlying data stored correctly, but there are additional metadata flags that can be set that are interpreted by some players, these are the NCLC color tags for color primaries, transfer function and conversion matrix.  This is defined as a ISO spec here (see https://www.ffmpegiso.org/ffmpeg-filters.html#scale-1

This is the recommended filter.

Color Metadata NCLC

The above gets the underlying data stored correctly, but there are additional metadata flags that can be set that are interpreted by some players, these are the NCLC color tags for color primaries, transfer function and conversion matrix.  This is defined as a ISO spec here (see https://www.iso.org/standard/73412.html). The numbers below are part of the definition.standard/73412.html). The numbers below are part of the definition.

NCLC stands for Non-Consistent Luminance Coding, a brief overview of its history is here. For MP4 files, its also known as NCLX. Additionally this metadata can also be represented in the h264 metadata stream in the video usability Information (VUI) block. 

You can read the metadata using mp4box.js which is a visual browser of the mp4 metadata, and look at moov/trak/mdia/minf/stbl/stsd/avc1/colr

NOTE: None of the flags below affect the encoding of the source imagery, they are meant to be used to guide how the mp4 file is decoded.

...

Color Transfer Characteristic aka color_trc

...

NOTE: -color_trc 1 - is not bt1886, but is actually the camera gamma, so has a gamma of ~1.95 rather than the 2.4 that is defined by bt1886. In order to get a gamma 2.4, you will need to use a quicktime hack (see below), but this only works on OSX. However, we suspect that chrome ignores the setting (see the following tests).

...