FFmpeg is frequently used by different studios for encoding their media, however the documentation for ffmpeg is often poor, or cryptic so its often harder than it should be to come up with a good starting point. We are aiming to come up with recommendations for different scenarios as well as document what the different flags are doing with the aim to make this easier to get to a good baseline.
...
TODO: Find examples of overall workflow.
Codec Selection.
| Name | Target Usage | Source | ffmpeg flags | Description | Size |
|---|---|---|---|---|---|
| x264 -pix_fmt yuv420p | Proxy playback, for web review, and non-color critical workflows, e.g. animation, modeling, etc. | This should be a lightweight compression, capable of supporting HD with a reasonable bit-rate, hopefully supporting a wide range of web browsers. Final review, e.g. lighting | |||
| x264 -pix_fmt yuv444p10le | Final review, e.g. lighting | https://trac.ffmpeg.org/wiki/colorspace | -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 | ||
| x264rgb | x264rgbFinal review, e.g. lighting | This is a variant of the above, its essentially using x264, but not converting to YCrCb. | |||
| Prores 4444 | For delivery to editorial | -c:v prores_ks -profile:v 4444 -qscale:v 1 -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 | |||
| -profile:v 4444 is equivalent to -profile:v 4 | |||||
| shotgun_diy_encode | https://support.shotgunsoftware.com/hc/en-us/articles/219030418-Do-it-yourself-DIY-transcoding', | -vcodec libx264 -pix_fmt yuv420p -g 30 -vprofile high -bf 0 -crf 2 | |||
DnxHD | For delivery to editorial | ||||
| Prores 422 HQ | For delivery to editorial | Some FFMpeg commands I need to remember for converting footage for video editing. http://bit.ly/vidsnippets · GitHub |
| ||
Note the -profile:v 3 is equivalent to -profile:v hq | |||||
x264
Key flags (see https://trac.ffmpeg.org/wiki/Encode/H.264 )
...
- Suggestions for max-bitrate?
- Suggestions for preset - ? slow
- Suggestions for tune
Color Preservation
Testing Methodology
Converting SMPTE color bars to the compressed movie, using ffmpeg to expand and then compare with OIIO. NOTE, for compression schemes that are not 444 we may need to mask the transitions.
Testing loading the compressed movie in to RV, Firefox, VLC, Avid, resolve, , to compare the resulting color transformation - not sure if there is a procedural way to run this?
For the tests below we are assuming that other tools are being used (e.g. oiiotool) to convert the rendered frames into an intermediate file (e.g. PNG) in the target color-space.
Q: Currently focusing just on color matching in vs. out, but should also do EXR ACEScg in to resulting movie. Feels like we should also bless full pipeline, e.g.: Reference "Dailies script" https://github.com/jedypod/generate-dailies
Test Sources
SMPTE test chart: https://commons.wikimedia.org/wiki/File:SMPTE_Color_Bars_16x9.svg
Download image sequence from: https://senkorasic.com/testmedia/ -
Explore netflix: https://opencontent.netflix.com/
Test Results:
taurich.org/encodingTests/results.html
Links
- An excellent starting point for this is: https://trac.ffmpeg.org/wiki/colorspace
- https://github.com/RxLaboratory/DuME/blob/master/src/FFmpeg_COLORS.md
- https://medium.com/invideo-io/talking-about-colorspaces-and-ffmpeg-f6d0b037cc2f
- https://docs.nvidia.com/video-technologies/video-codec-sdk/ffmpeg-with-nvidia-gpu/
- https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.1886-0-201103-I!!PDF-E.pdf - the BT1886 spec, esentially gamma 2.4 rec709 primaries.
- https://github.com/bbc/qtff-parameter-editor - A BBC open source app, for setting quicktime NCLC attributes.
- https://vimeo.com/349868875 - Video from baselight, reviewing setting the right NCLC tags for apple colorsync.
Notes
The big issue here is that by default if you start converting images to another format, and ffmpeg cannot determine the colorspace it will default to bt601. So many of the flags below are to:
A: Tell ffmpeg that the source media is in fact bt709
B: Add the metadata to the output, so that other future conversions also know how to convert it back.
C: Do as clean a conversion from RGB to YUV as possible.
Example Usages
Need to specify the build of ffmpeg. – ? - and specify build flags. -loglevel trace
...
Description
...
-pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colormatrix=bt470bg:bt709" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1...
The sws_flags are needed for the RGB to YUV conversion.
-color_range 1 # mpeg see: FFmpeg/pixfmt.h at master · FFmpeg/FFmpeg · GitHub
-colorspace 1 # BT709 FFmpeg/pixfmt.h at master · FFmpeg/FFmpeg · GitHub
-color_primaries 1 # BT709 FFmpeg/pixfmt.h at master · FFmpeg/FFmpeg · GitHub
-color_trc 1 # bt709 FFmpeg/pixfmt.h at master · FFmpeg/FFmpeg · GitHub - Color Transfer Characteristics.
...
VMAF
I did explore using VMAF - Video Multi-Method Assessment Fusion as a way to quantify the compression, the notes for setting this up are below, however I think we are going with a fairly high compression factor , so I think this is probably not really going to help us much.
https://github.com/Netflix/vmaf
https://jina-liu.medium.com/a-practical-guide-for-vmaf-481b4d420d9c
https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652
https://ottverse.com/vmaf-ffmpeg-ubuntu-compilation-installation-usage-guide/ - building VMAF on ubuntu.
Color Preservation
Testing Methodology
Converting SMPTE color bars to the compressed movie, using ffmpeg to expand and then compare with OIIO. NOTE, for compression schemes that are not 444 we may need to mask the transitions.
Testing loading the compressed movie in to RV, Firefox, VLC, Avid, resolve, , to compare the resulting color transformation - not sure if there is a procedural way to run this?
For the tests below we are assuming that other tools are being used (e.g. oiiotool) to convert the rendered frames into an intermediate file (e.g. PNG) in the target color-space.
Q: Currently focusing just on color matching in vs. out, but should also do EXR ACEScg in to resulting movie. Feels like we should also bless full pipeline, e.g.: Reference "Dailies script" https://github.com/jedypod/generate-dailies
Test Sources
SMPTE test chart: https://commons.wikimedia.org/wiki/File:SMPTE_Color_Bars_16x9.svg
Download image sequence from: https://senkorasic.com/testmedia/ -
Explore netflix: https://opencontent.netflix.com/
Test Results:
taurich.org/encodingTests/results.html
Links
- An excellent starting point for this is: https://trac.ffmpeg.org/wiki/colorspace
- https://github.com/RxLaboratory/DuME/blob/master/src/FFmpeg_COLORS.md
- https://medium.com/invideo-io/talking-about-colorspaces-and-ffmpeg-f6d0b037cc2f
- https://docs.nvidia.com/video-technologies/video-codec-sdk/ffmpeg-with-nvidia-gpu/
- https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.1886-0-201103-I!!PDF-E.pdf - the BT1886 spec, esentially gamma 2.4 rec709 primaries.
- https://github.com/bbc/qtff-parameter-editor - A BBC open source app, for setting quicktime NCLC attributes.
- https://vimeo.com/349868875 - Video from baselight, reviewing setting the right NCLC tags for apple colorsync.
Notes
The big issue here is that by default if you start converting images to another format, and ffmpeg cannot determine the colorspace it will default to bt601. So many of the flags below are to:
A: Tell ffmpeg that the source media is in fact bt709
B: Add the metadata to the output, so that other future conversions also know how to convert it back.
C: Do as clean a conversion from RGB to YUV as possible.
RGB/YCrVb Colorspace Conversion
As a rule of thumb, we would like ffmpeg to do as little as possible in terms of color space conversion. i.e. what comes in goes out. The problem is that most of the codecs are doing some sort of RGB to YUV conversion (technically YCrCb). The notable exception is x264rgb (see above). For more information, see: https://trac.ffmpeg.org/wiki/colorspace and https://ffmpeg.org/ffmpeg-filters.html#colorspace
While it is possible to do the encoding outside of ffmpeg (see later).
colormatrix filter
ffmpeg -y -i ../sourceimages/chip-chart-1080-noicc.png -sws_flags spline+accurate_rnd+full_chroma_int
...
-vf
...
"
...
colormatrix=bt470bg:bt709
...
" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -qscale:v 1 -color_range
...
1
...
-colorspace
...
1
...
-color_primaries
...
1
...
-color_trc
...
Compression quality
Testing Methodology
VMAF
I did explore using VMAF - Video Multi-Method Assessment Fusion as a way to quantify the compression, the notes for setting this up are below, however I think we are going with a fairly high compression factor , so I think this is probably not really going to help us much.
https://github.com/Netflix/vmaf
https://jina-liu.medium.com/a-practical-guide-for-vmaf-481b4d420d9c
https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652
https://ottverse.com/vmaf-ffmpeg-ubuntu-compilation-installation-usage-guide/ - building VMAF on ubuntu.
RGB/YCrVb Colorspace Conversion
As a rule of thumb, we would like ffmpeg to do as little as possible in terms of color space conversion. i.e. what comes in goes out. The problem is that most of the codecs are doing some sort of RGB to YUV conversion (technically YCrCb). The notable exception is x264rgb (see above). For more information, see: https://trac.ffmpeg.org/wiki/colorspace and https://ffmpeg.org/ffmpeg-filters.html#colorspace
While it is possible to do the encoding outside of ffmpeg (see later). Its easier if you do the encoding inside, using flags like:
-sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1"
Metadata flags -sws_flags spline+accurate_rnd+full_chroma_int helps with the YCrCb conversion. Its using the swscale filter, which has a number of options:
TODO: why spline? - Filtering applied to the decimation filtering to go from RGB to 444 to 422.
accurate_rnd - enables accurate rounding
full_chroma_int - Enable full chroma interpolation.
full_chroma_p?
The second part -vf "colorspace=bt709:iall=bt601-6-625:fast=1" encodes for the output being bt709, rather than the default bt601 matrix. iall=bt601-6-625 says to treat all the input (colorspace, primaries and transfer function) with the bt601-6-625 label). fast=1 skips gamma/primary conversion in a mathematically correct way.
TODO, CONFIRM - THIS IS TO GET THE CONVERSION TO TREAT THE RESULTING DATA AS BT709 rather than BT601? Why does fast=1 work? Surely I do want it to do all the flags?
...
1 ./chip-chart-yuvconvert/spline444colormatrix2.mp4
This is the most basic colorspace filtering. bt470bg is essentially part of the bt601 spec.
colorspace filter
ffmpeg -y -i ../sourceimages/chip-chart-1080-noicc.png -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -qscale:v 1 -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 ./chip-chart-yuvconvert/spline444colorspace.mp4
Using colorspace filter, better quality filter, SIMD so faster too, can support 10-bit too. The second part -vf "colorspace=bt709:iall=bt601-6-625:fast=1" encodes for the output being bt709, rather than the default bt601 matrix. iall=bt601-6-625 says to treat all the input (colorspace, primaries and transfer function) with the bt601-6-625 label). fast=1 skips gamma/primary conversion in a mathematically correct way.
libavscale filter
ffmpeg -y -i ../sourceimages/chip-chart-1080-noicc.png -sws_flags spline+accurate_rnd+full_chroma_int+full_chroma_inp -vf "scale=in_range=full:in_color_matrix=bt709:out_range=tv:out_color_matrix=bt709" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -qscale:v 1 -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 ./chip-chart-yuvconvert/spline444out_color_matrix.mp4
Using the libavscale library. Seems similar to colorspace, but with image resizing, and levels built in. https://www.ffmpeg.org/ffmpeg-filters.html#scale-1
This is the recommended filter.
Color Metadata NCLC
The above gets the underlying data stored correctly, but there are additional metadata flags that can be set that are interpreted by some players, these are the NCLC color tags for color primaries, transfer function and conversion matrix. This is defined as a ISO spec here (see https://www.iso.org/standard/73412.html). The numbers below are part of the definition.
...
The docs are pretty sparse for this, some of the better info is FFmpeg/pixfmt.h at master · FFmpeg/FFmpeg · GitHub
Color Range
Uses the flag -color_range e.g. -color_range 1 or -color_range tv
| Numeric value | String Values | Numeric range | Notes |
|---|---|---|---|
| 0 | Unspecified | ||
| 1 | tv mpeg | 16-135 | This is the default. |
| 2 | pc jpeg | 0-255 |
Color Space
This defines the YUV colorspace type, as defined by ISO/IEC 23091-2_2019 subclause 8.3
...
| Numeric Value | String values | Description |
|---|---|---|
| 0 | rgb | |
| 1 | bt709 | Typically set it to this. |
| 2 | unspecified | |
| 9 | bt2020nc bt2020_ncl | ITU-R BT2020 non-constant luminance system |
| 10 | bt2020c bt2020_cl | ITU-R BT2020 constant luminance system |
Color Primaries
Chromaticity coordinates of the source primaries. These values match the ones defined by ISO/IEC 23091-2_2019 subclause 8.1 and ITU-T H.273.
...
| Numeric Value | String Values | Description |
|---|---|---|
| 1 | bt709 | |
| 9 | bt2020 |
Color Transfer Characteristic aka color_trc
These values match the ones defined by ISO/IEC 23091-2_2019 subclause 8.2.
...