A Journey by way of Colour House with FFmpeg | by Canva Engineering | Apr, 2023

Video

For many who wish to perceive colour areas, remodel movies from one colour area to a different, or learn how I nearly went loopy figuring out why my Canva-generated movies appeared barely off, color-wise

by Sven Schindler

Hi there @cop a person reported that their exported video seems to be much less saturated in comparison with the way it seems to be like on the editor.

That is the sentence that began all of it. A person needed to export a lot of colourful Canva slides to a video file. I only recently joined Canva’s video group and thought this could be a fantastic starter activity. I imply, we’re speaking about changing a bunch of pictures to a video and apparently, there’s a minor colour area difficulty. No less than that’s what I assumed, with out actually greedy the complexity behind colour areas. Oh gosh, what had I executed!

My first naive method was, in fact, to seek out out whether or not another person on the web had the identical or an analogous downside. Fortunately, my superb workforce gave me a great place to start out. “Simply attempt to convert a picture to a video within the Rec. 709 colour area utilizing FFmpeg and also you’ll see what the issue is.” We’ll discuss what Rec. 709 is later within the article.

After a little bit of googling, I found the next wonderful weblog publish: Talking About Colorspaces and FFmpeg. Fortunately, the publish didn’t simply discuss how you can remodel colour areas with FFmpeg but in addition highlighted an analogous difficulty about pale movies generated utilizing pictures. The next command was supposed to repair all of it.

ffmpeg -i imgpercent03d.png -pix_fmt yuv420p 
-vf colorspace=all=bt709:iall=bt601-6-625:quick=1 
-colorspace 1 -color_primaries 1 -color_trc 1 video.mp4

I had one minor and one large difficulty with this command. My minor difficulty was that although the command helped make my video look barely higher, it was nonetheless a bit brighter and weaker when it comes to colours in comparison with the way it ought to look. My large difficulty was that I had no clue what it does! What’s colour area 1? And what are primaries? And if I ask what a trc is, will everybody chuckle at me? So, I began digging deeper to seek out out what precisely was taking place right here, why we’ve to make use of these actual values, and if there was an opportunity to make our movies look even higher.

Within the following sections, I’d wish to share my learnings with you and hope you’re as amazed by this area as I’m. Half I is that will help you perceive how we understand colours. In Half II, we’ll look into colour areas earlier than diving into FFmpeg-specific colour transformations in Half III.

As a software program engineer, I really like clearly outlined features and their boundaries, predictable outcomes, and derivable numbers. That is one purpose why I all the time keep away from much less secure or predictable sciences akin to biology. However, it seems, this time, we will’t escape. The design of colour areas, picture compression, and storage is simply too near the biology of human imaginative and prescient, so we will’t ignore it. So, earlier than we begin trying into colour areas, let’s shortly detour into the world of biology and learn how the human eye perceives colours and lights.

Sorry, however you’ll by no means know the way an actual inexperienced seems to be like

Mild includes of photons that transfer with particular wavelengths towards our eyes. Our eyes have cone cells (S, M, and L cone cells, to be exact) that decide up completely different wavelengths of sunshine. The next diagram reveals that the S-cone cells decide up the quick wavelengths, which make up the blue colours, the M cones, which decide up the inexperienced colours, and the L cones, which decide up the purple colours. I like to make use of the mnemonics S=quick, M=medium, and L=massive wavelengths to recollect all of this.

Be aware that the diagram knowledge is normalized, that means it doesn’t precisely symbolize the sensitivity of our cells for every of the wavelengths, which brings me to a different fascinating level. Your eyes are fairly biased in the case of colour. Your S cones are a lot much less delicate than your M cones, which suggests you want many extra photons for the blue mild to make a blue mild look as vibrant as a inexperienced mild. Equally, your L cones are barely much less delicate than your M cones. You might need seen up to now that, in a colour diagram, inexperienced all the time appears to look the brightest. That is no coincidence.

Clearly, you possibly can “see” greater than these three colours. Your mind lets you flip any mixture of indicators inside this wavelength vary right into a colour, together with a number of wavelengths directly (however extra on this later).

You might need seen one thing fascinating right here. If I needed to indicate you one thing inexperienced, I must decide one thing that emits or displays mild with a wavelength of about 550 nanometers. Though you’d see a inexperienced colour, your L cones, liable for the purple colours, nonetheless contribute. I can’t generate a wavelength that triggers solely a single cone kind in your eye. In idea, a pure inexperienced that wouldn’t set off another cones is known as hyper-green. Sadly, it’s inconceivable for me to indicate you what it might appear to be.

You may also be questioning, what’s up with the colours under roughly 400 or above 700 nanometers? The quick reply is you possibly can’t see them. Besides should you’re a Predator from the 1987 film, by which case you would possibly be capable to see infrared, which is the purple following our “reddest” seen purple at 700 nanometers. On the opposite aspect, issues don’t look significantly better for us people, as a result of we will’t see ultraviolet mild both. No less than circuitously. Keep in mind the flamboyant black lights put in in golf equipment and bars, and the phosphors in your shirt which turned these UV-A waves under 400 nm into essentially the most superb colours? That is one trick to make them seen however you possibly can’t see the UV lights at their authentic wavelengths.

You like the darkish aspect

Your imaginative and prescient isn’t linear. An fascinating facet of human imaginative and prescient is that you would be able to differentiate shadows and darker areas significantly better than brighter ones. That is a side to recollect after we discuss gamma correction later. From a software program engineering perspective, you possibly can say that should you needed to retailer an image, you’d want much more area in your disk for the darker colours than for the brighter ones as a result of you need to visualize much more darkish gray tones than vibrant ones.

Your eyes appear to care extra about brightness than colour

You won’t be sufficiently old to recollect black and white TVs. I used to be fairly younger when these had been round, however I keep in mind being fairly blissful watching a few of my favourite youngsters reveals on these. Black and white televisions basically solely present brightness, and it labored. Have you ever ever thought of watching TV on a tool that solely reveals completely different colours however all with the identical brightness? Imagine me, the expertise wouldn’t be nice. It’s because our human eye appears to have a better decision for brightness than for colour. This turns out to be useful after we discuss chroma subsampling later, the place we cut back the colour data we retailer however maintain as a lot of the brightness data as attainable.

Now that we all know how we will see issues let’s take into consideration how we will make others see issues the identical means we do.

Think about you’ve a time off, it’s a ravishing day, and also you resolve to benefit from the superb climate on the ocean. After all, you wish to share this expertise with your folks and colleagues, or perhaps you wish to make them jealous. Both means, you’re taking out your cell phone to take an image of the gorgeous ocean and ship it to your folks. Within the night, you’re reminiscing in regards to the lovely day, watching the image you took in your shiny new sensible TV, which you lastly received for Christmas. (I’ve to confess, this instance is somewhat far-fetched. I imply, who will get a sensible TV for Christmas? Anyway, let’s get to the purpose.)

Why does the image on the cell phone look just about equivalent to the image on the sensible TV? And to your pals and colleagues, all utilizing completely different units to view your ocean image, how will you make certain that it seems to be the identical for them? I imply, the ocean is blue, proper, however what precisely is blue? How does it get saved, and the way does my sensible TV know what sort of blue the ocean is? Is it darkish blue? Mild blue? Ocean blue? As a software program engineer, I do know I can symbolize pictures pixel by pixel utilizing a mix of purple, inexperienced, and blue (RGB values), every having a price between 0 and 255. So [R=0,G=0,B=255] represents a fairly robust blue and doubtlessly one colour in my ocean image. However how blue precisely is [0,0,255]? That is the place colour areas are available.

A colour area defines the vary of colours you’re allowed to make use of. It additionally tells us precisely how blue the ocean in our image must be. Completely different units would possibly assist completely different colour areas, and completely different colour areas would possibly outline completely different colour ranges. If we’re unfortunate, our ocean blue won’t be a part of a selected colour area. Fortunately, we will remodel between completely different colour areas to make issues look as near our authentic scene as attainable. Now you would possibly ask how that is attainable. Wouldn’t we’d like a reference colour area for that? A system that features all attainable colours and the place we will map our colour areas into? And also you’re proper. That is the place CIE 1931 is available in.

CIE 1931 for all seen colours, and extra

Lengthy earlier than individuals began googling FFmpeg filter chains, in 1931, the Commission on Illumination (Commission Internationale de l’éclairage, aka CIE) got here up with a chromaticity diagram that included all seen colours at their most luminance (don’t fear in regards to the time period luminance but, extra on this later).

You might need seen this diagram earlier than. The world throughout the horseshoe form represents all colours seen to the human eye, and you can begin utilizing it to outline the boundaries of your personal colour area. In observe, this diagram can’t present you all the colours, however extra about this later. What you see on this diagram is basically a flattened plot of the CIE XYZ colour area. First, let’s discuss how we received right here.

Sadly, the development of this diagram wasn’t so simple as it appears, no less than not in 1931. It began with experiments involving people attempting to match completely different attainable colours, which might be produced utilizing a single wavelength by solely utilizing three lights, a purple, a inexperienced, and a blue one. The quantity of purple, inexperienced, and blue required to match a lightweight of a selected wavelength was added to the chromaticity diagram, which ended up having this horseshoe-like form (please ignore the triangle across the area for now):

Discover the numbers across the horseshoe. They symbolize the wavelength for every colour on the sting of this shoe. Right here’s a enjoyable truth, all outer colours, aside from the underside ones, are basically the colours of a rainbow. All different colours throughout the form are combos of the outer colours. So if somebody tells you they noticed magenta in a rainbow, they’re both mendacity to you, or they noticed the colours of the rainbow mixed with different colours that occurred to be within the surroundings, such because the night sky or perhaps a second rainbow.

One other fascinating facet of this diagram is that some colour coordinates are adverse. It’s because some colours can’t be matched by including purple, inexperienced, and blue collectively. It’s important to take away a colour to get a match (in observe, this was executed by including the colour to the matching mild). We might cease right here and use this coordinate system to attract in our colour area boundaries, however then we’d should cope with adverse values when defining the colour area.

To repair this, the CIE remodeled (squeezing and stretching) this area into the one I confirmed you within the first CIE diagram, containing solely constructive values. Curiously, the CIE picked three inconceivable colours, Cr, Cg, and Cb, to make this linear transformation occur. However why? Do not forget that we stated we would like each colour to be representable by some purple, inexperienced, and blue values? In the event you’d have picked three colours from throughout the horseshoe form, you’d have by no means been in a position to generate all colours with out exceeding the area boundaries. Strive it your self; draw any triangle into the CIE 1931 diagram, and also you’ll see that you just’ll not be capable to match all attainable colours into your triangle.

We’re nearly there, I promise. What we’ve now could be the so-called CIE XYZ colour area. That is like the inspiration for all attainable colour areas, and also you’ll even see this one once more later after we discuss FFmpeg. So how do you outline a colour area? Easy, draw a triangle into the CIE diagram (there’s extra, really, however for now, this can do it).

That’s it. By drawing a triangle into the CIE XZY colour area, you simply outlined one thing referred to as primaries, an important property of your colour area. The primaries outline the precise boundaries of your area. On this case, we picked R=[x:0.64, y:0.33], G=[x:0.30, y:0.60], and B=[x:0.15, y:0.06]. In idea, you possibly can have picked any three factors within the CIE XYZ colour area, however it’s fairly frequent to choose some worth for blue, some for inexperienced, and a few for purple for many colour areas you’re coping with.

Are you interested in the dot within the center with the label “D65”? That is referred to as a white level to outline what’s white in our colour area and is one other essential property of our colour area. Completely different colour areas would possibly contemplate completely different colours as “white”. In our diagram, we picked a fairly frequent one, roughly a daylight white that sits at [x:0.3127, y:0.3290], with a lightweight temperature of 6500 Kelvin (which additionally explains the 65 in D65).

I’ve to confess that the values we picked right here usually are not completely random. They’re taken from the Rec. 709 colour area, which we’ll look into later.

You would possibly keep in mind from Half I of this weblog publish that our human imaginative and prescient isn’t linear.

Sadly, the foundations for digital camera sensors are a bit completely different. A digital camera would possibly certainly see our good ocean image very linearly. However what does this imply? As you possibly can see within the diagrams under, the human eye could be very delicate in low-light areas. It wants fewer photons to understand one thing as, let’s say, 20% vibrant but in addition much more power to understand one thing as twice as vibrant. In observe, it is a good factor. We gained’t go loopy when leaving a darkish room to seize a espresso on a sunny day and we will take within the lovely nature round us with none downside. The digital camera, nevertheless, retains it easy. It simply wants 50% of photons to make one thing look 50% vibrant and twice the quantity of power to make one thing seem twice as vibrant.

So what would occur if we seen our unprocessed, linear ocean image on an outdated CRT (cathode-ray tube) monitor? I intentionally selected a CRT monitor right here as a result of they want gamma correction. Coincidentally, CRT screens share a really related conduct with the human eye, simply the opposite means round, basically inverting the attention’s gamma curve. You possibly can see within the diagram that it requires a excessive voltage to achieve 50% brightness. So if we’d simply ship our unprocessed linear picture to this monitor, it might look fairly darkish as a result of it wants much more power than the digital camera picture signifies. However why am I speaking about outdated CRT displays right here? Haven’t we moved on from them? The truth is, in the case of this conduct, we haven’t actually. Newer LCD screens nonetheless implement this conduct, and it’s really not too unhealthy, as we’ll discover out later.

However for now, what can we do to repair this? It’s simple. We apply a metamorphosis to our picture in order that it has an inverse brightness curve in comparison with the curve of our eye. Subsequently, the ultimate picture on our display has a linear brightness. That is additionally referred to as gamma correction.

We stated earlier that to make the human eye blissful, we’d like extra knowledge/bits for the darker areas of an image than for the brighter ones. By making use of the inverse transformation earlier than storing the ultimate picture on our telephone, we will have each: an auto-corrected picture and extra particulars within the darker space. It’s because the inverse perform shortly pushes our darker values into the brighter area, permitting us to save lots of all shades in a fashion our eye considers good and evenly distributed.

And why did I inform you all of this? What does this should do with colour areas? To plot the graphs above, I cheekily used an exponent of two.5 (or 1/2.5). So I calculated y = x^(1/2.5) for all brightness values x between 0 and 1, that means I utilized a gamma correction worth of 1/2.5, which is fairly near how your eyes work. Different gamma values could be a more sensible choice for some units and use instances. Take into consideration an outdated film theatre projector, for instance. The underside line is there isn’t one gamma perform. Completely different colour area requirements would possibly outline completely different features for his or her gamma correction. These features are sometimes called switch traits (trc) or switch perform. They don’t essentially have this straightforward kind, and plenty of features are a mix of linear and exponential features, implementing numerous thresholds. Once we remodel between completely different colour areas later, we’ve to take this under consideration.

In the event you’d wish to be taught extra about this matter, I extremely suggest testing Unravel | Understanding Gamma and Understanding Gamma Correction. Each are wonderful articles on this matter.

Does it all the time should be purple, inexperienced, and blue? What about YUV and Y’CbCr?

Up to now, we’ve outlined the colours in our colour area utilizing some mixture of purple, inexperienced, and blue. However aren’t there different methods to symbolize colour? Y’CbCr might be one of the vital frequent alternate options. As a substitute of utilizing purple, inexperienced, and blue, Y’CbCr makes use of Y’ to symbolize the luma worth of a colour (the brightness) and the chroma values Cb and Cr to symbolize the colour. Let’s have a more in-depth look.

Earlier than explaining issues additional, it’s price noting we will remodel any of our RGB colour values into Y’CbCr values utilizing the next equations:

• Y’ = 0.299R + 0.587G + 0.114B
• Cb = -0.168935R — 0.331665G + 0.50059B
• Cr = 0.499813R — 0.418531G — 0.081282B

In the event you’re now much more confused, I used to be too after I noticed this the primary time. And what makes it extra complicated is that completely different sources have completely different equations. The truth is, I simplified this equation barely. For instance, Y’CbCr generally expects Y’ to be in a restricted vary between 16 and 235, whereas the worth produced right here is unscaled, between 0 and 255. However let’s ignore scaling for now. We’ll discuss ranges in a later part.

As a substitute, let’s take a look at what’s taking place right here. So Y’, our brightness worth, combines all three RGB values. The coefficients for our RGB values (0.299, 0.587, and 0.114) are referred to as luma coefficients (you possibly can consider them as weights). The coefficients can often be mathematically derived from the colour area primaries and white level. Nonetheless, not all colour area requirements observe this and would possibly use barely completely different coefficients. The little prime image (‘) in Y’ really issues. It means we’re working with gamma-corrected RGB knowledge, not linearised knowledge. In the event you don’t know what I’m speaking about, look again on the earlier part on gamma correction.

The values for Cb and Cr look much more complicated, even containing adverse values. Nonetheless, issues get somewhat simpler whenever you take a look at YUV. Y’CbCr and YUV are sometimes used interchangeably (even within the FFmpeg’s code base), however technically, they’re completely different. YUV is one thing from our good outdated analog tv world. In YUV, or Y’UV to be actual, U and V have the next definitions:

• U = B — Y’
• V = R — Y’

So all we’re doing is subtracting our luma (brightness) worth from our blue and purple values. That appears so much easier. And should you re-arrange the equations somewhat, you’ll see that Y’, U, and V present sufficient data to find out our authentic RGB values. Y’CbCr is predicated on this Y’UV illustration however goes a step additional. To get Cb and Cr for our Y’CbCr illustration out of the Y’UV one, we’re basically simply squeezing the U and V values into a variety of 16–240. That is defined intimately in About YUV Video — Win32 apps and is exterior the scope of this weblog publish.

You’ll typically discover the conversion equations properly packed right into a 3×3 matrix you possibly can multiply with the RGB values. If you wish to be taught extra about this matter, I like to recommend testing About YUV Video — Win32 apps and [MS-RDPRFX]: Color Conversion (RGB to YCbCr).

Oh yeah, after which there’s colour ranges

Keep in mind earlier after I stated RGB values are often between 0 and 255? It seems this isn’t all the time true. The truth is, many movies comprise RGB values between 16 and 235 (for instance, most Rec. 709 HD content material). On this case, 16 is the darkest black, and 235 is the brightest white. That is referred to as both full vary or restricted vary and was once more executed for historic causes. I don’t wish to dive too deep into this, however should you’re curious, I like to recommend the next article Full RGB vs. Limited RGB: Is There a Difference?.

What’s essential to know for our use case is that almost all pictures are encoded in full vary, whereas our anticipated Rec. 709 video output must be in restricted vary. If we neglect to take vary conversion under consideration, our video would possibly look somewhat washed out.

So what makes a colour area?

In the event you’ve made it this far, you just about have all of the instruments it’s worthwhile to outline your personal colour area. We talked about how one can outline the colours to your area, the anticipated gamma correction, and even what elements it’s best to use to rework the colours of your area from RGB to YUV or Y’CbCr. So in abstract, to create your personal colour area, it’s worthwhile to outline:

• A bunch of primaries (preferable purple, inexperienced and blue besides you wish to confuse individuals)
• A white level
• A switch perform / gamma correction

Moreover, should you outline your personal colour area customary, with the expectation that your colour values get represented in YUV or Y’CbCr, you would possibly wish to outline the anticipated luma coefficients to (these might be mathematically derived however don’t should be).

At this level, I have to admit that I’ve stored issues easy and hidden a couple of issues from you. Some video requirements that embrace a colour area definition outline many extra properties, together with body charge, bit depth per pixel, and extra. As a result of these are much less related to us, I made a decision to maintain these out of the scope of this doc. Now, let’s take a look at some precise colour areas.

sRGB — let’s discuss our very first colour area

The sRGB colour area might be one of the vital well-known colour areas. It’s the default colour area for most of the footage you’ll discover on-line and might be what your display is optimized for.

You’ll find the primaries, the switch perform, the white level, and the luma coefficients on the Wikipedia web page for sRGB, so there’s no want for me to repeat them on this weblog publish.

Nonetheless, I wish to point out an fascinating truth about this colour area. sRGB is similar to Rec. 709 (which we’ll focus on in a second) in that it makes use of the identical primaries and white level and, subsequently, the identical luma coefficients. Nonetheless, it’s somewhat complicated as a result of it defines its switch traits. Whereas researching this matter, I seen that the 2 areas are sometimes handled as equal with out contemplating the gamma correction, which ends up in outputs that look barely off.

Yet one more enjoyable truth earlier than we transfer on. When trying on the sRGB primaries, you’ll understand I lied after I confirmed you the CIE 1931 diagram. As I discussed earlier than, your display can most likely present you colours that stay near the sRGB area. However it may’t present you what’s exterior its little triangle. The truth is, and this could be disappointing, no RGB display on the earth can present you your entire CIE 1931 diagram. Keep in mind the primaries for CIE XYZ? They’re inconceivable colours that no machine can emit.

Rec. 709

Let’s take a look at yet one more colour area. Rec. 709, also referred to as BT.709 or ITU 709 (simply to make you keep in mind extra acronyms) is used for HD video materials. I’m mentioning this colour area as a result of it’s our default export colour area for movies at Canva.

The Rec. 709 primaries, white level, switch perform, and luma coefficients can all be discovered on Wikipedia, so we don’t have to dive into this once more. It’s price highlighting that Rec. 709 and sRGB share the identical primaries. Nonetheless, their switch features and luma coefficients differ.

Changing between colour areas

From the introduction of this weblog publish, you would possibly keep in mind that I’m attempting to transform an sRGB PNG picture right into a Rec. 709 video. This brings us to the subsequent matter, colour area transformations. Because the identify suggests, a colour area transformation is a metamorphosis of colour values from one colour area into one other colour area. Let’s take a step again and contemplate why and when these transformations are mandatory.

Every colour area is designed for a unique objective. For instance, you might need an HD video optimized to your Rec. 709 TV show. What should you needed to play this video on an SD TV? To make sure every little thing seems to be OK even in your SD show, you would possibly wish to convert your video into one with the Rec. 601 colour area. Equally, whenever you’re watching a video in a unique colour area than the one supported by your show, you may want your video participant to do the required conversions for you.

However how do you remodel a colour from one colour area into one other? You would possibly keep in mind the CIE XYZ colour area the place we stated this is sort of a tremendous colour area, in a position to symbolize any seen colour (and much more) from any area. The excellent news is that every frequent colour area defines a conversion matrix to and from CIE XYZ. Sadly, we will’t apply this matrix with our de-linearised (gamma-corrected) enter, so we should first linearise the information. It’s best to do that transformation with RGB knowledge. So in case you solely have Y’CbCr knowledge mendacity round, you’ll wish to remodel it to RGB first. This leaves us with the next steps to transform a colour CA from one colour area A to a different colour CB in area B (this is only one attainable means, in fact):

1. Convert the Y’CbCr values of colour CA to RGB utilizing the luma coefficients for A (skip this step if you have already got RGB knowledge).
2. Linearise CA utilizing the switch traits for A.
3. Rework CA to CIE XYZ utilizing the CIE XYZ transformation matrix for A.
4. Rework CIE XYZ to CB utilizing the CIE XZY transformation matrix for B.
5. De-linearise CB utilizing the switch traits for B.
6. Scale your RGB vary (in case your enter and output ranges are completely different, for instance, from restricted to full).
7. Convert RGB of CB to Y’CbCr utilizing the luma coefficients for B (skip this step should you don’t want a Y’CbCr knowledge stream, however for Rec. 709, you most likely need Y’CbCr values).

Now you understand how to transform colours from one colour area to a different. No less than in idea. Fortunately, FFmpeg is doing this work for us, so we solely should specify the completely different properties of our colour area. Earlier than diving into FFmpeg, although, let’s shortly make clear how the colours areas are saved.

The place is colour area data saved?

It’s essential to keep in mind that the RGB or Y’CbCr data of your picture doesn’t reveal what colour area they belong to. So if somebody tells you, “My new push bike has the colour R=24, G=55, B =255.”, it’s worthwhile to ask, “What colour area are we speaking about?”.

At first of this part, we requested how our pals’ TVs and cell phones know how you can current the image of our day on the ocean. Colour areas assist us make sure that the goal units know precisely what colours to make use of when presenting a picture and apply the required transformations. However the place is that this colour area data saved? The reply to this query is it relies upon.

Completely different file codecs have completely different assist for meta data. PNG, for instance, has a particular so-called “chunk” that may embed a colour profile. The mp4 video container format comprises one thing related, a so-called “colr” atom, which shops primaries, switch features, and so forth. Sadly, there’s no assure that this meta data across the colour area is definitely obtainable. You would, for instance, generate an image or a video with out including any colour data in any respect. On this case, the picture viewer or video participant will most likely run a best-effort try and guess the colour area, which doesn’t all the time yield the most effective outcomes.

A fast observe on chroma subsampling

I left this to final on this part as a result of chroma subsampling doesn’t match properly into the area of colour areas. However as we’re utilizing it in our FFmpeg command (-pix_fmt yuv420p), we must always no less than shortly make clear what it’s.

Keep in mind we stated that in the case of decision, your eyes are a lot much less fussed about decrease colour resolutions so long as the brightness decision is adequate? It is a property leveraged by chroma subsampling. Colour fashions, akin to Y’CbCr, make it simple for us to separate chroma from luma or brightness data. Keep in mind, the brightness being completely within the Y’? Now we will use the truth that our eyes and mind care much less in regards to the colour worth for every pixel and are extra taken with a excessive decision of luma values. How, you ask? Easy, we mix the chroma values for a bunch of neighboring pixels, successfully decreasing the decision, and on the identical time, holding the luma values of their excessive decision. For instance, if we’ve 2 pixels, p1 and p2, we maintain Y’p1 and Y’p2 separate however we mix Cbp1 and Cbp2 in addition to Crp1 and Crp2 into solely two chroma values Cbp12 and Crp12. We saved 2 knowledge factors by doing this.

In my instance, I made a decision to mix 2 pixels, however there are numerous other ways you possibly can accomplish this. You’ll typically see acronyms like yuv422 or yuv420. These are quick for various methods to mix pixel data, and so they outline precisely how you can mix two strains of pixels. For instance, yuv422 says, “take 4 pixels, within the first line and mix 2 of them into 1, within the second line mix 2 of them into 1”. yuv420 says, “mix 4 pixels, take 2 pixels from the primary line, take 0 pixels from the second line and mix them into 0”. yuv420 basically combines the chroma data for 4 completely different pixels (2 in every line) into solely 2 pixel values. The variety of strains that chroma subsampling impacts is all the time 2 in these instances. It could be a bit complicated, however this 2 just isn’t contained within the acronym.

There are a lot of different methods to mix pixels for chroma subsampling and much more particulars on storing and transferring the pixel knowledge. Nonetheless, that is out of the scope of this doc and isn’t required to grasp our FFmpeg colour area transformation examples.

Half II abstract

We opened this part with the query of how we will make sure that our pals and colleagues can get pleasure from our good ocean image on their units with the identical lovely colours as we do. We discovered how colour areas assist us and our units obtain this. We additionally discovered that colour area requirements outline a number of issues, akin to primaries, white level, gamma correction, and extra. After all, there’s no assure that your folks’ units will observe all the colour area definitions we outlined so passionately in our ocean image. However no less than we gave it our greatest.

This could give us sufficient enter to transform colour areas with FFmpeg. By now, you most likely have a common thought of what our initially complicated FFmpeg command at first of this weblog publish is doing. If not, don’t fear. We’ll get to it.

I wish to emphasize once more that I’m solely touching the floor on colour areas. There’s much more fascinating stuff to be taught. In the event you’d wish to dig deeper into colour areas usually, I like to recommend the next wonderful articles: Unravel | Understanding Color Spaces and The Hitchhiker’s Guide to Digital Colour.

Now that we’ve all the speculation to transform some colour areas with FFmpeg, this third a part of my weblog publish examines how we will do it.

Our use case is to transform an sRGB PNG file right into a Rec. 709 video. Right here once more is the preliminary FFmpeg command I discovered in Talking About Colorspaces and FFmpeg

ffmpeg -i imgpercent03d.png -pix_fmt yuv420p 
-vf colorspace=all=bt709:iall=bt601-6-625:quick=1 
-colorspace 1 -color_primaries 1 -color_trc 1 video.mp4

Be aware that the enter information within the linked weblog would possibly differ from ours, so we’ve to confirm if these parameters make sense for us. Let’s shortly stroll by way of a few of these parameters:

• -pix_fmt yuv420p: We’d like a yuv420p chroma subsampling.
• -vf colorspace=all=bt709:iall=bt601-6-625:quick=1: We’d like to make use of the colorspace filter, the output is about to Rec. 709 (or BT.709 as we discovered earlier), and the enter is Rec. 601 (aka BT.601, basically SD materials). Let’s discuss quick=1 in a second.
• -colorspace 1 -color_primaries 1 -color_trc 1: These parameters add the meta data for the chosen colour area to the output video. It’s essential to notice they don’t change something in regards to the video content material itself, in order that they don’t remodel the colour area of the video. That is the duty of the colorspace filter. As you might need already guessed, the 1 behind every of those parameters maps to Rec. 709. internally.

Within the following sections, we’ll overview the principle colour area properties and see what parameters and values greatest apply to our use case.

The primaries

So, what primaries ought to we be utilizing? Our enter is an sRGB PNG, so we must always use sRGB primaries. And for the output? Straightforward, we’re going to make use of Rec. 709 primaries. Let’s take a look at the unique filter definition.

colorspace=all=bt709:iall=bt601-6-625:quick=1

The output seems to be good, set to BT.709, the identical as Rec. 709. How in regards to the enter? This doesn’t work in our situation. The Rec. 601 colour area is completely different from the sRGB colour area. It’s additionally price mentioning that we don’t wish to do a quick conversion. Why? Let’s take a look at what the filter documentation says in regards to the quick parameter: “Do a quick conversion, which skips gamma/major correction.“. Let’s take into consideration this, might we skip the first and gamma corrections? Keep in mind, the sRGB and Rec. 709 primaries are the identical, so we must be high quality with the primaries choice. Sadly, the perform used for the gamma correction in each requirements differs. Subsequently, we’ve to take away the quick parameter.

My naive previous self thought, let’s give the next a strive (as a result of it completely is sensible proper?).

colorspace=all=bt709:iall=srgb

And that is what I received.

Some greater energy doesn’t need me to make use of the sRGB colour area as an enter. And to be trustworthy, this error message doesn’t inform me why. Wanting on the documentation, it appears that evidently for the iall parameter, we will solely select one of many following values: bt470m, bt470bg, bt601-6-525, _bt601-6-625, bt709, smpte170m, smpte240m, or bt2020. So it seems to be like what we’d like is simply not there.

However all just isn’t misplaced. Keep in mind what I stated earlier about sRGB utilizing the identical primaries, white level, and subsequently luma coefficients as Rec. 709, however a unique gamma perform? Fortunately, the colorspace filter permits us to outline all these properties individually (no less than not directly). So, let’s begin by explicitly defining our enter primaries.

colorspace=all=bt709:iprimaries=bt709

As you possibly can see, I dropped the iall parameter for now and changed it with iprimaries. At this level, I will point out that the all parameter is a bit complicated. I initially assumed that all set defaults for the enter and output parameters. Nonetheless, trying on the filter supply code, it appears that evidently all defines simply the output colour parameters, not the enter ones. iall defines the enter parameters and different parameters having the i prefix. And by the best way, you need to use the iprimaries parameter at the side of iall, by which case, iprimaries overrules no matter primaries iall would have outlined.

Gamma correction and TRC

Now that we’ve outlined our primaries, let’s take a look at how you can apply a gamma correction. Sadly, sRGB and Rec. 709 use completely different features for gamma correction. However this time, we’re in luck as a result of the colorspace filter instantly helps our sRGB switch perform and is as simple as the next.

colorspace=all=bt709:iprimaries=bt709:itrc=srgb

Luma coefficients

It is a complicated one. To start with, no parameter seems to be like it might explicitly outline luma coefficients. As we talked about earlier than, the luma coefficients must be mathematically derived from primaries and white level. Nonetheless, some requirements would possibly outline completely different values, and I needed to make sure we’re choosing the right values. So, being somewhat determined, I checked out the FFmpeg supply to see how the luma coefficient choice works internally. It seems the coefficients are chosen purely primarily based on the area/ispace parameters. And we must be high quality utilizing these, supplied we overwrite different colour area properties that don’t apply to us, akin to trc.

Much like the primaries, there’s no area parameter for sRGB. Wanting on the documentation, we will solely choose one of many following colour area values: bt709, fcc, bt470bg, smpte170m, smpte240m, ycgco, or bt2020ncl. As a result of sRGB and Rec. 709 share the identical primaries and white level, we will safely fall again to the Rec. 709 colour area once more to make the filter use the right luma coefficients.

colorspace=all=bt709:iprimaries=bt709:itrc=srgb:ispace=bt709

Colour vary

Defining the vary property nearly drove me loopy! Keep in mind, sRGB and Rec. 709 use completely different colour ranges. For Rec. 709, an RGB worth of [R=16, G=16, B=16] is absolute darkness, whereas for sRGB, it’s [R=0, G=0, B=0]. The colorspace filter permits us to outline enter and output colour ranges utilizing the irange and vary parameters. You should use the television worth to outline a restricted vary, whereas you need to use the computer worth to outline a full vary. Since we’re changing a full vary sRGB PNG picture to a restricted vary Rec. 709 video, this must be the filter expression, proper?

colorspace=all=bt709:iprimaries=bt709:itrc=srgb:ispace=bt709:vary=television:irange=computer

Oh, how mistaken I used to be. The right reply is the next.

colorspace=all=bt709:iprimaries=bt709:itrc=srgb:ispace=bt709:vary=television:irange=television

Alternatively, you possibly can drop the vary definition altogether. Are you as confused as I used to be? Try the subsequent part to seek out out precisely why that is taking place.

FFmpeg provides a scale filter with out telling you

That I might skip the colour vary transformation with out figuring out why bothered me so much. I needed to seek out out precisely what was taking place, so I began compiling my very own model of FFmpeg with further debug data. Because it seems, there’s much more occurring than I initially thought.

When trying on the colorspace filter supply, I found the next code snippet.

static int query_formats(AVFilterContext *ctx)
{
static const enum AVPixelFormat pix_fmts[] = {
AV_PIX_FMT_YUV420P,   AV_PIX_FMT_YUV422P,   AV_PIX_FMT_YUV444P,
AV_PIX_FMT_YUV420P10, AV_PIX_FMT_YUV422P10, AV_PIX_FMT_YUV444P10,
AV_PIX_FMT_YUV420P12, AV_PIX_FMT_YUV422P12, AV_PIX_FMT_YUV444P12,
AV_PIX_FMT_YUVJ420P,  AV_PIX_FMT_YUVJ422P,  AV_PIX_FMT_YUVJ444P,
AV_PIX_FMT_NONE
};
// ... extra setup{

To present you some context, every FFmpeg filter can outline the form of inputs it accepts in a perform referred to as query_formats. Do you discover one thing peculiar about this one? Our colorspace filter does not assist RGB inputs. So how might it soak up our PNG picture, which is clearly supplied in RGB? Because it seems, FFmpeg provides in a scale filter that converts our RGB knowledge to YUV for us. In the event you run FFmpeg with -report, you possibly can see this taking place.

[Parsed_colorspace_0 @ 0x7f7f8cf04b80] auto-inserting filter 'auto_scale_0' between the filter 'graph 0 enter from stream 0:0' and the filter 'Parsed_colorspace_0'

Keep in mind from our earlier sections that to rework to YUV/Y’CbCr, you want luma coefficients for the corresponding colour area. Sadly, FFmpeg’s PNG decoder doesn’t consider the colr chunk, which comprises our colour area data (enjoyable truth: the encoder does write it). This implies the scale filter falls again to an unspecified colour area. Fortunately, the coefficients used for an unspecified colour area equal those of our sRGB colour area. In the event you’re ever working with a PNG file that’s in a colour area completely different from sRGB, you’ll should maintain this in thoughts. It’s additionally price mentioning that the conduct would possibly differ for various enter file codecs. Within the case of a JPG file, for instance, the colour area data appears to be parsed and handed on to the filters appropriately.

And to make issues extra complicated, the scale filter makes use of a unique illustration of the luma coefficients than the colorspace filter internally. The scale filter makes use of a illustration outlined in yuv2rgb.c, whereas the colorspace filter makes use of the one in csp.c, and each units of values have a really completely different construction with completely different coefficients. The excellent news is it’s purely a unique illustration, and so they’re mathematically equal. Nonetheless, it’s essential to maintain this in thoughts when experimenting together with your luma coefficients.

One other fascinating facet of the dimensions filter is that it’s really much more highly effective than it pretends to be. I initially thought, “It’s referred to as scale, what can it attainable do? Scale one thing, proper?”. It seems the scale filter can do total colour area transformations too. It doesn’t appear in a position to do gamma correction, however it is aware of how you can cope with primaries and colour ranges, for instance. And this brings me to the subsequent fascinating level. In our situation, the scale filter translated our full vary RGB picture to a restricted vary one earlier than passing it on to our colorspace filter. I hope this explains why we don’t want the express vary definition within the colorspace filter itself.

Right here is the ultimate FFmpeg command I used to rework a PNG picture into an h264 video within the Rec. 709 colour area

ffmpeg -loop 1 -i picture.png -pix_fmt yuv420p -c:v libx264 -t 1 
-vf "colorspace=all=bt709:iprimaries=bt709:itrc=srgb:ispace=bt709" 
-color_range 1 -colorspace 1 -color_primaries 1 out.mp4

And if you wish to shorten this somewhat, you need to use the iall parameter as an alternative of explicitly defining the iprimaries and ispace parameters.

ffmpeg -loop 1 -i picture.png -pix_fmt yuv420p -c:v libx264 -t 1 
-vf "colorspace=all=bt709:iall=bt709:itrc=srgb" 
-color_range 1 -colorspace 1 -color_primaries 1 out.mp4

Right here’s a recap of a few of our FFmpeg learnings:

• -report is a helpful FFmpeg parameter to get some extra insights.
• If an enter is supplied in RGB (for instance, for PNG information) however a filter requires YUV, FFmpeg silently injects a scale filter (vf_scale).
• The vf_scale filter does far more than simply scaling, akin to YUV and colour area conversions (with out gamma correction).
• When utilizing the colorspace filter, don’t use quick=1 if you’d like gamma correction.
• The PNG decoder ignores the colr chunk on learn, resulting in an unspecified enter colour area, whereas the JPG decoder doesn’t. Nonetheless, the PNG encoder does write the colr chunk.
• -colorspace 1 -color_primaries 1 -color_trc 1 solely set meta colour profile data. They don’t trigger an precise change to the video.

I hope this weblog publish clarifies what colour areas are, why it’s best to care about them, and the way you need to use FFmpeg to rework colour areas. If in case you have any extra questions on this matter, please let me know and I’ll attempt to shed some mild on the lacking spots. Additionally, should you discover errors and really feel like issues want correcting, please let me know and I’ll get it fastened.

A fast guidelines for when your video seems to be off

Earlier than I allow you to go, right here’s a fast guidelines you need to use in case your generated video seems to be barely off when it comes to colours:

• Do the enter and output primaries look okay?
• What in regards to the enter and output trc and gamma correction?
• Have you considered the ranges?
• Are the luma coefficients appropriate? They need to should you picked the right primaries and white level.

Acknowledgements

I’d like to offer particular shout outs to Chris Cook, Bela Babik and Peter Camilleri for his or her assist and inspiration on this loopy colour area journey.

Some nice hyperlinks for additional studying

Listed here are some further sources of data I discovered helpful:

Need to work on the most recent in video tech? Join Us!