I by chance Blender VSE · Aras’ web site

Two months in the past I began to contribute a little bit of code to Blender’s Video Sequence Editor (VSE).
Do you know that Blender has a collection of video modifying instruments? Yeah, me neither 🙂 Even the
feature page for it on the web site appears to be like… a bit empty lol.

Do I do know something in any respect about video modifying, timelines, sequencers, shade grading, ffmpeg, audio mixing and so forth?
After all not! So naturally, which means I ought to begin to tinker with it.

Wait what?

How does one by chance begin engaged on VSE?

You try this since you determine to take a look at Unity’s
Unite 2023 convention
in Amsterdam, and to go to some associates. For a spare half-a-day after the convention, you determine to
take a look at Blender HQ. There, Francesco and Sergey, for some purpose,
ask you whether or not you’d wish to contribute to VSE, and in opposition to your higher judgement, you say “perhaps?”.

In order that’s how. After which it feels just about like this:

I began to tinker with it, largely attempting to do random “straightforward” issues. By straightforward,
I imply efficiency optimizations. Since, until the code complexicates rather a lot, they’re laborious
to argue in opposition to. “Right here, this factor is 2x quicker now”, in most locations everybody will react with “oh good!”. Hopefully.

So, two months of kinda-partime tinkering on this space that I didn’t even know existed earlier than, and Blender VSE
bought a small set of enhancements for upcoming Blender 4.1 (which simply turned beta and will be downloaded from traditional
daily builds). Right here they’re:

Snappier Timeline drawing

VSE timeline is the underside a part of the picture above. Right here it’s zoomed out into the entire
Sprite Fright edit venture, with about 3000 “strips” seen directly.
Simply scrolling and panning round in that timeline was updating the person interface at ~15 frames per second.

Now that’s 60+ frames per second (#115311). Seems,
submitting graphics API draw calls two triangles at a time just isn’t the quickest method, heh. Right here’s that course of
visualized inside probably the most wonderful Superluminal profiler – just about on a regular basis
is spent inside “start drawing one quad” and “end drawing one quad” capabilities 🤦

As a part of that, audio waveforms show additionally bought some weirdness about it mounted, some UI polish tweaks, and now could be on by default
(#115274).

Scopes

VSE has choices to show typical “scopes” you would possibly count on: picture histogram, waveform, vectorscope. Right here’s their look,
“earlier than” on the left aspect, “now” on the suitable.

Histogram was drawn as pixelated picture, with very saturated colours. Draw as nicer polygons, with a grid, and fewer saturation
(#116798):

Waveform (right here proven in “parade” choice) was saturating in a short time. Oh, and make it 15x quicker with some multi-threading
(#115579).

Vectorscope’s outer shade hexagon appeared very 90s with all of the pixelation. Copy the up to date picture editor vectorscope
design, and voilà
(#117738):

Whereas at it, the “present overexposed areas” (“zebra stripes”) choice was additionally sped up 2x-3x
(#115622).

All these scopes (and picture editor scopes) ought to in some unspecified time in the future be achieved on the GPU with compute shaders, after all.
Sometime.

ffmpeg bits

Blender primarily makes use of ffmpeg libraries for audio/video studying and writing. That suite is legendary
for the extraordinarily versatile and considerably intimidating command line tooling, however inside Blender the precise code libraries
like libavcodec are used. Amongst different issues, libswscale is used to do
film body RGB↔YUV conversion. Seems, libswscale can do these components multi-threaded for some time by now, it’s simply
not precisely intuitive how to realize that.

Earlier code was like:

// init
SwsContext *ctx = sws_getContext(...);
// convert RGB->YUV
sws_scale(ctx, ...);

however that finally ends up doing the conversion fully single-threaded. There’s a "threads" parameter which you can set on the
context, to make it be capable to multi-thread the conversion operation. However that parameter must be set at initialization time,
which suggests you may not use sws_getContext(), and as a substitute you need to initialize the context the laborious means:

SwsContext *get_threaded_sws_context(int width,
                                     int top,
                                     AVPixelFormat src_format,
                                     AVPixelFormat dst_format)
{
  /* sws_getContext doesn't enable passing flags that ask for multi-threaded
   * scaling context, so do it the laborious means. */
  SwsContext *c = sws_alloc_context();
  if (c == nullptr) {
    return nullptr;
  }
  av_opt_set_int(c, "srcw", width, 0);
  av_opt_set_int(c, "srch", top, 0);
  av_opt_set_int(c, "src_format", src_format, 0);
  av_opt_set_int(c, "dstw", width, 0);
  av_opt_set_int(c, "dsth", top, 0);
  av_opt_set_int(c, "dst_format", dst_format, 0);
  av_opt_set_int(c, "sws_flags", SWS_BICUBIC, 0);
  av_opt_set_int(c, "threads", BLI_system_thread_count(), 0);

  if (sws_init_context(c, nullptr, nullptr) < 0) {
    sws_freeContext(c);
    return nullptr;
  }

  return c;
}

And also you’d assume that’s sufficient? Haha, after all not. sws_scale() by no means does multi-threading internally. For that, you want
to make use of sws_scale_frame() as a substitute. And when you do, it crashes because it seems that you’ve got had created your
AVFrame objects only a tiny bit mistaken that was fully tremendous for sws_scale, however may be very a lot not tremendous for sws_scale_frame
because the latter tries to do varied kinds of reference counting and whatnot.

Youngsters, don’t design APIs like this!

So all that took some time to determine, however phew, achieved (#116008),
and the RGB→YUV conversion step whereas writing a film file is sort of a bit quicker now. After which do the identical within the different
course, i.e. when studying a film file, use multi-threaded YUV→RGB conversion, and fold vertical flip into the identical operation
as properly (#116309).

Audio resampling

Whereas taking a look at the place time is spent whereas rendering a film out of VSE, I observed a this feels extreme second the place
virtually half of the time that takes to “produce a video or audio body” is spent contained in the audio library utilized by Blender
(Audaspace). Not in encoding audio, simply in mixing it earlier than encoding! Seems,
most of that point is spent in resampling
audio clip information, for instance the film is ready to 48kHz audio, however a number of the audio strips are 44.1kHz or comparable.
I began to dig in.

Audaspace, the audio engine, had two modes that it may do sound resampling: for inside-blender playback, it was utilizing Linear
resampler, which simply linearly interpolates between samples. For rendering a film, it was utilizing
Julius O Smith’s algorithm with, what it appears like, “uhh, considerably overkill”
parameter sizes.

A method to take a look at resampler high quality is to take an artificial sound, e.g. one which has a single rising frequency, resample it,
and have a look at the spectrogram of it. Right here’s a “sweeping frequencies” sound, resampled
inside Audacity with “greatest”, “medium” and “low” resampling settings. What you need is the outcome
that appears just like the “greatest” one, i.e. as little further frequencies launched as doable.

Inside Blender, Audaspace was offering two choices: rendering vs. preview playback. Rendering one is sweet spectrogram certainly,
whereas preview, whereas being quick to compute resampling, does introduce rather a lot of additional frequencies.

What I’ve achieved, is add a brand new “medium” resampling high quality setting to Audaspace that, so far as I can inform, produces just about the identical outcome
whereas being about 3x quicker to calculate. And made Blender use that when rendering:

With that, rendering a portion (2000 frames) of Sprite Fright on Home windows Ryzen 5950X
PC went 92sec→73 sec (#116059). And I’ve discovered a factor or two about audio resampling. Not dangerous!

Picture transformations and filtering

Strips that produce a visible (pictures, films, textual content, scenes, …) in Blender VSE will be remodeled: positioned, rotated, scaled, and extra cropping
will be utilized. Every time that occurs, the picture that’s usually produced by the strip is remodeled into a brand new one. All of that’s achieved on the CPU,
and was multi-threaded already.

But it had some points/bugs, and components of the code might be optimized a bit. Plus another issues might be achieved.

“Why all of that’s achieved on the CPU?!” you would possibly ask. Good query! A part of the reason being, that nobody made or not it’s achieved on the GPU.
One other half, is that the CPU fallback nonetheless must exist (at the very least proper now), for the use case of: person desires to render issues on
a render farm that has no GPU.

“Off by half a pixel” errors

The code had varied “off by half a pixel” errors that in lots of instances cancel themselves out or are invisible. Till they don’t seem to be. This isn’t too dissimilar
to “half texel offset” issues that everybody needed to undergo in DirectX 9 occasions when doing any kind of picture postprocessing. Felt like youth once more 🙂

E.g. scaling a tiny picture up 16x utilizing Nearest and Bilinear filtering, respectively:

The Bilinear filter shifts the picture by half the supply pixel! (there’s additionally magenta – which is background shade right here – sneaking in; about that later)

Within the different course, scaling this picture down precisely by 2x utilizing Bilinear filtering does no filtering in any respect!

So issues like that (in addition to different “off by one thing” errors in different filters) bought mounted
(#116628). And the pictures above appear like this with Bilinear 16x upscaling and 2x downscaling:

Transparency border round Bilinear filtering

VSE had three filtering choices in Blender 4.0 and earlier: Nearest, Bilinear and Subsampled3x3. Of these, solely the Bilinear one was including
half a supply texel price of transparency across the ensuing picture. Which is considerably seen if you’re scaling your media up. Why this discrepancy,
nobody remembers at this level, nevertheless it was there “endlessly”, it appears.

There’s an identical subject in Blender (CPU) Compositor, the place Bilinear sampling of one thing blends in “transparency” when proper on the sting
of a picture, whereas Bicubic sampling doesn’t. Once more, nobody remembers why, and that must be addressed by somebody. Sometime.

I eliminated that “mix into transparency” from bilinear filtering code that’s utilized by VSE. Nonetheless! A aspect impact of this transparency
factor, is that if you don’t scale your picture however solely rotate it, the sting does get some kind of anti-aliasing. Which it might be dropping now, if simply
eradicating that from bilinear.

So as a substitute of mixing in transparency when filtering the supply picture, as a substitute I apply some kind of “transparency anti-aliasing” to the sting pixels of
the vacation spot picture (#117717).

Filtering additions and modifications

Common VSE strip transforms didn’t have a cubic filtering choice (it solely existed within the particular Remodel Impact strip), which sounded
like a curious omission. And that led right into a rabbit gap of attempting to determine what precisely does Blender imply after they say “bicubic”,
in addition to what different software program means by “bicubic”. It’s fairly a multitude lol! See an interactive comparability I made right here:
aras-p.info/img/misc/upsample_filter_comp_2024

Anyway, “Bicubic” in all places inside Blender truly means “Cubic B-Spline” filtering, i.e.
Mitchell-Netravali filter with B=1, C=0 coefficients,
also called “no ringing, however a lot of blur”. Whether or not that’s a good selection will depend on use case and what do the pictures signify.
For VSE particularly, it appeared like the standard “Mitchell” filter (B=C=1/3) might need been higher. Right here’s each of them for instance:

Each sorts of cubic filtering are an choice in VSE now (#117100,
#117517).

For downscaling the picture, Blender 3.5 added a “Subsampled 3×3” filter. What it truly is, is a field filter that’s hardcoded to 3×3 measurement.
Whether or not field filter is an efficient filter, is a query for one more day. However for now at the very least, I made it not be hardcoded to mounted 3×3 measurement
(#117584),
since for those who scale the picture down by not 3×3, it kinda begins to interrupt down. Right here, downscaling this attitude grid by 4x on every axis:
unique picture, downscaled with present Subsampled 3×3 filter, and downscaled with the adjusted Field filter. Barely higher:

All of that’s rather a lot of selections for the person, TBH! So I added an “Auto” filter choice
(#117853), that’s now the default for VSE strips.
It routinely picks the “most acceptable” filter primarily based on remodel information:

• When there isn’t a scaling or rotation: Nearest,
• When scaling up by greater than 2x: Cubic Mitchell,
• When cutting down by greater than 2x: Field,
• In any other case: Bilinear.

Moreover all that, the picture filtering course of bought a bit quicker:

• Do away with digital capabilities from the inside loop, and a few SIMD for bilinear filtering (#115653),
• Simplify cubic filtering, and add some SIMD (#117100),
• Simplify math utilized by Field (née Subsampled3x3) filter (#117125),
• Repair “does a strong picture that covers the entire display, and so we will skip all the things underneath it” optimization not working, when mentioned picture has scale
  (#117786).

As a sensible instance, on my PC having a single 1920×1080 picture in a 3840×2160 venture (scaled up 2x), utilizing Bilinear filtering: drawing the entire
sequencer preview space went from 36.8ms down to fifteen.9ms. I’ve some concepts tips on how to velocity it up additional.

Optimizing VSE Results

Whereas the precise film information units I’ve from Blender Studio
don’t use a lot/any results, I optimized them by noticing one thing within the code.
Most of that’s simply multi-threading.

• Glow impact: multi-threaded now, 6x-10x quicker (#115818).
• Wipe impact: multi-threaded now, and simplify extreme trigonometry in Clock wipe; 6x-20x quicker
  (#115837).
• Gamma Cross impact: was doing actually complicated desk + interpolation primarily based issues simply to keep away from a single sq. root name.
  Felt just like the code was written earlier than {hardware} floating level was invented 🙂 4x quicker now
  (#115801).
• Gaussian Blur impact: 1.5x quicker by avoiding some redundant calculations (#116089).

What does all of that imply for render occasions?

On the three information units I’ve from Blender Studio, the ultimate render time of a VSE film is about 2x quicker on my PC.
For instance, the identical Sprite Fright edit: rendering it went from virtually
13 minutes all the way down to 7 minutes.

I hope issues will be additional sped up. We “solely” have to do 2x speedup one other 3 times, after which it’s fairly good, proper? 😛

Ideas on precise work course of

Is the entire above a “good quantity of labor” achieved, for 2 months part-time effort?

I don’t know. I believe it’s fairly okay, particularly contemplating that the developer (me) knew nothing concerning the space or the codebase. Moreover the
user-visible modifications outlined above, I did a handful of pull requests that have been including exams, refactoring code, cleansing one thing up, and many others. In whole
37 pull requests bought achieved, reviewed and merged.

And right here’s the attention-grabbing bit: I’m fairly certain I may haven’t achieved this at an “precise job”. I don’t have many roles to match,
however e.g. at Unity between round 2015 and 2022, I believe I might have been capable of do like 30% of the above in the identical time. Possibly much less.
I in all probability may have achieved the above at “historical” Unity, i.e. round yr 2010 or so.

The explanations are quite a few and complicated, and must do with quantity of individuals inside the firm, processes, expectations, communication, politics and
whatnot. However it’s weirdly humorous, that if I’m capable of do “X quantity of labor in Y period of time” totally free, then at an organization the place it might
pay me comparatively lotsa cash for the work, varied forces would attempt to make me do the identical work slower. Or not end the work in any respect, since
because of (once more, complicated causes) the hassle would possibly get cancelled halfway!

I hope Blender doesn’t enterprise into that measurement/complexity/workflow the place it appears like The Course of just isn’t serving to, however somewhat is there to
demotivate and decelerate everybody (not on goal! it simply slowly turns into that means).

What’s subsequent?

Who is aware of! Blender 4.1 simply turned beta, which suggests feature-wise, it’s “achieved” and the VSE associated bits in 4.1 are going to be
as they are right now.

Nonetheless, work on Blender 4.2 begins now, after which 4.3, … For the close to future, I wish to maintain tinkering with it. However with no clear plan 🙂
As soon as I’ve one thing achieved, perhaps I’ll write about them. In the meantime, issues will be noticed within the
Weekly Updates discussion board part.

Till subsequent time! Blissful video sequence modifying!