Introducing graphics offload – GTK Improvement Weblog

A few of us within the GTK crew have spent the final month or so exploring the world of linux kernel graphics apis, particularly, dmabufs. We’re getting back from this journey with some frustrations and a few successes.

What’s a dmabuf?

A dmabuf is a reminiscence buffer in kernel house that’s recognized by a file descriptor. The concept is that you simply don’t have to repeat numerous pixel knowledge round, and as an alternative simply go a file descriptor between kernel subsystems.

Actuality is after all extra difficult that this rosy image: the reminiscence could also be system reminiscence that isn’t accessible in the identical method as ‘plain’ reminiscence, and there could also be multiple buffer (and multiple file descriptor), since graphics knowledge is commonly cut up into planes (e.g. RGB and A could also be separate, or Y and UV).

Why are dmabufs helpful?

I’ve already talked about that we hope to keep away from copying the pixel knowledge and feeding it by the GTK compositing pipeline (and with 4k video, that may be fairly a bit of knowledge for every body).

The use circumstances the place this sort of optimization issues are these the place continuously altering content material is displayed for a very long time, corresponding to

• Video gamers
• Digital machines
• Streaming
• Screencasting
• Video games

In the most effective case, we could possibly keep away from feeding the information by the compositing pipeline of the compositor as effectively, if the compositor helps direct scanout and the dmabuf is appropriate for it.  Specifically on cellular programs, this will keep away from utilizing the GPU altogether, thereby decreasing energy consumption.

Particulars

GTK has already been utilizing dmabufs since 4.0: When composing a body, GTK interprets all of the render nodes (sometimes a number of for every widget) into GL instructions, sends these to the GPU, and mesa then exports the ensuing texture as a dmabuf and attaches it to our Wayland floor.

But when the one factor that’s altering in your UI is the video content material that’s already in a dmabuf, it might be good to keep away from the detour by GL and simply hand the information on to the compositor, by giving it the file descriptor for the the dmabuf.

Wayland has the idea of subsurfaces that permit functions defer a few of their compositing must the compositor: The applying attaches a buffer to every (sub)floor, and it’s the job of the compositor to mix all of them collectively.

With what’s now in git foremost, GTK will create subsurfaces as-needed as a way to go dmabufs on to the compositor. We will do that in two other ways: If nothing is drawn on prime of the dmabuf (no rounded corners, or overlaid controls), then we will stack the subsurface above the primary floor with out altering any of the visuals.

That is the perfect case, because it permits the compositor to arrange direct scanout, which provides us a zero-copy path from the video decoder to the show.

If there’s content material that will get drawn on prime of the video, we could not have the ability to get that, however we will nonetheless get the good thing about letting the compositor do the compositing, by putting the subsurface with the video under the primary floor and poking a translucent gap in the primary floor to let it peek by.

The spherical play button is what forces the subsurface to be positioned under the primary floor right here.

GTK picks these modes routinely and transparently for every body, with out the applying developer having to do something. As soon as that play button seems in a body, we place the subsurface under, and as soon as the video is clipped by rounded corners, we cease offloading altogether. After all, some great benefits of offloading additionally disappear.

The graphics offload visualization within the GTK inspector reveals these adjustments as they occur:

Initially, the digital camera stream will not be offloaded as a result of the rounded corners clip it. The magenta define signifies that the stream is offloaded to a subsurface under the primary floor (as a result of the video controls are on prime of it). The golden define signifies that the subsurface is above the primary floor.

How do you utilize this?

GTK 4.14 will introduce a GtkGraphicsOffload widget, whose solely job it’s to provide a touch that GTK ought to attempt to offload the content material of its baby widget by attaching it to a subsurface as an alternative of letting GSK course of it prefer it normally does.

To create appropriate content material for offloading, the brand new GdkDmabufTextureBuilder wraps dmabufs in GdkTexture objects. Typical sources for dmabufs are pipewire, video4linux or gstreamer. The dmabuf help in gstreamer will probably be rather more stable within the upcoming 1.24 launch.

When testing this code, we used the GtkMediaStream implementation for pipewire by Georges Basile Stavracas Neto that may be present in pipewire-media-stream and libmks by Christian Hergert and Bilal Elmoussaoui.

What are the restrictions?

In the meanwhile, graphics offload will solely work with Wayland on Linux. There’s some hope that we could possibly implement comparable issues on MacOS, however for now, that is Wayland-only. It additionally relies on the content material being in dmabufs.

Purposes that need to make the most of this have to play alongside and keep away from doing issues that intrude with using subsurfaces, corresponding to rounding the corners of the video content material. The GtkGraphicsOffload docs have extra particulars for builders on constraints and debug issues with graphics offload.

Abstract

The GTK 4.14 launch may have some attention-grabbing new capabilities for media playback. You may strive it now, with the just-released 4.13.3 snapshot.

Please strive it and tell us what does and doesn’t give you the results you want.