Leveraging Rust and the GPU to render person interfaces at 120 FPS

A contemporary show’s refresh price ranges from 60 to 120 frames per second, which implies an utility solely has 8.33ms per body to push pixels to display. This contains updating the applying state, laying out UI components, and at last writing knowledge into the body buffer.

It is a tight deadline, and for those who’ve ever constructed an utility with Electron, it is a deadline which will really feel unattainable to constantly meet. Engaged on Atom, that is precisely how we felt: irrespective of how onerous we tried, there was all the time one thing in the best way of delivering frames on time. A random pause on account of rubbish assortment and we missed a body. An costly DOM relayout and we missed one other body. The body price was by no means constant, and lots of the causes have been past our management.

But whereas we struggled to micro-optimize Atom’s rendering pipeline consisting of straightforward containers and glyphs, we stared in awe of pc video games rendering lovely, advanced geometry at a continuing price of 120 frames per second. How may or not it’s that rendering a couple of <div>s was a lot slower than drawing a three-dimensional, photorealistic character?

After we got down to construct Zed, we have been decided to create a code editor so responsive it nearly disappeared. Impressed by the gaming world, we realized that the one strategy to obtain the efficiency we wanted was to construct our personal UI framework: GPUI.

GPUI: Rendering

After we began constructing Zed, arbitrary 2D graphics rendering on the GPU was nonetheless very a lot a analysis mission. We experimented with Patrick Walton’s Pathfinder crate, however it wasn’t quick sufficient to realize our efficiency objectives.

So we took a step again, and reconsidered the issue we have been attempting to unravel. Whereas a library able to rendering arbitrary graphics might have been good, the reality was that we did not actually need it for Zed. In apply, most 2D graphical interfaces break down into a couple of primary components: rectangles, shadows, textual content, icons, and pictures.

As an alternative of worrying a couple of common function graphics library, we determined to concentrate on writing a customized shader for every particular graphical primitive we knew we would must render Zed’s UI. By describing the properties of every primitive in a data-driven approach on the CPU, we may delegate all the heavy-lifting to the GPU the place UI components might be drawn in parallel.

Within the following sections, I’m going as an instance the methods utilized in GPUI to attract every primitive.

Drawing rectangles

The standard rectangle is a basic constructing block of graphical UIs.

To grasp how drawing rectangles works in GPUI, we first must take a detour into the idea of Signed Distance Features (SDFs for brief). As implied by the identify, an SDF is a perform that, given an enter place, returns the space to the sting of some mathematically-defined object. The space approaches zero because the place will get nearer to the article, and turns into unfavourable when stepping inside its boundaries.

The record of recognized SDFs is intensive, principally due to Inigo Quilez’s seminal work on the topic. On his web site, you may also discover a unending collection of methods that enable distortion, composition and repetition of SDFs to generate probably the most advanced and sensible 3D scenes. Significantly, check it out. It is fairly superb.

Again to rectangles: let’s derive a SDF for them. We are able to simplify the issue by centering the rectangle we wish to draw on the origin. From right here, it is comparatively easy to see the issue is symmetric. In different phrases, calculating the space for some extent mendacity in one of many 4 quadrants is equal to calculating the space for the mirror picture of that time in any of the opposite three quadrants.

This implies we solely want to fret concerning the top-right portion of the rectangle. Taking the nook as a reference, we are able to distinguish three circumstances:

• Case 1), the purpose is each above and to the left of the nook. On this case, the shortest distance between the purpose and the rectangle is given by the vertical distance from the purpose to the highest edge.
• Case 2), the purpose is each under and to the precise of the nook. On this case, the shortest distance between the purpose and the rectangle is given by the horizontal distance from level to the precise edge.
• Case 3), the purpose is above and to the precise of the nook. On this case, we are able to use the Pythagorean theorem to find out the space between the nook and the purpose.

Case 3 could be generalized to cowl the opposite two if we forbid the space vector to imagine unfavourable elements.

The foundations we simply sketched out are enough to attract a easy rectangle and, later on this submit, we’ll describe how that interprets to GPU code. Earlier than we get to that although, we are able to make a easy remark that permits extending these guidelines to calculate the SDF of rounded rectangles too!

Discover how in case 3) above, there are infinitely many factors situated on the identical distance from the nook. In actual fact, these aren’t simply random factors, they’re the factors that describe a circle originating on the nook and having a radius equal to the space.

Borders begin to get smoother as we transfer away from the straight rectangle. That is the important thing perception to drawing rounded corners: given a desired nook radius, we are able to shrink the unique rectangle by it, calculate the space to the purpose and subtract the nook radius from the computed distance.

Porting the rectangle SDF to the GPU could be very intuitive. As a fast recap, the traditional GPU pipeline consists of a vertex and a fraction shader.

The vertex shader is liable for mapping arbitrary enter knowledge into factors in three-d house, with every set of three factors defining a triangle that we wish to draw on display. Then, for each pixel contained in the triangles generated by the vertex shader, the GPU invokes the fragment shader, which is liable for assigning a coloration to the given pixel.

In our case, we use the vertex shader to outline the bounding field of the form we wish to draw on display utilizing two triangles. We can’t essentially fill each pixel inside this field. That’s left to the fragment shader, which we’ll focus on subsequent.

The next code is in Steel Shader Language, and is designed for use with instanced rendering to attract a number of rectangles to the display in a single draw name:

struct RectangleFragmentInput {
    float4 place [[position]];
    float2 origin [[flat]];
    float2 measurement [[flat]];
    float4 background_color [[flat]];
    float corner_radius [[flat]];
};

vertex RectangleFragmentInput rect_vertex(
    uint unit_vertex_id [[vertex_id]],
    uint rect_id [[instance_id]],
    fixed float2 *unit_vertices [[buffer(GPUIRectInputIndexVertices)]],
    fixed GPUIRect *rects [[buffer(GPUIRectInputIndexRects)]],
    fixed GPUIUniforms *uniforms [[buffer(GPUIRectInputIndexUniforms)]]
) {
    float2 place = unit_vertex * rect.measurement + rect.origin;
    
    float4 device_position = to_device_position(place, viewport_size);
    return RectangleFragmentInput {
      device_position,
      rect.origin,
      rect.measurement,
      rect.background_color,
      rect.corner_radius
    };
}

To find out the colour to assign to every pixel inside this bounding field, the fragment shader calculates the space from the pixel to the rectangle and fills the pixel solely when it lies contained in the boundaries (i.e., when the space is zero):

float rect_sdf(
    float2 absolute_pixel_position,
    float2 origin,
    float2 measurement,
    float corner_radius
) {
    float2 half_size = measurement / 2.;
    float2 rect_center = origin + half_size;

    
    
    float2 pixel_position = abs(absolute_pixel_position - rect_center);

    
    float2 shrunk_corner_position = half_size - corner_radius;

    
    
    float2 pixel_to_shrunk_corner = max(float2(0., 0.), pixel_position - shrunk_corner_position);

    float distance_to_shrunk_corner = size(pixel_to_shrunk_corner);

    
    
    float distance = distance_to_shrunk_corner - corner_radius;

    return distance;
}

fragment float4 rect_fragment(RectangleFragmentInput enter [[stage_in]]) {
    float distance = rect_sdf(
        enter.place.xy,
        enter.origin,
        enter.measurement,
        enter.corner_radius,
    );
    if (distance > 0.0) {
        return float4(0., 0., 0., 0.);
    } else {
        return enter.background_color;
    }
}

Drop shadows

To render drop shadows in GPUI, we adopted a technique developed by Evan Wallace, co-founder of Figma. For completeness, I’ll summarize the contents of the weblog submit right here, however it’s positively price studying the unique article.

Usually, drop shadows in functions are rendered utilizing a Gaussian blur. For each output pixel, the Gaussian blur is the results of a weighted common of all the encircling enter pixels, with the load assigned to every pixel reducing for farther pixels in a approach that follows a Gaussian curve.

If we transfer to the continual realm, we are able to consider the method above because the convolution of an enter sign (within the discrete case, the pixels of a picture) with a Gaussian function (within the discrete case, a matrix representing the values of a Gaussian likelihood distribution). Convolution is a particular mathematical operator that produces a brand new perform by taking the integral of the product of two features, the place one of many features (it does not matter which) is mirrored concerning the y axis. On an intuitive stage, it really works as if we’re sliding the Gaussian curve all around the picture, calculating for each pixel a transferring, weighted common that samples from the Gaussian curve to find out the load of the encircling pixels.

One fascinating facet of Gaussian blurs is that they’re separable. That’s, the blur could be utilized individually alongside the x and y axes and the ensuing output pixel is identical as making use of a single blur in two dimensions.

Within the case of a rectangle, there exists a closed-form answer to attract its blurred model with out sampling neighboring pixels. It’s because rectangles are additionally separable, and could be expressed because the intersection of two Boxcar functions, one for every dimension:

The convolution of a Gaussian with a step perform is equal to the integral of the Gaussian, which yields the error function (additionally referred to as erf). Subsequently, producing a blurred straight rectangle is identical as blurring every dimension individually after which intersecting the 2 outcomes:

float rect_shadow(float2 pixel_position, float2 origin, float2 measurement, float sigma) {
    float2 bottom_right = origin + measurement;
    float2 x_distance = float2(pixel_position.x - origin.x, pixel_position.x - bottom_right.x);
    float2 y_distance = float2(pixel_position.y - origin.y, pixel_position.y - bottom_right.y);
    float2 integral_x = 0.5 + 0.5 * erf(x_distance * (sqrt(0.5) / sigma));
    float2 integral_y = 0.5 + 0.5 * erf(y_distance * (sqrt(0.5) / sigma));
    return (integral_x.x - integral_x.y) * (integral_y.x - integral_y.y);
}

A closed-form answer just like the one above, nevertheless, does not exist for the 2D convolution of a rounded rectangle with a Gaussian, as a result of the method for a rounded rectangle isn’t separable. The cleverness of Evan Wallace’s approximation comes from performing a closed-form, actual convolution alongside one axis, after which manually sliding the Gaussian alongside the other axis a finite quantity of occasions:

float blur_along_x(float x, float y, float sigma, float nook, float2 half_size) {
    float delta = min(half_size.y - nook - abs(y), 0.);
    float curved = half_size.x - nook + sqrt(max(0., nook * nook - delta * delta));
    float2 integral = 0.5 + 0.5 * erf((x + float2(-curved, curved)) * (sqrt(0.5) / sigma));
    return integral.y - integral.x;
}

float shadow_rounded(float2 pixel_position, float2 origin, float2 measurement, float corner_radius, float sigma) {
    float2 half_size = measurement / 2.;
    float2 middle = origin + half_size;
    float2 level = pixel_position - middle;

    float low = level.y - half_size.y;
    float excessive = level.y + half_size.y;
    float begin = clamp(-3. * sigma, low, excessive);
    float finish = clamp(3. * sigma, low, excessive);

    float step = (finish - begin) / 4.;
    float y = begin + step * 0.5;
    float alpha = 0.;
    for (int i = 0; i < 4; i++) {
        alpha += blur_along_x(level.x, level.y - y, sigma, corner_radius, half_size) * gaussian(y, sigma) * step;
        y += step;
    }

    return alpha;
}

Textual content Rendering

Rendering glyphs effectively is essential for a text-intensive utility like Zed. On the identical time, it’s equally essential to provide textual content that matches the appear and feel of the goal working system. To grasp how we solved each issues in GPUI, we have to perceive how textual content shaping and font rasterization work.

Text shaping refers back to the strategy of figuring out which glyphs must be rendered and the place they need to be positioned given some sequence of characters and a font. There are several open-source shaping engines, and working techniques normally present related APIs out of the field (e.g., CoreText on macOS). Shaping is usually considered fairly costly, much more so when coping with languages which can be inherently harder to typeset, resembling Arabic or Devanagari.

One key remark about the issue is that textual content usually does not change a lot throughout frames. For instance, enhancing a line of code does not have an effect on the encircling traces, so it might be unnecessarily costly to form these once more.

As such, GPUI makes use of the working system’s APIs to carry out shaping (this ensures that textual content appears in keeping with different native functions) and maintains a cache of text-font pairs to formed glyphs. When some piece of textual content is formed for the primary time, it will get inserted into the cache. If the next body accommodates the identical text-font pair, the formed glyphs get reused. Vice versa, if a text-font pair disappears from the next body, it will get deleted from the cache. This amortizes the price of shaping and limits it solely to textual content that modifications from one body to the opposite.

Font rasterization, alternatively, refers back to the strategy of changing a glyph’s vector illustration into pixels. There are a number of methods to implement a rasterizer, with traditional CPU rasterizers such because the one offered by the working system (e.g., CoreText on macOS) or FreeType, and with some more moderen analysis tasks doing so totally on the GPU utilizing compute shaders (e.g., Pathfinder, Forma, or Vello).

As talked about earlier than, nevertheless, our speculation with GPUI was that we may obtain maximal efficiency by writing shaders for particular primitives versus having a single engine able to rendering arbitrary vector graphics. For textual content particularly, our objective was to render largely static content material with out interactive transformations that matched the platform’s native visible fashion. Furthermore, the set of glyphs that must be rendered is finite and could be cached fairly successfully, so rendering on the CPU does not actually develop into a bottleneck.

Identical to with textual content shaping, we let the working system deal with glyph rasterization in order that textual content completely matches different native functions. Specifically, we rasterize solely the alpha part (the opacity) of the glyph: we’ll get into why in a bit bit. We really render as much as 16 completely different variants of every particular person glyph to account for sub-pixel positioning, since CoreText subtly adjusts antialiasing of glyphs to provide them the visible look of being shifted barely within the X and Y path.

See Also

How the Bassmate Pc got here to be

The ensuing pixels are then cached into an atlas, a long-lived texture saved on the GPU. The situation of every glyph within the atlas is saved on the CPU, and glyphs are precisely positioned within the atlas to make use of as little house as doable utilizing the bin-packing algorithm offered by etagere.

Lastly, utilizing the previously-computed shaping data, these glyphs are assembled collectively to kind the unique piece of textual content that the applying needed to render.

The above is completed in a single, instanced draw name that describes the goal location of the glyph together with its place within the atlas:

typedef struct {
  float2 target_origin;
  float2 atlas_origin;
  float2 measurement;
  float4 coloration;
} GPUIGlyph;

Discover how GPUIGlyph permits specifying a coloration for the glyph. That is the explanation why we beforehand rasterized the glyph utilizing solely its alpha channel. By solely storing the glyph’s opacity, we are able to fill it with any coloration we wish utilizing a easy multiplication and keep away from storing one copy of the glyph within the atlas for every coloration used.

struct GlyphFragmentInput {
    float4 place [[position]];
    float2 atlas_position;
    float4 coloration [[flat]];
};

vertex GlyphFragmentInput glyph_vertex(
    uint unit_vertex_id [[vertex_id]],
    uint glyph_id [[instance_id]],
    fixed float2 *unit_vertices [[buffer(GPUIGlyphVertexInputIndexVertices)]],
    fixed GPUIGlyph *glyphs [[buffer(GPUIGlyphVertexInputIndexGlyphs)]],
    fixed GPUIUniforms *uniforms [[buffer(GPUIGlyphInputIndexUniforms)]]
) {
    float2 unit_vertex = unit_vertices[unit_vertex_id];
    GPUIGlyph glyph = glyphs[glyph_id];
    float2 place = unit_vertex * glyph.measurement + glyph.origin;
    float4 device_position = to_device_position(place, uniforms->viewport_size);
    float2 atlas_position = (unit_vertex * glyph.measurement + glyph.atlas_origin) / uniforms->atlas_size;

    return GlyphFragmentInput {
        device_position,
        atlas_position,
        glyph.coloration,
    };
}

fragment float4 glyph_fragment(
    GlyphFragmentInput enter [[stage_in]],
    texture2d<float> atlas [[ texture(GPUIGlyphFragmentInputIndexAtlas) ]]
) {
    constexpr sampler atlas_sampler(mag_filter::linear, min_filter::linear);
    float4 coloration = enter.coloration;
    float4 pattern = atlas.pattern(atlas_sampler, enter.atlas_position);
    coloration.a *= pattern.a;
    return coloration;
}

It is fascinating to notice how the efficiency of composing textual content utilizing the glyph atlas approximates the bandwidth of the GPU, as we are actually copying bytes from one texture to the opposite and performing a multiplication alongside the best way. It does not get any sooner than that.

Icons and Photographs

Rendering icons and pictures in GPUI follows an identical approach because the one described in textual content rendering, so we can’t be spending an excessive amount of time overlaying that right here. Precisely like textual content, SVG icons are parsed after which rasterized into pixels on the CPU utilizing solely their alpha channel, in order that they are often tinted. Alternatively, pictures do not want tinting and so they’re uploaded on a separate texture whereas preserving their coloration.

Icons and pictures are lastly assembled again into their goal place utilizing a shader just like the glyph one illustrated above.

GPUI: the Component trait

To date we have mentioned the low-level particulars of how rendering is carried out. That concern, nevertheless, is totally abstracted away when creating an utility with GPUI. As an alternative, customers of the framework work together with the Component trait when they should create new graphical affordances that may’t be already expressed as a composition of current components:

pub trait Component {
    fn format(&mut self, constraint: SizeConstraint) -> Measurement;
    fn paint(&mut self, origin: (f32, f32), measurement: Measurement, scene: &mut Scene);
}

Structure in GPUI was closely impressed by Flutter. Particularly, components nest right into a tree construction the place constraints circulate down and sizes circulate up. A constraint specifies the minimal and most measurement a given aspect can take:

pub struct SizeConstraint {
    pub min: Measurement,
    pub max: Measurement,
}

pub struct Measurement {
    pub width: f32,
    pub peak: f32,
}

Relying on the character of a component, the format methodology can determine to provide a brand new set of constraints for its youngsters to account for any further visible particulars that the aspect is including. For instance, if a component needs to attract a 1px border round its little one, it ought to shrink the max.width and max.peak equipped by the dad or mum by 1px and provide the shrunk constraint to its little one:

pub struct Border {
    little one: Field<dyn Component>,
    thickness: f32,
    coloration: Colour,
}

impl Component for Border {
    fn format(&mut self, mut constraint: SizeConstraint) -> Measurement {
        constraint.max.x -= self.thickness;
        constraint.max.y -= self.thickness;
        let (width, peak) = self.little one.format(constraint);
        Measurement {
            width: width + self.thickness,
            peak: peak + self.thickness,
        }
    }

    fn paint(&mut self, origin: (f32, f32), measurement: Measurement, scene: &mut Scene) {
        
    }
}

As soon as the dimensions of components has been established, the aspect tree could be lastly painted. Portray consists of positioning a component’s youngsters in response to the format in addition to drawing visible affordances belonging to the aspect itself. On the finish of this course of, all the weather could have pushed their very own graphical elements to a platform-neutral Scene struct, a set of the primitives described within the rendering part above:

pub struct Scene {
    layers: Vec<Layer>
}

struct Layer {
    shadows: Vec<Shadow>,
    rectangles: Vec<Rectangle>,
    glyphs: Vec<Glyph>,
    icons: Vec<Icon>,
    picture: Vec<Picture>
}

The renderer follows a particular order when drawing primitives. It begins by drawing all shadows, adopted by all rectangles, then all glyphs, and so forth. This prevents some primitives from being painted in entrance of others: for instance, a rectangle may by no means be rendered on high of a glyph.

There are circumstances, nevertheless, the place that habits isn’t fascinating. As an illustration, an utility might wish to paint a tooltip aspect in entrance of a button, and so the background of the tooltip must be rendered on high of the button’s textual content. To handle this, components can push a Layer to the scene, which ensures their graphical components might be rendered on high of their dad or mum.

GPUI additionally helps creating new stacking contexts, which permits for arbitrary z-index positioning in a approach that carefully resembles the painter’s algorithm.

Persevering with the instance of the border above, the paint methodology ought to first push a Rectangle containing the border it needs to attract after which place the kid to not overlap with the newly-drawn border:

impl Component for Border {
    fn format(&mut self, mut constraint: SizeConstraint) -> Measurement {
        
    }

    fn paint(&mut self, origin: (f32, f32), mut measurement: Measurement, scene: &mut Scene) {
        scene.push_rectangle(Rectangle {
            origin,
            measurement,
            border_color: self.coloration,
            border_thickness: self.thickness,
        });

        let (mut child_x, mut child_y) = origin;
        child_x += self.thickness;
        child_y += self.thickness;

        let mut child_size = measurement;
        child_size.width -= self.thickness;
        child_size.peak -= self.thickness;

        self.little one.paint((child_x, child_y), child_size, scene);
    }
}

GPUI affords a number of components out of the field to provide a wealthy visible expertise. Some components solely change the place and measurement of their youngsters (e.g., Flex which implements the flex-box mannequin), whereas different components add new graphical affordances (e.g., Label which renders a bit of textual content with the given fashion).

Conclusion

This submit was a whirlwind tour of GPUI’s rendering engine and the way it will get packaged into an API that permits encapsulation of format and portray. One other large function GPUI serves is reacting to person occasions, sustaining utility state, and translating that state into components.

We’re trying ahead to speaking about that in a future submit, so keep tuned to listen to extra about it quickly!