JPEG XL and the Pareto Entrance

Model 0.10 of libjxl, the reference implementation for JPEG XL, has simply been released. The principle enchancment this model brings, is that the so-called “streaming encoding” API has now been absolutely applied. This API permits encoding a big picture in “chunks.” As a substitute of processing your entire picture directly, which can require a major quantity of RAM if the picture is massive, the picture can now be processed in a extra memory-friendly method. As a aspect impact, encoding additionally turns into quicker, particularly when doing lossless compression of bigger photos.

Earlier than model 0.10, lossless JPEG XL encoding was fairly memory-intensive and will take fairly a while. This might pose a significant issue when attempting to encode massive photos, like for instance this 13500×6750 NASA image of Earth at night (64 MB as a TIFF file, 273 MB uncompressed).

Compressing this picture required about 8 gigabytes of RAM in libjxl model 0.9, on the default effort (e7). It took over two minutes, and resulted in a jxl file of 33.7 MB, which is slightly below 3 bits per pixel. Utilizing extra threads didn’t assist a lot: utilizing a single thread it took 2m40s, utilizing eight threads that was lowered to 2m06s. These timings had been measured on a November 2023 Macbook Professional with a 12-core Apple M3 Professional CPU with 36 GB of RAM.

Upgrading to libjxl model 0.10, compressing this identical picture now requires solely 0.7 gigabytes of RAM, takes 30 seconds utilizing a single thread (or 5 seconds utilizing eight threads), and ends in a jxl file of 33.2 MB.

For different effort settings, these are the outcomes for this explicit picture:

As you’ll be able to see within the desk above, compression is a recreation of diminishing returns: as you improve the quantity of cpu time spent on the encoding, the compression improves, however not in a linear style. Spending one second as a substitute of a tenth of a second (e2 as a substitute of e1) can on this case shave off 22 megabytes; spending 5 seconds as a substitute of 1 (e7 as a substitute of e2) shaves off one other 11 megabytes. However to shave off another megabyte, you’ll have to attend nearly two minutes (e9 as a substitute of e7).

So it’s very a lot a matter of trade-offs, and it is dependent upon the use case what makes essentially the most sense. In an authoring workflow, while you’re saving a picture regionally whereas nonetheless enhancing it, you usually don’t want robust compression and low-effort encoding is smart. However in a one-to-many supply state of affairs, or for long-term archival, it could be price it to spend a major quantity of CPU time to shave off some extra megabytes.

When evaluating completely different compression strategies, it doesn’t suffice to solely have a look at the compressed file sizes. The velocity of encoding additionally issues. So there are two dimensions to contemplate: compression density and encode velocity.

A particular methodology could be referred to as Pareto-optimal if no different methodology can obtain the identical (or higher) compression density in much less time. There is likely to be different strategies that compress higher however take extra time, or that compress quicker however lead to bigger information. However a Pareto-optimal methodology delivers the smallest information for a given time funds, which is why it’s referred to as “optimum.”

The set of Pareto-optimal strategies is named the “Pareto front.” It may be visualized by placing the completely different strategies on a chart that reveals each dimensions — encode velocity and compression density. As a substitute of taking a look at a single picture, which might not be consultant, we have a look at a set of photos and have a look at the typical velocity and compression density for every encoder and energy setting. For instance, for this set of test images, the chart appears to be like like this:

The vertical axis reveals the encode velocity, in megapixels per second. It’s a logarithmic scale because it has to cowl a broad vary of speeds, from lower than one megapixel per second to a whole bunch of megapixels per second. The horizontal axis reveals the typical bits per pixel for the compressed picture (uncompressed 8-bit RGB is 24 bits per pixel).

For AVIF, the darker factors point out a quicker however barely much less dense tiled encoder setting (utilizing –tilerowslog2 2 –tilecolslog2 2), which is quicker as a result of it could actually make higher use of multi-threading, whereas the lighter factors point out the default non-tiled setting. For PNG, the results of libpng with default settings is proven right here as a reference level; different PNG encoders and optimizers exist that attain completely different trade-offs.

The earlier model of libjxl already achieved Pareto-optimal outcomes throughout all speeds, producing smaller information than PNG and lossless AVIF or lossless WebP. The brand new model beats the earlier model by a major margin.

Not proven on the chart is QOI, which clocked in at 154 Mpx/s to realize 17 bpp, which can be “fairly OK” however is kind of removed from Pareto-optimal, contemplating the bottom effort setting of libjxl compresses right down to 11.5 bpp at 427 Mpx/s (so it’s 2.7 occasions as quick and the result’s 32.5% smaller).

After all in these charts, quite a bit is dependent upon the collection of check photos. Within the chart above, most photos are images, which are typically exhausting to compress losslessly: the naturally occurring noise in such photos is inherently incompressible.

For non-photographic photos, issues are considerably completely different. I took a random assortment of manga photos in numerous drawing types (41 photos with a mean dimension of seven.3 megapixels) and these had been the outcomes:

These sorts of photos compress considerably higher, to round 4 bpp (in comparison with round 10 bpp for photographic photos). For these photos, lossless AVIF just isn’t helpful — it compresses worse than PNG, and reaches about the identical density as QOI however is far slower. Lossless WebP alternatively achieves excellent compression for such photos. For a majority of these photos, QOI is certainly fairly OK for its velocity (and ease), although removed from Pareto-optimal: low-effort JPEG XL encoding is twice as quick and 31% smaller.

For non-photographic photos, the brand new model of libjxl once more improves upon the earlier model, by a major margin. The earlier model of libjxl may simply barely beat WebP: e.g. default-effort WebP compressed these photos to 4.30 bpp at 2.3 Mpx/s, whereas libjxl 0.9 at effort 5 compressed them to 4.27 bpp at 2.6 Mpx/s — solely a slight enchancment. Nevertheless libjxl 0.10 at effort 5 compresses the photographs to 4.25 bpp at 12.2 Mpx/s (barely higher compression however a lot quicker), and at effort 7 it compresses them to 4.04 bpp at 5.9 Mpx/s (considerably higher compression and nonetheless twice as quick). Zooming in on the medium-speed a part of the Pareto entrance on the above plot, the advance going from libjxl 0.9 to 0.10 turns into clear:

Lossless compression is comparatively straightforward to benchmark: all that issues is the compressed dimension and the velocity. For lossy compression, there’s a third dimension: picture high quality.

Lossy picture codecs and encoders can carry out in a different way at completely different high quality factors. An encoder that works very properly for high-quality encoding doesn’t essentially additionally carry out properly for low-quality encoding, and the opposite method round.

Of those three dimensions (compression, velocity and high quality), typically velocity is just ignored, and plots are manufactured from compression versus high quality (also referred to as bitrate-distortion plots). However this doesn’t actually enable evaluating the trade-offs between encode effort (velocity) and compression efficiency. So if we actually need to examine the Pareto entrance for lossy compression, a method of doing it’s to take a look at completely different “slices” of the three-dimensional area, at numerous high quality factors.

Picture high quality is a notoriously troublesome factor to measure: ultimately, it’s subjective and considerably completely different from one human to the subsequent. One of the best ways to measure picture high quality remains to be to run an experiment involving not less than dozens of people trying fastidiously at photos and evaluating or scoring them, in accordance with rigorously outlined check protocols. At Cloudinary, we’ve done such experiments up to now. However whereas that is one of the simplest ways to evaluate picture high quality, it’s a time-consuming and dear course of, and it isn’t possible to check all potential encoder configurations on this method.

For that purpose, so-called goal metrics are being developed, which permit algorithmic estimates of picture high quality. These metrics are usually not “extra goal” (within the sense of “extra right”) than scores obtained from testing with people, in actual fact they’re much less “right.” However they can provide a sign of picture high quality a lot quicker and cheaper (and extra simply reproducible and constant) than when people are concerned, which is what makes them helpful.

The most effective metrics presently publicly obtainable are SSIMULACRA2, Butteraugli, and DSSIM. These metrics attempt to mannequin the human visible system and have the best correlation with subjective results. Older, less complicated metrics like PSNR or SSIM may be used, however they don’t correlate very properly with human opinions about picture high quality. Care must be taken to not measure outcomes utilizing a metric an encoder is particularly optimizing for, as that will skew the ends in favor of such encoders. For instance, higher-effort libjxl optimizes for Butteraugli, whereas libavif can optimize for PSNR or SSIM. On this respect, SSIMULACRA2 is “secure” since not one of the encoders examined is utilizing it internally for optimization.

Totally different metrics will say various things, however there are additionally other ways to combination outcomes throughout a set of photos. To maintain issues easy, I chosen encoder settings such that when utilizing every setting on all photos within the set, the typical SSIMULACRA2 rating was equal to (or near) a selected worth. One other methodology would have been to regulate the encoder settings per picture so for every picture the SSIMULACRA2 rating is similar, or to pick out an encoder setting such that the worst-case SSIMULACRA2 rating is the same as a selected worth.
Aligning on worst-case scores is favorable for constant encoders (encoders that reliably produce the identical visible high quality given fastened high quality settings), whereas aligning on common scores is favorable for inconsistent encoders (encoders the place there’s extra variation in visible high quality when utilizing fastened high quality settings). From earlier research, we all know that AVIF and WebP are extra inconsistent than JPEG and HEIC, and that JPEG XL has essentially the most constant encoder:

Defining the standard factors the best way I did (utilizing fastened settings and aligning by common metric rating) is within the favor of WebP and AVIF; in sensible utilization you’ll seemingly need to align on worst-case metric rating (or fairly, worst-case precise visible high quality), however I selected not to try this, so as to not favor JPEG XL.

Lossless compression presents 2:1 to three:1 compression ratios (8 to 12 bpp) for photographic photos. Lossy compression can attain a lot better compression ratios. It’s tempting to see how lossy picture codecs behave when they’re pushed to their limits. Compression ratios of fifty:1 and even 200:1 (0.1 to 0.5 bpp) could be obtained, at the price of introducing compression artifacts. Right here is an instance of a picture compressed to achieve a SSIMULACRA2 rating of fifty, 30, and 10 utilizing libjpeg-turbo, libjxl and libavif:

This sort of high quality is fascinating to take a look at in experiments, however in most precise utilization, it isn’t fascinating to introduce such noticeable compression artifacts. In apply, the vary of qualities that’s related corresponds to SSIMULACRA2 scores starting from 60 (medium high quality) to 90 (visually lossless). These qualities appear to be this:

Visually lossless high quality (SSIMULACRA2 = 90) could be reached with a compression ratio of about 8:1 (3 bpp) with trendy codecs equivalent to AVIF and JPEG XL, or about 6:1 (4 bpp) with JPEG. At this level, the picture is visually not distinguishable from the uncompressed unique, even when trying very fastidiously. In cameras, when not taking pictures RAW, usually that is the form of high quality that’s desired. For internet supply, it’s overkill to make use of such a top quality.

Top quality (SSIMULACRA2 = 80) could be reached with a compression ratio of 16:1 (1.5 bpp). When trying fastidiously, very small variations is likely to be seen, however primarily the picture remains to be nearly as good as the unique. This, or maybe one thing in between prime quality and visually lossless high quality, is the best high quality helpful for internet supply, to be used instances the place picture constancy actually issues.

Medium-high high quality (SSIMULACRA2 = 70) could be reached with a compression ratio of 30:1 (0.8 bpp). There are some small artifacts, however the picture nonetheless appears to be like good. It is a good goal for many internet supply use instances, because it makes trade-off between constancy and bandwidth optimization.

Medium high quality (SSIMULACRA2 = 60) could be reached with a compression ratio of 40:1 (0.6 bpp). Compression artifacts begin to change into extra noticeable, however they’re not problematic for informal viewing. For non-critical photos on the net, this high quality could be “adequate.”

Any high quality decrease than that is doubtlessly dangerous: positive, bandwidth will likely be lowered by going even additional, however at the price of doubtlessly ruining the photographs. For the net, in 2024, the related vary is medium to prime quality: according to the HTTP Archive, the median AVIF picture on the net is compressed to 1 bpp, which corresponds to medium-high high quality, whereas the median JPEG picture is 2.1 bpp, which corresponds to prime quality. For many non-web use instances (e.g. cameras), the related vary is excessive to (visually) lossless high quality.

Within the following Pareto entrance plots, the next encoders had been examined:

These are the newest variations of every encoder on the time of writing (finish of February 2024).

Encode velocity was once more measured on a November 2023 Macbook Professional (Apple M3 Professional), utilizing 8 threads. For AVIF, each the tiled setting (with –tilerowslog2 2 –tilecolslog2 2) and the non-tiled settings had been examined. The tiled setting, indicated with “MT”, is quicker because it permits higher multi-threading, but it surely comes at a value in compression density.

Let’s begin by trying on the outcomes for medium high quality, i.e., settings that lead to a corpus common SSIMULACRA2 rating of 60. This is kind of the bottom high quality level that’s utilized in apply. Some photos could have seen compression artifacts with these encoder settings, so this high quality level is most related when saving bandwidth and decreasing web page weight is extra necessary than picture constancy.

To start with, word that even for a similar format, completely different encoders and completely different effort settings can attain fairly completely different outcomes. Traditionally, essentially the most generally used JPEG encoder was libjpeg-turbo — typically utilizing its default setting (no Huffman optimization, not progressive), which is the purpose all the best way within the prime proper. When Google first launched WebP, it outperformed libjpeg-turbo when it comes to compression density, as could be seen within the plot above. However Mozilla was not impressed, and so they created their very own JPEG encoder, mozjpeg, which is slower than libjpeg-turbo however presents higher compression outcomes. And certainly, we are able to see that mozjpeg is definitely extra Pareto-efficient than WebP (for this corpus, at this high quality level).

Extra just lately, the JPEG XL workforce at Google has constructed one more JPEG encoder, jpegli, which is each quicker and higher than even mozjpeg. It’s primarily based on classes discovered from guetzli and libjxl, and presents a really enticing trade-off: it is rather quick, compresses higher than WebP and even high-speed AVIF, whereas nonetheless producing good previous JPEG information which are supported all over the place.

Shifting on to the newer codecs, we are able to see that each AVIF and HEIC can acquire a greater compression density than JPEG and WebP, at the price of slower encoding. JPEG XL can attain an identical compression density however encodes considerably quicker. The present Pareto entrance for this high quality level consists of JPEG XL and the varied JPEG encoders for the “affordable” speeds, and AVIF on the slower speeds (although the extra financial savings over default-effort JPEG XL are small).

At considerably larger high quality settings the place the typical SSIMULACRA2 rating for the corpus is 70, the general outcomes look fairly comparable:

Shifting on to the best high quality level that’s related for the net (corpus common SSIMULACRA2 rating of 85, to make sure that most photos attain a rating above 80), the variations change into a bit extra pronounced.

At this level, mozjpeg not beats WebP, although jpegli nonetheless does. The Pareto entrance is now largely lined by JPEG XL, although for very quick encoding, good previous JPEG remains to be finest. At this high quality level, AVIF just isn’t on the Pareto entrance: at its slowest settings (at 0.5 Mpx/s or slower) it matches the compression density of the second-fastest libjxl setting, which is over 100 occasions as quick (52 Mpx/s).

Up to now, we’ve solely checked out compression density and encode velocity. Decode velocity just isn’t actually a major downside on trendy computer systems, however it’s fascinating to take a fast have a look at the numbers. The desk under reveals the identical outcomes because the plot above, however moreover bits per pixel and encode velocity, it additionally reveals the decode velocity. For completeness, the SSIMULACRA2 and Butteraugli 3-norm scores are additionally given for every encoder setting.

Sequential JPEG is unbeatable when it comes to decode velocity — not stunning for a codec that was designed within the Eighties. Progressive JPEG (e.g. as produced by mozjpeg and default jpegli) is considerably slower to decode, however nonetheless quick sufficient to load any reasonably-sized picture within the blink of an eye fixed. JPEG XL is someplace in between these two.

Apparently, the decode velocity of AVIF is dependent upon how the picture was encoded: it’s quicker when utilizing the faster-but-slightly-worse multi-tile encoding, slower when utilizing the default single-tile encoding. Nonetheless, even the slowest decode velocity measured right here might be “quick sufficient,” particularly in comparison with the encode speeds.

Lastly, let’s check out the outcomes for visually lossless high quality:

WebP just isn’t on this chart because it merely can not attain this high quality level, not less than not utilizing its lossy mode. It is because 4:2:0 chroma subsampling is compulsory in WebP. Additionally clearly mozjpeg was not designed for this high quality level, and performs worse than libjpeg-turbo in each compression and velocity.

At their default velocity settings, libavif is 20% smaller than libjpeg-turbo (although it takes an order of magnitude longer to encode), whereas libjxl is 20% smaller than libavif and a pair of.5 occasions as quick, at this high quality level. The Pareto entrance consists of largely JPEG XL however on the fastests speeds once more additionally consists of JPEG.

Within the plots above, the test set consisted of web-sized photos of about 1 megapixel every. That is related for the net, however for instance when storing digital camera photos, photos are bigger than this.

For a check set with bigger photos (the same set we used before to check lossless compression), at a top quality level, we get the next outcomes:

Now issues look fairly completely different than with the smaller, web-sized photos. WebP, mozjpeg, and AVIF are worse than libjpeg-turbo (for these photos, at this high quality level). HEIC brings vital financial savings over libjpeg-turbo, although so does jpegli, at a a lot better velocity. JPEG XL is the clear winner, compressing the photographs to lower than 1.3 bpp whereas AVIF, libjpeg-turbo, and WebP require greater than 2 bpp.

Whereas not as dramatic because the enhancements in lossless compression, additionally for lossy compression there have been enhancements between libjxl 0.9 and libjxl 0.10. On the default effort setting (e7), that is how the reminiscence and velocity modified for a big (39 Mpx) picture:

The brand new model of libjxl brings a really substantial discount in reminiscence consumption, by an order of magnitude, for each lossy and lossless compression. Additionally the velocity is improved, particularly for multi-threaded lossless encoding the place the default effort setting is now an order of magnitude quicker.

This consolidates JPEG XL’s place as the very best picture codec presently obtainable, for each lossless and lossy compression, throughout the standard vary however particularly for prime quality to visually lossless high quality. It’s Pareto-optimal throughout a variety of velocity settings.

In the meantime, the previous JPEG remains to be enticing thanks to raised encoders. The brand new jpegli encoder brings a major enchancment over mozjpeg when it comes to each velocity and compression. Maybe surprisingly, good previous JPEG remains to be a part of the Pareto entrance — when extraordinarily quick encoding is required, it could actually nonetheless be your best option.

At Cloudinary, we’re actively taking part in enhancing the cutting-edge in picture compression. We’re repeatedly making use of new insights and applied sciences so as to convey the absolute best expertise to our end-users. As new codecs emerge and encoders for present codes enhance, we maintain ensuring to ship media in accordance with the cutting-edge.