Now Reading
Changing HLS/DASH – Media over QUIC

Changing HLS/DASH – Media over QUIC

2023-11-14 09:10:03

Low-latency, excessive bitrate, mass fan-out is tough. Who knew?

See Replacing WebRTC for the earlier publish on this sequence.


Should you’re utilizing HLS/DASH and your principal precedence is…

  • price: wait till there CDN choices.
  • latency: you need to critically take into account MoQ.
  • options: it should take some time to implement the whole lot.
  • vod: it really works nice, why change it?


Thanks for the constructive reception on Hacker News!
Anyway, I’m again.

I spent the final 9 years engaged on actually all aspects of HLS and Twitch’s extension: LHLS.
We hit a latency wall and my process was to search out another, initially WebRTC however that finally pivoted into Media over QUIC.

Hopefully this time I received’t be “dunning-Krugerering off a cliff”. Thanks random Reddit consumer for that confidence enhance.


Easy reply: Apple

In case your app delivers video over mobile networks, and the video exceeds both 10 minutes length or 5 MB of information in a 5 minute interval, you might be required to make use of HTTP Reside Streaming.

It’s an anti-climactic reply, however Twitch migrated from RTMP to HLS to keep away from getting kicked off the App Retailer.
The subsequent sentence offers a touch as to why:

In case your app makes use of HTTP Reside Streaming over mobile networks, you might be required to supply not less than one stream at 64 Kbps or decrease bandwidth.

This was again in 2009 when the iPhone 3GS was launched and AT&T’s community was struggling to meet the demand.
The important thing function of HLS was ABR: a number of copies of the identical content material at completely different bitrates.
This allowed the Apple-controlled HLS participant to scale back the bitrate relatively than pummel a poor megacorp’s mobile community.

DASH got here afterwards in an try and standardize HLS minus the managed by Apple half.
There’s positively some cool options in DASH however the core concepts are the same and so they even share the identical media container now.
So the 2 get bundled collectively as HLS/DASH.

However I’ll focus extra on HLS since that’s my shit.

The Good Stuff

Whereas we had been pressured to modify protocols on the tech equal of gunpoint, HLS really has some wonderful advantages.
The largest one is that it makes use of HTTP.

HLS/DASH works by breaking media into “segments”, every containing just a few seconds of media.
The participant will individually request every phase through a HTTP request and seamlessly sew them collectively.
New segments are continually being generated and introduced to the participant through a “playlist”.


Thanks for the filer picture, DALL·E

As a result of HLS makes use of HTTP, a service like Twitch can piggyback on the prevailing infrastructure of the web.
There’s a plethora of optimized CDNs, servers, and shoppers that each one communicate HTTP and can be utilized to move media.
You do must do some additional work to therapeutic massage stay video into HTTP semantics, but it surely’s value it.

The bottom line is using economies of scale to make it low-cost to mass distribute stay media.
Crafting particular person IP packets would possibly the right option to ship stay media with minimal latency (ie. WebRTC), but it surely’s not probably the most price efficient.

The Unhealthy Stuff

I hope you weren’t anticipating a fluff piece.


We had been considerably unhappy to bid farewell to Flash (gasp).
Twitch’s latency went from one thing like 3 seconds with RTMP to fifteen seconds with HLS.

There’s a boatload of latency sources, wherever from the length of segments to the frequency of playlist updates.
Over time we had been in a position to slowly in a position to chip away on the drawback, finally extending HLS to get latency again right down to theoretical RTMP ranges.
I documented our journey in case you’re within the gritty particulars.

However one huge supply of latency stays: T C P

I went into extra element with my previous blog post, however the issue is head-of-line blocking.
When you flush a body to the TCP socket, will probably be delivered reliably and so as.
Nevertheless, when the community is congested, the encoded media bitrate will exceed the community bitrate and queues will develop.
Frames will take longer and longer to achieve the participant till the buffer is depleted and the viewer will get to see their least favourite spinny boye.


> tfw HLS/DASH

A HLS/DASH participant can detect queuing and swap to a decrease bitrate through ABR.
Nevertheless, it will possibly solely do that at rare (ex. 2s) phase boundaries, and it will possibly’t renege any frames already flushed to the socket.
So in case you’re watching 1080p video and your community takes a dump, nicely you continue to must obtain seconds of unsustainable 1080p video earlier than you may swap right down to an inexpensive 360p.

You may’t simply put the toothpaste again within the tube in case you squeeze out an excessive amount of.
You gotta use the entire toothpaste, even when it takes for much longer to brush your enamel.

TCP toothpaste

Source. The analogy falls aside however I
get to make use of this picture once more.


HLS makes use of “good” shoppers and “dumb” servers.
The shopper decides what, when, why, and the best way to obtain every media playlist, phase, and body.
In the meantime the server simply sits there and serves HTTP requests.

The issue actually will depend on your perspective. Should you management:

  • shopper solely: Life is nice!
  • shopper and server: Life is nice! You may even lengthen the protocol!
  • server solely: Life is ache.

For a service like Twitch, the answer may appear easy: construct your personal shopper and server!
And we did, together with a baremetal stay CDN designed solely for HLS.

However until quite recently, we’ve got been pressured to make use of the Apple HLS participant on iOS for AirPlay or Safari help.
And naturally TVs, consoles, casting units, and others have their very own HLS gamers.
And in case you’re providing your baremetal stay CDN to the public, you may’t precisely power prospects to make use of your proprietary participant.

So that you’re caught with a dumb server and a bunch of dumb shoppers.
These dumb shoppers make dumb selections with no cooperation with the server, based mostly on imperfect info.


I really like the simplicity of HLS in comparison with DASH.
There’s one thing so satisfying a couple of text-based playlist that you would be able to really learn, versus a XML monstrosity designed by committee.


However sadly Apple controls HLS.

There’s a misalignment of incentives between Apple and the remainder of the trade.
I’m not even positive how Apple makes use of HLS, or why they’d care about latency, or why they insist on being the only real arbiter of a stay streaming protocol.
Pantos has finished an incredible and thankless job, but it surely appears like a stand-off.

For instance, LL-HLS initially required HTTP/2 server push and it took almost the whole trade to persuade Apple that this was a foul concept.
The upside is that we bought a mailing list to allow them to announce modifications to builders first… however don’t anticipate the flexibility to suggest modifications any time quickly.

DASH is it’s personal can of worms because it’s managed by MPEG.
The specs are behind a paywall or require patent licensing?
I can’t even inform if I’m going to get sued for parsing a DASH playlist with out paying the troll toll.

troll toll

Source. 🎵 You gotta pay the Troll Toll 🎵

You’re given a clean canvas and a brush to color the greenest of fields, what do you make?

green field

Source. Wow. That’s fairly the
inexperienced area.


After my previous blog post, I had just a few individuals hit up my DMs and declare they will do real-time latency with TCP.
And I’m positive just a few extra individuals will too after this publish, so that you get your personal part that muddles the narrative.

Sure, you are able to do real-time latency with TCP (or WebSockets) beneath excellent situations.

Nevertheless, it simply received’t work nicely sufficient on poor networks.
Congestion and buffer-bloat will completely wreck your protocol on poor networks.
Quite a lot of my time spent at Twitch was optimizing for the ninetieth percentile; the shoddy mobile networks in Brazil or India or Australia.

But when you’ll reinvent RTMP, there are some ways to reduce queuing however they’re fairly restricted.
That is particularly true in a browser setting when restricted to HTTP or WebSockets.

See my subsequent weblog publish about Changing RTMP.


Notably absent to date has been any point out of LL-HLS and LL-DASH.
These two protocols are supposed to decrease HLS/DASH latency respectively by breaking media segments into smaller chunks.

The chunks could be smaller, however they’re nonetheless served sequentially over TCP.
The latency ground is decrease however the latency ceiling continues to be simply as excessive, and also you’re nonetheless going to buffer throughout congestion.



We’re additionally approaching the restrict of what you are able to do with HTTP semantics.

  • LL-HLS has configurable latency at the price of and exponential variety of sequential requests within the important path. For instance, 20 HTTP requests a second per monitor nonetheless solely will get you +100ms of latency, which isn’t even viable for real-time latency.
  • LL-DASH could be configured right down to +0ms added latency, delivering frame-by-frame with chunked-transfer. Nevertheless it completely wrecks client-side ABR algorithms. Twitch hosted a challenge to enhance this however I’m satisfied it’s not possible with out server suggestions.

HESP additionally will get a particular shout-out as a result of it’s cool.
It really works by canceling HTTP requests throughout congestion and frankensteining the video encoding which is sort of hacky intelligent, however suffers the same destiny.

We’ve hit a wall with HTTP over TCP.


Should you’re an astute hyper textual content transport protocol aficionado, you might need observed that I stated “HTTP over TCP” above.
However HTTP/3 makes use of QUIC as a substitute of TCP.
Drawback solved! We are able to change any point out of TCP with QUIC!

Properly, not fairly. To make use of one other sophisticated subject as a metaphor:

  • A TCP connection is a single-core CPU.
  • A QUIC connection is a multi-core CPU.

Should you take a single threaded program and run it on a multi-core machine, it should run simply as sluggish, and even perhaps slower.
That is the case with HLS/DASH as every phase request is made sequentially.
HTTP/3 will not be a magic bullet and solely has marginal advantages when used with HLS/DASH.

The important thing to utilizing QUIC is to embrace concurrency.

This implies using a number of, impartial streams that share a connection.
You may prioritize a stream so it will get extra bandwidth throughout congestion, very like you should use good on Linux to prioritize a course of when CPU starved.
If a stream is taking too lengthy, you may cancel it very like you may kill a course of.

See Also

For stay media, you need to prioritize new media over outdated media in an effort to skip outdated content material.
You additionally need to prioritize audio over video, so you may hear what somebody is saying with out essentially seeing their lips transfer.
Should you can solely transmit half of a media stream in time, be sure it’s crucial half.

To Apple/Pantos’ credit score, LL-HLS is exploring prioritization using HTTP/3.
It doesn’t go far sufficient (but!) and HTTP semantics get in the way in which, but it surely’s completely the precise course.
I’m satisfied that any individual will make a HTTP/3 only media protocol sooner or later.

However after all I’m biased in direction of…

MoQ makes use of WebTransport/QUIC on to keep away from TCP and HTTP.
However what about that entire economies of scale stuff?

Properly, there are some essential variations between Media over QUIC as in comparison with your normal not invented right here protocol:

Motive 0: QUIC

QUIC is the way forward for the web.
TCP is a relic of the previous.

You’re going to see loads of this emblem, though not crudely traced or inexperienced.

It’s a daring declare I do know.
However I wrestle to consider a single motive why you’ll use TCP over QUIC going ahead.
There are nonetheless some company firewalls that block UDP (utilized by QUIC) and {hardware} offload doesn’t exist but, however I imply that’s about it.

It is going to take just a few years, however each library, server, load balancer, and NIC will likely be optimized for QUIC supply.
Media over QUIC offloads as a lot as potential into this highly effective layer.
We profit additionally from any new options, together with proposals resembling multi-path, FEC, congestion control, and many others.
I don’t need community options in my media layer thanks very a lot (taking a look at you WebRTC).

It won’t be apparent is that HTTP/3 is definitely a skinny layer on high of QUIC.
Likewise MoQ can also be meant to be a skinny layer on high of QUIC, successfully simply offering pub/sub semantics.
We get the entire advantages of QUIC with out the luggage of HTTP, and but nonetheless obtain internet help through WebTransport.

As an alternative we will deal with the essential stuff as a substitute: stay media.

Motive 1: Relay Layer

To keep away from the mistakes of WebRTC, we have to decouple the applying from the transport.
If a relay (ie. CDN) is aware of something about media encoding, we’ve got failed.

The concept is to interrupt MoQ into layers.

MoqTransport is the bottom layer and is a typical pub/sub protocol, though catered towards QUIC.
The appliance splits information into “objects”, annotated with a header offering easy directions on how the relay must ship it.
These are generic indicators, together with stuff just like the precedence, reliability, grouping, expiration, and many others.

MoqTransport is designed for use for arbitrary purposes.
Some examples embody:

  • stay chat
  • end-to-end encryption
  • sport state
  • stay playlists
  • or perhaps a clock!

That is large draw for CDN distributors.
As an alternative of constructing a customized WebRTC CDN that targets one particular area of interest, you may solid a a lot wider internet with MoqTransport.
Akamai, Google, and Cloudflare have been concerned within the standardization course of to date and CDN help is inevitable.

There will likely be not less than one media layer on high of MoqTransport.
We’re targeted on the transport proper now so there’s no official “adopted” draft but.

Nevertheless, my proposal is Warp.
It makes use of CMAF so it’s backwards appropriate with HLS/DASH whereas nonetheless able to real-time latency.
I believe that is critically essential, as any migration must be finished piecewise, client-by-client and user-by-user.
The identical media segments could be served for a blended roll-out and for VoD.

This web site makes use of Warp! Try it out! Or watch one in all my presentations.

There’ll completely be different mappings and containers; MoQ will not be married to CMAF.
The essential half is that solely the encoder/decoder perceive this media layer and never any relays within the center.
There’s loads of cool concepts floating round, resembling a live playlist format and a low-overhead container.

Motive 3: IETF

Media over QUIC is an IETF working group.

I crudely traced and recolored this emblem too.

If you realize nothing in regards to the IETF, simply know that it’s the requirements physique behind favorites resembling HTTP, DNS, TLS, QUIC, and even WebRTC.
However I believe this part is particularly essential:

There is no such thing as a membership within the IETF. Anybody can take part by signing as much as a working group mailing checklist (extra on that under), or registering for an IETF assembly. All IETF individuals are thought of volunteers and anticipated to take part as people, together with these paid to take part.

It’s not a protocol owned by an organization.
It’s not a protocol owned by legal professionals.

Join the mailing list.

Okay cool so hopefully I bought you on MoQ.
What can’t you utilize it in the present day to exchange HLS/DASH?

  1. It’s not finished but: The IETF is many issues, however quick will not be one in all them.
  2. Value: QUIC is a brand new protocol that has but to be totally optimized to match TCP. It’s potential and apparently Google is near parity.
  3. Assist: Your favourite language/library/cdn/cloud/browser won’t even present HTTP/3 help but, not to mention WebTransport or QUIC.
  4. Options: Any individual has to reimplement the entire annoying HLS/DASH options like DRM and server-side commercials…
  5. VoD: MoQ is presently stay solely. HLS/DASH work nice, why change it?

We’ll get there finally.

Be happy to make use of our Rust or Typescript implementation is you need to experiment.
Be part of the Discord if you wish to assist!

Written by @kixelated.

Source Link

What's Your Reaction?
In Love
Not Sure
View Comments (0)

Leave a Reply

Your email address will not be published.

2022 Blinking Robots.
WordPress by Doejo

Scroll To Top