Re: [Moq] Latency @ Twitch

[Piers O'Hanlon <[e-mail]>]

I guess it depends whether a QUIC API would allow for obtaining per flow stats or just info for the entire connection? Using information from ACKs just for your flow may provide a clearer picture…

Best,

Piers

On 10 Nov 2021, at 15:28, Joerg Ott <[e-mail]<mailto:[e-mail]>> wrote:

If you can get to QUIC's readily available stats it seems you can
roughly approximate what you need for know for building some of the
RTP CC on top without extra RTCP feedback packets.

Per-packet timestamps mentioned by Justin would still be a nice to have.

Best,
Jörg

On 10.11.21 00:01, Kirill Pugin wrote:
Fwiw, for RUSH/QUIC we tested RMCAT as CC 😃
We did have to “re-implement” RTCP feedback, but that was relatively straightforward and we could re-use RMCAT implementation that was used for video calling applications.
Best,
Kirill
*From: *Luke Curley <[e-mail]<mailto:[e-mail]>>
*Date: *Tuesday, November 9, 2021 at 2:58 PM
*To: *"Mo Zanaty (mzanaty)" <mzanaty=[e-mail]<mailto:mzanaty=[e-mail]>>
*Cc: *Bernard Aboba <[e-mail]<mailto:[e-mail]>>, Justin Uberti <[e-mail]<mailto:[e-mail]>>, Ian Swett <[e-mail]<mailto:[e-mail]>>, "Ali C. Begen" <[e-mail]<mailto:[e-mail]>>, MOQ Mailing List <[e-mail]<mailto:[e-mail]>>
*Subject: *Re: [Moq] Latency @ Twitch
Maybe a dumb thought, but is the PROBE_RTT phase required when sufficiently application limited, as is primarily the case for live video? If I understand correctly, it's meant to drain the queue to remeasure the minimum RTT, but that doesn't seem necessary when the queue is constantly being drained due to a lack of data to send.
Either way, the issue is that existing TCP algorithms don't care about the live video use-case, and those are the ones that have been ported to QUIC thus far. But like Justin mentioned, this doesn't actually matter for the sake of standardizing a video over QUIC protocol provided the building blocks are in place.
The real question is: do QUIC ACKs contain enough signal to implement an adequate live video congestion control algorithm? If not, how can we increase that signal, potentially taking cues from RMCAT (ex. RTT on a per-packet basis)?
On Tue, Nov 9, 2021, 10:27 AM Mo Zanaty (mzanaty) <mzanaty=[e-mail]<mailto:mzanaty=[e-mail]> <mailto:[e-mail]>> wrote:
   All current QUIC CCs (BBRv1/2, CUBIC, NewReno, etc.) are not well
   suited for real-time media, even for a rough “envelope” or
   “circuit-breaker”. RMCAT CCs are explicitly designed for real-time
   media, but, of course, rely on RTCP feedback, so must be adapted to
   QUIC feedback.
   Mo
   On 11/9/21, 1:13 PM, "Bernard Aboba" <[e-mail]<mailto:[e-mail]>
   <mailto:[e-mail]>> wrote:
   Justin said:
   "As others have noted, BBR does not work great out of the box for
   realtime scenarios."
   [BA] At the ICCRG meeting on Monday, there was an update on BBR2:
   https://datatracker.ietf.org/meeting/112/materials/slides-112-iccrg-bbrv2-update-00.pdf
   <https://datatracker.ietf.org/meeting/112/materials/slides-112-iccrg-bbrv2-update-00.pdf>
   While there are some improvements, issues such as "PROBE_RTT" and
   rapid rampup after loss remain, and overall, it doesn't seem like
   BBR2 is going to help much with realtime scenarios. Is that fair?
   On Tue, Nov 9, 2021 at 12:46 PM Justin Uberti
   <[e-mail]<mailto:[e-mail]>
   <mailto:[e-mail]>> wrote:
       Ultimately we found that it wasn't necessary to standardize the
       CC as long as the behavior needed from the remote side (e.g.,
       feedback messaging) could be standardized.
       As others have noted, BBR does not work great out of the box for
       realtime scenarios. The last time this was discussed, the
       prevailing idea was to allow the QUIC CC to be used as a sort of
       circuit-breaker, but within that envelope the application could
       use whatever realtime algorithm it preferred (e.g, goog-cc).
       On Thu, Nov 4, 2021 at 3:58 AM Piers O'Hanlon
       <[e-mail]<mailto:[e-mail]> <mailto:[e-mail]>> wrote:
               On 3 Nov 2021, at 21:46, Luke Curley
               <[e-mail]<mailto:[e-mail]> <mailto:[e-mail]>> wrote:
               Yeah, there's definitely some funky behavior in BBR when
               application limited but it's nowhere near as bad as
               Cubic/Reno. With those algorithms you need to burst
               enough packets to fully utilize the congestion window
               before it can be grown. With BBR I believe you need to
               burst just enough to fully utilize the pacer, and even
               then this condition
               <https://source.chromium.org/chromium/chromium/src/+/master:net/third_party/quiche/src/quic/core/congestion_control/bbr_sender.cc;l=393> lets
               you use application-limited samples if they would
               increase the send rate.
           And there’s also the idle cwnd collapse/reset behaviour to
           consider if you’re sending a number of frames together and
           their inter-data gap exceeds the RTO - I’m not quite sure
           how the various QUIC stacks have translated RFC2861/7661
           advice on this…?
               I started with BBR first because it's simpler, but I'm
               going to try out BBR2 at some point because of the
               aforementioned PROBE_RTT issue. I don't follow the
               congestion control space closely enough; are there any
               notable algorithms that would better fit the live video
               use-case?
           I guess Google’s Goog_CC appears to be well used in the
           WebRTC space (e.g. WEBRTC
           <https://webrtc.googlesource.com/src/+/refs/heads/main/modules/congestion_controller/goog_cc> and
           aiortc
           <https://github.com/aiortc/aiortc/blob/1a192386b721861f27b0476dae23686f8f9bb2bc/src/aiortc/rate.py#L271>)
           despite the draft
           <https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc> never
           making it to RFC status… There's also SCREAM
           <https://datatracker.ietf.org/doc/rfc8298/> which has an
           open source implementation
           <https://github.com/EricssonResearch/scream> but not sure
           how widely deployed it is.
               On Wed, Nov 3, 2021 at 2:12 PM Ian Swett
               <[e-mail]<mailto:[e-mail]> <mailto:[e-mail]>> wrote:
                    From personal experience, BBR has some issues with
                   application limited behavior, but it is still able
                   to grow the congestion window, at least slightly, so
                   it's likely an improvement over Cubic or Reno.
                   On Wed, Nov 3, 2021 at 4:40 PM Luke Curley
                   <[e-mail]<mailto:[e-mail]> <mailto:[e-mail]>>
                   wrote:
                       I think resync points are an interesting idea
                       although we haven't evaluated them. Twitch did
                       push for S-frames in AV1 which will be another
                       option in the future instead of encoding a full
                       IDR frame at these resync boundaries.
                       An issue is you have to make the hard decision
                       to abort the current download and frantically
                       try to pick up the pieces before the buffer
                       depletes. It's a one-way door (maybe your
                       algorithm overreacted) and you're going to
                       be throwing out some media just to redownload it
                       at a lower bitrate.
                       Ideally, you could download segments in parallel
                       without causing contention. The idea is to spend
                       any available bandwidth on the new segment to
                       fix the problem, and any excess bandwidth on the
                       old segment in the event it arrives before the
                       player buffer actually depletes. That's more or
                       less the core concept for what we've built using
                       QUIC, and it's compatible with resync points if
                       we later go down that route.
                       And you're exactly right Piers. The fundamental
                       issue is that a web player lacks the low level
                       timing information required to infer
                       the delivery rate. You would want something like
                       BBR's rate estimation
                       <https://datatracker.ietf.org/doc/html/draft-cheng-iccrg-delivery-rate-estimation> which
                       inspects the time delta between packets to
                       determine the send rate. That gets really
                       difficult when the OS and browser delay flushing
                       data to the application, be it for performance
                       reasons or due to packet loss (to maintain
                       head-of-line blocking).
                       I did run into CUBIC/Reno not being able to grow
                       the congestion window when frames are sent one
                       at a time (application limited). I don't believe
                       BBR suffers from the same problem though due to
                       the aforementioned rate estimator.
                       On Wed, Nov 3, 2021 at 10:05 AM Ali C. Begen
                       <[e-mail]<mailto:[e-mail]>
                       <mailto:[e-mail]>> wrote:
                            > On Nov 3, 2021, at 6:50 PM, Piers
                           O'Hanlon <[e-mail]<mailto:[e-mail]>
                           <mailto:[e-mail]>> wrote:
                            >
                            >
                            >
                            >> On 2 Nov 2021, at 20:39, Ali C. Begen
                           <ali.begen=[e-mail]<mailto:ali.begen=[e-mail]>
                           <mailto:[e-mail]>>
                           wrote:
                            >>
                            >>
                            >>
                            >>> On Nov 2, 2021, at 3:39 AM, Luke Curley
                           <[e-mail]<mailto:[e-mail]>
                           <mailto:[e-mail]>> wrote:
                            >>>
                            >>> Hey folks, I wanted to quickly
                           summarize the problems we've run into at
                           Twitch that have led us to QUIC.
                            >>>
                            >>>
                            >>> Twitch is a live one-to-many product.
                           We primarily focus on video quality due to
                           the graphical fidelity of video games.
                           Viewers can participate in a chat room,
                           which the broadcaster reads and can respond
                           to via video. This means that latency is
                           also somewhat important to facilitate this
                           social interaction.
                            >>>
                            >>> A looong time ago we were using RTMP
                           for both ingest and distribution (Flash
                           player). We switched to HLS for distribution
                           to gain the benefit of 3rd party CDNs, at
                           the cost of dramatically increasing latency.
                           A later project lowered the latency of HLS
                           using chunked-transfer delivery, very
                           similar to LL-DASH (and not LL-HLS). We're
                           still using RTMP for contribution.
                            >>>
                            > I guess Apple do also have their
                           BYTERANGE/CTE mode for LL-HLS which is
                           pretty similar to LL-DASH.
                           Yes, Apple can list the parts (chunks in
                           LL-DASH) as byteranges in the playlist but
                           the frequent playlist refresh and part
                           retrieval process is inevitable in LL-HLS,
                           which is one of the main differences from
                           LL-DASH (no need for manifest refresh and
                           request per segment not chunk).
                            >
                            >>>
                            >>> To summarize the issues with our
                           current distribution system:
                            >>>
                            >>> 1. HLS suffers from head-of-line blocking.
                            >>> During congestion, the current segment
                           stalls and is delivered slower than the
                           encoded bitrate. The player has no recourse
                           than to wait for the segment to finish
                           downloading, risking depleting the buffer.
                           It can switch down to a lower rendition at
                           segment boundaries, but these boundaries
                           occur too infrequently (every 2s) to handle
                           sudden congestion. Trying to switch earlier,
                           either by canceling the current segment or
                           downloading the lower rendition in parallel,
                           only exacerbates the issue.
                            >>
                            > Isn't the HoL limitation more down to the
                           use of HTTP/1.1?
                            >
                            >> DASH has the concept of Resync points
                           that were designed exactly for this purpose
                           (allowing you to emergency downshift in the
                           middle of a segment).
                            >>
                            > I was curious if there are any studies or
                           experience of how resync points perform in
                           practice?
                           Resync points are pretty fresh out of the
                           oven. dash.js has it in the roadmap but not
                           yet implemented (and we also need to
                           generate test streams). So, there is no data
                           available yet with the real clients. But, I
                           suppose you can imagine how in-segment
                           switching can help in sudden bw drops
                           especially for long segments.
                            >
                            >>> 2. HLS has poor "auto" quality (ABR).
                            >>> The player is responsible for choosing
                           the rendition to download. This is a problem
                           when media is delivered frame-by-frame (ie.
                           HTTP chunked-transfer), as we're effectively
                           application-limited by the encoder bitrate.
                           The player can only measure the arrival
                           timestamp of data and does not know when the
                           network can sustain a higher bitrate without
                           just trying it. We hosted an ACM challenge
                           for this issue in particular.
                            >>
                            > The limitation here may also be down to
                           the lack of access to sufficiently accurate
                           timing information about data arrivals in
                           the browser - unfortunately the Streams API,
                           which provides data from the fetch API,
                           doesn’t directly timestamp the data arrivals
                           so the JS app has to timestamp it which can
                           suffer from noise such as scheduling etc -
                           especially a problem for small/fast data
                           arrivals.
                           Yes, you need to get rid of that noise (see
                           LoL+).
                            > I guess another issue could be that if
                           the system is only sending single frames
                           then the network transport may be operating
                           in application limited mode so the cwnd
                           doesn’t grow sufficiently to take advantage
                           of the available capacity.
                           Unless the video bitrate is too low, this
                           should not be an issue most of the time.
                            >
                            >> That exact challenge had three competing
                           solutions, two of which are now part of the
                           official dash.js code. And yes, the player
                           can figure what the network can sustain
                           *without* trying higher bitrate renditions.
                            >>
                           https://github.com/Dash-Industry-Forum/dash.js/wiki/Low-Latency-streaming
                           <https://github.com/Dash-Industry-Forum/dash.js/wiki/Low-Latency-streaming>
                            >> Or read the paper that even had “twitch”
                           in its title here:
                           https://ieeexplore.ieee.org/document/9429986
                           <https://ieeexplore.ieee.org/document/9429986>
                            >>
                            > There was a recent study that seems to
                           show that none of the current algorithms are
                           that great for low latency, and the two new
                           dash.js ones appear to lead to much higher
                           levels of rebuffering:
                            >
                           https://dl.acm.org/doi/pdf/10.1145/3458305.3478442
                           <https://dl.acm.org/doi/pdf/10.1145/3458305.3478442>
                           Brightcove’s paper uses the LoL and L2A
                           algorithms from the challenge where low
                           latency was the primary goal. For Twitch’s
                           own evaluation, I suggest you watch:
                           https://www.youtube.com/watch?v=rcXFVDotpy4
                           <https://www.youtube.com/watch?v=rcXFVDotpy4>
                           We later addressed the rebuffering issue,
                           developed LoL+, which is the version
                           included in dash.js now and explained at the
                           ieeexplore link I gave above.
                           Copying the authors in case they want to add
                           anything for the paper you cited.
                           -acbegen
                            >
                            > Piers
                            >
                            >>> I believe this is why LL-HLS opts to
                           burst small chunks of data (sub-segments) at
                           the cost of higher latency.
                            >>>
                            >>>
                            >>> Both of these necessitate a larger
                           player buffer, which increases latency. The
                           contribution system it's own problems, but
                           let me sync up with that team first before I
                           try to enumerate them.
                            >>> --
                            >>> Moq mailing list
                            >>> [e-mail]<mailto:[e-mail]> <mailto:[e-mail]>
                            >>>
                           https://www.ietf.org/mailman/listinfo/moq
                           <https://www.ietf.org/mailman/listinfo/moq>
                            >>
                            >> --
                            >> Moq mailing list
                            >> [e-mail]<mailto:[e-mail]> <mailto:[e-mail]>
                            >>
                           https://www.ietf.org/mailman/listinfo/moq
                           <https://www.ietf.org/mailman/listinfo/moq>
                       --                         Moq mailing list
                       [e-mail]<mailto:[e-mail]> <mailto:[e-mail]>
                       https://www.ietf.org/mailman/listinfo/moq
                       <https://www.ietf.org/mailman/listinfo/moq>
           --             Moq mailing list
           [e-mail]<mailto:[e-mail]> <mailto:[e-mail]>
           https://www.ietf.org/mailman/listinfo/moq
           <https://www.ietf.org/mailman/listinfo/moq>
       --         Moq mailing list
       [e-mail]<mailto:[e-mail]> <mailto:[e-mail]>
       https://www.ietf.org/mailman/listinfo/moq
       <https://www.ietf.org/mailman/listinfo/moq>
   --     Moq mailing list
   [e-mail]<mailto:[e-mail]> <mailto:[e-mail]>
   https://www.ietf.org/mailman/listinfo/moq
   <https://www.ietf.org/mailman/listinfo/moq>

--
Moq mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/moq