Re: [Mops] Low latency protocol comparision

[Alexandre GOUAILLARD <[e-mail]>]

Dear all,

I went around to the long time Hackathon attending IETF media guys and it
looks like it's going to be too short notice to prepare something for
Madrid.
Between AV1 RTP payload and implementation, the SVC extensions, the
QuicTransport / WebTransport , and the End-To-End encryption for real-time
media, everybody that would be otherwise interested and capable to
contribute is real busy.

However, I gathered sufficient interest for BKK (whether it will be virtual
or not), given the usual rules those people abide by are followed. Since
IETF does not have a set of proper rules nor a formal test suite, those
follow the rules that were used at IMTC superop events, previous IETF
Hackathon, W3C 's web-platform tests and/or AOMedia 's testing working
group. The usual IETF rules about the usage of MUST or SHOULD apply.
- Test cases must be discussed and defined early on, to make sure the
"everything else being equal" statement applies. The definition of starting
point and end point of latency measurement for example varies a lot
depending on the study.
- Test content (if any) must be defined and made publicly available to be
able to independently reproduce results.
- Test suite should be written in advance. The same test suite should be
used by all. The test code must be made publicly available to be able to
independently reproduce results.
- If the need for specific test bed appears, e.g. controlled network,
programmatic packet loss / jitter / bandwidth variation, we should speak
with the IETF network guys, and/or Charles E. to have that set up (in the
case of a real meeting), or for a common open source test infrastructure
like e.g. KITE to be set up and made available.

We usually consider it fair to provide 3 months to everyone after the first
comparative results are out, to try to optimize their implementations or
address bugs found in the process, so that the results are representative
of the capacity of a protocol, and not the deficiency of a given
implementation. I don't know if it makes sense to do this here, e.g. to
have temporary results in BKK, but give untill IETF 110 in Pragues to
freeze the data, since SRT is not an IETF protocol with fixed versions and
revisions at this stage.

If this sounds reasonable to at least one of the chair, and from maxim or
another SRT representative, I'm happy to facilitate for IETF 109 the set
up, bring enough IETF people and open source media server people to the
table to have critical mass of data and review, and to facilitate the
writing of a subsequent ID, in the same spirit we published a comparative
study of open-source media webrtc media servers:
https://www.semanticscholar.org/paper/Comparative-Study-of-WebRTC-Open-Source-SFUs-for-Andre-Breton/10b877cc13ffc9a1cabbf415cba73b22a1dd5442

Best regards,

Dr. Alex.

On Sun, Jun 14, 2020 at 10:05 PM Maxim Sharabayko <
[e-mail]> wrote:

> Hi,
>
>
>
> The idea of performing the comparison is very exciting! I would be glad to
> participate as a delegate from SRT.
>
>
>
> @Glen
>
> *It would be great if any SRT folks on this list could do something
> similar,  to to mention those working with CMAF and LL HLS.*
>
>
>
> There are of course some inaccuracies in the description of SRT.
>
> The receiver buffer latency of 120 ms is the default value, not the
> minimum. The minimum is 1 ms.
>
> There is no limit of the number of packet retransmissions, although the
> configurable limit might be added soon.
>
> HLS and DASH can be transmitted over SRT. And so on.
>
>
>
> But in general even with these inaccuracies, the overview of the existing
> protocols on the market is quite good.
>
>
>
> --
>
> Regards,
>
> Maxim Sharabayko
>
> Senior Software Developer | Haivision
>
>
>
> *From: *Mops <[e-mail]> on behalf of Alexandre GOUAILLARD <
> [e-mail]>
> *Date: *Sunday, 14. June 2020 at 01:28
> *To: *"Deen, Glenn (NBCUniversal)" <[e-mail]>
> *Cc: *Leslie Daigle <[e-mail]>, "[e-mail]" <
> [e-mail]>
> *Subject: *Re: [Mops] Low latency protocol comparision
>
>
>
> @Deen, agreed. I gave it a try though a couple of years back. Let me know
> if that would go in the right direction and I can update it with everything
> I learned in the mean time, and taking into account the updates to the
> different protocols (e.g. new HLS informational draft by pantos):
>
>
> http://webrtcbydralex.com/index.php/2018/05/15/streaming-protocols-and-ultra-low-latency-including-webrtc/
>
>
>
> @Leslie, I volunteer every year, every hackathon :-)
>
> However, I am more knowledgeable about webrtc than other protocols, so in
> that regard, I am biased.
>
> I think to be fair each protocol should have a
> reasonable representative/champion that at least validates that the testing
> protocol chosen is adequate, and garantees the accuracy and the fairness of
> the result.
>
>
>
> For webrtc, it was problematic as well, as each browser vendor did not
> want someone else to benchmark their implementation, so we ended up with
> one representant of each browser vendor at 104 I believe (pragues), to
> validate its result, and did it again in montreal:
>
> https://trac.ietf.org/trac/ietf/meeting/wiki/105hackathon/webrtc
>
> if we manage to get the same setting, I think it could be very successful
> in getting fair and accurate results.
>
>
>
> I think we could handle the webrtc testing (as in writing open source
> tests and providing results that would be deterministic and
> independently verifiable by any and all).
>
> - Maybe Veriskope, which is trying to push an RTMP 2.0, could be
> interested in joining for RTMP.
>
> - Magnus W for RTSP 2.0 ?
>
> - Then we would need one volunteer for SRT,
>
> - and another one for QUIC (maybe martin thompson can point us in the
> right direction). QUIC is always the biggest table so it should not be too
> difficult.
>
> - Jonathan Lennox, chairman of AVTCORE is always around and could help,
> especially on the congestion control, Error concealment, and bandwidth
> adaptation part
>
> - We could motivate EKR to comment on the security aspects.
>
> - I'm not sure if we want to asses IPv6 and NAT traversal readiness (ICE)
>
>
>
> What do you think?
>
>
>
> Regards,
>
>
>
> Dr. Alex
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Sun, Jun 14, 2020 at 1:05 AM Deen, Glenn (NBCUniversal) <
> [e-mail]> wrote:
>
> Yes, it’s a bit marketing, but it’s also one of the few write ups that
> attempts to put the popular media transports side by side for people to
> understand their strengths and abilities.   The marketing blogs also give
> us, the IETF, insight into how the outside world of adopters and sellers of
> our work see things.
>
>
>
> I really wish that all web documents and pages contained a posting date.
> This one doesn’t gave it a date.  However, the page must be from at least
> mid 2019 as it mentions things that were released then.
>
>
>
> I like your write up of the WebRTC inaccuracies.  It would be great if any
> SRT folks on this list could do something similar,  to to mention those
> working with CMAF and LL HLS.  There a lot of adopters out there who are
> working through what protocol best fits their particular needs - webRTC?
> SRT?  Something that bright summer intern wrote while she was here last
> summer?
>
>
>
>  Capturing something like this, and even better some sort of bake off as
> you suggest would be valuable to a lot of people.  Though it may have to
> wait to get critical mass of participants until next year.
>
>
>
> Glenn
>
>
>
> On Jun 13, 2020, at 3:34 PM, Alexandre GOUAILLARD <[e-mail]>
> wrote:
>
> There are multiple inaccurate statements in the linked documents. Most of
> them that have been carried around for many years by other SRT vendors like
> Wowza and already called out (here
> <http://webrtcbydralex.com/index.php/2019/05/18/wowzas-marketing-at-work-again-webrtc/>
> for example).
>
>
>
> I'm just going to point to three of the most egregious (read "verifiably
> wrong") statements, that by themselves undermine the credibility of the
> entire document.
>
>
>
> - "Maximum supported resolution: 720p, 30 frames per second with a bit
> rate of up to 2 Mbps."
> - There is no limit to the spatial and temporal resolution in
> webrtc/rtcweb protocol.
>
> Stadia for example send up to 4K@30fps.
>
>
>
> "WebRTC is inferior to its colleagues in terms of the coding quality and
> maximum amount of transmitted data.
>
> - There is no limit to the bandwidth used in the webrtc/rtcweb protocol.
>
> - There is a limit of 2Mbps per peer connection on the sending-side *only
> in the google chrome implementation*, only by default (web application
> can lift this limit), and only for webcams devices (screensharing has no
> bandwidth limit). This is a chrome implementation detail, and not a
> protocol limit.
>
> -While VP8 and H264 are both mandatory to implement (original IETF
> discussion
> <https://mailarchive.ietf.org/arch/msg/rtcweb/juVsZKp-AmRpYLCgy-uwartuNrY/>)
> anybody can add any codec to webrtc as long as an RTP payload specification
> exists.
>
> -- Google chrome, and firefox, both supports real-time VP9 mode 2 (4:4:4,
> 10 bits, HDR)
>
>
> https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/codecs/vp9/vp9_impl.cc;l=985
>
> -- Chrome has a real-time AV1 implementation in webrtc, which goes up to
> high profile (4:4:4, 8~10bits, HDR).
>
>
> https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/codecs/av1/
>
> In terms of quality, only codecs at high profile (i.e. 12bits) can do
> better, and none of those have a "real-time" version today.
>
> In terms of coding efficiency, AV1 is still the best codec available in
> production.
>
> "WebRTC is not available in Safari and partially unavailable in Bowser and
> Edge."
>
> - webrtc is fully available in Edge, and Safari.
>
> https://caniuse.com/#search=webrtc
>
> official daily W3C results:
> https://wpt.fyi/results/?label=master&label=experimental&aligned&q=webrtc
>
> - Safari has added H.265 support in addition to the mandatory to implement
> VP8 and H..264 in the latest safari tech preview.
>
>
>
> I'm not going into the Discussion about webrtc's underlying RTC/RTCP
> protocol and corresponding extension: NACK, RTX, FEC, RED, or Congestion
> Control (REMB, Google-cc, BBR, NADA), as IETF members I'm sure you re all
> aware of RFC3550 and all its derivatives. Many authors of those RFCs are
> part of this group.
>
>
>
> Maybe more overlooked is the most recent Draft for end-to-end encryption
> on top of webrtc co-authored by google:
>
> https://tools.ietf.org/html/draft-omara-sframe-00
>
>
>
> I'm not saying that SRT is worse, or better than WebRTC.. I'm just saying,
> this comparison is not egregiously incorrect, and I'm a little bit
> surprised that we are discussing marketing blogs that go in the face of so
> many IETF specs.
>
>
>
> If anybody is interested in going deep into a real  thorough comparison
> between all those protocols, why not doing it during the IETF hackathon in
> July, or in october (hopefully in BKK) just like we did group co-testing
> with IETF's AVTCORE/AVEXT/QUIC/RTCWEB/RMCAT during the previous IETF
> hackathons?
>
>
>
> Regards,
>
>
>
>
>
>
>
> On Sat, Jun 13, 2020 at 6:58 PM Leslie Daigle <[e-mail]>
> wrote:
>
> I thought this was a pretty interesting analysis:
>
> Low broadcast latency and protocols for implementation thereof:
>
> https://www.elecard.com/page/low_broadcast_latency_and_protocols_for_implementation_thereof
>
> Particularly telling is the chart at the end: WebRTC and SRT don’t look
> very similar, in this particular regard.
>
> Leslie.
>
> --
> ------------------------------
>
> Leslie Daigle
> Principal, ThinkingCat Enterprises
> [e-mail]
>
> --
> Mops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/mops
>
>
>
>
> --
>
> Alex. Gouaillard, PhD, PhD, MBA
>
>
> ------------------------------------------------------------------------------------
>
> President - CoSMo Software Consulting, Singapore
>
>
> ------------------------------------------------------------------------------------
>
> sg.linkedin.com/agouaillard
>
> ·
>
> --
> Mops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/mops
>
>
>
>
> --
>
> Alex. Gouaillard, PhD, PhD, MBA
>
>
> ------------------------------------------------------------------------------------
>
> President - CoSMo Software Consulting, Singapore
>
>
> ------------------------------------------------------------------------------------
>
> sg.linkedin.com/agouaillard
>
> ·
>


-- 
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------
sg.linkedin.com/agouaillard

   -