Re: [Mops] Low latency protocol comparision

Hi,

Speaking as only one of the chairs of this group, I have to say I like 
how this is shaping up.

My only question is whether there is work to be done even at the IETF 
108 hackathon, in order to establish the test cases and test suite?

Happy to hear from the other co-chair of this group :^) and any other 
group participants.

Leslie.

On 24 Jun 2020, at 9:41, Alexandre GOUAILLARD wrote:

> Dear all,
>
> I went around to the long time Hackathon attending IETF media guys and 
> it
> looks like it's going to be too short notice to prepare something for
> Madrid.
> Between AV1 RTP payload and implementation, the SVC extensions, the
> QuicTransport / WebTransport , and the End-To-End encryption for 
> real-time
> media, everybody that would be otherwise interested and capable to
> contribute is real busy.
>
> However, I gathered sufficient interest for BKK (whether it will be 
> virtual
> or not), given the usual rules those people abide by are followed. 
> Since
> IETF does not have a set of proper rules nor a formal test suite, 
> those
> follow the rules that were used at IMTC superop events, previous IETF
> Hackathon, W3C 's web-platform tests and/or AOMedia 's testing working
> group. The usual IETF rules about the usage of MUST or SHOULD apply.
> - Test cases must be discussed and defined early on, to make sure the
> "everything else being equal" statement applies. The definition of 
> starting
> point and end point of latency measurement for example varies a lot
> depending on the study.
> - Test content (if any) must be defined and made publicly available to 
> be
> able to independently reproduce results.
> - Test suite should be written in advance. The same test suite should 
> be
> used by all. The test code must be made publicly available to be able 
> to
> independently reproduce results.
> - If the need for specific test bed appears, e.g. controlled network,
> programmatic packet loss / jitter / bandwidth variation, we should 
> speak
> with the IETF network guys, and/or Charles E. to have that set up (in 
> the
> case of a real meeting), or for a common open source test 
> infrastructure
> like e.g. KITE to be set up and made available.
>
> We usually consider it fair to provide 3 months to everyone after the 
> first
> comparative results are out, to try to optimize their implementations 
> or
> address bugs found in the process, so that the results are 
> representative
> of the capacity of a protocol, and not the deficiency of a given
> implementation. I don't know if it makes sense to do this here, e.g. 
> to
> have temporary results in BKK, but give untill IETF 110 in Pragues to
> freeze the data, since SRT is not an IETF protocol with fixed versions 
> and
> revisions at this stage.
>
> If this sounds reasonable to at least one of the chair, and from maxim 
> or
> another SRT representative, I'm happy to facilitate for IETF 109 the 
> set
> up, bring enough IETF people and open source media server people to 
> the
> table to have critical mass of data and review, and to facilitate the
> writing of a subsequent ID, in the same spirit we published a 
> comparative
> study of open-source media webrtc media servers:
> https://www.semanticscholar.org/paper/Comparative-Study-of-WebRTC-Open-Source-SFUs-for-Andre-Breton/10b877cc13ffc9a1cabbf415cba73b22a1dd5442
>
> Best regards,
>
> Dr. Alex.
>
> On Sun, Jun 14, 2020 at 10:05 PM Maxim Sharabayko <
> maxsharabayko@haivision.com> wrote:
>
>> Hi,
>>
>>
>>
>> The idea of performing the comparison is very exciting! I would be 
>> glad to
>> participate as a delegate from SRT.
>>
>>
>>
>> @Glen
>>
>> *It would be great if any SRT folks on this list could do something
>> similar,  to to mention those working with CMAF and LL HLS.*
>>
>>
>>
>> There are of course some inaccuracies in the description of SRT.
>>
>> The receiver buffer latency of 120 ms is the default value, not the
>> minimum. The minimum is 1 ms.
>>
>> There is no limit of the number of packet retransmissions, although 
>> the
>> configurable limit might be added soon.
>>
>> HLS and DASH can be transmitted over SRT. And so on.
>>
>>
>>
>> But in general even with these inaccuracies, the overview of the 
>> existing
>> protocols on the market is quite good.
>>
>>
>>
>> --
>>
>> Regards,
>>
>> Maxim Sharabayko
>>
>> Senior Software Developer | Haivision
>>
>>
>>
>> *From: *Mops <mops-bounces@ietf.org> on behalf of Alexandre 
>> GOUAILLARD <
>> agouaillard@gmail.com>
>> *Date: *Sunday, 14. June 2020 at 01:28
>> *To: *"Deen, Glenn (NBCUniversal)" <Glenn.Deen@nbcuni.com>
>> *Cc: *Leslie Daigle <ldaigle@thinkingcat.com>, "mops@ietf.org" <
>> mops@ietf.org>
>> *Subject: *Re: [Mops] Low latency protocol comparision
>>
>>
>>
>> @Deen, agreed. I gave it a try though a couple of years back. Let me 
>> know
>> if that would go in the right direction and I can update it with 
>> everything
>> I learned in the mean time, and taking into account the updates to 
>> the
>> different protocols (e.g. new HLS informational draft by pantos):
>>
>>
>> http://webrtcbydralex.com/index.php/2018/05/15/streaming-protocols-and-ultra-low-latency-including-webrtc/
>>
>>
>>
>> @Leslie, I volunteer every year, every hackathon :-)
>>
>> However, I am more knowledgeable about webrtc than other protocols, 
>> so in
>> that regard, I am biased.
>>
>> I think to be fair each protocol should have a
>> reasonable representative/champion that at least validates that the 
>> testing
>> protocol chosen is adequate, and garantees the accuracy and the 
>> fairness of
>> the result.
>>
>>
>>
>> For webrtc, it was problematic as well, as each browser vendor did 
>> not
>> want someone else to benchmark their implementation, so we ended up 
>> with
>> one representant of each browser vendor at 104 I believe (pragues), 
>> to
>> validate its result, and did it again in montreal:
>>
>> https://trac.ietf.org/trac/ietf/meeting/wiki/105hackathon/webrtc
>>
>> if we manage to get the same setting, I think it could be very 
>> successful
>> in getting fair and accurate results.
>>
>>
>>
>> I think we could handle the webrtc testing (as in writing open source
>> tests and providing results that would be deterministic and
>> independently verifiable by any and all).
>>
>> - Maybe Veriskope, which is trying to push an RTMP 2.0, could be
>> interested in joining for RTMP.
>>
>> - Magnus W for RTSP 2.0 ?
>>
>> - Then we would need one volunteer for SRT,
>>
>> - and another one for QUIC (maybe martin thompson can point us in the
>> right direction). QUIC is always the biggest table so it should not 
>> be too
>> difficult.
>>
>> - Jonathan Lennox, chairman of AVTCORE is always around and could 
>> help,
>> especially on the congestion control, Error concealment, and 
>> bandwidth
>> adaptation part
>>
>> - We could motivate EKR to comment on the security aspects.
>>
>> - I'm not sure if we want to asses IPv6 and NAT traversal readiness 
>> (ICE)
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Regards,
>>
>>
>>
>> Dr. Alex
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Jun 14, 2020 at 1:05 AM Deen, Glenn (NBCUniversal) <
>> Glenn.Deen@nbcuni.com> wrote:
>>
>> Yes, it’s a bit marketing, but it’s also one of the few write ups 
>> that
>> attempts to put the popular media transports side by side for people 
>> to
>> understand their strengths and abilities.   The marketing blogs also 
>> give
>> us, the IETF, insight into how the outside world of adopters and 
>> sellers of
>> our work see things.
>>
>>
>>
>> I really wish that all web documents and pages contained a posting 
>> date.
>> This one doesn’t gave it a date.  However, the page must be from at 
>> least
>> mid 2019 as it mentions things that were released then.
>>
>>
>>
>> I like your write up of the WebRTC inaccuracies.  It would be great 
>> if any
>> SRT folks on this list could do something similar,  to to mention 
>> those
>> working with CMAF and LL HLS.  There a lot of adopters out there who 
>> are
>> working through what protocol best fits their particular needs - 
>> webRTC?
>> SRT?  Something that bright summer intern wrote while she was here 
>> last
>> summer?
>>
>>
>>
>>  Capturing something like this, and even better some sort of bake off 
>> as
>> you suggest would be valuable to a lot of people.  Though it may have 
>> to
>> wait to get critical mass of participants until next year.
>>
>>
>>
>> Glenn
>>
>>
>>
>> On Jun 13, 2020, at 3:34 PM, Alexandre GOUAILLARD 
>> <agouaillard@gmail.com>
>> wrote:
>>
>> There are multiple inaccurate statements in the linked documents. 
>> Most of
>> them that have been carried around for many years by other SRT 
>> vendors like
>> Wowza and already called out (here
>> <http://webrtcbydralex.com/index.php/2019/05/18/wowzas-marketing-at-work-again-webrtc/>
>> for example).
>>
>>
>>
>> I'm just going to point to three of the most egregious (read 
>> "verifiably
>> wrong") statements, that by themselves undermine the credibility of 
>> the
>> entire document.
>>
>>
>>
>> - "Maximum supported resolution: 720p, 30 frames per second with a 
>> bit
>> rate of up to 2 Mbps."
>> - There is no limit to the spatial and temporal resolution in
>> webrtc/rtcweb protocol.
>>
>> Stadia for example send up to 4K@30fps.
>>
>>
>>
>> "WebRTC is inferior to its colleagues in terms of the coding quality 
>> and
>> maximum amount of transmitted data.
>>
>> - There is no limit to the bandwidth used in the webrtc/rtcweb 
>> protocol.
>>
>> - There is a limit of 2Mbps per peer connection on the sending-side 
>> *only
>> in the google chrome implementation*, only by default (web 
>> application
>> can lift this limit), and only for webcams devices (screensharing has 
>> no
>> bandwidth limit). This is a chrome implementation detail, and not a
>> protocol limit.
>>
>> -While VP8 and H264 are both mandatory to implement (original IETF
>> discussion
>> <https://mailarchive.ietf.org/arch/msg/rtcweb/juVsZKp-AmRpYLCgy-uwartuNrY/>)
>> anybody can add any codec to webrtc as long as an RTP payload 
>> specification
>> exists.
>>
>> -- Google chrome, and firefox, both supports real-time VP9 mode 2 
>> (4:4:4,
>> 10 bits, HDR)
>>
>>
>> https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/codecs/vp9/vp9_impl.cc;l=985
>>
>> -- Chrome has a real-time AV1 implementation in webrtc, which goes up 
>> to
>> high profile (4:4:4, 8~10bits, HDR).
>>
>>
>> https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/codecs/av1/
>>
>> In terms of quality, only codecs at high profile (i.e. 12bits) can do
>> better, and none of those have a "real-time" version today.
>>
>> In terms of coding efficiency, AV1 is still the best codec available 
>> in
>> production.
>>
>> "WebRTC is not available in Safari and partially unavailable in 
>> Bowser and
>> Edge."
>>
>> - webrtc is fully available in Edge, and Safari.
>>
>> https://caniuse.com/#search=webrtc
>>
>> official daily W3C results:
>> https://wpt.fyi/results/?label=master&label=experimental&aligned&q=webrtc
>>
>> - Safari has added H.265 support in addition to the mandatory to 
>> implement
>> VP8 and H..264 in the latest safari tech preview.
>>
>>
>>
>> I'm not going into the Discussion about webrtc's underlying RTC/RTCP
>> protocol and corresponding extension: NACK, RTX, FEC, RED, or 
>> Congestion
>> Control (REMB, Google-cc, BBR, NADA), as IETF members I'm sure you re 
>> all
>> aware of RFC3550 and all its derivatives. Many authors of those RFCs 
>> are
>> part of this group.
>>
>>
>>
>> Maybe more overlooked is the most recent Draft for end-to-end 
>> encryption
>> on top of webrtc co-authored by google:
>>
>> https://tools.ietf.org/html/draft-omara-sframe-00
>>
>>
>>
>> I'm not saying that SRT is worse, or better than WebRTC.. I'm just 
>> saying,
>> this comparison is not egregiously incorrect, and I'm a little bit
>> surprised that we are discussing marketing blogs that go in the face 
>> of so
>> many IETF specs.
>>
>>
>>
>> If anybody is interested in going deep into a real  thorough 
>> comparison
>> between all those protocols, why not doing it during the IETF 
>> hackathon in
>> July, or in october (hopefully in BKK) just like we did group 
>> co-testing
>> with IETF's AVTCORE/AVEXT/QUIC/RTCWEB/RMCAT during the previous IETF
>> hackathons?
>>
>>
>>
>> Regards,
>>
>>
>>
>>
>>
>>
>>
>> On Sat, Jun 13, 2020 at 6:58 PM Leslie Daigle 
>> <ldaigle@thinkingcat.com>
>> wrote:
>>
>> I thought this was a pretty interesting analysis:
>>
>> Low broadcast latency and protocols for implementation thereof:
>>
>> https://www.elecard.com/page/low_broadcast_latency_and_protocols_for_implementation_thereof
>>
>> Particularly telling is the chart at the end: WebRTC and SRT don’t 
>> look
>> very similar, in this particular regard.
>>
>> Leslie.
>>
>> --
>> ------------------------------
>>
>> Leslie Daigle
>> Principal, ThinkingCat Enterprises
>> ldaigle@thinkingcat.com
>>
>> --
>> Mops mailing list
>> Mops@ietf.org
>> https://www.ietf.org/mailman/listinfo/mops
>>
>>
>>
>>
>> --
>>
>> Alex. Gouaillard, PhD, PhD, MBA
>>
>>
>> ------------------------------------------------------------------------------------
>>
>> President - CoSMo Software Consulting, Singapore
>>
>>
>> ------------------------------------------------------------------------------------
>>
>> sg.linkedin.com/agouaillard
>>
>> ·
>>
>> --
>> Mops mailing list
>> Mops@ietf.org
>> https://www.ietf.org/mailman/listinfo/mops
>>
>>
>>
>>
>> --
>>
>> Alex. Gouaillard, PhD, PhD, MBA
>>
>>
>> ------------------------------------------------------------------------------------
>>
>> President - CoSMo Software Consulting, Singapore
>>
>>
>> ------------------------------------------------------------------------------------
>>
>> sg.linkedin.com/agouaillard
>>
>> ·
>>
>
>
> -- 
> Alex. Gouaillard, PhD, PhD, MBA
> ------------------------------------------------------------------------------------
> President - CoSMo Software Consulting, Singapore
> ------------------------------------------------------------------------------------
> sg.linkedin.com/agouaillard
>
>    -

> -- 
> Mops mailing list
> Mops@ietf.org
> https://www.ietf.org/mailman/listinfo/mops

-- 

-------------------------------------------------------------------
Leslie Daigle
Principal, ThinkingCat Enterprises
ldaigle@thinkingcat.com
-------------------------------------------------------------------