Re: [Bier] WGLC: draft-ietf-bier-multicast-http-response

[Toerless Eckert <[e-mail]>]

Happy new year everybody!

Agree with Dirks points.

Let me address three of my own:

a) Personally i do like describing a solution and point out the BIER(-TE)
specific design components. IMHO, if well done, an actual solution description
is a lot more explanatory to operators than a purely architectural or protocol spec document.
Whether or not we have achieved the "well done" is of course a different question.

b) Process wise, BIER-WG has adopted the document with the scope it had when adopted.
If we want to change the scope, this is potentially a different process issue
than just improving on it in its current scope. I have seen WG chairs easily
approve without further adoption call split-up of documents into multiple parts,
but i am not sure about the IETF BCP when changing scope fundamentally.

c) Technically, i think there is a very strong tie between the solution/architecture
description and BIER as the delivery mechanism: 

If you wanted to do the same with IP Multicast, you have three options:

   I) For a stream offered in M different bitrates to N receivers,
   each of which could potentially want to receive any single one of
   those M bitrates at any point in time based on the path congestion
   towards that receiver, you statically create M*2^N ASM Multicast groups
   or SSM channels. Given how there may be K different media
   streaming simultaneous (K maybe short-tail of TV stations),
   it is easy to see how K*M*2^N easily exceeds any reasonable amount of
   hardware supported ASM/SSM multicast state in routers/switches for
   anything but an uninteresting small number of receivers (3 ? ;-).

   II) Instead of setting up those multicast trees statically, you let
   receivers dynamically join/leave an appropriate multicast group/channel
   when needed. This reduces the amount of group/channels to K*M, so
   its easy to see how this can scale to arbitrary amount of receivers.
   Alas, the amount of join/prune processing in the control plane (and
   pushing this into the forwarding plane) is extremely unlikely to be
   supportable in any router/switch not especially optimized in performance
   for this type of operation. Such as at least processing IGMP/MLS
   join/leave in distributed linecard CPU if not directly in FPE code.

   I have seen such prototypes for this in the past, but i am not aware of production
   code for this. I would be surprised to even find guaranteed join/leave per second
   performance numbers on any router/switch product. And todays IP Multicast
   IPTV deployments typically have sender-caches that also provide channel
   change acceleration whereby IP Multicast is joined to as an optimization
   while still receiving a unicast stream after channel change. Which means
   that in todays deployment actual join latency is very uncritical, making
   lazy/bad router/switch implementations a non-issue. And hence moving to
   a solution where such channel change would have to be fast and precise even
   more of a pain to expect from vendors.

   III) You can try to wiggle yourself through with a hybrid solution
   trying to mix&match I) and II) and assuming a lot more knowledge
   about about K, M and N, network topology and platform performances.
   Good luck! Probably could be a nice Master or PhD thesis, but not more.

   Long story short: It won't work in practice, and given the BIER alternative,
   it would IMHO be a colossal waste of development resources (other than
   a happy student in need of a paper idea ;-).

This is in the end what i tried to see contributed to the document.
I am happy to improve on that part  because it does IMHO give the core
promotion for BIER which is what i think we want. And of course if we feel
this type of core BIER promotion might be better perceived if we fork this
part out of the current document, and the WG chairs would support this as 
a split-off WG document, i could try to find the time to propose such text.
Wouldn't be just a few pages anyhow.  And could serve (one of) the "architectural" aspects
Dirk i think was referring to.

Cheers
    Toerless

On Mon, Jan 03, 2022 at 08:19:28AM +0000, Dirk Trossen wrote:
> Hi Gyan,
> 
> Many thanks for the comments (and a Happy New Year to you and the rest of the list).
> 
> The summary of the key point you make is clear: the draft does not outline an architecture but a specific solution. The focus on the latter makes it hard to recognize the underlying architectural concept that differs from IP multicast. In recent work, I called this ‘forward path requests, return path multicast”, making the implicit join operation for an instantaneous multicast return operation clearer (I think). The draft, unfortunately (largely due to historical reasons), does not refer to this.
> 
> One direction of moving this work forward could be on that architectural concept (with an HTTP-based solution being an example only). But it also begs the question whether BIER is the only realizing transport technology for this concept?
> 
> Best,
> 
> Dirk
> 
> From: BIER [mailto:[e-mail]] On Behalf Of Gyan Mishra
> Sent: 24 December 2021 08:24
> To: Dirk Trossen <[e-mail]>
> Cc: [e-mail]; BIER WG <[e-mail]>; Toerless Eckert <[e-mail]>; Alvaro Retana <[e-mail]>
> Subject: Re: [Bier] WGLC: draft-ietf-bier-multicast-http-response
> 
> Hi Dirk
> 
> Responses in-line
> 
> Kind Regards
> 
> Gyan
> 
> On Thu, Dec 23, 2021 at 4:24 AM Dirk Trossen <[e-mail]<mailto:[e-mail]>> wrote:
> Hi Gyan,
> 
> Many thanks for supporting this draft and its continuation as well as good to see your interest in collaborate on the continued draft.
>  Gyan> Welcome!
> See more inline.
> 
> Best,
> 
> Dirk
> 
> From: BIER [mailto:[e-mail]<mailto:[e-mail]>] On Behalf Of Gyan Mishra
> Sent: 23 December 2021 00:15
> To: Toerless Eckert <[e-mail]<mailto:[e-mail]>>
> Cc: [e-mail]<mailto:[e-mail]>; BIER WG <[e-mail]<mailto:[e-mail]>>; Alvaro Retana <[e-mail]<mailto:[e-mail]>>
> Subject: Re: [Bier] WGLC: draft-ietf-bier-multicast-http-response
> 
> 
> Dear BIER WG,
> 
> I support publication  of this informational draft which provides the framework and basis of how stateless BIER infrastructure can eliminate state and bandwidth usage with an aggregate http response when many client join a stream.  I would be happy to collaborate on the draft.
> [DOT] Key here is that, unlike in IP multicast, there is no direct notion of a stream but a joint interest (expressed in the request for a specific chunk, for instance), expressed as individual unicast requests but replied to as a single multicast response (there may be the same interest albeit express some time later, resulting in a the different response albeit with likely same content). This is key since it makes the multicast nature more instantaneous, not following a longer-lived group-like semantic. Take an example of clients pulling video chunks. The lack of synchronization in the unicast requests for chunks leads inevitably (due to processing and packet latencies) to changing batches of requests that may be responded to with relevant content, although the content itself may be the same across several responses. The batches likely change in membership since clients may encounter different relevant latencies etc.
>  Gyan>  You say “unlike IP Multicast”so what you are describing and optimizations is not a multicast solution but a completely new solution alternative for subscribers and operators.  Is there a document I can read that explains how the host signaling works as it does not use IGMP or PIM it seems and different process, procedures, FSM of how to express interest for a specific chunk and how that all works.  That would really help the reader in understanding the full scope of the use case.  Also what part of IP Multicast is used as far as host signaling if any?
> [DOT] This aspect is important in several technical aspects, such as timing used to create the multicast response (therefore combining a certain number of incoming requests), how to handle the congestion on multicast relations that possible differ in nature from one response to another (since possibly different clients are ‘combined’ into the individual responses). To me, those questions need dealing with although not in this draft IMO. What the draft provides is a base capability, as you also outline it, to allow for this aggregation and the resulting multicast delivery of aggregated responses.
> 
>      Gyan> Understood.  I am still a bit lost from your above paragraph that you state “unlike IP multicast”
> This sounds like a completely new architecture that does not use the multicast group concept for subscribers, but new framework to cutdown on IGMP and PIM chatter,
> I have some comments on the draft based on my experience working with multicast video delivery:
> 
> Is the waste in transmission bandwidth due to PIM Assert and double stream when multiple first hop routers are connected to the source?
> 
> Can you comment on the increase in signaling when there are a few subscribers?  I assume you are referring to IGMP join last hop router chatter.
> [DOT] When comparing with IP multicast, its explicit group membership constitutes waste compared to the proposed mechanism since in the latter, the ‘group’ is defined through batching requests (for the same semantic content) at the server. The request for the content is here the ‘signaling’ to compare against. Of course, for static groups, you can argue that the joining happens once and the sender can simply stream, while the proposed mechanism signals for every chunk, so to speak. But the objectives are, of course, different since the underlying chunk retrieval aims at a different semantic, not limited to a live stream. If you want to move to such ‘individual viewing experiences’, the proposed solution still yields in multicast delivery due to the mere statistical synchronicity of chunk requests (particular for the short tail of a catalogue). Also, if you do live event but provision them over an HTTP-based platform, you are simply pushed into the unicast request/reply pattern – this draft proposes a mechanism that is based on the pattern in the forward direction but not the return one.
>  Gyan> As I stated above and I think should be included in the draft is a detailed explanation of the proposed multicast replacement architecture called  “http level multicast” and how it works detailed procedures and state machine.  With HLS and DASH the video file is broken up into chunks segments so if you have to signal interest in the steam for evey chunk that would be a lot of control plane signaling overhead where with IGMP you join the group which is for the video feed not the individual chunks of the video file being streamed.  Am I missing something?
> The goal of BIER is to eliminate state in the core for MVPN P-tree and C-Tree tree building technologies and not the last hop router as that is on the customer side signaling.
> 
> I read through this RFC related to IRTF ICN deployments scenario discussed related to 80 nodes test, however not much detail is provided related to the adaptive multicast streaming testing.
> 
> https://datatracker.ietf.org/doc/html/rfc8763
> [DOT] Right, RFC8763 mentions this scenario but indeed lacks details. The test here was done as part of past efforts I led. The content was not live but stored content.
>  Gyan> Understood.  I think overall details of http level multicast is lacking in the dradt
> I read through this draft section 3.10 and it also does not provide details into how multicast adaptive video delivery works without having a gateway or server that does an ingest of the adaptive video to source the multicast distribution stream over UDP transport.
> 
> https://datatracker.ietf.org/doc/html/draft-ietf-bier-use-cases-12#section-3.10
> [DOT] You seem to assume a live streaming type of scenario, sourced from an ingest point (over UDP) and relayed via this mechanism? But the draft outlines something more akin to an HLS/DASH model, i.e., clients individually request chunks and the server replies albeit aggregating several same content responses into multicast ones – see also below.
>  Gyan> Understood.  Ok so I think I get it this is a completely new architecture multicast paradigm shift
> [DOT] the aspects below are indeed all valid but it’s this part where I would expect input and positions from the application (protocol) community since the draft is merely meant to provide a base delivery mechanism, not an application level solution. In my past work, we had some insights into the aspects you outline below but it’s not captured in this draft for the reasons already mentioned.
>  Gyan> Understood
> The key point I am missing is how adaptive video is natively possible over UDP transport.
> 
> Comments below are related to multicast and adaptive streaming.  The problem with multicast is it uses UDP transport and is a one way stream and not bidirectional so no feedback mechanism exists for adaptive rate limiting.  I believe QUIC with sideband TCP channel  is a possibility which I have investigated as an alternate transport to UDP.  PGM pragmatic multicast is another possibility.
> 
> https://datatracker.ietf.org/doc/html/rfc3208
> 
> 
> My understanding and experience with HLS and DASH adaptive streaming which uses chunks and segments natively both video delivery methods only support Unicast video delivery.  Does not support native delivery via multicast.
> [DOT] It is the HLS/DASH type of delivery mechanism. The lack of support for multicast here motivated the draft initially but still leaves questions open on issues like timing of aggregation, interaction with upper layer mechanism, how to handle congestion control on the return path (if at all)…
>   Gyan> Understood
> [DOT] as an example: DASH adaptive streaming mimics TCP in the sense of using latency information as an indication for network conditions, thereby lowering the stream quality when such latency occurs. If you look into the idea of aggregated chunk responses (as in the draft here), you may notice quickly that ‘delaying’ chunk responses is beneficial here since doing so increases your window to ‘catch’ more incoming client requests for the same content, thereby increasing the size of the response group. How long to delay? Well, could be something like 75% of the chunk length (just to give a number) since it gives you a good delay budget for returning the response and playing it out (with chunk length of 4 or more seconds). Problem is that if you do that, DASH adaptive stream will kick in and reduce your streaming quality…so there are interactions that need to be understood.
>  Gyan> Understood
> Most all modern web browsers Chrome, Firefox and Safari etc do not support NPAPI plug in and thus cannot play multicast natively.  That is a major industry wide problem.
> 
> There are standard solutions that exist that involve a UDP wrapper at the source multicast gateway that ingests unicast and distributes  multicast.
> 
> Ramp is one vendor that provides such a solution.
> Ramp uses a source sender that ingests HLS or DASH unicast stream and packages in UDP envelope and at the host receiver endpoint Window or Mac it strips UDP encapsulation and plays out HLS or DASH video natively on loopback 127.0.0.1.
> 
> https://www.rampecdn.com/
> 
> Wowza is another vendor that does support multicast streaming but requires ingest as well of adaptive steam and then sources multicast distribution stream.
> 
> https://www.wowza.com/community/t/multicast-options-with-wowza-streaming-engine/44969
> 
> 
> There has been development work done with WebRTC messaging that has the ability to provide a similar solution to natively play out multicast in a web browser.
> 
> https://www.w3.org/2020/09/webrtc-charter.html
> 
> https://datatracker.ietf.org/wg/rtcweb/documents/
> 
> This is the only link below I could find on DASH multicast.
> 
> https://res-www.zte.com.cn/mediares/magazine/publication/com_en/article/201803/PARK.pdf
> 
> Some history on DVB:
> 
> DVB is basically MPEG video delivery over an IP network.  There was a DVB IETF WG which concluded decades ago and since had been a standards forum for digital video delivery.
> 
> https://datatracker.ietf.org/wg/ipdvb/documents/
> 
> 
> https://dvb.org/
> 
> This document was published in 2018 and talks about adaptive streaming over IP multicast.
> 
> This document talks about how ingest occurs if adaptive video source stream done by a multicast server which encapsulated and delivers the multicast over the network.
> 
> https://dvb.org/wp-content/uploads/2019/12/a176_adaptive_media_streaming_over_ip_multicast_2018-02-16_draft_bluebook.pdf
> 
> 3GPP 5G to support MBS Multicast and Broadcast Services with release 17 adaptive video delivery MPEG DASH
> 
> https://www.smpte.org/webcast/an-introduction-to-3gpp-5g-multicast-and-broadcast-services?hsLang=en
> 
> https://dashif.org/docs/workshop-2019/12-Stockhammer%20Multicast%20December%202019.pdf
> 
> https://www.etsi.org/deliver/etsi_ts/103700_103799/103720/01.01.01_60/ts_103720v010101p.pdf
> 
> This solution to use BIER could apply to any network that utilizes adaptive video on FBB, MBB as well as an other network so  could have ubiquitous use cases.
> [DOT] Note again, that the point here is that the solution is based on a request/response semantic with the former in unicast and the latter possibly in multicast (if more than one request to the  same content). Most of the solutions you outline above are joining an explicit group for a stream. Now you could, of course, model this via an HTTP/2 request and a stream of HTTP DATA frames. So this would make the forward request your explicit group membership signaling.
>  Gyan> Understood.  I am getting it now a completely new multicast architecture
> I can provide a further review of the draft and input to help advance the document.
> [DOT] Great, thanks! Your email was already extremely helpful.
>  Gyan> Most welcome!!
> Many Thanks!
> 
> Gyan
> 
> On Mon, Dec 20, 2021 at 3:13 PM Toerless Eckert <[e-mail]<mailto:[e-mail]>> wrote:
> Dear BIER WG-chairs
> 
> a) As authors, we are pinging other WG members to ask for more reviews within BIER-WG.
> 
> b) Picking up on Alvaros suggestion (thanks a lot), i would like to request
>   on behalf of the authors a TSVART (Transport Area Review Team).
> 
> (we will ask for Application Area review later.. working ourselves up the stack).
> 
> Here is the suggested text for the datatracker form through which
> you WG chairs could reques
> 
> Deadline:
> 
>   02-15-2022
> 
> Document revision:
> 
>   draft-ietf-bier-multicast-http-response-06
> 
> Requester's comments and instructions
> 
>   The authors are requesting early review of the document from transport area.
> 
>   The following is a motivational summary of the purpose of this document:
> 
>   Adaptive bitrate streaming (such as today DASH) for live-streaming or short-tail can
>   use Multicast to improve network efficiency by eliminating duplicate transmission
>   of the same data segments to multiple users requiring the same bitrate. This
>   was shown and used in products as early as 200x (e.g.: Digital Fountain).
>   These solutions never proliferated because of the intrinsic signaling problems
>   of IP Multicast causing severe performance/scale/operational issues.
> 
>   These IP Multicast limitations can today be overcome with the novel BIER
>   multicast mechanism. This document describes an application solution layer
>   architecture in which BIER is used as the multicast mechanism for HTTP
>   adaptive bitrate streaming including the use of BIER-TE for including
>   path steering optimizations into the solution. This document is solely
>   informational and does not intend to establish any standards for the IETF.
> 
> Thank you so much
>     Toerless (for the authors)
> 
> On Fri, Dec 17, 2021 at 05:09:36AM -0800, Alvaro Retana wrote:
> > Greg:
> >
> > Hi!
> >
> > It is important in light of rfc8789, which requires that all IETF documents
> > reach consensus, to demonstrate that the WG cares and has reviewed,
> > commented, etc.  Silence is not a good indicator of consensus. Thanks for
> > following up on this!
> >
> > If there is interest in the WG, and given that this document deals with
> > HTTP level multicast, please request an early review from both the ART and
> > Transport areas review teams before sending it off for publication.
> >
> >
> > A couple of quick comments for the authors.  Please use the rfc8174
> > template (not rfc2119).  I am surprised that there are no Normative
> > references (
> > https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
> > ).
> >
> > Thanks!
> >
> > Alvaro.
> >
> > On December 9, 2021 at 12:19:05 PM, Greg Shepherd ([e-mail]<mailto:[e-mail]>) wrote:
> >
> > Zero response for this WGLC, not even from the authors. Is this work still
> > of interest to anyone in the WG?
> >
> > - Shep
> >
> > On Mon, Nov 22, 2021 at 8:45 AM Greg Shepherd <[e-mail]<mailto:[e-mail]>> wrote:
> >
> > > This email starts a 2 week WGLC timer for
> > > draft-ietf-bier-multicast-http-response. Please read and review:
> > >
> > > https://datatracker.ietf.org/doc/draft-ietf-bier-multicast-http-response/
> > >
> > > We also need a doc Shepherd before we can progress this work. Can someone
> > > please step up to help.
> > >
> > > Thanks,
> > > - Shep
> > > (Chairs)
> > >
> > _______________________________________________
> > BIER mailing list
> > [e-mail]<mailto:[e-mail]>
> > https://www.ietf.org/mailman/listinfo/bier
> 
> > _______________________________________________
> > BIER mailing list
> > [e-mail]<mailto:[e-mail]>
> > https://www.ietf.org/mailman/listinfo/bier
> 
> 
> --
> ---
> [e-mail]<mailto:[e-mail]>
> 
> _______________________________________________
> BIER mailing list
> [e-mail]<mailto:[e-mail]>
> https://www.ietf.org/mailman/listinfo/bier
> --
> 
> [http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<http://www.verizon.com/>
> 
> Gyan Mishra
> 
> Network Solutions Architect
> 
> Email [e-mail]<mailto:[e-mail]>
> 
> M 301 502-1347
> 
> --
> 
> [http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<http://www.verizon.com/>
> 
> Gyan Mishra
> 
> Network Solutions Architect
> 
> Email [e-mail]<mailto:[e-mail]>
> 
> M 301 502-1347
> 

-- 
---
[e-mail]