Re: [ippm] [MBONED] WGLC for draft-ietf-mboned-multicast-telemetry

[Toerless Eckert <[e-mail]>]

On Thu, Sep 14, 2023 at 08:01:31AM -0700, Leonard Giuliano wrote:
> MBONED (and IPPM),
> 
> We would like to officially begin working group last call for Multicast
> On-path Telemetry using IOAM.  We had a previous WGLC for this doc earlier
> this year, and at the time there was not enough response from the WG to
> support advancement, but the draft has since been updated and in the meeting
> in IETF117 there was support in the room for another WGLC.  It was also
> suggested that IPPM be included for their input and review.
> 
> Please post whether you support/oppose the advancement of this draft as well
> as any comments you may have to the list by Sept 29.
> 
> Most recent version of the draft can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-mboned-multicast-telemetry/
> 
> -MBONED Chairs

Thanks a lot folks for the work. Some comments, mostly textual inline below in
idnits output. I did not check the idnits header created. One terminology / functional
suggestion about the use of pre-existing rfc9197 terminology (ingress/egress interfaces)
as well as tracing the ingress interface as well.


idnits 2.17.1 

draft-ietf-mboned-multicast-telemetry-07.txt:

  Checking boilerplate required by RFC 5378 and the IETF Trust (see
  https://trustee.ietf.org/license-info):
  ----------------------------------------------------------------------------

     No issues found here.

  Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt:
  ----------------------------------------------------------------------------

     No issues found here.

  Checking nits according to https://www.ietf.org/id-info/checklist :
  ----------------------------------------------------------------------------

  == There are 1 instance of lines with multicast IPv4 addresses in the
     document.  If these are generic example addresses, they should be changed
     to use the 233.252.0.x range defined in RFC 5771


  Miscellaneous warnings:
  ----------------------------------------------------------------------------

  -- The document date (6 September 2023) is 15 days in the past.  Is this
     intentional?


  Checking references for intended status: Proposed Standard
  ----------------------------------------------------------------------------

     (See RFCs 3967 and 4897 for information about using normative references
     to lower-maturity documents in RFCs)

  == Missing Reference: 'A' is mentioned on line 220, but not defined

  -- Looks like a reference, but probably isn't: '0' on line 220

  -- Looks like a reference, but probably isn't: '1' on line 220

  == Missing Reference: 'RFC XXXX' is mentioned on line 452, but not defined

  == Unused Reference: 'RFC4687' is defined on line 469, but no explicit
     reference was found in the text

  == Unused Reference: 'RFC6037' is defined on line 482, but no explicit
     reference was found in the text

  == Unused Reference: 'I-D.xie-bier-ipv6-encapsulation' is defined on line
     590, but no explicit reference was found in the text

  ** Downref: Normative reference to an Informational RFC: RFC 4687

  ** Downref: Normative reference to an Historic RFC: RFC 6037


     Summary: 2 errors (**), 0 flaws (~~), 6 warnings (==), 3 comments (--).

     Run idnits with the --verbose option for more detailed information about
     the items above.

--------------------------------------------------------------------------------


2	MBONED                                                           H. Song
3	Internet-Draft                                                M. McBride
4	Intended status: Standards Track                  Futurewei Technologies
5	Expires: 9 March 2024                                          G. Mirsky
6	                                                               ZTE Corp.
7	                                                               G. Mishra
8	                                                            Verizon Inc.
9	                                                               H. Asaeda
10	                                                                    NICT
11	                                                                 T. Zhou
12	                                                     Huawei Technologies
13	                                                        6 September 2023

15	                 Multicast On-path Telemetry using IOAM
16	                draft-ietf-mboned-multicast-telemetry-07

18	Abstract

20	   This document specifies the requirements of on-path telemetry for
21	   multicast traffic using In-situ OAM.  While In-situ OAM is
22	   advantageous for multicast traffic telemetry, some unique challenges
23	   present.  This document provides the solutions based on the In-situ

"some unique challenges present" - does not parse.

24	   OAM trace option and direct export option to support the telemetry
25	   data correlation and the multicast tree reconstruction without
26	   incurring data redundancy.

28	Requirements Language

30	   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
31	   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
32	   "OPTIONAL" in this document are to be interpreted as described in BCP
33	   14 [RFC2119][RFC8174] when, and only when, they appear in all
34	   capitals, as shown here.

36	Status of This Memo

38	   This Internet-Draft is submitted in full conformance with the
39	   provisions of BCP 78 and BCP 79.

41	   Internet-Drafts are working documents of the Internet Engineering
42	   Task Force (IETF).  Note that other groups may also distribute
43	   working documents as Internet-Drafts.  The list of current Internet-
44	   Drafts is at https://datatracker.ietf.org/drafts/current/.

46	   Internet-Drafts are draft documents valid for a maximum of six months
47	   and may be updated, replaced, or obsoleted by other documents at any
48	   time.  It is inappropriate to use Internet-Drafts as reference
49	   material or to cite them other than as "work in progress."

51	   This Internet-Draft will expire on 9 March 2024.

53	Copyright Notice

55	   Copyright (c) 2023 IETF Trust and the persons identified as the
56	   document authors.  All rights reserved.

58	   This document is subject to BCP 78 and the IETF Trust's Legal
59	   Provisions Relating to IETF Documents (https://trustee.ietf.org/
60	   license-info) in effect on the date of publication of this document.
61	   Please review these documents carefully, as they describe your rights
62	   and restrictions with respect to this document.  Code Components
63	   extracted from this document must include Revised BSD License text as
64	   described in Section 4.e of the Trust Legal Provisions and are
65	   provided without warranty as described in the Revised BSD License.

67	Table of Contents

69	   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
70	   2.  Requirements for Multicast Traffic Telemetry  . . . . . . . .   4
71	   3.  Issues of Existing Techniques . . . . . . . . . . . . . . . .   4
72	   4.  Modifications to Existing Solutions . . . . . . . . . . . . .   5
73	     4.1.  Per-hop postcard using IOAM DEX . . . . . . . . . . . . .   5
74	     4.2.  Per-section postcard for IOAM Trace . . . . . . . . . . .   7
75	   5.  Application Considerations for Multicast Protocols  . . . . .   8
76	     5.1.  Mtrace verson 2 . . . . . . . . . . . . . . . . . . . . .   8
77	     5.2.  Application in PIM  . . . . . . . . . . . . . . . . . . .   9
78	     5.3.  Application of MVPN X-PMSI Tunnel Encapsulation
79	           Attribute . . . . . . . . . . . . . . . . . . . . . . . .   9
80	     5.4.  Application in BIER . . . . . . . . . . . . . . . . . . .  10
81	   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  10
82	   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  10
83	   8.  Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  10
84	   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  10
85	     9.1.  Normative References  . . . . . . . . . . . . . . . . . .  10
86	     9.2.  Informative References  . . . . . . . . . . . . . . . . .  12
87	   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  13

89	1.  Introduction

91	   Multicast is used by residential broadband customers across operator

suggest /used by/used for example by, but not limited to/
aka: there's a lot more.

92	   networks, private MPLS customers, and internal customers within
93	   corporate intranet.  Multicast provides real time interactive online
94	   meetings or podcasts, IPTV, and financial markets real-time data,
95	   which all have a reliance on UDP's unreliable transport.  End-to-end
96	   QOS, therefore, should be a critical component of multicast
97	   deployment in order to provide a good end user experience.  In
98	   multicast video streaming, if a packet is dropped, and that packet
99	   happens to be a reference frame (i.e., I-Frame) in the video feed,
100	   the client receiver of the multicast feed goes into buffering mode
101	   resulting in a frozen window. 

Suggest rewording of last sentence:

In multicast real-time media streaming, loss of a single packet containing a
reference frame can result in the inability of all thousands of receivers to
decode a whole so-called Group-of-Picture sequence of packets, introducing
black picture for periods often as much as 4 seconds. Unexpected long delay
in propagation of a packet in such real-time media streaming may equally result
oin the packet not being received and cxreating the same results.

101	   Multicast packet drops and delay can

...therefore

102	   severely affect the application performance and user experience.

104	   It is important to monitor the performance of the multicast traffic.
105	   New on-path telemetry techniques such as In-situ OAM (IOAM)
106	   [RFC9197], IOAM Direct Export (DEX) [RFC9326] IOAM Marking-based
107	   Postcard (PBT-M) [I-D.song-ippm-postcard-based-telemetry], and Hybrid
108	   Two-Step (HTS) [I-D.mirsky-ippm-hybrid-two-step] are useful and
109	   complementary to the existing active OAM performance monitoring
110	   methods, provide promising means to directly monitor the network

"existing active OAM performance monitoring" - such as ? One or two references
would help.

111	   experience of multicast traffic.  However, multicast traffic has some
112	   unique characteristics which pose some challenges on applying such
113	   techniques in an efficient way.

115	   The IP multicast packet data for a particular (S, G) state is
116	   identical from one branch to another on its way to multiple
117	   receivers.  When adding IOAM trace data to multicast packets, each
118	   replicated packet would keep the telemetry data for its entire
119	   forwarding path.  Since the replicated packets all share common path
120	   segments, redundant data will be collected for the same original
121	   multicast packet.  Such redundancy consumes extra network bandwidth
122	   unnecessarily.  For a large multicast tree, such redundancy is
123	   considerable.  Alternatively, it could be more efficient to collect
124	   the telemetry data using solutions such as IOAM DEX to eliminate the
125	   data redundancy.  However, IOAM DEX is lack of a branch identifier,

s/is lack of/lacks/

126	   making telemetry data correlation and multicast-tree reconstruction
127	   difficult.

129	   This draft provides two solutions to the IOAM data redundancy problem
130	   based on the IOAM standards.  The requirements for multicast traffic
131	   telemetry are discussed along with the issues of the existing on-path
132	   telemetry techniques.  We propose modifications to make these
133	   techniques adapt to multicast in order for the original multicast
134	   tree to be correctly reconstructed while eliminating redundant data.

136	2.  Requirements for Multicast Traffic Telemetry

138	   Multicast traffic is forwarded through a multicast tree.  With PIM
139	   and P2MP (e.g., MLDP, RSVP-TE) the forwarding tree is established and

Please provide expansion of terms and references for them on first use, aka: for
MLDP, RSVP-TE here (not later). Please check for all abbreviations in the document,
i may overlook them as i am not doing a thorough search.

140	   maintained by the multicast routing protocol.  With BIER, no state is
141	   created in the network to establish a forwarding tree; instead, a
142	   bier header provides the necessary information for each packet to
143	   know the egress points.  Multicast packets are only replicated at

And with BIER-TE the replication points too ;-))

144	   each tree branch fork node for efficiency.

146	   There are several requirements for multicast traffic telemetry, a few
147	   of which are:

149	   *  Reconstruct and visualize the multicast tree through data plane
150	      monitoring.

152	   *  Gather the multicast packet delay and jitter performance on each
153	      path.

155	   *  Find the multicast packet drop location and reason.

157	   *  Gather the VPN state and tunnel information in case of P2MP
158	      multicast.

160	   In order to meet these requirements, we need the ability to directly
161	   monitor the multicast traffic and derive data from the multicast
162	   packets.  The conventional OAM mechanisms, such as multicast ping and
163	   trace, are not sufficient to meet these requirements.

references for ping and trace here please.

165	3.  Issues of Existing Techniques

167	   On-path Telemetry techniques that directly retrieve data from
168	   multicast traffic's live network experience are ideal for addressing
169	   the aforementioned requirements.  The representative techniques
170	   include In-situ OAM (IOAM) Trace option [RFC9197], IOAM Direct Export
171	   (DEX) option [RFC9326], and PBT-M
172	   [I-D.song-ippm-postcard-based-telemetry].  However, unlike unicast,
173	   multicast poses some unique challenges to applying these techniques.

175	   Multicast packets are replicated at each branch fork node in the
176	   corresponding multicast tree.  Therefore, there are multiple copies
177	   of the original multicast packet in the network.

179	   If the IOAM trace option is used for on-path data collection, the
180	   partial trace data will also be replicated into the copy for each
181	   branch.  The end result is that, at the multicast tree leaves, each
182	   copy of the multicast packet has a complete trace.  Most of the data
183	   (except data from the last leaf branch), however, has redundant
184	   copies.  Data redundancy introduces unnecessary header overhead,
185	   wastes network bandwidth, and complicates the data processing.  The
186	   larger the multicast tree is or the longer the multicast path is, the
187	   more severe the redundancy problem becomes.

It is not clear to me if i am incorrectly understanding how IAM trace is supposed to work.
I thought that it would allow to collect trace information from more than one hop along the
path. So if i assume there is replication on every hop with multicast, and collection of
trace data on every hop, then the packet on every receiver would still have different
trace data, and only initial parts of the trace would be the same. But if lets
say i am multicast to 100 receivers, i am still saving a huge amount of data on
the wire by multicasting as compared to the source sending 100 unicast packets.

Whether this is true or false, it seems such a simple example to summarize how
IOAM trace works would help the text.

189	   The postcard-based solutions (e.g., IOAM DEX), can be used to
190	   eliminate such data redundancy, because each node on the tree only
191	   sends a postcard covering local data.  However, they cannot track and
192	   correlate the tree branches properly due to the lack of branching
193	   information, so they can bring confusion about the multicast tree
194	   topology.  For example, in a multicast tree, Node A has two branches,
195	   one to Node B and the other to node C; further, Node B leads to Node
196	   D and Node C leads to Node E.  When applying postcard-based methods,
197	   one cannot tell whether or not Node D(E) is the next hop of Node B(C)
198	   from the received postcards.

Unless one correlates the exporting nodes with knowledge about the tree collected
by other means, such as mtrace or SNMP/YANG. Aka: May be useful to include an
argument why such correlation is undesirable or inferior to what's proposed here.

May i also note that BIER-TE, RFC9262 does contain the tree information in the
packet header, so it would be possible to directly deduce the tree a packet was
passed over when correlating received IOAM DEX information.

200	   The fundamental reason for this problem is that there is not an
201	   identifier (either implicit or explicit) to correlate the data on
202	   each branch.

204	4.  Modifications to Existing Solutions

206	   We provide two solutions to address the above issues.  One is based
207	   on IOAM DEX and requires an extension to the instruction header of
208	   the IOAM DEX Option.  The second solution combines the IOAM trace
209	   option and postcards for redundancy removal.

211	4.1.  Per-hop postcard using IOAM DEX

213	   One way to mitigate the postcard-based telemetry's tree tracking
214	   weakness is to augment it with a branch identifier field.  Note that
215	   this works for the IOAM DEX option but not for PBT-M because the IOAM
216	   DEX option has an instruction header which can be used to hold the
217	   branch identifier.  To make the branch identifier globally unique,
218	   the branch fork node ID plus an index is used.  For example, Node A
219	   has two branches: one to Node B and the other to Node C.  Node A will
220	   use [A, 0] as the branch identifier for the branch to B, and [A, 1]
221	   for the branch to C.  The identifier is carried with the multicast
222	   packet until the next branch fork node.  Each node MUST export the
223	   branch identifier in the received IOAM DEX header in the postcards it
224	   sends.  The branch identifier, along with the other fields such as
225	   flow ID and sequence number, is sufficient for the data collector to
226	   reconstruct the topology of the multicast tree.

228	   Figure 1 shows an example of this solution.  "P" stands for the
229	   postcard packet.  The square brackets contains the branch identifier.
230	   The curly brace contains the telemetry data about a specific node.

232	     P:[A,0]{A}  P:[A,0]{B} P:[B,1]{D}  P:[B,0]{C}   P:[B,0]{E}
233	          ^            ^          ^        ^           ^
234	          :            :          :        :           :
235	          :            :          :        :           :
236	          :            :          :      +-:-+       +-:-+
237	          :            :          :      |   |       |   |
238	          :            :      +---:----->| C |------>| E |-...
239	        +-:-+        +-:-+    |   :      |   |       |   |
240	        |   |        |   |----+   :      +---+       +---+
241	        | A |------->| B |        :
242	        |   |        |   |--+   +-:-+
243	        +---+        +---+  |   |   |
244	                            +-->| D |--...
245	                                |   |
246	                                +---+

248	                         Figure 1: Per-hop Postcard

250	   Each branch fork node needs to generate a unique branch identifier
251	   (i.e., branch ID) for each branch in its multicast tree instance and
252	   include it in the IOAM DEX option header.  The branch ID remains
253	   unchanged until the next branch fork node.  The branch ID contains
254	   two parts: the branch fork node ID and an interface index.

256	   Conforming to the node ID specification in IOAM [RFC9197], the node
257	   ID is a 3-octet unsigned integer.  The interface index is a two-octet
258	   unsigned integer.  As shown in Figure 2, the branch ID consumes 8
259	   octets in total.  The three unused octets MUST be set to 0.

261	       0                   1                   2                   3
262	       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
263	      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
264	      |                 node_id                       |     unused    |
265	      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
266	      |       Interface Index         |           unused              |
267	      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

269	                    Figure 2: Multicast Branch ID format

271	   Figure 3 shows that the branch ID is carried as an optional field
272	   after the flow ID and sequence number optional fields in the IOAM DEX
273	   option header.  Two bits "N" and "I" (i.e., the third and fourth bits
274	   in the Extension-Flags field) are reserved to indicate the presence
275	   of the optional branch ID field.  "N" stands for the Node ID and "I"
276	   stands for the interface index.  If "N" and "I" are both set to 1,
277	   the optional multicast branch ID field is present; otherwise it is
278	   absent.

280	        0                   1                   2                   3
281	        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
282	       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
283	       |        Namespace-ID           |     Flags     |F|S|N|I|E-Flags|
284	       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
285	       |               IOAM-Trace-Type                 |   Reserved    |
286	       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
287	       |                         Flow ID (optional)                    |
288	       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
289	       |                     Sequence Number  (Optional)               |
290	       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
291	       |                       Multicast Branch ID                     |
292	       |                            (optional)                         |
293	       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

295	          Figure 3: Carry Branch ID in IOAM DEX option header

297	   Once a node gets the branch ID information from the upstream, it MUST
298	   carry this information in its telemetry data export postcards, so the
299	   original multicast tree can be correctly reconstructed based on the
300	   postcards.

Couple of considerations coming to mind for this mechanism:

1. It looks as if the trace information is done on egress, but this is not said.
Is it obvious to anybody who has read throughall IOAM specs that ONLY egress trace
data is generated ? Aka: in IPFIX we did define both ingress and egress data capture.
If in doubt, then maybe add some sentence in this section explaining that this tracing
is done on egress, aka: after replication - and hence the branch is assumed to be known.

2. I do think it would be extremely useful to also
know the incoming interface, not only the brach ID. And you have the 16 bit
"Unused" where the incoming interface would nicely fit into without increasing the
exported payload. For example when using MP2MP in MLDP or Bidir-PIM, the incoming
interface is not a given, aka: can be different.

3. Of couse, if/when you have the incoming interface, then you could also state
that you do want to optionally do trace on ingress, namely when the packet is received
by discarded by RPF-check or Bidir-PIM Acceptance check. Aka: Packets that are being
discarded will have no output branch, but are a great feature to trace. And of course
there could be the simple logic of setting the branch-id to 0 to indicate the packet
was discarded, and to imply this was ingress trace output.

4. RFC9197 already uses terms ingress_if_id and egress_if_id. If its not too much
trouble i would suggest to avoid inventing new terms such as "branch_id" but instead
reuse those pre-existing terms.

5. There is another reason, why i do not like branch-id as a term: Together with
the examples that use branch-id 0, 1 do make me worry that implementers could mis-interpret
this to mean that the branch-id is allocated starting from 0 independently for
every multicast tree and renumbered. Aka: In one tree, the OIF to router C is the
first branch, so it gets branch_id 0, and on another tree, it is the second branch,
so it gets branch_id 1. Aka: There should be text reconfirming that the same outoing
interface needs to use the same identifier across all trees, and the examples
ideally should use non-consecutive identifiers, not starting at 0.

302	4.2.  Per-section postcard for IOAM Trace

304	   The second solution is a combination of the IOAM trace option and the
305	   postcard-based telemetry.  To avoid data redundancy, at each branch
306	   fork node, the trace data accumulated up to this node is exported by
307	   a postcard before the packet is replicated.  In this solution, each
308	   branch also needs to maintain some identifier to help correlate the
309	   postcards for each tree section.  The natural way to accomplish this
310	   is to simply carry the branch fork node's data (including its ID) in
311	   the trace of each branch.  This is also necessary because each
312	   replicated multicast packet can have different telemetry data
313	   pertaining to this particular copy (e.g., node delay, egress
314	   timestamp, and egress interface).  As a consequence, the local data
315	   exported by each branch fork node can only contain partial data
316	   (e.g., ingress interface and ingress timestamp).

318	   Figure 4 shows an example in a segment of a multicast tree.  Node B
319	   and D are two branch fork nodes and they will export a postcard
320	   covering the trace data for the previous section.  The end node of
321	   each path will also need to export the data of the last section as a
322	   postcard.

324	                P:{A,B'}            P:{B1,C,D'}
325	                   ^                     ^
326	                   :                     :
327	                   :                     :
328	                   :                     :    {D1}
329	                   :                     :    +--...
330	                   :        +---+      +---+  |
331	                   :   {B1} |   |{B1,C}|   |--+ {D2}
332	                   :    +-->| C |----->| D |-----...
333	       +---+     +---+  |   |   |      |   |--+
334	       |   | {A} |   |--+   +---+      +---+  |
335	       | A |---->| B |                        +--...
336	       |   |     |   |--+   +---+             {D3}
337	       +---+     +---+  |   |   |{B2,E}
338	                        +-->| E |--...
339	                       {B2} |   |
340	                            +---+

342	                       Figure 4: Per-section Postcard

344	   There is no need to modify the IOAM trace option header format as
345	   specified in [RFC9197].  We just need to configure the branch fork
346	   nodes to export the postcards and refresh the IOAM header and data
347	   (e.g., clear the node data list and reset the Remaining Length
348	   field).

Not quite clear: If you do not apply thre section 4.1 proposal here, how
would you know from the P:{B1,C,D'} postcard that it should be stitched below
the P:{A,B'} postcard ? Wouldn't you need the same type of ingress/egress
interface information to do the stitching ?

350	5.  Application Considerations for Multicast Protocols

352	5.1.  Mtrace verson 2

354	   Mtrace version 2 (Mtrace2) [RFC8487] is a protocol that allows the
355	   tracing of an IP multicast routing path.  Mtrace2 provides additional
356	   information such as the packet rates and losses, as well as other
357	   diagnostic information.  Unlike unicast traceroute, Mtrace2 traces
358	   the path that a packet would take from a source to a receiver.  It is

Not really, right ? Mtrace2 traces the path that the tree building messages
follow from receiver to source. The reverse of this traced path can then be
assumed to be the path taken by multicast packets on the branch of the tree
to the receiver fdrom which the Mtrace2 was started.

359	   usually initiated from an Mtrace2 client by sending an Mtrace2 Query
360	   to a Last-Hop Router (LHR) or to a Rendezvous Point (RP).  The LHR/RP
361	   turns the Query packet into an Mtrace2 Request, appends a Standard
362	   Response Block containing its interface addresses and packet
363	   statistics to the Request packet, and forwards the packet towards the
364	   source/RP.  In a similar fashion, each router along the path to the
365	   source/RP appends a Standard Response Block to the end of the Request
366	   packet and forwards it to its upstream router.  When the First-Hop
367	   Router (FHR) receives the Request packet, it appends its own Standard
368	   Response Block, turns the Request packet into a Reply, and unicasts
369	   the Reply back to the Mtrace2 client.

371	   New on-path telemetry techniques will enhance Mtrace2, and other
372	   existing OAM solutions, with more granular and realtime network
373	   status data through direct measurements.  There are various multicast
374	   protocols that are used to forward the multicast data.  Each will
375	   require their own unique on-path telemetry solution.

At this point in time, readers like me will wonder:

Is there actually any integration between Mtrace2 and IOAM, and how does this
document itself help.

Maybe replace last paragraph by something like: At this point, Mtrace2 does not
integrate with IOAM2 directly, but network management systems may use Mtrace2
to learn about routers of interest, on which to instantiate for example the
IOM mechanisms introdcued in 4.1 and 4.2 ?

(just guessing. Some practical reason why / how Mtrace can be relevant in a solution
 applying this documents work would be lovely. Else only explicit notion that
 additional work is needed for integration).


377	5.2.  Application in PIM

379	   PIM-SM [RFC7761] is the most widely used multicast routing protocol
380	   deployed today.  Of the various PIM modes (PIM-SM, PIM-DM, BIDIR-PIM,
381	   PIM-SSM), PIM-SSM is the preferred method due to its simplicity and
382	   removal of network source discovery complexity.  With all PIM modes,
383	   control plane state is established in the network in order to forward
384	   multicast UDP data packets.  All PIM modes utilize network based
385	   source discovery except for PIM-SSM, which utilizes application based
386	   source discovery.  IP Multicast packets fall within the range of

I don't think we qualify Bidir-PIM as "source-discovery".

387	   224.0.0.0 through 239.255.255.255.  The telemetry solution will need
388	   to work within this address range and provide telemetry data for this
389	   UDP traffic.

391	   A proposed solution for encapsulating the telemetry instruction
392	   header and metadata in IPv6 packets is described in
393	   [I-D.ietf-ippm-ioam-ipv6-options].

So... is there any IPv4 proposal, or do we only have IPv6 options ? 387-389
does not mention an IPv4 encapsulation. Would be useful to be explicit. And
maybe start with IPv6, and then only mention IPv4. 

395	5.3.  Application of MVPN X-PMSI Tunnel Encapsulation Attribute

397	   Multipoint Label Distribution Protocol (mLDP), P2MP RSVP-TE, Ingress
398	   Replication (IR), PIM MDT SAFI with GRE Transport, are commonly used
399	   within a Multicast VPN (MVPN) environment utilizing MVPN procedures
400	   Multicast in MPLS/BGP IP VPNs [RFC6513] and BGP Encoding and
401	   Procedures for Multicast in MPLS/BGP IP VPNs [RFC6514].  MLDP LDP
402	   Extension for P2MP and MP2MP LSPs [RFC6388] provides extensions to
403	   LDP to establish point-to-multipoint (P2MP) and multipoint-to-
404	   multipoint (MP2MP) label switched paths (LSPs) in MPLS networks.
405	   P2MP RSVP-TE provides extensions to RSVP-TE RSVP-TE for P2MP LSPs
406	   [RFC4875] for establish traffic-engineered P2MP LSPs in MPLS
407	   networks.  Ingress Replication (IR) P2MP Trees Ingress Replication
408	   Tunnels in Multicast VPNs [RFC7988] using unicast replication from
409	   parent node to child node over MPLS Unicast Tunnel.  PIM MDT SAFI
410	   Multicast in BGP/MPLS IP VPNs [RFC6037]utilizes PIM modes PIM-SSM,
411	   PIM-SM, PIM-BIDIR control plane with GRE transport data plane in the
412	   core for X-PMSI P-Tree using MVPN procedures.  Replication SID SR
413	   Replication Segment for Multi-point Service Delivery
414	   [I-D.ietf-spring-sr-replication-segment] replication segments for
415	   P2MP multicast service delivery in Segment Routing SR-MPLS networks.
416	   The telemetry solution will need to be able to follow these P2MP and
417	   MP2MP paths.  The telemetry instruction header and data should be
418	   encapsulated into MPLS packets on P2MP and MP2MP paths.  A
419	   corresponding proposal is described in
420	   [I-D.song-mpls-extension-header].

This is unreadable.

Correct me if i am wrong, but IAM even with the extensions
proposed here is primarily a forwarding plane thing. So i would suggest
to start the section by stating that IOAM and the recommendations of this
document (4.1, 4.2) are equally applicable to multicast MPLS forwarded packets,
and include Hayou's draft as a possible forwarding plane mechanism.

Then a paragraph listing the relevant forwarding plane RFCs.

Then another paragraph stating that network management will need to correlate
and potentially use/expand equally the multicast MPLS control plane - and list
the according RFCs for control-plane.

422	5.4.  Application in BIER

424	   BIER [RFC8279] adds a new header to multicast packets and allows the
425	   multicast packets to be forwarded according to the header only.  By
426	   eliminating the requirement of maintaining per multicast group state,
427	   BIER is more scalable than the traditional multicast solutions.

429	   OAM Requirements for BIER [I-D.ietf-bier-oam-requirements] lists many
430	   of the requirements for OAM at the BIER layer which will help in the
431	   forming of on-path telemetry requirements as well.

433	   Depending on how the BIER header is encapsulated into packets with
434	   different transport protocols, the method to encapsulate the
435	   telemetry instruction header and metadata also varies.  It is also
436	   possible to make the instruction header and metadata a part of the
437	   BIER header itself, such as in a TLV.

If you can fit my prior note about BIER-TE, rfc9262 here, that would be lovely.

Cheers
    Toerless

439	6.  Security Considerations

441	   No new security issues are identified other than those discovered by
442	   the IOAM trace and DEX options.

444	7.  IANA Considerations

446	   The document requests two new extension flag registrations in the
447	   "IOAM DEX Extension-Flags" registry, as described in Section 4.1.

449	   Bit 2 "Multicast Branching Node ID [RFC XXXX] [RFC Editor: please
450	   replace with the RFC number of the current document]".

452	   Bit 3 "Multicast Branching Interface Index [RFC XXXX] [RFC Editor:
453	   please replace with the RFC number of the current document]".

455	8.  Acknowledgments

457	   The authors would like to thank Frank Brockners, Nils Warnke, Jake
458	   Holland, and Dino Farinacci for the comments and suggestions.

460	9.  References

462	9.1.  Normative References

464	   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
465	              Requirement Levels", BCP 14, RFC 2119,
466	              DOI 10.17487/RFC2119, March 1997,
467	              <https://www.rfc-editor.org/info/rfc2119>.

469	   [RFC4687]  Yasukawa, S., Farrel, A., King, D., and T. Nadeau,
470	              "Operations and Management (OAM) Requirements for Point-
471	              to-Multipoint MPLS Networks", RFC 4687,
472	              DOI 10.17487/RFC4687, September 2006,
473	              <https://www.rfc-editor.org/info/rfc4687>.

475	   [RFC4875]  Aggarwal, R., Ed., Papadimitriou, D., Ed., and S.
476	              Yasukawa, Ed., "Extensions to Resource Reservation
477	              Protocol - Traffic Engineering (RSVP-TE) for Point-to-
478	              Multipoint TE Label Switched Paths (LSPs)", RFC 4875,
479	              DOI 10.17487/RFC4875, May 2007,
480	              <https://www.rfc-editor.org/info/rfc4875>.

482	   [RFC6037]  Rosen, E., Ed., Cai, Y., Ed., and IJ. Wijnands, "Cisco
483	              Systems' Solution for Multicast in BGP/MPLS IP VPNs",
484	              RFC 6037, DOI 10.17487/RFC6037, October 2010,
485	              <https://www.rfc-editor.org/info/rfc6037>.

487	   [RFC6388]  Wijnands, IJ., Ed., Minei, I., Ed., Kompella, K., and B.
488	              Thomas, "Label Distribution Protocol Extensions for Point-
489	              to-Multipoint and Multipoint-to-Multipoint Label Switched
490	              Paths", RFC 6388, DOI 10.17487/RFC6388, November 2011,
491	              <https://www.rfc-editor.org/info/rfc6388>.

493	   [RFC6513]  Rosen, E., Ed. and R. Aggarwal, Ed., "Multicast in MPLS/
494	              BGP IP VPNs", RFC 6513, DOI 10.17487/RFC6513, February
495	              2012, <https://www.rfc-editor.org/info/rfc6513>.

497	   [RFC6514]  Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP
498	              Encodings and Procedures for Multicast in MPLS/BGP IP
499	              VPNs", RFC 6514, DOI 10.17487/RFC6514, February 2012,
500	              <https://www.rfc-editor.org/info/rfc6514>.

502	   [RFC7761]  Fenner, B., Handley, M., Holbrook, H., Kouvelas, I.,
503	              Parekh, R., Zhang, Z., and L. Zheng, "Protocol Independent
504	              Multicast - Sparse Mode (PIM-SM): Protocol Specification
505	              (Revised)", STD 83, RFC 7761, DOI 10.17487/RFC7761, March
506	              2016, <https://www.rfc-editor.org/info/rfc7761>.

508	   [RFC7988]  Rosen, E., Ed., Subramanian, K., and Z. Zhang, "Ingress
509	              Replication Tunnels in Multicast VPN", RFC 7988,
510	              DOI 10.17487/RFC7988, October 2016,
511	              <https://www.rfc-editor.org/info/rfc7988>.

513	   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
514	              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
515	              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

517	   [RFC8279]  Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A.,
518	              Przygienda, T., and S. Aldrin, "Multicast Using Bit Index
519	              Explicit Replication (BIER)", RFC 8279,
520	              DOI 10.17487/RFC8279, November 2017,
521	              <https://www.rfc-editor.org/info/rfc8279>.

523	   [RFC8487]  Asaeda, H., Meyer, K., and W. Lee, Ed., "Mtrace Version 2:
524	              Traceroute Facility for IP Multicast", RFC 8487,
525	              DOI 10.17487/RFC8487, October 2018,
526	              <https://www.rfc-editor.org/info/rfc8487>.

528	   [RFC9197]  Brockners, F., Ed., Bhandari, S., Ed., and T. Mizrahi,
529	              Ed., "Data Fields for In Situ Operations, Administration,
530	              and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197,
531	              May 2022, <https://www.rfc-editor.org/info/rfc9197>.

533	   [RFC9326]  Song, H., Gafni, B., Brockners, F., Bhandari, S., and T.
534	              Mizrahi, "In Situ Operations, Administration, and
535	              Maintenance (IOAM) Direct Exporting", RFC 9326,
536	              DOI 10.17487/RFC9326, November 2022,
537	              <https://www.rfc-editor.org/info/rfc9326>.

539	9.2.  Informative References

541	   [I-D.ietf-bier-oam-requirements]
542	              Mirsky, G., Nainar, N. K., Chen, M., and S. Pallagatti,
543	              "Operations, Administration and Maintenance (OAM)
544	              Requirements for Bit Index Explicit Replication (BIER)
545	              Layer", Work in Progress, Internet-Draft, draft-ietf-bier-
546	              oam-requirements-13, 10 August 2023,
547	              <https://datatracker.ietf.org/doc/html/draft-ietf-bier-
548	              oam-requirements-13>.

550	   [I-D.ietf-ippm-ioam-ipv6-options]
551	              Bhandari, S. and F. Brockners, "In-situ OAM IPv6 Options",
552	              Work in Progress, Internet-Draft, draft-ietf-ippm-ioam-
553	              ipv6-options-12, 7 May 2023,
554	              <https://datatracker.ietf.org/doc/html/draft-ietf-ippm-
555	              ioam-ipv6-options-12>.

557	   [I-D.ietf-spring-sr-replication-segment]
558	              Voyer, D., Filsfils, C., Parekh, R., Bidgoli, H., and Z.
559	              J. Zhang, "SR Replication segment for Multi-point Service
560	              Delivery", Work in Progress, Internet-Draft, draft-ietf-
561	              spring-sr-replication-segment-19, 28 August 2023,
562	              <https://datatracker.ietf.org/doc/html/draft-ietf-spring-
563	              sr-replication-segment-19>.

565	   [I-D.mirsky-ippm-hybrid-two-step]
566	              Mirsky, G., Lingqiang, W., Zhui, G., Song, H., and P.
567	              Thubert, "Hybrid Two-Step Performance Measurement Method",
568	              Work in Progress, Internet-Draft, draft-mirsky-ippm-
569	              hybrid-two-step-15, 15 June 2023,
570	              <https://datatracker.ietf.org/doc/html/draft-mirsky-ippm-
571	              hybrid-two-step-15>.

573	   [I-D.song-ippm-postcard-based-telemetry]
574	              Song, H., Mirsky, G., Zhou, T., Li, Z., Graf, T., Mishra,
575	              G. S., Shin, J., and K. Lee, "On-Path Telemetry using
576	              Packet Marking to Trigger Dedicated OAM Packets", Work in
577	              Progress, Internet-Draft, draft-song-ippm-postcard-based-
578	              telemetry-16, 2 June 2023,
579	              <https://datatracker.ietf.org/doc/html/draft-song-ippm-
580	              postcard-based-telemetry-16>.

582	   [I-D.song-mpls-extension-header]
583	              Song, H., Zhou, T., Andersson, L., Zhang, Z. J., and R.
584	              Gandhi, "MPLS Network Actions using Post-Stack Extension
585	              Headers", Work in Progress, Internet-Draft, draft-song-
586	              mpls-extension-header-12, 14 April 2023,
587	              <https://datatracker.ietf.org/doc/html/draft-song-mpls-
588	              extension-header-12>.

590	   [I-D.xie-bier-ipv6-encapsulation]
591	              Xie, J., Geng, L., McBride, M., Asati, R., Dhanaraj, S.,
592	              Zhu, Y., Qin, Z., Shin, M., Mishra, G. S., and X. Geng,
593	              "Encapsulation for BIER in Non-MPLS IPv6 Networks", Work
594	              in Progress, Internet-Draft, draft-xie-bier-ipv6-
595	              encapsulation-10, 22 February 2021,
596	              <https://datatracker.ietf.org/doc/html/draft-xie-bier-
597	              ipv6-encapsulation-10>.

599	Authors' Addresses

601	   Haoyu Song
602	   Futurewei Technologies
603	   2330 Central Expressway
604	   Santa Clara,
605	   United States of America
606	   Email: [e-mail]
607	   Mike McBride
608	   Futurewei Technologies
609	   2330 Central Expressway
610	   Santa Clara,
611	   United States of America
612	   Email: [e-mail]

614	   Greg Mirsky
615	   ZTE Corp.
616	   Email: [e-mail]

618	   Gyan Mishra
619	   Verizon Inc.
620	   Email: [e-mail]

622	   Hitoshi Asaeda
623	   National Institute of Information and Communications Technology
624	   4-2-1 Nukui-Kitamachi, Tokyo
625	   184-8795
626	   Japan
627	   Email: [e-mail]

629	   Tianran Zhou
630	   Huawei Technologies
631	   Beijing
632	   China
633	   Email: [e-mail]