Re: [ippm] [MBONED] WGLC for draft-ietf-mboned-multicast-telemetry

[Toerless Eckert <[e-mail]>]

Thanks, Hitoshi

I guess i was irritated by skipping the step that explains how mtrace traces
a control-plane path, which to me is a core difference of interest.

Cheers
    Toerless

On Thu, Sep 21, 2023 at 01:16:18PM +0900, Hitoshi Asaeda wrote:
> Hi Toerless,
> 
> I reply for your Mtrace2’s comment.
> 
> > 352 5.1.  Mtrace verson 2
> > 
> > 354    Mtrace version 2 (Mtrace2) [RFC8487] is a protocol that allows the
> > 355    tracing of an IP multicast routing path.  Mtrace2 provides additional
> > 356    information such as the packet rates and losses, as well as other
> > 357    diagnostic information.  Unlike unicast traceroute, Mtrace2 traces
> > 358    the path that a packet would take from a source to a receiver.  It is
> > 
> > Not really, right ? Mtrace2 traces the path that the tree building messages
> > follow from receiver to source. The reverse of this traced path can then be
> > assumed to be the path taken by multicast packets on the branch of the tree
> > to the receiver fdrom which the Mtrace2 was started.
> 
> You are right. I don’t see any contradiction of your statement and the statement written in the draft.
> 
> Mtrace2 request packets are forwarded from a receiver to a source, while it enables the receiver to trace the path for a packet forwarded from the source to the receiver after receiving the Mtrace2 reply as the forwarding path is constructed with the RPF method.
> 
> Regards,
> 
> Hitoshi
> 
> 
> > On Sep 21, 2023, at 10:47, Toerless Eckert <[e-mail]> wrote:
> > 
> > On Thu, Sep 14, 2023 at 08:01:31AM -0700, Leonard Giuliano wrote:
> >> MBONED (and IPPM),
> >> 
> >> We would like to officially begin working group last call for Multicast
> >> On-path Telemetry using IOAM.  We had a previous WGLC for this doc earlier
> >> this year, and at the time there was not enough response from the WG to
> >> support advancement, but the draft has since been updated and in the meeting
> >> in IETF117 there was support in the room for another WGLC.  It was also
> >> suggested that IPPM be included for their input and review.
> >> 
> >> Please post whether you support/oppose the advancement of this draft as well
> >> as any comments you may have to the list by Sept 29.
> >> 
> >> Most recent version of the draft can be found here:
> >> https://datatracker.ietf.org/doc/draft-ietf-mboned-multicast-telemetry/
> >> 
> >> -MBONED Chairs
> > 
> > Thanks a lot folks for the work. Some comments, mostly textual inline below in
> > idnits output. I did not check the idnits header created. One terminology / functional
> > suggestion about the use of pre-existing rfc9197 terminology (ingress/egress interfaces)
> > as well as tracing the ingress interface as well.
> > 
> > 
> > idnits 2.17.1 
> > 
> > draft-ietf-mboned-multicast-telemetry-07.txt:
> > 
> >  Checking boilerplate required by RFC 5378 and the IETF Trust (see
> >  https://trustee.ietf.org/license-info):
> >  ----------------------------------------------------------------------------
> > 
> >     No issues found here.
> > 
> >  Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt:
> >  ----------------------------------------------------------------------------
> > 
> >     No issues found here.
> > 
> >  Checking nits according to https://www.ietf.org/id-info/checklist :
> >  ----------------------------------------------------------------------------
> > 
> >  == There are 1 instance of lines with multicast IPv4 addresses in the
> >     document.  If these are generic example addresses, they should be changed
> >     to use the 233.252.0.x range defined in RFC 5771
> > 
> > 
> >  Miscellaneous warnings:
> >  ----------------------------------------------------------------------------
> > 
> >  -- The document date (6 September 2023) is 15 days in the past.  Is this
> >     intentional?
> > 
> > 
> >  Checking references for intended status: Proposed Standard
> >  ----------------------------------------------------------------------------
> > 
> >     (See RFCs 3967 and 4897 for information about using normative references
> >     to lower-maturity documents in RFCs)
> > 
> >  == Missing Reference: 'A' is mentioned on line 220, but not defined
> > 
> >  -- Looks like a reference, but probably isn't: '0' on line 220
> > 
> >  -- Looks like a reference, but probably isn't: '1' on line 220
> > 
> >  == Missing Reference: 'RFC XXXX' is mentioned on line 452, but not defined
> > 
> >  == Unused Reference: 'RFC4687' is defined on line 469, but no explicit
> >     reference was found in the text
> > 
> >  == Unused Reference: 'RFC6037' is defined on line 482, but no explicit
> >     reference was found in the text
> > 
> >  == Unused Reference: 'I-D.xie-bier-ipv6-encapsulation' is defined on line
> >     590, but no explicit reference was found in the text
> > 
> >  ** Downref: Normative reference to an Informational RFC: RFC 4687
> > 
> >  ** Downref: Normative reference to an Historic RFC: RFC 6037
> > 
> > 
> >     Summary: 2 errors (**), 0 flaws (~~), 6 warnings (==), 3 comments (--).
> > 
> >     Run idnits with the --verbose option for more detailed information about
> >     the items above.
> > 
> > --------------------------------------------------------------------------------
> > 
> > 
> > 2 MBONED                                                           H. Song
> > 3 Internet-Draft                                                M. McBride
> > 4 Intended status: Standards Track                  Futurewei Technologies
> > 5 Expires: 9 March 2024                                          G. Mirsky
> > 6                                                                ZTE Corp.
> > 7                                                                G. Mishra
> > 8                                                             Verizon Inc.
> > 9                                                                H. Asaeda
> > 10                                                                     NICT
> > 11                                                                  T. Zhou
> > 12                                                      Huawei Technologies
> > 13                                                         6 September 2023
> > 
> > 15                  Multicast On-path Telemetry using IOAM
> > 16                 draft-ietf-mboned-multicast-telemetry-07
> > 
> > 18 Abstract
> > 
> > 20    This document specifies the requirements of on-path telemetry for
> > 21    multicast traffic using In-situ OAM.  While In-situ OAM is
> > 22    advantageous for multicast traffic telemetry, some unique challenges
> > 23    present.  This document provides the solutions based on the In-situ
> > 
> > "some unique challenges present" - does not parse.
> > 
> > 24    OAM trace option and direct export option to support the telemetry
> > 25    data correlation and the multicast tree reconstruction without
> > 26    incurring data redundancy.
> > 
> > 28 Requirements Language
> > 
> > 30    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
> > 31    "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
> > 32    "OPTIONAL" in this document are to be interpreted as described in BCP
> > 33    14 [RFC2119][RFC8174] when, and only when, they appear in all
> > 34    capitals, as shown here.
> > 
> > 36 Status of This Memo
> > 
> > 38    This Internet-Draft is submitted in full conformance with the
> > 39    provisions of BCP 78 and BCP 79.
> > 
> > 41    Internet-Drafts are working documents of the Internet Engineering
> > 42    Task Force (IETF).  Note that other groups may also distribute
> > 43    working documents as Internet-Drafts.  The list of current Internet-
> > 44    Drafts is at https://datatracker.ietf.org/drafts/current/.
> > 
> > 46    Internet-Drafts are draft documents valid for a maximum of six months
> > 47    and may be updated, replaced, or obsoleted by other documents at any
> > 48    time.  It is inappropriate to use Internet-Drafts as reference
> > 49    material or to cite them other than as "work in progress."
> > 
> > 51    This Internet-Draft will expire on 9 March 2024.
> > 
> > 53 Copyright Notice
> > 
> > 55    Copyright (c) 2023 IETF Trust and the persons identified as the
> > 56    document authors.  All rights reserved.
> > 
> > 58    This document is subject to BCP 78 and the IETF Trust's Legal
> > 59    Provisions Relating to IETF Documents (https://trustee.ietf.org/
> > 60    license-info) in effect on the date of publication of this document.
> > 61    Please review these documents carefully, as they describe your rights
> > 62    and restrictions with respect to this document.  Code Components
> > 63    extracted from this document must include Revised BSD License text as
> > 64    described in Section 4.e of the Trust Legal Provisions and are
> > 65    provided without warranty as described in the Revised BSD License.
> > 
> > 67 Table of Contents
> > 
> > 69    1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
> > 70    2.  Requirements for Multicast Traffic Telemetry  . . . . . . . .   4
> > 71    3.  Issues of Existing Techniques . . . . . . . . . . . . . . . .   4
> > 72    4.  Modifications to Existing Solutions . . . . . . . . . . . . .   5
> > 73      4.1.  Per-hop postcard using IOAM DEX . . . . . . . . . . . . .   5
> > 74      4.2.  Per-section postcard for IOAM Trace . . . . . . . . . . .   7
> > 75    5.  Application Considerations for Multicast Protocols  . . . . .   8
> > 76      5.1.  Mtrace verson 2 . . . . . . . . . . . . . . . . . . . . .   8
> > 77      5.2.  Application in PIM  . . . . . . . . . . . . . . . . . . .   9
> > 78      5.3.  Application of MVPN X-PMSI Tunnel Encapsulation
> > 79            Attribute . . . . . . . . . . . . . . . . . . . . . . . .   9
> > 80      5.4.  Application in BIER . . . . . . . . . . . . . . . . . . .  10
> > 81    6.  Security Considerations . . . . . . . . . . . . . . . . . . .  10
> > 82    7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  10
> > 83    8.  Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  10
> > 84    9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  10
> > 85      9.1.  Normative References  . . . . . . . . . . . . . . . . . .  10
> > 86      9.2.  Informative References  . . . . . . . . . . . . . . . . .  12
> > 87    Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  13
> > 
> > 89 1.  Introduction
> > 
> > 91    Multicast is used by residential broadband customers across operator
> > 
> > suggest /used by/used for example by, but not limited to/
> > aka: there's a lot more.
> > 
> > 92    networks, private MPLS customers, and internal customers within
> > 93    corporate intranet.  Multicast provides real time interactive online
> > 94    meetings or podcasts, IPTV, and financial markets real-time data,
> > 95    which all have a reliance on UDP's unreliable transport.  End-to-end
> > 96    QOS, therefore, should be a critical component of multicast
> > 97    deployment in order to provide a good end user experience.  In
> > 98    multicast video streaming, if a packet is dropped, and that packet
> > 99    happens to be a reference frame (i.e., I-Frame) in the video feed,
> > 100    the client receiver of the multicast feed goes into buffering mode
> > 101    resulting in a frozen window. 
> > 
> > Suggest rewording of last sentence:
> > 
> > In multicast real-time media streaming, loss of a single packet containing a
> > reference frame can result in the inability of all thousands of receivers to
> > decode a whole so-called Group-of-Picture sequence of packets, introducing
> > black picture for periods often as much as 4 seconds. Unexpected long delay
> > in propagation of a packet in such real-time media streaming may equally result
> > oin the packet not being received and cxreating the same results.
> > 
> > 101    Multicast packet drops and delay can
> > 
> > ...therefore
> > 
> > 102    severely affect the application performance and user experience.
> > 
> > 104    It is important to monitor the performance of the multicast traffic.
> > 105    New on-path telemetry techniques such as In-situ OAM (IOAM)
> > 106    [RFC9197], IOAM Direct Export (DEX) [RFC9326] IOAM Marking-based
> > 107    Postcard (PBT-M) [I-D.song-ippm-postcard-based-telemetry], and Hybrid
> > 108    Two-Step (HTS) [I-D.mirsky-ippm-hybrid-two-step] are useful and
> > 109    complementary to the existing active OAM performance monitoring
> > 110    methods, provide promising means to directly monitor the network
> > 
> > "existing active OAM performance monitoring" - such as ? One or two references
> > would help.
> > 
> > 111    experience of multicast traffic.  However, multicast traffic has some
> > 112    unique characteristics which pose some challenges on applying such
> > 113    techniques in an efficient way.
> > 
> > 115    The IP multicast packet data for a particular (S, G) state is
> > 116    identical from one branch to another on its way to multiple
> > 117    receivers.  When adding IOAM trace data to multicast packets, each
> > 118    replicated packet would keep the telemetry data for its entire
> > 119    forwarding path.  Since the replicated packets all share common path
> > 120    segments, redundant data will be collected for the same original
> > 121    multicast packet.  Such redundancy consumes extra network bandwidth
> > 122    unnecessarily.  For a large multicast tree, such redundancy is
> > 123    considerable.  Alternatively, it could be more efficient to collect
> > 124    the telemetry data using solutions such as IOAM DEX to eliminate the
> > 125    data redundancy.  However, IOAM DEX is lack of a branch identifier,
> > 
> > s/is lack of/lacks/
> > 
> > 126    making telemetry data correlation and multicast-tree reconstruction
> > 127    difficult.
> > 
> > 129    This draft provides two solutions to the IOAM data redundancy problem
> > 130    based on the IOAM standards.  The requirements for multicast traffic
> > 131    telemetry are discussed along with the issues of the existing on-path
> > 132    telemetry techniques.  We propose modifications to make these
> > 133    techniques adapt to multicast in order for the original multicast
> > 134    tree to be correctly reconstructed while eliminating redundant data.
> > 
> > 136 2.  Requirements for Multicast Traffic Telemetry
> > 
> > 138    Multicast traffic is forwarded through a multicast tree.  With PIM
> > 139    and P2MP (e.g., MLDP, RSVP-TE) the forwarding tree is established and
> > 
> > Please provide expansion of terms and references for them on first use, aka: for
> > MLDP, RSVP-TE here (not later). Please check for all abbreviations in the document,
> > i may overlook them as i am not doing a thorough search.
> > 
> > 140    maintained by the multicast routing protocol.  With BIER, no state is
> > 141    created in the network to establish a forwarding tree; instead, a
> > 142    bier header provides the necessary information for each packet to
> > 143    know the egress points.  Multicast packets are only replicated at
> > 
> > And with BIER-TE the replication points too ;-))
> > 
> > 144    each tree branch fork node for efficiency.
> > 
> > 146    There are several requirements for multicast traffic telemetry, a few
> > 147    of which are:
> > 
> > 149    *  Reconstruct and visualize the multicast tree through data plane
> > 150       monitoring.
> > 
> > 152    *  Gather the multicast packet delay and jitter performance on each
> > 153       path.
> > 
> > 155    *  Find the multicast packet drop location and reason.
> > 
> > 157    *  Gather the VPN state and tunnel information in case of P2MP
> > 158       multicast.
> > 
> > 160    In order to meet these requirements, we need the ability to directly
> > 161    monitor the multicast traffic and derive data from the multicast
> > 162    packets.  The conventional OAM mechanisms, such as multicast ping and
> > 163    trace, are not sufficient to meet these requirements.
> > 
> > references for ping and trace here please.
> > 
> > 165 3.  Issues of Existing Techniques
> > 
> > 167    On-path Telemetry techniques that directly retrieve data from
> > 168    multicast traffic's live network experience are ideal for addressing
> > 169    the aforementioned requirements.  The representative techniques
> > 170    include In-situ OAM (IOAM) Trace option [RFC9197], IOAM Direct Export
> > 171    (DEX) option [RFC9326], and PBT-M
> > 172    [I-D.song-ippm-postcard-based-telemetry].  However, unlike unicast,
> > 173    multicast poses some unique challenges to applying these techniques.
> > 
> > 175    Multicast packets are replicated at each branch fork node in the
> > 176    corresponding multicast tree.  Therefore, there are multiple copies
> > 177    of the original multicast packet in the network.
> > 
> > 179    If the IOAM trace option is used for on-path data collection, the
> > 180    partial trace data will also be replicated into the copy for each
> > 181    branch.  The end result is that, at the multicast tree leaves, each
> > 182    copy of the multicast packet has a complete trace.  Most of the data
> > 183    (except data from the last leaf branch), however, has redundant
> > 184    copies.  Data redundancy introduces unnecessary header overhead,
> > 185    wastes network bandwidth, and complicates the data processing.  The
> > 186    larger the multicast tree is or the longer the multicast path is, the
> > 187    more severe the redundancy problem becomes.
> > 
> > It is not clear to me if i am incorrectly understanding how IAM trace is supposed to work.
> > I thought that it would allow to collect trace information from more than one hop along the
> > path. So if i assume there is replication on every hop with multicast, and collection of
> > trace data on every hop, then the packet on every receiver would still have different
> > trace data, and only initial parts of the trace would be the same. But if lets
> > say i am multicast to 100 receivers, i am still saving a huge amount of data on
> > the wire by multicasting as compared to the source sending 100 unicast packets.
> > 
> > Whether this is true or false, it seems such a simple example to summarize how
> > IOAM trace works would help the text.
> > 
> > 189    The postcard-based solutions (e.g., IOAM DEX), can be used to
> > 190    eliminate such data redundancy, because each node on the tree only
> > 191    sends a postcard covering local data.  However, they cannot track and
> > 192    correlate the tree branches properly due to the lack of branching
> > 193    information, so they can bring confusion about the multicast tree
> > 194    topology.  For example, in a multicast tree, Node A has two branches,
> > 195    one to Node B and the other to node C; further, Node B leads to Node
> > 196    D and Node C leads to Node E.  When applying postcard-based methods,
> > 197    one cannot tell whether or not Node D(E) is the next hop of Node B(C)
> > 198    from the received postcards.
> > 
> > Unless one correlates the exporting nodes with knowledge about the tree collected
> > by other means, such as mtrace or SNMP/YANG. Aka: May be useful to include an
> > argument why such correlation is undesirable or inferior to what's proposed here.
> > 
> > May i also note that BIER-TE, RFC9262 does contain the tree information in the
> > packet header, so it would be possible to directly deduce the tree a packet was
> > passed over when correlating received IOAM DEX information.
> > 
> > 200    The fundamental reason for this problem is that there is not an
> > 201    identifier (either implicit or explicit) to correlate the data on
> > 202    each branch.
> > 
> > 204 4.  Modifications to Existing Solutions
> > 
> > 206    We provide two solutions to address the above issues.  One is based
> > 207    on IOAM DEX and requires an extension to the instruction header of
> > 208    the IOAM DEX Option.  The second solution combines the IOAM trace
> > 209    option and postcards for redundancy removal.
> > 
> > 211 4.1.  Per-hop postcard using IOAM DEX
> > 
> > 213    One way to mitigate the postcard-based telemetry's tree tracking
> > 214    weakness is to augment it with a branch identifier field.  Note that
> > 215    this works for the IOAM DEX option but not for PBT-M because the IOAM
> > 216    DEX option has an instruction header which can be used to hold the
> > 217    branch identifier.  To make the branch identifier globally unique,
> > 218    the branch fork node ID plus an index is used.  For example, Node A
> > 219    has two branches: one to Node B and the other to Node C.  Node A will
> > 220    use [A, 0] as the branch identifier for the branch to B, and [A, 1]
> > 221    for the branch to C.  The identifier is carried with the multicast
> > 222    packet until the next branch fork node.  Each node MUST export the
> > 223    branch identifier in the received IOAM DEX header in the postcards it
> > 224    sends.  The branch identifier, along with the other fields such as
> > 225    flow ID and sequence number, is sufficient for the data collector to
> > 226    reconstruct the topology of the multicast tree.
> > 
> > 228    Figure 1 shows an example of this solution.  "P" stands for the
> > 229    postcard packet.  The square brackets contains the branch identifier.
> > 230    The curly brace contains the telemetry data about a specific node.
> > 
> > 232      P:[A,0]{A}  P:[A,0]{B} P:[B,1]{D}  P:[B,0]{C}   P:[B,0]{E}
> > 233           ^            ^          ^        ^           ^
> > 234           :            :          :        :           :
> > 235           :            :          :        :           :
> > 236           :            :          :      +-:-+       +-:-+
> > 237           :            :          :      |   |       |   |
> > 238           :            :      +---:----->| C |------>| E |-...
> > 239         +-:-+        +-:-+    |   :      |   |       |   |
> > 240         |   |        |   |----+   :      +---+       +---+
> > 241         | A |------->| B |        :
> > 242         |   |        |   |--+   +-:-+
> > 243         +---+        +---+  |   |   |
> > 244                             +-->| D |--...
> > 245                                 |   |
> > 246                                 +---+
> > 
> > 248                          Figure 1: Per-hop Postcard
> > 
> > 250    Each branch fork node needs to generate a unique branch identifier
> > 251    (i.e., branch ID) for each branch in its multicast tree instance and
> > 252    include it in the IOAM DEX option header.  The branch ID remains
> > 253    unchanged until the next branch fork node.  The branch ID contains
> > 254    two parts: the branch fork node ID and an interface index.
> > 
> > 256    Conforming to the node ID specification in IOAM [RFC9197], the node
> > 257    ID is a 3-octet unsigned integer.  The interface index is a two-octet
> > 258    unsigned integer.  As shown in Figure 2, the branch ID consumes 8
> > 259    octets in total.  The three unused octets MUST be set to 0.
> > 
> > 261        0                   1                   2                   3
> > 262        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> > 263       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > 264       |                 node_id                       |     unused    |
> > 265       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > 266       |       Interface Index         |           unused              |
> > 267       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > 
> > 269                     Figure 2: Multicast Branch ID format
> > 
> > 271    Figure 3 shows that the branch ID is carried as an optional field
> > 272    after the flow ID and sequence number optional fields in the IOAM DEX
> > 273    option header.  Two bits "N" and "I" (i.e., the third and fourth bits
> > 274    in the Extension-Flags field) are reserved to indicate the presence
> > 275    of the optional branch ID field.  "N" stands for the Node ID and "I"
> > 276    stands for the interface index.  If "N" and "I" are both set to 1,
> > 277    the optional multicast branch ID field is present; otherwise it is
> > 278    absent.
> > 
> > 280         0                   1                   2                   3
> > 281         0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> > 282        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > 283        |        Namespace-ID           |     Flags     |F|S|N|I|E-Flags|
> > 284        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > 285        |               IOAM-Trace-Type                 |   Reserved    |
> > 286        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > 287        |                         Flow ID (optional)                    |
> > 288        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > 289        |                     Sequence Number  (Optional)               |
> > 290        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > 291        |                       Multicast Branch ID                     |
> > 292        |                            (optional)                         |
> > 293        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > 
> > 295           Figure 3: Carry Branch ID in IOAM DEX option header
> > 
> > 297    Once a node gets the branch ID information from the upstream, it MUST
> > 298    carry this information in its telemetry data export postcards, so the
> > 299    original multicast tree can be correctly reconstructed based on the
> > 300    postcards.
> > 
> > Couple of considerations coming to mind for this mechanism:
> > 
> > 1. It looks as if the trace information is done on egress, but this is not said.
> > Is it obvious to anybody who has read throughall IOAM specs that ONLY egress trace
> > data is generated ? Aka: in IPFIX we did define both ingress and egress data capture.
> > If in doubt, then maybe add some sentence in this section explaining that this tracing
> > is done on egress, aka: after replication - and hence the branch is assumed to be known.
> > 
> > 2. I do think it would be extremely useful to also
> > know the incoming interface, not only the brach ID. And you have the 16 bit
> > "Unused" where the incoming interface would nicely fit into without increasing the
> > exported payload. For example when using MP2MP in MLDP or Bidir-PIM, the incoming
> > interface is not a given, aka: can be different.
> > 
> > 3. Of couse, if/when you have the incoming interface, then you could also state
> > that you do want to optionally do trace on ingress, namely when the packet is received
> > by discarded by RPF-check or Bidir-PIM Acceptance check. Aka: Packets that are being
> > discarded will have no output branch, but are a great feature to trace. And of course
> > there could be the simple logic of setting the branch-id to 0 to indicate the packet
> > was discarded, and to imply this was ingress trace output.
> > 
> > 4. RFC9197 already uses terms ingress_if_id and egress_if_id. If its not too much
> > trouble i would suggest to avoid inventing new terms such as "branch_id" but instead
> > reuse those pre-existing terms.
> > 
> > 5. There is another reason, why i do not like branch-id as a term: Together with
> > the examples that use branch-id 0, 1 do make me worry that implementers could mis-interpret
> > this to mean that the branch-id is allocated starting from 0 independently for
> > every multicast tree and renumbered. Aka: In one tree, the OIF to router C is the
> > first branch, so it gets branch_id 0, and on another tree, it is the second branch,
> > so it gets branch_id 1. Aka: There should be text reconfirming that the same outoing
> > interface needs to use the same identifier across all trees, and the examples
> > ideally should use non-consecutive identifiers, not starting at 0.
> > 
> > 302 4.2.  Per-section postcard for IOAM Trace
> > 
> > 304    The second solution is a combination of the IOAM trace option and the
> > 305    postcard-based telemetry.  To avoid data redundancy, at each branch
> > 306    fork node, the trace data accumulated up to this node is exported by
> > 307    a postcard before the packet is replicated.  In this solution, each
> > 308    branch also needs to maintain some identifier to help correlate the
> > 309    postcards for each tree section.  The natural way to accomplish this
> > 310    is to simply carry the branch fork node's data (including its ID) in
> > 311    the trace of each branch.  This is also necessary because each
> > 312    replicated multicast packet can have different telemetry data
> > 313    pertaining to this particular copy (e.g., node delay, egress
> > 314    timestamp, and egress interface).  As a consequence, the local data
> > 315    exported by each branch fork node can only contain partial data
> > 316    (e.g., ingress interface and ingress timestamp).
> > 
> > 318    Figure 4 shows an example in a segment of a multicast tree.  Node B
> > 319    and D are two branch fork nodes and they will export a postcard
> > 320    covering the trace data for the previous section.  The end node of
> > 321    each path will also need to export the data of the last section as a
> > 322    postcard.
> > 
> > 324                 P:{A,B'}            P:{B1,C,D'}
> > 325                    ^                     ^
> > 326                    :                     :
> > 327                    :                     :
> > 328                    :                     :    {D1}
> > 329                    :                     :    +--...
> > 330                    :        +---+      +---+  |
> > 331                    :   {B1} |   |{B1,C}|   |--+ {D2}
> > 332                    :    +-->| C |----->| D |-----...
> > 333        +---+     +---+  |   |   |      |   |--+
> > 334        |   | {A} |   |--+   +---+      +---+  |
> > 335        | A |---->| B |                        +--...
> > 336        |   |     |   |--+   +---+             {D3}
> > 337        +---+     +---+  |   |   |{B2,E}
> > 338                         +-->| E |--...
> > 339                        {B2} |   |
> > 340                             +---+
> > 
> > 342                        Figure 4: Per-section Postcard
> > 
> > 344    There is no need to modify the IOAM trace option header format as
> > 345    specified in [RFC9197].  We just need to configure the branch fork
> > 346    nodes to export the postcards and refresh the IOAM header and data
> > 347    (e.g., clear the node data list and reset the Remaining Length
> > 348    field).
> > 
> > Not quite clear: If you do not apply thre section 4.1 proposal here, how
> > would you know from the P:{B1,C,D'} postcard that it should be stitched below
> > the P:{A,B'} postcard ? Wouldn't you need the same type of ingress/egress
> > interface information to do the stitching ?
> > 
> > 350 5.  Application Considerations for Multicast Protocols
> > 
> > 352 5.1.  Mtrace verson 2
> > 
> > 354    Mtrace version 2 (Mtrace2) [RFC8487] is a protocol that allows the
> > 355    tracing of an IP multicast routing path.  Mtrace2 provides additional
> > 356    information such as the packet rates and losses, as well as other
> > 357    diagnostic information.  Unlike unicast traceroute, Mtrace2 traces
> > 358    the path that a packet would take from a source to a receiver.  It is
> > 
> > Not really, right ? Mtrace2 traces the path that the tree building messages
> > follow from receiver to source. The reverse of this traced path can then be
> > assumed to be the path taken by multicast packets on the branch of the tree
> > to the receiver fdrom which the Mtrace2 was started.
> > 
> > 359    usually initiated from an Mtrace2 client by sending an Mtrace2 Query
> > 360    to a Last-Hop Router (LHR) or to a Rendezvous Point (RP).  The LHR/RP
> > 361    turns the Query packet into an Mtrace2 Request, appends a Standard
> > 362    Response Block containing its interface addresses and packet
> > 363    statistics to the Request packet, and forwards the packet towards the
> > 364    source/RP.  In a similar fashion, each router along the path to the
> > 365    source/RP appends a Standard Response Block to the end of the Request
> > 366    packet and forwards it to its upstream router.  When the First-Hop
> > 367    Router (FHR) receives the Request packet, it appends its own Standard
> > 368    Response Block, turns the Request packet into a Reply, and unicasts
> > 369    the Reply back to the Mtrace2 client.
> > 
> > 371    New on-path telemetry techniques will enhance Mtrace2, and other
> > 372    existing OAM solutions, with more granular and realtime network
> > 373    status data through direct measurements.  There are various multicast
> > 374    protocols that are used to forward the multicast data.  Each will
> > 375    require their own unique on-path telemetry solution.
> > 
> > At this point in time, readers like me will wonder:
> > 
> > Is there actually any integration between Mtrace2 and IOAM, and how does this
> > document itself help.
> > 
> > Maybe replace last paragraph by something like: At this point, Mtrace2 does not
> > integrate with IOAM2 directly, but network management systems may use Mtrace2
> > to learn about routers of interest, on which to instantiate for example the
> > IOM mechanisms introdcued in 4.1 and 4.2 ?
> > 
> > (just guessing. Some practical reason why / how Mtrace can be relevant in a solution
> > applying this documents work would be lovely. Else only explicit notion that
> > additional work is needed for integration).
> > 
> > 
> > 377 5.2.  Application in PIM
> > 
> > 379    PIM-SM [RFC7761] is the most widely used multicast routing protocol
> > 380    deployed today.  Of the various PIM modes (PIM-SM, PIM-DM, BIDIR-PIM,
> > 381    PIM-SSM), PIM-SSM is the preferred method due to its simplicity and
> > 382    removal of network source discovery complexity.  With all PIM modes,
> > 383    control plane state is established in the network in order to forward
> > 384    multicast UDP data packets.  All PIM modes utilize network based
> > 385    source discovery except for PIM-SSM, which utilizes application based
> > 386    source discovery.  IP Multicast packets fall within the range of
> > 
> > I don't think we qualify Bidir-PIM as "source-discovery".
> > 
> > 387    224.0.0.0 through 239.255.255.255.  The telemetry solution will need
> > 388    to work within this address range and provide telemetry data for this
> > 389    UDP traffic.
> > 
> > 391    A proposed solution for encapsulating the telemetry instruction
> > 392    header and metadata in IPv6 packets is described in
> > 393    [I-D.ietf-ippm-ioam-ipv6-options].
> > 
> > So... is there any IPv4 proposal, or do we only have IPv6 options ? 387-389
> > does not mention an IPv4 encapsulation. Would be useful to be explicit. And
> > maybe start with IPv6, and then only mention IPv4. 
> > 
> > 395 5.3.  Application of MVPN X-PMSI Tunnel Encapsulation Attribute
> > 
> > 397    Multipoint Label Distribution Protocol (mLDP), P2MP RSVP-TE, Ingress
> > 398    Replication (IR), PIM MDT SAFI with GRE Transport, are commonly used
> > 399    within a Multicast VPN (MVPN) environment utilizing MVPN procedures
> > 400    Multicast in MPLS/BGP IP VPNs [RFC6513] and BGP Encoding and
> > 401    Procedures for Multicast in MPLS/BGP IP VPNs [RFC6514].  MLDP LDP
> > 402    Extension for P2MP and MP2MP LSPs [RFC6388] provides extensions to
> > 403    LDP to establish point-to-multipoint (P2MP) and multipoint-to-
> > 404    multipoint (MP2MP) label switched paths (LSPs) in MPLS networks.
> > 405    P2MP RSVP-TE provides extensions to RSVP-TE RSVP-TE for P2MP LSPs
> > 406    [RFC4875] for establish traffic-engineered P2MP LSPs in MPLS
> > 407    networks.  Ingress Replication (IR) P2MP Trees Ingress Replication
> > 408    Tunnels in Multicast VPNs [RFC7988] using unicast replication from
> > 409    parent node to child node over MPLS Unicast Tunnel.  PIM MDT SAFI
> > 410    Multicast in BGP/MPLS IP VPNs [RFC6037]utilizes PIM modes PIM-SSM,
> > 411    PIM-SM, PIM-BIDIR control plane with GRE transport data plane in the
> > 412    core for X-PMSI P-Tree using MVPN procedures.  Replication SID SR
> > 413    Replication Segment for Multi-point Service Delivery
> > 414    [I-D.ietf-spring-sr-replication-segment] replication segments for
> > 415    P2MP multicast service delivery in Segment Routing SR-MPLS networks.
> > 416    The telemetry solution will need to be able to follow these P2MP and
> > 417    MP2MP paths.  The telemetry instruction header and data should be
> > 418    encapsulated into MPLS packets on P2MP and MP2MP paths.  A
> > 419    corresponding proposal is described in
> > 420    [I-D.song-mpls-extension-header].
> > 
> > This is unreadable.
> > 
> > Correct me if i am wrong, but IAM even with the extensions
> > proposed here is primarily a forwarding plane thing. So i would suggest
> > to start the section by stating that IOAM and the recommendations of this
> > document (4.1, 4.2) are equally applicable to multicast MPLS forwarded packets,
> > and include Hayou's draft as a possible forwarding plane mechanism.
> > 
> > Then a paragraph listing the relevant forwarding plane RFCs.
> > 
> > Then another paragraph stating that network management will need to correlate
> > and potentially use/expand equally the multicast MPLS control plane - and list
> > the according RFCs for control-plane.
> > 
> > 422 5.4.  Application in BIER
> > 
> > 424    BIER [RFC8279] adds a new header to multicast packets and allows the
> > 425    multicast packets to be forwarded according to the header only.  By
> > 426    eliminating the requirement of maintaining per multicast group state,
> > 427    BIER is more scalable than the traditional multicast solutions.
> > 
> > 429    OAM Requirements for BIER [I-D.ietf-bier-oam-requirements] lists many
> > 430    of the requirements for OAM at the BIER layer which will help in the
> > 431    forming of on-path telemetry requirements as well.
> > 
> > 433    Depending on how the BIER header is encapsulated into packets with
> > 434    different transport protocols, the method to encapsulate the
> > 435    telemetry instruction header and metadata also varies.  It is also
> > 436    possible to make the instruction header and metadata a part of the
> > 437    BIER header itself, such as in a TLV.
> > 
> > If you can fit my prior note about BIER-TE, rfc9262 here, that would be lovely.
> > 
> > Cheers
> >    Toerless
> > 
> > 439 6.  Security Considerations
> > 
> > 441    No new security issues are identified other than those discovered by
> > 442    the IOAM trace and DEX options.
> > 
> > 444 7.  IANA Considerations
> > 
> > 446    The document requests two new extension flag registrations in the
> > 447    "IOAM DEX Extension-Flags" registry, as described in Section 4.1.
> > 
> > 449    Bit 2 "Multicast Branching Node ID [RFC XXXX] [RFC Editor: please
> > 450    replace with the RFC number of the current document]".
> > 
> > 452    Bit 3 "Multicast Branching Interface Index [RFC XXXX] [RFC Editor:
> > 453    please replace with the RFC number of the current document]".
> > 
> > 455 8.  Acknowledgments
> > 
> > 457    The authors would like to thank Frank Brockners, Nils Warnke, Jake
> > 458    Holland, and Dino Farinacci for the comments and suggestions.
> > 
> > 460 9.  References
> > 
> > 462 9.1.  Normative References
> > 
> > 464    [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
> > 465               Requirement Levels", BCP 14, RFC 2119,
> > 466               DOI 10.17487/RFC2119, March 1997,
> > 467               <https://www.rfc-editor.org/info/rfc2119>.
> > 
> > 469    [RFC4687]  Yasukawa, S., Farrel, A., King, D., and T. Nadeau,
> > 470               "Operations and Management (OAM) Requirements for Point-
> > 471               to-Multipoint MPLS Networks", RFC 4687,
> > 472               DOI 10.17487/RFC4687, September 2006,
> > 473               <https://www.rfc-editor.org/info/rfc4687>.
> > 
> > 475    [RFC4875]  Aggarwal, R., Ed., Papadimitriou, D., Ed., and S.
> > 476               Yasukawa, Ed., "Extensions to Resource Reservation
> > 477               Protocol - Traffic Engineering (RSVP-TE) for Point-to-
> > 478               Multipoint TE Label Switched Paths (LSPs)", RFC 4875,
> > 479               DOI 10.17487/RFC4875, May 2007,
> > 480               <https://www.rfc-editor.org/info/rfc4875>.
> > 
> > 482    [RFC6037]  Rosen, E., Ed., Cai, Y., Ed., and IJ. Wijnands, "Cisco
> > 483               Systems' Solution for Multicast in BGP/MPLS IP VPNs",
> > 484               RFC 6037, DOI 10.17487/RFC6037, October 2010,
> > 485               <https://www.rfc-editor.org/info/rfc6037>.
> > 
> > 487    [RFC6388]  Wijnands, IJ., Ed., Minei, I., Ed., Kompella, K., and B.
> > 488               Thomas, "Label Distribution Protocol Extensions for Point-
> > 489               to-Multipoint and Multipoint-to-Multipoint Label Switched
> > 490               Paths", RFC 6388, DOI 10.17487/RFC6388, November 2011,
> > 491               <https://www.rfc-editor.org/info/rfc6388>.
> > 
> > 493    [RFC6513]  Rosen, E., Ed. and R. Aggarwal, Ed., "Multicast in MPLS/
> > 494               BGP IP VPNs", RFC 6513, DOI 10.17487/RFC6513, February
> > 495               2012, <https://www.rfc-editor.org/info/rfc6513>.
> > 
> > 497    [RFC6514]  Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP
> > 498               Encodings and Procedures for Multicast in MPLS/BGP IP
> > 499               VPNs", RFC 6514, DOI 10.17487/RFC6514, February 2012,
> > 500               <https://www.rfc-editor.org/info/rfc6514>.
> > 
> > 502    [RFC7761]  Fenner, B., Handley, M., Holbrook, H., Kouvelas, I.,
> > 503               Parekh, R., Zhang, Z., and L. Zheng, "Protocol Independent
> > 504               Multicast - Sparse Mode (PIM-SM): Protocol Specification
> > 505               (Revised)", STD 83, RFC 7761, DOI 10.17487/RFC7761, March
> > 506               2016, <https://www.rfc-editor.org/info/rfc7761>.
> > 
> > 508    [RFC7988]  Rosen, E., Ed., Subramanian, K., and Z. Zhang, "Ingress
> > 509               Replication Tunnels in Multicast VPN", RFC 7988,
> > 510               DOI 10.17487/RFC7988, October 2016,
> > 511               <https://www.rfc-editor.org/info/rfc7988>.
> > 
> > 513    [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
> > 514               2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
> > 515               May 2017, <https://www.rfc-editor.org/info/rfc8174>.
> > 
> > 517    [RFC8279]  Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A.,
> > 518               Przygienda, T., and S. Aldrin, "Multicast Using Bit Index
> > 519               Explicit Replication (BIER)", RFC 8279,
> > 520               DOI 10.17487/RFC8279, November 2017,
> > 521               <https://www.rfc-editor.org/info/rfc8279>.
> > 
> > 523    [RFC8487]  Asaeda, H., Meyer, K., and W. Lee, Ed., "Mtrace Version 2:
> > 524               Traceroute Facility for IP Multicast", RFC 8487,
> > 525               DOI 10.17487/RFC8487, October 2018,
> > 526               <https://www.rfc-editor.org/info/rfc8487>.
> > 
> > 528    [RFC9197]  Brockners, F., Ed., Bhandari, S., Ed., and T. Mizrahi,
> > 529               Ed., "Data Fields for In Situ Operations, Administration,
> > 530               and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197,
> > 531               May 2022, <https://www.rfc-editor.org/info/rfc9197>.
> > 
> > 533    [RFC9326]  Song, H., Gafni, B., Brockners, F., Bhandari, S., and T.
> > 534               Mizrahi, "In Situ Operations, Administration, and
> > 535               Maintenance (IOAM) Direct Exporting", RFC 9326,
> > 536               DOI 10.17487/RFC9326, November 2022,
> > 537               <https://www.rfc-editor.org/info/rfc9326>.
> > 
> > 539 9.2.  Informative References
> > 
> > 541    [I-D.ietf-bier-oam-requirements]
> > 542               Mirsky, G., Nainar, N. K., Chen, M., and S. Pallagatti,
> > 543               "Operations, Administration and Maintenance (OAM)
> > 544               Requirements for Bit Index Explicit Replication (BIER)
> > 545               Layer", Work in Progress, Internet-Draft, draft-ietf-bier-
> > 546               oam-requirements-13, 10 August 2023,
> > 547               <https://datatracker.ietf.org/doc/html/draft-ietf-bier-
> > 548               oam-requirements-13>.
> > 
> > 550    [I-D.ietf-ippm-ioam-ipv6-options]
> > 551               Bhandari, S. and F. Brockners, "In-situ OAM IPv6 Options",
> > 552               Work in Progress, Internet-Draft, draft-ietf-ippm-ioam-
> > 553               ipv6-options-12, 7 May 2023,
> > 554               <https://datatracker.ietf.org/doc/html/draft-ietf-ippm-
> > 555               ioam-ipv6-options-12>.
> > 
> > 557    [I-D.ietf-spring-sr-replication-segment]
> > 558               Voyer, D., Filsfils, C., Parekh, R., Bidgoli, H., and Z.
> > 559               J. Zhang, "SR Replication segment for Multi-point Service
> > 560               Delivery", Work in Progress, Internet-Draft, draft-ietf-
> > 561               spring-sr-replication-segment-19, 28 August 2023,
> > 562               <https://datatracker.ietf.org/doc/html/draft-ietf-spring-
> > 563               sr-replication-segment-19>.
> > 
> > 565    [I-D.mirsky-ippm-hybrid-two-step]
> > 566               Mirsky, G., Lingqiang, W., Zhui, G., Song, H., and P.
> > 567               Thubert, "Hybrid Two-Step Performance Measurement Method",
> > 568               Work in Progress, Internet-Draft, draft-mirsky-ippm-
> > 569               hybrid-two-step-15, 15 June 2023,
> > 570               <https://datatracker.ietf.org/doc/html/draft-mirsky-ippm-
> > 571               hybrid-two-step-15>.
> > 
> > 573    [I-D.song-ippm-postcard-based-telemetry]
> > 574               Song, H., Mirsky, G., Zhou, T., Li, Z., Graf, T., Mishra,
> > 575               G. S., Shin, J., and K. Lee, "On-Path Telemetry using
> > 576               Packet Marking to Trigger Dedicated OAM Packets", Work in
> > 577               Progress, Internet-Draft, draft-song-ippm-postcard-based-
> > 578               telemetry-16, 2 June 2023,
> > 579               <https://datatracker.ietf.org/doc/html/draft-song-ippm-
> > 580               postcard-based-telemetry-16>.
> > 
> > 582    [I-D.song-mpls-extension-header]
> > 583               Song, H., Zhou, T., Andersson, L., Zhang, Z. J., and R.
> > 584               Gandhi, "MPLS Network Actions using Post-Stack Extension
> > 585               Headers", Work in Progress, Internet-Draft, draft-song-
> > 586               mpls-extension-header-12, 14 April 2023,
> > 587               <https://datatracker.ietf.org/doc/html/draft-song-mpls-
> > 588               extension-header-12>.
> > 
> > 590    [I-D.xie-bier-ipv6-encapsulation]
> > 591               Xie, J., Geng, L., McBride, M., Asati, R., Dhanaraj, S.,
> > 592               Zhu, Y., Qin, Z., Shin, M., Mishra, G. S., and X. Geng,
> > 593               "Encapsulation for BIER in Non-MPLS IPv6 Networks", Work
> > 594               in Progress, Internet-Draft, draft-xie-bier-ipv6-
> > 595               encapsulation-10, 22 February 2021,
> > 596               <https://datatracker.ietf.org/doc/html/draft-xie-bier-
> > 597               ipv6-encapsulation-10>.
> > 
> > 599 Authors' Addresses
> > 
> > 601    Haoyu Song
> > 602    Futurewei Technologies
> > 603    2330 Central Expressway
> > 604    Santa Clara,
> > 605    United States of America
> > 606    Email: [e-mail]
> > 607    Mike McBride
> > 608    Futurewei Technologies
> > 609    2330 Central Expressway
> > 610    Santa Clara,
> > 611    United States of America
> > 612    Email: [e-mail]
> > 
> > 614    Greg Mirsky
> > 615    ZTE Corp.
> > 616    Email: [e-mail]
> > 
> > 618    Gyan Mishra
> > 619    Verizon Inc.
> > 620    Email: [e-mail]
> > 
> > 622    Hitoshi Asaeda
> > 623    National Institute of Information and Communications Technology
> > 624    4-2-1 Nukui-Kitamachi, Tokyo
> > 625    184-8795
> > 626    Japan
> > 627    Email: [e-mail]
> > 
> > 629    Tianran Zhou
> > 630    Huawei Technologies
> > 631    Beijing
> > 632    China
> > 633    Email: [e-mail]
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > MBONED mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/mboned
> 
> 
> 

-- 
---
[e-mail]