Re: [ippm] WGLC for IOAM Flags

Greg Mirsky <> Fri, 08 October 2021 16:04 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 42B163A0553; Fri, 8 Oct 2021 09:04:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id ETHUk2uk3z92; Fri, 8 Oct 2021 09:04:13 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4864:20::52e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 9AD4D3A058F; Fri, 8 Oct 2021 09:04:12 -0700 (PDT)
Received: by with SMTP id d3so10749436edp.3; Fri, 08 Oct 2021 09:04:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=9ja3DgGevHSkbXsk0hM71Gw7aR1fjFEhETlm1U3iYDg=; b=SzDVuUnIv/K/rzL1wdwOxRlwidlnTbMsgLj9AOBzG5injEDWawycFRYex6cXAcuFOD tkwfWEOeO1RvQUo2CCn9ZSEzQNywGmB+Tpi2geTYUugTlKkfLmWkLGEGcNT9LBX1IfR2 1sPfY3o4Xr/fT3m684d4Vfbajk1resl3GbaRTFJd0tjXCCuKDSkgzmjBrZh/vaoxNanC 6bUBWediQXnquWmgl0vH0L9WwHt7H6N/byG6JNoe1I07Dfj4oA5UtrZ4BreotsRn1h2E cjb6K61sJFSiZJQqqVeIzBY6LfA6u1qX0KWMdU8Kgcr2+l8Wbw3xMr+paxYt1xFmHxux TjiA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=9ja3DgGevHSkbXsk0hM71Gw7aR1fjFEhETlm1U3iYDg=; b=YFlfjpXSnPIOK5mmI6QyljvuqI9IsWgVd9cCegsSA96shmhXa8gBLAvVSqrD1ASeNT aSyCwLCfrkl+N88IDovrVppSbI0PXi/xm+QWil3Py7IBy8j3wVXL1C3m/ccCWtMc9px4 T906mYNBSYMhP9XtBpxRLAlQlvsLmpbbJz5eEKW66dS4R3ICZCJvDD3PjEUnv4vpeIAo ZPg6khd/o6vaf8SHsmvu9l7o6x+4/2etG7rwfM0uVrI3ixNSKsnrM56u6EDyeZf4RHbG F+kYPimAr3a11Q4qNfBsEcez4rwitjDnZ3pprzZi87OHrPZvlwNlV4aIkbZ2g3MdhF7n itMg==
X-Gm-Message-State: AOAM530eCTk5HWbPtlSJRkdLpDMKC3a8GEByFprmJRHGG1UVw4vQ29hQ v/7TpHi0vGvrUCkrCJ/BRMMqPTx+/71hAuMn7iE=
X-Google-Smtp-Source: ABdhPJz+hcwe6Gq6/vNg0baYA8v/Bt5x8QUtRE3JirmPmjlxfesA6Mt5qYd5oMx6azYIGDQcnwbL2f0UE5IHA6O9KcQ=
X-Received: by 2002:a17:906:60c3:: with SMTP id f3mr4977770ejk.561.1633709049118; Fri, 08 Oct 2021 09:04:09 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <>
In-Reply-To: <>
From: Greg Mirsky <>
Date: Fri, 8 Oct 2021 09:03:57 -0700
Message-ID: <>
To: Tal Mizrahi <>
Cc: Tommy Pauly <>, IPPM Chairs <>, "IETF IPPM WG (" <>
Content-Type: multipart/alternative; boundary="0000000000008a03b605cdd987b9"
Archived-At: <>
Subject: Re: [ippm] WGLC for IOAM Flags
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 08 Oct 2021 16:04:18 -0000

Hi Tal,
thank you for your responses to my comments and questions. Please find my
follow-up notes in-lined below under the GIM>> tag.


On Thu, Sep 30, 2021 at 12:42 AM Tal Mizrahi <>

> Hi Greg,
> Many thanks for the detailed comments.
> Please see the responses below, as well as suggested text edits in some
> cases.
> Cheers,
> The authors.
> On Wed, Sep 8, 2021 at 1:00 AM Greg Mirsky <> wrote:
> >
> > Dear Authors, WG Chairs, and All,
> > I've read the latest version of the draft and have several comments and
> questions to share with you. I would much appreciate the opportunity to
> discuss these before I can state my support for progressing this draft to
> publication.
> >
> > The Abstract states that:
> >
> >    In-situ Operations, Administration, and Maintenance (IOAM) records
> >    operational and telemetry information in packets while they traverse
> >    a path between two points in the network.
> >
> > I think that is not a complete description of IOAM options as it
> excludes the Direct Export option. It might be easier to improve by
> s/records/collects/. Also, what, in your opinion, is the distinction
> between operational and telemetry information? I think that operational
> information collected from remote nodes is one example of telemetry
> information. What do you think?
> [TM] Agreed. Your proposal makes sense: s/records/collects/
> Regarding the difference between operational and telemetry
> information, the way I see it "telemetry" refers to measurements that
> are being exported to an external/remote entity, whereas operational
> information is not measurement-based, such as the Node ID or Interface
> ID.
GIM>> I think that there's no standardized definition of "telemetry" and
"telemetry information". I share the idea expressed in the following:

Telemetry is automated communication that processes from multiple data
sources. Telemetry data improves customer experiences, monitors security,
application health, quality, and performance.

For different cases, the set of information referred to as telemetry data
variates. For example, CPU and buffer utilization are valuable for
virtualized environments. In other words, I understand telemetry data as
information extracted from a remote system that characterizes the system's
state. In any case, providing a definition would be helpful to a reader of
the document.

> >
> > nit s/the the/the/
> [TM] Agreed.
> > I think that Abstract and Introduction can be improved by explaining the
> purpose of defining the Loopback and Active flags. What each of them is
> intended to achieve and improve?
> [TM] The point is well-taken, and the following text will be added to
> the introduction. In order to avoid text repetition, I believe it
> would be best not to add it to the abstract as well.
> OLD:
> -
> NEW:
> The Loopback flag is used to request that each transit device along
> the path loops back a copy of the data packet to the sender. The
> Active flag indicates that a packet is used for active measurement.
GIM>> Thank you for the proposed text. I have several notes to the new text:

   - As I understand the behavior required from a node processing the
   Loopback flag, the returned packet is not the complete but truncated data
   - Is "active measurement" used in the context of RFC 7799? If "yes",
   does that mean that the packet is a specifically generated test packet?

> > It is stated in the Introduction that:
> >
> >    This document defines
> >    two new flags in the Pre-allocated and Incremental Trace options: the
> >    Loopback and Active flags.
> > Does that mean that neither of the two flags is applicable in the Direct
> Export option? Should draft-ietf-ippm-ioam-direct-export have an explicit
> statement regarding the applicability of the Loopback and Active flags? And
> if any is applicable, describe the behavior of an IOAM node receiving it.
> [TM] Right.
> The direct exporting draft explicitly states the behavior of each
> field in the DEX option header including the flags, so there seems to
> be no need to specify flags that are supported in other options.
GIM>> I'm a bit confused. Can you help me with the text from the DEX draft
that specifies the use of the Loopback and Active flags in combination with
the DEX trace option?

> >
> > In Section 4 the purpose of using the Loopback flag is explained as:
> >
> >    Loopback allows an IOAM encapsulating node to trace the path to a
> >    given destination, and to receive per-hop data about both the forward
> >    and the return path.
> > In my previous comments, I've pointed out that the Loopback mode is just
> a special case of the Direct Export option. In addition to that, I would
> appreciate your explanation of how the IOAM encapsulating node receives
> per-hop data about the return path. How does that work, for example, in an
> MPLS or SFC NSH environment?
> [TM] It is slightly similar to direct exporting, but is not a special
> case of direct exporting. In direct exporting telemetry/operational
> data is not pushed into the data packet, but the data packet with the
> DEX option triggers the data collection and optionally also exporting,
> but exporting is not mandatory and may be performed later, for example
> by the management plane. On the other hand, a trace option with the
> loopback flag causes all IOAM transit nodes to push IOAM data onto the
> trace option, and also loop a copy of the packet back to the sender.
GIM>> Thank you for providing that comparison between the result of
processing DEX and the Loopback. As I understand it, the result of DEX,
i.e., immediate export or local processing with exporting a batch of
telemetry data, is determined by the local policy. If that is the case, the
local policy can produce the same result as for the Loopback flag, which
could be the default behavior for the DEX.

> >
> > Further it Section 4, it is stated that:
> >
> >    The two IOAM looped back copies are
> >    terminated by the encapsulating node.
> > But I don't see any technical explanation that could sustain that
> assumption. As I understand the diagram in figure 1, a transit node doesn't
> know the identity of the IOAM encapsulating node. Correct? If that is the
> case, where does the transit node direct the looped packet?
> [TM] As described in the draft, the loopback flag is not necessarily
> applicable to all possible encapsulations. The way the return path is
> defined is encapsulation-specific. The draft currently states:
>    Loopback can be used only if a return path from transit nodes and
>    destination nodes towards the source (encapsulating node) exists.
>    Specifically, loopback is only applicable in encapsulations in which
>    the identity of the encapsulating node is available in the
>    encapsulation header.  If an encapsulating node receives a looped
>    back packet that was not originated from the current encapsulating
>    node, the packet is dropped.
GIM>> Please help me understand, the looped back packet is transmitted
using the same encapsulation as the data packet with the Loopback flag?
What is the rationale? If we use MPLS LSP Ping as an example, the response
can be sent in several ways, but the default is in IP/UDP encapsulation as
the commonly available transport. Why not use that and send the looped
packet over IP/UDP? I believe that the applicability of this mechanism will
be much broader.

> > It seems that the text in the last paragraph attempts to walk around my
> questions. But I don't find that paragraph sufficient. It lacks the
> normative language to guide a developer to produce resilient
> implementation. For example, how to interpret:
> >    Loopback can be used only if a return path from transit nodes and
> >    destination nodes towards the source (encapsulating node) exists.
> > That reads like a recommendation, not a requirement. As a result, an
> implementation may transmit an IOAM packet with the Loopback fag set
> regardless of whether there's a return path or not. Then, how must a
> transit node process such an IOAM packet?
> >
> [TM] Right, the last response and quoted text is relevant here as
> well. Please also note the following text from the draft:
> it is assumed that the
>    source address is available in the encapsulation header.  Thus, the
>    source address of the original packet is used as the destination
>    address in the copied packet.
> I propose to add the following sentence:
> The loopback flag MUST NOT be set if it is not guaranteed that there
> is a return path from each of the IOAM transit and IOAM decapsulating
> nodes, or if the encapsulating node's address is not available in the
> encapsulation header.
GIM>> Thank you for the proposed text. I'd note that referring to the
"encapsulating node's address" and "source address" might be interpreted as
applying only to IP. In some networks, for example, BIER, the identity of
the IOAM encapsulating node, BFIR, is known from its identifier that, in
turn, can be mapped to the IP address.

> > A very concerning statement I've found in Section 4.1
> >
> >    The encapsulating node either generates synthetic packets with an
> >    IOAM trace option that has the Loopback flag set ...
> > IOAM has been defined as a hybrid measurement protocol. This statement
> implies that it is to be used as an active OAM protocol (all per RFC 7799
> definitions). The requirements for active OAM protocols in terms of their
> applicability and, most of, security are very different from those set for
> a hybrid measurement protocol. I believe that extending IOAM into an active
> mode of measurement requires more discussion and the audience from other
> IETF WGs, e.g., 6man, SFC, BIER, MPLS, BFD, RTGWG.
> [TM] This draft does not define an active measurement protocol. The
> point in the draft is that the loopback flag can be used in active
> measurement packets.
GIM>> I have to disagree. Per RFC 7799, the use of specially
constructed test packets is the primary characteristic of an active
measurement method. By introducing the Active flag, this drat makes the
IOAM into an active measurement protocol with all the requirements to an
active measurement protocol to be applied.

> The following text edit is proposed:
> OLD:
>    The encapsulating node either generates synthetic packets with an
>    IOAM trace option that has the Loopback flag set, or sets the loopack
>    flag in a subset of the in-transit data packets.  Loopback is used
>    either proactively or on-demand, i.e., when a failure is detected.
> NEW:
>    The loopback flag can be set in the IOAM trace option either in a
>    subset of the in-transit data packets, or in synthetic packets that
>    are generated by the encapsulating node. Loopback can be used
>    either proactively or on-demand, i.e., when a failure is detected.
> >
> > The next statement explains the use case for the Loopback flag:
> >
> >    Loopback is used
> >    either proactively or on-demand, i.e., when a failure is detected.
> > That is the same use of ICMP and LSP Ping. What is the benefit of using
> the Loopback flag instead of readily available ICMP? Does it work as L2
> Linktrace? I don't see that as a sufficient benefit considering the
> complexity, exclusions, and risks associated and identified in the draft.
> It is clear that there are tools like ICMP and LSP Ping that adequately
> address scenarios listed in the draft. Don't see any technical reason for
> the introduction of another mechanism that does not have any significant
> benefit compared to the existing OAM tools. Especially, considering the
> limited information that is allowed to be collected using the Loopback IOAM
> flag ("every transit node that processes this trace option only adds a
> single data field, which is the Hop_Lim and node_id data field"). With the
> abundance of information and additional verification achieved using ICMP
> and LSP Ping, there's even less rationale in the introduction of this
> over-complex and unnatural mechanism.
> [TM] Right, as the draft says, the loopback flag is an "...alternative
> to Traceroute, that allows the encapsulating node to receive responses
> from multiple transit nodes along the path". Indeed, it slightly
> resembles ETH-LT (Ethernet Linktrace) in the sense that a single
> packet from the encapsulating node allows to quickly trace the entire
> path, but as with IOAM in general, using the loopback flag does not
> mandate a specific encapsulation.
GIM>> Also, I must note that "pro-active" use of the Loopback is possible
only as part of the active measurement protocol.

> >
> > The last paragraph states the requirement:
> >
> >    IOAM encapsulating nodes MUST NOT push an IOAM encapsulation with the
> >    Loopback flag onto data packets that already include an IOAM
> >    encapsulation.
> > I couldn't find any guidance on how an IOAM node that receives such
> invalid combination processes the packet.
> >
> [TM] The current draft only specifies the flags. Specifically in the
> scenario described above, the encapsulating node does not set the
> loopback flag. This is intended to reduce the impact of potential
> amplification attacks.
GIM>> I am still not clear about "MUST NOT push an IOAM encapsulation with
the Loopback flag" in the draft and your explanation "the encapsulating
node does not set the Loopback flag". As I understand it, in the given
network for the monitored data flow, only one IOAM encapsulating node can
exist. If that is correct, where does the existing IOAM encapsulation
come from?

> >
> > It is assumed that the identity of the IOAM encapsulating node is known
> to the node that is expected to sent a copy of the received packet back.
> But what if that is not the case? How must that node act?
> [TM] Agreed. The following new text is proposed to be added to Section 4.1:
> If the address of the encapsulating node is not available in the
> encapsulation header, then the transit/decapsulating node does not
> loop back a copy of the original packet.
GIM>> I think that points to the significant limitation of the proposed
mechanism. Comparing with MPLS LSP Ping, we see that it allows tracing the
LSP even though MPLS encapsulation does not identify the ingress LER while
the proposed Loopback would not work at all.

> > I believe that there's inconsistency between Figure 1 and the following
> text in Section 4.2
> >
> >    Thus, the
> >    source address of the original packet is used as the destination
> >    address in the copied packet.
> > It seems that the text assumes that the IOAM encapsulating node adds
> also a network layer encapsulation. But how would it work, for example, in
> SFC NSH data plane? Or MPLS?
> [TM] Right, the paragraph beginning with "Loopback can be used
> only..." (discussed in response to one of the previous comments) makes
> that restriction.
> >
> > Section 4.4 states that
> >
> >    If the Node ID matches the current
> >    IOAM node, it indicates that this is a looped back packet that was
> >    initiated by the current IOAM node, and processed accordingly.
> > But according to the definition of node_id in Section of
> draft-ietf-ippm-ioam-data, "Node identifier field to uniquely identify a
> node within the IOAM-Namespace and associated IOAM-Domain." It appears that
> simply comparing Node IDs is not sufficient to verify that the node
> received its looped back packet. What is your opinion?
> [TM] Agreed. The following update is proposed:
> OLD:
> If the Node ID matches
> NEW:
> If the Node ID and IOAM-Namespace match
> >
> > If, as stated in Section 5,
> >
> >    It [Active flag] is not intended as a replacement for existing
> >    active OAM protocols, which may run in higher layers and make use of
> >    the Active flag.
> > Then the question, What is the purpose of the Active flag? Existing
> active protocols, Fault Management and Performance Monitoring, are
> identified by, for example, use of a well-known destination port number or,
> in case of MPLS, Generic Associated Channel type. There's no apparent need
> of using IOAM option header to identify the payload as a control message of
> an active OAM protocol. On the contrary, that would likely cause more
> confusion and challenges to interoperation among different implementations.
> >
> > Three cases analyzed in Section 5 require special consideration, as
> these likely present the motivation for the introduction of the Active flag:
> >
> > the first use case is not really using the Active flag at all and thus
> the text doesn't seem adding any value to the draft;
> > it is not clear what is viewed as the benefit of using the Active flag
> compared with using any of the existing active OAM protocols in the second
> use case. I would appreciate it if you can clarify that for me. Since an
> SFC NSH network is used as an example, is it required that a Service
> Function (SF) or an SF Proxy process the IOAM header with Active flag set?
> Or would a Service Function Forwarder (SFF) avoid sending a packet with the
> IOAM header that has an Active flag set? And can you compare that with the
> procedures defined in the Active OAM for SFC draft?
> > My questions for the third use case described in Section 5 are the same
> as above because I don't see the benefit of using a replicated data packet
> (less benefits if it is truncated) compared to a special test packet.
> >
> > In my conclusion on the Active flag, I don't see any clear benefit of
> introducing it and using whether in place or in addition to the existing
> active OAM protocols.
> [TM] General response to all the Active flag comments: this was
> extensively discussed on the mailing list and in previous IETF
> meetings, and your comments about the previous versions of the
> document have contributed to greatly improving Section 5, which
> describes the main use cases of the Active flag.
GIM>> I have to disagree with you here. I don't find the explanation in the
document sufficient to justify the introduction of another mechanism to
achieve what has already been satisfactorily addressed in IETF. Without a
more detailed specification of the problem solved by the Active flag and
how that is achieved, this mechanism appears like the answer is looking for
a question, like a loaded gun without safety.

> >
> > I am looking forward to your feedback and the continued discussion.
> >
> > Regards,
> > Greg
> >
> > On Mon, Aug 30, 2021 at 1:58 PM Tommy Pauly <tpauly=
>> wrote:
> >>
> >> Hello IPPM,
> >>
> >> This email starts a Working Group Last Call on two related IOAM
> documents, draft-ietf-ippm-ioam-flags and
> draft-ietf-ippm-ioam-direct-export.
> >>
> >> In-situ OAM Loopback and Active Flags
> >>
> >>
> >>
> >> In-situ OAM Direct Exporting
> >>
> >>
> >>
> >> Please provide feedback by replying to this email with your reviews and
> if you think these documents are ready to progress, by Wednesday, September
> 15.
> >>
> >> Best,
> >> Tommy & Ian
> >>
> >> _______________________________________________
> >> ippm mailing list
> >>
> >>
> >
> > _______________________________________________
> > ippm mailing list
> >
> >