Re: [nvo3] [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

[Tianran Zhou <[e-mail]>]

I think it's better that Frank or Shwetha can explain the multi-layer use case in detail.

Tianran
> -----Original Message-----
> From: Tom Herbert [mailto:[e-mail]]
> Sent: Monday, April 16, 2018 10:40 PM
> To: Tianran Zhou <[e-mail]>
> Cc: Shwetha Bhandari (shwethab) <[e-mail]>; Frank Brockners
> (fbrockne) <[e-mail]>; Mickey Spiegel
> <[e-mail]>; NVO3 <[e-mail]>; int-area
> <[e-mail]>; Service Function Chaining IETF list <[e-mail]>;
> IETF IPPM WG <[e-mail]>
> Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
> protocols - follow up from WG discussion in London
> 
> On Mon, Apr 16, 2018 at 6:31 AM, Tianran Zhou <[e-mail]> wrote:
> > Hi Shwetha,
> >
> > You are talking about the outer encapsution. It is straight forward
> > for the underlay to record by the header. But what about the overlay,
> > i.e., inner encapsulation(e.g. geneve)? Without special configuration,
> > intermediate node will not read the inner header, hence not be able to
> > process IOAM.e
> 
> Hi Tianran,
> 
> I believe that is also not protocol conformant. Intermediate nodes should
> not be processing transport layer data as this can lead to misinterpretation
> and possibly silent data corruption.
> 
> For instance, Geneve is a UDP encapsulation protocol with assigned port 6081.
> In order for an intermediate device to process the Geneve encapsulation header
> it would need to look for packets with destination port of 6081 since that
> is the only possible discriminator. However, transport port numbers do not
> have global meaning and hosts may use port numbers for other purposes (RFC7605
> describes this). So a packet to port 6081 might be something other than Geneve
> and may be misinterpreted. If a misinterpreted packet is changed (like ippm
> data is written) then that would be systematic silent data corruption.
> 
> As far as I know, hop-by-hop options is the only protocol confirming mechanism
> that allows an intermediate note to change data of packet in flight.
> Encpasulation is the only conforming mechanism that allows an intermediate
> node to add data (like extension headers) to a packet in flight.
> 
> Tom
> 
> > Maybe we are not synced by this overlay/underlay use case. :-)
> >
> > Tianran
> >
> >
> >
> > ________________________________
> > Sent from WeLink
> >
> > 发件人： Shwetha Bhandari (shwethab)
> > 收件人： Tianran Zhou<[e-mail]>;Frank Brockners
> > (fbrockne)<[e-mail]>;Mickey
> > Spiegel<[e-mail]>;Tom
> > Herbert<[e-mail]>
> > 抄送： NVO3<[e-mail]>;int-area<[e-mail]>;Service Function
> > Chaining IETF list<[e-mail]>;IETF IPPM WG<[e-mail]>
> > 主题： Re: [ippm] [Int-area] encapsulation of IOAM data in various
> > protocols - follow up from WG discussion in London
> > 时间： 2018-04-16 18:17:01
> >
> > Hi Tianran,
> >
> >> If I recall right, it is not written in the ioam data draft.
> >
> > Data draft is defining the data to be carried in IOAM in an
> > encapsulation agnostic way, it does not specify how the encapsulation
> > protocol is configured.
> >
> >
> >
> >> Yes, node by node configuration is an easy way.
> >
> > While it is, it does not have to be a node by node configuration. It
> > can be part of the encapsulation definition.
> >
> > For e.g. If the encapsulation is IPv6 and if we define the data to be
> > carried as HbH options, then based on the Option Type with highest
> > order 2 bits set to 00 then the v6 nodes that implement IOAM will
> > process the option and others will skip over.
> >
> >
> >
> >
> >
> > Thanks,
> >
> > Shwetha
> >
> >
> >
> > From: ippm <[e-mail]> on behalf of Tianran Zhou
> > <[e-mail]>
> > Date: Monday, April 16, 2018 at 2:36 PM
> > To: "Frank Brockners (fbrockne)" <[e-mail]>, Mickey Spiegel
> > <[e-mail]>, Tom Herbert <[e-mail]>
> > Cc: NVO3 <[e-mail]>, "[e-mail]" <[e-mail]>,
> > Service Function Chaining IETF list <[e-mail]>, IETF IPPM WG
> > <[e-mail]>
> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
> > protocols - follow up from WG discussion in London
> >
> >
> >
> > Hi Frank,
> >
> >
> >
> > If I recall right, it is not written in the ioam data draft.
> >
> > Yes, node by node configuration is an easy way. In the
> > draft-zhou-ippm-ioam-yang, we have the “protocol-type” to indicate the
> > layering.
> >
> >    +--rw ioam
> >
> >       +--rw ioam-profiles
> >
> >          +--rw enabled?        boolean
> >
> >          +--rw ioam-profile* [profile-name]
> >
> >             +--rw profile-name                    string
> >
> >             +--rw filter
> >
> >             |  +--rw filter-type?   ioam-filter-type
> >
> >             |  +--rw acl-name?      -> /acl:acls/acl/name
> >
> >             +--rw protocol-type?                  ioam-protocol-type
> >
> >             +--rw incremental-tracing-profile {incremental-trace}?
> >
> >             |  ...
> >
> >             +--rw preallocated-tracing-profile {preallocated-trace}?
> >
> >             |  ...
> >
> >             +--rw pot-profile {proof-of-transit}?
> >
> >             |  ...
> >
> >             +--rw e2e-profile {edge-to-edge}?
> >
> >                ...
> >
> >
> >
> >
> >
> > Tianran
> >
> > From: Frank Brockners (fbrockne) [mailto:[e-mail]]
> > Sent: Monday, April 16, 2018 4:51 PM
> > To: Tianran Zhou <[e-mail]>; Mickey Spiegel
> > <[e-mail]>; Tom Herbert <[e-mail]>
> > Cc: NVO3 <[e-mail]>; [e-mail]; Service Function Chaining
> > IETF list <[e-mail]>; IETF IPPM WG <[e-mail]>
> > Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various
> > protocols - follow up from WG discussion in London
> >
> >
> >
> > Hi Tianran,
> >
> >
> >
> > IOAM is a domain specific feature (see also
> > draft-ietf-ippm-ioam-data-02 sections 3 and 4), which allows an
> > operator to control by means of configuration where and for which
> > traffic IOAM data fields are added/updated/removed from the customer
> > traffic. Using your example of Geneve over IPv6 – with IOAM data in
> > both the Geneve and the IPv6 protocol, one would expect that the
> > operator configures the endpoints of the Geneve tunnel to operate on
> > the IOAM data in Geneve, and the IPv6 routers that the Geneve tunnel
> traverses to operate on the IOAM data in IPv6.
> >
> >
> >
> > Frank
> >
> >
> >
> > From: Tianran Zhou <[e-mail]>
> > Sent: Montag, 16. April 2018 10:37
> > To: Frank Brockners (fbrockne) <[e-mail]>; Mickey Spiegel
> > <[e-mail]>; Tom Herbert <[e-mail]>
> > Cc: NVO3 <[e-mail]>; [e-mail]; Service Function Chaining
> > IETF list <[e-mail]>; IETF IPPM WG <[e-mail]>
> > Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various
> > protocols - follow up from WG discussion in London
> >
> >
> >
> > Hi Frank,
> >
> >
> >
> > How does a forwarder know when and where to insert the data?
> >
> > In the case of Geneve over IPv6, do you mean the device need to scan
> > all the protocol stack? Or just the outer encapsulation?
> >
> >
> >
> > Tianran
> >
> >
> >
> > From: ippm [mailto:[e-mail]] On Behalf Of Frank Brockners
> > (fbrockne)
> > Sent: Monday, April 16, 2018 3:08 PM
> > To: Mickey Spiegel <[e-mail]>; Tom Herbert
> > <[e-mail]>
> > Cc: NVO3 <[e-mail]>; [e-mail]; Service Function Chaining
> > IETF list <[e-mail]>; IETF IPPM WG <[e-mail]>
> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
> > protocols - follow up from WG discussion in London
> >
> >
> >
> >
> >
> > Tom,
> >
> >
> >
> > a quick addition to what Mickey mentioned below: What you seem to have
> > in mind is what draft-ietf-ippm-ioam-data-02 refers to as “layering”
> > (see section 3.), i.e. if you’re running for example Geneve over IPv6,
> > then IOAM data could be encapsulated in both protocols, Geneve and
> > IPv6 – giving you visibility into the “underlay” (IPv6) and the “overlay”
> (Geneve).
> >
> >
> >
> > Frank
> >
> >
> >
> > From: ippm <[e-mail]> On Behalf Of Mickey Spiegel
> > Sent: Freitag, 13. April 2018 20:22
> > To: Tom Herbert <[e-mail]>
> > Cc: NVO3 <[e-mail]>; [e-mail]; Service Function Chaining
> > IETF list <[e-mail]>; IETF IPPM WG <[e-mail]>
> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
> > protocols - follow up from WG discussion in London
> >
> >
> >
> > Tom,
> >
> >
> >
> > On Thu, Apr 12, 2018 at 10:17 PM, Tom Herbert <[e-mail]> wrote:
> >
> > Mickey,
> >
> > Looking at these ippm drafts more closely, I have a much more
> > fundamental concern.
> >
> > In draft-brockners-ippm-ioam-geneve-00 for instance, there is the text
> > in the introduction:
> >
> > "In-situ OAM (IOAM) records OAM information within the packet while
> > the packet traverses a particular network domain.  The term "in-situ"
> > refers to the fact that the IOAM data fields are added to the data
> > packets rather than is being sent within packets specifically
> > dedicated to OAM.  This document defines how IOAM data fields are
> > transported as part of the Geneve [I-D.ietf-nvo3-geneve]
> > encapsulation."
> >
> > I assume this means that as packets with Geneve encapsulation traverse
> > the network they are interpreted by intermediate nodes as being
> > Geneve. Since Geneve is a UDP encapsulation, then the destination UDP
> > port number would be used to identify packets as being Geneve. So an
> > intermediate device might be looking for UDP packets destined to port
> > 6081 (the assigned UDP port for Geneve). If my understanding is
> > correct, then this is a problem.
> >
> > UDP port numbers do not have global meaning. An intermediate device
> > may very well see UDP packets destined to port 6081 that are not
> > actually Geneve. This scenario is discussed in RFC7605:
> >
> > "...intermediate device interprets traffic based on the port number.
> > It is important to recognize that any interpretation of port numbers
> > -- except at the endpoints -- may be incorrect, because port numbers
> > are meaningful only at the endpoints."
> >
> > If the UDP data is modified, as the draft would imply, then
> > misinterpretation may also mean silent data corruption of packets. A
> > protocol that would allow this seems pretty incorrect! Note that this
> > would be true also for any UDP encapsulation that the network tries to
> > interpret.
> >
> >
> >
> > The intention is to allow for multiple nodes that a packet traverses
> >
> > to be able to insert IOAM node information in the same trace option,
> >
> > but leave some flexibility regarding which nodes actually do the
> >
> > IOAM processing and the node information. This may vary
> >
> > depending on the transport.
> >
> >
> >
> > In case of a tunneled encapsulation such as Geneve or VXLAN,
> >
> > there may still be multiple hops. For example a network may use
> >
> > Geneve or VXLAN, but only do L2 processing at ToRs, with L3
> >
> > processing done at aggregation or core switches. In this case
> >
> > many packets would do 2 Geneve or VXLAN hops, so the packet
> >
> > would contain IOAM node information from two nodes.
> >
> >
> >
> > Another example is service function chaining using Geneve or
> >
> > VXLAN rather than NSH.
> >
> >
> >
> >
> > I am also wondering if hop-by-hop options been considered for this
> > application? Their interpretation in the network is unabiguous and
> > they also have the advantage that the work with any IP protocol or
> > encapsulation.
> >
> >
> >
> > IPv6 hop-by-hop options has been considered. See
> >
> > draft-brockners-inband-oam-transport-05. This has not yet been
> >
> > broken out into a separate draft.
> >
> >
> >
> > Mickey
> >
> >
> >
> >
> > Thanks,
> > Tom
> >
> >
> > On Thu, Apr 12, 2018 at 3:31 PM, Mickey Spiegel
> > <[e-mail]> wrote:
> >
> >> Tom,
> >>
> >> On Thu, Apr 12, 2018 at 2:46 PM, Tom Herbert <[e-mail]> wrote:
> >>>
> >>> On Thu, Apr 12, 2018 at 9:54 AM, Greg Mirsky <[e-mail]>
> >>> wrote:
> >>> > Hi Frank,
> >>> > thank you for sharing your points. Please find my notes in-line
> >>> > and tagged
> >>> > GIM>>. I believe that this is very much relevant to work of other
> >>> > working
> >>> > groups that directly work on the overlay encapsulations in the
> >>> > center of the discussion and hence I've added them to the list.
> >>> > Hope we'll have more opinions to reach the conclusion that is
> >>> > acceptable to all.
> >>> >
> >>> > Regards,
> >>> > Greg
> >>> >
> >>> > On Wed, Apr 11, 2018 at 12:02 PM, Frank Brockners (fbrockne)
> >>> > <[e-mail]> wrote:
> >>> >>
> >>> >> Back at the IPPM meeting in London, we discussed several drafts
> >>> >> dealing with the encapsulation of IOAM data in various protocols
> >>> >> (draft-brockners-ippm-ioam-vxlan-gpe-00,
> >>> >> draft-brockners-ippm-ioam-geneve-00,
> >>> >> draft-weis-ippm-ioam-gre-00). One discussion topic that we
> >>> >> decided to take to the list was the question on whether
> >>> >> draft-ooamdt-rtgwg-ooam-header could be leveraged..  After
> >>> >> carefully considering draft-ooamdt-rtgwg-ooam-header, I came to
> >>> >> the conclusion that the “OOAM header” does not meet the needs of
> >>> >> IOAM:
> >>> >>
> >>> >> * Efficiency: IOAM adds data to live user traffic. As such, an
> >>> >> encapsulation needs to be as efficient as possible. The “OOAM header”
> >>> >> is 8
> >>> >> bytes long. The approach for IOAM data encapsulation in the above
> >>> >> mentioned drafts only requires 4 bytes. Using the OOAM header
> >>> >> approach would add an unnecessary overhead of 4 bytes – which is
> >>> >> significant.
> >>> Greg,
> >>>
> >>> I'm missing something here. I looked at the drafts you referenced
> >>> and each of them looks like the overhead for OAM is greater that
> >>> four bytes. In each there is some overhead equivalent to
> >>> type/length, for instance in Geneve four bytes are needed for option
> >>> class, type, and length. Unless the the OAM data is zero length, I
> >>> don't see how this adds up to only four bytes of overhead.
> >>
> >>
> >> The four versus eight bytes just refers to the fields in the four
> >> bytes of IOAM info, that is common to all IOAM options. Beyond that,
> >> there are IOAM option specific fields. For example if doing one of
> >> the IOAM trace options, there are four bytes of trace option header,
> >> including the IOAM-trace-type, NodeLen, Flags, and RemainingLen
> >> fields. These are followed by the node data list containing the per
> >> hop IOAM information.
> >>
> >> In looking at the OOAM header content, it has nothing to do with any
> >> of the IOAM information after the first four bytes. It contains
> >> another variant of the information in the first four bytes of IOAM
> >> info, spread out over eight bytes.
> >>
> >>>
> >>> Tom
> >>>
> >>> >
> >>> > GIM>> The difference in four octets is because OOAM Header:
> >>> >
> >>> > provides more flexibility, e.g. Flags field and Reserved fields;
> >>
> >>
> >> The flags field only has one defined flag at the moment, for a
> >> timestamp block. For IOAM trace we need per hop timestamps, which the
> >> timestamp block cannot address, i.e. the timestamp block is redundant for
> IOAM.
> >>
> >>>
> >>> > supports larger OAM packets than iOAM header;
> >>
> >>
> >> For IOAM purposes, 1020 octets is more than enough.
> >>
> >>>
> >>> > is future proof by supporting versioning (Version field).
> >>
> >>
> >> IMO, taking the first two bits of the IOAM-Type to define a Version
> >> field would be a good thing. This does not require adding four more
> >> bytes of overhead. 64 IOAM-Types is more than enough.
> >>
> >>>
> >>> >>
> >>> >> * Maturity: IOAM has several implementations, which were also
> >>> >> shown at recent IETF hackathons – and we’re expecting additional
> >>> >> implementations to be publicized soon. Interoperable
> >>> >> implementations need timely specifications. Despite the question
> >>> >> being asked, the recent thread on OOAM in the NVO3 list hasn’t
> >>> >> revealed any implementation of the OOAM header.
> >>> >> In
> >>> >> addition, the thread revealed that several fundamental questions
> >>> >> about the OOAM header are still open, such as whether or how
> >>> >> active OAM mechanisms within protocols such as Geneve would apply
> >>> >> to the OOAM header. This ultimately means that we won’t get to a
> >>> >> timely specification.
> >>> >
> >>> > GIM>> May I ask which encapsulations supported by the
> >>> > GIM>> implementations
> >>> > you
> >>> > refer to. Until very recently all iOAM proposals were to use
> >>> > meta-data TLV in, e.g. Geneve and NSH. And if these or some of
> >>> > these implementations already updated to the newly proposed iOAM
> >>> > shim, I don't see problem in making them use OOAM Header. Would
> >>> > you agree?
> >>> >
> >>> >>
> >>> >> * Scope: It isn’t entirely clear to which protocols the OOAM
> >>> >> header would ultimately apply to. The way the OOAM header is
> >>> >> defined, OOAM uses a 8-bit field for “Next Prot”, the next
> >>> >> protocol. Some protocols that IOAM data needs to be encapsulated
> >>> >> into use 16-bits for their next protocol code points. See e.g.
> >>> >> the GRE encapsulation – as specified in
> >>> >> draft-weis-ippm-ioam-gre-00.
> >>> >
> >>> > GIM>> The first paragraph of the Introduction section states:
> >>> >    New protocols that support overlay networks like VxLAN-GPE
> >>> >    [I-D.ietf-nvo3-vxlan-gpe], GUE [I-D.ietf-nvo3-gue], Geneve
> >>> >    [I-D.ietf-nvo3-geneve], BIER [I-D.ietf-bier-mpls-encapsulation],
> and
> >>> >    NSH [I-D.ietf-sfc-nsh] support multi-protocol payload, e.g.
> >>> >    Ethernet, IPv4/IPv6, and recognize Operations, Administration, and
> >>> >    Maintenance (OAM) as one of distinct types.  That ensures that
> >>> >    Overlay OAM (OOAM)packets are sharing fate with Overlay data packet
> >>> >    traversing the underlay.
> >>> > I'm updating the OOAM Header draft and along with cleaning nits
> >>> > will update reference to GUE. I think that the list and the
> >>> > statemnt are quite clear in identifying the scope of networks that
> >>> > may benefit from using not only common OOAM Header but common OOAM
> >>> > mechanisms, e.g. Echo Request/Reply.
> >>> >
> >>> >> With the above in mind, I’d suggest that the WG moves forward
> >>> >> with specific definitions for encapsulating IOAM data into
> >>> >> protocols – per the above mentioned drafts.
> >>> >>
> >>> >>
> >>> >>
> >>> >> Regards, Frank
> >>> >>
> >>> >>
> >>> >> _______________________________________________
> >>> >> ippm mailing list
> >>> >> [e-mail]
> >>> >> https://www.ietf.org/mailman/listinfo/ippm
> >>> >>
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Int-area mailing list
> >>> > [e-mail]
> >>> > https://www.ietf.org/mailman/listinfo/int-area
> >>> >
> >>>
> >>> _______________________________________________
> >>> ippm mailing list
> >>> [e-mail]
> >>> https://www.ietf.org/mailman/listinfo/ippm
> >>
> >>
> >
> >