Re: [nvo3] [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

["Frank Brockners (fbrockne)" <[e-mail]>]

Hi Tianran,

Tom's note already includes the hint: You'll add IOAM data to the protocol/layer that you're interested in monitoring. Again using Geneve over IPv6 as an example: 
* If you're interested in the overlay, i.e. Geneve (e.g. timestamping the packet when it enters and exists the tunnel) - you'd add IOAM data to Geneve
* If you're interested in the underlay, i.e. IPv6 (e.g. you'd like to understand which path packets take in the v6 network) - you'd add IOAM data to IPv6
* If you're interested in both, then you'd add IOAM data to Geneve and IPv6

Draft draft-ietf-ippm-ioam-data-02 already mentions layering (see section 3):
"Layering: If several encapsulation protocols (e.g., in case of tunneling) are stacked on top of each other, IOAM data-records could be present at every layer.  The behavior follows the ships-in-the-night model."

Given the discussion here, we'll add some additional text in the next revision to make things crisper (e.g. adding an example might help).

Frank

-----Original Message-----
From: Tianran Zhou <[e-mail]> 
Sent: Dienstag, 17. April 2018 03:18
To: Tom Herbert <[e-mail]>
Cc: Shwetha Bhandari (shwethab) <[e-mail]>; Frank Brockners (fbrockne) <[e-mail]>; Mickey Spiegel <[e-mail]>; NVO3 <[e-mail]>; Service Function Chaining IETF list <[e-mail]>; IETF IPPM WG <[e-mail]>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

I think it's better that Frank or Shwetha can explain the multi-layer use case in detail.

Tianran
> -----Original Message-----
> From: Tom Herbert [mailto:[e-mail]]
> Sent: Monday, April 16, 2018 10:40 PM
> To: Tianran Zhou <[e-mail]>
> Cc: Shwetha Bhandari (shwethab) <[e-mail]>; Frank Brockners
> (fbrockne) <[e-mail]>; Mickey Spiegel 
> <[e-mail]>; NVO3 <[e-mail]>; int-area 
> <[e-mail]>; Service Function Chaining IETF list 
> <[e-mail]>; IETF IPPM WG <[e-mail]>
> Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various 
> protocols - follow up from WG discussion in London
> 
> On Mon, Apr 16, 2018 at 6:31 AM, Tianran Zhou <[e-mail]> wrote:
> > Hi Shwetha,
> >
> > You are talking about the outer encapsution. It is straight forward 
> > for the underlay to record by the header. But what about the 
> > overlay, i.e., inner encapsulation(e.g. geneve)? Without special 
> > configuration, intermediate node will not read the inner header, 
> > hence not be able to process IOAM.e
> 
> Hi Tianran,
> 
> I believe that is also not protocol conformant. Intermediate nodes 
> should not be processing transport layer data as this can lead to 
> misinterpretation and possibly silent data corruption.
> 
> For instance, Geneve is a UDP encapsulation protocol with assigned port 6081.
> In order for an intermediate device to process the Geneve 
> encapsulation header it would need to look for packets with 
> destination port of 6081 since that is the only possible 
> discriminator. However, transport port numbers do not have global 
> meaning and hosts may use port numbers for other purposes (RFC7605 
> describes this). So a packet to port 6081 might be something other 
> than Geneve and may be misinterpreted. If a misinterpreted packet is changed (like ippm data is written) then that would be systematic silent data corruption.
> 
> As far as I know, hop-by-hop options is the only protocol confirming 
> mechanism that allows an intermediate note to change data of packet in flight.
> Encpasulation is the only conforming mechanism that allows an 
> intermediate node to add data (like extension headers) to a packet in flight.
> 
> Tom
> 
> > Maybe we are not synced by this overlay/underlay use case. :-)
> >
> > Tianran
> >
> >
> >
> > ________________________________
> > Sent from WeLink
> >
> > 发件人： Shwetha Bhandari (shwethab)
> > 收件人： Tianran Zhou<[e-mail]>;Frank Brockners 
> > (fbrockne)<[e-mail]>;Mickey
> > Spiegel<[e-mail]>;Tom
> > Herbert<[e-mail]>
> > 抄送： NVO3<[e-mail]>;int-area<[e-mail]>;Service Function 
> > Chaining IETF list<[e-mail]>;IETF IPPM WG<[e-mail]>
> > 主题： Re: [ippm] [Int-area] encapsulation of IOAM data in various 
> > protocols - follow up from WG discussion in London
> > 时间： 2018-04-16 18:17:01
> >
> > Hi Tianran,
> >
> >> If I recall right, it is not written in the ioam data draft.
> >
> > Data draft is defining the data to be carried in IOAM in an 
> > encapsulation agnostic way, it does not specify how the 
> > encapsulation protocol is configured.
> >
> >
> >
> >> Yes, node by node configuration is an easy way.
> >
> > While it is, it does not have to be a node by node configuration. It 
> > can be part of the encapsulation definition.
> >
> > For e.g. If the encapsulation is IPv6 and if we define the data to 
> > be carried as HbH options, then based on the Option Type with 
> > highest order 2 bits set to 00 then the v6 nodes that implement IOAM 
> > will process the option and others will skip over.
> >
> >
> >
> >
> >
> > Thanks,
> >
> > Shwetha
> >
> >
> >
> > From: ippm <[e-mail]> on behalf of Tianran Zhou 
> > <[e-mail]>
> > Date: Monday, April 16, 2018 at 2:36 PM
> > To: "Frank Brockners (fbrockne)" <[e-mail]>, Mickey 
> > Spiegel <[e-mail]>, Tom Herbert 
> > <[e-mail]>
> > Cc: NVO3 <[e-mail]>, "[e-mail]" <[e-mail]>, 
> > Service Function Chaining IETF list <[e-mail]>, IETF IPPM WG 
> > <[e-mail]>
> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various 
> > protocols - follow up from WG discussion in London
> >
> >
> >
> > Hi Frank,
> >
> >
> >
> > If I recall right, it is not written in the ioam data draft.
> >
> > Yes, node by node configuration is an easy way. In the 
> > draft-zhou-ippm-ioam-yang, we have the “protocol-type” to indicate 
> > the layering.
> >
> >    +--rw ioam
> >
> >       +--rw ioam-profiles
> >
> >          +--rw enabled?        boolean
> >
> >          +--rw ioam-profile* [profile-name]
> >
> >             +--rw profile-name                    string
> >
> >             +--rw filter
> >
> >             |  +--rw filter-type?   ioam-filter-type
> >
> >             |  +--rw acl-name?      -> /acl:acls/acl/name
> >
> >             +--rw protocol-type?                  ioam-protocol-type
> >
> >             +--rw incremental-tracing-profile {incremental-trace}?
> >
> >             |  ...
> >
> >             +--rw preallocated-tracing-profile {preallocated-trace}?
> >
> >             |  ...
> >
> >             +--rw pot-profile {proof-of-transit}?
> >
> >             |  ...
> >
> >             +--rw e2e-profile {edge-to-edge}?
> >
> >                ...
> >
> >
> >
> >
> >
> > Tianran
> >
> > From: Frank Brockners (fbrockne) [mailto:[e-mail]]
> > Sent: Monday, April 16, 2018 4:51 PM
> > To: Tianran Zhou <[e-mail]>; Mickey Spiegel 
> > <[e-mail]>; Tom Herbert <[e-mail]>
> > Cc: NVO3 <[e-mail]>; [e-mail]; Service Function 
> > Chaining IETF list <[e-mail]>; IETF IPPM WG <[e-mail]>
> > Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various 
> > protocols - follow up from WG discussion in London
> >
> >
> >
> > Hi Tianran,
> >
> >
> >
> > IOAM is a domain specific feature (see also
> > draft-ietf-ippm-ioam-data-02 sections 3 and 4), which allows an 
> > operator to control by means of configuration where and for which 
> > traffic IOAM data fields are added/updated/removed from the customer 
> > traffic. Using your example of Geneve over IPv6 – with IOAM data in 
> > both the Geneve and the IPv6 protocol, one would expect that the 
> > operator configures the endpoints of the Geneve tunnel to operate on 
> > the IOAM data in Geneve, and the IPv6 routers that the Geneve tunnel
> traverses to operate on the IOAM data in IPv6.
> >
> >
> >
> > Frank
> >
> >
> >
> > From: Tianran Zhou <[e-mail]>
> > Sent: Montag, 16. April 2018 10:37
> > To: Frank Brockners (fbrockne) <[e-mail]>; Mickey Spiegel 
> > <[e-mail]>; Tom Herbert <[e-mail]>
> > Cc: NVO3 <[e-mail]>; [e-mail]; Service Function 
> > Chaining IETF list <[e-mail]>; IETF IPPM WG <[e-mail]>
> > Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various 
> > protocols - follow up from WG discussion in London
> >
> >
> >
> > Hi Frank,
> >
> >
> >
> > How does a forwarder know when and where to insert the data?
> >
> > In the case of Geneve over IPv6, do you mean the device need to scan 
> > all the protocol stack? Or just the outer encapsulation?
> >
> >
> >
> > Tianran
> >
> >
> >
> > From: ippm [mailto:[e-mail]] On Behalf Of Frank 
> > Brockners
> > (fbrockne)
> > Sent: Monday, April 16, 2018 3:08 PM
> > To: Mickey Spiegel <[e-mail]>; Tom Herbert 
> > <[e-mail]>
> > Cc: NVO3 <[e-mail]>; [e-mail]; Service Function 
> > Chaining IETF list <[e-mail]>; IETF IPPM WG <[e-mail]>
> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various 
> > protocols - follow up from WG discussion in London
> >
> >
> >
> >
> >
> > Tom,
> >
> >
> >
> > a quick addition to what Mickey mentioned below: What you seem to 
> > have in mind is what draft-ietf-ippm-ioam-data-02 refers to as “layering”
> > (see section 3.), i.e. if you’re running for example Geneve over 
> > IPv6, then IOAM data could be encapsulated in both protocols, Geneve 
> > and
> > IPv6 – giving you visibility into the “underlay” (IPv6) and the “overlay”
> (Geneve).
> >
> >
> >
> > Frank
> >
> >
> >
> > From: ippm <[e-mail]> On Behalf Of Mickey Spiegel
> > Sent: Freitag, 13. April 2018 20:22
> > To: Tom Herbert <[e-mail]>
> > Cc: NVO3 <[e-mail]>; [e-mail]; Service Function 
> > Chaining IETF list <[e-mail]>; IETF IPPM WG <[e-mail]>
> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various 
> > protocols - follow up from WG discussion in London
> >
> >
> >
> > Tom,
> >
> >
> >
> > On Thu, Apr 12, 2018 at 10:17 PM, Tom Herbert <[e-mail]> wrote:
> >
> > Mickey,
> >
> > Looking at these ippm drafts more closely, I have a much more 
> > fundamental concern.
> >
> > In draft-brockners-ippm-ioam-geneve-00 for instance, there is the 
> > text in the introduction:
> >
> > "In-situ OAM (IOAM) records OAM information within the packet while 
> > the packet traverses a particular network domain.  The term "in-situ"
> > refers to the fact that the IOAM data fields are added to the data 
> > packets rather than is being sent within packets specifically 
> > dedicated to OAM.  This document defines how IOAM data fields are 
> > transported as part of the Geneve [I-D.ietf-nvo3-geneve] 
> > encapsulation."
> >
> > I assume this means that as packets with Geneve encapsulation 
> > traverse the network they are interpreted by intermediate nodes as 
> > being Geneve. Since Geneve is a UDP encapsulation, then the 
> > destination UDP port number would be used to identify packets as 
> > being Geneve. So an intermediate device might be looking for UDP 
> > packets destined to port
> > 6081 (the assigned UDP port for Geneve). If my understanding is 
> > correct, then this is a problem.
> >
> > UDP port numbers do not have global meaning. An intermediate device 
> > may very well see UDP packets destined to port 6081 that are not 
> > actually Geneve. This scenario is discussed in RFC7605:
> >
> > "...intermediate device interprets traffic based on the port number.
> > It is important to recognize that any interpretation of port numbers
> > -- except at the endpoints -- may be incorrect, because port numbers 
> > are meaningful only at the endpoints."
> >
> > If the UDP data is modified, as the draft would imply, then 
> > misinterpretation may also mean silent data corruption of packets. A 
> > protocol that would allow this seems pretty incorrect! Note that 
> > this would be true also for any UDP encapsulation that the network 
> > tries to interpret.
> >
> >
> >
> > The intention is to allow for multiple nodes that a packet traverses
> >
> > to be able to insert IOAM node information in the same trace option,
> >
> > but leave some flexibility regarding which nodes actually do the
> >
> > IOAM processing and the node information. This may vary
> >
> > depending on the transport.
> >
> >
> >
> > In case of a tunneled encapsulation such as Geneve or VXLAN,
> >
> > there may still be multiple hops. For example a network may use
> >
> > Geneve or VXLAN, but only do L2 processing at ToRs, with L3
> >
> > processing done at aggregation or core switches. In this case
> >
> > many packets would do 2 Geneve or VXLAN hops, so the packet
> >
> > would contain IOAM node information from two nodes.
> >
> >
> >
> > Another example is service function chaining using Geneve or
> >
> > VXLAN rather than NSH.
> >
> >
> >
> >
> > I am also wondering if hop-by-hop options been considered for this 
> > application? Their interpretation in the network is unabiguous and 
> > they also have the advantage that the work with any IP protocol or 
> > encapsulation.
> >
> >
> >
> > IPv6 hop-by-hop options has been considered. See
> >
> > draft-brockners-inband-oam-transport-05. This has not yet been
> >
> > broken out into a separate draft.
> >
> >
> >
> > Mickey
> >
> >
> >
> >
> > Thanks,
> > Tom
> >
> >
> > On Thu, Apr 12, 2018 at 3:31 PM, Mickey Spiegel 
> > <[e-mail]> wrote:
> >
> >> Tom,
> >>
> >> On Thu, Apr 12, 2018 at 2:46 PM, Tom Herbert <[e-mail]> wrote:
> >>>
> >>> On Thu, Apr 12, 2018 at 9:54 AM, Greg Mirsky 
> >>> <[e-mail]>
> >>> wrote:
> >>> > Hi Frank,
> >>> > thank you for sharing your points. Please find my notes in-line 
> >>> > and tagged
> >>> > GIM>>. I believe that this is very much relevant to work of 
> >>> > GIM>>other
> >>> > working
> >>> > groups that directly work on the overlay encapsulations in the 
> >>> > center of the discussion and hence I've added them to the list.
> >>> > Hope we'll have more opinions to reach the conclusion that is 
> >>> > acceptable to all.
> >>> >
> >>> > Regards,
> >>> > Greg
> >>> >
> >>> > On Wed, Apr 11, 2018 at 12:02 PM, Frank Brockners (fbrockne) 
> >>> > <[e-mail]> wrote:
> >>> >>
> >>> >> Back at the IPPM meeting in London, we discussed several drafts 
> >>> >> dealing with the encapsulation of IOAM data in various 
> >>> >> protocols (draft-brockners-ippm-ioam-vxlan-gpe-00,
> >>> >> draft-brockners-ippm-ioam-geneve-00,
> >>> >> draft-weis-ippm-ioam-gre-00). One discussion topic that we 
> >>> >> decided to take to the list was the question on whether 
> >>> >> draft-ooamdt-rtgwg-ooam-header could be leveraged..  After 
> >>> >> carefully considering draft-ooamdt-rtgwg-ooam-header, I came to 
> >>> >> the conclusion that the “OOAM header” does not meet the needs 
> >>> >> of
> >>> >> IOAM:
> >>> >>
> >>> >> * Efficiency: IOAM adds data to live user traffic. As such, an 
> >>> >> encapsulation needs to be as efficient as possible. The “OOAM header”
> >>> >> is 8
> >>> >> bytes long. The approach for IOAM data encapsulation in the 
> >>> >> above mentioned drafts only requires 4 bytes. Using the OOAM 
> >>> >> header approach would add an unnecessary overhead of 4 bytes – 
> >>> >> which is significant.
> >>> Greg,
> >>>
> >>> I'm missing something here. I looked at the drafts you referenced 
> >>> and each of them looks like the overhead for OAM is greater that 
> >>> four bytes. In each there is some overhead equivalent to 
> >>> type/length, for instance in Geneve four bytes are needed for 
> >>> option class, type, and length. Unless the the OAM data is zero 
> >>> length, I don't see how this adds up to only four bytes of overhead.
> >>
> >>
> >> The four versus eight bytes just refers to the fields in the four 
> >> bytes of IOAM info, that is common to all IOAM options. Beyond 
> >> that, there are IOAM option specific fields. For example if doing 
> >> one of the IOAM trace options, there are four bytes of trace option 
> >> header, including the IOAM-trace-type, NodeLen, Flags, and 
> >> RemainingLen fields. These are followed by the node data list 
> >> containing the per hop IOAM information.
> >>
> >> In looking at the OOAM header content, it has nothing to do with 
> >> any of the IOAM information after the first four bytes. It contains 
> >> another variant of the information in the first four bytes of IOAM 
> >> info, spread out over eight bytes.
> >>
> >>>
> >>> Tom
> >>>
> >>> >
> >>> > GIM>> The difference in four octets is because OOAM Header:
> >>> >
> >>> > provides more flexibility, e.g. Flags field and Reserved fields;
> >>
> >>
> >> The flags field only has one defined flag at the moment, for a 
> >> timestamp block. For IOAM trace we need per hop timestamps, which 
> >> the timestamp block cannot address, i.e. the timestamp block is 
> >> redundant for
> IOAM.
> >>
> >>>
> >>> > supports larger OAM packets than iOAM header;
> >>
> >>
> >> For IOAM purposes, 1020 octets is more than enough.
> >>
> >>>
> >>> > is future proof by supporting versioning (Version field).
> >>
> >>
> >> IMO, taking the first two bits of the IOAM-Type to define a Version 
> >> field would be a good thing. This does not require adding four more 
> >> bytes of overhead. 64 IOAM-Types is more than enough.
> >>
> >>>
> >>> >>
> >>> >> * Maturity: IOAM has several implementations, which were also 
> >>> >> shown at recent IETF hackathons – and we’re expecting 
> >>> >> additional implementations to be publicized soon. Interoperable 
> >>> >> implementations need timely specifications. Despite the 
> >>> >> question being asked, the recent thread on OOAM in the NVO3 
> >>> >> list hasn’t revealed any implementation of the OOAM header.
> >>> >> In
> >>> >> addition, the thread revealed that several fundamental 
> >>> >> questions about the OOAM header are still open, such as whether 
> >>> >> or how active OAM mechanisms within protocols such as Geneve 
> >>> >> would apply to the OOAM header. This ultimately means that we 
> >>> >> won’t get to a timely specification.
> >>> >
> >>> > GIM>> May I ask which encapsulations supported by the 
> >>> > GIM>> implementations
> >>> > you
> >>> > refer to. Until very recently all iOAM proposals were to use 
> >>> > meta-data TLV in, e.g. Geneve and NSH. And if these or some of 
> >>> > these implementations already updated to the newly proposed iOAM 
> >>> > shim, I don't see problem in making them use OOAM Header. Would 
> >>> > you agree?
> >>> >
> >>> >>
> >>> >> * Scope: It isn’t entirely clear to which protocols the OOAM 
> >>> >> header would ultimately apply to. The way the OOAM header is 
> >>> >> defined, OOAM uses a 8-bit field for “Next Prot”, the next 
> >>> >> protocol. Some protocols that IOAM data needs to be 
> >>> >> encapsulated into use 16-bits for their next protocol code points. See e.g.
> >>> >> the GRE encapsulation – as specified in 
> >>> >> draft-weis-ippm-ioam-gre-00.
> >>> >
> >>> > GIM>> The first paragraph of the Introduction section states:
> >>> >    New protocols that support overlay networks like VxLAN-GPE
> >>> >    [I-D.ietf-nvo3-vxlan-gpe], GUE [I-D.ietf-nvo3-gue], Geneve
> >>> >    [I-D.ietf-nvo3-geneve], BIER 
> >>> > [I-D.ietf-bier-mpls-encapsulation],
> and
> >>> >    NSH [I-D.ietf-sfc-nsh] support multi-protocol payload, e.g.
> >>> >    Ethernet, IPv4/IPv6, and recognize Operations, Administration, and
> >>> >    Maintenance (OAM) as one of distinct types.  That ensures that
> >>> >    Overlay OAM (OOAM)packets are sharing fate with Overlay data packet
> >>> >    traversing the underlay.
> >>> > I'm updating the OOAM Header draft and along with cleaning nits 
> >>> > will update reference to GUE. I think that the list and the 
> >>> > statemnt are quite clear in identifying the scope of networks 
> >>> > that may benefit from using not only common OOAM Header but 
> >>> > common OOAM mechanisms, e.g. Echo Request/Reply.
> >>> >
> >>> >> With the above in mind, I’d suggest that the WG moves forward 
> >>> >> with specific definitions for encapsulating IOAM data into 
> >>> >> protocols – per the above mentioned drafts.
> >>> >>
> >>> >>
> >>> >>
> >>> >> Regards, Frank
> >>> >>
> >>> >>
> >>> >> _______________________________________________
> >>> >> ippm mailing list
> >>> >> [e-mail]
> >>> >> https://www.ietf.org/mailman/listinfo/ippm
> >>> >>
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Int-area mailing list
> >>> > [e-mail]
> >>> > https://www.ietf.org/mailman/listinfo/int-area
> >>> >
> >>>
> >>> _______________________________________________
> >>> ippm mailing list
> >>> [e-mail]
> >>> https://www.ietf.org/mailman/listinfo/ippm
> >>
> >>
> >
> >