Re: [nvo3] [sfc] [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

["Carlos Pignataro (cpignata)" <[e-mail]>]

Tom,

On Apr 17, 2018, at 10:22 AM, Tom Herbert <[e-mail]<mailto:[e-mail]>> wrote:

On Tue, Apr 17, 2018 at 12:51 AM, Frank Brockners (fbrockne)
<[e-mail]<mailto:[e-mail]>> wrote:

Hi Tianran,

Tom's note already includes the hint: You'll add IOAM data to the protocol/layer that you're interested in monitoring. Again using Geneve over IPv6 as an example:
* If you're interested in the overlay, i.e. Geneve (e.g. timestamping the packet when it enters and exists the tunnel) - you'd add IOAM data to Geneve
* If you're interested in the underlay, i.e. IPv6 (e.g. you'd like to understand which path packets take in the v6 network) - you'd add IOAM data to IPv6
* If you're interested in both, then you'd add IOAM data to Geneve and IPv6

Frank,

In that case why not just use a hop-by-hop option for measuring the
underlay and a destination option for measuring the overlay? The
advantage is that this works _any_ IP encapsulation method or any IP
protocol for that matter.

Because you want to instrument the layer that you want to measure.
Because there’s cases with more unnatural layering where there’s a desire to correlate and compare measurements across layers (in a way in which, for example, the Service layer is tested in a service chaining scenario, not the IPv6 hop-by-hop.
Because different topologies expose different Hops and IPv6 HBH goes by the IPv6 node topology.
Because not everything is IPv6, and because you have cases of IPv6 over something as well.
Those are quick ones that come to mind.

I don't believe adding ippm to every
encapsulation protocol is straightforward: e.g.
draft-brockners-ippm-ioam-geneve describe but notes the limited size
of header, draft-weis-ippm-ioam-gre states that a new EtherType would
be needed just for this purpose. This also entails additional
encapsulation-specific HW support also, whereas support destination
and hbh options could be more generic.


Engineering is about trade-offs. If you want to measure Geneve, there are limitations. But instead of trying to prove why it does not work, I’ll point to working demos of where it does — many of which on different HW/SW and encaps, shown at various IETF events.

Thanks,

— Carlos Pignataro

Tom

Draft draft-ietf-ippm-ioam-data-02 already mentions layering (see section 3):
"Layering: If several encapsulation protocols (e.g., in case of tunneling) are stacked on top of each other, IOAM data-records could be present at every layer.  The behavior follows the ships-in-the-night model."

Given the discussion here, we'll add some additional text in the next revision to make things crisper (e.g. adding an example might help).

Frank

-----Original Message-----
From: Tianran Zhou <[e-mail]<mailto:[e-mail]>>
Sent: Dienstag, 17. April 2018 03:18
To: Tom Herbert <[e-mail]<mailto:[e-mail]>>
Cc: Shwetha Bhandari (shwethab) <[e-mail]<mailto:[e-mail]>>; Frank Brockners (fbrockne) <[e-mail]<mailto:[e-mail]>>; Mickey Spiegel <[e-mail]<mailto:[e-mail]>>; NVO3 <[e-mail]<mailto:[e-mail]>>; Service Function Chaining IETF list <[e-mail]<mailto:[e-mail]>>; IETF IPPM WG <[e-mail]<mailto:[e-mail]>>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

I think it's better that Frank or Shwetha can explain the multi-layer use case in detail.

Tianran
-----Original Message-----
From: Tom Herbert [mailto:[e-mail]]
Sent: Monday, April 16, 2018 10:40 PM
To: Tianran Zhou <[e-mail]<mailto:[e-mail]>>
Cc: Shwetha Bhandari (shwethab) <[e-mail]<mailto:[e-mail]>>; Frank Brockners
(fbrockne) <[e-mail]<mailto:[e-mail]>>; Mickey Spiegel
<[e-mail]<mailto:[e-mail]>>; NVO3 <[e-mail]<mailto:[e-mail]>>; int-area
<[e-mail]<mailto:[e-mail]>>; Service Function Chaining IETF list
<[e-mail]<mailto:[e-mail]>>; IETF IPPM WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London

On Mon, Apr 16, 2018 at 6:31 AM, Tianran Zhou <[e-mail]<mailto:[e-mail]>> wrote:
Hi Shwetha,

You are talking about the outer encapsution. It is straight forward
for the underlay to record by the header. But what about the
overlay, i.e., inner encapsulation(e.g. geneve)? Without special
configuration, intermediate node will not read the inner header,
hence not be able to process IOAM.e

Hi Tianran,

I believe that is also not protocol conformant. Intermediate nodes
should not be processing transport layer data as this can lead to
misinterpretation and possibly silent data corruption.

For instance, Geneve is a UDP encapsulation protocol with assigned port 6081.
In order for an intermediate device to process the Geneve
encapsulation header it would need to look for packets with
destination port of 6081 since that is the only possible
discriminator. However, transport port numbers do not have global
meaning and hosts may use port numbers for other purposes (RFC7605
describes this). So a packet to port 6081 might be something other
than Geneve and may be misinterpreted. If a misinterpreted packet is changed (like ippm data is written) then that would be systematic silent data corruption.

As far as I know, hop-by-hop options is the only protocol confirming
mechanism that allows an intermediate note to change data of packet in flight.
Encpasulation is the only conforming mechanism that allows an
intermediate node to add data (like extension headers) to a packet in flight.

Tom

Maybe we are not synced by this overlay/underlay use case. :-)

Tianran



________________________________
Sent from WeLink

发件人： Shwetha Bhandari (shwethab)
收件人： Tianran Zhou<[e-mail]<mailto:[e-mail]>>;Frank Brockners
(fbrockne)<[e-mail]<mailto:[e-mail]>>;Mickey
Spiegel<[e-mail]<mailto:[e-mail]>>;Tom
Herbert<[e-mail]<mailto:[e-mail]>>
抄送： NVO3<[e-mail]<mailto:[e-mail]>>;int-area<[e-mail]<mailto:[e-mail]>>;Service Function
Chaining IETF list<[e-mail]<mailto:[e-mail]>>;IETF IPPM WG<[e-mail]<mailto:[e-mail]>>
主题： Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London
时间： 2018-04-16 18:17:01

Hi Tianran,

If I recall right, it is not written in the ioam data draft.

Data draft is defining the data to be carried in IOAM in an
encapsulation agnostic way, it does not specify how the
encapsulation protocol is configured.



Yes, node by node configuration is an easy way.

While it is, it does not have to be a node by node configuration. It
can be part of the encapsulation definition.

For e.g. If the encapsulation is IPv6 and if we define the data to
be carried as HbH options, then based on the Option Type with
highest order 2 bits set to 00 then the v6 nodes that implement IOAM
will process the option and others will skip over.





Thanks,

Shwetha



From: ippm <[e-mail]<mailto:[e-mail]>> on behalf of Tianran Zhou
<[e-mail]<mailto:[e-mail]>>
Date: Monday, April 16, 2018 at 2:36 PM
To: "Frank Brockners (fbrockne)" <[e-mail]<mailto:[e-mail]>>, Mickey
Spiegel <[e-mail]<mailto:[e-mail]>>, Tom Herbert
<[e-mail]<mailto:[e-mail]>>
Cc: NVO3 <[e-mail]<mailto:[e-mail]>>, "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>,
Service Function Chaining IETF list <[e-mail]<mailto:[e-mail]>>, IETF IPPM WG
<[e-mail]<mailto:[e-mail]>>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London



Hi Frank,



If I recall right, it is not written in the ioam data draft.

Yes, node by node configuration is an easy way. In the
draft-zhou-ippm-ioam-yang, we have the “protocol-type” to indicate
the layering.

  +--rw ioam

     +--rw ioam-profiles

        +--rw enabled?        boolean

        +--rw ioam-profile* [profile-name]

           +--rw profile-name                    string

           +--rw filter

           |  +--rw filter-type?   ioam-filter-type

           |  +--rw acl-name?      -> /acl:acls/acl/name

           +--rw protocol-type?                  ioam-protocol-type

           +--rw incremental-tracing-profile {incremental-trace}?

           |  ...

           +--rw preallocated-tracing-profile {preallocated-trace}?

           |  ...

           +--rw pot-profile {proof-of-transit}?

           |  ...

           +--rw e2e-profile {edge-to-edge}?

              ...





Tianran

From: Frank Brockners (fbrockne) [mailto:[e-mail]]
Sent: Monday, April 16, 2018 4:51 PM
To: Tianran Zhou <[e-mail]<mailto:[e-mail]>>; Mickey Spiegel
<[e-mail]<mailto:[e-mail]>>; Tom Herbert <[e-mail]<mailto:[e-mail]>>
Cc: NVO3 <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>; Service Function
Chaining IETF list <[e-mail]<mailto:[e-mail]>>; IETF IPPM WG <[e-mail]<mailto:[e-mail]>>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London



Hi Tianran,



IOAM is a domain specific feature (see also
draft-ietf-ippm-ioam-data-02 sections 3 and 4), which allows an
operator to control by means of configuration where and for which
traffic IOAM data fields are added/updated/removed from the customer
traffic. Using your example of Geneve over IPv6 – with IOAM data in
both the Geneve and the IPv6 protocol, one would expect that the
operator configures the endpoints of the Geneve tunnel to operate on
the IOAM data in Geneve, and the IPv6 routers that the Geneve tunnel
traverses to operate on the IOAM data in IPv6.



Frank



From: Tianran Zhou <[e-mail]<mailto:[e-mail]>>
Sent: Montag, 16. April 2018 10:37
To: Frank Brockners (fbrockne) <[e-mail]<mailto:[e-mail]>>; Mickey Spiegel
<[e-mail]<mailto:[e-mail]>>; Tom Herbert <[e-mail]<mailto:[e-mail]>>
Cc: NVO3 <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>; Service Function
Chaining IETF list <[e-mail]<mailto:[e-mail]>>; IETF IPPM WG <[e-mail]<mailto:[e-mail]>>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London



Hi Frank,



How does a forwarder know when and where to insert the data?

In the case of Geneve over IPv6, do you mean the device need to scan
all the protocol stack? Or just the outer encapsulation?



Tianran



From: ippm [mailto:[e-mail]] On Behalf Of Frank
Brockners
(fbrockne)
Sent: Monday, April 16, 2018 3:08 PM
To: Mickey Spiegel <[e-mail]<mailto:[e-mail]>>; Tom Herbert
<[e-mail]<mailto:[e-mail]>>
Cc: NVO3 <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>; Service Function
Chaining IETF list <[e-mail]<mailto:[e-mail]>>; IETF IPPM WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London





Tom,



a quick addition to what Mickey mentioned below: What you seem to
have in mind is what draft-ietf-ippm-ioam-data-02 refers to as “layering”
(see section 3.), i.e. if you’re running for example Geneve over
IPv6, then IOAM data could be encapsulated in both protocols, Geneve
and
IPv6 – giving you visibility into the “underlay” (IPv6) and the “overlay”
(Geneve).



Frank



From: ippm <[e-mail]<mailto:[e-mail]>> On Behalf Of Mickey Spiegel
Sent: Freitag, 13. April 2018 20:22
To: Tom Herbert <[e-mail]<mailto:[e-mail]>>
Cc: NVO3 <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>; Service Function
Chaining IETF list <[e-mail]<mailto:[e-mail]>>; IETF IPPM WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London



Tom,



On Thu, Apr 12, 2018 at 10:17 PM, Tom Herbert <[e-mail]<mailto:[e-mail]>> wrote:

Mickey,

Looking at these ippm drafts more closely, I have a much more
fundamental concern.

In draft-brockners-ippm-ioam-geneve-00 for instance, there is the
text in the introduction:

"In-situ OAM (IOAM) records OAM information within the packet while
the packet traverses a particular network domain.  The term "in-situ"
refers to the fact that the IOAM data fields are added to the data
packets rather than is being sent within packets specifically
dedicated to OAM.  This document defines how IOAM data fields are
transported as part of the Geneve [I-D.ietf-nvo3-geneve]
encapsulation."

I assume this means that as packets with Geneve encapsulation
traverse the network they are interpreted by intermediate nodes as
being Geneve. Since Geneve is a UDP encapsulation, then the
destination UDP port number would be used to identify packets as
being Geneve. So an intermediate device might be looking for UDP
packets destined to port
6081 (the assigned UDP port for Geneve). If my understanding is
correct, then this is a problem.

UDP port numbers do not have global meaning. An intermediate device
may very well see UDP packets destined to port 6081 that are not
actually Geneve. This scenario is discussed in RFC7605:

"...intermediate device interprets traffic based on the port number.
It is important to recognize that any interpretation of port numbers
-- except at the endpoints -- may be incorrect, because port numbers
are meaningful only at the endpoints."

If the UDP data is modified, as the draft would imply, then
misinterpretation may also mean silent data corruption of packets. A
protocol that would allow this seems pretty incorrect! Note that
this would be true also for any UDP encapsulation that the network
tries to interpret.



The intention is to allow for multiple nodes that a packet traverses

to be able to insert IOAM node information in the same trace option,

but leave some flexibility regarding which nodes actually do the

IOAM processing and the node information. This may vary

depending on the transport.



In case of a tunneled encapsulation such as Geneve or VXLAN,

there may still be multiple hops. For example a network may use

Geneve or VXLAN, but only do L2 processing at ToRs, with L3

processing done at aggregation or core switches. In this case

many packets would do 2 Geneve or VXLAN hops, so the packet

would contain IOAM node information from two nodes.



Another example is service function chaining using Geneve or

VXLAN rather than NSH.




I am also wondering if hop-by-hop options been considered for this
application? Their interpretation in the network is unabiguous and
they also have the advantage that the work with any IP protocol or
encapsulation.



IPv6 hop-by-hop options has been considered. See

draft-brockners-inband-oam-transport-05. This has not yet been

broken out into a separate draft.



Mickey




Thanks,
Tom


On Thu, Apr 12, 2018 at 3:31 PM, Mickey Spiegel
<[e-mail]<mailto:[e-mail]>> wrote:

Tom,

On Thu, Apr 12, 2018 at 2:46 PM, Tom Herbert <[e-mail]<mailto:[e-mail]>> wrote:

On Thu, Apr 12, 2018 at 9:54 AM, Greg Mirsky
<[e-mail]<mailto:[e-mail]>>
wrote:
Hi Frank,
thank you for sharing your points. Please find my notes in-line
and tagged
GIM>>. I believe that this is very much relevant to work of
GIM>>other
working
groups that directly work on the overlay encapsulations in the
center of the discussion and hence I've added them to the list.
Hope we'll have more opinions to reach the conclusion that is
acceptable to all.

Regards,
Greg

On Wed, Apr 11, 2018 at 12:02 PM, Frank Brockners (fbrockne)
<[e-mail]<mailto:[e-mail]>> wrote:

Back at the IPPM meeting in London, we discussed several drafts
dealing with the encapsulation of IOAM data in various
protocols (draft-brockners-ippm-ioam-vxlan-gpe-00,
draft-brockners-ippm-ioam-geneve-00,
draft-weis-ippm-ioam-gre-00). One discussion topic that we
decided to take to the list was the question on whether
draft-ooamdt-rtgwg-ooam-header could be leveraged..  After
carefully considering draft-ooamdt-rtgwg-ooam-header, I came to
the conclusion that the “OOAM header” does not meet the needs
of
IOAM:

* Efficiency: IOAM adds data to live user traffic. As such, an
encapsulation needs to be as efficient as possible. The “OOAM header”
is 8
bytes long. The approach for IOAM data encapsulation in the
above mentioned drafts only requires 4 bytes. Using the OOAM
header approach would add an unnecessary overhead of 4 bytes –
which is significant.
Greg,

I'm missing something here. I looked at the drafts you referenced
and each of them looks like the overhead for OAM is greater that
four bytes. In each there is some overhead equivalent to
type/length, for instance in Geneve four bytes are needed for
option class, type, and length. Unless the the OAM data is zero
length, I don't see how this adds up to only four bytes of overhead.


The four versus eight bytes just refers to the fields in the four
bytes of IOAM info, that is common to all IOAM options. Beyond
that, there are IOAM option specific fields. For example if doing
one of the IOAM trace options, there are four bytes of trace option
header, including the IOAM-trace-type, NodeLen, Flags, and
RemainingLen fields. These are followed by the node data list
containing the per hop IOAM information.

In looking at the OOAM header content, it has nothing to do with
any of the IOAM information after the first four bytes. It contains
another variant of the information in the first four bytes of IOAM
info, spread out over eight bytes.


Tom


GIM>> The difference in four octets is because OOAM Header:

provides more flexibility, e.g. Flags field and Reserved fields;


The flags field only has one defined flag at the moment, for a
timestamp block. For IOAM trace we need per hop timestamps, which
the timestamp block cannot address, i.e. the timestamp block is
redundant for
IOAM.


supports larger OAM packets than iOAM header;


For IOAM purposes, 1020 octets is more than enough.


is future proof by supporting versioning (Version field).


IMO, taking the first two bits of the IOAM-Type to define a Version
field would be a good thing. This does not require adding four more
bytes of overhead. 64 IOAM-Types is more than enough.



* Maturity: IOAM has several implementations, which were also
shown at recent IETF hackathons – and we’re expecting
additional implementations to be publicized soon. Interoperable
implementations need timely specifications. Despite the
question being asked, the recent thread on OOAM in the NVO3
list hasn’t revealed any implementation of the OOAM header.
In
addition, the thread revealed that several fundamental
questions about the OOAM header are still open, such as whether
or how active OAM mechanisms within protocols such as Geneve
would apply to the OOAM header. This ultimately means that we
won’t get to a timely specification.

GIM>> May I ask which encapsulations supported by the
GIM>> implementations
you
refer to. Until very recently all iOAM proposals were to use
meta-data TLV in, e.g. Geneve and NSH. And if these or some of
these implementations already updated to the newly proposed iOAM
shim, I don't see problem in making them use OOAM Header. Would
you agree?


* Scope: It isn’t entirely clear to which protocols the OOAM
header would ultimately apply to. The way the OOAM header is
defined, OOAM uses a 8-bit field for “Next Prot”, the next
protocol. Some protocols that IOAM data needs to be
encapsulated into use 16-bits for their next protocol code points. See e.g.
the GRE encapsulation – as specified in
draft-weis-ippm-ioam-gre-00.

GIM>> The first paragraph of the Introduction section states:
  New protocols that support overlay networks like VxLAN-GPE
  [I-D.ietf-nvo3-vxlan-gpe], GUE [I-D.ietf-nvo3-gue], Geneve
  [I-D.ietf-nvo3-geneve], BIER
[I-D.ietf-bier-mpls-encapsulation],
and
  NSH [I-D.ietf-sfc-nsh] support multi-protocol payload, e.g.
  Ethernet, IPv4/IPv6, and recognize Operations, Administration, and
  Maintenance (OAM) as one of distinct types.  That ensures that
  Overlay OAM (OOAM)packets are sharing fate with Overlay data packet
  traversing the underlay.
I'm updating the OOAM Header draft and along with cleaning nits
will update reference to GUE. I think that the list and the
statemnt are quite clear in identifying the scope of networks
that may benefit from using not only common OOAM Header but
common OOAM mechanisms, e.g. Echo Request/Reply.

With the above in mind, I’d suggest that the WG moves forward
with specific definitions for encapsulating IOAM data into
protocols – per the above mentioned drafts.



Regards, Frank


_______________________________________________
ippm mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/ippm



_______________________________________________
Int-area mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/int-area


_______________________________________________
ippm mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/ippm





_______________________________________________
sfc mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/sfc