Re: [ippm] Compare IOAM Trace Mode Header and PBT Mode Header RE: IPPM - IOAM Side Meeting @IETF 104

[Haoyu song <[e-mail]>]

Hi Mickey,

Please see my response inline.

From: Mickey Spiegel [mailto:[e-mail]]
Sent: Tuesday, March 26, 2019 3:40 PM
To: Haoyu song <[e-mail]>
Cc: Shwetha Bhandari (shwethab) <[e-mail]>; Frank Brockners (fbrockne) <[e-mail]>; Ramesh Sivakolundu (sramesh) <[e-mail]>; Tal Mizrahi <[e-mail]>; Barak Gafni <[e-mail]>; IETF IPPM WG <[e-mail]>
Subject: Re: [ippm] Compare IOAM Trace Mode Header and PBT Mode Header RE: IPPM - IOAM Side Meeting @IETF 104

Please see comments inline.

On Tue, Mar 26, 2019 at 6:07 AM Haoyu song <[e-mail]<mailto:[e-mail]>> wrote:
Dear all,

Per the side meeting discussion on IOAM last night, we decide to describe and compare the IOAM trace mode header and PBT header (updated to align with IOAM spec) to spur discussions within the WG, so we can decide if we should make PBT-I another mode of IOAM ( in addition to Incremental Trace, Pre-allocated Trace, E2E, and POT modes). If so, the new mode could be named PHP (Per Hop Postcard).

Thanks for putting this together. It makes it much easier to discuss commonalities and differences, so that we can decide whether this should be indicated by another IOAM-Type field value or a flag in one of the existing trace types.

IMO, there is no need for all these additional three letter acronyms and terminology such as PBT or PHP. I have also heard some objections to the word "immediate" in "immediate export". If we go with a new IOAM-Type field value, my suggestion would be "IOAM Trace with Direct Export Option". If we go with a flag, my suggestion would be just "Direct Export" or "Direct Reports". Anyway, we can decide on terminology later.

[HS] We do need a name to describe the header and feature. I don’t see a problem here. I talked with Barack and thought about a case for using “immediate export” in the trace mode: it change the router behavior whenever there’s no room for extra trace data: If this bit is not set, no data will be added and the end result is that you will only get a partial trace (kind of the currently defined behavior); if the bit is set, then a postcard will be generate to export the current trace data so room is made to accommodate new trace data. And PHP is another mode option for which each node must generate a postcard. Now the semantics for each option is clean.

Today we have some further discussion with Barak and below reflects the discussion results. I’ll update our PBT draft as well to reflect these considerations.

Please feel free to provide your thoughts and inputs. Thank you very much!

Best regards,
Haoyu

IOAM Trace Mode (Incremental and Pre-allocated) Header format:
    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        Namespace-ID           |NodeLen  | Flags | RemainingLen|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |               IOAM-Trace-Type                 |  Reserved     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+
   |                                                               |  |
   |                        node data list [0]                     |  |
   |                                                               |  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  D
   |                                                               |  a
   |                        node data list [1]                     |  t
   |                                                               |  a
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   ~                             ...                               ~  S
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  p
   |                                                               |  a
   |                        node data list [n-1]                   |  c
   |                                                               |  e
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
   |                                                               |  |
   |                        node data list [n]                     |  |
   |                                                               |  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+


PHP Header Format:

    0                   1                   2                   3

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |        Namespace-ID           | Extended Flags  | Hop Count   |

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |               IOAM-Trace-Type                 |  Reserved     |

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |                     Flow ID                                   |

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |                     Sequence Number                           |

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


Key Differences between PBT-I and IOAM Trace:

1)      PBT-I: No need for “NodeLen” field; bits saved for flags
Agree that there is no need for "NodeLen" in this case.

2)      PBT-I: No need for “immediate export” flag; mode identified through “mode” field before the header
We need to decide whether to indicate this case using a new IOAM-Type field value or a flag.

3)      PBT-I: No need for “remaining length” field, since no IOAM data will be included in user packet
Agree.

4)      PBT-I: No need for “Node Data List” for obvious reason.
Agree.

5)      PBT-I: Have a “flow ID” field which helps to identify and correlate a flow

a.      Avoid the need to truncate the original packet header

b.      Remove the flow parsing burden of the data collector and analyzer
While a "flow ID" may seem like a good idea at first glance, IMO it is not all that useful. Reiterating what I stated during the meeting:

  *   I don't believe in central assignment of "flow ID" because the granularity at which you configure what you are interested in monitoring (e.g. ACLs) differs from the granularity that the monitoring/analytics systems care about (typically 5-tuple). Typically such ACLs will not have individual entries for every 5-tuple, they will use masks such as /24 or /16.
  *   While "flow ID" may be useful for correlation of a particular packet as it traverses the network, with metadata being exported separately at each hop, it is not that useful for correlating different packets belonging to the same flow (e.g. 5-tuple).

     *   If you buy the argument in the first bullet, then "flow ID" will be assigned algorithmically at the encapsulating node. This has many implications, captured in the next few bullets.
     *   The "flow ID" does not remove the flow parsing burden because the monitoring/analytics systems just see "flow ID" as a number that at best assists with correlation. Flow identity, in the sense that flows are presented to users of the monitoring/analytics systems, is determined from the packet header fields. Since it is possible for packets containing exported metadata to be lost, this means that every packet containing exported metadata should contain sufficient packet header fields to identify the flow, in addition to the "flow ID".
     *   The encapsulating node may change during the lifetime of a flow. Trying to define a common "flow ID" assignment algorithm that could work across multiple different vendor implementations is futile, even for a simple encapsulation. These types of algorithms tend to be hardware specific. For more complex encapsulations, it would also be difficult to reach consensus on which fields should be looked at by the "flow ID" assignment algorithm. This means that the "flow ID" will change whenever the the encapsulating node changes, e.g. due to changes in the paths determined by routing protocols in the previous network.
     *   The monitoring/analytics system can simplify its operation by always relying on packet header fields to identify the flow, rather than bouncing back and forth between relying on packet header fields and relying on "flow ID". The latter is susceptible to corner cases that will significantly increase the complexity of the implementation.

6)      PBT-I: Have a “sequence number” field which helps to correlate postcards for a same user packet

a.      Without it,  it’s very difficult to correctly correlate postcards (see detailed discussion about this issue on PBT-M in the PBT draft)
If the system carrying out the correlation knows the network topology, it is possible to correlate without presence of a sequence number. We have an implementation that carries out such correlation based on knowledge of the network topology, and using the ingress and egress interface IDs present in the telemetry metadata.

[HS] The knowledge of network topology is a too strong requirement which doesn’t apply for carrier networks. So sequence number is necessary for the general case. Again, I have talked with Barak about the possibility to make both Flow ID and sequence number optional in case some user really keens to reduce the overhead and can do without them. Bust still, I believe in most cases, these two fields can make life much easier.

Any implementation relying on "sequence number" will have to deal with the case where the encapsulating node changes. Either the identity of the encapsulating node would have to be included in the packet, or some other approach would need to be taken such as randomizing the initial sequence number, in order to minimize the possibility of sequence number collisions.

[HS] I didn’t see the need to change the encapsulation nodes in my use cases. But if there is, the flowID and sequence number can change. I don’t see a big trouble here. Maybe I miss something here?

Even if we conclude that "sequence number" is useful, we would still have to decide whether it is necessary to put the "sequence number" directly in this "IOAM Trace Option", or whether it is good enough to include a separate "IOAM Edge-to-Edge Option" as is already defined in draft-ietf-ippm-ioam-data-05.

[HS] We have discussed about this. It’s improper to include E2E with Trace Option to just get the sequence number. That’s an abuse of the modes and doesn’t bring any benefit. A new mode is the right way to go.

In summary, I agree that there are some fields that are not applicable in this case. However, given the need for 4 byte alignment, defining a new IOAM-type field value does not save any bytes in the resulting data packets. Moreover, the use case for the proposed new fields such as "flow ID" and "sequence number" is not that strong. I do not yet see sufficient justification for a new IOAM-type field value.

Mickey


7)      PBT-I: Have an optional “hop count” field which helps to correlate and track the order of the postcards

a.      If unused, set to all “0”

b.      If used, start from 1, and every node increments it by 1.
Common fields

1)      Namespace-ID

2)      Flags

a.      More flag bits are available due to removing the “NodeLen” field

b.      No “immediate export” flag needed

3)      Trace Type

a.      Although we think a template ID is a better way to replace the bitmap-based trace type, ultimately different modes should be made consistent for this.

Barak suggests that Flow ID and Sequence Number can also be made optional by setting some flags, if user prefer to use other means to correlate packets. This subjects to further discussion.

_______________________________________________
ippm mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/ippm