Re: [ippm] Lars Eggert's Discuss on draft-ietf-ippm-ioam-data-12: (with DISCUSS and COMMENT)

["Frank Brockners (fbrockne)" <[e-mail]>]

Hi Lars,

> -----Original Message-----
> From: Lars Eggert <[e-mail]>
> Sent: Montag, 12. April 2021 08:59
> To: Frank Brockners (fbrockne) <[e-mail]>
> Cc: The IESG <[e-mail]>; [e-mail]; ippm-
> [e-mail]; [e-mail]; Al Morton <[e-mail]>
> Subject: Re: Lars Eggert's Discuss on draft-ietf-ippm-ioam-data-12: (with
> DISCUSS and COMMENT)
> 
> Hi,
> 
> On 2021-4-11, at 19:46, Frank Brockners (fbrockne) <[e-mail]>
> wrote:
> >> Not requiring at least one mandatory-to-implement trace option type
> >> is highly problematic, since it creates two incompatible flavors of this
> standard.
> >> Preventing bifurcation seems to trump the desire for allowing
> >> (minor?) optimizations.
> >
> > ...FB: There seems to be some confusion and potential misunderstanding here:
> The two different Trace Option-Types would not lead to two incompatible
> flavors of the standard:
> > Each Trace Option-Type fills in the exact same IOAM data-fields - i.e. " node
> data list [i]" looks the exact same whether pre-allocated or incremental trace
> option types are used.
> 
> I was with you until here, thinking I had misunderstood the draft.
> 
> > Each Trace Option-Type just handles a different “bucket” in the packet.
> 
> But then this sentence confused me again.
> 
> This is what I am trying to understand: Let's assume IOM data gets added to a
> packet by an implementation using the pre-allocated format. It now gets
> forwarded to another node with an implementation using the incremental
> format. Can that second implementation, access, parse and update the IOAM
> information in the packet? Is the reverse also possible (first stack is using
> incremental format, second stack is using preallocated format?)
> 
> If the answer is yes to both of these, I have misunderstood the spec, and my
> discuss is moot.
> 
> If not, then I don't see how a deployment can mix hardware- and software-
> based forwarders?

... FB: Perhaps a picture can help - see attached. The example shows a 5 node setup:
* Node A is the encap node; node E is the decap node; nodes B,C,D are transit nodes. The node data list [i] array is abbreviated with ndl[i].
* We have a setup with a mix of software (SW - using preallocated trace Option-Type) and hardware (HW - using incremental trace Option-Type) forwarders.
* The encap node inserts trace option types for both flavors (pre-allocated and incremental). For pre-allocated, we make room for an array of three hops. 
* The picture gives a view of the packet at 3 tap points:
1) After nodes A and B: You see the two buckets, one for incremental-tracing, one for pre-allocated tracing. In both buckets, there one field of the node data list filled.
2) After node C, which is a software forwarder, an additional field in the node data list of the pre-allocated tracing bucket got filled.
3) After node D, which is a hardware forwarder, an additional field in the node data list of the incremental tracing bucket got allocated and filled.
* Node E would simply strip the IOAM information from the packet and forward it to some controller/collector. draft-spiegel-ippm-ioam-rawexport is one example of how to handle export using IPFIX. Note that node E would not do any interpretation of the data. It just strips it from the packet and ships it.

> 
> > And each Trace Option-Type caters for a different type of transit node:
> Software-based forwarders and Hardware-based forwarders. The differences
> between the two are not negligible. Memcopy is expensive for a software
> forwarder, as is a pointer-lookup in the packet for a hardware forwarder. The
> fact that there are two Option-Types that cater for different types of transit
> nodes is the result of long discussions. The two different ways to carry the data-
> fields is to ensure that IOAM can be deployed on a wide variety of devices. I'm
> afraid, that if we'd drop one variant, or make one variant mandatory, all we'd
> get is a slowed down adoption of the standard, because we would restrict IOAM
> tracing to either software forwarders or hardware forwarders.
> 
> I fully understand that at the moment, there is a performance difference. If this
> technology actually takes off, I believe that both hardware and software
> techniques will be developed that will mitigate those performance differences.
> This has played out time and time again for different standars.

...FB: I would really wish you were right. The issue is that we all have the IPv6 extension header example in mind, which started to put pointers into packets, requiring pointer lookup to properly parse extension headers... You know how long it took the industry to come up with basic support for HbyH or routing extension headers in hardware. We're still in the middle of this journey.
This is pretty much what drove the design for two different types of "buckets" - so that forwarders can insert the information into the bucket that suits them.

> 
> > That said, one thing that we should underline in the document is the fact that
> encap and decap nodes SHOULD be capable of inserting and removing both
> Trace-Option types. That way, there won’t be any compatibility challenge at all.
> Transit nodes of different type (HW or SW) would write the information into
> "their" type of bucket. Extracting the information at the decap node would be
> common in any case.
> 
> You'd need to mandate that both must be inserted into the packets, and on
> decap, you'd need to specify how an implementation is supposed to merge the
> information contained in the two different option types to construct
> information about what happened on the path.

...FB: Agree that we'd need to add a statement that encap/decap nodes SHOULD be able to insert/remove both trace Option-Types, because on encap/decap the effort between forwarders is comparable.
Per my note above, there is no need for the decap node to do any type of consolidation. It would just strip the different option types, and there could be additional ones, like POT or E2E Option-Types and ship the information to some controller/management station for further processing. 

> 
> >
> >>
> >> ---------------------------------------------------------------------
> >> -
> >> COMMENT:
> >> ---------------------------------------------------------------------
> >> -
> >>
> >> Section 1, paragraph 4, comment:
> >>>   IOAM use cases and mechanisms have expanded as this document
> matured,
> >>>   resulting in additional flags and options that could trigger creation
> >>>   of additional packets dedicated to OAM.  The term IOAM continues to
> >>>   be used for such mechanisms, in addition to the "in-situ" mechanisms
> >>>   that motivated this terminology.
> >>
> >> Suggest to rephrase this expanded view on IAM in a way that does not
> >> tie the description to the time period during which this
> >> soon-to-be-archival document was edited.
> >
> > ...FB: Thanks. Good suggestion - which we'll reflect in the next revision.
> >
> >>
> >> Section 5.2, paragraph 6, comment:
> >>>   A transit node MUST ignore IOAM-Option-Types that it does not
> >>>   understand.  A transit node MUST NOT add new IOAM-Option-Types to a
> >>>   packet, MUST NOT remove IOAM-Option-Types from a packet, and MUST
> >> NOT
> >>>   change the IOAM-Data-Fields of an IOAM Edge-to-Edge Option-Type.
> >>
> >> I'm surprised that IOAM data isn't authenticated or even
> >> integrity-protected at all. Relying on RFC2119 language alone seems a pretty
> weak protection.
> >
> > ...FB: Integrity protection of IOAM data fields is handled in a sister document:
> draft-brockners-ippm-ioam-data-integrity.
> 
> Thanks, I missed that. But that is an informational reference to an individual I-D.
> I'm still surprised that integrity protection of IOAM data is not more of a concern
> for the architecture that is being proposed here?

...FB: The integrity discussion only started recently - likely because the current deployments of early IOAM / or similar implementations are within "limited domains". We also need to ack the cost of signing the IOAM data, which is not negligible. 

> 
> >>
> >> I'll note that cryptographically authenticating IOM data would
> >> probably result in a system that wouldn't need the concept of
> >> namespaces, because keys would automatically serve that purpose. (A
> >> device can't update an IOAM data item if it doesn't have the key to
> >> authenticate the update with.)
> >
> > ...FB: Looking at the way you interpreted the meaning of namespaces, we'll
> probably need to add some additional explanation of  the role of namespaces.
> 
> I think that would be very helpful. I think Ben's ballot position goes into more
> detail here, so please check with him on this.

... FB: We'll do. I'm still in catch-up mode after the break.
> 
> > Section 5.4, paragraph 11, comment:
> >>>   o  Time of day when the packet was processed by the node as well as
> >>>      the transit delay.  Different definitions of processing time are
> >>>      feasible and expected, though it is important that all devices of
> >>>      an in-situ OAM domain follow the same definition.
> >>
> >> I think "important" is an understatement, this seems required? Also,
> >> capturing time-of-day seems to require synchronized clocks.
> >
> > ...FB: As usual, the answer is "in depends". If one is just interested in capturing
> jitter, then one might also do with non-synchronized clocks.
> 
> I'd suggest that clock sync should then be recommended, with the caveat that
> unsync'ed clocks may be allowed when jitter is all that will be evaluated for.

...FB: We can add this context/explanation to the document. 
> 
> > Section 5.4.2.12, paragraph 2, comment:
> >>> 5.4.2.12.  buffer occupancy
> >>>
> >>>   The "buffer occupancy" field is a 4-octet unsigned integer field.
> >>>   This field indicates the current status of the occupancy of the
> >>>   common buffer pool used by a set of queues.  The units of this field
> >>>   are implementation specific.  Hence, the units are interpreted within
> >>>   the context of an IOAM-Namespace and/or node-id if used.  The authors
> >>>   acknowledge that in some operational cases there is a need for the
> >>>   units to be consistent across a packet path through the network,
> >>>   hence it is RECOMMENDED for implementations to use standard units
> >>>   such as Bytes.
> >>
> >> There are other "standard units" here, such as packets. You'd need to
> >> recommend a specific standard unit and not just give an example.
> >
> > ...FB: The topic of "units" for IOAM data fields has been discussed quite a bit
> and not prescribing units is the result of this discussion. The approach that the
> draft takes is to "avoid specifying a unit unless you have to", because it is quite
> easy to translate units in backend analytics systems. As long as one properly
> understands which unit is used for a particular field, backend systems can easily
> interpret the data and there is no need to prescribe a particular unit.
> 
> Yes, but that "as long as" qualification is a pretty big one. It would mean that the
> backend systems need to have detailed knowledge of what each device along
> the path is configured to use as units. That seems a pretty dangerous
> assumoption?

...FB: You can group devices which use a particular unit for buffers into the same namespace. This is what namespaces are for. 
The backend system would maintain a mapping of what unit is used for buffers for a particular namespace. 

> 
> >> Given that the max. value for microseconds is 999999, using a 32-bit
> >> field leaves the top eight bits unused.
> >
> > ...FB: True. Are you suggesting that we'd rather make the top 8-bits "reserved"
> because they will never be used anyway?
> 
> I'm just pointing it out, in case you'd want to use those eight bits for second-
> level info.

...FB: Good point. Let's see what other folks think - whether we should keep things as they are, or leverage the fields to grow the seconds range.

Thanks again, Frank
> 
> Thanks,
> Lars