Re: [ippm] Carlos' Comments regarding draft-amf-ippm-route

["Carlos Pignataro (cpignata)" <[e-mail]>]

Hi, Al,

Thanks for considering these comments and suggestions, and for your response! Please see inline.

On Oct 16, 2017, at 8:08 PM, MORTON, ALFRED C (AL) <[e-mail]<mailto:[e-mail]>> wrote:

Hi Carlos,

Thanks again for your comments!
I’ve been working through them incrementally
when I could make time, and finally reached the bottom.

There were a few comments I feel we need to clarify
and/or discuss further before taking action.
I marked these with @@@@.

There is a question of how broad to make the
“technology” scope, to address some of your comments below,
and whether the work proposed here:
- metric definitions
- methods of measurement
- RTD Measurement and Analysis
should move ahead together or in parts, to yield a
memo at a length that will serve various purposes,
and reviewers will find appealing.
So, this is something more to discuss!

Correct, some of my comments are “testing” that scope.

I am skipping through some responses, as an implicit Ack.


thanks again, and regards,
Al


From: Carlos Pignataro (cpignata) [mailto:[e-mail]]
Sent: Monday, October 02, 2017 8:03 PM
To: [e-mail]<mailto:[e-mail]>
Subject: Mail regarding draft-amf-ippm-route

Hi, J. Ignacio, Al,

As promised, I sent some time reviewing draft-amf-ippm-route-00 and wanted to share the following comments, observations, and suggestions.

I hope you find these clear and useful. These include both substantive and more editorial comments.

I believe this is a very important topic and would like to further collaborate on it with you.

https://datatracker.ietf.org/doc/draft-amf-ippm-route/?include_text=1<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Damf-2Dippm-2Droute_-3Finclude-5Ftext-3D1&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=PIySZ-JwS55B9576sFOKjczIZtauxElH1LOE7ctBD8w&s=NQxRwUJnjz6rMJlnmzusoYA2L-y9WyLoOJIcpopF7yc&e=>




                Advanced Unidirectional Route Assessment

Did you realize that the title acronyms to AURA? :-)
[ACM]
No, but that is an acronym certainly worth recognizing in the
draft, and hopefully in the WG version of the filename. ☺

CMP: :-)

-=-=-=-=-=-=-=-

Abstract

   This memo introduces an advanced unidirectional route assessment
   metric and associated measurement methodology, based on the IP
   Performance Metrics (IPPM) Framework RFC 2330.  This memo updates RFC
   2330 in the areas of path-related terminology and path description,
   primarily to include the possibility of parallel subpaths.

Do you mean “parallel subpaths” or “parallel paths”? I think at least the latter, at most both, but not only ‘parallel subpaths’.
[ACM]
This is certainly a point that we need to address in terminology.
Well-spotted.

I think that if we define path measurement as originating from
a single source IP address, then all measured paths would include
the same subnet, and any path variation would have the initial
subnet in common (and the destination subnet, because we are testing
between fixed Src and Dst addresses). So, parallel subpaths I think.

But, if the originating and terminating measurement points are
hosts with several different interfaces and connectivity
to different subnets, then we would certainly have the
possibility of completely unique parallel paths.
-=-=-=-=-=-=-=-


2.  Scope

   The purpose of this memo is to add new route metrics and methods of
   measurement to the existing set of IPPM metrics.

   The scope is to define route metrics that can identify the path taken
by a packet traversing the Internet between any two hosts.  Also, to

Given the exploration of parallel paths (i.e., multi-path), it is important to talk about “packets and flows” in the sentence above and throughout, given the flow-awareness of many parallel path implementations. In fact it is a desired feature to load-balance while not creating packet misordering within a given flow.
[ACM]
You make a good point, today’s mechanisms effectively allocate
flows among the parallel resources with a good outcome.
However, each packet in the flow is routed individually,
and when we manipulate the packet fields during measurement
to preserve flow characteristics, we do so on a per-packet basis.

CMP: Not necessarily. Take e.g., RFC 6438, in which a “packet” field defines a flow, and a potential mapping to sub-paths.

CMP: In that sense, things like rates and delays (even using a coloring scheme) can be done per-flow.

Nevertheless, I think we should mention flows here in the scope.

CMP: Thanks!

-=-=-=-=-=-=-=-

   specify a framework for active methods of measurement which use the
   techniques described in [PT] at a minimum, and a framework for hybrid
   active-passive methods of measurement, such as the Hybrid Type I
   method [RFC7799] described in [I-D.brockners-inband-oam-data]
   (currently intended only for single administrative domains).
   Combinations of active methods and hybrid active-passive methods are
   also in-scope.

   Further, this memo provides additional analysis of the round-trip
   delay measurements made possible by the methods, in an effort to
   discover more details about the path, such as the link technology in
   use.

Does this really imply “guessing” the link technology based on the reported delay? So a congested GigE can be confused for a 2G interface? If that’s not what is meant, somehow I misunderstood. If it is what is meant, I think it shouldn’t. We can discover the link technology directly via interrogation and not with heuristics.
[ACM]
Yes, I think the term I would use is “inferring” link technology
from delay measurements. But general queue delay and apparent
tail drop alone do not indicate anything but congestion.

CMP: Correct. That was my point.

The
task is simple for some technologies like satellite. 2G would
likely have a strong quantization of delays from the time-slot
service discipline. I doubt GigE can be identified.
But this is Ignacio’s area of investigation, and he should
add to or correct what I’ve said.

@@@@ I’m interested to know what forms of link interrogation you
are thinking of (part of IOAM?) ...


CMP: Great question! With iOAM you can have explicit interface identification via interface_id. But there’s nothing preventing you from things like RFC 5837 including name and MTU. Just an observer that influences more.

CMP: Traceroute can easily use RFC 5837 (I had at one point an implementation in my machine)

You make another good point, though. We cannot anticipate all
the ways which the metrics and measurement results can be
*used badly*, to make incorrect inferences or other mistakes.
The problem of making bad decisions using good measurements
is not ours to solve, but we can certainly add appropriate
cautions for the methods we specify and possibly head-off
first-order stupidity.

CMP: :-)

-=-=-=-=-=-=-=-=-=-=-


   This memo updates Section 5 of [RFC2330] in the areas of path-related
   terminology and path description, primarily to include the
   possibility of parallel subpaths.

Parallel Paths and/or Subpaths?
[ACM]
...parallel subpaths between a given Source and Destination pair.
(as mentioned above, this is with fixed endpoint IPs)



CMP: Ack.

Also, I believe it is important to acknowledge the “Multi-Path” terminology and include it in the definitions. Further, ECMP (Equal-cost multi path) and UCMP (unequal-cost …) can be named.
[ACM]
Yes, something like:
...parallel subpaths between a given Source and Destination pair,
owing to the presence of Multi-path technologies (such as ECMP and UCMP).

spelling out the acronyms of course.

CMP: Thanks!


-=-=-=-=-=-=-=-=-=-=-=-=-=-


   There are several simple non-goals of this memo.  There is no attempt
   to assess the reverse path from any host on the path to the host
   attempting the path measurement.  The reverse path contribution to
   delay will be that experienced by ICMP packets (in active methods),
   and may be different from UDP or TCP packets.  Also, the round trip
   delay will include an unknown contribution of processing time at the
   host that generates the ICMP response.  Therefore, the active methods
   are not supposed to yield accurate, reproducible estimations of the
   round-trip delay that UDP or TCP packets will experience.

This should clarify that there are other active methods that do not use ICMP. And in particular, I am interested in using MPLS LSP Ping and MPLS LSP Traceroute here (RFC 8029), which provide:
1.       Timestamp facilities and
2.       deterministic Multipath awareness.
I understand this is “IP”PM, but see https://tools.ietf.org/html/rfc8029<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc8029&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=PIySZ-JwS55B9576sFOKjczIZtauxElH1LOE7ctBD8w&s=t8iMJjpIlZGQevkw53tCRPrBKd_FwCTUIp4Wf_J9abc&e=>
The same techniques can be applied to IP, and are applied to IPv6 in Data Center environments. Including a Downstream Mapping.
[ACM]
Adding some generality to include MPLS traces in the definitions
is probably worth looking into. I imagine there would be more++
work to describe the corresponding methods. In any case,
the two features you describe above take much of the
uncertainty/measurement challenge away, I think.

CMP: Correct, they add determinism to it. And perhaps there are easy ways of generalizing without loosing the initial point, and without massive re-writes.


-=-=-=-=-=-=-=-=-=-=-


3.  Route Metric Definitions

   Section 7 of [RFC2330] presented a simple example of a "route" metric
   along with several other examples.  The example is reproduced below
   (where the reference is to Section 5 of [RFC2330]):

   "route The path, as defined in Section 5, from A to B at a given
   time."

   This example provides a starting point to develop a more complete
   definition of route.  Areas needing clarification include:

   Time:  In practice, this will be a time interval, because active path
      detection methods like [PT] rely on TTL limits for their operation
      and cannot accomplish discovery of all hosts using a single
      packet.

This could also call out the methods in RFC 8029 and explicit Multipath identification https://tools.ietf.org/html/rfc8029#section-6.2.6<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc8029-23section-2D6.2.6&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=PIySZ-JwS55B9576sFOKjczIZtauxElH1LOE7ctBD8w&s=GqxQd3nbg14yYyWs8ut24JDSIhZDDZRu8cNBlN8rIds&e=>
Further, this definition should speak to Flow-awareness.

[ACM]
@@@@ I’m not clear on how the discussion Time intervals would benefit from
mentioning MPLS Multipath or flow-awareness, and note that the
route definition above is just an example from the Framework RFC2330.
But I may be missing your point…

CMP: Sorry I was not clear, only that time interfaval measurement can, in some use cases, be flow-aware. A minor point, however.


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-



   Parallel Paths:  This a reality of Internet paths and a strength of
      advanced route assessment methods, so the metric must acknowledge
      this possibility.

It would be great for this to speak to ECMP and UCMP.
[ACM]
Yes, encountering  ECMP or UCMP makes this a certainty.
And flow-awareness, as you mention below, these terms
and definitions should be referenced in this section.


   Cloud Subpath:  May contain hosts that do not decrement TTL or Hop
      Count, but have two or more exchange links connecting
      "discoverable" hosts or routers.  Parallel subpaths contained
      within clouds cannot be discovered.  The assessment methods only
      discover hosts or routers on the path that decrement TTL or Hop
      Count.

This definition says “Hop Count” but it should say “Hop Limit” [RFC 8200].
[ACM]
Yes, we need to fix that everywhere now.


CMP: Thanks!


   The refined definition of Route metrics begins with the sections that
   follow.


There is one Operationally important consideration that is not discussed here: Tunneling and nesting of Tunnels. And in particular, different Tunnel modes that can “Hide Hops” by not copying-up the TTL/Hop Limit.
[ACM]
I agree, this is worth describing, the effect of tunnels
can be profound.

@@@@ In my OPS-DIR review of draft-ietf-bier-mpls-encapsulation
I found that they planned to decrement TTL and discard when
TTL=1, but had no requirement for notification
(MAY tell other layers).
I have pushed-back on the possibility for silent discard,
and I’m looking for a “SHOULD tell other layers”, at least.

CMP: Indeed!

-=-=-=-=-=-=-=-=-=-



3.2.  Parameters

   o  Src, the IP address of a host

   o  Dst, the IP address of a host

   o  i, the TTL or Hop Count of a packet sent from the host at Src to
      the host at Dst.

This should say “Hop Limit”, not “Hop Count”.
[ACM]
Yep.


   o  MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops).

   o  T0, a time (start of measurement interval)

   o  Tf, a time (end of measurement interval)

   o  T, the host time of a packet as measured at MP(Src), meaning
      Measurement Point at the Source.

   o  Ta, the host time of a reply packet's *arrival* as measured at
      MP(Src), assigned to packets that arrive within a "reasonable"
      time (see parameter below).

   o  Tmax, a maximum waiting time for reply packets to return to the
      source, set sufficiently long to disambiguate packets with long
      delays from packets that are discarded (lost), thus the
      distribution of delay is not truncated.

   o  F, the number of different flows simulated by the method and
      variant.

   o  flow, the stream of packets with the same n-tuple of designated
      header fields that (when held constant) results in identical
      treatment in a multi-path decision (such as that taken in load
      balancing).

It is important to come back to the Flow-awareness piece mentioned above, and multipath. Hence these should be defined earlier, as explicit considerations. (ECMP, not LAGs)
[ACM]
I agree.


4.  Route Assessment Methodologies

4.1.  Active Methodologies

   We have chosen to describe the method based on that employed in
   current open source tools, thereby providing a practical framework
   for further advanced techniques to be included as method variants.

The practicality of using these tools should not opaque the fact that there are others that provide flow-awareness different ways. One is using UDP Probes as Facebook does. Another is using Downstream-mapping techniques as proposed in NVO3.
[ACM]
I agree, perhaps after some survey and investigation,
we should recommend a method that may be more efficient
and effective than PT for the general IP network case.
We could list other special circumstances and references
for their tracing solutions/methods.

I have a lingering concern about the breadth of scope
if we go into NVO3 and similar technologies (there are
many).

CMP: I am testing the boundary here. :-)


-=-=-=-=-

   Paris-traceroute [PT] provides some measure of protection from path
   variation generated by ECMP load balancing, and it ensures traceroute
   packets will follow the same path in 98% of cases according to
   [SCAMPER].  If it is necessary to find every path possible between
   two hosts, Paris-traceroute provides "exhaustive" mode while scamper
   provides "tracelb" (stands for traceroute load balance).

   The Type-P of packets used could be ICMP (as ones in the original
   traceroute), UDP and TCP.  The later are used when a particular
   characteristic is needed to verify, such as filtering or traffic
   shaping on specific ports (i.e., services).

   The advanced route assessment method used by Paris-traceroute keeps
   these fields constant for every packet that it sends to maintain the
   appearance of the same flow in routers.  Since route assessment can
   be conducted using TCP, UDP or ICMP packets, this method keeps the
ToS, the protocol, IP source and destination addresses, and the port

All these field names need updaring (there’s no “ToS”, protocol is “protocol number” or “next header”) and IPv6-proof.
[ACM]
I agree.


   settings for TCP or UDP intact.  For ICMP probes, the method requires
   to keep the type, code, and ICMP checksum constant; which take the
   same position in the header of an IP packet, e.g., bytes 20 to 23
   when the header IP has no options.

   The checksum is most challenging because the ICMP sequence number is
   part of the checksum calculation.  The advanced traceroute method
   requires calculation of the IP identification field, yielding in a
   constant ICMP checksum.

   For TCP and UDP packets, the checksum must also be kept constant.
   Therefore, the first four bytes of UDP data field are modified to
   compensate for fields that change from packet to packet.  This
   variant of the advanced traceroute method is called Paris traceroute
   [PT].  Note: other variants of advanced traceroute are planned be
described.

The IPv6 Field “Flow Label” is not covered here, but it really should. See RFC 6438, using the IPv6 Flow Label for flow-aware ECMP/LAG.
[ACM]
Good point, I wonder if the PT paper considered IPv6,
it’s something to check, and certainly something to address
(otherwise, there will be pain to go back and do it later).



CMP: Thanks!

   Finally, the return path is also important to check.  Taking into
   account that it is an ICMP time exceeded (during transit) packet, the
   source and destination IP are constant for every reply.  Then, we
   should consider the fields in the first 32 bits of the protocol on
   the top of IP: the type and code of ICMP packet, and its checksum.
   Again, to maintain the ICMP checksum constant for the returning
   packets, we need to consider the whole ICMP message.  It contains the
   IP header of the discarded packet plus the first 8 bytes of the IP
   payload; that is some of the fields of TCP header, the UDP header
   plus four data bytes, the ICMP header plus four bytes.  Therefore,
   for UDP case the data field is used to maintain the ICMP checksum
   constant in the returning packet.  For the ICMP case, the identifier
   and sequence fields of the sent ICMP probe are manipulated to be
   constant.  The TCP case presents no problem because its first eight
   bytes will be the same for every packet probe.

   Formally, to maintain the same flow in the measurements to a certain
   hop, the Type-P-Route-Ensemble-Method-Variant packets should be[PT]:

   o  TCP case: Fields Src, Dst, port-Src, port_Dst, and ToS should be
      the same.

   o  UDP case: Fields Src, Dst, port-Src, port-Dst, and ToS should be
      the same, the UDP-checksum should change to maintain constant the
      IP checksum of the ICMP time exceeded reply.  Then, the data
      length should be fixed, and the data field is used to fixing it
      (consider that ICMP checksum uses its data field, which contains
      the original IP header plus 8 bytes of UDP, where TTL, IP
      identification, IP checksum, and UDP checksum changes).

   o  ICMP case: The Data field should compesate variations on TTL, IP
identification, and IP checksum for every packet.

Typo – “compensate”
[ACM]
thanks.


   Then, the way to identify different hops and attempts of the same
   flow is:

   o  TCP case: The IP identification field.

   o  UDP case: The IP identification field.

   o  ICMP case: The IP idtentification field, and ICMP Sequence number.


Typo, “identification”.
[ACM]
thanks.


4.2.  Hybrid Methodologies

   The Hybrid Type I methods provide an alternative method for Route
   Member assessment.  As mentioned in the Scope section,
   [I-D.brockners-inband-oam-data] provides a possible set of data
   fields that would support route identification.

Here, I would highlight path congruency given the Hybrid method, is built it.
[ACM]
@@@@ OK, I just looked-up path congruency, I think you mean
bidirectional path congruency, or reverse path congruency,
right? and you meant “built-in” ?

CMP: No, sorry. I meant the idea of using iOAM within a data packet itself.


We currently have “unidirectional” in the “AURA” title.

“Congruent” doesn’t appear in [I-D.brockners-inband-oam-data]
which we now need to update to the WG version:
https://tools.ietf.org/html/draft-ietf-ippm-ioam-data-00

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


   In general, nodes in the measured domain would be equipped with
   specific abilities:

   1.  The ingress node adds one or more fields to the measurement
       packets, and identifies to other nodes in the domain that a route
       assessment will be conducted using one or more specific packets.
       The packets typically originate from a host outside the domain,
       and constitute normal traffic on the domain.

   2.  Each node visited by the specific packet within in the domain
       identifies itself in a data field of the packet (the field has
       been added for this purpose).

   3.  When a measurement packet reaches the edge node of the domain,
       the edge node adds its identity to the list, removes all the
       identities from the packet, forwards the packet onward, and
       communicates the ordered list of node identities to the intended
       receiver.

While this method is described above to capture Nodes / hosts within each hop, it can (and does) also capture links, including potentially the “type of link”..
[ACM]
Thanks, that’s something we (I) missed.


6.  Tools to Measure Delays in the Internet

Source-routed technologies, like Segment Routing for Ipv6, allow this for arbitrary sources and destinations. From NMS to source, to destination to NMS, with source and dest as arbitrary hosts.
[ACM]
Clearly, we have more to consider here.
The possibility of Segment Routing raises a new area of
investigation, and a first-order question for the scope
of the draft.


8.  Conclusions

   Combining the method proposed in Section 4 and statistics in
   Section 7, we can measure the performance of paths interconnecting
   two endpoints in Internet, and attempt the categorization of link
   types and congestion presence based on RTD.

I trust these will be useful.
[ACM]
I hope this answer is redundant after my in-line replies,
but YES!!

CMP: Thanks! :-) Great to hear.

— Carlos.


THANKS!

Carlos Pignataro.
.