Re: [Detnet] AD review of draft-ietf-detnet-oam-framework-08

[Greg Mirsky <[e-mail]>]

Hi John,
thank you for your thorough review and thoughtful comments. We used the
DOCX copy for the authors' discussion of your questions and suggestions.
Attached, please find a new working version and diffs with the earlier
version of the draft and TXT copy with your comments. Please let me know if
we satisfactorily addressed your comments.

Regards,
Greg


On Fri, May 26, 2023 at 9:22 AM John Scudder <[e-mail]> wrote:

> Hi Authors, WG,
>
> Thanks for this spec.
>
> I’ve supplied my questions and comments in the form of an edited copy of
> the draft. Minor editorial suggestions I’ve made in place without further
> comment, more substantive questions and comments are done in-line and
> prefixed with “jgs:”. You can use your favorite diff tool to review them;
> I’ve attached the iddiff output for your convenience if you’d like to use
> it. I’ve also pasted a traditional diff below in case you want to use it
> for in-line reply.
>
> Thanks,
>
> —John
>
> --- draft-ietf-detnet-oam-framework-08.txt      2023-05-25
> 15:47:46.000000000 -0400
> +++ draft-ietf-detnet-oam-framework-08-jgs-comments.txt 2023-05-26
> 12:10:44.000000000 -0400
> @@ -22,8 +22,8 @@
>
>  Abstract
>
> -   Deterministic Networking (DetNet), as defined in RFC 8655, is aimed
> -   to provide a bounded end-to-end latency on top of the network
> +   Deterministic Networking (DetNet), as defined in RFC 8655, aims
> +   to provide bounded end-to-end latency on top of the network
>     infrastructure, comprising both Layer 2 bridged and Layer 3 routed
>     segments.  This document's primary purpose is to detail the specific
>     requirements of the Operation, Administration, and Maintenance (OAM)
> @@ -139,7 +139,7 @@
>     Objectives (SLO), such as in-order packet delivery, packet delay,
>     delay variation, and packet loss ratio, assigned to each DetNet flow.
>
> -   This document lists the functional requirements toward OAM for DetNet
> +   This document lists the functional requirements toward OAM for a DetNet
>     domain.  The list can further be used for gap analysis of available
>     OAM tools to identify possible enhancements of existing or whether
>     new OAM tools are required to support proactive and on-demand path
> @@ -176,17 +176,31 @@
>
>     *  Maintenance Intermediate Point (MIP): an OAM instance along the
>        DetNet flow in the particular sub-layer of the DetNet OAM domain.
> -      A MIP MAY respond to an OAM message generated by the MEP at its
> +      A MIP may respond to an OAM message generated by the MEP at its
>        sub-layer of the same DetNet OAM domain.
> +--
> +jgs: Do you really want the MAY above? The note in Section 1.3 implies
> +you'll only be using RFC 2119 keywords in Section 6, so I've tentatively
> +changed it to "may".
> +--
>
>     *  Control and management plane: the control and management planes
>        are used to configure and control the network (long-term).
>        Relative to a DetNet flow, the control and/or management plane can
>        be out-of-band.
> +--
> +jgs: I'm not sure what work the parenthetical comment "(long-term)" is
> +doing here. Would it do any harm to remove it?
> +--
>
>     *  Active measurement methods (as defined in [RFC7799]) modify a
>        DetNet flow by inserting novel fields, injecting specially
>        constructed test packets [RFC2544]).
> +--
> +jgs: Consider "new", "extra", or "additional" instead of "novel", which
> +is a bit of a trigger word for some reviewers and doesn't seem to be
> +needed here.
> +--
>
>     *  Passive measurement methods [RFC7799] infer information by
>        observing unmodified existing flows.
> @@ -216,6 +230,16 @@
>     OAM: Operations, Administration, and Maintenance
>
>     DetNet: Deterministic Networking
> +--
> +jgs: "DetNet" is not, strictly speaking, an acronym, it's an
> +abbreviation. For that matter, neither is "OAM", it's an initialism,
> +since it's read as "oh ay em". "PREOF" is a legit acronym. [*] I was
> +going to suggest you consider changing the subsection name to
> +"terminology" except you've already used that. I guess you could use
> +"definitions".
> +
> +[*]
> https://www.rd.com/article/acronym-vs-abbreviation-whats-the-difference/
> +--
>
>     PREOF: Packet Replication, Elimination and Ordering Functions
>
> @@ -228,7 +252,11 @@
>
>     SLO: Service Level Objective
>
> -   PM: Perfromance Monitoring
> +   PM: Performance Monitoring
> +--
> +jgs: You only use the term "PM" in one place. Consider removing it and
> just
> +writing out "Performance Monitoring" in that place.
> +--
>
>  1.3.  Requirements Language
>
> @@ -244,6 +272,10 @@
>     DetNet networks expect to provide communications with predictable low
>     packet delay and packet loss.  Most critical applications will define
>     an SLO to be required for the DetNet flows it generates.
> +--
> +jgs: You identify bounding delay and loss as primary goals of DetNet, but
> +leave out misordering, which RFC 8655 also identifies as a first-order
> goal.
> +--
>
>     To respect strict guarantees, DetNet can use an orchestrator able to
>     monitor and maintain the network.  Typically, a Software-Defined
> @@ -252,27 +284,38 @@
>     the regular operation of the network.  OAM represents the essential
>     elements of the network operation and necessary for OAM resources
>     that need to be accounted for to maintain the network operational.
> +--
> +jgs: the last sentence above doesn't work. Are you using "represents" in
> +the sense of "is defined as"? If so, let me implore you not to redefine
> +OAM in-line in a single sentence, after all we have an entire RFC (6291,
> +which you cite) to do this. It seems to me you can just delete the
> +sentence.
> +
> +If you're doing something other here than (re)defining "OAM", I
> +seriously don't get it, please propose a rewrite or help me understand
> +what you mean.
> +--
>
>     Many legacy OAM tools can be used in DetNet networks, but they are
>     not able to cover all the aspects of deterministic networking.
>     Fulfilling strict guarantees is essential for DetNet flows, resulting
> -   in new DetNet specific functionalities that must be covered with OAM.
> +   in new DetNet-specific functionalities that must be covered with OAM.
>     Filling these gaps is inevitable and needs accurate consideration of
> -   DetNet specifics.  Similar to DetNet flows itself, their OAM needs
> +   DetNet specifics.  Similar to DetNet flows themselves, their OAM needs
>     careful end-to-end engineering as well.
>
>     For example, appropriate placing of MEPs along the path of a DetNet
> -   flow is not always a trivial task and may require proper design
> +   flow is not always a trivial task and may require proper design,
>     together with the design of the service component of a given DetNet
>     flow.
>
> -   There are several DetNet specific challenges for OAM.  Bounded
> +   There are several DetNet-specific challenges for OAM.  Bounded
>     network characteristics (e.g., delay, loss) are inseparable service
>     parameters; therefore, PM is a key topic for DetNet.  OAM tools are
>     needed to prove the SLO without impacting the DetNet flow
> -   characteristics.  A further challenge is the strict resource
> +   characteristics.  A further challenge is strict resource
>     allocation.  Resources used by OAM must be considered and allocated
> -   to avoid disturbing DetNet flow(s).
> +   to avoid disturbing DetNet flows.
>
>
>
> @@ -292,12 +335,12 @@
>        network.
>
>     OAM mechanisms exist for the DetNet forwarding sub-layer,
> -   nonetheless, OAM for the service sub-layer requires new OAM
> +   but the service sub-layer requires new OAM
>     procedures.  These new OAM functions must allow, for example, to
>     recognize/discover DetNet relay nodes, to get information about their
>     configuration, and to check their operation or status.
>
> -   DetNet service sub-layer functions use a sequence numbers for PREOF.
> +   DetNet service sub-layer functions use a sequence number for PREOF.
>     That creates a challenge for inserting OAM packets in the DetNet
>     flow.
>
> @@ -312,13 +355,17 @@
>     also control PREOF in those scenarios where DetNet solutions involve
>     more than one single central controller.
>
> -   DetNet forwarding sub-layer is based on legacy technologies and has a
> +   The DetNet forwarding sub-layer is based on preexisting technologies
> and has
>     much better coverage regarding OAM.  However, the forwarding sub-
>     layer is terminated at DetNet relay nodes, so the end-to-end OAM
>     state of forwarding may be created only based on the status of
>     multiple forwarding sub-layer segments serving a given DetNet flow
>     (e.g., in case of DetNet MPLS, there may be no end-to-end LSP below
>     the DetNet PW).
> +--
> +jgs: Please write out PW. You could expand on first use instead, but...
> +this is the only use.
> +--
>
>  3.  Operation
>
> @@ -326,8 +373,8 @@
>     forwarding and routing purposes.
>
>     It is worth noting that the test and data packets are expected to
> -   follow the same path, i.e., the connectivity verification has to be
> -   conducted in-band without impacting the data traffic.  It is expected
> +   follow the same path, i.e., connectivity verification has to be
> +   conducted in-band without impacting data traffic.  It is expected
>     that test packets share fate with the monitored data traffic without
>     introducing congestion in normal network conditions.
>
> @@ -371,7 +418,7 @@
>  3.2.  Continuity Check
>
>     Continuity check is used to monitor the continuity of a path, i.e.,
> -   that there exists a way to deliver the packets between two MEP A and
> +   that there exists a way to deliver packets between MEP A and
>     MEP B.  The continuity check detects a network failure in one
>     direction, from the MEP transmitting test packets to the remote
>     egress MEP.  Continuity check in a DetNet OAM domain monitors the
> @@ -381,11 +428,21 @@
>  3.3.  Connectivity Verification
>
>     In addition to the Continuity Check, DetNet solutions have to verify
> -   the connectivity.  This verification considers additional
> -   constraints, i.e., the absence of misconnection.  The misconnection
> +   connectivity.  This verification considers additional
> +   constraints, e.g., the absence of misconnection.  The misconnection
>     error state is entered after several consecutive test packets from
> -   other DetNet flows are received.  The definition of the conditions of
> -   entry and exit for misconnection error state is outside the scope of
> +   other DetNet flows are received.  The definition of the conditions for
> +   entry and exit of misconnection error state is outside the scope of
> +--
> +jgs: Above I changed "i.e." to "e.g.", since I *think* you mean "for
> +example", and not "in other words".
> +
> +On the other hand if you really did mean "i.e." then I guess it's not
> +"additional constraints" but "an additional constraint", singular, and
> +you should revert my edit and make that edit instead, for agreement in
> +number. (In that case you could also remove "i.e." entirely without
> +loss of clarity.)
> +--
>
>
>
> @@ -396,40 +453,48 @@
>
>     this document.  Connectivity verification in a DetNet OAM domain
>     monitors the DetNet forwarding sub-layer and thus is not affected by
> -   PREOF that operates at the DetNet service sub-layer ([RFC8655].
> +   PREOF that operates at the DetNet service sub-layer ([RFC8655]).
>
>  3.4.  Route Tracing
>
>     Ping and traceroute are two ubiquitous tools that help localize and
> -   characterize a failure in the network using echo request/reply
> +   characterize a failure in the network using an echo request/reply
>     mechanism.  They help to identify a subset of the list of routers in
>     the route.  However, to be predictable, resources are reserved per
> +--
> +jgs: Would it be correct to rewrite as "They help to identify a subset of
> +the routers in the path"? (If not, why not?)
> +--
>     flow in DetNet.  Thus, DetNet needs to define route tracing tools
> -   able to track the route for a specific flow.  Also, tracing can be
> +   able to trace the route for a specific flow.  Also, tracing can be
>     used for the discovery of the Path Maximum Transmission Unit or
>     location of elements of PREOF for the particular route in the DetNet
>     domain.
>
>     DetNet is not expected to use Equal-Cost Multipath (ECMP) [RFC8939].
> -   As the result, DetNet OAM in ECMP environment is outside the scope of
> +   As the result, DetNet OAM in an ECMP environment is outside the scope
> of
>     this document.
>
>  3.5.  Fault Verification/Detection
>
>     DetNet expects to operate fault-tolerant networks.  Thus, mechanisms
> -   able to detect faults before they impact the network performance are
> +   able to detect faults before they impact network performance are
>     needed.
>
> -   The network has to detect when a fault occurred, i.e., the network
> +   The network has to detect when a fault has occurred, i.e., the network
>     has deviated from its expected behavior.  Fault detection can be
> -   based on pro-active OAM protocols like continuity check or on-demand
> +   based on proactive OAM protocols like continuity check or on-demand
> methods
>     like ping.  While the network must report an alarm, the cause may not
>     be identified precisely.  For instance, the end-to-end reliability
>     has decreased significantly, or a buffer overflow occurs.
> +--
> +jgs: Would it be right to replace "For instance" with "Examples of such
> +alarms are"?
> +--
>
>  3.6.  Fault Localization and Characterization
>
> -   An ability to localize the network defect and provide its
> +   An ability to localize a network defect and provide its
>     characterization are necessary elements of network operation.
>
>        Fault localization, a process of deducing the location of a
> @@ -437,7 +502,7 @@
>        be achieved, for example, by tracing the route of the DetNet flow
>        in which the network failure was detected.  Another method of
>        fault localization can correlate reports of failures from a set of
> -      interleaving sessions monitoring path continuity.
> +      interleaved sessions monitoring path continuity.
>
>
>
> @@ -464,11 +529,14 @@
>        of Active Methods and Passive Methods.
>
>     A hybrid measurement method may produce metrics as close to passive,
> +--
> +jgs: I don't know what you mean by "as close to passive"
> +--
>     but it still alters something in a data packet even if that is the
>     value of a designated field in the packet encapsulation.  One example
>     of such a hybrid measurement method is the Alternate Marking method
>     (AMM) described in [RFC8321].  As with all on-path telemetry methods,
> -   AMM in a DetNet domain with the IP data plane is natively in-band in
> +   AMM in a DetNet domain with the IP data plane is natively in-band with
>     respect to the monitored DetNet flow.  Because the marking is applied
>     to a data flow, measured metrics are directly applicable to the
>     DetNet flow.  AMM minimizes the additional load on the DetNet domain
> @@ -478,26 +546,34 @@
>
>  4.  Administration
>
> -   The ability to expose a collection of metrics to support an operator
> -   making proper decisions is essential.  Following perfromance metrics
> +   The ability to expose a collection of metrics to support an operator's
> +   decision-making is essential.  The following performance metrics
>     are useful:
>
> -   *  Queuing Delay: the time elapsed between a packet enqueued and its
> +   *  Queuing Delay: the time elapsed between enqueuing a packet and its
>        transmission to the next hop.
>
>     *  Buffer occupancy: the number of packets present in the buffer, for
>        each of the existing flows.
>
> -   *  Per a DetNet flow to measure the end-to-end performance for a
> +   *  Per DetNet flow, a measure of the end-to-end performance for a
>        given flow.  Each of the paths has to be isolated in multipath
>        routing strategies.
> +--
> +jgs: I'm not sure my proposed edit correctly captures your meaning,
> +please check.
> +--
>
> -   *  Per path to detect misbehaving path(s) when multiple paths are
> -      used for the service protection.
> +   *  Per path, detection of misbehaving path(s) when multiple paths are
> +      used for service protection.
>
> -   *  Per device to detect misbehaving device, when it relays the
> +   *  Per device, detection of a misbehaving device, when it relays the
>        packets of several flows.
> -
> +--
> +jgs: Same above. Also, I don't understand what work "when it relays the
> +packets of several flows" is doing. Is it OK for a device to misbehave
> when
> +it relays the packets of only a single flow?
> +--
>
>
>
> @@ -514,6 +590,15 @@
>     Periodic and event-triggered collection information characterizing
>     the state of a network is an example of of of mechanisms to achieve
>     the optimization.
> +--
> +jgs: I'm not entirely sure what the first sentence above means, but I
> +suspect the "important operational function" text is muddling things
> +up. Perhaps,
> +
> +It is important to optimize the volume and frequency of statistics/
> +measurement collection, whether the mechanisms are distributed,
> +centralized, or both.
> +--
>
>  4.2.  Worst-case metrics
>
> @@ -530,7 +615,25 @@
>     Service protection (provided by the DetNet Service sub-layer) is
>     designed to cope with simple network failures and mitigates the
>     DetNet Controller Plane's immediate reaction to network events.  In
> -   the face of events that impact the network operation (e.g., link up/
> +--
> +jgs: I don't understand what you mean by "mitigates the DetNet
> +Controller Plane's immediate reaction". Are you trying to say
> +that it mitigates the inevitable latency between a network event,
> +and the Controller Plane's ability to respond to it, analogous
> +to the relationship between IPFRR and traditional routing
> +protocols?
> +
> +On the assumption that's what you're getting at, perhaps something
> +like,
> +
> +"... designed to mitigate simple network failures more rapidly
> +than the expected response time of the DetNet Controller Plane."
> +
> +If you wanted to you could even add the analogy to the relationship
> +between IPFRR and the IP control plane, but this might just be
> +clutter.
> +--
> +   the face of events that impact network operation (e.g., link up/
>     down, device crash/reboot, flows starting and ending), the DetNet
>     Controller Plane needs to perform repair and re-optimization actions
>     in order to permanently ensure the SLO of all active flows with
> @@ -541,6 +644,17 @@
>
>        the cost of the sub-optimality: resources may not be used
>        optimally (e.g., a better path exists).
> +--
> +jgs: I'm trying to think of an example where resources are not used
> +optimally but no better path exists. I guess it depends on exactly
> +how one defines "better" and "optimal" but essentially, it seems to me
> that
> +"resources used optimally" is equivalent to "there exists no better
> +path" and therefore "resources not used optimally" must mean "there does
> +exist a better path".
> +
> +Which, if you agree, means either the "e.g." should be an "i.e.", or
> +the parenthetical comment should go away.
> +---
>
>        the reconfiguration cost: the DetNet Controller Plane needs an
>        ability to trigger some reconfigurations.  For this transient
> @@ -604,6 +718,10 @@
>     2.   It MUST be possible to initialize a DetNet OAM session from
>          using any of DetNet Controller Plane solution, e.g., centralized
>          controller.
> +--
> +jgs: I don't understand what the above means. Please explain/propose a
> +rewrite. Possibly you mean "initiate" instead of "initialize"?
> +--
>
>     3.   DetNet OAM MUST support proactive OAM monitoring and measurement
>          methods.
> @@ -649,11 +767,11 @@
>         the DetNet OAM instance performing continuity checking.
>
>     3.  DetNet OAM MUST support monitoring levels of resources allocated
> -       for the particular DetNet flow.  Such resources include but not
> -       limited to buffer utilization, scheduler transmission calendar.
> +       for a particular DetNet flow.  Such resources include but are not
> +       limited to buffer utilization and scheduler transmission calendar.
>
> -   4.  DetNet OAM MUST support monitoring any sub-set of paths traversed
> -       through the DetNet domain by the DetNet flow.
> +   4.  DetNet OAM MUST support monitoring any subset of paths traversed
> +       through the DetNet domain by a DetNet flow.
>
>  6.2.  Requirements on OAM for DetNet Service Sub-layer
>
> @@ -680,21 +798,47 @@
>     3.  DetNet OAM MUST support the discovery of Packet Replication,
>         Elimination, and Order preservation sub-functions locations in
>         the domain.
> +--
> +jgs: Probably that should be "sub-function" singular. Also, this is the
> +rare case where I suggest inserting an acronym -- since we talk about
> +"PREOF" so much in DetNet, it probably would be worth adding "(PREOF)"
> +here.
> +--
>
> -   4.  DetNet OAM MUST support the collection of the DetNet service sub-
> +   4.  DetNet OAM MUST support the collection of DetNet service sub-
>         layer specific (e.g., configuration/operation/status) information
>         from DetNet relay nodes.
> +--
> +jgs: I'm willing to accept that there *might* be sub-layer specific info
> +other than configuration, operation, and status, but I sure don't know
> +what it is. So I wonder if the "e.g." can/should be dropped.
> +--
>
> -   5.  DetNet OAM MUST support excercising functionality of Packet
> +   5.  DetNet OAM MUST support exercising functionality of Packet
>         Replication, Elimination, and Order preservation sub-functions in
>         the domain.
> +--
> +jgs: again it might be worth adding "(PREOF)". One reason for this is to
> +accommodate people doing text search within the document for relevant
> +requirements.
> +--
>
>     6.  DetNet OAM MUST work for DetNet data planes - MPLS and IP.
>
>     7.  DetNet OAM MUST support defect notification mechanism, like Alarm
> -       Indication Signal.  Any DetNet relay node within the given DetNet
> +       Indication Signal.  Any DetNet relay node within a given DetNet
>         flow MAY originate a defect notification addressed to any subset
>         of DetNet relay nodes within that flow.
> +--
> +jgs: there needs to be some article before "defect notification
> mechanism",
> +I'm not sure which to propose, but maybe "a", as in "... must support a
> +defect notification mechanism, like Alarm Indication Signal".
> +
> +Also, are relay nodes "within" a flow? I would think they are "along" a
> +flow or possibly "providing service for" a flow.
> +
> +The MAY seems a little funny in this context but I think it is OK.
> +--
>
>     8.  DetNet OAM MUST be able to measure metrics (e.g. delay) inside a
>         collection of OAM sessions, specially for complex DetNet flows,
> @@ -703,13 +847,13 @@
>  7.  IANA Considerations
>
>     This document has no actionable requirements for IANA.  This section
> -   can be removed before the publication.
> +   can be removed before publication.
>
>  8.  Security Considerations
>
>     This document lists the OAM requirements for a DetNet domain and does
>     not raise any security concerns or issues in addition to ones common
> -   to networking and those specific to a DetNet discussed in Section 9
> +   to networking and those specific to DetNet discussed in Section 9
>     of [RFC9055].
>
>  9.  Acknowledgments
>
>
>