Re: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05

[János Farkas <[e-mail]>]

Hi David,

Thank you very much for your review and comments!

Please see in-line.

On 6/29/2018 4:10 PM, Black, David wrote:
>
> The -06 version of this draft is much improved over the -05 version – 
> lots of diligent work by the authors is evident and appreciated.
>
> I have two minor content concerns and some smaller editorial items.
>
> ** Content
>
> -- Section 2.2
>
> The definition of “bridged path” looks like a definition of “bridged 
> forwarding” – the definition needs to encompass the notion of a path 
> that involves more than one bridge.
>
>    bridged path
>            A VLAN bridge uses the VLAN ID and the destination MAC
>            address to select the outbound port hence the path for a
>            frame.
>
You are right, it looks like the emphasis is on bridge forwarding rather 
than path.
However, the superposition of the outbound ports provides a path, which 
was tried to be captured by "hence the path".
The idea was to keep it simple not dive into the details, which are not 
necessary in this document.

Would it be good to update to:

    bridged path

            A VLAN bridge uses the VLAN ID and the destination MAC

            address to select the outbound port. The superposition of 
the outbound ports is the path for a frame.

?

Or should we do something else?
There is only one occurrence of "bridged path" in the text in brackets.

> -- Section 4
>
> Figure 2 in Section 4.1.1 introduces the Detnet service and transport 
> layers.   The associated text quickly dives into explaining the 
> functionality in those layers, but it would be helpful to start with 
> an initial high level description of each of those layers that helps 
> motivate their separation.  It appears that:
>
> - The DetNet Service layer provides timely reliable in-order DetNet 
> flow delivery to applications that use DetNet.   A DetNet flow at the 
> top of the DetNet Service layer may or may not be a compound flow at 
> the bottom of that layer.
>
> - The DetNet Transport layer insulates DetNet flows (at the bottom of 
> the DetNet Service layer) from interference by network mechanisms, 
> primarily queuing and routing, where congestion is an indication of 
> queueing interference.  A DetNet flow at the top of the DetNet 
> Transport layer may or may not be a member flow that is part of a 
> compound flow, depending on what is done in the DetNet Service layer.
>

You are right, section 4 jumps into the details without much of an 
introduction. I guess we had a kind of chicken and egg problem here. 
Figure 4 provides a higher level view than Figure 2. However, Figure 4 
captures different aspects, dives into the details of data plane 
options. So, it seems better to keep the order.

We could add some introductory text to 4.1 or 4.1.1, for instance:
The functionality needed for DetNet (Section 3) are implemented in the 
DetNet layer which comprises the DetNet service layer and the DetNet 
transport layer. The DetNet service layer functions rely on flow 
identification and sequencing. The DetNet transport layer function may 
rely on flow identification.

What do you think?

Should we also rearrange, e.g., indent the text after Figure 2 such that
Packet sequencing, Duplicate elimination, Flow replication, Flow 
merging, Packet encoding, and Packet decoding get under Service layer
and
Congestion protection and Explicit routes get under Transport layer?


> There’s a related paragraph of text between Figures 3 and 4 in Section 
> 4.1.2, but it doesn’t provide useful intuition about which 
> functionality belongs in which layer.
>
> ** Editorial
>
> -- Section 2.1
>
> “expected” seems off in the definition of reservation:
>
> reservation
>
> The set of resources allocated between a source and one or
>
> more destinations through transit nodes and subnets
>
> associated with a DetNet flow, to provide the expected DetNet
>
> Service.
>
> Suggest either “to provide the provisioned DetNet Service to that flow”
>
I like this one, will update to this one.

> or “to provide the specified DetNet Service to that flow.”
>
> -- Section 3.1
>
>    o  An upper bound on out-of-order packet delivery.  It is worth
>       noting that some DetNet applications are unable to tolerate any
>       out-of-order delivery.
>
> Is that a sequence bound or a time bound or both?   I suspect that a 
> sequence bound (maximum degree of reordering in a flow, independent of 
> flow rate) was intended.  An analogous clarification would be helpful 
> in section 3.2.2.1 .
>
It is intentionally kept high level in the architecture draft. Details 
are to be provided by the flow information model draft. Please see my 
previous mail: 
https://mailarchive.ietf.org/arch/msg/detnet/AWG2eemg_AWsvF2FwtREyd_xtC8.

> -- Section 3.2.3
>
> I don’t understand “defined” in:
>
>    Even the use of
>    redundant paths through a network defined, e.g., by [RFC6372] do not
>    eliminate the chances of packet loss.
>
> Was “e.g., as defined by [RFC6372],” intended?
>
Yes, this was the intention, will update to this one.

>    Many real-time networks rely on physical rings or chains of two-port
>    devices, with a relatively simple ring control protocol.
>
> How does a “ring control protocol” control “chains”?  Suggest: “ring 
> control protocol” -> “control protocol”
>
> Also, are paired chains between common endpoints intended, or a more 
> general notion of chains?
>
This text refers to typical factory deployment, which is chain or ring.
It seems better to keep “ring control protocol” because they are often 
very different from generic control protocols applicable to mesh networks.

Would flipping chain and ring resolve your concern?

    Many real-time networks rely on physical chains or rings of two-port

    devices, with a relatively simple ring control protocol.


>    Explicit routes can be established various ways,
>
> established various -> established in various
>
OK.

>    This is
>    irrespective of the distribution method used, also applies to service
>    protection over explicit routes.
>
> used, also -> used, and also
>
OK.

> -- Figure 5 (Section 4.1.3)
>
> The UNI acronym is introduced without expansion or definition – please 
> correct that, e.g., by defining UNI in Section 2.1.
>
DetNet-UNI is there among the definitions. It seems enough to me.

> -- Figure 9 (Section 4.7.2)
>
> I think I understand what’s going on, but the use of “add/remove” in 
> the figure should be explained in associated text.
>
What about updating the sentence above Figure 9 to:
A packet with corresponding Flow-IDs is illustrated in Figure 9, which 
also indicates where Flow-ID can be added or removed.

> Thanks, --David
>

Thank you!
Janos


> *From:*detnet [mailto:[e-mail]] *On Behalf Of *János Farkas
> *Sent:* Thursday, June 28, 2018 9:09 PM
> *To:* Lou Berger; Stewart Bryant; [e-mail] >> DetNet WG
> *Subject:* Re: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05
>
> Dear all,
>
> Off-line discussions among Lou, Stewart, and authors followed the 
> discussions to properly address the WGLC comments, including the 
> detailed comments.
>
> A new revision of the draft has been uploaded: 
> draft-ietf-detnet-architecture-06.
>
> In addition to the changes already described in this thread, the 
> following bigger changes have been made to the draft:
>
>
> *Section 2.1 Terms used in this document*
>
> Some definitions refined as suggested by the detailed comments
>
> New definitions have been added:
>
>    "allocation
>            Resources are dedicated to support a DetNet flow.  Depending
>            on an implementation, the resource may be reused by non-
>            DetNet flows when it is not used by the DetNet flow.
>
>
>    PEF     A Packet Elimination Function (PEF) eliminates duplicate
>            copies of packets to prevent excess packets flooding the
>            network or duplicate packets being sent out of the DetNet
>            domain.  PEF can be implemented by an edge node, a relay
>            node, or an end system.
>
>   PRF     A Packet Replication Function (PRF) replicates DetNet flow
>            packets and forwards them to one or more next hops in the
>            DetNet domain.  The number of packet copies sent to each next
>            hop is a DetNet flow specific parameter at the node doing the
>            replication.  PRF can be implemented by an edge node, a relay
>            node, or an end system.
>
>    PREOF   Collective name for Packet Replication, Elimination, and
>            Ordering Functions.
>
>    POF     A Packet Ordering Function (POF) re-orders packets within a
>            DetNet flow that are received out of order.  This function
>            can be implemented by an edge node, a relay node, or an end
>            system.
>
>   DetNet service proxy
>            Maps between App-flows and DetNet flows.
>
>    bridged path
>            A VLAN bridge uses the VLAN ID and the destination MAC
>            address to select the outbound port hence the path for a
>            frame."
>
>
> *Section 3.1 Primary goals defining the DetNet QoS*
>
> A new QoS aspect has been added:
>    "o  An upper bound on out-of-order packet delivery.  It is worth
>       noting that some DetNet applications are unable to tolerate any
>       out-of-order delivery."
>
>
> The 3rd paragraph on loss on page 8 after the bullet list has been 
> extended to:
>    "After congestion, the most important contributions to packet loss are
>    typically from random media errors and equipment failures.  Service
>    protection is the name for the mechanisms used by DetNet to address
>    these losses.  The mechanisms employed are constrained by the
>    requirement to meet the users' latency requirements. Packet
>    replication and elimination (Section 3.2.2) and packet encoding
>    (Section 3.2.2.3) are described in this document to provide service
>    protection; others may be found.  For instance, packet encoding can
>    be used to provide service protection against random media errors,
>    packet replication and elimination can be used to provide service
>    protection against equipment failures.  This mechanism distributes
>    the contents of DetNet flows over multiple paths in time and/or
>    space, so that the loss of some of the paths does need not cause the
>    loss of any packets."
>
>
> *3.2.2.  Service Protection*
>
> Service protection is used as a more generic term. Introductory text 
> added:
>    "Service protection aims to mitigate or eliminate packet loss due to
>    equipment failures, random media and/or memory faults. These types
>    of packet loss can be greatly reduced by spreading the data over
>    multiple disjoint forwarding paths.  Various service protection
>    methods are described in [RFC6372], e.g., 1+1 linear protection.
>    This section describes the functional details of an additional method
>    in Section 3.2.2.2, which can be implemented as described in
>    Section 3.2.2.3 or as specified in [I-D.ietf-detnet-dp-sol-mpls] in
>    order to provide 1+n hitless protection.  The appropriate service
>    protection mechanism depends on the scenario and the requirements."
>
>
> New sub-section added:
>
> "3.2.2.1.  In-Order Delivery
>
>    Out-of-order packet delivery can be a side effect of service
>    protection.  Packets delivered out-of-order impact the amount of
>    buffering needed at the destination to properly process the received
>    data.  Such packets also influence the jitter of a flow. The DetNet
>    service includes maximum allowed misordering as a constraint.  Zero
>    misordering would be a valid service constraint to reflect that the
>    end system(s) of the flow cannot tolerate any out-of-order delivery.
>    Service protection may provide a mechanism to support in-order
>    delivery."
>
>
> *3.2.2.2. Packet Replication and Elimination*
>
> New bullet added as the last one:
>    "o  The Packet Ordering Function (POF) uses the sequencing information
>       to re-order a DetNet flow's packets that are received out of
>       order."
>
> New sentence added after the bullet list:
> "The order in which a node applies the PEF and the PRF to a DetNet
> flow is implementation specific."
>
> 2nd paragraph after the bullet list has been updated to:
> "Some service protection mechanisms rely on switching from one flow to
>    another when a failure of a flow is detected. Contrarily, packet
>    replication and elimination combines the DetNet member flows sent
>    along multiple different paths, and performs a packet-by-packet
>    selection of which to discard, e.g., based on sequencing information."
>
>
> *3.2.3.  Explicit routes*
>
> Out-of-order aspect added to the first paragraph, which is about 
> distributed routing:
> "Furthermore, out-of-order
> packet delivery can be a side effect of route changes."
>
> New sentence added to the 3rd paragraph:
> "Explicit routes can be established various
> ways, e.g., with RSVP-TE [RFC3209], with Segment Routing (SR)
> [I-D.ietf-spring-segment-routing], via a Software Defined Networking
> approach [RFC7426], with IS-IS [RFC7813], etc."
>
> New paragraph added:
>    "Out-of-order packet delivery can be a side effect of distributing a
>    single flow over multiple paths especially when there is a change
>    from one path to another when combining the flow.  This is
>    irrespective of the distribution method used, also applies to service
>    protection over explicit routes.  As described in Section 3.2.2.1,
>    out-of-order packets influence the jitter of a flow and impact the
>    amount of buffering needed to process the data; therefore, DetNet
>    service includes maximum allowed misordering as a constraint.  The
>    use of explicit routes helps to provide in-order delivery because
>    there is no immediate route change with the network topology, but the
>    changes are plannable as they are between the different explicit
>    routes."
>
> *
> 4.1.1. Representative Protocol Stack Model*
>
> "Explicit routes" have been added to Figure 2 with the corresponding 
> explanation:
>  "Explicit routes
>            The DetNet transport layer provides mechanisms to ensure that
>            fixed paths are provided for DetNet flows.  These explicit
>            paths avoid the impact of network convergence."
>
>
> Section 4.11 Connected islands vs. networks of v05 has been deleted 
> because it was just a leftover from early drafts on what DetNet WG 
> should do; which are covered by the charter anyways.
>
>
> References have been cleaned up and brought up-to-date.
>
>
> Refinements have been implemented in the draft according to Lou's 
> detailed comments. They are not listed here because they are minor 
> changes.
>
>
> Best regards,
> Janos
>
> On 6/12/2018 2:48 PM, Lou Berger wrote:
>
>
>     Balázs,
>
>     Thanks for the response -- please see below.
>
>     ----------
>     On June 12, 2018 4:07:35 AM Balázs Varga A
>     <[e-mail]> <mailto:[e-mail]>
>     wrote:
>
>
>         Hi Lou, Thanks for the comments. See reactions inline.
>         Document update in progress. Cheers Bala'zs
>
>         -----Original Message-----
>         From: detnet <[e-mail]>
>         <mailto:[e-mail]> On Behalf Of Lou Berger
>         Sent: 2018. június 1. 22:42
>         To: DetNet WG <[e-mail]> <mailto:[e-mail]>;
>         [e-mail]
>         <mailto:[e-mail]>
>         Subject: [Detnet] Promised comments on
>         draft-ietf-detnet-architecture
>
>         Hi,
>
>              I have a number of high level comments on the document
>         that I'd like to  raise below.  I also have a number of more
>         editorial/specific comments that  I'd like to review directly
>         with the authors and then have them report back  on changes --
>         if any turn out to be more substantive discussions from the
>          author's perspective, I'll raise these on the list separately.
>
>     ...
>
>
>         - WRT Section 4.4.3, I think the inclusion of a distributed
>         control plane in the "Network Plane" is inconsistent with
>         other functional definitions and conflates where a function
>         resides from the actual function and that whether control is
>         implemented in a fully centralized, fully distributed or some
>         hybrid form doesn't fundamentally change  the control
>         function's role -- therefore I think the sections 4.4.2 and .3
>         should be revised accordingly
>         [Balázs Varga A] Agree in principal. Let's discuss the details.
>
>     Okay - I'll work with you off line and we can report back the
>     results/proposed changes.
>
>
>         - In several places it's not clear that DetNet service is
>         always a L3 service which is controlled using L3 identifiers,
>         i.e., IP addresses -- this is true even in the MPLS service
>         case and the TSN over MPLS case. I think this is an important
>         point to be clear on in the document.
>         [Balázs Varga A] I am not sure. DetNet service is always
>         provided over a L3 network (IP or MPLS), that is fine. However
>         the service itself can be L2 or L3. In case of TSN Ethernet
>         frames are transported, so it is a L2 service. In case of IP
>         end systems IP packets are transported so it is a L3 service.
>
>
>     Humm - While I agree that DetNet is providing an (enhanced) L2VPN
>     service, it is not itself providing control or service of L2
>     devices -- this is TSN's job. The fact that DetNet is all about
>     behavior of L3 nodes (i.e., IP and PW/MPLS) and not L2 nodes
>     (i.e., TSN bridges) is something the architecture should make
>     unambiguous.
>
>     Thanks,
>     Lou
>
>
>         Please let me know what you think.
>
>         Cheers,
>
>         Lou
>
>
>         _______________________________________________
>         detnet mailing list
>         [e-mail] <mailto:[e-mail]>
>         https://www.ietf.org/mailman/listinfo/detnet
>         _______________________________________________
>         detnet mailing list
>         [e-mail] <mailto:[e-mail]>
>         https://www.ietf.org/mailman/listinfo/detnet
>
>
>
>
> On 6/12/2018 6:27 PM, Stewart Bryant wrote:
>
>     Dear Bala'zs
>
>     Thank you your for your consideration of these points. I will just
>     pick up a few that need some further thought:
>
>
>
>             DetNet transit node
>
>                    A node operating at the DetNet transport layer,
>         that utilizes
>
>                    link layer and/or network layer switching across
>         multiple
>
>                    links and/or sub-networks to provide paths for
>         DetNet service
>
>                    layer functions. Optionally provides congestion
>         protection
>
>                    over those paths.  An MPLS LSR is an example of a
>         DetNet
>
>                    transit node.
>
>         SB> In that example it would have to be a DetNet enable/aware
>         LSR. An
>
>         SB> ordinary LSR would not know anything about DetNet.
>
>         [Balázs Varga A] No, A DetNet aware LSR would be a relay node
>         (S-PE).
>
>
>     I think the confusion is what "DetNet Transport Layer" means. This
>     technology touches on Transport Layer in  the L4 sense, and the
>     Transport Network Layer as in the packet network that carries
>     L3 packets.
>
>     So I think that we need to clarify whether a DetNet transit node
>     is an S-PE (i.e. a a transit node in the DetNet layer), or a P node
>     (i.e. a transit node that is carrying DetNet packets but could be
>     carrying any sort of MPLS packet)
>
>
>         ============
>
>            These three techniques can be applied independently, giving
>         eight
>
>            possible combinations, including none (no DetNet), although
>         some
>
>            combinations are of wider utility than others.  This
>         separation keeps
>
>            the protocol stack coherent and maximizes interoperability with
>
>            existing and developing standards in this (IETF) and other
>         Standards
>
>            Development Organizations. Some examples of typical expected
>
>            combinations:
>
>            o  Explicit routes plus service protection are exactly the
>         techniques
>
>               employed by [HSR-PRP]. Explicit routes are achieved by
>         limiting
>
>               the physical topology of the network, and the
>         sequentialization,
>
>               replication, and duplicate elimination are facilitated
>         by packet
>
>               tags added at the front or the end of Ethernet frames.
>
>         SB> ER can be done virtually as well as physically. RSVP is a
>         type of
>
>         SB> ER, as is Adj-SID based SR, and we can design other types.
>
>         [Balázs Varga A] Agree, but these are examples. Statement is
>         for HSR-PRP.
>
>     So the question is whether we should expand the set of examples,
>     particularly
>     to more accessible ones.
>
>     ============
>
>                             Packet replication and elimination
>
>                         > > > > > > > > > relay > > > > > > > >
>
>                        > /------------+ R node E +------------\ >
>
>                       > /                  v + ^               \ >
>
>               end    R +                   v | ^                + E end
>
>               system +                   v | ^                +   system
>
>                       > \                  v + ^               / >
>
>                        > \------------+ R relay E +-----------/ >
>
>                         > > > > > > > > >  node > > > > > > > >
>
>         Figure 1
>
>            Packet replication and elimination does not react to and
>         correct
>
>            failures; it is entirely passive.  Thus, intermittent failures,
>
>         SB> I think it copes with intermittent failures OK.
>
>         [Balázs Varga A] Yes, PRF and PEF can eliminate the effect of
>         such failures. But text
>
>         intends to say that it is passive. It is not reacting to such
>         failures. No change in text.
>
>
>     You say that PREF does not correct failures. I would contend that
>     is exactly
>     what it does. Sure it does not react but it does correct, and it does
>     address intermittent failures.
>
>         ===========
>
>            transported between the peer end systems.  Therefore, the
>         following
>
>            possible types / formats of a DetNet flow are distinguished
>         in this
>
>            document:
>
>            o  App-flow: native format of a DetNet flow.  It does not
>         contain any
>
>               DetNet related attributes.
>
>            o  DetNet-t-flow: specific format of a DetNet flow.  Only
>         requires
>
>               the congestion / latency features provided by the Detnet
>         transport
>
>               layer.
>
>            o  DetNet-s-flow: specific format of a DetNet flow.  Only
>         requires
>
>               the replication/elimination feature ensured by the
>         DetNet service
>
>               layer.
>
>            o  DetNet-st-flow: specific format of a DetNet flow.  It
>         requires
>
>               both DetNet service layer and DetNet transport layer
>         functions
>
>               during forwarding.
>
>         SB> I find the relisting of these types confusing. Wheren't
>         they defined
>
>         SB> in the section above?
>
>         [Balázs Varga A] This is inline with the previous section "
>         Grouping of end systems ".
>
>     Surely if we have defined them we never need to do anything but
>     name them in
>     later sections. Redefinition is never a good idea because it often
>     leads to
>     conflicting definitions.
>
>
>         ============
>
>            [HSR-PRP]  IEC, "High availability seamless redundancy
>         (HSR) is a
>
>                       further development of the PRP approach,
>         although HSR
>
>                       functions primarily as a protocol for creating media
>
>                       redundancy while PRP, as described in the previous
>
>                       section, creates network redundancy.  PRP and
>         HSR are both
>
>                       described in the IEC 62439 3 standard.",
>
>                       <http://webstore.iec.ch/webstore/webstore.nsf/
>
>         artnum/046615!opendocument>.
>
>         SB> Not available at the time of this review, but my
>         recollection is
>
>         SB> that this is not readily available without paying a large fee.
>
>
>     If we decide that this is essential as a key reference, there
>     needs to be
>     some suitable text, as this will get raised a number of times between
>     here an publication as first the directorates and then the ADs run
>     into this.
>
>
>         ===========
>
>            [ISA95]    ANSI/ISA, "Enterprise-Control System Integration
>         Part 1:
>
>                       Models and Terminology", 2000,
>
>                       <https://www.isa.org/isa95/>.
>
>         SB> Should that be 2000, or 2010.
>
>         SB> Note that this seems to be a very expensive document to
>         access.
>
>     You did not comment on the correctness of the reference.
>
>
>     Best Regards
>
>     Stewart
>