Re: [Detnet] Some initial comments on draft-dt-detnet-dp-sol-00

[Balázs Varga A <[e-mail]>]

Hi Stewart,
Further feedback on your detailed notes inline in your original mail.
Cheers
Bala’zs

From: Balázs Varga A
Sent: 2017. június 6. 18:51
To: 'Stewart Bryant' <[e-mail]>; [e-mail]; [e-mail]
Subject: RE: Some initial comments on draft-dt-detnet-dp-sol-00

Hi Stewart,

Thanks for the detailed review.

Based on several good comments in the Chicago meeting we have started to update the draft.
The major changes are:
1, changing encapsulation
- PW-based encapsulation: The PW-based data plane can be run over an MPLS PSN.
- Native-IP encapsulation: This solution is based on IP header fields, namely on IPv6 Flow label
and a new DetNet Control Word extension header option. It is targeted for native IPv6 networks.
2, PW-label specific requirement is simplified (regarding uniqueness):
- DetNet flows that need to undergo PREF processing MUST have the same PW Label when
they arrive at the DA-*-PE node.

We have also identified that sequence numbering related functions may be challenging, however
it is a must for the PREF (Packet Replication and Elimination Function) implementation. PREF provides
per packet level redundancy and not the so far defined per-PW-segment redundancy.

Similarly, your concerns on constraining DetNet packets to a path and an interface in a pure IP network,
is absolutely a candidate topic for further improvements.

I will be back soon with further feedback regarding your detailed notes.

Thanks & Cheers
Bala’zs

From: Stewart Bryant [mailto:[e-mail]]
Sent: 2017. június 5. 22:02
To: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>
Subject: Some initial comments on draft-dt-detnet-dp-sol-00


I took an initial look at the detnet dp document.



I think that PWs are certainly a good way to go, but I am concerned

about a number of aspects of the proposal.



My biggest concern is doing the sequence number checking. There have

not been many attempts to do this, and from work that we have done in

the OAM space all the feedback I get is that this is a hard problem.

It might be useful if you constrained the packets to a path and an

interface using MPLS-TP or MPLS-TE, however I don't see a practical

way to do that in a pure IP network. You could of course do it

with SR, but that is v6 only, and the packets could potentially need

a very large SR header.



Please see the notes inline.



-Stewart





   The PW-based data plane can be run over either an IP

   or MPLS [RFC4448][RFC6658] Packet Switched Network (PSN).



SB> I comment on this in detail later



============





5.1.  DetNet Control Word



   The DetNet control word (d-CW) is identical to the control word

   defined for Ethernet over MPLS networks in [RFC4448].  The DetNet

   control word is illustrated in Figure 4.



      0                   1                   2                   3

      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     |0 0 0 0|  reserved - set to 0  |   16 bit Sequence Number      |

     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+





                       Figure 4: DetNet Control Word



   [Editor's note: Shoudl we care about high speed links, here 16 bits

   of sequence number wraps fast?  For example, in a case of 100Gb/s

   link, 16 bits of sequence number will wrap in ~6.6ms assuming 1250

   octets of packets and ~3.3ms for 625 octets packets.  Both numbers

   mean quite long fiber distances, though.]



SB> This worries me. Doing read modify write on the sequence number is

SB> difficult in the general case, particularly without path constraint

SB> since a packet can arrive on any interface, and this interface can

SB> change.

SB>

SB> Are you thinking that there will be so few DN PWs that you can

SB> put the counters in registers? That might fly at the T-PEs, but

SB> I would be worried at the S-PEs.

SB>

SB> BTW shouldn't you consider much smaller packets, or do you imagine

SB> that DN will be constrained to applications using large packets.



[Balázs Varga A]

Yes, Doing read modify write on the sequence number is difficult in the general case.

However here we do write only at ingress node (e.g., T-PE). No modify operation defined.

Read operation is needed on S-PE nodes, doing replication and elimination and they need

a counter per flow. That is the tax of per packet redundancy. Not all flow may require PREF

function of DetNet. Resources have to be considered during flow setup.



===========



5.3.  DetNet encapsulation



   The DetNet data plane follows PW encapsulation.  This document

   specifies a single encapsulation that can be used over both MPLS and

   IP packet switched Networks (PSN).  The DetNet data plane

   encapsulation consists of a



   o  DetNet control word (d-CW): contains sequencing information for

      packet replication and duplicate elimination purposes.  There is a

      separate sequence number space per each DetNet label.



SB> Do you mean per DetNet flow ID, or per PW label?



[Balázs Varga A]

In case of MPLS DetNet flow ID = PW label.



 o  DetNet flow-ID (f-ID): uniquely identifies a DetNet flow within a

      DetNet network.  Multiple DetNet PWs with different PW labels may

      have the same f-ID, which then implies the PWs are actually

      subflows of one compound flow.



SB> I am not sure I understand the definition of a DetNet network yet.

SB> I presume that it is Fig 5 from the architecture draft, which

SB> is a tunnel between two service instances.

SB> However I am having difficulty understanding the scope of the

SB> uniqueness. It sounds as if it needs to be unique between a

SB> pair of service instances is that the case, or does the

SB> uniqueness have greater scope?

SB> PWs are not subflows, they are "A mechanism that carries the

SB> essential elements of an emulated service from one PE to one or

SB> more other PEs over a PSN."



[Balázs Varga A]

the scope of the uniqueness is that it needs to be unique between a pair of service instances.

[Balázs Varga A]

In DetNet, PWs are used to transport the subflows (replica flows) of a compound flow. Inline

with your quoted definition.



 o  PseudoWire Label (PW Label;): a standard PW label that identifies

      a PW Instance within a (DA-)T-PE or (DA-)S-PE device.



SB> Just so we are all clear the PW label changes at S-PEs



[Balázs Varga A]

Yes, the PW label may change.



===========



 o  DetNet topology overlay label (L-label): an optional label used

      between (DA-)T-PE or (DA-)S-PE nodes.  The main use of L-labels is

      to tunnel PWs through a PE node and therefore effectively making a

      PE node to behave like a P node.



SB> This needs more thought. The reason that S-PEs were created

SB> was to minimise the burden of running PWs between different

SB> administrative domains. To make this feasible it was necessary

SB> for the T-PEs to allocate their own PW label and have the

SB> S-PEs swap, that way only the boundary nodes (S-PEs) needed

SB> be worried about the mapping between the PE identity and the

SB> PW label in the data-plane.

SB> If a data-plane identifier is used, then we don't really need

SB> S-PEs as such. So I think that we have to define the new

SB> purpose of the S-PE more clearly when they are used for Detnet.



[Balázs Varga A]

There is dedicated chapter in the new data-plane draft version to describe in more detail S-PE

nodes. I think that will be a good starting point for further clarification.





 In a case of MPLS-based PSN, the tunnel labels between LSRs are

   referred as T-labels.



SB> I think that they are really LSP labels.

[Balázs Varga A]

Yes, they are LSP labels.



 The DetNet CW and the Detnet flow-ID together constitute the DetNet

   PseudoWire encapsulation header.



      [Editor's note: The current design has the DetNet flow-ID as part

      of the every DetNet flow packet.  The flow-ID identifies the flow

      uniquely within the DetNet network and together with the sequence

      number information from the DetNet control word is used for PREF

      purposes.  The flow-ID makes is easy for the DA-*-PE node to

      associate different PWs into one compound flow and perform the

      elimination of duplicate packets.



SB> I am not sure this is needed. The DA-*-PE knows the relationship

SB> between the PW labels and can make the decision based on that.

[Balázs Varga A]

That is a point where the design team had a long discussion. Table lookups were calculated and

resulted in this note. I agree it may be implementation specific, so further details would be great

to discuss.





      The flow-ID would point at the

      node internal construct that holds the received packet history for







Korhonen, et al.       Expires September 14, 2017               [Page 8]



Internet-Draft         DetNet data plane solution             March 2017





      each DetNet flow of interest.  However, it could also be possible

      to associate multiple PWs into one DetNet flow just using the

      control plane provided information.  In this case different PWs

      (using any PW label) would be mapped internally within a node to a

      local-ID (or similar construct), which again points at the

      internal per DetNet flow received packets history construct.



SB> I think you have to do this anyway



      The

      explicit in-band flow-ID is easy from the processing and control

      plane point of view.

SB> Isn't this a bigger change to the forwarder? What normally

SB> happens is you vector to the instructions and context using

SB> the PW label as the identifier. So I think the flow-ID just adds

SB> complexity. Given that it needs to be unique and known at

SB> each PW node on the path, I do not see what has been gained in

SB> terms of reduction in control plane activity.



[Balázs Varga A]

That would be worth for further discussions with chip vendors onboard.





      The local-ID approach does not need the in-

      band information (thus has less overhead) but requires more from

      the control plane and the mapping information has to be stored

      into the LFIB.  Current design decision is the in-band flow-ID but

      may be changed to local-ID if there is a strong reason to do the

      change.]



   Figure 6 illustrates a DetNet PseudoWire encapsulation using an MPLS

   PSN.  Similarly, Figure 7 illustrates the DetNet PseudoWire

   encapsulation when IP PSN is used.  The encapsulation is uniform

   above the PSN.



   Depending on the network topology the "overlay label" (L-label) may

   be part of the label stack.  The L-label tunnels guarantee PW labels

   remain unchanged between DA-*-PE nodes.

SB> Please see earlier



   Furthermore, L-labels

   tunnels allow selectively exposing the PW label to DA-*-PE nodes,

   which means some overlay topologies may just pass through specific

   DA-S-PEs without any DetNet specific processing.



SB> They can do this anyway. At an SPE we normally just swap the

SB> PW label, and other than for OAM reasons (trapped by TTL expiry)

SB> we do no processing. So if you do not teach an SPE that the PW

SB> is to be processed, it will just pass through.

SB>

SB> I think that you need to look seriously at deleting this component

SB> of your design and building something much closer to a normal

SB> PW design.



[Balázs Varga A]

This component is optional. There are some scenarios when it can be useful.

E.g., on Flow identification for OAM purposes. We have discussed some troubleshooting scenario,

where it was necessary on a link to being able to distinguish which DetNet packet belongs to which

sub-flow of a compound flow.



Example:

sub-flow1: A --> D

sub-flow2: B --> D



So on link C-D we can NOT distinguish packets of the two sub-flow, as they have the same PW-label.



A--+

       \

        C-----D

       /

B--+



One possible solution is the use of the "Topology overlay Label".





===============



   When IP PSN is used, the label stack it transports is only inspected

   when the IP packet destination address equals to the IP address of a

   DA-*-PE or a P node.  Essentially there are one more IP tunnels

   between a number of DA-*-PE and/or P nodes.  The LFIB and the

   forwarding information base (FIB) combination determines whether a PW

   gets terminated at the node or forwarded to another node within a new

   IP tunnel.



SB> So, setting aside for the moment the work that we are doing on

SB> unifying SRv6 and MPLS SR, work which as yet has no official status

SB> you should understand that there really is no deployment of

SB> MPLS-PW over IP. All of the deployed PWs are either pure PW over

SB> MPLS, or LT2Pv3. L2TPv3 has no concept of an S-PE.

SB>

SB> Now I think the unified approach is the right one, but so far

SB> there are no real specifications.

SB>

SB> Also you need to think about whether you want ECMP or not, because

SB> if you do you really do need the interstitial UDP layer (RFC7510)

SB> shown below.



[Balázs Varga A]

This is fixed now in the new version of the draft. See my previous mail.



==============





6.1.  Forwarded clarifications



   [Editor's note: The Detnet-aware "extended forwarder" does the heavy

   lifting on maintaining the sequence numbers associated with the

   DetNet labels.  Extended forwarder is also responsible for packet

   replication and duplicate elimination.  See the excerpt from RFC3985

   Section 4.2.1. about forwarder's functions.  We extend that to PREF:



      Some applications have to forward payload elements selectively

      from one or more ACs to one or more PWs.  In such cases, there

      will also be a need to perform the inverse function on PWE3-PDUs

      received by a PE from the PSN.  This is the function of the

      forwarder.



   ]

SB> But note that forwarder function only appears in the T-PE, we

SB> never included it in the S-PE which can be better thought of

SB> as a two layer MPLS switch - it's not how it works although

SB> a modern two label lookup MPLS system could do it like that

SB> but all that happens in an SPE is that you swap two labels

SB> simultaneously - the LSP label and the PW label.





    The DetNet specific new functionality in a DA-*-PE PW processing is

   the packet replication and duplication elimination function (PREF).

   This functional is a part of the "extended" forwarder.  The PREF

   processing is triggered by the LFIB actions i.e., not all PWs receive







Korhonen, et al.       Expires September 14, 2017              [Page 11]



Internet-Draft         DetNet data plane solution             March 2017





   DetNet specific processing.  Basically the LFIB has to be extended

   with a "PREF enabled" boolean configuration switch that is associated

   with the normal label actions (e.g., swap, push, pop, ..).  The

   output of the PREF elimination function is always a single packet.

   The output of the PREF replication function is always one or more

   packet (i.e., 1:M replication).  The replicated packets MUST share

   the same DetNet PW control word sequence number and flow identity

   word flow-id.



   The complex part of the DetNet PREF processing is tracking the

   history of received packets for multiple PWs.  These PWs do not have

   the same PW label value while they still share the same PW sequence

   number counter and the history information.  That is where the DetNet

   encapsulation header flow-ID plays an important role and binds the

   control word sequence number to the flow specific shared counter and

   history information within the PREF function.



SB> That is certainly one way of doing it, although given that you need

SB> to provision the PW anyway, I an not sure it is needed.



[Balázs Varga A]

This is fixed in the new version. PWs of the same compound flow have the same PW label.



   The DetNet flow word contains a D flag bit (see Section 5.2), which

   makes the DA-*-PE node aware of the direction the flow-ID arrived

   from.  If the node, based on the local policy, checks for the D bit

   setting that effectively means the sequence number history has to

   contain also the D bit information.



SB> I am really not sure why you need the D bit a PW received on a given PW

SB> label only goes one way.



[Balázs Varga A]

If I remember correctly the scenario was for rings with clockwise and anticlockwise directions.





 ==============











7.  Other DetNet considerations



7.1.  Class of Service



   [Editor's note: Discuss the CoS.. and how that is archived when using

   MPLS or IP PSN.]





SB> Don't all your packets need to go on the highest class of service?



[Balázs Varga A]

It is scenario and network operator specific. One may decide that some OAM traffic may have higher

precedence than anything else.



===========



7.3.  Time synchronization





   o  PTP with on-path support: in this approach PTP packets are sent as

      DetNet flows, and intermediate nodes take part in the protocol as

      Transparent Clocks or Boundary Clocks [IEEE1588].  The on-path PTP

      support by intermediate nodes provides a higher degree of accuracy

      than the previous approach.  The actual accuracy depends on

      whether all intermediate nodes are PTP-capable, or only a subset

      of them.



SB> RFC8169 shows how to do TC in an MPLS network.

SB> I am not sure anyone knows how to do this in an IP network.

[Balázs Varga A]

I think for IP hop-by-hop PTP support would be preferred.





===========



7.4.  Bidirectional traffic



   Some DetNet applications generate bidirectional traffic and may

   require symmetric flows.  There are already mechanisms that can be

   used to create bidirectional tunnels at the transport network level,

   such as MPLS-TP.  The data plane solution SHOULD allow establishing

   bidirectional symmetric flows.  Control plane mechanisms would need

   to also support this, though this is out of scope of this document.

   [Summary of existing mechanisms to create bidirectional tunnels that

   can be used.]



SB> PWs are always bidirectional of course.



[Balázs Varga A]

Yes.



 8.1.  PW Label assignment and distribution



   The PW label distribution follows the same mechanisms specified for

   MS-PW [RFC6073].



SB> This will need extensions to support DETNET



[Balázs Varga A]

Yes.