Re: [Detnet] consolidated response: IP Data Plane draft comments

["Black, David" <[e-mail]>]

Lou,

Rather than continue the nested inline, I'll try to write a high-level commentary starting from this helpful goal summary:

> And I believe we are trying to ensure that the that the DetNet data plane
> can be supported on the widest variety of existing and future hardware.
> Which to me means not introducing any restrictions that don't exist today
> or overly specify behaviors not required by detnet.

In my view, that also means not departing significantly from the network architecture on which that hardware is based.  The underlying network architecture aspect that matters here is that both DSCP and ECN are intended to provide multiple markings within individual flows, not subdivide flows into sub-flows.  In more detail ...

-- DSCP --

The proposal to use DSCP for path selection is certainly "creative" :-), as Diffserv is primarily concerned with forwarding behaviors, see RFC 2475 (An Architecture for Differentiated Services).  That said, in light of the warnings in RFC 7657 (Differentiated Services (Diffserv) and Real-Time Communication) about use of multiple DSCPs, suitable "there be dragons here" warnings, i.e., "SHOULD NOT" language with explanations of what can go wrong, ought to likewise suffice for DetNet.

An important RFC 7657 warning (in Section 6) to keep in mind is:

   o  Cannot rely on end-to-end preservation of DSCPs as network node
      remarking can change DSCPs and remove drop precedence distinctions
      (see Section 3.2).  For example, if a source uses drop precedence
      distinctions within an AF class to identify different types of
      video frames, using those DSCP values at the receiver to identify
      frame type is inherently unreliable.

There is a lot of deployed network equipment that remarks DSCPs, so relying on DSCPs to identify DetNet flows (or even DetNet traffic), and especially to distinguish DetNet traffic from non-DetNet traffic, has potentially problematic hardware interactions.  In contrast, if any 5-tuple is either all-DetNet or all-not-DetNet, network DSCP remarking ought not to result in moving DetNet traffic into a non-DetNet data plane at network nodes or moving non-DetNet traffic into a DetNet data plane, both of which look like robustness benefits for DetNet.  There is also an analogous robustness benefit for DetNet network edge filtering that prevents DetNet IP traffic from escaping into non-DetNet networks - if the filters are 5-tuple-based, DSCP remarking ought not to result in DetNet traffic bypassing an edge filter that is intended to block Detnet traffic.

Overall, this DSCP topic still feels like it merits "SHOULD/SHOULD NOT" requirements, not "MUST/MUST NOT" requirements (reminder: what I'm looking for is roughly - SHOULD NOT mix DetNet and non-DetNet traffic within a single 5-tuple).

OTOH ...

-- ECN --

ECN values are used to label packets in a single transport protocol session, not to multiplex transport protocol sessions, period.   The proposal to use ECN for path selection is fundamentally inconsistent with RFC 3168 (The Addition of Explicit Congestion Notification (ECN) to IP), a long-time Proposed Standard.  Taking up the example mentioned below:

> Although, if a system is smart enough to route ecn enabled packets over ecn
> supporting routers I'm not sure why this would be a bad thing. But if the
> restriction already exists, it exists...

It would definitely be a bad thing if it is based on the value of the ECN field in the IP header.  One problem arises from the requirement (in Section 6.1.5 of RFC 3168) that when a TCP session uses ECN, all retransmitted packets have to be marked with Not-ECT, i.e., as not using ECN.   As a result, sending ECN and non-ECN traffic on different network paths invites reordering of packets in a TCP session, which is known to be problematic, e.g., if retransmitted packets are subject to more delay than non-retransmitted packets.  There's much more where this explanation came from (please do read RFC 3168).  The upshot is that DetNet needs to prohibit use of ECN for path selection, i.e., "MUST NOT" language that cites RFC 3168.

For completeness, I should not that there is experimental work in progress to support ECN for retransmitted packets, e.g., see RFC 8311 (Relaxing Restrictions on Explicit Congestion Notification (ECN) Experimentation), but there is also plenty of "running code" that does not use ECN for retransmitted packets in traffic that otherwise uses ECN.

Thanks, --David (co-author of RFCs 2475, 3168, 7657 & 8311)

> -----Original Message-----
> From: Lou Berger <[e-mail]>
> Sent: Friday, October 11, 2019 7:00 AM
> To: Black, David
> Cc: [e-mail]
> Subject: RE: [Detnet] consolidated response: IP Data Plane draft comments
> 
> 
> [EXTERNAL EMAIL]
> 
> David,
> 
> 
> ----------
> On October 10, 2019 11:13:54 AM "Black, David" <[e-mail]>
> wrote:
> 
> > Lou,
> >
> >> I think there's some cross usage of the term flow here.   DetNet flow
> >> identification is the traffic classification function that enables
> >> DetNet to deliver different service, including paths through a network
> >> to different flows.  ECN markings already impact how packets are handled
> >> in ECN enabled routers (DROP for 00 for DE packets).
> >
> > I'm drawing a bright-line distinction between different service in network
> > nodes (both DSCP and ECN are used for this purpose) and selection of
> paths.
> >
> 
> And I believe we are trying to ensure that the that the DetNet data plane
> can be supported on the widest variety of existing and future hardware.
> Which to me means not introducing any restrictions that don't exist today
> or overly specify behaviors not required by detnet.
> 
> > In other words, if "including paths through the network" were removed
> from
> > the above text snippet, I would strongly agree.
> >
> >> Also, I don't see
> >> anything in the RFCs that would preclude path selection based on
> >> different DSCPs or different ECN field values
> >
> > I firmly disagree on those two:
> > DSCP:- Selection of different paths based on DSCP is problematic for
> > network operations and management tools.
> 
> I don't disagree, but we don't prevent  operators to 'play with scissors'.
> Sometimes such flexibility even results in new capabilities.
> 
> Adding text that points this out seems completely reasonable.
> 
> > ECN: I believe that different path selection based on ECN values is
> > effectively prohibited by RFC 3168, and is completely inconsistent with
> > prior and ongoing Transport Area work.
> >
> 
> Where is this prohibition stated? Referencing existing specification
> certainly makes sense.
> 
> Although, if a system is smart enough to route ecn enabled packets over ecn
> supporting routers I'm not sure why this would be a bad thing. But if the
> restriction already exists, it exists...
> 
> > To summarize what I think ought to be done:
> > - Path selection based on DSCP ought to be a "SHOULD NOT" requirement.
> RFC
> > 7657 is relevant to this.
> 
> 
> > - Path selection based on ECN has to be a "MUST NOT" requirement - see
> RFC
> > 3168.
> >
> >> As a possible example, I could see ECT and non-ect
> >> traffic routed over different paths and am not sure where this is
> >> precluded by the reference RFCs.
> >
> > Please read RFC 3168 to better understand why that is a seriously bad idea.
> >
> 
> Perhaps this is a good discussion point for Monday's call.
> 
> >> > Is it at least possible to prohibit (or at least
> >> > strongly discourage, i.e., "SHOULD NOT") use of the same 5-tuple for
> DetNet
> >> > and non-DetNet traffic?  That would at least contain the resulting
> network
> >> > management and operational effects to DetNet?
> >>
> >> I'm not sure how we can preclude what operators choose to do.  Vendors
> >> on the other hand...
> >
> > I'm not looking for "preclude" which would be expressed as "MUST NOT."
> > What I'm looking for is a strong "here be dragons" "SHOULD NOT" warning
> > about mixing DetNet and non-DetNet traffic on the same 5-tuple (e.g., that
> > sure looks like an opportunity to misdirect DetNet OAM traffic into
> > non-DetNet paths and forwarding planes).  Then operators who choose to
> do
> > otherwise have only themselves to credit for the consequences.
> >
> 
> Okay. I'll work on some language for this and send it to the list before
> Monday's call.
> 
> >> > Also, be aware that if two DetNet flows differ only in DSCP, most
> existing
> >> > network transport protocols will have problems dealing with that
> situation
> >> > - see RFC 7657 for details.
> >>
> >> sure.  but then it should be using different DSCP values.
> >
> > I'm not sure what "it" refers to, but if that was meant to imply that
> > existing network transport protocols are not or never appropriate for use
> > with DetNet, then I would strongly disagree.
> >
> > The rest of the context is that in end systems, DSCP is not used to choose
> > the "socket" or equivalent - traffic is muxed/demuxed to/from transport
> > protocol sessions/instances based on 5-tuple (not including DSCP) with
> > DSCPs being used to label flows, but not to subdivide any 5-tuple flow into
> > multiple 6-tuple sub-flows.   If splitting a 5-tuple flow into multiple
> > sub-flows is what's wanted, then the WebRTC example ought to be
> followed
> > where header info above the IP header (and above the UDP header) is
> used to
> > do the mux/demux so that multiple WebRTC sub-flows can use the same 5-
> tuple.
> >
> 
> I hear you and agree you certainly wouldn't expect this to happen for a
> single 5-tuple.  This said,  I have seen some real networks, and suspect
> there are more, that route traffic differently based on their dscps - and
> this is nothing more than we are allowing for in detnet.
> 
> I look forward to talking to you on Monday
> 
> Lou
> > Thanks, --David
> >
> >> -----Original Message-----
> >> From: detnet <[e-mail]> On Behalf Of Lou Berger
> >> Sent: Wednesday, October 9, 2019 10:42 PM
> >> To: Black, David
> >> Cc: [e-mail]
> >> Subject: Re: [Detnet] consolidated response: IP Data Plane draft
> comments
> >>
> >>
> >> [EXTERNAL EMAIL]
> >>
> >> Hi David,
> >>
> >> Thanks for the response, see below.
> >>
> >>
> >> ----------
> >> On October 8, 2019 10:04:00 AM "Black, David" <[e-mail]>
> >> wrote:
> >>
> >> > Lou,
> >> >
> >> >> Please let me know if this addresses all your comments in an
> acceptable
> >> >> fashion or if you'd like to discuss further.
> >> >
> >> > The proposed text does not address the comments.
> >> >
> >> > [A] ECN
> >> >
> >> > The proposed text effectively ignores the comments, and fails to
> address
> >> > the problem.
> >> >
> >> > The ECN field MUST NOT be used for flow identification, period (e.g.,
> see
> >> > RFC 3290, which does not include ECN in the 6-tuple).  In particular, it
> >> > would be wrong for two DetNet flows to differ solely on the values used
> in
> >> > the ECN field.  If there's a disagreement, please cite text from RFC 3168
> >> > that justifies this proposed ECN field usage.
> >> >
> >>
> >> I think there's some cross usage of the term flow here.   DetNet flow
> >> identification is the traffic classification function that enables
> >> DetNet to deliver different service, including paths through a network
> >> to different flows.  ECN markings already impact how packets are handled
> >> in ECN enabled routers (DROP for 00 for DE packets).  Also, I don't see
> >> anything in the RFCs that would preclude path selection based on
> >> different DSCPs or different ECN field values -- It is even covered
> >> under 'split paths' albeit from the perspective of a broken
> >> implementation.  As a possible example, I could see ECT and non-ect
> >> traffic routed over different paths and am not sure where this is
> >> precluded by the reference RFCs.
> >>
> >>
> >> > [B] 5-tuples and 6-tuples
> >> >
> >> > As I noted in one of the referenced messages in response to a question:
> >> >
> >> >> > Are you okay with dropping "While 5-tuples suffice to identify
> DetNet
> >> flows,"
> >> >> Not by itself - if that text is dropped, I'd want to add a couple of
> >> sentences
> >> >> (perhaps not in Section 3) to explain how DetNet flows differ from
> >> Diffserv
> >> >> flows,
> >> >> i.e., "microflows" in defined in RFC 2475 (Diffserv Architecture).
> >> >
> >> > The heads up for everyone is that if multiple DetNet flows differ only in
> >> > DSCP (i.e., IP addresses, transport protocol and ports are the same),
> then
> >> > non-DetNet network management tools that deal with traffic flows will
> >> lump
> >> > those flows together.  Is it at least possible to prohibit (or at least
> >> > strongly discourage, i.e., "SHOULD NOT") use of the same 5-tuple for
> >> DetNet
> >> > and non-DetNet traffic?  That would at least contain the resulting
> network
> >> > management and operational effects to DetNet?
> >>
> >> I'm not sure how we can preclude what operators choose to do.  Vendors
> >> on the other hand...
> >>
> >> How about adding:
> >>
> >> It is important to note that the DSCP values used within a DetNet domain
> >> may not be meaningful outside of that DetNet domain as the values may
> >> have no relationship to standard PHB and DSCP values.  It is REQUIRED
> >> that routers that operate in both DetNet and non-DetNet domains
> support
> >> remapping or setting the DSCP value of any flow originating in a DetNet
> >> domain.
> >>
> >>
> >> >
> >> > The use of DSCP for flow identification makes the OAM problem much
> >> harder
> >> > for the IP data plane by dramatically increasing the difficulty of getting
> >> > DetNet IP OAM traffic to fate-share with the DetNet traffic for which
> the
> >> > OAM functionality is intended (it may be necessary to resort to shoving
> an
> >> > MPLS label in there to identify OAM traffic and requiring nodes that
> >> > originate or terminate DetNet OAM packets to understand enough of
> >> MPLS to
> >> > deal with this).
> >> For the MPLS data plane, I suspect this will be required.  For IP, given
> >> the complexities of ECMP, I don't see how this behavior notably
> >> complicates OAM, but also agree it will need to be considered.
> >>
> >> >
> >> > Also, be aware that if two DetNet flows differ only in DSCP, most
> existing
> >> > network transport protocols will have problems dealing with that
> situation
> >> > - see RFC 7657 for details.
> >>
> >> sure.  but then it should be using different DSCP values.
> >>
> >> Lou
> >>
> >> PS WRT your offer to join one of the data plane calls, that would be great!
> >>
> >> >
> >> > Thanks, --David
> >> >
> >> >> -----Original Message-----
> >> >> From: Lou Berger <[e-mail]>
> >> >> Sent: Sunday, October 6, 2019 4:47 PM
> >> >> To: Black, David
> >> >> Cc: [e-mail]
> >> >> Subject: consolidated response: IP Data Plane draft comments
> >> >>
> >> >>
> >> >> [EXTERNAL EMAIL]
> >> >>
> >> >> David,
> >> >>
> >> >>      My apologies on the time to get back to this topic.  Before the
> >> >> last IETF you raised a number of points on the IP data plane document.
> >> >> For context, I believe the threads are found in:
> >> >>
> >>
> https://mailarchive.ietf.org/arch/msg/detnet/ykpI6gbH2L705BBjmHbnsZraxi
> >> g
> >> >>
> >>
> https://mailarchive.ietf.org/arch/msg/detnet/phi4OBWijlp7T4_HML2YvWfoa
> >> as
> >> >>
> >>
> https://mailarchive.ietf.org/arch/msg/detnet/th2Yj5Z4_8VtBsEOF1UfLejMh
> >> pM
> >> >> https://mailarchive.ietf.org/arch/msg/detnet/qvs-
> >> S0Nrm1B8w8JYO7zDLFwpTFQ
> >> >>
> >> >> In re reviewing the discussions, I think we missed the ECN field. Based
> >> >> on this and a general reread, I have put together a proposed set of
> >> >> changes to the published draft.  The changes are  available at
> >> >>
> >> >> https://github.com/detnet-wg/data-plane-
> >> >> drafts/commit/d3a733556804f11abbedbd128e3c62786418ddc8
> >> >>
> >> >> (text format can be received at
> >> >> https://xml2rfc.tools.ietf.org/cgi-
> >> >> bin/xml2rfc.cgi?url=https://raw.githubusercontent.com/detnet-
> wg/data-
> >> >> plane-drafts/working/lb/ip-tos-update/ip/draft-ietf-detnet-ip.xml)
> >> >>
> >> >> the following changes are proposed:
> >> >> OLD183,195c185,202
> >> >> <    The DetNet IP data plane uses "6-tuple" based flow identification,
> >> >> <    where 6-tuple refers to information carried in IP and higher layer
> >> >> <    protocol headers.  The 6-tuple referred to in this document is the
> >> >> <    same as that defined in [RFC3290].  Specifically 6-tuple is
> >> >> <    (destination address, source address, IP protocol, source port,
> >> >> <    destination port, and differentiated services (DiffServ) code point
> >> >> <    (DSCP).  General background on the use of IP headers, and 5-
> tuples,
> >> >> <    to identify flows and support Quality of Service (QoS) can be found
> >> >> <    in [RFC3670].  [RFC7657] also provides useful background on the
> >> >> <    delivery of DiffServ and "tuple" based flow identification.
> >> >> <    Referring to a 6-tuple allows DetNet nodes to forward packets with
> >> >> <    the 6-tuple as is or remap the DSCP where required by the DetNet
> >> >> <    service.
> >> >> NEW
> >> >>  >    The DetNet IP data plane primarily uses "6-tuple" based flow
> >> >>  >    identification, where 6-tuple refers to information carried in IP and
> >> >>  >    higher layer protocol headers.  The 6-tuple referred to in this
> >> >>  >    document is the same as that defined in [RFC3290]. Specifically
> >> >>  >    6-tuple is (destination address, source address, IP protocol, source
> >> >>  >    port, destination port, and differentiated services (DiffServ) code
> >> >>  >    point (DSCP).  General background on the use of IP headers, and
> >> >>  >    5-tuples, to identify flows and support Quality of Service (QoS) can
> >> >>  >    be found in [RFC3670].  [RFC7657] also provides useful background
> on
> >> >>  >    the delivery of DiffServ and "tuple" based flow identification.
> >> >>  >
> >> >>  >    The DetNet IP data plane also allows for optional matching on two
> >> >>  >    additional data fields.  The optional fields are the ECN Field, as in
> >> >>  >    [RFC3168], and the IPv6 flow label field, as defined in [RFC8200].
> >> >>  >
> >> >>  >    Generally the fields used in flow identification are forward
> >> >>  >    unmodified but modification is allowed, for example to a DSCP
> value,
> >> >>  >    when required by the DetNet service.
> >> >>
> >> >> OLD
> >> >> <    masks, prefixes and ranges.  IP tunnels may also be used to support
> >> >> NEW
> >> >>  >    masks, lists, prefixes and ranges.  IP tunnels may also be used to
> >> >>
> >> >> OLD
> >> >> 437,438c444,446
> >> >> <    of the 6-tuple i.e., the typical 5-tuple enhanced with the DSCP
> code,
> >> >> <    uniquely identifies a DetNet service flow.
> >> >> NEW
> >> >>  >    of the 6-tuple i.e., the typical 5-tuple enhanced with the DSCP and
> >> >>  >    previously mentioned two optional fields, uniquely identifies a
> >> >>  >    DetNet service flow.
> >> >>
> >> >> OLD
> >> >> 590,593c598,608
> >> >> <    MUST support bitmask based matching, where bits set to one (1) in
> >> the
> >> >> <    bitmask indicate which subset of the bits in the field are to be used
> >> >> <    in determining a match.  Note that all bits set to zero (0) value as
> >> >> <    a bitmask effectively means that these fields are ignored.
> >> >> NEW
> >> >>  >    MUST support list based matching of DSCP values, where the list is
> >> >>  >    composed of possible field values that are to be considered when
> >> >>  >    identifying a specific DetNet flow.  Implementations SHOULD allow
> for
> >> >>  >    this field to be ignored for a specific DetNet flow.
> >> >>  >
> >> >>  >    Implementations of this document MUST allow the ECN field to be
> >> >>  >    ignored as part of DetNet flow identification. Additionally,
> >> >>  >    implementations SHOULD support identification of DetNet flows
> >> based
> >> >>  >    on the value carried in the ECN field.  When this field is used to
> >> >>  >    identify a specific DetNet flow, implementations MUST support a
> list
> >> >>  >    of ECN values that match a specific slow.
> >> >>
> >> >> OLD
> >> >> 705c720,729
> >> >> <    o  IPv4 Type of Service and IPv6 Traffic Class Fields.
> >> >> <
> >> >> <    o  IPv4 Type of Service and IPv6 Traffic Class Field Bitmask, where a
> >> >> <       zero (0) effectively means that theses fields are ignored.
> >> >> NEW
> >> >>  >    o  For the IPv4 Type of Service and IPv6 Traffic Class Fields:
> >> >>  >
> >> >>  >       *  If the DSCP field is to be used in flow identification.
> >> >>  >          Ignoring the DSCP filed is optional.
> >> >>  >
> >> >>  >       *  When the DSCP field is used in flow identification, a list of
> >> >>  >          field values that may be used by a specific flow.
> >> >>  >
> >> >>  >       *  If the ECN field is to be used in flow identification.
> >> >>  >          Matching based on ECN filed values is optional.
> >> >>  >
> >> >>  >       *  When ECN field is used in flow identification, a list of field
> >> >>  >          values that may be used by a specific flow.
> >> >>
> >> >> Please let me know if this addresses all your comments in an
> acceptable
> >> >> fashion or if you'd like to discuss further.
> >> >>
> >> >> (BTW I think this is the sole open set of issues on the document.)
> >> >>
> >> >> Thank you!
> >> >>
> >> >> Lou
> >> >>
> >> >
> >> > _______________________________________________
> >> > detnet mailing list
> >> > [e-mail]
> >> > https://www.ietf.org/mailman/listinfo/detnet
> >>
> >> _______________________________________________
> >> detnet mailing list
> >> [e-mail]
> >> https://www.ietf.org/mailman/listinfo/detnet
>