Re: [sfc] Status of draft-ietf-sfc-nsh-ecn-support

[Donald Eastlake <[e-mail]>]

Hi Med,

Thanks for your detailed review.

Many of your suggested wording changes seem fine to me and I plan to
adopt them. Below are my responses to the thread with Bob Briscoe and
here are responses on a couple of other points:

Abstract: What's wrong with the occurrence of "RFC 8300" and "RFC
7011" in the Abstract? Square bracketed references are prohibited in
Abstracts but such non-square-backeted names of RFCs are routinely
included in Abstracts. Such a text pointer to an RFC MUST be included
in the Abstract when a document Updates or Obsoletes that RFC. Here
are three example RFCs with text RFC pointers tat are not the Updates
and Obsoletes cases: 9018, 8822, and 8397. I believe the pointers to
RFC 8300 and RFC 7011 would be helpful to some readers.

Is it really necessary to always say "SFC-enabled domain" rather than
"SFC domain"? I'd be happy to say "enabled" in the abstract and on the
first use in the Introduction and to add an entry in the Terminology
(Conventions used in this Document) section saying that "SFC domain"
means "SFC-enabled domain". But always including the "enabled" in
"SFC-enabled" just seems like excess verbiage. What would an "SFC
domain" be other than an "SFC-enabled domain"?

On the thread with Bob Briscoe, see more detailed response below at
<de> but, as a general point, while it is possible to handle complex
cases it requires a more complex feature to do so. For example, an SF
doing so could feed-forward to the egress information about how it had
changed the flow. It seems better to propose a simpler feature, even
if it does not handle all cases. But it's limitations (scope) should
be made clear.

On Fri, Oct 29, 2021 at 8:38 AM Bob Briscoe <[e-mail]> wrote:
> Med,
>
> On 29/10/2021 07:45, [e-mail] wrote:
> Hi Bob,
>
> Thank you much for the follow up.
>
> Please see inline.
>
> Cheers,
> Med
>
> De : sfc <[e-mail]> De la part de Bob Briscoe
> Envoyé : jeudi 28 octobre 2021 19:11
> À : BOUCADAIR Mohamed INNOV/NET <[e-mail]>
> Cc : Donald Eastlake <[e-mail]>; [e-mail]; [e-mail]
> Objet : Re: [sfc] Status of draft-ietf-sfc-nsh-ecn-support
>
> [BB] Mohamed
>
> Thank you for your extensive review.
> Glossing over all the places where you would like the wording to be different, I detect the following main points of technical significance:
>
> The scheme is not how you imagine it - deliberately
>
> Draft (§1.1): "The NSH is a natural place, in a domain where traffic is NSH encapsulated, to note congestion"
> MB: "I would remove this mention as the outer-transport encapsulation would be “more” natural, IMO. If maintained, you may consider s/natural/candidate "
>
> [BB] I believe you have 'imagined' a different scheme (that wouldn't work). Then, from this point on, you edit the document to describe the scheme that you imagine.
> But I think you've misunderstood. The outer transports /are/ usually ECN-capable.
> [Med] They should if they follow the various guidelines there (6020, 6020update)
>
> [BB] However, unless the congestion information propagates onward to something that can feed it back, it is just black-holed, and useless.
> [Med] Sure, but this is not specific where the notification is carried.
>
> [BB] This is why the NSH header is arranged to accumulate any ECN marking from each outer transport along an SFP. I'm not sure which of the following two options you are imagining, but neither work:
> 1) EITHER: the header encapsulated within the NSH could be used to accumulate congestion information across the SFC-enabled domain. Then NSH would have to be defined to require that, as SFs and SFFs strip each outer transport header, they propagate any ECN markings into the header encapsulated within the NSH. But this would only be legal for those packets where that inner header indicates that the e2e transport supports ECN. That makes this #2 option depend on e2e ECN deployment, which has now reached about 20%, but not close enough to 100% to be useful for controlling congestion routing within an SFC-enabled domain.
> 2) OR: the scheme you imagine consists of a sequence of outer transport headers, that are disjoint from the ECN point of view. Then whenever there was congestion marking of one of these outer transport headers, it would get dumped at the terminating SFF, which would then have to find somewhere to feed it back to. If each SFF fed it back to the ingress classifier, there would be a huge amount of feedback streams all converging on each ingress classifier, compared to the scheme as described, which accumulates the information forwards, then feeds it back in one message per SFP, from the egress.
>
> [Med] What I had in mind as an alternative is:
>
> * uniform SFF capabilities with regards to ECN in an SFC-enabled domain
> * a consistent setting of ECN at both outer and inner headers (this allows in particular a terminating SF when present)
> * SFFs to glue the ECN during encap/decap along an SFP.
> * A control plane that monitors the SFC-enabled domain.
>
> As indicated in my comment, I suggest to use a more factual wording instead of indicating this is “natural”.
>
> OLD:
> The NSH is a
>    natural place, in a domain where traffic is NSH encapsulated, to note
>    congestion, avoiding possible confusion due, for example, to changes
>    in the outer transport header in different parts of the domain.
>
> NEW:
> This document discusses how the NSH can be used to note congestion.

<de> Carrying the ECN just through the outer transport header is
brittle. (Actually, if SFC is transport agnostic, you could have a
region where the transport protocol was one not providing for ECN
marking.)  Any transport link or SFF or SF (or SF proxy) that cleared
ECN would remove all record of any earlier part of the SFP. On the
other hand, using the NSH as the continuing record or glue is more
robust. Clearing ECN or failure to update it or failure to recombine
the outer trasport ECN with the continuing NSH ECN generally only
loses congestion information for a small segment of the SFP. Of
course, a sufficiently badly behaved node can still mess things up but
generally it seems to me that ECN indication in the NSH will be better
protected.

> [BB] I think we've confused you by describing the partially deployed ECN case first. Instead, perhaps we should describe the case with ECN deployed everywhere within the SFC-enabled domain first, then step back and describe partial deployment.
>
> In case it helps, I'll describe the scheme with full ECN deployment:
> The scheme uses the NSH (the one that remains associated with the packets/frames travelling along each SFP) to collect up any ECN marking added by each outer transport encapsulation. These all accumulate{Note 1} in the NSH headers, for delivery to the egress of the SFC-enabled domain, which then regularly feeds back summaries of each path's congestion to the ingress. The ingress is then continually collecting a view of the congestion of all the SFPs it is using through the SFC-enabled domain, which it can use to inform routing decisions (or pass this info on to where these decisions are made).
>
> [Med] This text is better, indeed. The main difference with what I described above is that NSH is the glue rather than relying on a local treatment at SFFs.
>
>
> [BB] I find it rather unusual that you talk relative to the scheme you describe. Surely you should be reviewing the scheme described in the draft, which doesn't need these magic "local treatments"?
>
> Still, I don’t see how an egress can know the ingress of an SFP. There is no such thing in the NSH and in the lookup tables.
>
> [BB] I know my stuff on ECN, and on NFV. But I haven't been involved in SFC specifically, so I rely on my co-authors for a response on this question.

<de> There should be more text in the document on how the egress, when
processing a packet, will know the ingress. In more complex cases, it
is probably reasonable to be able to include a way to get to the
ingress in a TLV in the HSH metadata.

> [Med] One side question: when NSH-in-NSH (RFC 8459) is used, do you propagate the notification in all NSH levels?
>
> [BB] Always only propagate into and out of whatever header is the outer.
>
> In fact, the draft is primarily about partial ECN deployment - in two senses:
> 1) In theory, for packets to have ECN enabled, the two end-points (e2e) have to check that they both support ECN first. {Note 2} Nonetheless, if an SFC-enabled domain wants to use ECN to manage itself, it doesn't have to wait for all Internet endpoints to support ECN. It can fake ECN support edge-to-edge: by the ingress of the SFC-enabled domain turning on ECN support in the proposed NSH header it adds, then the egress forwards on any congestion markings in the header it decapsulates, unless ot sees that the decapsulated inner indicates no ECN support. In which case the egress converts an incoming congestion mark into a drop, which is the only congestion signal that this particular e2e transport understands.
>
> [Med] I liked the faked ECN part even if it was “lost” in the text. I hope you will rework the draft to present this better. The “faked ECN” part is actually what motivates my comment about raising the comment about the intended status to consider Experimental track.
>
> [BB] That's not new. It's how MPLS checks for the ECN capability [RFC5129]. There it's called "per-domain ECT checking", so we should probably use that term here:
> https://datatracker.ietf.org/doc/html/rfc5129#section-3
>
>
> 2) Also, some of the SFFs or SFs might not support ECN. So the draft has to require all SFFs and SFs, as endpoints of an outer transport encapsulation, to check they both support ECN, and otherwise disable it in the outer between them. One assumes such cases would be rare in an SFC-enabled domain set up to use ECN. But, one has to cater for this possibility.
>
> [Med] I would assume that uniform capabilities at least for SFFs.
>
> [BB] Just SFFs is not enough. I was imagining that many SFs would already be written, and not necessarily already support ECN.
>
> {Note 1}: When I say "accumulate", I mean in the same way ECN accumulates congestion marking along any path of network elements. For instance, if there are two congestion points on a path, the first one might mark 0.1% of packets, and the second might mark 1.2% of packets. Then roughly 1.3% of packets will be marked. Actually, the idea is that it's combinatorial probability, not additive, which happens naturally, because probabilistic marking at each congested point gives you
>     [1 - (1-0.001)(1-0.012)] = 1.2988% marking.
> For instance, if there was much higher congestion, e.g. 60% marking at three points, you would get 1-(1-0.6)(1-0.6)(1-0.6) = 93.6% marking, not 180% which would be impossible.
>
> {Note 2}: There's about 85% server support of ECN now, but only perhaps 25% client support (don't quote me on that - it's a guess). So 25%*85% = 21% of connections are ECN-capable.
>
> Connection end-points within an SFC-enabled domain
>
> MB (re. §3): "Some of the legacy SFs may terminate TCP connections. These SFs will still follow the behavior in Section 6.1 of 3168. No? "
> MB: (re. §3.2.2): "I’m afraid there is a deviation when the SF terminates the connection. The inner packet will thus be processed as per RFC5681 "
> MB: (re. Figure 2 in §1.3) An SF may terminate the connection. It can be considered as a “final rcvr”.
>
> Correct, but these TCP connections are control or management connectionsfor network operations, not forwarded connections.{Note 3} In these cases, all the layers are decapsulated on the SF all at once, and the path does not traverse an egress of the SFC-enabled domain
>
> [Med] Not sure to get your point as the egress of an SFP is the last SFF of that chain, which is the SFF that services the terminating SF in this case.
>
>[BB] , so the only appropriate congestion feedback loop is the outer one (e2e). This document is about creating and using an inner (edge-to-edge) loop, to gather congestion information to inform the /SFC/ routing decisions. Yes, there will be some control and management connections into that domain that are not under that control, but they are still congestion controlled end-to-end, they just can't contribute to the edge-to-edge congestion feedback.
>
> [Med] The edge-to-edge is somehow “virtual” as any SFF in a domain be the terminating SFF of a chain.
>
>
> [BB] OK. We would need to use the correct terminology - I'll leave that to Donald as editor. But I'm saying that we don't need to include 'degenerate' points that act as the egress of the domain at the /management/ interfaces of the functions within the domain. But we do want {Note 4} to include points where traffic exits the SFC-enabled domain into an end-point (see discussion about context head-ends later).
>
> It makes effectively no difference to the overall traffic control picture whether management interfaces are included or not, except it unnecessarily adds loads of extra feedback report loops about tiny amounts of traffic.
>
> {Note 4}: 'We want' is not always 'we need'. In theory a congestion routing scheme (aka traffic engineering) can work and remain stable even if only a fraction of the traffic is under its control (there's a paper by Peter Key on that in the context of multipath routing - can't find it right now though).
>
> [Med] At least, the text should call the case of terminating SFs, what happens when ECN conflicts are observed at various levels (outer, inner, nsh), etc.
>
> [BB] I don't know what you mean here by 'ECN conflicts'.
> Guessing what you mean, I can say that ECN has to propagate up and down as layers are added and stripped, and the rules in the draft ensure that is so. Then everything should work. And if anything doesn't follow the rules (bugs etc), drop will still act as the well-understood congestion signal of last resort, because a drop at any layer is a drop at all layers.
>
> {Note 3}: An SF by definition forwards packets.
>
> [Med] An SF is not a forwarder from the SFC architecture.
>
> [BB] Both 'service' and 'function' are ambiguous words, that would describe a web server, or a sandwich dispensing machine for that matter. But, put the words together, and you get an SF, which provides a /network/ function. It doesn't terminate connections, other than for management and control. Yes, it might terminate some packets, as in the DDoS scrubbing machine you mention. And it might also drop packets due to congestion. But none of this acts as a connection termination at the user plane.
>
> [Med] Hmm, we don’t make any assumption about the nature of SFs deployed in an SFC-enabled domain. But there are plenty SFs that terminates a connection: this can be content head-end, for example.
>
> [BB] OK, I was assuming SFs were forwarding functions not endpoints. I actually checked the SFC architecture when you said this in your review, but I couldn't find anything about it, so I (apparently wrongly) assumed that the definition must be the same as for NFV, where forwarding is a defining feature of a 'network function'. Whatever, this means we will need to distinguish forwarding SFs from endpoint SFs. And we will want{Note 4} to define a domain egress at the interface to each endpoint function.
>
> [Med] Responses to some specific comments
>
> [BB] §2 "the IPv4 or IPv6 header as defined in Section 23.1 {?} of [RFC3168]"
> Section 23.1, is the IANA registration of the codepoints, not thier definition. Section 5 is the definition.
>
> [Med] OK. I hesitated when indicated that section; hence the “?” in the comment. I had a preference for 23.1 because it is really direct to point about the position of ECN bits.
>
> [BB] Fig 1 & Fig 2 in Section 5 of RFC3168 define the codepoints and the position of the field, resp.
>
>
> [BB] * §3. Reason for citing RFC6040, not draft-ietf-tsvwg-rfc6040update-shim: the latter updates one specific part of RFC6040, so the convention is to cite the base RFC, unless only the update is relevant.
>
> [Med] The reason for suggesting to cite the update I-D is that it covers many transports that are considered in SFC, especially the following ones:
>
>    o  GRE (Generic Routing Encapsulation) [RFC2784] and NVGRE (Network
>       Virtualization using GRE) [RFC7637];
>    o  CAPWAP (Control And Provisioning of Wireless Access Points)
>       [RFC5415];
>    o  LISP (Locator/Identifier Separation Protocol) [RFC6830];
>    o  VXLAN (Virtual eXtensible Local Area Network) [RFC7348] and VXLAN-
>       GPE [I-D.ietf-nvo3-vxlan-gpe];
>    o  Geneve [RFC8926];
>
> MB: "Why not simply say that these nodes must adhere to the reco in RFC6040? "
> [BB] Because one cannot mandate that the past should not have happened (let alone MUST NOT happen!). This draft is all about incremental deployment
>
> [Med] see next point
>
> MB: "Why not just saying “all SFFs” as the egress node is the last SFF of an SFP?"
> [BB] When ECN for the SFC-enabled domain is in the NSH (as we have deliberately defined it, rather than as you have 'imagined' it) decap /is/ only needed and supported at the egress of the domain. That's the whole point.
>
> [Med] I was actually echoing your text when saying “Section 3.3 requires that all the egress nodes …”. Knowing that every SFF can in theory terminate a chain, and thus behave as an, egress for that chain, one can conclude that all SFFs must support it!

<de> As we have said, the document is intended to support incremental
deployment. There is significnat configuration involved as to what
IPFIX templates to use, etc., so the Network Manager just needs to not
try to use this ECN support where it will not work. And, the current
document says, somewhat redundantly, in various places:

- "It requires that all ingress and egress nodes of the SFC domain
  implement ECN."

- "Section 3.3 requires that all the egress nodes of the SFC domain
  support Compliant ECN Decapsulation in conjunction with tunnel
  congestion feedback, otherwise the scheme in this document will not
  work."

- "All the egress nodes of the SFC domain MUST support Compliant ECN
  Decapsulation as specified in this section."

Probably there should be some wording adjustment to make it clear that
multiple SFFs can be the egress for a flow or set of flows being
congestion controlled.

> [BB] See earlier about forwarding vs endpoint SFs.
>
>
> Draft: "This approach will inherently support all known variants of ECN, including the experimental L4S capability [RFC8311] [ecnL4S]."
> MB: "I’m afraid some more elaboration is needed to back this statement "
> [BB] Agreed. We need to describe this (probably in an appendix, rather like the similar draft about ECN support in TRILL).
>
> [Med] Thanks.
>
>
> MB: "If we want a pluggable detection logic, I would not require AQM as required, but just as an example. That approach is what is adopted by DC-TCP for example (RFC 8257)."
> [BB] DCTCP uses an AQM that fits the recommendations of RFC7567, which are deliberately already generalized. You only have to read the list of recommendation headings from the Contents of RFC7567 to see that it is very general. The functions within an SFC-enabled domain still have to work with e2e congestion control. So I would strongly advise against going off into abstract hyperspace by saying some pluggable function is possible that does some other magic sort of active queue management.
>
> [Med] OK.
>
> Draft: "Congestion encountered in the SFF to SF and SF to SFF paths will be unmanaged. "
> MB: "Why not handling this at the outer transport? "
> [BB] This is defined as a case where the SF is ECN-unware. So it can't be handled in the outer transport either, because it terminates at a node that is ECN-unaware.
>
> [Med] You are right, even if I see it unlikely to have an SF that is NSH-aware but ECN-unaware.
>
>
> §3.2.3
> Draft: "Other forwarding nodes, that are non-NSH forwarding nodes between NSH forwarding nodes, such as IP or label switched routers, might also contain potential bottlenecks. If so, they SHOULD implement an AQM"
>
> MB: "This is not specific to SFC. Do we have a pointer where such reco is provided for routers? "
> [BB] Answer: RFC7567
>
> [Med] I’m aware of BCP197, but I was looking for a profile document for routers such as draft-ietf-v6ops-ipv6rtr-reqs. But that ones expired and does not include AQM reco.

<de> I see no reason why we need to find a recommendation elsewhere
although I would be happy to point to it if there is one.

The goal is to monitor congestion in the entire ingress to egress path
which can only be complete if packets/frames are marked as to
experienced congestion at every bottleneck, which is about the same as
every place there is a queue, including queues inside SFFs,
SF(SF-Proxy)s, routers, bridges, ...

> §4
> "How the egress knows the ingress for a given chain, a given packet, flow, etc?"
> Using a lookup against the Service Path Identifier?
>
> [Med] We don’t have such feature in NSH. As currently defined, the feedback mechanism is broken. It is for the same reasons that, for example, the OAM work defines a TLV to indicate where to send a reply back. I’m afraid that you need such a TLV (draft-ietf-sfc-multi-layer-oam-16#section-6.3.1). Please note that there might be several classifiers, reclassification happening on path, etc.
>
> [BB] OK.

<de> Agreed, as per my response above.

> Draft: "a flow that would preserve the number of packets in the absence of congestion"
> MB: "How owns this information?"
> [BB] This is an important open question that the draft raises, but does not answer, I believe. But when I was first involved with this draft, I imagined that SFs would have to be registered with a requirement to provide this information (boolean).
>
> [Med] This is an important point to leave without elaboration. Although expired, you may refer to https://datatracker.ietf.org/doc/html/draft-ietf-sfc-control-plane#section-3.3.3 as a candidate to feed that information.
>
>
> MB: "When a congestion is observed, to what extend these messages will further exacerbate the congestion conditions? "
>[BB] The use IPFIX over partially reliable SCTP is designed to address this (the draft doesn't say the following but it should):
> * This designs leads to compact congestion summaries sent regularly (so congestion control of user traffic will have already yielded the little amount of space this control traffic needs.
> * IPFIX uses cumulative counters, so any losses can be retransmitted, but no need to repeatedly retransmit because the next one will catch up the counter.
> * SCTP's partial reliability mode allows retransmission to be abandoned after a deadline passes.
>
> [Med] These are good additions. Looking forward seeing them integrated.

<de> OK.

> §5
> Draft: "IPFIX template records are exchanged between ingress and egress "
> MB: "Which may be between every SFF and a classifier. "
> [BB] No. This is the scheme you've imagined, not the one described.

<de> A particular IPFIX temple for the flow or set of flows being
monitored from a particular ingress need only get to the egresses used
by those flows, not all nodes that might be an egress for some flow.

> [Med] Hmm. I guess this is because you assume there are few nodes that behave as egress nodes, but in the SFC architecture the SFF that terminates a chain is the egress!

<de> I agree that the document is worded to somewhat imply that there
are relatively few egress nodes but that can be fixed.

> [BB] Need to distinguish forwarding and endpoint SFs, as already discussed.
>
> Thank you for all this. V helpful.

I hope to post a revised version at some point during the IETF meeting
next week.

Thanks,
Donald
===============================
 Donald E. Eastlake 3rd   [phone] (cell)
 2386 Panoramic Circle, Apopka, FL 32703 USA
 [e-mail]

> Bob
>
> Thank you very much for the time you've taken to review this.
>
> Cheers
> Bob
>
>
>
> On 28/10/2021 12:52, [e-mail] wrote:
> Hi Donald,
>
>
> Thank you for your effort to mature the document.
>
>
> FWIW, please find my review of this version:
>
>
> * pdf: https://raw.githubusercontent.com/boucadair/IETF-Drafts-Reviews/master/draft-ietf-sfc-nsh-ecn-support-08-rev%20Med.pdf
> * doc: https://github.com/boucadair/IETF-Drafts-Reviews/raw/master/draft-ietf-sfc-nsh-ecn-support-08-rev%20Med.docx
>
>
> Cheers,
> Med
>
>
> -----Message d'origine-----
>
> De : sfc <[e-mail]> De la part de Donald Eastlake
> Envoyé : mercredi 27 octobre 2021 20:31
> À : [e-mail]
> Cc : [e-mail]
> Objet : [sfc] Status of draft-ietf-sfc-nsh-ecn-support
>
>
> Hi,
>
>
> I believe this draft is ready for WG Last Call. The most recent revisions
> have been minor editorial improvements and clarifications.
>
>
> However, the IETF meeting is coming up and I understand there may be a
> Transport Area review. So I suggest there should be a WG Last Call
> starting in two or three weeks.
>
>
> Thanks,
> Donald
> ===============================
>  Donald E. Eastlake 3rd   [phone] (cell)
>  2386 Panoramic Circle, Apopka, FL 32703 USA  [e-mail]
>
>
> ---------- Forwarded message ---------
> From: <[e-mail]>
> Date: Thu, Oct 21, 2021 at 9:36 PM
> Subject: New Version Notification for draft-ietf-sfc-nsh-ecn-support-
> 08.txt
>
>
> A new version of I-D, draft-ietf-sfc-nsh-ecn-support-08.txt
> has been successfully submitted by Donald E. Eastlake and posted to the
> IETF repository.
>
>
> Name:           draft-ietf-sfc-nsh-ecn-support
> Revision:       08
> Title:          Explicit Congestion Notification (ECN) and Congestion
> Feedback Using the Network Service Header (NSH) and IPFIX Document date:
> 2021-10-21
> Group:          sfc
> Pages:          33
> URL:
> https://www.ietf.org/archive/id/draft-ietf-sfc-nsh-ecn-support-08.txt
> Status:         https://datatracker.ietf.org/doc/draft-ietf-sfc-nsh-ecn-
> support/
> Htmlized:
> https://datatracker.ietf.org/doc/html/draft-ietf-sfc-nsh-ecn-support
> Diff:
> https://www.ietf.org/rfcdiff?url2=draft-ietf-sfc-nsh-ecn-support-08
>
>
> Abstract:
>    Explicit congestion notification (ECN) allows a forwarding element to
>    notify downstream devices of the onset of congestion without having
>    to drop packets. Coupled with a means to feed information about
>    congestion back to upstream nodes, this can improve network
>    efficiency through better congestion control, frequently without
>    packet drops. This document specifies ECN and congestion feedback
>    support within a Service Function Chaining (SFC) domain through use
>    of the Network Service Header (NSH, RFC 8300) and IP Flow Information
>    Export (IPFIX, RFC 7011).
>
>
>
> The IETF Secretariat
> _______________________________________________
> sfc mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/sfc