Re: [mpls] [Detnet] DetNet: draft-eckert-detnet-mpls-tc-tcqf-03 request for feedback

["Black, David" <[e-mail]>]

Hi Toerless,

> > I am not saying we should never have ECN in DetNet. I am just saying that we don't have it now:

We’re on the same page - some earlier remarks seemed to imply that it's safe to assume that ECN will never be used with DetNet.  I have no problem with deferring work on how ECN could be used with DetNet - I just wanted to avoid an up-front prohibition.  Heck, Pascal even found a use case for ECN with DetNet ...

> DetNet inside does not eliminate congestion, though. It places it between the application and the DetNet network ingress,
> in the stack in the source end system and across the UNI, be it the bus to the NIC or a first Ethernet hop.

For which ECN and scalable congestion control are likely to be useful for congestion-responsive protocols such as TCP.  This is one way to avoid a DetNet network ingress version of buffer-bloat if a capacity-seeking protocol (e.g., TCP) is run over DetNet.  Beyond with, ECN may be useful inside DetNet, as you've describe:

> > I am happy to see us "abuse" ECN bits as a form of OAM: I calculate by DetNet traffic queue
> > size based on my calculus for my admitted traffic (whether its guaranteed bandwidth or
> > bounded latency), and then i make my queues somewhat longer and trigger ECN when the queue gets into that
> > range.
> > 
>  > Technically i call this an "abuse", because i do not expect the sender/receiver app to reduce their
> > bitrate, instead i am signaling that i (the network service) have screwed up. It's more like an
> > "Apology Bit" then. But i would very much like that.

That's close enough to what I'm suggesting, although it need not be "abuse" - if the apps involved are congestion-responsive (e.g., web stacks for device management), ECN congestion indications will cause a bit rate reduction. Firmware download comes to mind as an example where that might be useful.

Thanks, --David

-----Original Message-----
From: Pascal Thubert (pthubert) <[e-mail]> 
Sent: Friday, August 5, 2022 6:10 AM
To: Toerless Eckert
Cc: Black, David; [e-mail]; DetNet WG; [e-mail]; draft-eckert-detnet-mpls-tc-tcqf
Subject: Re: [Detnet] DetNet: draft-eckert-detnet-mpls-tc-tcqf-03 request for feedback


[EXTERNAL EMAIL] 

Hello Toerless 

I agree that the current definition of DetNet makes ECN useless inside the DetNet network which from the outside emulates an ingress to egress serial wireline. It’s not only that we do not need ECN, there’s no bit grim reaper either because loss, even organized loss, is not acceptable.

DetNet inside does not eliminate congestion, though. It places it between the application and the DetNet network ingress, in the stack in the source end system and across the UNI, be it the bus to the NIC or a first Ethernet hop.

We can either a) make all that deterministic but that will be hard for a common OS or 2) provide congestion control, like ECN or whatever else. Ideally in the form of traffic backwards from the ingress to the application. Think Qbb PFC relayed all the way up the stack in the source end system.

An alternative is to make the transport layer aware of DetNet and terminating the congestion signaling so the app would just see a more traditional flow control. The benefit of that approach is that the application is agnostic of whether we do a) or b) below. 

If we do nothing to pace the application then the excess data will be squeezed at the source. Note that if the pacing could be either in the from of a tick to the application or in the form of the pull of a burst by the transport. The same goes over the UNI.

 I see your TCQF as one cool way to pull the burst and absorb the jitter the jitter in the application, the stack, and the access network to the DetNet ingress. Using 3 (or even more) cycles like we do for CQF over long lines, we could absorb the UNI jitter. The transport doing CQF would just need a rough syntonization to avoid drifting.

I hope I make sense to you,

Pascal

> Le 5 août 2022 à 09:46, Toerless Eckert <[e-mail]> a écrit :
> 
> On Thu, Aug 04, 2022 at 08:21:59PM +0000, Black, David wrote:
>>> I think the initial customers for DetNet are those who have the no-loss, no-jitter, no-congestion
>>> requirements. Hence i want the first round that we can do without new packet headers
>> 
>> Sorry, that's not an answer to the question that was asked.
>> 
>> An architectural decision now that DetNet/MPLS will *never* support ECN could be rather short-sighted (e.g., if the management stack use case is reasonable).  OTOH, the approach of locating TCQF in the Service layer might be extensible to also locating ECN marking in that layer.
> 
> I am not saying we should never have ECN in DetNet. I am just saying that we don't have it now:
> For any traffic with the DetNet guaranteed bandwidth or bounded latency services,
> there is no DetNet definition what ECN would mean, because neither of these services is
> defined to have any congestion. Instead, both of them are defined to have no congestion loss
> or ECN.
> 
> If you have a good definition of a proposed DetNet guaranteed bandwidth or DetNet bounded latency
> service with ECN, please elaborate. I did have ideas of my own,  but given how those ideas even
> before DetNet where considered by IETF - and you - to be too complex on non-compliant with standard
> PHB, i'll let others step forward first now (if you remember, it was the AF41/AF42 discuss).
> 
> And just because DetNet IP has an ECN field in the IP header does not mean we have a DetNet definition
> as to what it should mean with DetNet guaranteed throughput / bounded latency traffic.
> 
> I am happy to see us "abuse" ECN bits as a form of OAM: I calculate by DetNet traffic queue
> size based on my calculus for my admitted traffic (whether its guaranteed bandwidth or
> bounded latency), and then i make my queues somewhat longer and trigger ECN when the queue gets into that
> range.
> 
> Technically i call this an "abuse", because i do not expect the sender/receiver app to reduce their
> bitrate, instead i am signalling that i (the network service) have screwed up. Its more like an
> "Apology Bit" then. But i would very much like that.
> 
> And finally, if i just use DetNet for no-bandwidth guarantee traffic, for example because
> i just want to do PREOF, then i definitely want to support ECN, but for that traffic i can not
> use TCQF or any other bounded latency option, because they all depend on reservation of
> path resources (bitrate, buffers).
> 
> Cheers
>    Toerless
> 
>> Thanks, --David
>> 
>> -----Original Message-----
>> From: Toerless Eckert <[e-mail]> 
>> Sent: Thursday, August 4, 2022 2:14 PM
>> To: Black, David
>> Cc: [e-mail]; DetNet WG; [e-mail]; draft-eckert-detnet-mpls-tc-tcqf
>> Subject: Re: [Detnet] DetNet: draft-eckert-detnet-mpls-tc-tcqf-03 request for feedback
>> 
>> 
>> [EXTERNAL EMAIL] 
>> 
>>> On Wed, Aug 03, 2022 at 01:27:33PM +0000, Black, David wrote:
>>> Hi Toerless,
>>> 
>>> I'll observe that the rather small MPLS TC field is being used to serve a lot of purposes.  A question to consider (and one that likely reflects my limited MPLS expertise) is whether cycle tag information is always needed in the forwarding layer F-label vs. an alternative where the cycle tag is in the service layer S-label, resulting in an approach where the service layer handles tagged cycles, relying on the unmodified forwarding layer of the MPLS data plane between service nodes.  That could result in a hierarchy where service nodes are deployed to bound the level of cycle skew exposed to the unmodified forwarding layer that does not see the cycle tags.  Does that make sense?
>> 
>> The goal definitely is to come up with the most easily first-round deployable
>> solution, so i am happy to put the bounded latency function into whatever layer
>> (Service or Forwarding) where it is most easily implemented. But: a) not sure
>> what the other detnet folks think about that, or b) we probably would need
>> to ask the MPLS WG what they think aout looking up QoS from the second label
>> in the stack - complexity wise.
>> 
>>> On ECN ...
>>> 
>>>> Wrt. ECN: For TCQF, we do not need ECN: There CANNOT be any TCQF/DetNet traffic
>>>> with bounded latency that would need ECN markings. That would simply be a failure
>>>> of the bounded latency service.
>>> 
>>> There may be a "never say never" reality check lurking in there ;-).
>>> 
>>> While use of ECN to signal actual congestion is almost certainly indicative of a DetNet failure, I'll repeat my comments to Pascal on this topic:
>>> 
>>>>> I think we have two related goals in this discussion that ought to be distinguished:
>>>>>    - What's the minimal/necessary/ideal L4 transport functionality for DetNet?
>>>>>    - What would it take to run an existing transport stack (L4 and above) over DetNet?
>>>>> This discussion started around the first goal, where I agree with you that the ECN-based dynamic windowing
>>>>> functionality is not needed.  That functionality may be useful towards the second goal (e.g., run HTTP/TLS/QUIC
>>>>> over a DetNet data plane for device management).
>>> 
>>> I suggest more consideration of that second goal.
>>> 
>>> Are you taking the position that it is *never* appropriate to run an existing web-based management stack (e.g., based on HTTP/TLS/TCP or HTTP/TLS/QUIC) over a DetNet MPLS data plane?
>> 
>> I think the initial customers for DetNet are those who have the no-loss, no-jitter, no-congestion
>> requirements. Hence i want the first round that we can do without new packet headers to be
>> focussing on that, so we (DetNet) gets a good foot in the market. And especially that we
>> can establish simpler DetNet-Only solutions, instead of creating an unfortunately even
>> more complex solution we have now: layering DetNet on top of TSN to get bounded latency.
>> 
>> So, i primarily would not want to spare bit for more loftiger goals in this first-round
>> encoding options (reuse TC, DSCP), unless we know we have them available without  cutting
>> into that first goal.
>> 
>> Once we know we have the packet header bits to do more:
>> 
>> First of all, i would like us to think about OAM bits to recognize when there are errors,
>> e.g.: when there is packet drop even though there should be non, because something
>> malfunctioned. How would we do this ? Aka: Whats the best OAM for failure of
>> bounded latency that we can give those first-round applications.
>> 
>> If that OAM turns out to also be usable for classical congestion controlled applications
>> (which i don't think), then we're done. Otherwise we'd have to look for encoding
>> of bits for both those OAM goals and the ECN goals.
>> 
>> If we go with SR-MPLS, our LSE would turn into SIDs, so we could have SIDs for just
>> DetNet, not shared by other traffic. Meaning wew have the full 3 bit free. And we
>> could then also easier (IMHO) hack around even more, redesignating e.g.: some more
>> bits from TTL (if Cisco gives license to IETF on its patent for that...).
>> 
>> Aka: Lots of options, which is why i was trying to figure out whats hopefully most important
>> first so we can get more deployment experience from a first round RFC.
>> 
>> Cheers
>>    Toerless
>> 
>>> Thanks, --David
>>> 
>>> -----Original Message-----
>>> From: detnet <[e-mail]> On Behalf Of Toerless Eckert
>>> Sent: Tuesday, August 2, 2022 10:41 PM
>>> To: [e-mail]
>>> Cc: DetNet WG; [e-mail]; draft-eckert-detnet-mpls-tc-tcqf
>>> Subject: Re: [Detnet] DetNet: draft-eckert-detnet-mpls-tc-tcqf-03 request for feedback
>>> 
>>> 
>>> [EXTERNAL EMAIL] 
>>> 
>>> Thanks for the interest Zongpeng,
>>> 
>>> Given how the TC-TCQF solution is now within scope of DetNet charter, i think this
>>> is also a good time to also get more input/review from the MPLS WG, hence Cc'ing it
>>> (the chair where already suggesting to engage with th MPLS WG on this in the past).
>>> 
>>> For background: The draft currently covers what i think is the most easy deployable option to
>>> enable tagged-cyclic-queuing-and-forwarding (TCQF) with the MPLS forwarding plane -
>>> without changes on the wire, by reusing the existing MPLS header field TC.
>>> 
>>> If we choose to adopt this work in DetNet it would be good work to document not
>>> only this most simple TC encoding, but also any other encoding that do not require
>>> on-the-wire-changes, but that may be necessary when this most simple method is
>>> considered to be not sufficient for some MPLS networks.
>>> 
>>> TCQF encoding that would encode the cycle in a new heder field (change on the wire)
>>> should IMHO be work reserved for another draft. Primarily because i think we can
>>> much faster finish specifying and deploying the mechanism with the options that do
>>> not require changes on the wire - and collect experience with it (this is true for
>>> MPLS as well as IP/IPv6, where we can use DSCP).
>>> 
>>> The draft therefore proposed allocation of MPLS TC values for use with TC-TCQF is
>>> described in section 3.3 of the draft TC-TCQF draft. We would need 3 or 4 TC values.
>>> 
>>> Here is a bit more detail:
>>> 
>>> RFC3270 defines how to use the TC field
>>> (actually, it calls it EXP, the field was later renamed to TC in RFC5462)
>>> 
>>> What our TCQF draft proposes is to propose to use the so-called 
>>> "TC- Inferred-PSC LSP (E-LSP) behavior". This is described in section 1.2 of
>>> RFC3270.
>>> 
>>> This is IMHO the most widely deployed method for using DiffServ with MPLS
>>> and also the most simple to deploy, because it can get away without additional
>>> control plane / labels. What this effectively means is that the network operator
>>> configures in a DiffServ style 
>>> 
>>> When a deployment has already used more than 4/5 TC and therefore
>>> does not have 3 or 4 TC free, then (i think!) the operator could allocate additional
>>> labels space where the TC values are free. Depending on how MPLS experts read
>>> RFC3270, this might require resorting to calling those LSPs SIDs and call that
>>> encoding then an SR-MPLS mechanism.
>>> 
>>> We could also look into allocating 3 or 4 additional L-LSP, but that looks like
>>> the most complicated option to me.
>>> 
>>> IMHO, a first RFC does not have to be complete with all possible encoding
>>> options. I'd rather see an RFC finished that can be deployed, and when that
>>> raises interest in the industry, but the vendors recognize some customers who
>>> already used up all TC, then this would be good justification to do a -bis of
>>> the RFC with the more complicated encodings.
>>> 
>>> Wrt. ECN: For TCQF, we do not need ECN: There CANNOT be any TCQF/DetNet traffic
>>> with bounded latency that would need ECN markings. That would simply be a failure
>>> of the bounded latency service. The non-TCQF TC values may indicate ECN
>>> for non-TCQF traffic, but to the best of my knowledge ECN with TC is practically
>>> not deployed.
>>> 
>>> Hope this help, please comment!
>>> 
>>> Cheers
>>>    Toerless
>>> 
>>> On Tue, Aug 02, 2022 at 09:14:24AM +0800, [e-mail] wrote:
>>>> Hi, Toerless   
>>>> 
>>>>    I am Zongpeng Du from China Mobile.
>>>> 
>>>>    We are interested in the draft. However, we have a question about the method to convey the tag in the TOS of an MPLS label.
>>>> 
>>>>    MPLS label: 32bits
>>>> 
>>>>    00-19     20-21                                                 23                                    24-31
>>>>    label       TC: Traffic Class(QoS and ECN)        S: Bottom-of-Stack          TTL: Time-to-live
>>>> 
>>>> As shown in the Figure, the three bits for TC have been defined as the format of "QoS and ECN". 
>>>>    IMO, unless the IETF agrees to disuse it, we can not use the bits that have been used.
>>>>    Do I have any misunderstanding here?
>>>> 
>>>>   Or, the label is special defined, such as an S-Label, so that we can define new use of the TC field ? 
>>>> 
>>>> Best Regards
>>>> Zongpeng Du
>>>> 
>>>> 
>>>> 
>>>> [e-mail] & [e-mail]
>>>> 
>>>> From: Toerless Eckert
>>>> Date: 2022-07-12 03:20
>>>> To: detnet
>>>> CC: draft-eckert-detnet-mpls-tc-tcqf
>>>> Subject: [Detnet] DetNet: draft-eckert-detnet-mpls-tc-tcqf-03 request for feedback
>>>> Dear DetNet WG
>>>> 
>>>> I just posted rev 3 of https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-eckert-detnet-mpls-tc-tcqf/__;!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiORb915kQ$ [datatracker[.]ietf[.]org]
>>>> 
>>>> Diff over previous version 2:
>>>>  https://urldefense.com/v3/__http://www.ietf.org/*rfcdiff?url1=https:**Awww.ietf.org*archive*id*draft-eckert-detnet-mpls-tc-tcqf-02.txt&url2=https:**Awww.ietf.org*archive*id*draft-eckert-detnet-mpls-tc-tcqf-03.txt__;Ly8vLy8vLy8vLy8!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiNYE2hEcw$ [ietf[.]org]
>>>> 
>>>> This version adds new text about the per-flow ingres enqueueing into the
>>>> cycles, informational text about high-speed HW implementation considerations
>>>> and a pointer to a research paper also describing such HW validation.
>>>> 
>>>> I think this makes the proposed work complete, and as authors, we would
>>>> like to ask the WG for adoption - and the participants for feedback.
>>>> 
>>>>  Btw: The authors are also happy to discuss any restructuring, for example
>>>>  Eshan told me it might be better to separaate the genreic mechanism from
>>>>  the MPLS encoding (separate drafts). 
>>>> 
>>>> Quick summary, if you have not kept track:
>>>> 
>>>>  TCQF stands for Tagged Cyclic Queuing and Forwarding, and is a proposal
>>>>  how to adopt the IEEE TSN "Cyclic Queuing and Forwarding" Queuing for
>>>>  'native' use in DetNet, specifically with MPLS forwarding plane.
>>>> 
>>>>  CQF is a great queuing mechanism for DetNet because it does not only offer
>>>>  bounded latency, but also tight-bounded jitter, important for all
>>>>  industrial "synchronous" applications. It also is most easily scalable
>>>>  in HW and operations for metropolitan or larger size WAN DetNet deployments
>>>>  because it does not have per-hop state on every router.
>>>> 
>>>>  By tagging packets with the cycle number, TCQF overcomes also the issues
>>>>  that keeps CQF limited to small-scale (few Km) Ethernet: it can support
>>>>  large link delays and link-delay variation, and it reduces relatively
>>>>  the required clock accuracy, therefore making it feasible to use this
>>>>  on 100, 400 Gbs or faster networks.
>>>> 
>>>>  There is an expired problem description draft i wrote some time ago,
>>>>  if you need more background: https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-eckert-detnet-bounded-latency-problems/__;!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiOUluv-oQ$ [datatracker[.]ietf[.]org]
>>>> 
>>>> Cheers
>>>>    Toerless
>>>> 
>>>> _______________________________________________
>>>> detnet mailing list
>>>> [e-mail]
>>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/detnet__;!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiMZSWyvmw$ [ietf[.]org]
>>> 
>>> -- 
>>> ---
>>> [e-mail]
>>> 
>>> _______________________________________________
>>> detnet mailing list
>>> [e-mail]
>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/detnet__;!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiMZSWyvmw$ [ietf[.]org]
>>> 
>> 
>> -- 
>> ---
>> [e-mail]
>> 
> 
> -- 
> ---
> [e-mail]
> 
> _______________________________________________
> detnet mailing list
> [e-mail]
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/detnet__;!!LpKI!hQadGsWXaITIMhg4SO1TEGGzt-SPRSKfjpCt37cObco49GhhBlacEqxPAHGJfg1ZAH4NJ1O_rrqXSxv1$ [ietf[.]org]