Re: [Detnet] TSN scheduling/queuing algorithms and DetNet

[[e-mail]]

Hi Florian, Toerless,

Thanks for your good discussions about this topic.
I have some understandings, please see #PSF inline.

Regards,
PSF



Original


From: FlorianKauer <[e-mail]>
To: Toerless Eckert <[e-mail]>;
Cc: Black, David <[e-mail]>;[e-mail] <[e-mail]>;
Date: 2023年10月16日 04:40
Subject: Re: [Detnet] TSN scheduling/queuing algorithms and DetNet

Hi Toerless,
thanks! That already clears up some things for me. Please see my comments inline.
 
On 09.10.23 18:39, Toerless Eckert wrote:
> Thank, Florian, inline
>  
> On Thu, Oct 05, 2023 at 09:43:47PM +0200, Florian Kauer wrote:
>> Thank you a lot David and Toerless for your explanations!
>> 
>> On 05.10.23 18:39, Toerless Eckert wrote:
>>> As i tried to explain in the last email reply to Florian:
>>> 
>>> I think for large-scale/high-speed detnets, that the TSN algorithms will not work well,
>>> but with proposal like we have on the tavble, i think we know how to adopt what we learned from TSN.
>>> (aka: TCQF/CSQF vs. CQF/ECQF).
>> 
>> Yes, as you have explained detailed in your first mail to me, just TSN is certainly not sufficient for
>> the large-scale/high-speed cases and I think we all agree on that and it is the primary motivation
>> for the "open working meetings".
>> 
>>> Even if something like timed gates on ingres is something we could reuse directly from TSN, i still
>>> prefer we have IETF docs explaining how to adopt to IETF networks.> For low-speed/limited-bandwdith DetNet, such as in manufacturing floors, i wonder if we are going to see
>>> any insance where we need to adopt TSN mechanisms directly to L3:
>> 
>> I think the discussion is "just" about terminology. So what it really means in the end to
>> "adopt TSN mechanisms directly to L3".
>  
> I think it's more about specifications. But i first need to read up more on the stuff
> to have more of an opinion. Ultimately i would just like to see that DetNet makes DetNet-only
> implementations of whatever method we gree on as easy as possible, without implementers
> having to delve into TSN spec stuff they don't need to know. But maybe that's not the
> only useful goal...
 
I think that is already the central point here.
My assumption for the origin of this discussion was a deployment where we already HAVE underlying
L2 TSN and the question is just if we can use the techniques that are provided already on L2
to fulfill the L3 requirements. And in that situation I would agree with David that this is already
possible without a dedicated DetNet standard.
 
Now you seem to come from a different perspective where your L2 does not provide any
services equivalent to TSN, but you still might want to use the TSN algorithms on L3 (like ECQF).
In that scenario I see your point that it does not really make sense to say
"You are not actually using anything from 802.* in your system, but still for implementing DetNet
you have to read and understand 802.1Q..." 
 
My assumption (as I never worked with anything faster than 2.5G NICs ;-) ) was that it is actually
more convenient if we reuse as much on DetNet level from TSN because the implementers are anyway
already familiar with TSN and now just want to bring it one layer higher.
 
But you are coming from the other direction of very high speed L3 routers and it would be more effort to
also consider L2 techniques that are, if I understand you correctly, not really relevant in the first
place in your area. It would be very interesting to me if you would explain a little bit how your
L2 looks like. Does it follow any existing standard or is it proprietary?
 
#PSF: 
1) For queue mechanisms that do not require traffic to coordinate between nodes, such as 
ATS/CBS/ECQF, it is indeed only necessary to introduce a simple mapping between L3 
encapsulated scheduling parameters and L2 encapsulated scheduling parameters (covered 
by Detnet over TSN), and there is indeed no need to redefine the queue scheduling algorithm. 
However, I also believe that other reasonable opinions are that we cannot look at queueing 
mechanisms in isolation. They actually involve necessary components required for successful 
work, including scheduling parameters encapsulation, resource reservation, etc., which obviously 
cannot be directly used in L3 by reusing L2's standard definition. That is, it is necessary to refer 
to existing L2 scheduling algorithms to redefine L3's scheduling mechanism, that is my 
understanding of the algorithm mentioned by David.
2) For queue mechanisms that require traffic to coordinate between nodes, such as TAS, the 
more important task in this case is how the entire network's traffic interleaves and orchestrates, 
while queueing scheduling on the data side is extremely simple (only a single buffer is needed 
for deterministic Scheduled Traffic). IMO This is clearly not something that Detnet over TSN can cover.


  
>>> I do not think we are going to see any such limited-scale networks without L2 switches involved. I for
>>> once am not aware of any campus/industrial environments where sender/receivers directly attach to
>>> a router. IETF has just done a (sh#$y - censored) job to make deployment of L3 easy. You immediately
>>> need an IP/IPv6 addrssing expert, manage addressing space,  
>> 
>> Following your argument, you need that addressing expert anyway either for configuring the senders/receivers
>> or for configuring the bridges, so what is the difference?
>> 
>> Why does that imply that the senders/receiver will not directly attach to a router?
>> Do you refer to the complexity to implement such functionality on the end stations?
>  
> So you have a couple of hosts sender/receivers, you want to use ip address space
> 192.168.100.x/24 for their addresses. How do you do that without an ieee ethernet
> switch between these hosts and a router ?
>  
> a) WiFi. Thats a whole other poblem we can think about with RAW.
 
Agreed, out of scope for this discussion.
 
> b) Every host sits on a differen router port and you configure every router
>    port with a different 192.168.100.x/31, more likely 192.168.100.x/30, wasting
>    50% address space. You will even hae a hard time finding routers supporting this
>    (in SMB/home/industrial routers...).
 
IPv6 should at least mitigate that, but we should not dive into the IPv4 vs. 6 discussion here ;-)
But I totally see what you mean by that! We were just using a different universal quantification ;-)
Of course, most pairs of nodes go though L2 switches. I just would assume that there are still individual
cases where this is not the case (e.g. in the backbone of a campus network), to connect a particular
end station...
 
But in the end that is actually not very relevant. Even in such mixed networks, where most
nodes are connected via (TSN) L2 switches, point-to-point L3 connections will very likely also
use the same (TSN) L2, even if there is only two nodes in the network and no actual switch inbetween.
 
And that was actually my point: We most likely don't need any "DetNet-not-over-TSN" in these cases
and so we can rely on TSN. Still, as we are discussing below, stitching multiple TSN networks
together without any consideration on L3 will likely not work properly IMHO.
 
> c) You go to an antiquities store and find some 10 or 100Mbps ethernet hub or yellow cable.
 
Agreed, we don't want to go back to that.
 
> Aka: every realistic wired solution today has an 802.1 ethernet switch between hosts and
> routers, and if that switch does not do TSN, then we would even need to do workarounds
> for that to support DetNet between host and router.
 
What kind of workarounds do you think about?
I would assume if we have a non-TSN switch on the path, it will throw our RT guarantees out of the window
regardless of what we do on L3. Without L2 RT guarantees, no L3 RT guarantees.
 
 #PSF: Can traffic collisions between host and router be basically ignored? For example, 
in the data center, there is no focus on latency, but more on the disorder caused by ECMP.


>>> and if you want to have device mobility you
>>> need complex solutions that most likely will break any services guarantees (like TSN or DetNet).
>> 
>> I think device mobility for time-sensitive/deterministic networks is a completely different topic
>> where IPv6 addressing might be even the smallest problem you need to solve...
>  
> Just mentioning that there are today more widely used L2 mobility solutions than L3
> mobility solutions (even though IETF did define L3 mobility).
>  
>>> Aka: between any sender/receiver and any router will be L2 switches whcih would then have to be TSN
>>> anyhow. And we have the RFCs explaining how to run DetNet "over" such L2 TSN subnets.
>>> 
>>> How about a larger industrial network/campus:  
>>> 
>>>     Src --- L2sw --- Rtr1 ---- backbone --- Rtr2 --- L2sw -- Rcvr
>>> 
>>> Would we not we need "DetNet only" between Rtr1 and Rtr2  - no TSN involved ?
>>> 
>>> No. Because any Rtr equipment that vendors would sell into such environment will already have an
>>> L2 switch integrated inyo any of these routers. So in reality the actual forwarding would be:
>>> 
>>>     Src --- L2sw --- [L2sw-Rtr1-L2sw] ---- backbone --- [L2sw-Rtr2-L2sw] --- L2sw -- Rcvr
>> 
>> What exactly do you mean by "DetNet only"? Even in the "large-scale/high-speed case" 
>> there will always be SOME L2 layer (and L1), otherwise you won't be able to transmit
>> anything on the wire (or wireless).
>> 
>> Of course, that does not necessarily need to be TSN according to IEEE 802.*, but at least
>> SOME L2 layer that provides basic RT guarantees (such as not delaying or dropping packets
>> arbitrarily).
>  
> Links betweeen routers typically do not use L2 switching between them. Aka: its always
> about L2 802.1 switching, not just some point-to-point L2 layer. Yes, that is always there,
> sory if that wasn't clear.
 
Yes, so apparently the difference between my use case and your use case is that I also
have TSN even between two routers (even if there is no switch inbetween) and in your
case there is no TSN L2 below that you can rely on for certain functionality, right?
 
 
>>> In effect we do have 3 TSN domains where Rtr1 / Rtr2 really just need to do non-queuing
>>> forwarding, make sure that queuing happens on the outgoing TSN L2sw side on the same box,
>>> and the only DetNet job on the sending side is to aply the right L2 tagging to ensure the
>>> packet is put into th right TSN service instance.
>> 
>> I don't know if I understand you correctly, but JUST connecting two completely uncoordinated
>> TSNs together with a L3 forwarding function, will IMHO not provide you deterministic latency.
>  
> That is an interesting question. Admittedly, i am not sure how TSN without DetNet
> supposedly works on hosts. Aka:
>  
>     Sender       TSN       TSN          Receiver
>     Host   ---->Switch1--->Switch2 ---> Host
>  
> Sender and Receiver have a UDP/IP stack, are on the same IP subnet, and the Sender
> now has UDP/IP socket on which it sends packets to Receive Host. And TSN give that
> traffic latency/throughput guarantees.
>  
> Now i postulate the following:
>  
>     Sender       TSN       TSN             TSN          TSN                               Receiver
>     Host   ---->Switch1--->Switch2 ---> (Switch3->Rtr->Switch4) --->Switch5--->Switch6 -> Host2
>  
>                 TSN Domain 1                                TSN Domain 2
>  
> Receiver host is replaced by a typical industrial or SMB router,which all have L2 switches
> in front of their logical router ports, shown in picture as TSN Switch3 and TSN Switch4.
>  
> So, in this router, the TSN-Switch3 terminates TSN Domain 1, and the received packets
> go up to the Rtr L3 level, and then down again into TSN Switch4 processing, which acts
> as sender for TSN Domain 2.
>  
> So, the only question is whether there is a chance for the L3 Router portion to mess up
> the bandwidth and latency guarantees. And theoretically this is of course possible,
> but practically it very likly that the only reasons why the L3 router level could mess
> up guarantes is for reasons that we would equally have problems fixing with any of the
> direct DetNet mechanisms. Aka: If for example there is a performance limit to receive
> and route/forward L3 packets, and there is no priorization at that level of DetNet or TSN
> packets received.
 
[see below]
 
>> So if, just for the sake of the example, both networks use 1 second cycle time, but the
>> gate for a certain traffic class only opens for 1 ms in every cycle, it depends on the relative
>> time position of the gate opening if the latency over the hop is ~1 ms or ~1 s.
>  
> Right. But not sure i understand the point you're trying to make.
 
[see below]
 
>> So in this case you need both time synchronization between the networks as well as some
>> overarching coordination that is aware of the fact that there are two networks that shall be
>> connected.
>  
> Your example is using a relative posiutioning of gate times which is 100 times more granular
> (1 msec) than the granularity of the forwarding cycles (1 cycle). This is not what i am
> describing in draft-eckert-detnet-flow-interleaving. Hence my problem understanding the
> point you're trying to make.
>  
> But if we turn it around: cycle time 10 usec, relative gate opening times within a window
> 10 msec for a period of 10 usec - then we're in the territory i am describing
>  
> And in this environment i could even connect two non-synchronized CQF or TCQF/CSQF domains
> across a router: Assume the clock of domain 2 drifts continuously against the clock of
> domain 1. Even the interleaving still works. I just loose the ability to accuratiely determine
> at a 10 usec level when a packet will be at which router or switch. The granualiry now
> is likely about 30 usec because of the asynchronous connection.
 
Maybe there is just a confusion with the terminology? E.g. do we use "cycle time" differently?
I try to illustrate my example in the appended picture and please let me know if the terminology
matches (sorry, I am not fluent enough in ASCII art for that picture...)
 
So the point that I am trying to make with that example is just that it is not sufficient
to only connect two TSN domains by a "L3 forwarding function", but (if you use gates)
you also need to synchronize the time between the domains and align the gate schedule.
 
Of course, if you use techniques that does not require time synchronization in the first place,
you obviously do not need to apply one...
 
 
>> Nevertheless, there is nothing fancy that this "coordination" needs to consider
>> that is DetNet-specific or uses other algorithms that are different from those
>> applied in a "pure" TSN.
>  
> For everything where we do ofer new "algorithms" i think this is not true. Whereever we
> just want to adopt TSN algorihms, e.g.: for not-large-scale Qcr for example, it is mostly
> an issue of removed the guesswork of implementers by giving them explicit spec.
 
Yes, agreed! I am only talking about a scenario where the data plane techniques from TSN
are sufficient (like the use of gates) and there is actually an underlying TSN available.
In that case, no data plane specifications are IMHO required for DetNet, but still control
plane specifications (like the need for time synchronization and alignment of gate schedules).
 
#PSF: 
1) The rate based scheduling defined by TSN provides less friendly latency per hop, 
which may be acceptable in cases of few L2 hops, but needs to be pay in large scaling 
L3 networks. In addition, the cost of Interleaved Regulators used to avoid burstiness 
cascading also needs to be taken seriously.
2) If we want to totally reuse an existing L2 TSN mechanism, but it requires time 
synchronization, which results in all L3 nodes also requiring time synchronization, then 
IMO it only means that we cannot reuse that TSN mechanism and should modify it or find 
other mechanisms.


>>> Only when the "backbone" links become fast / lage enough that the L2sw approach on the box
>>> won't work would we need "direct L3" interface queuing.  
>> 
>> Probably the same question as the "DetNet only" one above, but what exactly do you mean "direct L3" here?
>  
> I think that routers with interfaces of 100Gbps or faster will not have any TSN
> capable hardwre for those interfaces (they may have on lower-speed interfaces linecards),
> so they would be looking for DetNet for guidance what to do across those interfaces.
> And that guidance should be as simple as possible.
>  
>  
> And i would think it would be prudent too to think about how well those devices could
> implement DetNet functionality without TSN on the low-speed access interfaces.
 
As above, it would be good to know if in this high-speed scenario there is actually a
L2 available that could provide something similar to TSN or if this is completely
left to the implementation.
 
 
>> In principle, I can just implement a single silicon and put it in a black box
>> that looks from outside just the same as two separate L2 bridges and a separate "L3 forwarding device" 
>> in a box, but does that very efficiently and fast internally.
>  
> Yes, its' actually quite difficult/complex, but thats what devices often do.
 
Agreed :-)
 
 
>> How would that be different than a "direct L3" in silicon given that also this one needs
>> to implement some L2 + L1 functionality to output any meaningful electrical signals?
>  
> One problem is how to flatten and unify forwarding plane functionality across L2 and L3.
> Aka: imagine you are forwarding from one host to another host in your router/switch, and
> in one case both hosts are on the same subnet, so that forwarding is L2 and any
> latency/throughput guarantees could therefore be TSN, or both hosts are in diffeent
> subnets, so L3 is involved.
>  
> The main issue IMHO is that TSN does not have IP-flows, so there is this complex mapping
> of IP DetNet flows to TSN, which uses other packet tags. But whenever you have one of those
> complex ASICs that can do L2 and L3, you could likely equally do IP-flow based classification also
> for the L2 forwarding. And then you can give up on all the TSN classification and logically
> end up at a DetNet only solution, whether the device does L3 or L2 forwading.  
 
AFAIK already in TSN we have the possibility for IP stream identification (802.1CB)
and even more we have the "Extended Stream identification function" with mask & match (802.1CBdb).
You could argue that it is not a clear layer separation, but it is what we have today and with
that you can quite seamlessly handle a L3 flow as TSN stream without a complex mapping.
 
 
> We've ran through another instance of this mixed mess with multicast and "snooping" on L2
> switches, and because it was first implemented and then (badly) specified, its still a mess (IMHO).
 
So to avoid the mess for TSN and DetNet, we should IMHO consider three cases separately:
 
DetNet over TSN:
We already have RFC 9023 that basically already allows to use the TSN techniques to implement a DetNet
(without respecifying each of them separately as RFC - and that was how I understood Davids inital question).
However, tere are most likely gaps that are underspecified when knitting together several TSN domains and
they should be addressed on DetNet level.
 
DetNet over "Some other standardized real-time capable L2":
If there is an L2 where implementing similar techniques to TSN are feasible, I would argue
that this should be done (instead of just respecifying TSN techniques on L3).
 
DetNet over "proprietary L2":
If it is not feasible to standardize a real-time capable L2 (how I understand your scenario),
I would agree with you that there is the danger of underspecification. But I first need to
understand the background of that scenario...
 
Thanks,
Florian
 
 
> Cheers
>     Toerless
>  
>> Thanks,
>> Florian
>> 
>>> 
>>> Cheers
>>>     toerless
>>> 
>>> 
>>> On Thu, Oct 05, 2023 at 01:46:44PM +0000, Black, David wrote:
>>>> Hi Florian,
>>>> 
>>>> Still posting as an individual - the two of us are on the same page ....
>>>> 
>>>>> Since I was not able to attend the meeting and I can not find any minutes or recording, I would love to get more background about this question, especially why you explicitly marked **algorithms** as bold.
>>>> 
>>>> I was drawing a distinction between the entire scope of TSN technology, which is a layer-2 subnet when fully deployed, versus just the TSN scheduling/queuing algorithms (e.g., Qbv) which are readily deployable at layer 3.
>>>> 
>>>>> And for those scenarios my current working assumption is actually that all DetNet nodes that route on L3 between L2 TSN
>>>>> networks would be able to directly apply all TSN techniques (like Qbv and CQF) without any special consideration for them on L3.
>>>> 
>>>> +1 on "directly apply all TSN techniques (like Qbv and CQF) without any special consideration for them on L3" - and in particular, I believe that this direct application is possible now (i.e., no need to wait for IETF to publish a detnet RFC that says it's ok to do this obvious thing ....).
>>>> 
>>>>> The only exception is that the L3 processing MIGHT take more time than L2 bridging and this might have to be taken into account e.g. when calculating a Qbv schedule.
>>>>> Do you agree or is there something I overlook?
>>>> 
>>>> I don't think you've missed anything major ...
>>>> 
>>>> Thanks, --David
>>>> 
>>>> -----Original Message-----
>>>> From: Florian Kauer <[e-mail]>  
>>>> Sent: Thursday, October 5, 2023 3:22 AM
>>>> To: Black, David; [e-mail]
>>>> Cc: Black, David
>>>> Subject: Re: [Detnet] TSN scheduling/queuing algorithms and DetNet
>>>> 
>>>> 
>>>> [EXTERNAL EMAIL]  
>>>> 
>>>> Hi David,
>>>> thanks for bringing up this point!
>>>> Since I was not able to attend the meeting and I can not find any minutes or recording, I would love to get more background about this question, especially why you explicitly marked **algorithms** as bold.
>>>> 
>>>> For the use cases I am currently considering for DetNet (industrial control/vPLC and pro audio) the scaling requirements (time asynchrony, propagation latency...) are IMHO of lower importance at the moment. And for those scenarios my current working assumption is actually that all DetNet nodes that route on L3 between L2 TSN networks would be able to directly apply all TSN techniques (like Qbv and CQF) without any special consideration for them on L3. The only exception is that the L3 processing MIGHT take more time than L2 bridging and this might have to be taken into account e.g. when calculating a Qbv schedule. Do you agree or is there something I overlook?
>>>> 
>>>> Thanks,
>>>> Florian
>>>> 
>>>> On 04.10.23 23:30, Black, David wrote:
>>>>> For clarity, this is posted as an individual contributor …
>>>>> 
>>>>>  
>>>>> 
>>>>> In the most recent open working meeting, Toerless and I had a discussion about whether DetNet nodes could use the TSN scheduling/queuing algorithms.
>>>>> 
>>>>>  
>>>>> 
>>>>> Checking the DetNet RFCs, what I believe I see is:
>>>>> 
>>>>>   * Toerless’s assertion that all DetNet RFC references to TSN refer to layer-2 TSN subnets appears to be correct.
>>>>>   * OTOH, I did not see anything that prohibits implementation and use of the TSN scheduling/queuing **algorithms** at layer 3 in DetNet nodes.
>>>>> 
>>>>>  
>>>>> 
>>>>> I would hope that it’s not necessary to publish an RFC to bless the use of the TSN scheduling/queuing algorithms at layer 3 in DetNet Nodes …
>>>>> 
>>>>>  
>>>>> 
>>>>> Thanks, --David
>>>>> 
>>>>> * *
>>>>> 
>>>>> *David L. Black, *Sr. Distinguished Engineer, Technology & Standards
>>>>> 
>>>>> Infrastructure Solutions Group,*Dell Technologies*
>>>>> 
>>>>> mobile [phone] [e-mail] <mailto:[e-mail]> 
>>>>> 
>>>>>  
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> detnet mailing list
>>>>> [e-mail]
>>>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/detnet__;!!LpKI!nabqKqkaJqASqxUmFFXsdWshrCNzaQHg7m-v-hGX_vjxDkaPwEjaN_7LLu4bVsAoreoYzg2j8oO_zNFgpKpK_IqND_rO3Q$ [ietf[.]org]
>>> 
>  
_______________________________________________
detnet mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/detnet