Re: [Detnet] TSN scheduling/queuing algorithms and DetNet

[Toerless Eckert <[e-mail]>]

Thank, Florian, inline

On Thu, Oct 05, 2023 at 09:43:47PM +0200, Florian Kauer wrote:
> Thank you a lot David and Toerless for your explanations!
> 
> On 05.10.23 18:39, Toerless Eckert wrote:
> > As i tried to explain in the last email reply to Florian:
> > 
> > I think for large-scale/high-speed detnets, that the TSN algorithms will not work well,
> > but with proposal like we have on the tavble, i think we know how to adopt what we learned from TSN.
> > (aka: TCQF/CSQF vs. CQF/ECQF).
> 
> Yes, as you have explained detailed in your first mail to me, just TSN is certainly not sufficient for
> the large-scale/high-speed cases and I think we all agree on that and it is the primary motivation
> for the "open working meetings".
> 
> > Even if something like timed gates on ingres is something we could reuse directly from TSN, i still
> > prefer we have IETF docs explaining how to adopt to IETF networks.> For low-speed/limited-bandwdith DetNet, such as in manufacturing floors, i wonder if we are going to see
> > any insance where we need to adopt TSN mechanisms directly to L3:
> 
> I think the discussion is "just" about terminology. So what it really means in the end to
> "adopt TSN mechanisms directly to L3".

I think it's more about specifications. But i first need to read up more on the stuff
to have more of an opinion. Ultimately i would just like to see that DetNet makes DetNet-only
implementations of whatever method we gree on as easy as possible, without implementers
having to delve into TSN spec stuff they don't need to know. But maybe that's not the
only useful goal...

> > I do not think we are going to see any such limited-scale networks without L2 switches involved. I for
> > once am not aware of any campus/industrial environments where sender/receivers directly attach to
> > a router. IETF has just done a (sh#$y - censored) job to make deployment of L3 easy. You immediately
> > need an IP/IPv6 addrssing expert, manage addressing space, 
> 
> Following your argument, you need that addressing expert anyway either for configuring the senders/receivers
> or for configuring the bridges, so what is the difference?
>
> Why does that imply that the senders/receiver will not directly attach to a router?
> Do you refer to the complexity to implement such functionality on the end stations?

So you have a couple of hosts sender/receivers, you want to use ip address space
192.168.100.x/24 for their addresses. How do you do that without an ieee ethernet
switch between these hosts and a router ?

a) WiFi. Thats a whole other poblem we can think about with RAW.

b) Every host sits on a differen router port and you configure every router
   port with a different 192.168.100.x/31, more likely 192.168.100.x/30, wasting
   50% address space. You will even hae a hard time finding routers supporting this
   (in SMB/home/industrial routers...).

c) You go to an antiquities store and find some 10 or 100Mbps ethernet hub or yellow cable.

Aka: every realistic wired solution today has an 802.1 ethernet switch between hosts and
routers, and if that switch does not do TSN, then we would even need to do workarounds
for that to support DetNet between host and router.
   
> > and if you want to have device mobility you
> > need complex solutions that most likely will break any services guarantees (like TSN or DetNet).
> 
> I think device mobility for time-sensitive/deterministic networks is a completely different topic
> where IPv6 addressing might be even the smallest problem you need to solve...

Just mentioning that there are today more widely used L2 mobility solutions than L3
mobility solutions (even though IETF did define L3 mobility).

> > Aka: between any sender/receiver and any router will be L2 switches whcih would then have to be TSN
> > anyhow. And we have the RFCs explaining how to run DetNet "over" such L2 TSN subnets.
> > 
> > How about a larger industrial network/campus: 
> > 
> >     Src --- L2sw --- Rtr1 ---- backbone --- Rtr2 --- L2sw -- Rcvr
> > 
> > Would we not we need "DetNet only" between Rtr1 and Rtr2  - no TSN involved ?
> > 
> > No. Because any Rtr equipment that vendors would sell into such environment will already have an
> > L2 switch integrated inyo any of these routers. So in reality the actual forwarding would be:
> > 
> >     Src --- L2sw --- [L2sw-Rtr1-L2sw] ---- backbone --- [L2sw-Rtr2-L2sw] --- L2sw -- Rcvr
> 
> What exactly do you mean by "DetNet only"? Even in the "large-scale/high-speed case"
> there will always be SOME L2 layer (and L1), otherwise you won't be able to transmit
> anything on the wire (or wireless).
>
> Of course, that does not necessarily need to be TSN according to IEEE 802.*, but at least
> SOME L2 layer that provides basic RT guarantees (such as not delaying or dropping packets
> arbitrarily).

Links betweeen routers typically do not use L2 switching between them. Aka: its always
about L2 802.1 switching, not just some point-to-point L2 layer. Yes, that is always there,
sory if that wasn't clear.

> > In effect we do have 3 TSN domains where Rtr1 / Rtr2 really just need to do non-queuing
> > forwarding, make sure that queuing happens on the outgoing TSN L2sw side on the same box,
> > and the only DetNet job on the sending side is to aply the right L2 tagging to ensure the
> > packet is put into th right TSN service instance.
> 
> I don't know if I understand you correctly, but JUST connecting two completely uncoordinated
> TSNs together with a L3 forwarding function, will IMHO not provide you deterministic latency.

That is an interesting question. Admittedly, i am not sure how TSN without DetNet
supposedly works on hosts. Aka:

    Sender       TSN       TSN          Receiver
    Host   ---->Switch1--->Switch2 ---> Host

Sender and Receiver have a UDP/IP stack, are on the same IP subnet, and the Sender
now has UDP/IP socket on which it sends packets to Receive Host. And TSN give that
traffic latency/throughput guarantees.

Now i postulate the following:

    Sender       TSN       TSN             TSN          TSN                               Receiver
    Host   ---->Switch1--->Switch2 ---> (Switch3->Rtr->Switch4) --->Switch5--->Switch6 -> Host2

                TSN Domain 1                                TSN Domain 2

Receiver host is replaced by a typical industrial or SMB router,which all have L2 switches
in front of their logical router ports, shown in picture as TSN Switch3 and TSN Switch4.

So, in this router, the TSN-Switch3 terminates TSN Domain 1, and the received packets
go up to the Rtr L3 level, and then down again into TSN Switch4 processing, which acts
as sender for TSN Domain 2.

So, the only question is whether there is a chance for the L3 Router portion to mess up
the bandwidth and latency guarantees. And theoretically this is of course possible,
but practically it very likly that the only reasons why the L3 router level could mess
up guarantes is for reasons that we would equally have problems fixing with any of the
direct DetNet mechanisms. Aka: If for example there is a performance limit to receive
and route/forward L3 packets, and there is no priorization at that level of DetNet or TSN
packets received.

> So if, just for the sake of the example, both networks use 1 second cycle time, but the
> gate for a certain traffic class only opens for 1 ms in every cycle, it depends on the relative
> time position of the gate opening if the latency over the hop is ~1 ms or ~1 s.

Right. But not sure i understand the point you're trying to make.

> So in this case you need both time synchronization between the networks as well as some
> overarching coordination that is aware of the fact that there are two networks that shall be
> connected.

Your example is using a relative posiutioning of gate times which is 100 times more granular
(1 msec) than the granularity of the forwarding cycles (1 cycle). This is not what i am
describing in draft-eckert-detnet-flow-interleaving. Hence my problem understanding the
point you're trying to make.

But if we turn it around: cycle time 10 usec, relative gate opening times within a window
10 msec for a period of 10 usec - then we're in the territory i am describing

And in this environment i could even connect two non-synchronized CQF or TCQF/CSQF domains
across a router: Assume the clock of domain 2 drifts continuously against the clock of
domain 1. Even the interleaving still works. I just loose the ability to accuratiely determine
at a 10 usec level when a packet will be at which router or switch. The granualiry now
is likely about 30 usec because of the asynchronous connection.

> Nevertheless, there is nothing fancy that this "coordination" needs to consider
> that is DetNet-specific or uses other algorithms that are different from those
> applied in a "pure" TSN.

For everything where we do ofer new "algorithms" i think this is not true. Whereever we
just want to adopt TSN algorihms, e.g.: for not-large-scale Qcr for example, it is mostly
an issue of removed the guesswork of implementers by giving them explicit spec.

> > Only when the "backbone" links become fast / lage enough that the L2sw approach on the box
> > won't work would we need "direct L3" interface queuing. 
> 
> Probably the same question as the "DetNet only" one above, but what exactly do you mean "direct L3" here?

I think that routers with interfaces of 100Gbps or faster will not have any TSN
capable hardwre for those interfaces (they may have on lower-speed interfaces linecards),
so they would be looking for DetNet for guidance what to do across those interfaces.
And that guidance should be as simple as possible.


And i would think it would be prudent too to think about how well those devices could
implement DetNet functionality without TSN on the low-speed access interfaces.

> In principle, I can just implement a single silicon and put it in a black box
> that looks from outside just the same as two separate L2 bridges and a separate "L3 forwarding device"
> in a box, but does that very efficiently and fast internally.

Yes, its' actually quite difficult/complex, but thats what devices often do.

> How would that be different than a "direct L3" in silicon given that also this one needs
> to implement some L2 + L1 functionality to output any meaningful electrical signals?

One problem is how to flatten and unify forwarding plane functionality across L2 and L3.
Aka: imagine you are forwarding from one host to another host in your router/switch, and
in one case both hosts are on the same subnet, so that forwarding is L2 and any
latency/throughput guarantees could therefore be TSN, or both hosts are in diffeent
subnets, so L3 is involved.

The main issue IMHO is that TSN does not have IP-flows, so there is this complex mapping
of IP DetNet flows to TSN, which uses other packet tags. But whenever you have one of those
complex ASICs that can do L2 and L3, you could likely equally do IP-flow based classification also
for the L2 forwarding. And then you can give up on all the TSN classification and logically
end up at a DetNet only solution, whether the device does L3 or L2 forwading. 

We've ran through another instance of this mixed mess with multicast and "snooping" on L2
switches, and because it was first implemented and then (badly) specified, its still a mess (IMHO).

Cheers
    Toerless

> Thanks,
> Florian
> 
> > 
> > Cheers
> >     toerless
> > 
> > 
> > On Thu, Oct 05, 2023 at 01:46:44PM +0000, Black, David wrote:
> >> Hi Florian,
> >>
> >> Still posting as an individual - the two of us are on the same page ...
> >>
> >>> Since I was not able to attend the meeting and I can not find any minutes or recording, I would love to get more background about this question, especially why you explicitly marked **algorithms** as bold.
> >>
> >> I was drawing a distinction between the entire scope of TSN technology, which is a layer-2 subnet when fully deployed, versus just the TSN scheduling/queuing algorithms (e.g., Qbv) which are readily deployable at layer 3.
> >>
> >>> And for those scenarios my current working assumption is actually that all DetNet nodes that route on L3 between L2 TSN
> >>> networks would be able to directly apply all TSN techniques (like Qbv and CQF) without any special consideration for them on L3.
> >>
> >> +1 on "directly apply all TSN techniques (like Qbv and CQF) without any special consideration for them on L3" - and in particular, I believe that this direct application is possible now (i.e., no need to wait for IETF to publish a detnet RFC that says it's ok to do this obvious thing ...).
> >>
> >>> The only exception is that the L3 processing MIGHT take more time than L2 bridging and this might have to be taken into account e.g. when calculating a Qbv schedule.
> >>> Do you agree or is there something I overlook?
> >>
> >> I don't think you've missed anything major ...
> >>
> >> Thanks, --David
> >>
> >> -----Original Message-----
> >> From: Florian Kauer <[e-mail]> 
> >> Sent: Thursday, October 5, 2023 3:22 AM
> >> To: Black, David; [e-mail]
> >> Cc: Black, David
> >> Subject: Re: [Detnet] TSN scheduling/queuing algorithms and DetNet
> >>
> >>
> >> [EXTERNAL EMAIL] 
> >>
> >> Hi David,
> >> thanks for bringing up this point!
> >> Since I was not able to attend the meeting and I can not find any minutes or recording, I would love to get more background about this question, especially why you explicitly marked **algorithms** as bold.
> >>
> >> For the use cases I am currently considering for DetNet (industrial control/vPLC and pro audio) the scaling requirements (time asynchrony, propagation latency...) are IMHO of lower importance at the moment. And for those scenarios my current working assumption is actually that all DetNet nodes that route on L3 between L2 TSN networks would be able to directly apply all TSN techniques (like Qbv and CQF) without any special consideration for them on L3. The only exception is that the L3 processing MIGHT take more time than L2 bridging and this might have to be taken into account e.g. when calculating a Qbv schedule. Do you agree or is there something I overlook?
> >>
> >> Thanks,
> >> Florian
> >>
> >> On 04.10.23 23:30, Black, David wrote:
> >>> For clarity, this is posted as an individual contributor …
> >>>
> >>>  
> >>>
> >>> In the most recent open working meeting, Toerless and I had a discussion about whether DetNet nodes could use the TSN scheduling/queuing algorithms.
> >>>
> >>>  
> >>>
> >>> Checking the DetNet RFCs, what I believe I see is:
> >>>
> >>>   * Toerless’s assertion that all DetNet RFC references to TSN refer to layer-2 TSN subnets appears to be correct.
> >>>   * OTOH, I did not see anything that prohibits implementation and use of the TSN scheduling/queuing **algorithms** at layer 3 in DetNet nodes.
> >>>
> >>>  
> >>>
> >>> I would hope that it’s not necessary to publish an RFC to bless the use of the TSN scheduling/queuing algorithms at layer 3 in DetNet Nodes …
> >>>
> >>>  
> >>>
> >>> Thanks, --David
> >>>
> >>> * *
> >>>
> >>> *David L. Black, *Sr. Distinguished Engineer, Technology & Standards
> >>>
> >>> Infrastructure Solutions Group,*Dell Technologies*
> >>>
> >>> mobile [phone] [e-mail] <mailto:[e-mail]>
> >>>
> >>>  
> >>>
> >>>
> >>> _______________________________________________
> >>> detnet mailing list
> >>> [e-mail]
> >>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/detnet__;!!LpKI!nabqKqkaJqASqxUmFFXsdWshrCNzaQHg7m-v-hGX_vjxDkaPwEjaN_7LLu4bVsAoreoYzg2j8oO_zNFgpKpK_IqND_rO3Q$ [ietf[.]org]
> > 

-- 
---
[e-mail]