Re: [Detnet] FYI: draft-eckert-detnet-glbf-01.txt

["Andrea Francini (Nokia)" <[e-mail]>]

Hi Jinoo,

As I understand it, C-SCORE can be seen as a network-wide instantiation of the Virtual Clock scheduler by Lixia Zhang.

When realized In a single scheduling node, Virtual Clock is a Latency-Rate server and also a Rate-Proportional Server. Is it possible to provide formal demonstration that C-SCORE has the same end-to-end latency properties as a network of Virtual Clock servers (or, more generally, of Rate-Proportional Servers), possibly within a finite margin of error?

One basic element of concern from the draft that describes the scheme is that the real-time clocks of the ingress nodes are not required to be synchronized. Single-node virtual clock works (for latency enforcement only, the service fairness index was instead unbounded, with the negative effect on implementation that I bring up below) because all packets are timestamped based on a common time reference. This can generally be assumed to be not the case for C-SCORE, where the local clocks of the ingress nodes can drift apart from each other. Why not require at least a periodic re-sync of the ingress-node clocks?

Also, have you looked into a detailed specification for the range of the time representation in the timestamp (finishing-potential) field carried by the packets? A major issue with the implementation of Virtual Clock in a single node was that the timestamp of a single flow could grow much larger than the system potential (the real time) if it was the only flow supplying back-to-back packets for an extended time. The time representation allowed by a finite number of bits could then wrap around, causing the timestamps of packets of different flows to be no longer comparable in a meaningful way. It seems that the same could happen with C-SCORE.

Best regards,

Andrea

From: detnet <[e-mail]> On Behalf Of Jinoo Joung
Sent: Friday, September 22, 2023 7:19 PM
To: Toerless Eckert <[e-mail]>
Cc: [e-mail]; [e-mail]; [e-mail]
Subject: Re: [Detnet] FYI: draft-eckert-detnet-glbf-01.txt


CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See the URL nok.it/ext for additional information.


Toerless,

Your argument to me and Shaofu is summarized in the following statement.

"The latency calculusfor C-SCORE is based on the assumption of a TSpec limit
for each flow. To me it seems clear that this calcullus does not hold
true when the TSpec increases across hops."

Another way to put it would be:
The increased burst size of a flow will harm the latency bound calculated based on the initial burst size of the flow.

My answer is:
The increased burst size of a flow does NOT harm the latency bound calculated based on the initial burst size of the flow.

Let's say a flow has TSpec rt+b.
The worst case happens is that the actual arrival follows rt+b;
that is at t=0 b arrives, and then packets arrive one after another, with rate r.

CLAIM: The largest E2E latency is that of the LAST packet of the initial burst b. Let's call this packet p(b).
Proof:
1) For any packet that arrives before p(b), it departs the network before p(b), so CLAIM holds.
2) Consider a packet that arrives later than p(b). Let's call it p(b)+n.
For simplicity of discussion, let's assume that every packet's length is 1.
Assume that p(b)+n joins the burst b at a point of the network.
Then by definition of the burst, at that point the arrival time difference between p(b) and p(b)+n is larger than or equal to n/r.
Assume that the burst to which these two packets belong is resolved at another point of the network.
Between the join point and the resolve point, they go together.
Then the departure time difference is less than or equal to n/r, Since the flow is served at a rate larger than or equal to r.
Therefore the latency difference is the departure time difference minus the arrival time difference,
which is less than or equal to zero.
CLAIM holds

So, for both cases CLAIM holds.

I hope this helps.
The burst accumulation effect is well known from decades ago.
Do you really believe that the network calculus theory has not been aware of that?

Best,
Jinoo


On Sat, Sep 23, 2023 at 12:39 AM Toerless Eckert <[e-mail]<mailto:[e-mail]>> wrote:
(eliminating many lines of empty space - your mailer is curious that way...)

On Fri, Sep 22, 2023 at 04:36:16PM +0800, [e-mail]<mailto:[e-mail]> wrote:
> Hi Toerless,
>
>
> You provide a good example in previous mail to show how does bursts accumulation
> of a single flow generate due to bursts aggregation from other flows. Here, I apologize
> to invent a new word "bursts aggregation", to distinguish it with "bursts accumulation".
> The former is something about multiple flows, while the later is something about multiple
> service-burst-intervals of a single flow.
>
> Yes, bursts accumulation may cause TSpec distortion of the observed flow. I believe we
> have agreement on this.
>
> However, the difference in our views is, the solution effect of different schedulings on
> bursts accumulation.
>
> If I get your point, you think that re-shaping (including IR), or Damper, must be introduced
> to eliminate bursts accumulation.
>
>
> IMO, IR/Damper is just one option (note that I take Damper as similar design of IR, e.g, per
> "traffic class + incoming port", except with different calculation method of eligibility time for
> packets), other options may be time-rank, such as latency compensation in Deadline,
> visual finish time in C-SCORE.

Let me repeat what i just replied to Jinou:

None of the scheduling mechanisms considered for rfc2212 treats a flow's adjacent packets
differently just because they are clustered closer togeter than they should (because of
prior hops burst aggregation/accumulation).

Shaper/IR and Damper rectify this problem directly, albeit differently - resulting
also to an in-time/work-preserving vs on-time/zero-jitter service experience. And
a difference in whether we have per-hop state/processing scaling issues.

> We can simply understand the difference between IR/Damper option and time-rank option
>
>
> as below:
>
>
> In IR/Damper option, early arrived packet (note: compared to the previous arrived
> packet) is delayed in the shaping buffer for a period of time untill its eligibility time, before

compared to the previousarrived packet ... from same flow .. only in shaper/IR, not damper
damper delays all packets up to quasi synchronous latency.

> that, it is REFUSED entry into the queueing sub-system.
>
>
> In time-rank option, early arrived packet is TOLERATED entry into the queueing
> sub-system but just with a large rank (or low priority) to not affect the urgency of scheduling
> eligibility arrivals (thus get bounded latency for them).

I remember discussions with Jakoov Stein re. draft-stein-srtsn, which is AFAIK also an
instance of such a time-rank forwarding with deadlines, and he pointed out in later
parts of the discussion that his (students) research also showed that the mechanism
is a heuristic bounded latency solution, but not a deterministic.

> That is, even in the case of bursts accumulation, a well-designed scheduling can still get a
> bounded latency.

And i fundamentally disagree, because i can not find a principle in these mechanisms
that restores per-flow TSpec increase correctly. To elaborate iin more detail on my
4 * 100 Mbps flows via Gbps link: WFQ would not restore a 1 Mbps time difference between
distorted packets p-1 and p of my example flow, it would only limit the burst to be
equal or less to that of a 2.5 Mbps flow in the best case, because this is effectively what packet
bursts of 400 * 1 Mbps across a 1 Gbps link do get with WFQ. But if the distorted p-1 and
p packet happen to get into a burst of less than 400 flows packets, then each of p-1 and
p would get an even higher share of the available 1 Gbps bandwidth and thus their distortion
would stay higher. When there is no contention on some following hop, their distortion would
not be eliminated at all.

> However, I agree with you that there is still difference result of the amount of bursts accumulation
> for in-time and on-time mode, with different buffer cost.

My unfortunate feeling is that there is no way to build an in-time/work-preserving
bounded latency solution that is per-hop stateless, because i need shaper/IR for
in-time/work-preserving, and i can not build IR/shaper per-hop stateless.

Note that our LBF work  (https://ieeexplore.ieee.org/document/9110431) equally attempted to
provide such a deadline, and departure priority calculation based latency management - in-time.
But we also failed to figure out how to provide a bounded latency calculus because of this
TSpec distortion issue. Which is why we then went to the gLBF approach.

And of course it is extremely annoying because as soon as you do such
an advanced scheduler (PIFO, per-packet calculation of departure priority/time), you do
minimize the effect of course (like WQF and C-SCORE and LBF do), but you don't eliminate it.

Aka: close, but i fear no deterministic cigar.

And btw: I'd be happy to be proven wrong, but the per-hop math/calculus that is used in rfc2212
and elsewhere is all based on the premise of per-hop known and same TSpec  for the flows.

Cheers
    Toerless
>
>
>
>
>
>
> Regards,
>
>
> PSF
>
>
>
>
>
>
>
>
>
>
>
> Original
>
>
>
> From: ToerlessEckert <[e-mail]<mailto:[e-mail]>>
> To: 彭少富10053815;
> Cc: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>;[e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>;[e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>;
> Date: 2023年09月22日 01:15
> Subject: Re: [Detnet] FYI: draft-eckert-detnet-glbf-01.txt
>
>
> Peng,
>
> I am always trying to escape having to get into the depts of the math, and instead
> trying to find as simple as possible example. And the text from rfc2212 may be somewhat
> misleading when it talks about classes.
>
> The simple logic, why i think that WFQ does not help the math over FIFO are this:
>
> - I can have all flows with the same r_i and b_i, and then all flows are treated equal.
> - I can all bursts be single packets , in which case the queue i build up is just
>   one packet/flow.
>
> In these cases, i think WFQ will not treat packets differently from FIFO. Correct
> me if i am wrong. And the problem of TSpec distortion still exists.
>
> Aka:  WFC (and in extrapolation C-SCORE), woul likely have benefits reducing the
> TSpec distortion to some degree if b_i is multiple packets, because then a back-to-back
> burst of packets from one flow would be broken up, but if the interface serving
> rate is still significantly higher than the sum(r_i), then i will still continue
> to pass bursts hop-by-hop that lead to TSpec distortion (IMHO).
>
> Cheers
>     Toerless
>
>
> On Thu, Sep 21, 2023 at 12:02:01PM +0800, [e-mail]<mailto:[e-mail]> wrote:
> > Hi Jinoo,
> >
> >
> >
> >
> >
> >
> > Thanks for your explanation.
> >
> >
> >
> >
> >
> >
> > I agree with you that for a single observed flow, especialy with ideal fluid model,
> >
> >
> > each scheduler in the network can provide a guaranteed service for it, even in the case of
> >
> >
> > flow aggregation.
> >
> >
> >
> >
> >
> >
> > For example, multiple flows, each with arrival curve A_i(t) = b_i + r_i * t, may belongs to the
> >
> >
> > same traffic class and consume the resources (burst and bandwidth) of the same
> >
> >
> > out-interface on the intermediate node. Suppose that the scheduler provide a rate-latency
> >
> >
> > service curve R*(t - T) for that traffic class, where R >= sum(r_i). Then, if each flow arrived
> >
> >
> > idealy, i.e., complied its own arrival curve, for the oberserved flow, it will be ensured to get a
> >
> >
> > guaranteed service rate R' from the total service rate R.
> >
> >
> >
> >
> >
> >
> > Suppose that on each node the guaranteed service curve for the observed flow (e.g, flow 1)
> >
> >
> > is R'*(t - T') , then, according to "Pay Bursts Only Once" rule, the E2E latency may be:
> >
> >
> > T' * hops + b_1/R'. It seems that E2E latency just consider the burst of flow 1 (i.e., b_1) and
> >
> >
> > only once, and never consider other flows..
> >
> >
> >
> >
> >
> >
> > However,  the truth is hidden in T'.
> >
> >
> >
> >
> >
> >
> > According to section 6 "Aggregate Scheduling" discussion in netcal book,  the guaranteed
> >
> >
> > service curve for flow 1, i.e., R'*(t - T'), may be deduced by:
> >
> >
> >     R*(t - T) - sum(b_2 + r_2*t, b_3 + r_3*t, ..., b_n + r_n*t)
> >
> >
> > = (R - r_2 - r_3 -...- r_n) * (t - ((b_2 + b_3 + ... + b_n + R*T) /  (R - r_2 - r_3 -...- r_n)))
> >
> >
> >
> >
> >
> >
> > Thus, R' equals to (R - r_2 - r_3 -...- r_n),
> >
> >
> > and T' equals to ((b_2 + b_3 + ... + b_n + R*T) /  (R - r_2 - r_3 -...- r_n)).
> >
> >
> > It can be seen that all bursts of other flows contribute the result of T', on each node,
> >
> >
> > again and again.
> >
> >
> >
> >
> >
> >
> > If we take each b_i as 1/n of B, r_i as 1/n of R, T as 0, to simply compare the above PBOO
> >
> >
> > based latency estiamation and traffic class based latency estimation, we may find that
> >
> >
> > the former is b_1/r_1 + hops*(n-1)*b_1/r_1, while the later is b_1/r_1. It is amazing that
> >
> >
> > the former is about n times the latter .
> >
> >
> >
> >
> >
> > Please correct me if I misunderstand.
> >
> >
> >
> >
> >
> >
> > Regards,
> >
> >
> > PSF
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Original
> >
> >
> >
> > From: JinooJoung <[e-mail]<mailto:[e-mail]>>
> > To: 彭少富10053815;
> > Cc: Toerless Eckert <[e-mail]<mailto:[e-mail]>>;DetNet WG <[e-mail]<mailto:[e-mail]>>;[e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>;
> > Date: 2023年09月19日 16:13
> > Subject: Re: [Detnet] FYI: draft-eckert-detnet-glbf-01.txt
> >
> >
> >
> >
> >
> >
> > Hello Shaofu.
> >
> >
> > Thanks for the most relevant question, which, as you have said, is related to Toerless' last question regarding "reshaping on merge points".
> >
> >
> > If I may repeat Toerless' question below:
> >
> > "If you are relying on the same math as what rfc2212 claims, just replacing
> > stateful WFQ with stateless, then it seems to me that you would equally need
> > the shapers demanded by rfc2212 on merge-points. I do not see them in C-SCORE."
> >
> > And your question is:
> >
> > " "Pay Bursts Only Once"  may be only applied in the case that the network provides
> > a dedicated service rate to a flow. Our network naturally aggregates flows at every node,
> > therefore does not dedicate a service rate to a flow, and PBOO does not apply."
> > My answer is:
> >
> > "Pay Bursts Only Once" is applied in the case that the network provides
> > a GUARANTEED service rate to a flow.
> >
> > Fair queuing, C-SCORE, or even FIFO scheduler can guarantee a service rate to a flow.
> >
> > As long as a flow, as a single entity, is guaranteed a service rate, it is not considered aggregated or merged.
> > Therefore reshaping is not necessary, and PBOO holds.
> >
> > Below is my long answer. Take a look if you'd like.
> >
> > Fair queuing, Deficit round robin, or even FIFO scheduler guarantees a service rate to a flow,
> > if the total arrival rate is less than the link capacity.
> > The only caveat is that FIFO can only guarantee the service rate that is equal to the arrival rate,
> > while FQ and DRR can adjust the service rate to be larger than the arrival rate.
> > If such rate-guaranteeing schedulers are placed in a network,
> > then a flow is guaranteed to be served with a certain service rate,
> > and not considered to be "aggregated", in the intermediate nodes.
> >
> > RFC2212, page 9~10, in subsection "Policing", states the followings:
> >
> > "Reshaping is done at all heterogeneous source branch points and at all source merge points."
> >
> >
> > "Reshaping need only be done if ..."
> >
> >
> >
> > "A heterogeneous source branch point is a spot where the
> > multicast distribution tree from a source branches to multiple
> > distinct paths, and the TSpec’s of the reservations on the various
> > outgoing links are not all the same."
> >
> > "A source merge point is where the distribution paths or trees from two different sources
> > (sharing the same reservation) merge."
> >
> > In short, RFC2212 states that reshaping CAN be necessary at the flow aggregation and deaggregation points.
> >
> > Flow aggregation and deaggregation are something happening usually at the network boundary, between networks, etc,
> > with careful planning. Flow multiplexing into a FIFO is not considered an aggregation.
> >
> > Best,
> >
> > Jinoo
> >
> >
> >
> >
> > On Tue, Sep 19, 2023 at 10:57 AM <[e-mail]<mailto:[e-mail]>> wrote:
> >
> >
> >
> >
> >
> >
> > Hi Jinoo, Toerless,
> >
> >
> >
> >
> >
> >
> > Sorry to interrupt your discussion.
> >
> >
> >
> >
> >
> >
> > According to NetCal book (https://leboudec.github.io/netcal/latex/netCalBook.pdf),
> >
> >
> > "Pay Bursts Only Once"  may be only applied in the case that the network provide
> >
> >
> > a dedicate service rate (may be protected by a dedicate queue, or even a dedicate
> >
> >
> > sub-link) for the observed flow, such as the guarantee service defined in RFC2212.
> >
> >
> > In brief, there is no other flows sharing service rate with the observed flow. That is,
> >
> >
> > there is no fan-in, no so-called " competition flows belongs to the same traffic class
> >
> >
> > at the intermediate node".
> >
> >
> >
> >
> >
> >
> > Traffic class is often used to identify flow aggregation, which is not the case that
> >
> >
> > "Pay Bursts Only Once" may be applied. It seems that in IP/MPLS network, flow
> >
> >
> > aggregation is naturely. The picture Toerless showed in the previous mail is exactly
> >
> >
> > related with flow aggregation, i.e, the observed flow may be interfered by some
> >
> >
> > competition flows belongs to the same traffic class at node A and again interfered by
> >
> >
> > other competition flows belongs to the same traffic class at node B separately.
> >
> >
> >
> >
> >
> >
> > Please correct me if I misunderstood.
> >
> >
> >
> >
> >
> >
> > Regards,
> >
> >
> > PSF
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > detnet mailing list
> > [e-mail]<mailto:[e-mail]>
> > https://www.ietf.org/mailman/listinfo/detnet
> >
> >
> > Toerless,
> > It seems that you argue two things in the last email.
> >
> > 1) Your first argument: The E2E latency is the sum of per-hop latencies..
> >
> > You are half-right.
> > According to RFC2212, page 3, the generic expression for the E2E latency bound of a flow is:
> >
> > [(b-M)/R*(p-R)/(p-r)] + (M+Ctot)/R + Dtot.    (1)
> >
> >
> > Let's call this expression (1).
> >
> > Here, b is the max burst, M the max packet length, R the service rate, p the peak rate, r the arrival rate;
> > and Ctot and Dtot are the sums of C and D, which are so called "error terms",  over the hops.
> > Thus, the E2E latency bound can be a linear function of the hop count,
> > since Ctot and Dtot are functions of hop count.
> > However, the first term [(b-M)/R*(p-R)/(p-r)], which includes b, is not..
> > So you can see the E2E latency is NOT just the sum of per-hop latencies..
> >
> > 2) Your second argument: C-SCORE cannot be free from burst accumulation and other flows' burst.
> > My short answer: It is free from burst accumulation or other flows' burst.
> >
> > Imagine an ideal system, in which your flow is completely isolated.
> > It is ALONE in the network, whose link has the rate R in every hop.
> > No other flows at all.
> >
> > Assume the flow's arrival process is indeed rt+b. At time 0, b arrives instantly.
> > (This is the worst arrival, such that it makes the worst latency.)
> > Then your flow experiences the worst latency b/R at the first node,
> > and M/R (transmission delay of a packet) at the subsequent nodes.
> >
> >
> > I.e.  (b-M)/R + H*M/R, where H is the hop count.
> >
> > This is a special case of (1), where R is the same with r, C is M, and D is zero.
> >
> >
> > Fair queuing and C-SCORE are the best approximations of such an ideal system, with D equals Lmax/LC,
> >
> > where Lmax and LC are the max packet length in the link and the capacity of the link, respectively.
> > Therefore the E2E latency bound of C-SCORE is
> >
> > (b-M)/R + H*(M/R + Lmax/LC),
> >
> >
> > which is again another special case of  (1).
> > Note that b is divided by R, not LC.
> >
> > It is well known that a FIFO scheduler's D (error term) is a function of the sum of other flows' bursts.
> > Their E2E latency bounds are approximately
> > (b-M)/R + H*[M/R + (Sum of Bursts)/LC].
> >
> >
> > ATS or UBS also relies on FIFO, but the bursts are suppressed as the initial values,
> >
> > therefore enjoys a much better E2E latency expression.
> >
> > Hope this helps.
> >
> > Best,
> > Jinoo
> >
> >
> >
> >
> >
> >
> > On Sun, Sep 17, 2023 at 1:42 AM Toerless Eckert <[e-mail]<mailto:[e-mail]>> wrote:
> >
> > On Thu, Sep 14, 2023 at 08:08:24PM +0900, Jinoo Joung wrote:
> >  > Toerless, thanks for the reply.
> >  > In short:
> >  >
> >  > C-SCORE's E2E latency bound is NOT affected by the sum of bursts, but only
> >  > by the flow's own burst.
> >
> >                      +----+
> >            If1   --->|    |
> >            ...       |  R |x--->
> >            if100 --->|    |
> >                      +----+
> >
> >  If you have a router with for example 100 input interfaces, all sending packets
> >  to the same ooutput interfaces, they all will have uncorrelated flows arriving
> >  from the different interfaces, and at least each interface can have a packet
> >  arriving t the same time in R.
> >
> >  The basic caluculus of UBS is simply the most simple and hence conservative,
> >  assuming all flows packet can arrive from different interface without
> >  rate limits. But of course you can do the latency calculations in a tighter
> >  fashion for UBS. WOuld be interesting to learn if IEEE for TSN-ATS (Qcr) was
> >  looking into any of this. E.g.: applying line shaping for the aggregate of
> >  flows from the same input interface and incorporating the service curve of
> >  the node.
> >
> >  In any case, whether the tighter latency calculus is you do for Qcr, it
> >  is equally applicable to gLBF.
> >
> >  > It is bounded by (B-L)/r + H(L/r + Lmax/R), where B is the flow's max burst
> >  > size.
> >
> >  See above picture. How could the packet in question not also suffer the
> >  latency introduced by the other 99 packets randomnly arriving at almos
> >  the same time, just a tad earlier.
> >
> >  > You can also see that B appears only once, and not multiplied by the hop
> >  > count H.  So the burst is paid only once.
> >
> >  C-SCORE is a stateless fair queuing (FQ) mechanism, but making FQ state,
> >  does not change the fact that FQ itself does not eliminate the
> >  burst accumulation on merge points such as shon in the picture. rfc2212
> >  which for example also recommends FQ independently mandates the use of
> >  reshaping.
> >
> >  > Please see inline marked with JJ.
> >
> >  yes, more inline.
> >
> >  > On Thu, Sep 14, 2023 at 3:34 AM Toerless Eckert <[e-mail]<mailto:[e-mail]>> wrote:
> >  [cutting off dead leafs]
> >
> >  > > I was only talking about the E2E latency bound guaranteeable by DetNet
> >  > > Admission Control (AC).
> >  >
> >  > JJ: Admission control has two aspects.
> >  > First, guaranteeing the total arrival rate (or allocated service rate) does
> >  > not exceed the link capacity.
> >  > Second, assuring a service level agreement (SLA), or a requested latency
> >  > bound (RSpec) to a flow.
> >  > By "E2E latency bound guaranteeable by DetNet Admission Control (AC)",
> >  > I think you mean that negotiating the SLA first, and then, based on the
> >  > negotiated latency, allocating per-hop latency to a flow.
> >  > I suggest this per-hop latency allocation, and enforcing that value in a
> >  > node, is not a good idea,
> >  > since it can harm the advantage of "pay burst only once (PBOO)" property.
> >
> >  Its the only way AFAIK to achieve scalability in number of hops and number of
> >  flows by being able to calculate latency as a linear composition of per-hop
> >  latencies. This is what rfc2212 does, this is what Qcr does, *CQF, gLBF and so on.
> >
> >  > PBOO can be interpreted roughly as: If your burst is resolved at a point in
> >  > a network, then it is resolved and does not bother you anymore.
> >  > However, in the process of resolving, the delay is inevitable.
> >
> >  Remember tht the problemtic burstyness is an increase in the intra-flow
> >  burstyness resulting from merge-point based unexpected latency
> >  to p-1 of the flow followed by no such delay for packet p of the same flow,
> >  and hence clustering p-1 and p closer together, making it exceed its
> >  reserved burst size.
> >
> >  This can not be compensated for in a work-conserving way by simply capturing
> >  the per-hop latency of each packet p-1 and p alone, but it requires a shaper
> >  (or UR) to get rid off.
> >
> >  > JJ: Because you don't know exactly where the burst is resolved,
> >  > when you calculate the per-node latency bound, the latency due to the burst
> >  > has to be added as a portion of the latency bound.
> >  > Thus the sum of per-node bounds is much larger than the E2E latency bound
> >  > calculated with seeing the network as a whole.
> >
> >  Consider a flow passing through 10 hops. On every hop, you have potentially
> >  merge-point with new traffic flows and new burst collision. All that we do
> >  in the simple UBS/Qcr calculus is to take the worst case into account,
> >  where worst case may not even be admitted now, but which could be admitted
> >  in the future, and at that point in time you do not want to go back and
> >  change the latency guarantee for your already admitted flow.
> >
> >          Src2  Src3  Src4  Src5  Src6  Src7  Src8   Src9 Src10
> >           |     |     |     |     |     |     |     |     |
> >           v     v     v     v     v     v     v     v     v
> >   Src1 -> R1 -> R2 -> R3 -> R4 -> R5 -> R6 -> R7 -> R8 -> R9 -> R10 -> Rcv1
> >                  |     |     |     |     |     |     |     |     |
> >                  v     v     v     v     v     v     v     v     v
> >                 Rcv2  Rcv3  Rcv4  Rcv5  Rcv6  Rcv7  Rcv8  Rcv9  Rcv10
> >
> >  Above is example, where Flow 1 from Src1 to Rcv1 will experience such
> >  merge-point burst accumulation issue on every hop - worst case.  And as mentioned
> >  before, yes, when you use the simple calculus, you're also overcalculating
> >  the per-hop latency for flows that e.g.: all run in parallel to Src1, but
> >  that is just a matter of using stricter network calculus. And because
> >  Network Calculus is complex, and i didn't want to start becoming an expert on
> >  it, i simply built the stateless solution in a way where i can reuse a
> >  pre-existing, proven and used-in-standards (Qcr) queuing-model, calculus.
> >
> >
> >  > JJ: If you enforce resolving it at the network entrance by a strict
> >  > regulation, then you may end up resolving while not actually needed to.
> >  > However, this approach is feasible. I will think more about it.
> >
> >  I think the flow-interleaving is a much more fundamental issue of higher
> >  utilization with large number of flows all with low bitrates. Look at
> >  the examples of the draft-eckert-detnet-flow-interleaving, and tell me
> >  how else but time-division-multiplexing one would be able to solve this.
> >  Forget the complex option where flows from diffeent ingres routers to
> >  different egres routers are involved. Just the most simple problem of
> >  one ingres router PE1, maybe 10 hops through thre network to a PE2, and 10,000
> >  acyclic flows going to thre same egress router PE2.
> >
> >  Seems to me quite obvious tht you can as well resolve the burst on ingress
> >  PE1 instead of hoping, and getting complex math by trying to do this
> >  on further hops along the path.
> >
> >  Cheers
> >      Toerless
> >
> >  > >
> >  > > > It can be a function of many factors, such as number of flows, their
> >  > > > service rates, their max bursts, etc.
> >  > > > The sum of service rates is a deciding factor of utilization.
> >  > >
> >  > > Given how queuing latency always occurs from collision of packets in
> >  > > buffers,
> >  > > the sum of burst sizes is the much bigger problem for DetNet than service
> >  > > rates. But this is a side discuss.
> >  > >
> >  >
> >  > JJ: That is a good point. So we should avoid queuing schemes whose latency
> >  > bounds are affected by the sum of bursts.
> >  > Fortunately, C-SCORE's E2E latency bound is NOT affected by the sum of
> >  > bursts, but only by the flow's own burst.
> >  > It is bounded by (B-L)/r + H(L/r + Lmax/R), where B is the flow's max burst
> >  > size.
> >  > You can also see that B appears only once, and not multiplied by the hop
> >  > count H.
> >  > So the burst is paid only once.
> >  >
> >  >
> >  > > > So, based on an E2E latency bound expression, you can guess the bound at
> >  > > > 100% utilization.
> >  > > > But you can always fill the link with flows of any burst sizes, therefore
> >  > > > the guess can be wrong.
> >  > >
> >  > > "guess" is not a good work for DetNet.
> >  > >
> >  > > A DetNet bounded latency mechanisms needs a latency bound expression
> >  > > (calculus)
> >  > > to be a guaranteeable (over)estimate of the bounded latency independent of
> >  > > what other competing traffic there may be in the future. Not a "guess".
> >  > >
> >  >
> >  > JJ: Right. We should not guess. We should be able to provide an exact
> >  > mathematical expression for latency bound.
> >  > Because you argued in the previous mail that the latency bound should be
> >  > obtained based on 100% utilization,
> >  > I was trying to explain why that should not be done.
> >  >
> >  >
> >  > > > Admission control, on the other hand, can be based on assumption of high
> >  > > > utilization level, but not necessarily 100%.
> >  > > >
> >  > > > You do not assume 100% utilization when you slot-schedule, don't you?
> >  > >
> >  > > I don't understand what you mean with slot-schedule, can you please
> >  > > explain ?
> >  > >
> >  >
> >  > JJ: Slot-scheduling, which is a common term in the research community,
> >  > is a mapping of flow into a slot (or cycle) in a slot (or cycle) based
> >  > queuing methods, such as CQF.
> >  > When we say a schedulability, it usually means whether we can allocate
> >  > requesting flows into slots (cycles) with a preconfigured cycle's length
> >  > and number.
> >  >
> >  >
> >  > > > So "incremental scheduling" is now popular.
> >  > >
> >  > > Not sure what you mean with that term.
> >  >
> >  >
> >  > JJ: The incremental scheduling means, when a new flow wants to join the
> >  > network, then the network examines the schedulability of the flow,
> >  > without altering the existing flows' schedule.
> >  >
> >  >
> >  > >
> >  > >
> >  > > I am only thinking about "admitting" when it comes to bounded end-to-end
> >  > > latency, aka: action by the AC of the DetNet controller-plane, and
> >  > > yes, that needs to support "on-demand" (incremental?), aka: whenever
> >  > > a new flow wants to be admitted.
> >  > >
> >  > > > 2) In the example I gave, the two flows travel the same path, thus the
> >  > > > second link's occupancy is identical to the first one.
> >  > > > Thus the competency levels in two links are the same, contrary to your
> >  > > > argument.
> >  > >
> >  > > I guss we started from different assumptions about details not explicitly
> >  > > mentioned.  For example, i guess we both assume that the sources connect
> >  > > to the
> >  > > first router with arbitrary high interfaces so that we could ever get close
> >  > > to 2B/R on the first interface.
> >  > >
> >  > > But then we differed in a core detail. here is my assumption for
> >  > > the topology / admission control:
> >  > >
> >  > >                      Src3          to R4
> >  > >               +----+   \  +----+  /  +----+
> >  > >       Src1 ---|    |    \-|    |-/   |    |
> >  > >       Src2 ---| R1 |x-----| R2 |x----| R3 |
> >  > >               |    |.     |    |.    |    |
> >  > >               +----+.     +----+.    +----+
> >  > >                     |           |
> >  > >                 2B buffer    2B buffer
> >  > >                 R srv rate   R srv rate
> >  > >
> >  > > Aka: in my case, i was assuming that there could be case where
> >  > > the interface from R2 to R3 could have a 2B/R queue (and not
> >  > > assuming further optimizations in calculus). E.g.: in some
> >  > > other possible scenario, Src2 sends to R2, and Src3 and Src1 to
> >  > > R3 for example.
> >  > >
> >  >
> >  > JJ: You can come up with an example that your scheme works well.
> >  > But that does not negate the counterexample I gave.
> >  >
> >  > JJ: Again, there are only two flows.
> >  > And, B is not the buffer size. B is the max burst size of a flow.
> >  > R is the link capacity.
> >  > Please review carefully the example I gave.
> >  >
> >  >
> >  > >
> >  > > You must have assumed that the totality of the DetNet admission control
> >  > > relevant topology is this:
> >  > >
> >  > >               +----+      +----+     +----+
> >  > >       Src1 ---|    |      |    |     |    |
> >  > >       Src2 ---| R1 |x-----| R2 |x----| R3 |
> >  > >               |    |.     |    |.    |    |
> >  > >               +----+.     +----+.    +----+
> >  > >                     |           |
> >  > >                 2B buffer    2B buffer
> >  > >                 R srv rate   R srv rate
> >  > >
> >  > > Aka: DetNet admission control would have to be able to predict that
> >  > > under no permitted admission scenario, R2 would build a DetNet queue,
> >  > > so even when Src1 shows up as the first and only flow, the admission
> >  > > control could permit a latency to R3 of 2B/R - only for the maximum
> >  > > delay through R1 queue and 0 for R2 queue.
> >  > >
> >  > > But if this is the whole network and the admission control logic
> >  > > can come to this conclusion, then of course it could equally do the
> >  > > optimization and not enable gLBF Dampening on R2 output
> >  > > interface, or e.g.: set MAX=0 or the like. An e'voila, gLBF
> >  > > would also give 2B/R - but as said, i think it's not a deployment
> >  > > relevant example.
> >  >
> >  >
> >  > JJ: If you can revise and advance the gLBF, that would be great.
> >  > I am willing to join that effort, if you would like to.
> >  >
> >  >
> >  > >
> >  > >
> >  > > Cheers
> >  > >     Toerless
> >  > >
> >  > > > Please see inline with JJ.
> >  > > >
> >  > > > Best,
> >  > > > Jinoo
> >  > > >
> >  > > > On Wed, Sep 13, 2023 at 9:06 AM Toerless Eckert <[e-mail]<mailto:[e-mail]>> wrote:
> >  > > >
> >  > > > > On Fri, Jul 21, 2023 at 08:47:18PM +0900, Jinoo Joung wrote:
> >  > > > > > Shaofu, thanks for the reply.
> >  > > > > > It is my pleasure to discuss issues like this with you.
> >  > > > > >
> >  > > > > > The example network I gave is a simple one, but the scenario is the
> >  > > worst
> >  > > > > > that can happen.
> >  > > > > > The E2E latency bounds are thus,
> >  > > > > >
> >  > > > > > for Case 1: ~ 2B/R
> >  > > > > > for Case 2: ~ 2 * (2B/R)
> >  > > > >
> >  > > > > This is a bit terse, let me try to expand:
> >  > > > >
> >  > > > > Case 1 is FIFO or UBS/ATS, right / Case 2 is gLBF ?
> >  > > > >
> >  > > >
> >  > > > JJ: Correct.
> >  > > >
> >  > > >
> >  > > > >
> >  > > > > Assuming i am interpreting it right, then this is inconsistent with
> >  > > your
> >  > > > > setup: You said all links are the same so both hops do have the same
> >  > > > > buffer and rates, so the admission controller also expect to
> >  > > > > have to put as many flows on second link/queue that it fills up 2B.
> >  > > > >
> >  > > >
> >  > > > JJ: Yes, two links are identical. However, as I have mentioned,
> >  > > > an E2E latency bound is calculated based on a given network environment.
> >  > > > We don't always consider a filled up link capacity.
> >  > > > BTW, B is the max burst size of the flow.
> >  > > >
> >  > > >
> >  > > >
> >  > > > >
> >  > > > > You just then made an example, where there was never such an amount
> >  > > > > of competing traffic on the second hop. But tht does not mean that
> >  > > > > the admission controller could guarantee in UBS/ATS would have
> >  > > > > less per-hop latency than 2B/R.
> >  > > >
> >  > > >
> >  > > > JJ: Again, two links are identical and two flows travel both links.
> >  > > > The difference between Case 1 and Case 2 is not because of the different
> >  > > > competition level (they are identical.)
> >  > > > but because of the non-work conserving behaviour of the second link in
> >  > > Case
> >  > > > 2.
> >  > > >
> >  > > >
> >  > > > >
> >  > > > > If the admission controller knew there would never be a queue on the
> >  > > > > second hop, then gLBF likewise would not need to do a Damper on the
> >  > > > > second hop. Hence as i said previously, the per-hop and end-to-end
> >  > > > > bounded latency guarantee is the same between UBS and gLBF.
> >  > > > >
> >  > > > > > And again, these are the WORST E2E latencies that a packet can
> >  > > experience
> >  > > > > > in the two-hop network in the scenario.
> >  > > > >
> >  > > > > Its not the worst case latency for the UBS case. you just did not have
> >  > > > > an example to create the worst case amount of competing traffic. Or you
> >  > > > > overestimed the amount of buffering and hence per-hop latency for the
> >  > > > > UBS/ATS casee.
> >  > > > >
> >  > > > > > In any network that is more complex, the E2E latency bounds of two
> >  > > > > schemes
> >  > > > > > are very different.
> >  > > > >
> >  > > > > Counterexample:
> >  > > > >
> >  > > > > You have a network with TSN-ATS. You have an admission controller.
> >  > > > > You only have one priority for simplicity of example.
> >  > > > >
> >  > > > > You do not want to dymamically signal changed end-to-end latencies
> >  > > > > to applications... because its difficult. So you need to plan
> >  > > > > for worst-case bounded latencies under maximum amount of traffic
> >  > > > > load. In a simple case this means you give each interface
> >  > > > > a queue size B(i)/r = 10usec. Whenever a new flow needs to be
> >  > > > > added to the network, you find a path where all the  buffers
> >  > > > > have enough space for your new flows burst and you signal
> >  > > > > to the application that the end-t-end guaranteed latency is
> >  > > > > P(path)+N*10usec, where P is the physical propagation latecy of
> >  > > > > the path and N is the number of hops it has.
> >  > > > >
> >  > > > > And in result, all packets from the flow will arrive with
> >  > > > > a latency between P(path)...P(path)+N*10usec - depending
> >  > > > > on network load/weather.
> >  > > > >
> >  > > > > Now we replace UBS in the routers with gLBF. What changes ?
> >  > > > >
> >  > > > > 1) With UBS the controller still had to signal every new and
> >  > > > > to-be-deleted flow to every router along it path to set up the
> >  > > > > IR for the flow. This goes away (big win).
> >  > > > >
> >  > > > > 2) The forwarding is in our opinion cheaper/faster to implement
> >  > > > > (because of lack of memory read/write cycle of IR).
> >  > > > >
> >  > > > > 3) The application now sees all packets arrive at fixed latency
> >  > > > > of P(path)+N*10usec. Which arguably to the application that
> >  > > > > MUST have bounded latency is from all examples i know
> >  > > > > seen rather as a benefit than as a downside.
> >  > > > >
> >  > > > > Cheers
> >  > > > >     Toerless
> >  > > > >
> >  > > > >
> >  > > > > >
> >  > > > > > Best,
> >  > > > > > Jinoo
> >  > > > > >
> >  > > > > > On Fri, Jul 21, 2023 at 8:31 PM <[e-mail]..cn<mailto:[e-mail]..cn>> wrote:
> >  > > > > >
> >  > > > > > >
> >  > > > > > > Hi Jinoo,
> >  > > > > > >
> >  > > > > > >
> >  > > > > > > I tried to reply briefly. If Toerless have free time, can confirm
> >  > > it.
> >  > > > > > >
> >  > > > > > >
> >  > > > > > > Here, when we said latency bound formula, it refers to worst-case
> >  > > > > latency.
> >  > > > > > >
> >  > > > > > >
> >  > > > > > > Intuitively, the worst-case latency for gLBF (damper + shaper +
> >  > > > > scheduler)
> >  > > > > > >
> >  > > > > > > is that:
> >  > > > > > >
> >  > > > > > >     damping delay per hop is always 0. (because scheduling delay =
> >  > > MAX)
> >  > > > > > >
> >  > > > > > >     shaping delay is always 0. (because all are eligibility
> >  > > arrivals)
> >  > > > > > >
> >  > > > > > >     scheduling delay is always MAX (i.e., concurent full burst
> >  > > from all
> >  > > > > > > eligibility arrivals on each hop)
> >  > > > > > >
> >  > > > > > >
> >  > > > > > > Similarly, the worst-case latency for UBS (shaper + scheduler) is
> >  > > that:
> >  > > > > > >
> >  > > > > > >     shaping delay is always 0. (because all are eligibility
> >  > > arrivals)
> >  > > > > > >
> >  > > > > > >     scheduling delay is always MAX (i.e., concurent full burst
> >  > > from all
> >  > > > > > > eligibility arrivals on each hop)
> >  > > > > > >
> >  > > > > > >
> >  > > > > > > Thus, the worst-case latency of gLBF and UBS is the same.
> >  > > > > > >
> >  > > > > > > Your example give a minimumal latency that may be expierenced by
> >  > > UBS,
> >  > > > > but
> >  > > > > > >
> >  > > > > > > it is not the worst-case latency. In fact, your example is a simple
> >  > > > > > > topology that only
> >  > > > > > >
> >  > > > > > > contains a line without fan-in, that cause scheduling delay almost
> >  > > a
> >  > > > > > > minimumal
> >  > > > > > >
> >  > > > > > > value due to no interfering flows.
> >  > > > > > >
> >  > > > > > >
> >  > > > > > > Regards,
> >  > > > > > >
> >  > > > > > > PSF
> >  > > > > > >
> >  > > > > > >
> >  > > > > > >
> >  > > > > > > Original
> >  > > > > > > *From: *JinooJoung <[e-mail]<mailto:[e-mail]>>
> >  > > > > > > *To: *彭少富10053815;
> >  > > > > > > *Cc: *[e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>;[e-mail]<mailto:[e-mail]> <
> >  > > [e-mail]<mailto:[e-mail]>>;
> >  > > > > > > [e-mail]<mailto:[e-mail]> <
> >  > > [e-mail]<mailto:[e-mail]>>;
> >  > > > > > > *Date: *2023年07月21日 14:10
> >  > > > > > > *Subject: **Re: [Detnet] FYI: draft-eckert-detnet-glbf-01.txt*
> >  > > > > > > _______________________________________________
> >  > > > > > > detnet mailing list
> >  > > > > > > [e-mail]<mailto:[e-mail]>
> >  > > > > > > https://www.ietf.org/mailman/listinfo/detnet
> >  > > > > > >
> >  > > > > > > Hello Toerless,
> >  > > > > > > I have a comment on your argument.
> >  > > > > > > This is not a question, so you don't have to answer.
> >  > > > > > >
> >  > > > > > > You argued that the gLBF + SP has the same latency bound formula
> >  > > with
> >  > > > > UBS
> >  > > > > > > (equivalently ATS IR + SP).
> >  > > > > > > The IR is not a generalized gLBF, so they do not have the same
> >  > > bound.
> >  > > > > > >
> >  > > > > > > In short, ATS IR is a rate-based shaper so it enjoys "Pay burst
> >  > > only
> >  > > > > once"
> >  > > > > > > property.
> >  > > > > > > gLBF is not. So it pays burst every node.
> >  > > > > > >
> >  > > > > > > Consider a simplest example, where there are only two identical
> >  > > flows
> >  > > > > > > travelling the same path.
> >  > > > > > > Every node and link in the path are identical.
> >  > > > > > >
> >  > > > > > > Case 1: Just FIFO
> >  > > > > > > Case 2: gLBF + FIFO
> >  > > > > > >
> >  > > > > > > In the first node, two flows' max bursts arrive almost at the same
> >  > > time
> >  > > > > > > but your flow is just a little late.
> >  > > > > > > Then your last packet in the burst (packet of interest, POI)
> >  > > suffers
> >  > > > > > > latency around 2B/R, where B is the burst size and R is the link
> >  > > > > capacity.
> >  > > > > > > This is true for both cases.
> >  > > > > > >
> >  > > > > > > In the next node:
> >  > > > > > > In Case 1, the POI does not see any packet queued. so it is
> >  > > delayed by
> >  > > > > its
> >  > > > > > > own transmission delay.
> >  > > > > > > In Case 2, the burst from the other flow, as well as your own
> >  > > burst,
> >  > > > > > > awaits the POI. So the POI is again delayed around 2B/R.
> >  > > > > > >
> >  > > > > > > In the case of UBS, the max bursts are legitimate, so the regulator
> >  > > > > does
> >  > > > > > > not do anything,
> >  > > > > > > and the forwarding behavior is identical to Case 1.
> >  > > > > > >
> >  > > > > > > Best,
> >  > > > > > > Jinoo
> >  > > > > > >
> >  > > > > > > On Fri, Jul 21, 2023 at 10:58 AM <[e-mail]<mailto:[e-mail]>> wrote:
> >  > > > > > >
> >  > > > > > >>
> >  > > > > > >> Hi Toerless,
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> Thanks for your response, and understand your busy situation.
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> A quick reply is that gLBF is really an interested proposal,
> >  > > which is
> >  > > > > > >> very
> >  > > > > > >>
> >  > > > > > >> similar to the function of Deadline on-time per hop.  Our views
> >  > > are
> >  > > > > > >>
> >  > > > > > >> consistent on this point. The key beneficial is to avoid burst
> >  > > > > cumulation.
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> The following example originated from the analysis of deadline
> >  > > on-time
> >  > > > > > >>
> >  > > > > > >> mode. I believe it also makes sense for gLBF. When you have free
> >  > > time,
> >  > > > > > >>
> >  > > > > > >> may verify it. The result may be helpful both for gLBF and
> >  > > deadline
> >  > > > > > >>
> >  > > > > > >> on-time mode.  Note that I didn't question mathematical proof
> >  > > about
> >  > > > > UBS,
> >  > > > > > >>
> >  > > > > > >> which get the worst-case latency based on the combination of
> >  > > > > > >>
> >  > > > > > >> "IR shaper + SP scheduler".
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> Regards,
> >  > > > > > >>
> >  > > > > > >> PSF
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> Original
> >  > > > > > >> *From: *ToerlessEckert <[e-mail]<mailto:[e-mail]>>
> >  > > > > > >> *To: *彭少富10053815;
> >  > > > > > >> *Cc: *[e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>;[e-mail]<mailto:[e-mail]> <
> >  > > > > > >> [e-mail]<mailto:[e-mail]>>;[e-mail]<mailto:[e-mail]> <
> >  > > > > > >> [e-mail]<mailto:[e-mail]>>;
> >  > > > > > >> *Date: *2023年07月21日 06:07
> >  > > > > > >> *Subject: **Re: [Detnet] FYI: draft-eckert-detnet-glbf-01..txt*
> >  > > > > > >>
> >  > > > > > >> Thanks folks for the question and discussion, I have some WG chair
> >  > > > > vultures hovering over me
> >  > > > > > >>
> >  > > > > > >> making sure i prioritize building slides now (the worst one is
> >  > > myself
> >  > > > > ;-), so i will only
> >  > > > > > >> give a brief answer and will get back to it later when i had more
> >  > > > > time.
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> The calculus that i used is from the [UBS] research paper from
> >  > > > > Johannes Specht, aka: it has
> >  > > > > > >>
> >  > > > > > >> the mathematical proof, full reference is in the gLBF draft.
> >  > > There is
> >  > > > > another, later proof of the
> >  > > > > > >>
> >  > > > > > >> calculus from Jean Yves Le Boudec in another research paper which
> >  > > i'd
> >  > > > > have to dig up, and
> >  > > > > > >>
> >  > > > > > >> depending on whom you ask one or the other is easier to read. I
> >  > > am on
> >  > > > > the UBS research paper
> >  > > > > > >>
> >  > > > > > >> side because i have not studied Jean Yves calculus book. But its
> >  > > > > really beautifully simple
> >  > > > > > >>
> >  > > > > > >> that as soon as you think of flows with only burst-size and rate
> >  > > (or
> >  > > > > period) of those burst,
> >  > > > > > >>
> >  > > > > > >> then your delay through the queue is really just the sum of
> >  > > bursts.
> >  > > > > And i just find beauty
> >  > > > > > >>
> >  > > > > > >> in simplicity. And that can not be the full answer to Jinoo, but i
> >  > > > > first need to read up more
> >  > > > > > >> on his WRR options.
> >  > > > > > >>
> >  > > > > > >> The need for doing per-hop dampening is really as i said from two
> >  > > > > points:
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> 1. Unless we do per-hop dampening, we will not get such a simple
> >  > > > > calculus and equally low latency.
> >  > > > > > >>
> >  > > > > > >> The two validation slides of the gLBF presentation show that one
> >  > > can
> >  > > > > exceed the simple
> >  > > > > > >>
> >  > > > > > >> calculated bounded latency already with as few as 9  flows across
> >  > > a
> >  > > > > single hop and arriving
> >  > > > > > >>
> >  > > > > > >> into one single queue -  unless there is per-hop dampening (or
> >  > > > > per-flow-shaper).
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> 2. I can not imagine how to safely sell router equipment and build
> >  > > > > out all desirable topologies without
> >  > > > > > >>
> >  > > > > > >> every node is able to do the dampening. And i also see it as the
> >  > > > > right next-generation challenge
> >  > > > > > >>
> >  > > > > > >> and option to make that happen in high speed hardware.
> >  > > Specifically
> >  > > > > in metro rings, every big aggregation
> >  > > > > > >>
> >  > > > > > >> ring node has potentially 100 incoming interfaces and hence can
> >  > > > > create a lot of bursts onto ring interfaces.
> >  > > > > > >>
> >  > > > > > >> Cheers
> >  > > > > > >>    Toerless
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> P.S.: The validation picture in our slides was from our Springer
> >  > > > > Journal article, so
> >  > > > > > >>
> >  > > > > > >> i can not simply put a copy on the Internet now, but ping me in
> >  > > PM if
> >  > > > > you want an authors copy.
> >  > > > > > >>
> >  > > > > > >> On Wed, Jul 12, 2023 at 11:48:36AM +0800, peng.shaofu@zte..com.cn<http://com.cn>
> >  > > > > wrote:
> >  > > > > > >> > Hi Jinoo, Toerless
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > Also thank Toerless for bringing us this interested draft.
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > For the question Jinoo pointed out, I guess, based on the
> >  > > similar
> >  > > > > analysis
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > of deadline on-time per hop, that even if all flows departured
> >  > > from
> >  > > > > the damper
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > and arrived at the queueing subsystem at the same time, each
> >  > > flow
> >  > > > > can still
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > have its worst-case latency, but just consume the next round of
> >  > > > > budget (i.e.,
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > the MAX value mentioned in the document).
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > However, consuming the next round of budget, means that it
> >  > > relies
> >  > > > > on the
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > downstream node to compensate latency, and may result a jitter
> >  > > with
> >  > > > > MAX
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > (i.e., worst-case latency). Due to this reason, deadline on-time
> >  > > > > per hop is
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > temperately removed in version-6, waiting for more strict
> >  > > proof and
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > optimization.
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > Anyway, gLBF can do same things that deadline on-time per hop
> >  > > done.
> >  > > > > The
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > following instuitive exaple is common for these two solutions.
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > Assuming that at the last node, all received flows have
> >  > > expierenced
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > almost 0 queueding delay on the upstream nodes. Traffic class-8
> >  > > has
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > per hop worst case latency 80 us (just an example, similar to
> >  > > delay
> >  > > > > level
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > of deadline),  traffic class-7 has 70 us, ... ..., traffic
> >  > > class-1
> >  > > > > has 10 us.
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > Then, at time T0, traffic class-8 arrived at the last node, it
> >  > > will
> >  > > > > dampen 80us;
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > at time T0+10us, traffic class-7 arrived, it will dampen 70us,
> >  > > and
> >  > > > > so on. At
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > T0+80us, all traffic class flows will departure from the damper,
> >  > > > > and send to
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > the same outgoing port. So, an observed packet may expierence
> >  > > > > another
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > round of worst case lantecy if other higher priority flows
> >  > > > > existing, or
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > expierence best case latency (almost 0) if other higher priority
> >  > > > > flows not
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > existing. That is, a jitter with value of worst case latency
> >  > > still
> >  > > > > exists.
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > Regards,
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > PSF
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > Original
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > From: JinooJoung <[e-mail]<mailto:[e-mail]>>
> >  > > > > > >> > To: Toerless Eckert <[e-mail]<mailto:[e-mail]>>;
> >  > > > > > >> > Cc: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>;
> >  > > > > [e-mail]<mailto:[e-mail]>
> >  > > > > > >>  <[e-mail]<mailto:[e-mail]>>;
> >  > > > > > >> > Date: 2023年07月09日 09:39
> >  > > > > > >> > Subject: Re: [Detnet] FYI: draft-eckert-detnet-glbf-01.txt
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > _______________________________________________
> >  > > > > > >> > detnet mailing list
> >  > > > > > >> > [e-mail]<mailto:[e-mail]>
> >  > > > > > >> > https://www.ietf.org/mailman/listinfo/detnet
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > Dear Toerless; thanks for the draft.
> >  > > > > > >>
> >  > > > > > >> > gLBF is an interesting approach, similar in concept to the
> >  > > Buffered
> >  > > > > Network (BN) I have introduced in the ADN Framework document.
> >  > > > > > >>
> >  > > > > > >> > The difference seems that the BN buffers only once at the
> >  > > network
> >  > > > > boundary, while gLBF buffers at every node.
> >  > > > > > >>
> >  > > > > > >> > Therefore in the BN, a buffer handles only a few flows, while in
> >  > > > > the gLBF a buffer needs to face millions of flows.
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > The implementation complexity should be addressed in the future
> >  > > > > draft, I think.
> >  > > > > > >> >
> >  > > > > > >> > I have a quick question below.
> >  > > > > > >> >
> >  > > > > > >> >    +------------------------+      +------------------------+
> >  > > > > > >> >    | Node A                  |        | Node B
> >  > >   |
> >  > > > > > >> >    |   +-+   +-+   +-+      |        |   +-+   +-+   +-+      |
> >  > > > > > >> >    |-x-|D|-y-|F|---|Q|----z -|------ |-x-|D|-y-|F|---|Q|----z- |
> >  > > > > > >> >    |   +-+   +-+   +-+      | Link |   +-+   +-+   +-+      |
> >  > > > > > >> >    +------------------------+      +------------------------+
> >  > > > > > >> >            |<- A/B in-time latency ->|
> >  > > > > > >> >            |<--A/B on-time latency ------->|
> >  > > > > > >> >
> >  > > > > > >> >                Figure 3: Forwarding with Damper and measuring
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > In Figure 3, how can F and Q guarantee the nodal latency below
> >  > > MAX?
> >  > > > > > >>
> >  > > > > > >> > Does the gLBF provide the same latency bound as that of UBS, as
> >  > > it
> >  > > > > is argued?
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > In UBS, an interleaved regulator (IR) works as the damper D in
> >  > > the
> >  > > > > gLBF.
> >  > > > > > >>
> >  > > > > > >> > IR is essentially a FIFO, whose HoQ packet is examined and
> >  > > leaves
> >  > > > > if eligible.
> >  > > > > > >>
> >  > > > > > >> > A packet's eligible time can be earlier than the time that it
> >  > > > > became the HoQ.
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > However, in gLBF, a packet has a precise moment that it needs
> >  > > to be
> >  > > > > forwarded from D.
> >  > > > > > >> > (Therefore, UBS is not a generalized gLBF.)
> >  > > > > > >>
> >  > > > > > >> > In the worst case, all the flows may want to send the packets
> >  > > from
> >  > > > > D to F at the same time.
> >  > > > > > >>
> >  > > > > > >> > If it can be implemented as such, bursts may accumulate, and the
> >  > > > > latency cannot be guaranteed.
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> > If it cannot be implemented that way, you may introduce another
> >  > > > > type of delay.
> >  > > > > > >> >
> >  > > > > > >> > Don't you need an additional mechanism for latency guarantee?
> >  > > > > > >> >
> >  > > > > > >> > Thanks a lot in advance, I support this draft.
> >  > > > > > >> >
> >  > > > > > >> > Best,
> >  > > > > > >> > Jinoo
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> >
> >  > > > > > >> > On Sat, Jul 8, 2023 at 12:05 AM Toerless Eckert <[e-mail]<mailto:[e-mail]>>
> >  > > > > wrote:
> >  > > > > > >> >
> >  > > > > > >> > Dear DetNet WG,
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> >  FYI on a newly posted  bounded latency method/proposal draft
> >  > > that
> >  > > > > we call gLBF.
> >  > > > > > >> >  (guaranteed Latency Based Forwarding).
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> >  gLBF, as compared to TCQF and CSQF is proposed from our side
> >  > > to be
> >  > > > > a more long-term
> >  > > > > > >>
> >  > > > > > >> >  solution, because it has not been validated with high-speed
> >  > > > > forwarding hardware and requires
> >  > > > > > >>
> >  > > > > > >> >  new network header information for the damper value, whereas
> >  > > > > TCQF/CSQF of course can operate
> >  > > > > > >>
> >  > > > > > >> >  without new headers, have proven high-speed implementations PoC
> >  > > > > and are therefore really
> >  > > > > > >> >  ready for adoption now.
> >  > > > > > >> >
> >  > > > > > >>
> >  > > > > > >> >  gLBF is a specific variant of the damper idea that is meant to
> >  > > be
> >  > > > > compatible with the
> >  > > > > > >>
> >  > > > > > >> >  TSN-ATS latency calculus so that it can use the same
> >  > > > > controller-plane/path-computation
> >  > > > > > >>
> >  > > > > > >> >  algorithms and implementations one would use for TSN-ATS. It
> >  > > also
> >  > > > > allows to eliminate
> >  > > > > > >>
> >  > > > > > >> >  the need for hop-by-hop clock synchronization and (we hope)
> >  > > should
> >  > > > > be well implementable
> >  > > > > > >> >  in high-speed hardware.
> >  > > > > > >> >
> >  > > > > > >> >  Any feedback welcome.
> >  > > > > > >> >
> >  > > > > > >> >  Cheers
> >  > > > > > >> >      Toerless
> >  > > > > > >> >
> >  > > > > > >> >  In-Reply-To: <
> >  > > > > [e-mail]<mailto:[e-mail]>>
> >  > > > > > >> >
> >  > > > > > >> >  On Fri, Jul 07, 2023 at 07:37:56AM -0700,
> >  > > [e-mail]<mailto:[e-mail]>
> >  > > > > > >>  wrote:
> >  > > > > > >> >  >
> >  > > > > > >> >  > A new version of I-D, draft-eckert-detnet-glbf-01.txt
> >  > > > > > >> >  > has been successfully submitted by Toerless Eckert and
> >  > > posted to
> >  > > > > the
> >  > > > > > >> >  > IETF repository.
> >  > > > > > >> >  >
> >  > > > > > >> >  > Name:         draft-eckert-detnet-glbf
> >  > > > > > >> >  > Revision:     01
> >  > > > > > >>
> >  > > > > > >> >  > Title:                Deterministic Networking (DetNet) Data
> >  > > > > Plane - guaranteed Latency Based Forwarding (gLBF) for bounded latency
> >  > > with
> >  > > > > low jitter and asynchronous forwarding in Deterministic Networks
> >  > > > > > >> >  > Document date:        2023-07-07
> >  > > > > > >> >  > Group:                Individual Submission
> >  > > > > > >> >  > Pages:                39
> >  > > > > > >> >  > URL:
> >  > > > > > >> https://www.ietf.org/archive/id/draft-eckert-detnet-glbf-01.txt
> >  > > > > > >> >  > Status:
> >  > > > > > >> https://datatracker.ietf.org/doc/draft-eckert-detnet-glbf/
> >  > > > > > >> >  > Htmlized:
> >  > > > > > >> https://datatracker.ietf.org/doc/html/draft-eckert-detnet-glbf
> >  > > > > > >> >  > Diff:
> >  > > > > > >>
> >  > > https://author-tools.ietf.org/iddiff?url2=draft-eckert-detnet-glbf-01
> >  > > > > > >> >  >
> >  > > > > > >> >  > Abstract:
> >  > > > > > >> >  >    This memo proposes a mechanism called "guaranteed Latency
> >  > > > > Based
> >  > > > > > >>
> >  > > > > > >> >  >    Forwarding" (gLBF) as part of DetNet for hop-by-hop packet
> >  > > > > forwarding
> >  > > > > > >> >  >    with per-hop deterministically bounded latency and minimal
> >  > > > > jitter.
> >  > > > > > >> >  >
> >  > > > > > >> >  >    gLBF is intended to be useful across a wide range of
> >  > > networks
> >  > > > > and
> >  > > > > > >> >  >    applications with need for high-precision deterministic
> >  > > > > networking
> >  > > > > > >>
> >  > > > > > >> >  >    services, including in-car networks or networks used for
> >  > > > > industrial
> >  > > > > > >> >  >    automation across on factory floors, all the way to
> >  > > ++100Gbps
> >  > > > > > >> >  >    country-wide networks.
> >  > > > > > >> >  >
> >  > > > > > >> >  >    Contrary to other mechanisms, gLBF does not require
> >  > > network
> >  > > > > wide
> >  > > > > > >>
> >  > > > > > >> >  >    clock synchronization, nor does it need to maintain
> >  > > per-flow
> >  > > > > state at
> >  > > > > > >> >  >    network nodes, avoiding drawbacks of other known methods
> >  > > while
> >  > > > > > >> >  >    leveraging their advantages.
> >  > > > > > >> >  >
> >  > > > > > >> >  >    Specifically, gLBF uses the queuing model and calculus of
> >  > > > > Urgency
> >  > > > > > >> >  >    Based Scheduling (UBS, [UBS]), which is used by TSN
> >  > > > > Asynchronous
> >  > > > > > >> >  >    Traffic Shaping [TSN-ATS]. gLBF is intended to be a
> >  > > plug-in
> >  > > > > > >> >  >    replacement for TSN-ATN or as a parallel mechanism beside
> >  > > > > TSN-ATS
> >  > > > > > >>
> >  > > > > > >> >  >    because it allows to keeping the same controller-plane
> >  > > design
> >  > > > > which
> >  > > > > > >> >  >    is selecting paths for TSN-ATS, sizing TSN-ATS queues,
> >  > > > > calculating
> >  > > > > > >> >  >    latencies and admitting flows to calculated paths for
> >  > > > > calculated
> >  > > > > > >> >  >    latencies.
> >  > > > > > >> >  >
> >  > > > > > >>
> >  > > > > > >> >  >    In addition to reducing the jitter compared to TSN-ATS by
> >  > > > > additional
> >  > > > > > >>
> >  > > > > > >> >  >    buffering (dampening) in the network, gLBF also eliminates
> >  > > > > the need
> >  > > > > > >> >  >    for per-flow, per-hop state maintenance required by
> >  > > TSN-ATS.
> >  > > > > This
> >  > > > > > >> >  >    avoids the need to signal per-flow state to every hop
> >  > > from the
> >  > > > > > >> >  >    controller-plane and associated scaling problems.  It also
> >  > > > > reduces
> >  > > > > > >> >  >    implementation cost for high-speed networking hardware
> >  > > due to
> >  > > > > the
> >  > > > > > >>
> >  > > > > > >> >  >    avoidance of additional high-speed speed read/write memory
> >  > > > > access to
> >  > > > > > >> >  >    retrieve, process and update per-flow state variables for
> >  > > a
> >  > > > > large
> >  > > > > > >> >  >    number of flows.
> >  > > > > > >> >  >
> >  > > > > > >>
> >  > > > > > >> >  >
> >  > > > > > >> >  >
> >  > > > > > >> >  >
> >  > > > > > >> >  > The IETF Secretariat
> >  > > > > > >> >
> >  > > > > > >> >  _______________________________________________
> >  > > > > > >> >  detnet mailing list
> >  > > > > > >> >  [e-mail]<mailto:[e-mail]>
> >  > > > > > >> >  https://www.ietf.org/mailman/listinfo/detnet
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >> --
> >  > > > > > >> ---
> >  > > > > > >> [e-mail]<mailto:[e-mail]>
> >  > > > > > >>
> >  > > > > > >> _______________________________________________
> >  > > > > > >> detnet mailing list
> >  > > > > > >> [e-mail]<mailto:[e-mail]>
> >  > > > > > >> https://www.ietf.org/mailman/listinfo/detnet
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >>
> >  > > > > > >
> >  > > > >
> >  > > > > --
> >  > > > > ---
> >  > > > > [e-mail]<mailto:[e-mail]>
> >  > > > >
> >  > >
> >  > > --
> >  > > ---
> >  > > [e-mail]<mailto:[e-mail]>
> >  > >
> >
> >  --
> >  ---
> >  [e-mail]<mailto:[e-mail]>
>
>
>
> --
> ---
> [e-mail]<mailto:[e-mail]>



--
---
[e-mail]<mailto:[e-mail]>