Re: [DLNEX] 答复: Various sources contributing to E2E latency/delay-- Discussion of reliable and deterministic latency attributes

[Bob Briscoe <[e-mail]>]

Ning,

On 07/11/16 08:52, Zongning wrote:
>
> Hi, Bob,
>
> Thanks for pointing out the text for reading. It is an excellent and 
> useful survey indeed. J
>
> While I am still reading the paper, I have quick (maybe naïve) 
> questions as below.
>
> 1)I see you adopted “the amount of data transferred” as one parameter 
> for benchmarking.
>
If you don't know the context, see {Note 1}
>
> Would you think it is also important to consider the “characteristic 
> of the flow”? e.g. some flow may have regular burst or even be totally 
> unexpected.
>

The answer is no. Because, we chose to to use the division: 
(session-completion-time / original-session-completion-time), so the 
characteristic of the flow would be the same in both cases. Only the 
latency reducing technology would be different.

> 2)I am especially interested in DBCC (Delay Based Congestion Control). 
> The paper argues it is difficult to deploy over the Internet due to 
> potential starvation by loss-based flows. Is this one of the 
> motivation of L4S that separates the queues for different (loss or 
> latency sensitive) flows?
>
No. There are two misunderstandings here.

1/ Without the "Coupled" aspect of the AQM that we recommend, L4S 
congestion controls would starve the bandwidth of loss-based flows, not 
the other way round. Nonetheless, perhaps what you were thinking was 
that, without the DualQ aspect of the AQM, loss-based flows would ruin 
the low latency of L4S flows.

2/ L4S is intended to have all traffic together in the L4S queue. The 
second 'Classic' queue is only for transition, to partition off legacy 
TCP variants until they upgrade. You are incorrectly thinking of the two 
queues as one for low latency and one not. That is what DIffserv gave. 
L4S gives you both ultra-low latency and capacity-seeking flows in the 
same queue, so you don't need a "non-low-latency" queue (except for 
transition).
>
> Again, I’d think it is interesting to have L4S (or similar) capability 
> abstracted and exposed to handle potentially unbounded flows, e.g. how 
> many latency bounded flows could be supported in certain network node.
>
We have overloaded L4S with very large numbers of TCP flows and we still 
get ultra-low queuing delay.
Nonetheless, we identified what is effectively a bug in TCP, where it 
becomes unresponsive to congestion when the window is less than two 
segments. We are working on implementing a TCP with this problem fixed, 
which will be a requirement for L4S congestion controls.

Cheers


Bob

{Note 1}

For the benefit of those in cc, this is about the ranking of a selection 
of techniques by the reduction in latency they offer, and the reduction 
in variability of latency. The problem we were trying to solve was to 
find a metric that allowed comparison between techniques across a huge 
range of scenarios.

Quoting:
"We decided on percent reduction in session completion time
= 100% - (session-completion-time / original-session-completion-time)
‘Session’ is a deliberately general concept that can be stretched to 
mean a message, a connection or a set of connections that fulfil a task. 
This allows us to compare techniques that address a range of 
interactions, as long as the content of the session is the same in the 
before and after scenarios (denominator and numerator):

  * a one-way end-to-end event notification (message), e.g. a price update;
  * a two-way exchange (connection) including connection set-up and
    optionally security set-up, e.g. retrieval of a simple Web page;
  * a set of exchanges of information (session), e.g. retrieval of a
    geographical map identifying hardware shops in a locality.

"
[...]
"
Despite the numerous parameters above that could characterize a 
scenario, two parameters in particular strongly affect the outcome in 
nearly all cases:

  * the amount of data transferred (‘session-size’);
  * how far apart the end-points are (or were originally), e.g. WAN, LAN.

It should be sufficient to visualize just two cases for each of these 
two dimensions, leading to the 2x2 matrix of cases shown.
"

In other words, even though our chosen metric was already normalized to 
the amount of data transferred, we could only compare results for 
similar "amounts of data transferred", so we divided the bubble plots in 
Fig 21 between small sessions and large sessions to give two example 
ranges of data points.


> Anyway, this is a good paper for gap analysis and build new ideas upon.
>
> Thanks.
>
> -Ning
>
> *发件人:*DLNEX [mailto:[e-mail]] *代表 *Bob Briscoe
> *发送时间:*2016年10月28日0:40
> *收件人:*Linda Dunbar; Lou Berger; Pat Thaler
> *抄送:*Pascal Thubert (pthubert); [e-mail]; Patrick Wetterwald 
> (pwetterw); Alia Atlas; [e-mail]; [e-mail]
> *主题:*Re: [DLNEX] Various sources contributing to E2E latency/delay-- 
> Discussion of reliable and deterministic latency attributes
>
> Linda,
>
> Probably the most useful part of that paper (other than saving you 
> reading over 300 references) is Chapter IX. It attempts to select some 
> of the most interesting approaches, and to quantify how much of the 
> overall latency (in selected scenarios) they remove. Both as text, and 
> as a visualization of their relative merits (mean and variance) in 
> some nice (deliberately approximate) bubble diagrams, using the colour 
> coding of the categories Linda outlined below.
>
>
>
> Bob
>
> On 20/10/16 21:18, Linda Dunbar wrote:
>
>     Lou,
>
>     Change the subject to make the discussion more focused.
>
>     As of now, the DLNex is to explore what kind of characteristics
>     can be exposed by network, or can be imposed by upper layer, to
>     make the delay/latency of E2E services more predictable or more
>     optimal. The final work may eventually fall into some existing
>     IETF WGs. Here, I am just brainstorming some potential exposures:
>
>     This paper "Reducing Internet Latency: A Survey of Techniques and
>     their Merits" (http://www.bobbriscoe.net/pubs.html#latency_survey
>     ) listed multiple sources that contributing to Latency/delay for
>     E2E services:
>
>     /The sources of delay are classified into five main categories:
>     structural delays, interaction between endpoints, delays along
>     transmission paths, delays related to link capacities, and
>     intra-end-host delays./
>
>     /Structural delays arise from the structure of the network or the
>     communication path that is used. /
>
>     The paper shows that one E2E service can use DNS multiple times.
>     Therefore, where DNS is located is a big contributor to the
>     overall latency. This could be a potential exposure by the network.
>
>     /Delays resulting from the interaction between endpoints include
>     delays due to transport initialization and secure session
>     initialization, as well as delays from recovering lost packets and
>     from message-aggregation techniques/
>
>     Is it another exposure? Is there secure channel used?
>
>     /Delays along transmission paths captures the delays that may be
>     encountered as data travels between a sender and a receiver./
>
>     If some segments along the transmission paths are DetNet enabled
>     or OIF’s FlexEthernet enabled, this definitely should be exposed.
>     Then the upper layer can control/reserve the resource as CCAMP has
>     done for MPLS paths and optical channels.
>
>     /Delays related to link capacities include both delays resulting
>     from sharing limited capacity and delays from protocol
>     inefficiencies that under-utilize capacity and therefore
>     communication takes longer than necessary./
>
>     I assume this is the type of work that CCAMP is to address.
>
>     /Intra-end-host delays are delays that occur internally within
>     host endpoints. This includes delays due to buffering in the
>     transport protocol stack and delays within the operating system./
>
>     This is the end hosts TCP/UDP layer.
>
>     Linda
>
>     -----Original Message-----
>     From: Lou Berger [mailto:[e-mail]]
>     Sent: Thursday, October 20, 2016 2:36 PM
>     To: Linda Dunbar <[e-mail]>
>     <mailto:[e-mail]>; Pat Thaler
>     <[e-mail]> <mailto:[e-mail]>
>     Cc: Pascal Thubert (pthubert) <[e-mail]>
>     <mailto:[e-mail]>; [e-mail]
>     <mailto:[e-mail]>; Patrick Wetterwald (pwetterw)
>     <[e-mail]> <mailto:[e-mail]>; Alia Atlas
>     <[e-mail]> <mailto:[e-mail]>;
>     [e-mail] <mailto:[e-mail]>;
>     [e-mail] <mailto:[e-mail]>
>     Subject: Re: [Detnet] FW: New Non-WG Mailing List: DLNEX --
>     Discussion of reliable and deterministic latency attributes
>
>     Hi Linda,
>
>         Could you (or anyone on DLNEX) elaborate on the type of
>     documents you're thinking of delivering?
>
>     In detnet, we've so far been more focused on applications/use
>     cases and data plane options.  But we're really looking forward to
>     contributions/drafts on the information that is needed to support
>     the control of deterministic flows.  The charter states this as:
>
>         Data flow information model: This work will identify the
>
>         information needed for flow establishment and control and be
>
>         used by reservation protocols and YANG data models. The work
>
>         will be independent from the protocol(s) used to control the
>
>         flows (e.g. YANG+NETCONF/RESTCONF, PCEP or GMPLS).
>
>     Any drafts/contributions on this topic or willingness to help
>     define these would be great.
>
>     If there are sufficiently developed ideas, we could probably even
>     find time to discuss in our Seoul session.  (Or perhaps just
>     informally, if there aren't.)
>
>     Thanks,
>
>     Lou
>
>     On 10/19/2016 7:09 PM, Linda Dunbar wrote:
>
>     >
>
>     > Pat,
>
>     >
>
>     >
>
>     >
>
>     > What you said is so true: “The network can't provide a bounded or
>
>     > deterministic latency to an unbounded flow”.
>
>     >
>
>     >
>
>     >
>
>     > DETNET charter starts with
>
>     >
>
>     > The Deterministic Networking (DetNet) Working Group focuses on
>
>     > deterministic data paths that operate over Layer 2 bridged and Layer 3
>
>     > routed segments, where such paths can provide bounds on latency, loss,
>
>     > and packet delay variation (jitter), and high reliability
>
>     >
>
>     >
>
>     >
>
>     > If I understand it correctly, DETNET focuses on the enabling
>
>     > technologies to ensure deterministic latency path across a set of nodes.
>
>     >
>
>     >
>
>     >
>
>     > E2E low latency services has many issues, such as:
>
>     >
>
>     > ·        There could be “bounded flows” that need low latency services
>
>     > mixed with other traffic in the network.
>
>     >
>
>     > ·        E2E services may have to go through many hops, links with
>
>     > some links not supporting DETNET.
>
>     >
>
>     > ·        there could be various protection/re-route mechanisms which
>
>     > all increase latency.
>
>     >
>
>     > ·        etc
>
>     >
>
>     >
>
>     >
>
>     > Some problems specific to WAN include how to isolate the various
>
>     > service traffic, and provide maximum manageability to upper layers.
>
>     >
>
>     >
>
>     >
>
>     > One company in NANOG asked why not use ICMP to mark the delay along
>
>     > the path to give upper link, ..
>
>     >
>
>     >
>
>     >
>
>     > Linda
>
>     >
>
>     >
>
>     >
>
>     >
>
>     >
>
>     > Linda
>
>     >
>
>     >
>
>     >
>
>     > *From:*Pat Thaler [mailto:[e-mail]]
>
>     > *Sent:* Wednesday, October 19, 2016 5:25 PM
>
>     > *To:* Lou Berger <[e-mail] <mailto:[e-mail]>>
>
>     > *Cc:* Alia Atlas <[e-mail] <mailto:[e-mail]>>; Patrick Wetterwald
>     (pwetterw)
>
>     > <[e-mail] <mailto:[e-mail]>>;
>     [e-mail] <mailto:[e-mail]>; Pascal
>     Thubert
>
>     > (pthubert) <[e-mail] <mailto:[e-mail]>>; [e-mail]
>     <mailto:[e-mail]>; [e-mail] <mailto:[e-mail]>;
>
>     > Linda Dunbar <[e-mail] <mailto:[e-mail]>>
>
>     > *Subject:* Re: [Detnet] FW: New Non-WG Mailing List: DLNEX --
>
>     > Discussion of reliable and deterministic latency attributes
>
>     >
>
>     >
>
>     >
>
>     > We have been having recent discussions about latency such as what type
>
>     > of latency is important.
>
>     >
>
>     >
>
>     >
>
>     > As far as the "feasibility exercise for achieving reliable and
>
>     > deterministic latency through a network element or a segment" that
>
>     > overlaps very much what DetNet and IEEE 802.1 TSN have been doing. I
>
>     > believe we have already shown that it is feasible and are not on
>
>     > specifying how to do that. I'm concerned about the overlap between
>
>     > what that paragraph describes and our work.
>
>     >
>
>     >
>
>     >
>
>     > There could be useful work on communication from the upper layer of
>
>     > the flow characteristics, behavior and requirements. There probably is
>
>     > some gap to fill that would help our work. I'm concerned that the list
>
>     > description focuses only on exposing lower layer characteristics to
>
>     > the upper layer. The network can't provide a bounded or deterministic
>
>     > latency to an unbounded flow - to determine what latency can be
>
>     > provided, one has to know something about the bandwidth utilization of
>
>     > the flow.
>
>     >
>
>     >
>
>     >
>
>     > Regards,
>
>     >
>
>     > Pat
>
>     >
>
>     >
>
>     >
>
>     > On Wed, Oct 19, 2016 at 1:59 PM, Lou Berger <[e-mail] <mailto:[e-mail]>
>
>     > <mailto:[e-mail]>> wrote:
>
>     >
>
>     >     Hi Alia,
>
>     >
>
>     >         Overlap looks likely.  I hope that when it does occur, you send
>
>     >     folks over here to contribute -- additional input / WG participation
>
>     >     would be great!  (For folks not familiar with DetNet, check out our
>
>     >     charter at https://datatracker.ietf.org/wg/detnet/charter/
>
>     >     <https://datatracker.ietf.org/wg/detnet/charter/>)
>
>     >
>
>     >     I expect we'll have some good data plane centric discussions at this
>
>     >     meeting, for those who may be interested (there's even still time to
>
>     >     submit related drafts!)
>
>     >
>
>     >     Cheers,
>
>     >
>
>     >     Lou
>
>     >
>
>     >     DetNet Co-chair
>
>     >
>
>     >     On 10/19/2016 11:46 AM, Alia Atlas wrote:
>
>     >     > Hi Patrick,
>
>     >     >
>
>     >     > The discussion intended on dlnex, a non-WG mailing list, is, I
>
>     >     think,
>
>     >     > broader - spanning the range of communications between upper layer
>
>     >     > applications and the network.  It is not clear yet what work might
>
>     >     > develop.  Be sure that I will be watching for any specific overlap
>
>     >     > with DetNet.
>
>     >     >
>
>     >     > I feel that it is useful to make it easy for groups to have a
>
>     >     list to
>
>     >     > brainstorm about their ideas before work is mature enough to
>
>     >     consider
>
>     >     > for standardization.
>
>     >     >
>
>     >     > Regards,
>
>     >     > Alia
>
>     >     >
>
>     >     > On Wed, Oct 19, 2016 at 8:17 AM, Patrick Wetterwald (pwetterw)
>
>     >     > <[e-mail] <mailto:[e-mail]
>     <mailto:[e-mail]%20%3Cmailto:[e-mail]>>
>
>     >     <mailto:[e-mail] <mailto:[e-mail]
>     <mailto:[e-mail]%20%3Cmailto:[e-mail]>>>> wrote:
>
>     >     >
>
>     >     >     Can someone explain me why this mailing list is created
>
>     >     outside of
>
>     >     >     DetNet?
>
>     >     >
>
>     >     >     This sounds like a subset of Detnet no?
>
>     >     >
>
>     >     >     Thanks,
>
>     >     >
>
>     >     >     Patrick
>
>     >     >
>
>     >     >
>
>     >     >     On 19/10/2016, 07:56, "detnet on behalf of Pascal Thubert
>
>     >     >     (pthubert)" <[e-mail] <mailto:[e-mail]>
>
>     >     <mailto:[e-mail]>
>
>     >     >     <mailto:[e-mail]
>
>     >     <mailto:[e-mail]>> on behalf of [e-mail]
>     <mailto:[e-mail]>
>
>     >     <mailto:[e-mail]>
>
>     >     >     <mailto:[e-mail] <mailto:[e-mail]
>     <mailto:[e-mail]%20%3Cmailto:[e-mail]>>>> wrote:
>
>     >     >
>
>     >     >         From: IETF-Announce
>
>     >     [mailto:[e-mail]
>
>     >     <mailto:[e-mail]>
>
>     >     >     <mailto:[e-mail]
>
>     >     <mailto:[e-mail]>>] On Behalf Of IETF
>
>     >     Secretariat
>
>     >     >         Sent: mardi 18 octobre 2016 22:36
>
>     >     >         To: IETF Announcement List <[e-mail] <mailto:[e-mail]>
>
>     >     <mailto:[e-mail]>
>
>     >     >     <mailto:[e-mail] <mailto:[e-mail]
>     <mailto:[e-mail]%20%3Cmailto:[e-mail]>>>>
>
>     >     >         Cc: [e-mail] <mailto:[e-mail]>
>
>     >     <mailto:[e-mail]> <mailto:[e-mail]
>
>     >     <mailto:[e-mail]>>;
>
>     >     >[e-mail] <mailto:[e-mail]> <mailto:[e-mail]>
>
>     >     <mailto:[e-mail] <mailto:[e-mail]
>     <mailto:[e-mail]%20%3Cmailto:[e-mail]>>>;
>
>     >[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail]>
>
>     >     >     <mailto:[e-mail]
>
>     >     <mailto:[e-mail]>>
>
>     >     >         Subject: New Non-WG Mailing List: DLNEX -- Discussion of
>
>     >     >     reliable and deterministic latency attributes
>
>     >     >
>
>     >     >
>
>     >     >         A new IETF non-working group email list has been created.
>
>     >     >
>
>     >     >         List address:[e-mail] <mailto:address:[e-mail]>
>
>     >     <mailto:[e-mail]> <mailto:[e-mail]
>
>     >     <mailto:[e-mail]>>
>
>     >     >         Archive:
>
>     >     >https://mailarchive.ietf.org/arch/search/?email_list=dlnex
>
>     >
>
>     >     >     <https://mailarchive.ietf.org/arch/search/?email_list=dlnex>
>
>     >     >         To subscribe:
>
>     >https://www.ietf.org/mailman/listinfo/dlnex
>
>     >     <https://www.ietf.org/mailman/listinfo/dlnex>
>
>     >     >     <https://www.ietf.org/mailman/listinfo/dlnex>
>
>     >     >
>
>     >     >         Purpose:
>
>     >     >         DLNEX is to discuss various latency characteristics that can
>
>     >     >     be exposed by network elements or segments and to explore if
>
>     >     there
>
>     >     >     are any latency related attributes that can be utilized by upper
>
>     >     >     layer. For example, could there be latency exposure that upper
>
>     >     >     layer can utilize to plan how to distribute their content to the
>
>     >     >     right edges to achieve optimal user experience? Or something
>
>     >     used
>
>     >     >     by Interactive AR controller to optimize their services? Is
>
>     >     there
>
>     >     >     any value gained by upper layer expressing that they would
>
>     >     rather
>
>     >     >     have fixed latency than losing packets?
>
>     >     >
>
>     >     >         The discussion is to answer questions like: are there any
>
>     >     >     effective interaction/coordination between upper layer and lower
>
>     >     >     layer to achieve more efficient optimization for latency
>
>     >     sensitive
>
>     >     >     services?
>
>     >     >
>
>     >     >         This discussion group is also a place to showcase the
>
>     >     state of
>
>     >     >     the arts in latency optimized switching architecture and
>
>     >     interface
>
>     >     >     designs, as the feasibility exercise for achieving reliable and
>
>     >     >     deterministic latency through a network element or a segment.
>
>     >     >     Those latency exposures are the foundation for (future) latency
>
>     >     >     optimized control plane.
>
>     >     >
>
>     >     >
>
>     >     >         For additional information, please contact the list
>
>     >     >     administrators.
>
>     >     >
>
>     >     >_______________________________________________
>
>     >     >         detnet mailing list
>
>     >
>
>     >     >[e-mail] <mailto:[e-mail]> <mailto:[e-mail]>
>
>     >     <mailto:[e-mail] <mailto:[e-mail]
>     <mailto:[e-mail]%20%3Cmailto:[e-mail]>>>
>
>     >     >https://www.ietf.org/mailman/listinfo/detnet
>
>     >     >     <https://www.ietf.org/mailman/listinfo/detnet>
>
>     >     >
>
>     >     >
>
>     >     >_______________________________________________
>
>     >     >     detnet mailing list
>
>     >     >[e-mail] <mailto:[e-mail]> <mailto:[e-mail]>
>
>     >     <mailto:[e-mail] <mailto:[e-mail]
>     <mailto:[e-mail]%20%3Cmailto:[e-mail]>>>
>
>     >     >https://www.ietf.org/mailman/listinfo/detnet
>
>     >
>
>     >     >     <https://www.ietf.org/mailman/listinfo/detnet>
>
>     >     >
>
>     >     >
>
>     >     >
>
>     >     >
>
>     >     > _______________________________________________
>
>     >     > detnet mailing list
>
>     >     > [e-mail] <mailto:[e-mail]> <mailto:[e-mail]>
>
>     >     > https://www.ietf.org/mailman/listinfo/detnet
>
>     >
>
>     >_______________________________________________
>
>     >     detnet mailing list
>
>     >[e-mail] <mailto:[e-mail]> <mailto:[e-mail]>
>
>     >https://www.ietf.org/mailman/listinfo/detnet
>
>     >
>
>     >
>
>     >
>
>     >
>
>     >
>
>     > _______________________________________________
>
>     > detnet mailing list
>
>     > [e-mail] <mailto:[e-mail]>
>
>     > https://www.ietf.org/mailman/listinfo/detnet
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoe http://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/