Re: [bmwg] Martin Duke's Discuss on draft-ietf-bmwg-b2b-frame-03: (with DISCUSS and COMMENT)

[Martin Duke <[e-mail]>]

Looks good to me.

On Wed, Dec 16, 2020 at 10:32 AM MORTON, ALFRED C (AL) <[e-mail]>
wrote:

> Hi Scott,
>
> I appreciate your practical insights shared below, as always!
>
> Let me propose some text, expressing both the concerns you and Martin
> raised, and the caution that's in my mind (buffer-bloat sizes change the
> time scale of testing, but the problem we currently face with the "latest
> technology" is a very small buffer <<1 sec and difficulty increasing the
> size through configuration changes = we need to re-run tests often to
> determine implementation success/failure).
>
> The duration of the trial includes three REQUIRED components:
> ...
> 3. At least 2 seconds not overlapping the time to receive the burst (2.),
> to ensure that DUT buffers have depleted. Longer times MUST be used when
> conditions warrant, such as when buffer times >2 seconds are measured or
> when burst sending times are >2 seconds, but care is needed since this time
> component directly increases trial duration and many trials and tests
> comprise a complete benchmarking study.
>
> hope this works,
> Al
>
>
> > -----Original Message-----
> > From: Scott O. Bradner [mailto:[e-mail]]
> > Sent: Wednesday, December 16, 2020 6:44 AM
> > To: MORTON, ALFRED C (AL) <[e-mail]>
> > Cc: Martin Duke <[e-mail]>; The IESG <[e-mail]>;
> > [e-mail]; [e-mail]; [e-mail]
> > Subject: Re: [bmwg] Martin Duke's Discuss on
> draft-ietf-bmwg-b2b-frame-03:
> > (with DISCUSS and COMMENT)
> >
> >
> >
> > > On Dec 15, 2020, at 5:04 PM, MORTON, ALFRED C (AL)
> > <[e-mail]> wrote:
> > >
> > > Hi Scott,
> > >
> > > Please see my replies below, marked [acm], with a couple of questions.
> > > I hope I'm not missing something obvious, so trying to be very clear in
> > all replies!
> > > But I could be overlooking something, and if so I will learn something
> > very soon...
> > >
> > >> -----Original Message-----
> > >> From: Scott O. Bradner [mailto:[e-mail]]
> > >> Sent: Tuesday, December 15, 2020 9:06 AM
> > >> To: MORTON, ALFRED C (AL) <[e-mail]>
> > >> Cc: Martin Duke <[e-mail]>; The IESG <[e-mail]>;
> > >> [e-mail]; [e-mail];
> [e-mail]
> > >> Subject: Re: [bmwg] Martin Duke's Discuss on
> draft-ietf-bmwg-b2b-frame-
> > 03:
> > >> (with DISCUSS and COMMENT)
> > >>
> > >> I basically understood that but it seemed to me that using a fixed (2
> > >> second) extra time, which is unrelated
> > >> to whatever time that the burst might have taken to be sent seemed
> > risky
> > >> since I could
> > >> imagine cases where the play out speed was less than the receive speed
> > >
> > > [acm]
> > > I guess I don't understand your example, where (buffer?) play-out speed
> > plays a role in the results, and how play-out speed could be less than
> the
> > receive speed in the multi-second time scale of the buffer-bloat example.
> > I think (buffer) play-out speed and receive speed should be nominally the
> > same.
> >
> >
> > I expect that generally they would be about the same but its all software
> > and different routines would
> > handle input & output so one can not be sure - in addition the system
> > could be adding keep-alive packets,
> > routing updates etc to the output stream (not that they would take much
> > time to send)
> > >
> > > Although RFC 2544 Throughput definition is based on offered load
> > delivered loss-free to the receiver, we use it here as the best
> > approximation available for packet header processing rate (equal to
> > playout rate from the buffer?), egress from the DUT, and the speed at
> > which the test system receives packets.
> > >
> > > So, in our diagram from the memo:
> > >
> > >                        |------------ DUT --------|
> > >   Generator -> Ingress -> Buffer -> HeaderProc -> Egress -> Receiver
> > >
> > > Is your play-out speed the HeaderProc speed, or Egress speed?
> > >
> > > And how can the (buffer) play-out speed be less than the speed at a
> > subsequent interface (for very long)?
> >
> > "very long" is a relative term :-)
> >
> > I agree that there should not be any issue if 2 seconds is long relative
> > to the burst length but
> > maybe not so if the burst length is long relative to 2 seconds (e.f. it
> > takes a minute or two
> > to fill the buffer)
> >
> > Scott
> > >
> > > help me understand the mechanics I'm overlooking, my friend!
> > >
> > >>
> > >> but if you are convinced that the 2 seconds extra time would cover all
> > >> possible cases then go to it
> > > [acm]
> > >
> > > Well, we say "at least 2 seconds" and allow for customization if
> > necessary.
> > >
> > > As you know, I've conducted LOTS of production network testing, where
> we
> > have used static waiting times to distinguish packet loss from long
> delay,
> > and prescribed the same in IPPM RFCs, etc. A static waiting time "Tmax"
> > has served us well.
> > >
> > > Here, we have the added stability of the Isolated Test Environment
> (ITE,
> > as Kevin Dubray called it), and the three time-component definition of
> > trial duration, where we wait 2 seconds after the last packet on seen
> > egress (it is more like a cool-down interval between trials). I think all
> > the adaptation we need comes from explicit recognition that the time for
> > the Test Receiver to receive the entire burst depends on the buffer size,
> > the DUT header processing rate, the actual interface speed, etc. IOW, all
> > the unknown variables.
> > >
> > > Thanks again for your time, Scott!
> > > Al
> > >
> > >>
> > >> Scott
> > >>
> > >>
> > >>> On Dec 14, 2020, at 7:24 PM, MORTON, ALFRED C (AL)
> > >> <[e-mail]> wrote:
> > >>>
> > >>> Hi Scott, thanks for helping with this discussion.
> > >>>
> > >>> I'm trying to formulate adaptive extra time based on the time it
> takes
> > >> to *receive* the burst, with the additional "at least 2 seconds"
> > waiting
> > >> time to be sure we received all the packets that might arrive.  Let me
> > try
> > >> drawing the timeline that's in my mind, and I'll use a buffer-bloat
> > case
> > >> example of a 1 second buffer (which dominates all other buffers in the
> > >> DUT).
> > >>>
> > >>> One of the key contributions of this memo is recognizing that the
> > buffer
> > >> is being emptied while the burst of back-to-back frames is
> > simultaneously
> > >> trying to fill the buffer.
> > >>>
> > >>> Assume that the RFC 2544 Throughput is only half of the back-to-back
> > >> frame rate for the frame size used.
> > >>>
> > >>> From the draft:
> > >>>  4.  A helpful concept is the buffer filling rate, which is the
> > >>>      difference between the Max Theoretical Frame Rate (ingress) and
> > >>>      the Measured Throughput (HeaderProc on egress).  If the actual
> > >>>      buffer size in frames was known, the time to fill the buffer
> > >>>      during a measurement can be calculated using the filling rate as
> > >>>      a check on measurements.  However, the Buffer in the model
> > >>>      represents many buffers of different sizes in the DUT data path.
> > >>>
> > >>> So (danger: calculating while typing and drawing!), a 1 second burst
> > of
> > >> B2B frames only raises the occupation buffer to 50%, and another
> second
> > of
> > >> transmission is needed before reaching 100% occupation.
> > >>>
> > >>> Trial
> > >>> Time, sec: 0          1          2          3         4          5
> > >> 6
> > >>>
> > >>> Sender:    |==========|==========|
> > >>> Receiver:  |= = = = = |= = = = = |= = = = = |= = = = =|
> > >>> Waiting Time                                          |          |
> > >> |
> > >>>
> > >> Trial
> > >>>
> > >> Ends
> > >>>
> > >>> In the ideal example timeline above, the back-to-back burst stopped
> > >> exactly when the buffer reached capacity, so there is no loss. The
> > buffer
> > >> fill rate is half the back-to-back rate. Also, it takes 2 seconds to
> > >> deplete the buffer and for frames to stop arriving at the receiver.
> > Only
> > >> then do we start the 2 second waiting time to ensure no more frames
> > will
> > >> arrive!
> > >>>
> > >>> While we're here, let's look at a calculation from the memo:
> > >>>
> > >>>  Corrected DUT Buffer Time =
> > >>>                         /                                         \
> > >>>          Implied DUT    |Implied DUT       Measured Throughput    |
> > >>>       =  Buffer Time -  |Buffer Time * -------------------------- |
> > >>>                         |              Max Theoretical Frame Rate |
> > >>>                         \                                         /
> > >>>       =  2 - [ 2 * 0.5 ] seconds
> > >>>       =  1 second
> > >>>
> > >>> and we avoid the error of calculating buffer time based on the
> > sender's
> > >> burst duration alone.
> > >>>
> > >>> hope this helps,
> > >>> Al
> > >>>
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: Scott O. Bradner [mailto:[e-mail]]
> > >>>> Sent: Saturday, December 12, 2020 5:18 PM
> > >>>> To: MORTON, ALFRED C (AL) <[e-mail]>
> > >>>> Cc: Martin Duke <[e-mail]>; The IESG <[e-mail]
> >;
> > >>>> [e-mail]; [e-mail];
> > [e-mail]
> > >>>> Subject: Re: [bmwg] Martin Duke's Discuss on draft-ietf-bmwg-b2b-
> > frame-
> > >> 03:
> > >>>> (with DISCUSS and COMMENT)
> > >>>>
> > >>>> this would seem to work if 2 seconds is significantly longer than it
> > >> takes
> > >>>> to send the burst - but if it takes 2 second to send the burst
> > >>>> then 2 seconds extra buffer could easily lose packets - seems to me
> > >> that
> > >>>> he extra time should be related to the time it takes to send the
> > burst
> > >>>>
> > >>>> e.g 50% of the burst time but not less than 2 seconds
> > >>>>
> > >>>> Scott
> > >>>>
> > >>>>
> > >>>>> On Dec 12, 2020, at 10:18 AM, MORTON, ALFRED C (AL)
> > >>>> <[e-mail]> wrote:
> > >>>>>
> > >>>>> Hi Martin, thanks for your review and comment,
> > >>>>> please see my reply, [acm] below,
> > >>>>> Al
> > >>>>>
> > >>>>>> -----Original Message-----
> > >>>>> ...
> > >>>>>>
> > >>>>>>
> -------------------------------------------------------------------
> > --
> > >> -
> > >>>>>> DISCUSS:
> > >>>>>>
> -------------------------------------------------------------------
> > --
> > >> -
> > >>>>>>
> > >>>>>> Thank you for engaging with the TSVART review. Despite the
> > >> wordsmithing
> > >>>> that
> > >>>>>> has gone on, I am not sure that we have captured the correct text.
> > >>>>>>
> > >>>>>> The proposed change is:
> > >>>>>>> I clarified:
> > >>>>>>> The duration of the trial MUST include at least 2 seconds in
> > >> addition
> > >>>> to the time
> > >>>>>>> required to send and receive each burst of frames, to ensure that
> > >> DUT
> > >>>> buffers to deplete.
> > >>>>>>> and I'll add:
> > >>>>>>> The upper search limit for the time to send each burst MUST be
> > >>>> configurable as
> > >>>>>>> high as 30 seconds (buffer time results
> > >>>>>>> reported at the configured upper limit are likely invalid, and
> the
> > >>>> test MUST
> > >>>>>>> be repeated with a higher search limit).
> > >>>>>>
> > >>>>>> But IIUC it's the additional time that needs to scale up.
> > >>>>> [acm]
> > >>>>>
> > >>>>> In the revised text where David and I reached agreement, we
> > identified
> > >> 3
> > >>>> time components of the trial duration, making the duration variable:
> > no
> > >>>> longer static and at "at least 2 seconds".
> > >>>>>
> > >>>>> 1. the time to send the burst of frames (at the back-to-back rate),
> > >>>> determined by the search algorithm
> > >>>>> 2. the time to receive the transferred burst of frames (at the
> > RFC2544
> > >>>> Throughput rate), possibly truncated by buffer overflow, but
> > certainly
> > >>>> including the latency of the DUT with or without buffer-bloat
> > >>>>> 3. at least 2 seconds in addition to the time to receive the burst
> > >> (2.),
> > >>>> to ensure that DUT buffers have depleted.
> > >>>>>
> > >>>>> So, both components 1. and 2. are variables, and the burst receive
> > >> time
> > >>>> component (2.) compensates for large buffers, non-back-to-back burst
> > >>>> egress, and anything else that contributes to DUT latency. The final
> > >> "at
> > >>>> least 2 seconds" is simply about making sure the trial is really
> over
> > >>>> before moving on in an automated test - we won't make an error if
> > >> frames
> > >>>> trickle-out very late for some unfortunate reason.
> > >>>>>
> > >>>>>> A layman's reading of
> > >>>>>> the document, IMO, suggests that the burst length has a binary
> > search
> > >>>> but the 2
> > >>>>>> seconds of waiting can be fixed.
> > >>>>> [acm]
> > >>>>> Yes, that's right, plus all the other factors above.
> > >>>>>
> > >>>>> So, let's try this, but I'm trying not to extend or complicate the
> > >>>> buffer time << 2 seconds testing for the sake of the buffer-bloat
> > case:
> > >>>>>
> > >>>>> -=-=-=-=-=-=-
> > >>>>>
> > >>>>> The duration of the trial includes three REQUIRED components:
> > >>>>>
> > >>>>> 1. the time to send the burst of frames (at the back-to-back rate),
> > >>>> determined by the search algorithm
> > >>>>> 2. the time to receive the transferred burst of frames (at the
> > RFC2544
> > >>>> Throughput rate), possibly truncated by buffer overflow, and
> > certainly
> > >>>> including the latency of the DUT
> > >>>>> 3. at least 2 seconds not overlapping the time to receive the burst
> > >>>> (2.), to ensure that DUT buffers have depleted.
> > >>>>>
> > >>>>> The upper search limit for the time to send each burst MUST be
> > >>>> configurable as high as 30 seconds (buffer time results reported at
> > or
> > >>>> near the configured upper limit are likely invalid, and the test
> MUST
> > >> be
> > >>>> repeated with a higher search limit).
> > >>>>>
> > >>>>> -=-=-=-=-=-=-=-=-
> > >>>>>
> > >>>>> Does that wording do it?
> > >>>>>
> > >>>>>>
> > >>>>>>
> -------------------------------------------------------------------
> > --
> > >> -
> > >>>>>> COMMENT:
> > >>>>>>
> -------------------------------------------------------------------
> > --
> > >> -
> > >>>>>>
> > >>>>>> Other than that, this a well-written document. Thanks!
> > >>>>> [acm]
> > >>>>> Thank you!
> > >>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> bmwg mailing list
> > >>>>> [e-mail]
> > >>>>>
> > >>>>
> > >>
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__
> ;!
> > >>>>
> > !BhdT!1uRJDJBUadSunB4ZCkgOTzg3ZssPtiufcyrsTcxEc1F67df5q4YNUa9IYHacnsA$
> > >>>
> > >
>
>