Re: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt

[Lencse Gábor <[e-mail]>]

Dear Al,

Thank you very much for your reply. Please see my answers inline. (I 
keep only those parts of the text that I reply to.)

2020.05.23. 19:40 keltezéssel, MORTON, ALFRED C (AL) írta:
>
>> As for the non-overlapping areas of our draft and the documents you
>> cited, I have not found anything in them about our suggestion for
>> "Improved Throughput and Frame Loss Rate Measurement Procedures using
>> Individual Frame Timeout".
>>
>> If you think this one joins well into your efforts to update RFC 2544,
>> then we could focus on this one first, and deal with some others later
>> one by one in (in different documents).
>>
>> What do you think of it?
> [acm]
> Using a constant frame timeout for declaration of Loss (and delay) is
> much like the IPPM WG metrics and methods (see RFC 7679 and RFC 7680)
> for the production network measurements (Tmax). A constant waiting time for
> frames to arrive at the receiver simply excludes frames on an on-going
> basis. RFC 2544 has a waiting time at the end of the trial, where the
> tester must wait for 2 seconds for buffers to clear but the first frame
> sent has the entire trial duration+2sec to arrive.
>

Yes, this is why I feel that some of the frames we count as "received" 
may be completely useless for the applications.

> Perhaps the biggest potential change to the Throughput definition is
> whether or not we demand frame correspondence between sender and receiver,
> so that we can calculate one-way delay. RFC 2544 and ETSI NFV TST009 only
> require equal send and receive frame counts to satisfy the zero loss
> criteria in the Throughput benchmark definition. But in ETSI NFV TST009,
> the Capacity at X% Loss Ratio metric-variant definition begins to infer
> frame correspondence. Later, the definitions of (one-way) Latency and
> Delay Variation and Loss allow a Sample Filter, which could be a constant
> maximum time-out for individual frames, as you suggest. Note that a Sample
> Filter could be applied in post-measurement processing, assuming all the
> delays are available.

I have implemented siitperf-pdv (part of siitpef, available from: 
https://github.com/lencsegabor/siitperf ) in a way that depending on the 
value of its last parameter (*frame timeout*), it can do the 
post-processing in two different ways:
- If the value of *frame timeout* is 0, then proper PDV measurement is 
done.
- If the value of *frame timeout* is higher than zero, then rather a 
special throughput (or frame loss rate) measurement is performed, where 
the tester checks *frame timeout* for each frame individually: if the 
measured delay of a frame is longer than the timeout, then the frame is 
reclassified as lost.

> But the real question in my mind now, after looking into the possibilities
> to take frame correspondence with a fixed timeout into account, is whether
> we can find some examples where adding timeout criteria makes a significant
> difference for Throughput measurements that would not be accounted for by
> revised/modernized Latency and new Delay Variation Benchmarks and
> metric variants?

With other words (as my co-author, Keiichi Shima expressed):

"Can defining a per-frame timeout test provide more sense than the delay 
variation test?"

IMHO, we can answer "yes".

Let us consider the following example. A delay sensitive application can 
tolerate at most 100ms delay and 0.01% frame loss. (The "lost" frames 
include also frames with higher than 100ms delay.)

Section 7.3.1. of RFC 8219 ( 
https://tools.ietf.org/html/rfc8219#section-7.3.1) defines PDV as follows:

    PDV = D99.9thPercentile - Dmin

    Where:

    o  D99.9thPercentile = the 99.9th percentile (as described in
       [RFC5481  <https://tools.ietf.org/html/rfc5481>]) of the one-way delay for the stream

    o  Dmin = the minimum one-way delay in the stream


There can be two problems with using PDV:
- As  PDV uses 99.9th percentile, and even if we measure PDV as 99ms, it 
gives a guarantee for 99.9% of the frames and 0.1% of them may have 
higher than 100ms latency, whereas our system tolerates only 0.01% frame 
loss (including frames with higher latency than 100ms)
- Dmin is subtracted from the 99.9th percentile, thus the final result 
of the PDV measurement is not exactly, what we need.

IMHO, our suggested method can provide a better solution:
- The *frame timeout* parameter of siitperf-pdv should be set to 100ms
- The bash shell script that performs the binary search (and executes 
siitperf-pdv in every single step) should allow 0.01% frame loss.
Thus the measurement can be easily performed.

What do you think of it?

Best regards,

Gábor