Re: [bmwg] My review of draft-ietf-bmwg-mlrsearch-01

["Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)" <[e-mail]>]

Dear Gábor,

thank you very much for your comments.
Sorry for the late reply. I have been busier than usual
(and I still am, except for today).

> I wonder, if there is some more formal method that could help.

Personally, I try to track different classes of algorithms,
some more specific than others (e.g. different subclasses).
Imagine "bisection using arithmetic average of load bounds" being a subclass of
"bisection using some deterministic function of load bounds",
or "MLRsearch with RFC2544-compatible parameters" being a subclass of
"MLRsearch with all parameters customizable",
but different than its subclass "MLRsearch with CSIT parameters".

But that is too many possible classes, I am not sure
which ones are important enough to define in the draft.
And there is still a question how to properly define
a class of algorithms in an English text,
without the result being too vague.

(Not likely to be improved in the next draft version.)

> Issue #1 -- The measurement result depends on the performance of the Tester

> I believe that final result of benchmarking measurements
> SHOULD depend only on the performance of the DUT/SUT
> and it SHOULD NOT depend on the performance of the Tester.

This is a hard issue.
The algorithm as described has no way of knowing
which part of the whole system is limiting the performance.

It could be DUT only (no problem, testing DUT as expected),
it could be TG only (can be mitigated by TG self-test
and using small enough loads).

But it could also be an interaction between DUT and TG.
Imagine a TG (the Traffic Analyzer part) which is only able
to handle incoming traffic up to some rate,
but passes the self-test as the Generator part has maximal rate
not larger than that. But what if DUT turns that steady rate
into long-enough bursts of a higher rate (with delays between bursts
large enough, so average forwarding rate matches the load).
This way TA will see some packets as missing, even though DUT
has processed them correctly. And the loss ratio may depend
on trial duration (if TA has some buffers before its bottleneck).

Not sure if I described the scenario well enough,
but my point is that even with TG self-tests,
the MLRsearch results may not describe DUT only.

I will probably state that detection and mitigation
is important, but outside of the scope.
(Next draft version may or may not fix this.)

> https://datatracker.ietf.org/doc/html/rfc8219#section-9.2.1

On the first read it looks too specific,
but it may be used as and example for TG bottleneck mitigation.
I will read the whole RFC and decide
(but maybe not in the very next version of this draft).

> Issue # 2 -- well behaving and bad behaving systems

> I am not sure, but perhaps it would be worth mentioning
> some potential root causes of the "bad behavior" of systems.

I have seen two types of "bad behavior".
In most cases, it is hard to prove the real cause,
but it is easy to form a hypothesis of what would cause this behavior.

One possible cause is an infrequent short pause in DUT processing.
(Imagine a Java application doing a "stop the world" garbage collection.)
The behavior is bad (inconsistent), because some trial measurements
get lucky and avoid the pause, and the probability and/or severity
of the pause increases with trial duration.

Another possible cause is a resource leak of some kind.
After a period of good performance, the performance starts declining
and the decline accelerates until DUT stops working entirely.

> Such a thing(s) can be the buffering of the SUT

Possible, but the DUTs we test in CSIT have all small enough buffers
(thousands of packets, not millions).

> Perhaps the issue could be mitigated

Mitigations would probably depend on the type of behavior,
and it may be hard to detect properly which type of behavior is happening.

I will think about it, but I guess I will just make sure the draft states
anything related to bad behavior detection, causes and mitigation
is important, but out of scope.

(Next draft version may or may not fix this.)

> We have described this issue in the case of DNS64 testing in this paper

Looks interesting on a first glance.
I need more time to read it properly,
maybe the next draft version will be improved based on that.

> Issue #3
> I wonder about the correctness of the following constants

Some of the values are caused by hardware/firmware/software restrictions,
mainly on the Traffic Generator side.

In other cases, we intentionally deviate (in fd.io CSIT)
from RFC2544 requirements, to save more time than otherwise possible.

I will think about how to make that clearer.
(Next draft version may or may not fix this.)

> <title abbrev="MLRsearch"> Multiple Loss Ratio Search for Packet Throughput (MLRsearch)</title>
> imput --> input
> tighterst --> tightest
> bandwith --> bandwidth (appears twice)
> In practice two rates --> In practice, two rates
> In general MLRsearch --> In general, MLRsearch
> overal --> overall
> byt --> ???
> overriden --> overridden
> offes --> offers
> start --> starts
> immediatelly --> immediately
> purposes.Any --> purposes. Any

All of the above will be fixed in next draft version.

> It may be correct, I just wonder, who is the author, if both authors are (only) editors. :-)

Only? The "editor" is a role for the "author".
How do I search for the document that clarifies the roles?
(Next draft version may or may not fix this.)

Vratko.

From: bmwg <[e-mail]> On Behalf Of Gabor LENCSE
Sent: Monday, 2021-October-04 22:34
To: [e-mail]
Subject: [bmwg] My review of draft-ietf-bmwg-mlrsearch-01


Dear Authors and BMWG members,

I have reviewed https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-mlrsearch-01

General comments
-------------------------

- I have found only one significant issue, and the majority of my findings are only nits (usually typos).

- I think it was a good idea to give first a general overview of the algorithm, then present the details (in a simplified way), then give some more details, and finally include an example with numbers. I humbly admit that I still had parts that I did not understand for the first reading. I wonder, if there is some more formal method that could help. (I am sorry, I cannot propose one.)



Discuss:
----------

Issue #1 -- The measurement result depends of the performance of the Tester

Page 17:

          +  The final bounds describe the performance of not just SUT,

             but of the whole system, including the traffic generator.

I believe that final result of benchmarking measurements SHOULD depend only on the performance of the DUT/SUT and it SHOULD NOT depend on the performance of the Tester.

I am aware that this situation is the consequence of the condition described on page 16:

   5.  Duration stretching.



       *  In some cases, traffic generator may get overloaded, causing

          it to take significantly longer (than duration) to send all

          packets.

Yes, it may happen, especially, if we use software based traffic generators and the performance of the hardware is not enough for a given rate. However, it can be (and I think it should be) handled before the measurements. A possible solution can be to use a self-test of the Tester. E.g. if one would like to use the Tester let us say up to 5 million frames per second rate, then first the Tester should be looped back and be tested at 5.5M fps rate, if it can surely transmit and receive the frames within the required duration. (The 10% is the performance reserve. It is a high handed choice, I usually use this value.)

We recommended this solution for DNS64 benchmarking measurement in RFC 8219: https://datatracker.ietf.org/doc/html/rfc8219#section-9.2.1

All further things are much less significant, they are rather questions.



Issue # 2 -- well behaving and bad behaving systems

As far as I understood, there are some well behaving systems: their throughput is nearly the same with different trial duration lengths. Whereas this property is highly utilized to reduce the overall execution time, the correctness of the results is guaranteed by using the proper trial duration length at the final measurement series.

I am not sure, but perhaps it would be worth mentioning some potential root causes of the "bad behavior" of systems. Such a thing(s) can be the buffering of the SUT and the 2s timeout Required by Section 23 of  RFC 2544:

   d) Wait for two seconds for any residual frames to be received.

If the trial is only 1s long and the SUT can forward packets and 1Mfps, but it can store 2M frames in its buffers, then its throughput may be (wrongfully) measured as 3Mfps.

(We have described this issue in the case of DNS64 testing in this paper: http://www.hit.bme.hu/~lencse/publications/ECC-2017-B-M-DNS64-revised.pdf See page 8 and search for the word "gaming".)

Perhaps the issue could be mitigated by significantly decreasing the timeout from 2s for the measurements using shorter test duration than the final one.



Issue #3

I wonder about the correctness of the following constants:

page 17

5.1.1.  FD.io CSIT Input Parameters

   1.  *max_rate* - Typical values: 2 * 14.88 Mpps for 64B 10GE link
       rate, 2 * 18.75 Mpps for 64B 40GE NIC (specific model).

Using 64-byte frames, 14.88Mpps is correct for 10GE, but 18.75 Mpps for 40GE sounds strange to me.


   2.  *min_rate* - Value: 2 * 9001 pps (we reserve 9000 pps for latency
       measurements).

I do not understand the comment in parenthesis. AFAIK, latency measurements are to be performed at the rate determined by the throughput measurement.


   3.  *final_trial_duration* - Value: 30.0 seconds.

RFC 2455 REQUIRES 60.0 seconds.

   4.  *initial_trial_duration* - Value: 1.0 second.

And the timeout is still 2 seconds as required by RFC 2455? Then one trial will still last at least 1+2=3 seconds.



Nits (usually typos)
-------------------------

page 2: (as well as on all the following pages)

Internet-DraMultiple Loss Ratio Search for Packet Throughput   July 2021

The title of the draft is too long to be displayed correctly in the page header. In this case, you can specify an abbreviated title in the XML file. It may look like this:

    <title abbrev="MLRsearch"> Multiple Loss Ratio Search for Packet Throughput (MLRsearch)</title>



page 3:

imput --> input



page 4:

tighterst --> tightest

bandwith --> bandwidth (appears twice)



page 5:

A comma is missing:

In practice two rates --> In practice, two rates



page 8:

A comma is missing:

In general MLRsearch --> In general, MLRsearch



page 13:

overal --> overall



page 16

byt --> ???

overriden --> overridden


page 17:

offes --> offers

start --> starts


page 18:

immediatelly --> immediately



page 20:

purposes.Any --> purposes. Any



page 21:

   Maciek Konstantynowicz (editor)



   Vratko Polak (editor)



It may be correct, I just wonder, who is the author, if both authors are (only) editors. :-)



That's all.

Best regards,

Gábor