Re: [bmwg] Throughput definition discussions - volunteers needed

[[e-mail]]

Thanks for your input Vratko. We are taking the comment rec’d on the call and will be proposing an updated definition.  Stay tuned.

 

Brian

 

From: bmwg <[e-mail]> On Behalf Of Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)
Sent: Wednesday, March 24, 2021 3:42 PM
To: [e-mail]
Subject: Re: [bmwg] Throughput definition discussions - volunteers needed

 

TL;DR:

1. Define "load" (the one quantity test equipment varies between trials within a search).

2. Define validity criterion for trial measurement result.

3. Confirm we are searching for max valid load (not for some medium load optimum).

4. Define which quantity of the best trial ends up in the spreadsheet.

(5. I care about definitions, I do not have strong opinions about names.)

 

The long version:

 

I had no comments during the meeting,

but here is my updated understanding.

 

You can generalize trial [3] in various ways

and get corresponding generalizations of throughput [4]

by selecting best trial that satisfies some condition.

 

We are interested in "continuous traffic" trials,

where trials have configurable duration and "load" as input values.

(Other trial type has configurable "size"

and duration is the result of measurement,

but we are not interested in that trial type yet.)

Of course, test equipment has more inputs than just duration and load,

but we group them under "traffic profile" and keep constant.

 

A direct result of a (continuous traffic) trial is a bunch of counters,

different vendors may use different counters (so we are not standardizing those).

Indirect results are numerous, but two are important.

Validity criterion, and "forwarding rate".

 

For the classic throughput, "load" is the Offered Load (in pps),

validity criterion is zero loss, and forwarding rate is again Offered Load.

 

The general approach for finding "throughput" is to vary the load,

and select the best "forwarding rate" from result that passed validation.

Usually, valid trial at increased load implies increased forwarding rate,

so it is enough to do binary search to find max load that still gives valid result.

Other algorithms (such as MLRsearch) can be used.

I will call this approach "the search".

(An example where increased load can decrease forwarding rate

is Maximum Forwarding Rate [5], as it has no validation criterion

related to packet loss.)

 

I think the generalization for NGFW would look like this.

Traffic profile defines what a "transaction" is.

Test equipment has "transaction rate" as its "load" parameter.

It initiates new transactions with a constant frequency,

regardless of status of previous transactions.

Transaction rate is given in TPS (transactions per second).

Test equipment has counters that can evaluate transaction success percentage (at least after the trial).

Validation criterion is that only 0.0001% of transactions can be other than success.

Test equipment also has some L2 counters.

Forwarding rate is computed from those counters, (probably in bits per second,

not sure about inter frame gap and other details).

The search is then finding the highest transaction rate that still leads to valid trial result.

The "throughut" is then the forwarding rate at highest valid transaction rate.

Even if slightly smaller loads happen to achieve higher forwarding rate

(which does not happen, L2 load usually accelerates more rapidly than transaction rate).

 

Even if all of the above is true, we still need to name two quantities.

One is the "highest load still leading to valid trial results".

(In PLRsearch draft I am calling that a "critical load",

but maybe we can come up with something less generic.)

Another one is the "previously reported as throughput",

an L2-centric quantity measured at the highest valid load.

Ideally, this name would refer to the first name in some way.

Examples: "Critical Throughput" or "Forwarding Rate at Maximal Usable Load (FRaMUL)".

 

Vratko.

 

[3] https://tools.ietf.org/html/rfc2544#section-23

[4] https://tools.ietf.org/html/rfc2544#section-26.1

[5] https://tools.ietf.org/html/rfc2285#section-3.6.3

 

From: bmwg <[e-mail] <mailto:[e-mail]> > On Behalf Of Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)
Sent: Monday, 2021-March-15 15:43
To: [e-mail] <mailto:[e-mail]> 
Subject: Re: [bmwg] Throughput definition discussions - volunteers needed

 

> Let me know if you are interested

 

I am interested.

 

I think we already mentioned (during the meeting)

the need to distinguish allowed vs rejected traffic and packets vs bits.

RFC 2647 also mentions traffic orientation and goodput.

I think (for future proofing) we should think also about situations

when the amount of data going in differs from the amount of data coming out

(e.g encapsulation/decapsulation, encryption/decryption, compression/decompression).

 

> RFC 1242, 3.17 – Throughput

 

That is defined as a load with some additional properties.

This assumes constant load [0] in packets per second.

Stateful traffic often leads to non-constant loads,

affecting different layers differently.

I think we first need to update or replace intended load [2] (and offered load),

and base the "application throughput" on that.

 

Finally, I repeat my earlier comment: Counters examined during the traffic

cannot distinguish an in-flight "transaction" from a failed one.

 

Vratko.

 

[0] https://tools.ietf.org/html/rfc1242#section-3.4

[1] https://tools.ietf.org/html/rfc2285#section-3.5.1

 

From: bmwg <[e-mail] <mailto:[e-mail]> > On Behalf Of [e-mail] <mailto:[e-mail]> 
Sent: Friday, 2021-March-12 17:39
To: [e-mail] <mailto:[e-mail]> 
Subject: [bmwg] Throughput definition discussions - volunteers needed

 

Folks,

 

The NGFW performance draft (https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-06) was reviewed in the virtual BMWG meeting yesterday. One item that proved to be a bit interesting  the discussion was around the definition of “Throughput” in section 6.3. We used what we thought was appropriate wording from RFC 2647, section 3.4 – Bit Forwarding Rate. The concern raised was specifically our use of the word Throughput. There was general consensus from the BMWG that this was confusing and for clarity sake we should use something else as the accepted BMWG definition of Throughput was defined well in RFC 1242, 3.17 – Throughput.  This discussion devolved into a circular one. It was suggested that some folks from NetSecOPEN and some folks from the BMWG get together on a call to come up with proposed wording that would be acceptable to everyone. I am looking for a few people from here to volunteer. Please let me know if you are interested.

 

Maciek Konstantynowicz and Al Morton have already volunteered.  The plan is to setup a call with the BMWG volunteers and a small number from NetSecOPEN to discuss. Our goal is to reach consensus on suitable wording that we can bring back to the group.

 

Let me know if you are interested and I will reach out to discuss timing.

 

Brian

 

 

---------

Brian Monkman

Executive Director, NetSecOPEN

Office: [phone] 

Fax: [phone]

Mobile: [phone]

 



https://www.netsecopen.org