Re: [bmwg] the draft "Benchmarking Methodology for IPv6 Transition Technologies"

[Marius Georgescu <[e-mail]>]

> On Aug 21, 2015, at 15:34, Fred Baker (fred) <[e-mail]> wrote:
> 
>>> I'd stay away from link layer issues. Token Thing is interesting in comparison to Ethernet due to token rotation. Have you ever seen a Token Thing interface? Would you do better to describe the IPv4/IPv6 interaction and leave the rest as an MTU calculation?
>> 
>> ###MG: I am to trying be as link layer agnostic as possible. However, the test data is generated as frames and the fragmentation behavior is one of the important performance aspects in my opinion. Maybe I’m missing the point?
> 
> I'm not sure I see the point of SONET testing. To begin with, in Japan I would expect it's SDH (SONET is one of our weird US standards), but in any event it's not the link layer you're testing, it's the IP layer or maybe two instances of it. I'd stop at Ethernet. But that's just me.
> 

I’ve received a similar opinion from Al. If the WG agrees, I’ll stop at Ethernet.


>>>> 8.2. Benchmarking Performance Degradation
>>>> 
>>>> Objective: To quantify the performance degradation introduced by n
>>>> parallel network flows.
>>>> 
>>>> Procedure: First the benchmarking tests presented in Section 6 have
>>>> to be performed for one network flow.
>>>> 
>>>> The same tests have to be repeated for n network flows. The
>>>> performance degradation of the X benchmarking dimension SHOULD be
>>>> calculated as relative performance change between the 1-flow results
>>>> and the n-flow results, using the following formula:
>>>> 
>>>> 
>>>>            Xn - X1
>>>>     Xpd= ----------- * 100, where: X1 - result for 1-flow
>>>>               X1                    Xn - result for n-flows
>>>> 
>>>> Reporting Format: The performance degradation SHOULD be expressed as
>>>> a percentage. The number of tested parallel flows n MUST be clearly
>>>> specified. For each of the performed benchmarking tests, there
>>>> SHOULD be a table containing a column for each frame size. The table
>>>> SHOULD also state the applied frame rate.
>>> 
>>> OK as far as it goes. You have two fundamental things you need to test, which are the creation of state (linear or episodic) and the use of state (if I have N active sessions, what is the implication for performance?). I suspect that's at least three tests.
>> 
>> ###MG: I am not sure what you mean by creation of state and use of state here.
> 
> Well, it takes a certain amount of work to associate an inside address or tuple with an outside address or tuple, which is what I'm calling "creation of state". The difference between linear growth and episodic growth is that one is a little state creation at a time and the other may be all at once - and may be affected by drops if the queue to the creation process is finite. Once state is created, you simply find it and use it, which also takes work but it's different work.
> 
> I'm just suggesting that you test sustainable state creation separately from episodic state creation, and the impact of translation of throughput/loss as a third test.

I was thinking you are referring only to stateful technologies, but I guess you mean we should have something like:

1 - measure throughput/loss for 1 + 1 +1 … = N network flows started over a period of time T at an interval I -> result S1
2 - measure  throughput/loss for N network flows started simultaneously  -> result S2
3 - calculate separately S1 and S2 considering the result for 1 network flow

Is this what you mean/closer to the mark?

Many thanks for the follow-ups,
Marius