Re: [bmwg] the draft "Benchmarking Methodology for IPv6 Transition Technologies"

["Fred Baker (fred)" <[e-mail]>]

On Aug 20, 2015, at 11:19 PM, Marius Georgescu <[e-mail]> wrote:
>>>  The performance of Dual-stack transition technologies can be fully
>>>  evaluated using the benchmarking methodologies presented by
>>>  [RFC2544] and [RFC5180]. Consequently, this document focuses on the
>>>  other 3 categories: Single-stack, Encapsulation-based, and
>>>  Translation-based transition technologies.
>> 
>> I think I would focus on translation and encapsulation. The single-stack case, if I understand it, is equivalent to dual stack within the network (the hosts can all communicate directly), and for other hosts, it comes down to translation.
> 
> ###MG: Considering the previous comment as well, I was thinking a more comprehensive generic classification could be:
> 
> - dual stack: dual IP operations as defined in RFC4213
> - single translation IPv6 transition technologies: same IPvX in the network and single translation at the edge
> - double translation: double translation between A -> C (both using IPvX) with B running IPvY
> - encapsulation: encapsulation between A -> C (IPvX) over B (IPvY)
> 
> With the scope on the last 3. What do you think?

That makes sense.

>>>  For the test setups presented in this memo dynamic routing SHOULD be
>>>  employed. However, the presence of routing and management frames can
>>>  represent unwanted background data that can affect the benchmarking
>>>  result. To that end, the procedures defined in [RFC2544] (Sections
>>>  11.2 and 11.3) related to routing and management frames SHOULD be
>>>  used here as well. Moreover, the "Trial description" recommendations
>>>  presented in [RFC2544] (Section 23) are valid for this memo as well.
>> 
>> Dumb question... why require dynamic routing? Yes, it's interesting (I'm a routing guy), but in the kinds of scenarios you mention, you will have routing in two and perhaps three different domains (A, B, and C in my parlance above), and I'll bet that once you have verified that it works, it doesn't change much.
> 
> ###MG: The 1st reason for having dynamic routing is less operational complexity for the testing team. The 2nd would be Scalability testing. You need a good deal of configuration, in my experience. This could easily be solved by the dynamic routing/filtering the routing traffic approach.

I'm not objecting, mind you. I just think you'll wind up doing as much configuration one way as the other - turning on RIP/OSPF/IS-IS or installing a static route.

>> I'd stay away from link layer issues. Token Thing is interesting in comparison to Ethernet due to token rotation. Have you ever seen a Token Thing interface? Would you do better to describe the IPv4/IPv6 interaction and leave the rest as an MTU calculation?
> 
> ###MG: I am to trying be as link layer agnostic as possible. However, the test data is generated as frames and the fragmentation behavior is one of the important performance aspects in my opinion. Maybe I’m missing the point?

I'm not sure I see the point of SONET testing. To begin with, in Japan I would expect it's SDH (SONET is one of our weird US standards), but in any event it's not the link layer you're testing, it's the IP layer or maybe two instances of it. I'd stop at Ethernet. But that's just me.

>>> 8.2. Benchmarking Performance Degradation
>>> 
>>>  Objective: To quantify the performance degradation introduced by n
>>>  parallel network flows.
>>> 
>>>  Procedure: First the benchmarking tests presented in Section 6 have
>>>  to be performed for one network flow.
>>> 
>>>  The same tests have to be repeated for n network flows. The
>>>  performance degradation of the X benchmarking dimension SHOULD be
>>>  calculated as relative performance change between the 1-flow results
>>>  and the n-flow results, using the following formula:
>>> 
>>> 
>>>             Xn - X1
>>>      Xpd= ----------- * 100, where: X1 - result for 1-flow
>>>                X1                    Xn - result for n-flows
>>> 
>>>  Reporting Format: The performance degradation SHOULD be expressed as
>>>  a percentage. The number of tested parallel flows n MUST be clearly
>>>  specified. For each of the performed benchmarking tests, there
>>>  SHOULD be a table containing a column for each frame size. The table
>>>  SHOULD also state the applied frame rate.
>> 
>> OK as far as it goes. You have two fundamental things you need to test, which are the creation of state (linear or episodic) and the use of state (if I have N active sessions, what is the implication for performance?). I suspect that's at least three tests.
> 
> ###MG: I am not sure what you mean by creation of state and use of state here.

Well, it takes a certain amount of work to associate an inside address or tuple with an outside address or tuple, which is what I'm calling "creation of state". The difference between linear growth and episodic growth is that one is a little state creation at a time and the other may be all at once - and may be affected by drops if the queue to the creation process is finite. Once state is created, you simply find it and use it, which also takes work but it's different work.

I'm just suggesting that you test sustainable state creation separately from episodic state creation, and the impact of translation of throughput/loss as a third test.