Re: [bmwg] the draft "Benchmarking Methodology for IPv6 Transition Technologies"

[Marius Georgescu <[e-mail]>]

Dear Fred,

Thank you very much for taking the time to give the draft a detailed review. I will  reflect on all the feedback and will do my best to address it in the next version of the draft. For now, here are some inline replies to make sure I understand what you’ve meant. The replies are marked with ###MG.

> On Aug 21, 2015, at 10:30, Fred Baker (fred) <[e-mail]> wrote:
> 
> 
> On Aug 20, 2015, at 1:05 AM, Marius Georgescu <[e-mail]> wrote:
>> I hope my e-mail finds you well. It was very nice meeting you in IETF93. We had a very short discussion about benchmarking in the context of IPv6 transition technologies. I have been working on a draft associated with BMWG on this topic for a while now (3rd iteration) and some feedback on it would be more than welcome. If time allows you, please let me know what you think about it. Here’s a link:
>> https://tools.ietf.org/html/draft-georgescu-bmwg-ipv6-tran-tech-benchmarking-01
> 
> OK. I have now had a chance to read this. I'm copying the working group, as it may be interested in the comments. Probably to tell me I'm wrong :-)
> 
> My question walking in was how this specifically differed from a benchmark of, say, an Ethernet Switch or a Ethernet-to-Ethernet Router, or maybe a firewall as the DUT. One might expect the results to be a little different, and if they are a lot different (wide variation in delay, dramatically reduced rate) that could be an important observation. There is also an interesting question on data sources and sinks, as they presumably have different addresses or address families than they might otherwise. But from a step back, I would expect the procedure to be very similar.
> 
> The good news, as I understand the draft, is that they are. In sections 6 and 7, you point to the same procedures one would use in a test of a standard intermediate DUT. Section 8 needs some work.
> When you say "transition technologies", I presume that you're addressing
> - RFC 4213's dual stack model,
> - RFC 6052/6144-6147/6791 IPv4/IPv6 translation,
> - RFC 6296 NPTv6,
> - RFC 6333 DS-Lite,
> - RFC 6740-6748 ILNP,
> - RFC 6877 464XLAT,
> - RFC 7597-7599 MAP (encapsulated and translated),
> - SIIT-DC
>   https://datatracker.ietf.org/doc/draft-ietf-v6ops-siit-dc
>   https://datatracker.ietf.org/doc/draft-ietf-v6ops-siit-dc-2xlat
>   https://datatracker.ietf.org/doc/draft-ietf-v6ops-siit-eam
> 
> and potentially others. One could get into NFV discussions relating to the use of "floating IP addresses" and their implied translators in OpenStack, and the use of load balancers of various kinds in service offerings.
> 

###MG: One could potentially use the specifications of this document (in a future more refined version) and use it for NFV transition techniques, of course in conjuncture  with NFV benchmarking specifications (currently in development in BMWG), but I would rather leave that scope for a different document.


> Thoughts...
> 
>> Abstract
>> 
>>   There are benchmarking methodologies addressing the performance of
>>   network interconnect devices that are IPv4- or IPv6-capable, but the
>>   IPv6 transition technologies are outside of their scope. This
>>   document provides complementary guidelines for evaluating the
>>   performance of IPv6 transition technologies.
> 
> To my mind, this is not obvious. I may not be understanding the scope you have in mind. RFC 2544, as you say, is mostly IP version agnostic; one could use the same procedures to benchmark an Ethernet switch, modulo the appendix on frame format (which specifies TCP and IPv4). The primary difference in consideration for RFC 5180 is the fact that it is IPv6 traffic, and the authors were concerned about the presence or absence of (sets of) extension headers.
> 
> What "scope" do you have in mind? Does it cover data content (when I send an IPvx message with components {Y}, I should see an IPvz message with components {Y'}), and are you complaining that the messages in question are not specified? If not that (and I don't see that in your draft), what's the issue?
> 

###MG: I should definitely make it clearer in the abstract. With this memo I am trying to address the overhead created by the mechanisms within the transition technologies (encapsulation, translation) which affect performance.


>> 1. Introduction
>>   ...
> 
>>   IPv6 is not backwards compatible, which means that IPv4-only nodes
>>   cannot directly communicate with IPv6-only nodes. To solve this
>>   issue, IPv6 transition technologies have been proposed and
>>   implemented, many of which are still in development.
> 
> I might quibble with that last sentence. Looking at the list of technologies I mentioned, all of them have RFC numbers apart from SIIT-DC (which is in WGLC), and almost all of them (including SIIT-DC but excluding ILNP) have commercial implementations deployed, or that have been deployed and since replaced with native deployment, in networks such as T-Mobile, China Mobile, Redpill-linpro, Facebook, Free, and others. The one that can be described as "in development" from an IETF perspective, SIIT-DC, was tested using three different translation implementations, two commercial and one open source, and for the most part documents a service using a feature they all implemented that wasn't actually in the NAT64 specification.
> 

###MG: I think this is an English problem. I should separate the technologies and implementations in that sentence. Most of the Technologies may very well be established. However, I think many of the Implementations are still being developed. This memo could also help developers get feedback on how their implementation’s performs/scales.

>> 1.1. IPv6 Transition Technologies
>>   ...
>>   Tentatively, we can consider a basic production IP-based network as
>>   being constructed using the following components:
>> 
>>   o  a Customer Edge (CE) segment
>> 
>>   o  a Core network segment
>> 
>>   o  a Provider Edge (PE) segment
> 
> A side note, more a quibble than anything, I'm not sure I would focus on customers and providers here. Yes, they are common, but every network isn't focused on them. My corporate network, to pick one at random, is far more concerned about campuses within the network, and treats the Great Unwashed as pretty much that. I think I would focus on the fact that the network contains version-specific domains and dual stack domains, and we're looking at the transitions between them. As you say, if the entire network runs IPvx, it really doesn't matter if it runs IPvy. Call the above A, B, and C; if A and C each use one stack, but different ones, either you will translate between A and B or B and C, or you will tunnel from a host in {A|C} to a host in {C|A}, and the tunnel will transit B and possibly parts of A or C. The "double translation" methodologies treat the translation points as endpoints to a tunnel. To me, the question of whether you pay for a service is beside the point in that context.

###MG: I think I concentrated on the wrong thing here. Like you say, the Great Unwashed should be treated as such. Ultimately, I was aiming at the generic classification of the transition technologies and the associated basic test setups.  

> 
>>   The performance of Dual-stack transition technologies can be fully
>>   evaluated using the benchmarking methodologies presented by
>>   [RFC2544] and [RFC5180]. Consequently, this document focuses on the
>>   other 3 categories: Single-stack, Encapsulation-based, and
>>   Translation-based transition technologies.
> 
> I think I would focus on translation and encapsulation. The single-stack case, if I understand it, is equivalent to dual stack within the network (the hosts can all communicate directly), and for other hosts, it comes down to translation.

###MG: Considering the previous comment as well, I was thinking a more comprehensive generic classification could be:

- dual stack: dual IP operations as defined in RFC4213
- single translation IPv6 transition technologies: same IPvX in the network and single translation at the edge
- double translation: double translation between A -> C (both using IPvX) with B running IPvY
- encapsulation: encapsulation between A -> C (IPvX) over B (IPvY)

With the scope on the last 3. What do you think?
 
> 
>>   Hence, for the stateful IPv6 transition technologies additional
>>   benchmarking tests are RECOMMENDED.
> 
> I don't know that it is actually different tests, but rather the same tests (packets going through the DUT with magic happening under the hood) with an expectation that there will be service issues (the translation can't happen until the state has been created).

###MG: The test are different considering the testa data(for Section 6 tests UDP, for Section 7 tests TCP) and the test metrics. You could be running the tests in Section 6 for the overall performance of the DUT, but if you want just the performance of the state table, the tests in Section 7 are needed.  

> 
>> 3. Test Setup
>> 
>>   The test environment setup options recommended for IPv6 transition
>>   technologies benchmarking are very similar to the ones presented in
>>   Section 6 of [RFC2544]. In the case of the tester setup, the options
>>   presented in [RFC2544] can be applied here as well. However, the
>>   Device under test (DUT) setup options should be explained in the
>>   context of the 3 targeted categories of IPv6 transition
> 
> Why not mention 5180 here? It in turn refers to 2544, but focuses on IPv6 packets and extension headers, which will be important in this context.
> 

###MG: I agree. Will make it so.

> BTW, 6145 specifies the translations of IPv6 extension teachers and IPv4 options. It mostly says "they don't map very well", and sees them disappear. However, fragmentation and IPsec/Security headers get translated, and there are specifics in that. If it was important for 5180 to talk about extension headers, I would at least ask about them in this case.
> 
>>   For the test setups presented in this memo dynamic routing SHOULD be
>>   employed. However, the presence of routing and management frames can
>>   represent unwanted background data that can affect the benchmarking
>>   result. To that end, the procedures defined in [RFC2544] (Sections
>>   11.2 and 11.3) related to routing and management frames SHOULD be
>>   used here as well. Moreover, the "Trial description" recommendations
>>   presented in [RFC2544] (Section 23) are valid for this memo as well.
> 
> Dumb question... why require dynamic routing? Yes, it's interesting (I'm a routing guy), but in the kinds of scenarios you mention, you will have routing in two and perhaps three different domains (A, B, and C in my parlance above), and I'll bet that once you have verified that it works, it doesn't change much.

###MG: The 1st reason for having dynamic routing is less operational complexity for the testing team. The 2nd would be Scalability testing. You need a good deal of configuration, in my experience. This could easily be solved by the dynamic routing/filtering the routing traffic approach. 

> 
>> 3.1. Single-stack Transition Technologies
>> 
>>   For the evaluation of Single-stack transition technologies a single
>>   DUT setup (see Figure 1) SHOULD be used. The DUT is responsible for
>>   translating the IPvX packets into IPvY packets. In this context, the
>>   tester device should be configured to support both IPvX and IPvY.
>> 
>>                           +--------------------+
>>                           |                    |
>>              +------------|IPvX   tester   IPvY|<-------------+
>>              |            |                    |              |
>>              |            +--------------------+              |
>>              |                                                |
>>              |            +--------------------+              |
>>              |            |                    |              |
>>              +----------->|IPvX     DUT    IPvY|--------------+
>>                           |     (translator)   |
>>                           +--------------------+
>>                           Figure 1. Test setup 1
> 
> Quibble point. I think the above tests a single-translation technology - for example RFC 6145 or 6146 or perhaps SIIT-DC. I don't think it worries much about whether the translation is between a network and what is outside it (the single stack case) or different points in the same network. It's about translation between IPvx-only and IPvy-only domains in a network.
> 
>> 3.2. Encapsulation/Translation Based Transition Technologies
>> 
>>   For evaluating the performance of Encapsulation-based and
>>   Translation-based transition technologies a dual DUT setup (see
>>   Figure 2) SHOULD be employed. The tester creates a network flow of
>>   IPvX packets. The DUT CE is responsible for the encapsulation or
>>   translation of IPvX packets into IPvY packets. The IPvY packets are
>>   decapsulated/translated back to IPvX packets by the DUT PE and
>>   forwarded to the tester.
>>                           +--------------------+
>>                           |                    |
>>     +---------------------|IPvX   tester   IPvX|<------------------+
>>     |                     |                    |                   |
>>     |                     +--------------------+                   |
>>     |                                                              |
>>     |      +--------------------+      +--------------------+      |
>>     |      |                    |      |                    |      |
>>     +----->|IPvX    DUT CE  IPvY|----->|IPvY   DUT PE   IPvX|------+
>>            |    trans/encaps    |      |    trans/decaps    |
>>            +--------------------+      +--------------------+
>>                            Figure 2. Test setup 2
>> 
>> 
>>   In the case of translation based transition technology, the DUT CE
>>   and DUT PE machines MAY be tested separately as well. These tests
>>   can represent a fine grain performance analysis of the IPvX to IPvY
>>   translation direction versus the IPvY to IPvX translation direction.
>>   The tests SHOULD follow the test setup presented in Figure 1.
> 
> I think the above doesn't test encapsulation vs translation, but encapsulation in its various forms vs double translation such as described in MAP-T or SIIT-DC-2XLAT. The point is that you have IPvx-only domains at either end connected by an IPvy-only domain, and either you tunnel (encapsulate and decapsulate) or translate twice to cross the intervening network.
> 

###MG: I agree, it’s encapsulation vs double translation. That’s why I named them differently (encapsulation-based, translation-based). Maybe as noted above, changing the generic categories naming and assumption will be better.


>> 4.1. Frame Formats and Sizes
>> 
>>   [RFC5180] describes the frame size requirements for two commonly
>>   used media types: Ethernet and SONET (Synchronous Optical Network).
>>   [RFC2544] covers also other media types, such as token ring and
>>   FDDI. The two documents can be referred for the dual-stack
>>   transition technologies. For the rest of the transition technologies
>>   the frame overhead introduced by translation or encapsulation MUST
>>   be considered.
> 
> This gets wildly hairy pretty quickly. For example, in a data center, VxLAN is pretty common. To my way of thinking, you might want to get the person testing to describe his/her scenario ("I'm using VxLAN, IPvx within an NFV neighborhood, and IPvy between neighborhoods"), calculate the resulting frame size and frame overhead, and document it in the test report.

###MG: That would be a very reasonable addition. 

> 
> I'd stay away from link layer issues. Token Thing is interesting in comparison to Ethernet due to token rotation. Have you ever seen a Token Thing interface? Would you do better to describe the IPv4/IPv6 interaction and leave the rest as an MTU calculation?

###MG: I am to trying be as link layer agnostic as possible. However, the test data is generated as frames and the fragmentation behavior is one of the important performance aspects in my opinion. Maybe I’m missing the point?
 
>> 8. Scalability
>> 
>>   Scalability has been often discussed; however, in the context of
>>   network devices, a formal definition or a measurement method has not
>>   yet been approached.
>> 
>>   Scalability can be defined as the ability of each transition
>>   technology to accommodate network growth.
> 
> You might be interested to read RFC 3489 on its simplicity and coupling principles. Not disagreeing, but this might give you some insight into the scaling issues people routinely point to.
> 

###MG: the scalability dimension in scope here is actually load scalability. Maybe I should make this clearer. Was also thinking about structural scalability (e.g. A+P mapping address sharing efficiency). As I glanced through RFC3489, I guess that’s one of the issues. Will be analyzing in detail though. 

>>   Poor scalability usually leads to poor performance. Considering
>>   this, scalability can be measured by quantifying the network
>>   performance degradation while the network grows.
> 
> Maybe. Scaling issues can be, and often are, episodic. To pick an example, if I turn on the power in a building that contains a number of routers, routing protocols can spend tens of minutes beating each other into the ground until their randomization features decouple timing. This tests the specific case of linear growth in complexity over time.
> 
> I think I might describe two instances of the test, one of which opens a new session every mumble time units, to test linear scaling properties, and another that opens the same number of sessions but all at once, to test the episodic properties.

###MG: The way I tested so far was the all at once scenario. Adding a “linear scaling test"  would be a good addition.

> 
>> 8.1.1. Single-stack Transition Technologies
> 
> As noted, "single translation" cases.
> 
>> 8.1.2. Encapsulation/Translation Transition Technologies
> 
> As noted, encapsulation and "double translation" cases.
> 
>> 8.2. Benchmarking Performance Degradation
>> 
>>   Objective: To quantify the performance degradation introduced by n
>>   parallel network flows.
>> 
>>   Procedure: First the benchmarking tests presented in Section 6 have
>>   to be performed for one network flow.
>> 
>>   The same tests have to be repeated for n network flows. The
>>   performance degradation of the X benchmarking dimension SHOULD be
>>   calculated as relative performance change between the 1-flow results
>>   and the n-flow results, using the following formula:
>> 
>> 
>>              Xn - X1
>>       Xpd= ----------- * 100, where: X1 - result for 1-flow
>>                 X1                    Xn - result for n-flows
>> 
>>   Reporting Format: The performance degradation SHOULD be expressed as
>>   a percentage. The number of tested parallel flows n MUST be clearly
>>   specified. For each of the performed benchmarking tests, there
>>   SHOULD be a table containing a column for each frame size. The table
>>   SHOULD also state the applied frame rate.
> 
> OK as far as it goes. You have two fundamental things you need to test, which are the creation of state (linear or episodic) and the use of state (if I have N active sessions, what is the implication for performance?). I suspect that's at least three tests.

###MG: I am not sure what you mean by creation of state and use of state here.

Best regards,
Marius