[bmwg] How to benchmark stateful middleboxes (e.g. CGN gateways)? -- Re: How to apply RFC 4814 pseudorandom source and destination port numbers, when benchmarking tunneling technologies?

[Lencse Gábor <[e-mail]>]

Dear David and all BMWG members!

First of all, David, thank you very much for your answer and advise!

Now I have started working on the extension of siitperf for benchmarking 
stateful NAT64 gateways. The problem I have encountered is much more 
general than IPv6 transition technologies, because a lot of ISP-s and 
network operators use CGN (Carrier-grade NAT).

Does anyone have experience with benchmarking CGN gateways in an RFC 
4814 compliant way?

The problem is the following:

                            +--------------------+
                            |                    |
               +------------|IPv4   Tester   IPv4|<-------------+
               |            |private       public|              |
               |            +--------------------+              |
               |                                                |
               |            +--------------------+              |
               |            |         DUT:       |              |
               +----------->|IPv4   CGN gw.  IPv4|--------------+
                            |private       public|
                            +--------------------+

Pseudorandom port numbers may not be used in the traffic forwarded 
through the DUT in the /right to left/ direction due to the stateful 
nature of the CGN. But this is not a very hard problem, because port 
numbers of the frames forwarded in the /left to right/ direction may be 
observed during a preliminary test and they may be stored and reused by 
randomly selecting of them as valid combinations (please see the details 
below).

However, using pseudorandom source and destination port numbers in the 
traffic forwarded through the DUT in the /left to right/ direction would 
be a DoS attack against the CGN gateway, because it would exhaust the 
connection tracking table of the CGN gateway!
Let us see some figures:
source port range: 1024-65535, its size is: 65535-1024+1=64512
destination port range: 1-49151, its size is: 49151-1+1=49151
The number of source and destination port combinations is: 
64512*49151=3,170,829,312
And yet we did not consider the requirement for testing with 256 
destination networks, which would further increase the number of 
connection table entries.

Considering the case of an ISP, it is likely that:
- the source port numbers will be quite different in the range of 
1024-65535 (maybe the distribution is not uniform, but skewed, however, 
I hope that using uniform distribution is perhaps not an absolutely bad 
model)
- there will be a few very popular ones among the destination port 
numbers, with the dominance of 80 (HTTP) and 443 (HTTPS), appearing also 
the port numbers of several other widely used protocols.

For me, the most important question is that: /Will these two 
distributions significantly influence the performance of the CGN gateway?/

Let me explain, why I consider it a key problem.

My first idea was to*implement stateful tests as follows:**
*
Let the user specify the port number ranges for the left to right 
traffic through the DUT (called forward direction in siitperf 
terminology) as follows:

|Fwd-sport-min 1024 Fwd-sport-max 65535 Fwd-dport-min 80 Fwd-dport-max 90|

That is the full RFC 4814 range for the source port numbers, and a 
narrow range for the destination port numbers.

Right before the actual benchmarking test (e.g. throughput measurement):
- the left side port of the Tester sends a specified number of frames 
(N) in the left to right direction through the DUT using pseudorandom 
source and destination port numbers from the specified ranges
- the right side port of the Tester collects the source IP address, 
source port number, destination IP address, destination port number 4 
tuples and stores them in a table, which has M entries, where M>=N.

During the actual benchmarking test:
- the left side port of the Tester sends frames with pseudorandom source 
and destination port numbers from the specified ranges
- the right side port of the Tester continuously updates its table of 
the collected 4 tuples, when it receives a frame. The table is used as a 
ring buffer. (It is filled up one by one, and if its last entry is 
written, then its write index is reset to 0.)
- the right side port of the Tester randomly selects an entry from the 
table of collected results for each test frame to generate, and uses the 
4 tuple (of course, exchanges source and destination) for test frame 
generation

Remarks, questions, concerns:
1. N should be large enough. How large? (Is it necessary to cover all 
possible source port number destination port number combinations?)
2. How large should M be? (E.g. M=N, or M>>N?)
3. What happens if a connection times out in the CGN gateway during the 
test, but its 4 tuple is still present in the table of the right port of 
the Tester? Generating and sending a test frame on the basis of it will 
cause frame loss at any low rate! Thus throughput measurement will be 
unreliable, and perhaps completely unusable! Therefore, M may not be 
arbitrarily large! How a good upper bound for M can be determined?

So this solution is not trivial, and may have several questions, but it 
seems to be manageable.

However, let us return to the question: /Will the rather skewed 
distribution of the frequency of the destination port numbers 
significantly influence the performance of the stateful CGN gateway?/

In one sense, the answer is surely YES. It it the necessary number of 
public IP addresses on the outside port of the CGN. (Users want to 
access the 80 or 443 port of a few very popular web sites.)

My question is if it make a difference in throughput, if the connections 
use let us say
- different destination ports from 80 to 90 and the same public source 
IP address
or
- destination port 80 and the necessary number of different public 
source IP address

If the throughput is not really different in the above two cases, then 
the above solution may be used. Otherwise, it may be necessary that the 
Tester inputs the distribution of destination port number required by 
the user and generates the traffic according to the distribution 
specified by the user. (I feel this solution much more complicated 
regarding both its implementation and also experimenting with it.)

What do you think?

If you know of any related guidelines (e.g. RFC-s, Internet Drafts, 
papers, etc.) published, then could you give me a pointer to them?

Thank you very much for all information, ideas, suggestions, experience, 
advise, etc. in advance!

Best regards,

Gábor

24/09/2020 00:02 keltezéssel, David Newman írta:
> We did not consider middleboxes when writing 4814, which was intended to
> address deficiencies in benchmarking of one or a small set of switches
> and routers.
>
> Per standard bmwg practice at the time, we treated the DUT/SUT as a
> single black box, where ingress and egress ports didn't know or care
> what happened inside the box.
>
> However, a possible problem with the pseudorandom-everywhere approach
> can arise when ingress/egress ports and one or more intermediate
> tunneling mechanisms require different and possibly conflicting patterns
> to obtain optimal traffic distributions at tunneling vs. egress ports.
>
> Obviously, if you're testing some sort of encapsulation scheme, then the
> specifics of that protocol are fixed, but you, as the tester, need to
> understand the traffic you're generating and whether it represents a
> fair test for arbitrary boxes or has been specifically tailored to your
> device or system.
>
> The spirit of 4814 is to use your knowledge of the test environment and
> protocols to provide a way to  scale test traffic that is free from bias
> when used to compare arbitrary boxes that perform the function that is
> tested.
>
> The original intent was to scale endpoints, e.g. MAC addresses, IP
> addresses, etc. If you're testing a protocol that is between two fixed
> endpoints, then scaling addresses might not be a reasonable approach for
> traffic scaling, even if it provides better performance because hashing
> is easier.
>
> You really have to use your best judgement, as a knowledgeable tester,
> to be fair and not cheat.
>
> Also, a wording nit: We have a strong allergy to the term "real-life" in
> the context of bmwg benchmarking. As former bmwg chair Scott Bradner
> once asked, "This begs the question: Real for whom?"
>
> As with approaches to encapsulation, there is no valid one-size-fits-all
> answer to that question.
>
> dn and tcp
>
>
> On 9/19/20 1:08 PM, Lencse Gábor wrote:
>> Dear Authors of RFC 4814 and also BMWG Members,
>>
>> RFC 4814 Section 4.5 recommends to use pseudorandom source and
>> destination port numbers. My experience show that their usage is an
>> essential precondition to receive realistic results, when multi-core
>> devices are benchmarked. (I have recently added this feature to
>> siitperf, my DPDK-based RFC 8219 compliant SIIT tester:
>> https://github.com/lencsegabor/siitperf/tree/varport Please see my
>> report on the BMWG mailing list for the details.)
>>
>> However, I am not sure, how random ports should be used, when tunneling
>> technologies are benchmarked. My MSc student is working on the
>> implementation of a DS-Lite tester. He is going to build two separate
>> Testers for the B4 and for the AFTR parts. For simplicity, let us
>> consider B4 first (as the stateful nature of the AFTR further
>> complicates the problem). The Single DUT test and traffic setup of RFC
>> 8219 would look like this:
>>
>>                             +--------------------+
>>                             |                    |
>>                +------------|IPv4   Tester   IPv6|<-------------+
>>                |            |              tunnel|              |
>>                |            +--------------------+              |
>>                |                                                |
>>                |            +--------------------+              |
>>                |            |                    |              |
>>                +----------->|IPv4     B4     IPv6|--------------+
>>                             |              tunnel|
>>                             +--------------------+
>>
>> Although the arrows are unidirectional, bidirectional traffic is
>> required. On the left side, the Tester may send IPv4 traffic with
>> pseudorandom source and destination port numbers. The DUT encapsulates
>> the IPv4 packets into IPv6 packets and forwards them to the IPv6 port of
>> the DUT. However, on the right side, the DUT has to use fixed UDP port
>> numbers, when sending IPv6 traffic, as there is a tunnel there. Of
>> course, the encapsulated IPv4 packets may have pseudorandom UDP port
>> numbers, but they are not taken into consideration by the NIC of the B4
>> device. Thus the interrupts caused by the arrival of the IPv6 packets
>> will not be hashed to all the cores of the B4 devices.
>>
>> IMHO, we may say that it is NOT a problem regarding the benchmarking
>> method, because of the following consideration:
>>
>> We use pseudorandom port numbers for benchmarking in order to
>> approximate the nature of the real life traffic, which usually has
>> different IP addresses and port numbers. However, in this case, the real
>> life traffic does have fixed port numbers due to the tunnel, thus having
>> fixed UDP port numbers of over IPv6 complies with the real life traffic.
>> (In addition to that, B4 is usually a small CE device, which perhaps
>> does not have too many cores.)
>>
>> Am I right, or do I miss something?
>>
>> The situation of the AFTR is very different.
>>
>>                             +--------------------+
>>                             |                    |
>>                +------------|IPv6   Tester   IPv4|<-------------+
>>                |            |tunnel              |              |
>>                |            +--------------------+              |
>>                |                                                |
>>                |            +--------------------+              |
>>                |            |                    |              |
>>                +----------->|IPv6    AFTR    IPv4|--------------+
>>                             |tunnel              |
>>                             +--------------------+
>>
>> On the left side, the Tester needs to send IPv6 packets that encapsulate
>> IPv4 packets. In real life, a high number of B4 devices are connected to
>> an AFTR. Thus although the port numbers are fixed due to the tunnels,
>> the IPv6 source addresses are different, and thus the interrupts caused
>> by the packets are hashed more or less equally to the several CPU cores
>> of the AFTR. (Of course, the encapsulated IPv4 packets have pseudorandom
>> port numbers, but the NIC does not "see" them.)
>>
>> Am I right, that here the real spirit of RFC 4814 is that the Tester
>> should use different source IPv6 addresses?
>>
>> Of course, their number (e.g. 256 or 65536) is a further question.
>>
>> As for the IPv4 traffic in the other direction, varying port number may
>> be (and should be) used. However, they may not be arbitrarily due to the
>> stateful NAPT44 in the AFTR. To that end, I have an idea: the tester
>> should send preliminary traffic in the left to right direction, where
>> the encapsulated IPv4 packets have varying (not random but rather
>> one-by-one increasing) port numbers, and it should store the observed
>> port numbers on the right side. Then it may randomly select from among
>> the observed port numbers, when generating traffic in the right to left
>> direction. But it is beyond the scope of my current question.
>>
>> I really appreciate any suggestions, comments, ideas!
>>
>> Best regards,
>>
>> Gábor Lencse
>>
>> -----------------------------------------------------
>> Dr. Gabor Lencse
>> Professor
>> Department of Telecommunications
>> Szechenyi Istvan University
>> Egyetem ter 1.  Gyor, H-9026, HUNGARY
>> FAX: [phone]
>> E-Mail: [e-mail]
>> URL: http://www.hit.bme.hu/people/lencse/index_en.htm
>>