[bmwg] Sequential vs. random -- Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

[Gábor LENCSE <[e-mail]>]

Dear Brian,

I think that you know firewall testing better then me, thus you can make 
a better decision than me, and I will accept your decision anything it 
will be.

However, I would like to support you decision by sharing my experience 
and results with you. Full disclosure: I find the question of sequential 
vs. random very interesting and important regarding my own research. I 
hope that our discussion may be beneficial for both of us.


STATELESS TESTING

1. DNS64 and authoritative DNS server benchmarking

I used sequential source port numbers in dns64perf++ (implemented by my 
BSc student Dániel Bakai) only to support RSS (Receive-Side Scaling). It 
made me possible to perform authoritative DNS server performance 
measurements over 3 million queries per second rate. (The details were 
published in my open access paper: G. Lencse, "Benchmarking 
Authoritative DNS Servers", /IEEE Access/, vol. 8. pp. 130224-130238, 
July 2020. DOI: 10.1109/ACCESS.2020.3009141)

2. SIIT (also called stateless NAT64) benchmarking

In the original version of my DPDK-based RFC 8219 compliant SIIT tester 
called siitperf, I implemented the possibility of using random 
destination networks to comply with the requirement of RFC 2544. But I 
used fixed source and destination port numbers by literally following 
the Test Frame Format supplied in the appendix of RFC 2544. Then Al 
Morton has called my attention to RFC 4814. Then I added the feature of 
pseudorandom port numbers to comply with RFC 4814. Besides pseudorandom 
port numbers, I also added the possibility of increasing (or decreasing) 
port numbers. (I have seen this feature also in commercial RFC 2544 
testers.)

I have very interesting measurement results with using pseudorandom vs. 
sequential port numbers in IPv4 router benchmarking. (IPv4 routers were 
chosen to be able to achieve high throughput.) Please see Table I. in my 
open access paper: G. Lencse, "Adding RFC 4814 Random Port Feature to 
Siitperf: Design, Implementation and Performance Estimation", 
/International Journal of Advances in Telecommunications, 
Electrotechnics, Signals and Systems/, vol 9, no 3, pp. 18-26, 2020, 
DOI: 10.11601/ijates.v9i3.291 available also from: 
http://www.hit.bme.hu/~lencse/publications/291-1113-1-PB.pdf
If you consider the median throughput rates using:
- random source ports (and fixed destination port): 3,469,891 fps
- increasing source ports (and fixed destination port): 3,485,627 fps
- random destination ports (and fixed source port): 3,400,966 fps
- increasing destination ports (and fixed source port): 3,447,165 fps
Then you can see that there is not much difference, but increasing port 
numbers result in somewhat higher throughput than random ones. (It is 
also interesting that varying source port numbers cause higher 
throughput than varying destination port numbers. I attribute this 
difference to an asymmetry in the hash function: varying source port 
numbers can distribute the interrupts more evenly among the CPU cores.)

The fact that all the above result were close to the 3,402,271 fps 
throughput measured using pseudorandom source and destination port 
numbers to comply with RFC 4814, is very important for me: using 
sequential source (or destination, but not both) port numbers can be a 
computationally cheaper alternative to using pseudorandom source and 
destination port numbers.


STATEFUL TESTING

This is my very recent research area, and my surprising results made me 
recommending you to consider pseudorandom port numbers.

When I designed the stateful extension of siitperf, my original idea was 
to use sequential sweep in the port number range, which I called as 
"port number enumeration", in the preliminary phase of the stateful 
testing to load the (source IP, source port, destination IP, destination 
port) four tuples into both the connection tracking table of the DUT and 
the state table of the "Responder" subsystem of the Tester.
However, my test results showed huge difference in the maximum 
connection establishment rate of iptables!
Parameters of the tests:
source port range: 10,000 - 49,999
destination port range: 80 - 179
(Number of all possible combinations: 40,000*100=4,000,000)
nf_conntrack_max: 4,194,304
hashsize: 524,288
Number of preliminary frames: 4,000,000
Maximum connection establishment rate was determined by binary search. 
The results were shocking:
- with enumerated port numbers: 669,587 fps
- with random port numbers: 1,406,320

I was aware that random port numbers did not fully exhaust the possible 
port number combinations (some combinations appeared twice or even more 
times). Therefore, I have repeated the measurement with 40,000,000 
preliminary frames using random port numbers. The result was 
1,271,023fps, which still about the double of the result with enumerated 
port numbers.

You can find all the details in my paper currently under review: 
http://www.hit.bme.hu/~lencse/publications/SFNAT64-tester-for-review.pdf

Regarding the possible root cause of the phenomenon, I have my own 
explanation, but I do not want influence you.

What do you think of it?

This experience was the reason, why we wrote in our I-D that:

    Important warning: in normal router testing, the port number
    selection algorithm (whether it is pseudo-random or enumerated) does
    not affect final results.  However, our experience with iptables
    shows that if the connection tracking table is filled using port
    number enumeration in increasing order, then the maximum connection
    establishment rate of iptables degrades significantly compared to its
    performance using pseudorandom port numbers [LEN2021  <https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-stateful#ref-LEN2021>].

Source: 
https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-stateful


APPLICATION TO FIREWALL TESTING

Of course, I am aware that your case may be different in several aspects 
than yours.

For example, I willfully increased the size of the connection tracking 
table of iptables from its default value to a value recommended for a 
high-loaded NAT server: https://ixnfo.com/en/tuning-nf_conntrack.html
In addition to that, I have nearly fully filled the connection tracking 
table of the DUT!

These parameters likely do not apply to firewalls preforming deep packet 
inspection. :-)

My aim was to test the abilities of my stateful Tester in a demanding 
situation. I used NAT44 instead of NAT64 to be able to achieve higher 
rates. NAT44 is much-much simpler than the tasks performed by your 
firewalls, thus the lookup of the connections may dominate the execution 
time in my case, but it may be marginal in your case.

And there may be several other differences, which I do not even think 
of. This is why wrote that you need to decide, what is suitable for you. 
The most I can do is to give ideas to consider. :-)

Best regards,

Gábor



18/05/2021 22:28 keltezéssel, [e-mail] írta:
>
> Hi Gabor,
>
> Apologies for the tardy reply. All of the comments/suggestions/nits 
> you submitted look fine, with one exception. In your original e-mail 
> you said:
>
> “page 16
>
>    [...]  The behavior of the
>    client is to sweep through the given server IP space, sequentially
>    generating a recognizable service by the DUT.
>
> “I am not sure if sequential sweep is the right choice for two reasons:
> - I feel that the spirit of RFC 4814 would rather suggest 
> pseudorandom. To surely include all of them, I recommend random 
> permutation.
> - According to my experience, iptables has shown significantly 
> different maximum connection establishment rate, when I used 
> pseudorandom port numbers or when I enumerated the port numbers in 
> increasing order. (Details available upon request.)”
>
> We feel that in order to meet the goal of reproducible results that 
> introducing randomness is not something we want to do.  While we do 
> understand your position, we hope you can support our desire to no 
> make this change.
>
> Brian
>