Re: [bmwg] Sequential vs. random -- Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

[[e-mail]]

HI Gabor,

 

We will update page 16 of the draft to allow for either sequential or
pseudorandom port numbers. 

 

Thanks for your detailed explanation.

 

Brian 

 

From: Gábor LENCSE <[e-mail]> 
Sent: Wednesday, May 19, 2021 3:49 AM
To: [e-mail]; 'MORTON JR., AL' <[e-mail]>
Cc: 'Bala Balarajah' <[e-mail]>; 'Bala Balarajah'
<[e-mail]>; [e-mail]; 'MORTON, ALFRED C (AL)'
<[e-mail]>; 'Sarah Banks' <[e-mail]>; Dr. Lencse
Gábor <[e-mail]>
Subject: Sequential vs. random -- Re: [bmwg] FW: WGLC on New version of
draft-ietf-bmwg-ngfw-performance

 

Dear Brian,

I think that you know firewall testing better then me, thus you can make a
better decision than me, and I will accept your decision anything it will
be.

However, I would like to support you decision by sharing my experience and
results with you. Full disclosure: I find the question of sequential vs.
random very interesting and important regarding my own research. I hope that
our discussion may be beneficial for both of us. 


STATELESS TESTING

1. DNS64 and authoritative DNS server benchmarking

I used sequential source port numbers in dns64perf++ (implemented by my BSc
student Dániel Bakai) only to support RSS (Receive-Side Scaling). It made me
possible to perform authoritative DNS server performance measurements over 3
million queries per second rate. (The details were published in my open
access paper: G. Lencse, "Benchmarking Authoritative DNS Servers", IEEE
Access, vol. 8. pp. 130224-130238, July 2020. DOI:
10.1109/ACCESS.2020.3009141) 

2. SIIT (also called stateless NAT64) benchmarking

In the original version of my DPDK-based RFC 8219 compliant SIIT tester
called siitperf, I implemented the possibility of using random destination
networks to comply with the requirement of RFC 2544. But I used fixed source
and destination port numbers by literally following the Test Frame Format
supplied in the appendix of RFC 2544. Then Al Morton has called my attention
to RFC 4814. Then I added the feature of pseudorandom port numbers to comply
with RFC 4814. Besides pseudorandom port numbers, I also added the
possibility of increasing (or decreasing) port numbers. (I have seen this
feature also in commercial RFC 2544 testers.)

I have very interesting measurement results with using pseudorandom vs.
sequential port numbers in IPv4 router benchmarking. (IPv4 routers were
chosen to be able to achieve high throughput.) Please see Table I. in my
open access paper: G. Lencse, "Adding RFC 4814 Random Port Feature to
Siitperf: Design, Implementation and Performance Estimation", International
Journal of Advances in Telecommunications, Electrotechnics, Signals and
Systems, vol 9, no 3, pp. 18-26, 2020, DOI: 10.11601/ijates.v9i3.291
available also from:
http://www.hit.bme.hu/~lencse/publications/291-1113-1-PB.pdf
If you consider the median throughput rates using:
- random source ports (and fixed destination port): 3,469,891 fps
- increasing source ports (and fixed destination port): 3,485,627 fps
- random destination ports (and fixed source port): 3,400,966 fps
- increasing destination ports (and fixed source port): 3,447,165 fps
Then you can see that there is not much difference, but increasing port
numbers result in somewhat higher throughput than random ones. (It is also
interesting that varying source port numbers cause higher throughput than
varying destination port numbers. I attribute this difference to an
asymmetry in the hash function: varying source port numbers can distribute
the interrupts more evenly among the CPU cores.)

The fact that all the above result were close to the 3,402,271 fps
throughput measured using pseudorandom source and destination port numbers
to comply with RFC 4814, is very important for me: using sequential source
(or destination, but not both) port numbers can be a computationally cheaper
alternative to using pseudorandom source and destination port numbers.


STATEFUL TESTING

This is my very recent research area, and my surprising results made me
recommending you to consider pseudorandom port numbers. 

When I designed the stateful extension of siitperf, my original idea was to
use sequential sweep in the port number range, which I called as "port
number enumeration", in the preliminary phase of the stateful testing to
load the (source IP, source port, destination IP, destination port) four
tuples into both the connection tracking table of the DUT and the state
table of the "Responder" subsystem of the Tester.
However, my test results showed huge difference in the maximum connection
establishment rate of iptables! 
Parameters of the tests:
source port range: 10,000 - 49,999 
destination port range: 80 - 179
(Number of all possible combinations: 40,000*100=4,000,000)
nf_conntrack_max: 4,194,304
hashsize: 524,288
Number of preliminary frames: 4,000,000
Maximum connection establishment rate was determined by binary search. The
results were shocking:
- with enumerated port numbers: 669,587 fps
- with random port numbers: 1,406,320

I was aware that random port numbers did not fully exhaust the possible port
number combinations (some combinations appeared twice or even more times).
Therefore, I have repeated the measurement with 40,000,000 preliminary
frames using random port numbers. The result was 1,271,023fps, which still
about the double of the result with enumerated port numbers.

You can find all the details in my paper currently under review:
http://www.hit.bme.hu/~lencse/publications/SFNAT64-tester-for-review.pdf

Regarding the possible root cause of the phenomenon, I have my own
explanation, but I do not want influence you. 

What do you think of it?

This experience was the reason, why we wrote in our I-D that:

   Important warning: in normal router testing, the port number
   selection algorithm (whether it is pseudo-random or enumerated) does
   not affect final results.  However, our experience with iptables
   shows that if the connection tracking table is filled using port
   number enumeration in increasing order, then the maximum connection
   establishment rate of iptables degrades significantly compared to its
   performance using pseudorandom port numbers [LEN2021
<https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-statef
ul#ref-LEN2021> ].

Source:
https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-statefu
l


APPLICATION TO FIREWALL TESTING

Of course, I am aware that your case may be different in several aspects
than yours. 

For example, I willfully increased the size of the connection tracking table
of iptables from its default value to a value recommended for a high-loaded
NAT server: https://ixnfo.com/en/tuning-nf_conntrack.html
In addition to that, I have nearly fully filled the connection tracking
table of the DUT!

These parameters likely do not apply to firewalls preforming deep packet
inspection. :-)

My aim was to test the abilities of my stateful Tester in a demanding
situation. I used NAT44 instead of NAT64 to be able to achieve higher rates.
NAT44 is much-much simpler than the tasks performed by your firewalls, thus
the lookup of the connections may dominate the execution time in my case,
but it may be marginal in your case.

And there may be several other differences, which I do not even think of.
This is why wrote that you need to decide, what is suitable for you. The
most I can do is to give ideas to consider. :-)

Best regards,

Gábor




18/05/2021 22:28 keltezéssel, [e-mail]
<mailto:[e-mail]>  írta:

Hi Gabor,

 

Apologies for the tardy reply. All of the comments/suggestions/nits you
submitted look fine, with one exception. In your original e-mail you said:

 

“page 16

   [...]  The behavior of the
   client is to sweep through the given server IP space, sequentially
   generating a recognizable service by the DUT.  

“I am not sure if sequential sweep is the right choice for two reasons:
- I feel that the spirit of RFC 4814 would rather suggest pseudorandom. To
surely include all of them, I recommend random permutation.
- According to my experience, iptables has shown significantly different
maximum connection establishment rate, when I used pseudorandom port numbers
or when I enumerated the port numbers in increasing order. (Details
available upon request.)”

 

We feel that in order to meet the goal of reproducible results that
introducing randomness is not something we want to do.  While we do
understand your position, we hope you can support our desire to no make this
change.

 

Brian