[bmwg] Reporting format -- Re: I-D Action: draft-ietf-bmwg-benchmarking-stateful-00.txt

[Gabor LENCSE <[e-mail]>]

Dear BMWG Chairs and List Members,

We have seen that RFC 8219 has a recommendation for reporting format, 
but our draft did not have it yet. Therefore, we drafted the following text:

4.5.1.  Typical Types of Measurement Series and Reporting Format

    Measurements are often taken as measurement series by changing the
    value of one or more parameter(s) to discover how the various values
    of the given parameter(s) influence the performance of the DUT.

    Scalability is a typical issue to be examined.  To that end, we
    RECOMMEND the following two measurement series:

    1.  Scalability against the number of sessions.  We RECOMMEND to use
        some representative values from the range of the potential number
        of sessions (that is, source port number destination port number
        combinations) the DUT may be faced with during its intended
        usage.

    2.  Scalability against the number of active CPU cores.  We RECOMMEND
        this measurement series if the number of active CPU cores of the
        DUT may be adjusted AND it is meaningful to use less than the
        maximum number of active CPU cores (e.g. when testing a software
        NATxy gateway implementation executed by a general purpose
        computer).

    Measurements MUST be executed multiple times.  The report of the
    results MUST contain the number of the repetitions of the
    measurements.  We RECOMMEND median as the summarizing function of the
    results complemented with the first percentile and the 99th
    percentile as indices of the dispersion of the results.  Average and
    standard deviation MAY also be reported.

    All parameters and settings that may influence the performance of the
    DUT MUST be reported.  Some of them may be specific to the given
    NATxy gateway implementation, like the "hashsize" (hash table size)
    and "nf_conntrack_max" (number of connection tracking table entries)
    values for iptables or the limit of the number of states for OpenBSD
    PF (set by the "set limit states number" command in the pf.conf
    file).


     number of sessions (req.)            0.4M       4M     40M     400M
     source port numbers (req.)         40,000   40,000  40,000   40,000
     destination port numbers (req.)        10      100   1,000   10,000
     "hashsize" (i.s.)                    2^17     2^20    2^23     2^27
     "nf_conntrack_max" (i.s.)            2^20     2^23    2^26     2^30
     num. sessions / "hashsize" (i.s.)    3.05     3.81    4.77     2.98
     number of experiments (req.)           10       10      10       10
     error of binary search (req.)       1,000    1,000   1,000    1,000
     connections/s median (req.)
     connections/s 1st perc. (req.)
     connections/s 99th perc. (req.)

       Figure 3: Non-validated maximum connection establishment rate of
         iptables against the number of sessions -- example for table
                                   headings


    Figure 3 shows an example for table headings for reporting the
    measurement results for the scalability of the iptables stateful
    NAT44 implementation against the number of sessions.  We have
    indicated the always required fields (req.) and the implementation
    specific ones (i.s.).  In row 6, we also added a computed value, the
    number of sessions per hashsize ratio, what helps the reader to
    interpret the achieved maximum connection establishment rate.  (A
    lower value results in shorter linked lists hanging on the entries of
    the hash table and thus facilitating higher performance.  The ratio
    is varying, because the number of sessions is always a power of 10,
    whereas hash table size is a power of 2.)  To reflect the accuracy of
    the results, we have also added the value of the error of binary
    search, that is, the difference between the higher bound and lower
    bound when the binary search stops.

    The table MUST be complemented with reporting the relevant parameters
    of the DUT.  If the DUT is a general purpose computer and some
    software NATxy gateway implementation is tested, then hardware
    description SHOULD include: computer type, CPU type and number of
    active CPU cores, memory type, size and speed, network interface card
    type (reflecting also the speed), the fact that direct cable
    connections were used or the type of the switch used for
    interconnecting the Tester and the DUT.  Operating system type and
    version, kernel version, and the version of the NATxy gateway
    implementation (including last commit date and number if applicable)
    SHOULD also be given.

What do you think of the text above?

Is it OK to use an example table here? Or should it go to the Appendix?

All comments are welcome!

Best regards,

Gábor


On 9/26/2022 11:18 AM, Gabor LENCSE wrote:
>
> Dear BMWG Chairs and List Members,
>
> First all, I am very happy for the adoption of our draft and thank you 
> for all your support!
>
> We have made the following main changes:
>
>     Added measurement setup for Stateful NAT64 gateways.
>
>     Consistency checking and corrections.
>
> As for the nit checker report regarding the non-RFC-6890 compliant IP 
> addresses, the following explanation was added:
>
>     Note: We are fully aware of [RFC6890  <https://datatracker.ietf.org/doc/html/rfc6890>] special purpose IP address
>     ranges.  The [RFC1918  <https://datatracker.ietf.org/doc/html/rfc1918>] private IP addresses are used to facilitate an
>     easy understanding of the example.  And we consider the usage of the
>     IP addresses reserved for benchmarking absolutely legitimate.
>
> We would be happy to receive reviews and recommendations how to 
> improve it. We are ready to publish an improved version before the 
> cutoff date of IETF 114 if needed.
>
> I am in Tokyo as a guest researcher at the Research Laboratory of 
> Internet Initiative Japan from September 15 to December 15 to do 
> research in the topic of benchmarking stateful NAT64 gateways. I plan 
> to validate the methodology described in our draft by benchmarking 
> various stateful NAT64 gateway implementations.
>
> So this is the best time, if you could recommend any improvement for 
> the methodology. :-)
>
> Best regards
>
> Gábor
>
>
> On 9/25/2022 3:02 AM, [e-mail] wrote:
>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>> This draft is a work item of the Benchmarking Methodology WG of the IETF.
>>
>>          Title           : Benchmarking Methodology for Stateful NATxy Gateways using RFC 4814 Pseudorandom Port Numbers
>>          Authors         : Gabor Lencse
>>                            Keiichi Shima
>>    Filename        : draft-ietf-bmwg-benchmarking-stateful-00.txt
>>    Pages           : 23
>>    Date            : 2022-09-24
>>
>> Abstract:
>>     RFC 2544 has defined a benchmarking methodology for network
>>     interconnect devices.  RFC 5180 addressed IPv6 specificities and it
>>     also provided a technology update, but excluded IPv6 transition
>>     technologies.  RFC 8219 addressed IPv6 transition technologies,
>>     including stateful NAT64.  However, none of them discussed how to
>>     apply RFC 4814 pseudorandom port numbers to any stateful NATxy
>>     (NAT44, NAT64, NAT66) technologies.  We discuss why using
>>     pseudorandom port numbers with stateful NATxy gateways is a difficult
>>     problem.  We recommend a solution limiting the port number ranges and
>>     using two phases: the preliminary test phase and the real test phase.
>>     We show how the classic performance measurement procedures (e.g.
>>     throughput, frame loss rate, latency, etc.) can be carried out.  We
>>     also define new performance metrics and measurement procedures for
>>     maximum connection establishment rate, connection tear down rate and
>>     connection tracking table capacity measurements.
>>
>>
>> The IETF datatracker status page for this draft is:
>> https://datatracker.ietf.org/doc/draft-ietf-bmwg-benchmarking-stateful/
>>
>> There is also an htmlized version available at:
>> https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-benchmarking-stateful-00
>>
>>
>> Internet-Drafts are also available by rsync at rsync.ietf.org::internet-drafts
>>
>>
>> _______________________________________________
>> bmwg mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/bmwg
>
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg