[bmwg] general network interconnect tester vs. siitpef -- Re: Agenda for BMWG session at IETF-109

Lencse Gábor <lencse@hit.bme.hu> Sat, 14 November 2020 11:09 UTC

Return-Path: <lencse@hit.bme.hu>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id EBDAE3A14D0 for <bmwg@ietfa.amsl.com>; Sat, 14 Nov 2020 03:09:50 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id GxKcKi6f6L6z for <bmwg@ietfa.amsl.com>; Sat, 14 Nov 2020 03:09:46 -0800 (PST)
Received: from frogstar.hit.bme.hu (frogstar.hit.bme.hu [IPv6:2001:738:2001:4020::2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 325123A14D1 for <bmwg@ietf.org>; Sat, 14 Nov 2020 03:09:45 -0800 (PST)
Received: from [] (host-79-121-41-201.kabelnet.hu []) (authenticated bits=0) by frogstar.hit.bme.hu (8.15.2/8.15.2) with ESMTPSA id 0AEB9Ldt085593 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sat, 14 Nov 2020 12:09:31 +0100 (CET) (envelope-from lencse@hit.bme.hu)
X-Authentication-Warning: frogstar.hit.bme.hu: Host host-79-121-41-201.kabelnet.hu [] claimed to be []
To: Vladimir Vassilev <vladimir@lightside-instruments.com>
Cc: bmwg@ietf.org
References: <4D7F4AD313D3FC43A053B309F97543CF0147645848@njmtexg5.research.att.com> <665bd46b-f8b6-6d4f-bcbe-92c958987b79@hit.bme.hu> <dc02d621-89c8-a631-ac09-4833cbb5291e@lightside-instruments.com>
From: =?UTF-8?Q?Lencse_G=c3=a1bor?= <lencse@hit.bme.hu>
Message-ID: <2d99fa8d-2c04-db9a-d9c7-9c7527eac872@hit.bme.hu>
Date: Sat, 14 Nov 2020 12:09:18 +0100
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.12.1
MIME-Version: 1.0
In-Reply-To: <dc02d621-89c8-a631-ac09-4833cbb5291e@lightside-instruments.com>
Content-Type: multipart/alternative; boundary="------------A8154D7E6EAD1C2AA3CB8027"
Content-Language: en-US
X-Virus-Scanned: clamav-milter 0.102.4 at frogstar.hit.bme.hu
X-Virus-Status: Clean
Received-SPF: pass (frogstar.hit.bme.hu: authenticated connection) receiver=frogstar.hit.bme.hu; client-ip=; helo=[]; envelope-from=lencse@hit.bme.hu; x-software=spfmilter 2.001 http://www.acme.com/software/spfmilter/ with libspf2-1.2.10;
X-DCC--Metrics: frogstar.hit.bme.hu; whitelist
X-Scanned-By: MIMEDefang 2.79 on
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/uliOFRCauRIEQ1t8fSdHDj1sKyg>
Subject: [bmwg] general network interconnect tester vs. siitpef -- Re: Agenda for BMWG session at IETF-109
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 14 Nov 2020 11:09:51 -0000

Dear Vladimir,

Thank you very much for your e-mail. Your general network interconnect 
tester based on a YANG model sounds interesting to me. (I was not aware 
of your draft [1] before.) As far as I now understand it, on the one 
hand, our projects are similar, and on the other hand, they are 

Both your YANG model based tester and my siitperf uses a split design. I 
think your concept with data plane and control plane is more clear, but 
I also used a similar idea: my C++ code is responsible for the data 
plane and my bash shell scripts are responsible for the control plane.

- Your project implements a general tester, whereas siitperf is very 
specific (designed for SIIT, including IPv4 or IPv6 routing) and it uses 
different executables for different measurements like throughput, 
latency, packet-delay variation.
- You used the two "extreme" solutions in my eyes: hardware and socket 
interface, whereas siitperf uses DPDK, which is in between of the two.

I think it can be a good idea for you to include also a DPDK version as 
a balanced compromise. Let me give you some performance data to support 
my suggestion:

On the one hand, I could test authoritative DNS servers by dns64perf++, 
which uses TCP/IP socket interface programming, by sending and receiving 
3.3 million queries per second using a DELL PowerEdge R430 server: I 
used 16 cores for sending and 16 cores for receiving. Please refer to my 
open access paper for the details: G. Lencse, "Benchmarking 
Authoritative DNS Servers", /IEEE Access/, vol. 8. pp. 130224–130238, 
July 2020. DOI: https://doi.org/10.1109/ACCESS.2020.3009141

On the other hand, the DPDK-based siitperf could send and receive more 
than 7 million frames per second _per direction_ using the same type of 
server and only a single core for sending and another for receiving. (In 
a bidirectional test, 4 cores in all, but then siitperf sent and 
received twice 7 million frames per second.) Please refer to my open 
access paper for the details: G. Lencse, "Design and Implementation of a 
Software Tester for Benchmarking Stateless NAT64 Gateways", /IEICE 
Transactions on Communications/, DOI: 

I think the numbers are convincing. :-)

I admit that DPDK has some requirements for the CPUs and especially for 
the NICs, but there are a lot of supported hardware: 

Thank you very much for inviting me into your project at Hackathon. 
Unfortunately, I am too busy to take part in it. However, I am happy to 
share my experience with you.

I have documented the design of siitperf in my above mentioned paper and 
the source code is also available for you as a free software. I would be 
happy if your team could reuse some part of it. The implementation of 
the RFC 4814 random port feature is currently in the "varport" branch, 
which I plan to merge into the master branch later on. As for random 
number generation, 64-bit Mersenne Twister (std::mt19937_64) is fast enough.

Currently the accuracy of frame timing is the greatest challenge for me. 
The inter-sending time of software testers is usually rather inaccurate 
at higher (e.g. several million
frames per second) rates. I have found a good paper about it, which 
compares some software packet generators. The authors did measurements 
with a NetFPGA device. If you plan to use DPDK, I highly recommend you 
to read their paper: P. Emmerich, S. Gallenmüller, G Antichi, A. W. 
Moore, G. Carle, "Mind the gap - A comparison of software packet 
generators", in: /2017 ACM/IEEE Symposium on Architectures for 
Networking and Communications Systems (ANCS)/, Beijing, China, 2017, 
doi: https://doi.org/10.1109/ANCS.2017.32

I have just finished the first round of testing the accuracy of siitperf 
in the sense, how the inaccurate timing may influence its results. I 
plan to finish my paper about it soon and it will be available from my 
list of publications: http://www.hit.bme.hu/~lencse/publications/

Perhaps I can also include some of its results into the slides of my 
presentation about siitperf at the BMWG session of IETF 109.

Just one advice concerning DPDK. Please be careful, when choosing its 
version to use. I used version 16.11.9-1+deb9u1 distributed in Debian 
9.9. However, my code does not compile with some newer DPDK versions.

I want to emphasize that I am not an expert of DPDK, but I am happy to 
help you, if I can. :-)

And let me ask you some questions about how your general tester could be 
used for my purposes. Siitperf is only the first step in the series of 
the RFC 8219 compliant testers necessary for benchmarking the five most 
important IPv4aaS technologies (464XLAT, DS-Lite, MAP-T, MAP-E, lw4o6), 
which we plan to use for the comparison of their performance as a part 
of our draft: 

Perhaps it is not so difficult to use your tester for benchmarking SIIT 
implementations. (Even if I have siitperf, it could be interesting to 
compare its results with that of your tester!) However, I cannot see, if 
it is possible to benchmark the B4 or AFTR devices of DS-Lite (RFC 6333) 
with your general tester. Is it possible to send encapsulated packets 
and/or to decapsulate them with your tester?

If you allow me some critical comments regarding your tester, let me 
mention two things.

1. It seems from your example that your tester sends bursts of frames. I 
understand that you can achieve higher frame rates in this way, however 
using a burst size higher than 1 makes your tests non compliant of RFC 
2544 / RFC 5180 / RFC 8219 as they require a constant frame rate.

2. It seems that you use integers to express the delay between frames 
(either within a burst or between bursts). Originally, I used the same 
solution to avoid a fixed point division: I used cpf (clock-ticks per 
frame), but it limited the resolution of the frame rates. (Just a 
numeric example: you have a 2GHz CPU and you want a 4 million frames per 
second rate. It means 2,000,000,000Hz/4,000,000fps=500cpf. The first 
possible highest frame rate is: 2,000,000,000Hz/499cpf=4,008,016fps.) I 
do not state that it is very bad, but I did not like it and I rather 
specify the required frame rate. (It also means that I use a fix point 
division in every single sending cycle. This is the cost of accuracy.)

Best regards,


11/11/2020 09:43 keltezéssel, Vladimir Vassilev írta:
> Hi Gábor,
> On 08/11/2020 19.21, Lencse Gábor wrote:
>> However, I have results regarding siitperf, my RFC 8219 compliant 
>> SIIT tester. (The usage of random port numbers is working well. I'm 
>> currently working on calibrating it with a legacy RFC 2544 tester by 
>> benhmarking the same DUT with the two devices and comparing their 
>> results.) If there is interest in the WG for it, I would be happy to 
>> present. But I do not want to push it.
>> So, what do you think?
> I looked into the implementation of siitperf. It is a well designed 
> reference implementation for RFC 8219 and a useful DPDK API usecase 
> and example in general.
> Maybe it is a long shot but I was wondering if you are participating 
> in the ongoing Hackathon? We are working on a relevant project that 
> implements a general network interconnect tester based on a YANG model 
> [1]. Instead of a single executable implementing a benchmark test 
> (e.g. RFC2544, RFC8219 etc.) one can split the design into control 
> plane (search logic, reconfiguration of the generator for different 
> trials, reading status) and data plane (the timing critical frame 
> input/output operations and timestamping). The control plane is 
> managed with a python script over NETCONF [2] and the data plane 
> workload is processed by software (native code in C) [3] or hardware 
> logic (Verilog) [4] that is started/enabled by the NETCONF server for 
> each interface when traffic generator configuration is committed.  The 
> command line looks like this (based on the YANG model for a generator 
> with single stream):
> $ ./traffic-generator --interface=eth0 --frame-size=64 
> --interframe-gap=20 --interburst-gap=124999852 \
>    --frames-per-burst=2 --total-frames=4 
> --realtime-epoch="2020-11-10T13:00:00.000000000Z" 
> --interface-speed=1000000000
> --frame-data="123456789ABCDEF01234567808004500002E000000000A112CBCC0000201C0000202C0200007001A0000000102030405060708090A0B0C0D0E0F10119CD50E0F" 
> We have an alternative traffic-generator-gmii tool [4] that instead of 
> using software to generate the packets writes to the registers of a 
> hardware core we have designed in Verilog. So replacing the command 
> name results in deterministic generation for the systems that have the 
> core available.
> We were thinking of a third option based on DPDK 
> (traffic-generator-dpdk) could be a compromise that is more 
> deterministic then the default (Unix userspace) option and would not 
> require programmable logic like the hardware dependent one.
> We could augment the YANG model adding choice and an alternative to 
> the static frame-data specifying the range and algorithm for 
> generation of a pseudorandom port numbers in the specified range as a 
> second reference implementation and do some testing of common DUTs and 
> compare the results produced by the different implementations.
> References:
> [1] 
> https://datatracker.ietf.org/doc/draft-vassilev-bmwg-network-interconnect-tester/
> [2] 
> https://github.com/vlvassilev/litenc/blob/master/tntapi/example/ietf-network-interconnect-tester/test-rfc2544-throughput.py 
> (seems the script diverged from the initial  throughput sec. 26.1 target)
> [3] 
> https://github.com/vlvassilev/yuma123/blob/master/example-modules/ietf-traffic-generator/traffic-generator.c
> [4] 
> https://github.com/vlvassilev/network-interconnect-tester-cores/blob/master/lib/hw/lsi/cores/traffic_generator_gmii/hdl/traffic_generator_gmii.v
> /Vladimir
>> Best regards,
>> Gábor