Re: [bmwg] general network interconnect tester vs. siitpef -- Re: Agenda for BMWG session at IETF-109

[Vladimir Vassilev <[e-mail]>]

Hi Gábor,

Thank you for the reply.


On 14/11/2020 12.09, Lencse Gábor wrote:
> Dear Vladimir,
>
> Thank you very much for your e-mail. Your general network interconnect 
> tester based on a YANG model sounds interesting to me. (I was not 
> aware of your draft [1] before.) As far as I now understand it, on the 
> one hand, our projects are similar, and on the other hand, they are 
> complementary.
>
> Similarity:
> Both your YANG model based tester and my siitperf uses a split design. 
> I think your concept with data plane and control plane is more clear, 
> but I also used a similar idea: my C++ code is responsible for the 
> data plane and my bash shell scripts are responsible for the control 
> plane.
>
> Differences:
> - Your project implements a general tester, whereas siitperf is very 
> specific (designed for SIIT, including IPv4 or IPv6 routing) and it 
> uses different executables for different measurements like throughput, 
> latency, packet-delay variation.
> - You used the two "extreme" solutions in my eyes: hardware and socket 
> interface, whereas siitperf uses DPDK, which is in between of the two.


Yes

The Unix raw socket solution [3] is just a very simple way to document 
the design pretending it will operate in an ideal world.

While the Verilog core [4] requires specialized hardware with 
programmable logic available but provides full determinism (runs at the 
GMII interface) which we needed to calibrate and evaluate other 
implementations.


>
> I think it can be a good idea for you to include also a DPDK version 
> as a balanced compromise. Let me give you some performance data to 
> support my suggestion:
>
> On the one hand, I could test authoritative DNS servers by 
> dns64perf++, which uses TCP/IP socket interface programming, by 
> sending and receiving 3.3 million queries per second using a DELL 
> PowerEdge R430 server: I used 16 cores for sending and 16 cores for 
> receiving. Please refer to my open access paper for the details: G. 
> Lencse, "Benchmarking Authoritative DNS Servers", /IEEE Access/, vol. 
> 8. pp. 130224–130238, July 2020. DOI: 
> https://doi.org/10.1109/ACCESS.2020.3009141
>
> On the other hand, the DPDK-based siitperf could send and receive more 
> than 7 million frames per second _per direction_ using the same type 
> of server and only a single core for sending and another for 
> receiving. (In a bidirectional test, 4 cores in all, but then siitperf 
> sent and received twice 7 million frames per second.) Please refer to 
> my open access paper for the details: G. Lencse, "Design and 
> Implementation of a Software Tester for Benchmarking Stateless NAT64 
> Gateways", /IEICE Transactions on Communications/, DOI: 
> https://doi.org/10.1587/transcom.2019EBN0010
>
> I think the numbers are convincing. :-)
>
> I admit that DPDK has some requirements for the CPUs and especially 
> for the NICs, but there are a lot of supported hardware: 
> https://core.dpdk.org/supported/


I agree with that.


>
>
> Thank you very much for inviting me into your project at Hackathon. 
> Unfortunately, I am too busy to take part in it. However, I am happy 
> to share my experience with you.
>
> I have documented the design of siitperf in my above mentioned paper 
> and the source code is also available for you as a free software. I 
> would be happy if your team could reuse some part of it. The 
> implementation of the RFC 4814 random port feature is currently in the 
> "varport" branch, which I plan to merge into the master branch later 
> on. As for random number generation, 64-bit Mersenne Twister 
> (std::mt19937_64) is fast enough.
>
> Currently the accuracy of frame timing is the greatest challenge for 
> me. The inter-sending time of software testers is usually rather 
> inaccurate at higher (e.g. several million
> frames per second) rates. I have found a good paper about it, which 
> compares some software packet generators. The authors did measurements 
> with a NetFPGA device. If you plan to use DPDK, I highly recommend you 
> to read their paper: P. Emmerich, S. Gallenmüller, G Antichi, A. W. 
> Moore, G. Carle, "Mind the gap - A comparison of software packet 
> generators", in: /2017 ACM/IEEE Symposium on Architectures for 
> Networking and Communications Systems (ANCS)/, Beijing, China, 2017, 
> doi: https://doi.org/10.1109/ANCS.2017.32


One of the requirements for the model design was that it can be used 
both with not very deterministic software testers as well as with 
deterministic hardware testers. The main benefit being that replacing a 
tester can change the precision without requiring alternative test to be 
coded. Currently existing hardware testers can mostly implement the base 
model with simple agent translating NETCONF transactions to the native 
interfaces they have. (SCPI etc.)


>
> I have just finished the first round of testing the accuracy of 
> siitperf in the sense, how the inaccurate timing may influence its 
> results. I plan to finish my paper about it soon and it will be 
> available from my list of publications: 
> http://www.hit.bme.hu/~lencse/publications/
>
> Perhaps I can also include some of its results into the slides of my 
> presentation about siitperf at the BMWG session of IETF 109.
>
> Just one advice concerning DPDK. Please be careful, when choosing its 
> version to use. I used version 16.11.9-1+deb9u1 distributed in Debian 
> 9.9. However, my code does not compile with some newer DPDK versions.
>
> I want to emphasize that I am not an expert of DPDK, but I am happy to 
> help you, if I can. :-)


Based on the siitperf code I went through I think you are being too 
humble. I will ask for help when we get there.


>
> And let me ask you some questions about how your general tester could 
> be used for my purposes. Siitperf is only the first step in the series 
> of the RFC 8219 compliant testers necessary for benchmarking the five 
> most important IPv4aaS technologies (464XLAT, DS-Lite, MAP-T, MAP-E, 
> lw4o6), which we plan to use for the comparison of their performance 
> as a part of our draft: 
> https://tools.ietf.org/html/draft-lmhp-v6ops-transition-comparison
>
> Perhaps it is not so difficult to use your tester for benchmarking 
> SIIT implementations. (Even if I have siitperf, it could be 
> interesting to compare its results with that of your tester!) However, 
> I cannot see, if it is possible to benchmark the B4 or AFTR devices of 
> DS-Lite (RFC 6333) with your general tester. Is it possible to send 
> encapsulated packets and/or to decapsulate them with your tester?


The concept of allowing external technology specific YANG modules to 
augment the base module (ref. ietf-interfaces from RFC8343 etc.) is 
available to add support for this kind of dynamic transformations of the 
stream frame.

One example would be a protocol field like the ipv4 udp destination port 
to increment (or variate pseudo randomly) in certain range. To address 
that requirement one would need to specify additional YANG module and 
implement the functionality. Another example would be dynamic frame-size 
(rfc6985) within the stream (not multi-stream which is supported but 
requires a stream for each alternative frame). I intend to add some 
examples of common trivial cases of dynamic transformations and 
hopefully the experts in different areas can build on top of that.


>
> If you allow me some critical comments regarding your tester, let me 
> mention two things.
>
> 1. It seems from your example that your tester sends bursts of frames. 
> I understand that you can achieve higher frame rates in this way, 
> however using a burst size higher than 1 makes your tests non 
> compliant of RFC 2544 / RFC 5180 / RFC 8219 as they require a constant 
> frame rate.


Since bursts are optional in the model (when 'interburst-gap' and 
'frames-per-burst' parameters are not present) one can use them only 
when needed (not use them in RFC2544 tests e.g. specify only 
'interframe-gap').


>
> 2. It seems that you use integers to express the delay between frames 
> (either within a burst or between bursts). Originally, I used the same 
> solution to avoid a fixed point division: I used cpf (clock-ticks per 
> frame), but it limited the resolution of the frame rates. (Just a 
> numeric example: you have a 2GHz CPU and you want a 4 million frames 
> per second rate. It means 2,000,000,000Hz/4,000,000fps=500cpf. The 
> first possible highest frame rate is: 
> 2,000,000,000Hz/499cpf=4,008,016fps.) I do not state that it is very 
> bad, but I did not like it and I rather specify the required frame 
> rate. (It also means that I use a fix point division in every single 
> sending cycle. This is the cost of accuracy.)


Clearly the supported frames per second (fps) with a CONSTANT interframe 
gap per stream is a discrete set of real numbers - for a 64 octet frame 
and 1Gb ethernet interface (1000000000/8)/(64+8+12+i) => 
{1488095.238095238, 1470588.235294118, 1453488.372093023 ...}. However 
the generation is specified in a 100% deterministic way for that set 
when one uses the model with integer frame-size/gap. There is a way to 
support fps outside of that set if one specifies a VARIABLE 
interframe-gap augmentation (e.g.  in case you need 12.5 interframe gap 
to achieve certain fps you need to iterate between 12 and 13 octets) or 
one can add a second stream for every second to add a slightly longer 
interframe gap at the end of each second (or other period) to get a 
integer number of frames in that period. If one does that as part of the 
model we will not loose the determinism in order to gain flexibility at 
some higher level abstraction.

That said a higher-level interface can still use alternative metrics but 
it should be able to map those to the deterministic model so that it can 
execute on a hardware tester.

Let me know if you plan some open test event we can join. Perhaps the 
next IETF hackathon?

/Vladimir


>
> Best regards,
>
> Gábor
>
>
> 11/11/2020 09:43 keltezéssel, Vladimir Vassilev írta:
>> Hi Gábor,
>>
>> On 08/11/2020 19.21, Lencse Gábor wrote:
>>> However, I have results regarding siitperf, my RFC 8219 compliant 
>>> SIIT tester. (The usage of random port numbers is working well. I'm 
>>> currently working on calibrating it with a legacy RFC 2544 tester by 
>>> benhmarking the same DUT with the two devices and comparing their 
>>> results.) If there is interest in the WG for it, I would be happy to 
>>> present. But I do not want to push it.
>>>
>>> So, what do you think?
>>
>>
>> I looked into the implementation of siitperf. It is a well designed 
>> reference implementation for RFC 8219 and a useful DPDK API usecase 
>> and example in general.
>>
>> Maybe it is a long shot but I was wondering if you are participating 
>> in the ongoing Hackathon? We are working on a relevant project that 
>> implements a general network interconnect tester based on a YANG 
>> model [1]. Instead of a single executable implementing a benchmark 
>> test (e.g. RFC2544, RFC8219 etc.) one can split the design into 
>> control plane (search logic, reconfiguration of the generator for 
>> different trials, reading status) and data plane (the timing critical 
>> frame input/output operations and timestamping). The control plane is 
>> managed with a python script over NETCONF [2] and the data plane 
>> workload is processed by software (native code in C) [3] or hardware 
>> logic (Verilog) [4] that is started/enabled by the NETCONF server for 
>> each interface when traffic generator configuration is committed.  
>> The command line looks like this (based on the YANG model for a 
>> generator with single stream):
>>
>> $ ./traffic-generator --interface=eth0 --frame-size=64 
>> --interframe-gap=20 --interburst-gap=124999852 \
>>    --frames-per-burst=2 --total-frames=4 
>> --realtime-epoch="2020-11-10T13:00:00.000000000Z" 
>> --interface-speed=1000000000
>> --frame-data="123456789ABCDEF01234567808004500002E000000000A112CBCC0000201C0000202C0200007001A0000000102030405060708090A0B0C0D0E0F10119CD50E0F" 
>>
>>
>> We have an alternative traffic-generator-gmii tool [4] that instead 
>> of using software to generate the packets writes to the registers of 
>> a hardware core we have designed in Verilog. So replacing the command 
>> name results in deterministic generation for the systems that have 
>> the core available.
>>
>> We were thinking of a third option based on DPDK 
>> (traffic-generator-dpdk) could be a compromise that is more 
>> deterministic then the default (Unix userspace) option and would not 
>> require programmable logic like the hardware dependent one.
>>
>> We could augment the YANG model adding choice and an alternative to 
>> the static frame-data specifying the range and algorithm for 
>> generation of a pseudorandom port numbers in the specified range as a 
>> second reference implementation and do some testing of common DUTs 
>> and compare the results produced by the different implementations.
>>
>> References:
>>
>> [1] 
>> https://datatracker.ietf.org/doc/draft-vassilev-bmwg-network-interconnect-tester/
>>
>> [2] 
>> https://github.com/vlvassilev/litenc/blob/master/tntapi/example/ietf-network-interconnect-tester/test-rfc2544-throughput.py 
>> (seems the script diverged from the initial  throughput sec. 26.1 
>> target)
>>
>> [3] 
>> https://github.com/vlvassilev/yuma123/blob/master/example-modules/ietf-traffic-generator/traffic-generator.c
>>
>> [4] 
>> https://github.com/vlvassilev/network-interconnect-tester-cores/blob/master/lib/hw/lsi/cores/traffic_generator_gmii/hdl/traffic_generator_gmii.v
>>
>>
>> /Vladimir
>>
>>
>>>
>>> Best regards,
>>>
>>> Gábor