TMux performance results

[Pete Cameron <[e-mail]>]

Here are the test results that we obtained during our
performance tests of TMux. These are the figures I promised
to post during the Houston IETF BOF.

I have a postscript version of this with the diagrams, it
can be obtained via anon ftp from

	xylint.co.uk:tmux/tmux_results.ps

Please note, the postscript version is setup for european A4
sized paper, but should print ok on American letter sized,
but you may have to convince your printer that is what is
loaded (hit the continue button?).

Pete Cameron                                            tel: [phone]
Consulting Engineer                                     fax: [phone]
Xylogics International Limited                  email: [e-mail]
Featherstone Rd, Wolverton Mill, MK12 5RD, UK.         [e-mail]

===========================================================================


1 Introduction
                                    
The purpose of this document is to collect together and analyze the
results which have been obtained from various tests performed on the
Tmux software.
                                    
2 The Test Environment
                                    
                                    
In order to generate useful test results involving large port counts and
high data throughput  an automated and programmable test rig was
required. A special operating image for a stand alone Annex terminal
server was created which could emulate upto 32 users. This image
generates characters at fixed intervals and checks the reply characters.
                                    
The serial output from this test Annex can then be fed into a second
Annex terminal server whose ports are running TELNET sessions into the
host system under test (see Fig 1).
                                    
3 Test Performed
                                    
                                    
The purpose of this test is to simulate reasonably good typists entering
data into an application such as a word processor.
                                    
In this type of application it is the word processor or editor that
performs the character echo so for these tests a simple application
program has been written which executes on the host system and performs
the echoing. To simplify starting the tests a dummy user was added to
the host machine. When this user is logged in the program is invoked by
the login script.
                                    
The test is controlled by a special Annex image run on a independent
Annex unit (ie. not a unit that may be involved in the test). This unit
repeatedly sends a single character ('a') out of the serial port, then
receives a single character in reply ('0', the echo simulation). The
time for this echo character to be received is also calculated, and 
used to calculate a sleep time before the next cycle is started. On
these tests, the cycle is repeated every 200ms, to simulate a 5
character per second typist. In addition, the time taken for the round
trip is recorded for later analysis to ensure that the cycles are being
repeated within the allotted time. As well as the time information, the
echo character received is checked, and any errors are also logged. The
test runs until stopped by the operator.
                                    
This test image will thus generate a constant predetermined and
repeatable load that is independent of the response time of the host
under test. The load is generated on the Annex serial ports, so that 
it can be passed to the host under test via another Annex unit.
                                    
                                    
4 Test Results
                                    
The tests were carried out on a single 33 MHz sparc processor with 16
MBytes of system memory.
                                    
The tests were carried out using the standard out of kernel TELNET
implementation and a Xylogics in kernel protocol.
                                    
Tests could only be performed upto a load of 25 ports as the system ran
out of memory at this point. In practice most systems have much more
memory than our test rig and can support much higher workloads before
they start to swap.
                                    
                                    
Statistics were collected over a 30 minute period having allowed 15
minutes initial settling time. All tests were run on a private network.
                                    
                                    
5 cps Results for Telnet
Ports     %usr      %sys      %idle     %host load     Response (ms)
5         1         8         91             9         20
10        2         19        79             21        22
15        4         37        58             42        24
20        5         43        52             48        29
25        6         53        40             60        40
                                    
5 cps Results for Telnet and TMux
Ports     %usr      %sys      %idle     %host load     Response (ms)
5         1         9         90             10        58
10        2         17        81             19        59
15        4         26        71             29        62
20        5         34        61             39        70
25        6         44        50             50        80
                                    
5 cps Results for in kernel protocol
Ports     %usr      %sys      %idle     %host load     Response (ms)
5         0         6         94             6         20
10        0         14        85             15        25
15        1         18        81             19        26
20        1         25        74             26        33
25        1         29        70             30        59
                                    
5 cps Results for in Kernel protocol and TMux
Ports     %usr      %sys      %idle     %host load     Response (ms)
5         0         5         95             5         58
10        0         9         91             9         60
15        0         12        88             12        65
20        1         15        84             16        76
25        1         19        80             20        96
                                    
Notes:
                                    
%usr, %sys, and %idle valuse were obtained from the average line
produced by the Unix 'sar' command.
                                    
%host load is calculated as 100-%idle.
                                    
Response time is measured from the time the character is sent by the
driver Annex until the echo character is received. Thus it includes the
time needed to transmit the original character and the echo (at 9600
baud).