Re: [tmrg] TCP Evaluation Suite

["SCHARF, Michael" <[e-mail]>]

Hi David,

I have never worked with the ns-2 implementation of the TCP test suite.
However, while I was at the University of Stuttgart, I implemented a
certain subset of the TCP test suite for an own TCP simulation tool that
uses the Network Simulation Cradle (NSC).

I mostly used the recommended TMIX traces in a dumbbell topology with
the recommended RTT values. Different to the recommended scenarios, I
was interested in broadband access with bottlenecks of the order of 10
Mbps. And I decided to ignore quite a number of the test cases, given
the lack of time to implement, simulate, and evaluate them.

Since my tool is not compatible to the ns-2 TCP test suite, and never
was meant to be so, many of my insights do not directly apply. Still,
please find some comments inline.

> 1.3 Simulation start up
> 
> Background:
> 
> To accelerate the system start up, the system is "prefilled" 
> to approximate a "steady state" of congestion. All 
> connections that would have started in time=0s to 
> time=prefill are started more closely together spaced over a 
> MaxRTT period before time=prefill.  The idea behind this is 
> to lessen the time taken before measurements can be taken. 
> Measurements are begin after time=2*prefill
> 
> 
> Fixed with font required.
> eg.
>                      <---->
> 			     MaxRTT
> |--------------------|----|-------------------------|
> t=0                       t=prefill                 t=2*prefill
>                      ^
>                      | start connections here.
> 
> 
> 
> *** Does any one foresee problems with attempting to reduce 
> the necessary
>   simulation warmup time with this method? ***

For my own tool, this method might have caused problems. Due to memory
constraints, my tool had to enforce an upper limit on the number of
active connections, which were taken out of a pool. If I correctly
understand your proposal, the number of parallel connections during the
warmup phase could be significantly larger than during the measurement.
If there is an upper limit on the number of connections, this suggestion
probably requires an increase of that limit. 

> 1.4. Packet sizes
> 
> The draft calls for 10% 536B and 90% 1500B TCP packets. To 
> achieve this, we have added a new MSS record has been added 
> to the Tmix traffic vector files, and Tmix enhanced to use 
> it. The original traces have been processed so that the TCP 
> MSS for each tmix connection is selected randomly (i.i.d.) 
> with 10% 496B and 90% 1460B.
> 
> Rationale: The maximum segment size is generally constant for 
> any particular
> 	      connection.
> 
> *** Is it enough to randomly select the MSS for each 
> connection in the trace
>     file without being concerned how much traffic each 
> connection generates?
>     ***

For whatever it is worth, I decided not to care about those 10% with
MSS=536B, like many other details... Shame on me ;)

> 2. SCALE SELECTION and LOSS TARGETS
> 
> The connection arrival times are scaled for each scenario so 
> as to achieve a certain average loss rate at the most 
> congested queue on the central link. The draft in general 
> omitted the target loss rates, but noted that moderate 
> congestion had about a 1-2% loss rate for the access link.
> 
> In these scenarios packet loss is not primarily caused by 
> long lived TCP sessions in congestion avoidance mode 
> cyclically filling the buffer. It is often caused by the 
> "collision" of multiple TCP sessions starting together.
> 
> It was difficult to achieve any stable results with these 
> targets for the dial-up and geostationary satellite cases. 
> Both of these scenarios experience very bursty loss, with the 
> dial-up scenario by being by far the worst. For these 
> scenarios we propose:
> 
> Geostationary satellite: Mild congestion 2%, Moderate 
> congestion 6% Dial-up link
> (64kbps): Mild congestion 5%, Moderate congestion 15%
> 
> Wireless link: to be studied.
> 
> The rule of thumb was moderate congestion is three times the 
> loss of mild congestion. For the non-congested link traffic 
> we propose a connection arrival rate of half that of the 
> mild-congestion scenario.
> 
> 
> *** Do these targets seem reasonably practical? Comments? 
> Suggestions? ***

I can't really comment on this since I studied different scenarios. But
in my experiments with 10 Mbps links and moderate load I measured a loss
rate of 1-2%. With nasty aggresssive congestion control schemes I
managed to measure up to 8% loss. I don't know whether any value beyond
that is really realistic.

> 3. SIMULATION TIMES
> 
> The draft recommended at least 100s. We have found that this 
> is not sufficient in any but the data center and transoceanic 
> scenarios.
> 
> The simulation times listed below are rough minima which 
> provide enough averaging for a reliable determination of the 
> connection arrival time scaling for the target loss rates. 
> Final values require further study.
> 
> Approximate simulation times:
>  - data center: ~85s (including 35s warmup)
>  - transoceanic: ~100s (40s warmup)
>  - access link: ~360s (60s warmup)
>  - geostationary: ~740s (40s warmup)
>  - dial-up: ~5100s (100s warmup)
> 
> NOTES
> 	1. The traffic is not stationary. 
> 	2. data center and transoceanic links have thousands of 
> concurrent
> 	TCP sessions and take a significant amount of real time.
> 
> 
> *** Is everyone happy with simulation times varying according 
> to the scenario?
>     Comments? Suggestions? ***

I used simulation times that were at least one order of magnitude
larger. As far as I recall, my minimum simulation time for 10 Mbps links
was at least 3600s with a warmup of about 100s.

BTW: I totally agree that it is an excellent idea to continue this work!

Michael