[Tmrg] tmix-linux: burst of FIN packets causes packet loss at end of experiment

[lstewart at room52.net (Lawrence Stewart)]

Hi Lachlan and all,

Lachlan Andrew wrote:

[snip]

> 
> There is a fundamental statistical need to ignore the first little part,
> so that the number of connections can reach "steady state", but I don't
> know any   fundamental   reason to ignore the last 1/3 of the
> experiment, provided that traffic generator ends flows cleanly.  Since
> our suite is very time-constrained, we want to cut out any unnecessary
> waiting.
> 
> I'm Cc'ing this to TMRG in case someone on the list knows of a strong
> reason to ignore the end of an experiment (or knows a way around the
> SNMP problem).

I certainly couldn't say that ignoring the last 1/3rd of an experiment 
is necessary in my experience. However, I have observed some behaviour 
with the FreeBSD TCP implementation which also might be relevant to 
other TCPs as well and might be pertinent to this discussion.

Increasing the tx socket buffer size at the sender can lead to a 
situation where at the end of the connection, the userland process 
closes the socket, but the kernel finds itself with a large buffer of 
data still needing to be sent. Going on memory here, I recall observing 
that sometimes (haven't taken the time to narrow down when/why etc) some 
of the TCP variables can get apparently messed up e.g. cwnd can take on 
some unexpected values while the buffer is being flushed. I can't be 
more specific than that right now, but I hope to sit down and nut it out 
at some point.

Stepping back further, one might reasonably ask why you'd need to 
increase the tx socket buffer to a size where this problem is 
possible... I noticed by trial and error that when trying to use a 
non-real-time OS to do traffic generation, sometimes vagaries in kernel 
scheduling meant that you could end up with an empty tx socket buffer 
for periods of time during transmission if you didn't have the buffer 
sized a substantial amount larger than the BDP of the path.

I've worked around the issue by chopping the end off files after the 
time at which the traffic generation process (iperf in my case) closes 
the socket (normally a few seconds at most depending on the test 
parameters). Definitely not ideal, I know, but it works around the issue 
and I thought it was a story from the coal face worth sharing.

Cheers,
Lawrence