Re: Fwd: Re: [tcpm] FW: Call for Adoption: TCP Tuning for HTTP

[Joe Touch <[e-mail]>]

On 3/2/2016 10:55 PM, Willy Tarreau wrote:
> On Wed, Mar 02, 2016 at 03:34:42PM -0800, Joe Touch wrote:
>>
>>
>> On 3/2/2016 3:21 PM, Willy Tarreau wrote:
>>> On Wed, Mar 02, 2016 at 02:34:38PM -0800, Joe Touch wrote:
>>>> - it has significant errors
>>>>
>>>> 	TIME-WAIT issues apply to servers, not clients.
>>>
>>> Sorry but no it's the opposite. 
>>
>> TIME-WAIT is a state caused by the side that closes the connection.
> 
> ... that closes the connection *first* (since both sides close it).

Of course.

> This point is important because it means some proxies often should
> better wait for a passive close from a server than deciding to
> close themselves.

Transparent proxies don't have that choice - they're governed by the
semantics of the connection (whether EOF == close or not).

Non-transparent proxies shouldn't be opening one connection per
transaction anyway; they ought to use one or more persistent connections
and leave them open while they are interacting with the proxy. If they
do this, there won't be an issue with who closes the connection because
the close frequency should be very low.

>> In the bulk of HTTP connections, the server closes the connection,
>> either to drop a persistent connection or to indicate "EOF" for a transfer.
> 
> Yes.
> 
>> Clients generally don't enter TIME-WAIT, so reducing the time they spend
>> in a state they don't enter has no effect.
> 
> They can if they close first and that's exactly the problem we absolutely
> want to avoid.

TW buildup has two effects:

	1) limits the number connection rate to a given IP address

	2) consumes memory space (and potentially CPU resources)

Neither is typically an issue for HCI-based clients. Servers have much
higher rate requirements for a given address when they act as a proxy
and consume more memory overall because they interact with a much larger
set of addresses.

> There are certain cases where we had to put warnings in
> rfc7230/7540, especially in relation with proxies. The typical case is
> when a client closes a connection to a proxy (eg: a CONNECT tunnel) and
> the proxy is supposed to in turn close the connection to the server. In
> this case the proxy is the connection initiator, and it can very quickly
> condemn all of its source ports by accumulating TIME_WAITs there. 

That speaks to a mismanagement of port resources. If they are allocated
on a per-IP basis, they won't run out. The error is in treating the pool
of source ports as global across all IP addresses, which TW does not
require.

> But the
> same problem exists with idle persistent connections that clients must
> avoid to close themselves if there's any hope the server will close soon.

I agree - this is the same problem if it exists -- port mismanagement.

>>> A server has no issue with knowing that
>>> a SYN belongs to a new session by seeing its ISN greater than the end
>>> of the previous window. 
>>
>> That's exactly the reason the server keeps information in the TIME-WAIT
>> state.
>>
>>> On the opposite, a client cannot know if the
>>> remote server it wants to connect to is safe for reuse 
>>
>> TIME-WAIT isn't just for new connections; it's to protect against
>> injecting traffic from previous connections that is delayed into new
>> connections...
> 
> Yes I'm well aware of this :-)
> 
>>> and will refrain
>>> from establishing a new connection during the whole TIME_WAIT state,
>>> effectively preventing itself from doing its job.
>>
>> If that's what it doesn, that's not TIME-WAIT - it's some new state in
>> the OS to avoid the possibility of hitting a TIME-WAIT at the server.
>> That's mislabeled at best, and defeats the entire purpose of the
>> TIME-WAIT at the server anyway.
> 
> No I'm not saying any such thing,

OK - glad to hear that..

> I'm saying that by all means the
> server must close first to keep the TIME_WAIT on its side and never
> on the client side. A TIME_WAIT on a server is very cheap (a few tens
> of bytes of memory at worst) 

It costs exactly the same on the client and the server when implemented
correctly.

> and can be recycled when a new valid SYN
> arrives.

The purpose of TW is to inhibit new SYNs involving the same port. When a
new SYN arrives on another port, that has no impact on existing TWs.

> A TIME_WAIT on the client is not recyclable. That's why
> TIME_WAIT is a problem for the client and not for the server.

See above; TW is *never* recyclable.

> The problem is that in some cases it's suggested that the client
> closes first and this causes such problems.

That actually helps the server (see our 99 Infocom paper).

> The only workaround for
> the client is to close with an RST by disabling lingering,

That's not what SO_LINGER does. See:
http://man7.org/linux/man-pages/man7/socket.7.html

> but that's
> really ugly and unreliable : if the RST is lost while the server is
> in LAST_ACK (and chances are that it will happen if the ACK was lost
> already), the new connection will not open until this connection
> expires.

TCP has a significant error regarding RSTs; the side that throws a RST
on an existing connection should really go into TW - for all the same
reasons that TW exists in the first place, to protect new connections
from old data still in the network.

> So by all means we must do whatever we can to avoid to cause
> TIME_WAITs to be accumulated on the client side and that was the
> point mentionned in the document since it's supposed to be used as
> a reference for future protocol designs. 

This is the error I mentioned, and it should not be recommended.

> For example in HTTP/2, the
> GOAWAY frame makes this a bit easier to take care of, since we can
> declare an intent to close that will cause the other side to close.

Receiving a GOAWAY says "don't start new connections", not "shut this
one down now". That's the right behavior, because it says nothing about
the semantics of existing connections.

> Also, there are people who face this issue and work around them using
> some OS-specific tunables which allow to blindly recycle some of these
> connections and these people don't understand the impacts of doing so.

They really ought to read the literature. It's been out there so long it
can probably apply for a driver's license by now.

> The doc will have to be clear enough to discourage them from doing so,
> and to adapt the client code instead.

We don't need a new doc to address this, especially (IMO) incorrectly.

Joe