Re: [hybi] Reliable message delivery (was Re: Technical feedback.)

[Jamie Lokier <[e-mail]>]

Maciej Stachowiak wrote:
> > On Fri, Jan 29, 2010 at 11:25 PM, Maciej Stachowiak <[e-mail]> wrote:
> >>> It depends upon what level of "reliability" you are looking for.  If
> >>> you are aiming for the "common" case, the answer to both is "yes".
> >>> 
> >>> However, edge cases make the answer "no" - it is quite possible to
> >>> have "lost" responses that a server actually sends, but the client
> >>> will never see.
> >> 
> >> So you could lose messages, but can you tell in this case that you are not guaranteed yet that they have been delivered?
> > 
> > No, not really - the client simply thinks the server close()'d the
> > connection but it has no way of knowing there were other data packets
> > that the server really meant for the client to see.  Correspondingly,
> > the server did everything in the right order - it wrote all the data
> > it expected and then it close()'d the socket.  Yet...oops.
> 
> Presumably the server could know that at least all the packets ACK'd
> at the TCP level have been successfully delivered, right? So I
> assume the only problem is the remaining packets after that, if you
> don't do a lingering close.

No.

The server sends a TCP RST when it recieves data after it has called
close().

That TCP RST causes the client to *discard* data is has previously
received and ACK'd at the TCP level.  The client application does not
see that data, if it hasn't already read it from the OS.

The client should get a socket error, but that's not very useful.
Depending on how the client is used, the practical effect is sometimes
a truncated message, or missing messages.

This is why Apache must implement a rather complicated "lingering
close", which uses shutdown(SHUT_WR) instead of close(), and then
reads and discards any further data recieved from the HTTP client.

HTTP servers (and proxies) which don't do this are prone to unreliable
response delivery if the client sends any more data, such as a
pipelined request.  It only happens under some network and load
conditions, and with some clients, and some configurations, which is
why there are a lot of implementations that get it wrong.

Some applications piggybacked on WebSocket look likely to get it wrong
and suffer this problem in corner cases.

> I would like to understand the lingering close issue better. Does it consist of waiting for TCP ACKs for all your packets before closing the TCP connection?

No, it consists of calling shutdown(SHUT_WR) immediately, and then
reading and discarding whatever the client sends until you recieve a
client-side close (read() returns EOF), or you think you have waited
long enough for the client application to have read the response
(e.g. 2 minutes).

Due to the time heuristic, and the client's internal delays, there is
no guarantee that the client application will actually have received
the response, but it is ok in practice with normal HTTP clients.

> I think you can do better than just orderly close. Either from
> TCP-level acks or from WebSocket-protocol-level acks, you could tell
> that some number of your messages have definitely been delivered,
> even in the face of a service interruption. Right?

I agree.
 
> Maybe I'm thinking of reliable message delivery differently than
> you, but I assumed a major goal would be to know what might need to
> be retransmitted even if there is an unexpected disconnect.

Yes, that is a major one, because you often want automatic
retransmission when possible.  (See: HTTP pipelining problems).

Unexpected disconnected can happen for many reasons, including the
network itself which endpoints have no control over.  E.g. NAT router
resets (happens daily on one network I'm aware of).

Orderly close does not help with network-level disconnects, so other
techniques like duplicate elimination are valuable too.

-- Jamie