Re: [hybi] NAT reset recovery? Was: Extensibility mechanisms?

[Jamie Lokier <[e-mail]>]

Vladimir Katardjiev wrote:
> 
> On 19 apr 2010, at 23.05, Pieter Hintjens wrote:
> 
> > On Mon, Apr 19, 2010 at 10:54 PM,  <[e-mail]> wrote:
> > 
> >> Hmm... Even if the protocol as a whole is asynchronous (both ends can generate frames at any time), isn't it still possible to exchange requests and responses as well over the same transport connection?
> > 
> > Here is the logic we used when designing AMQP's KA mechanism.
> > 
> > Let's say we use a req-resp model over a symmetric async connection...
> > 
> > * if the other party is sending us stuff, KA responses are redundant
> > * If we're sending stuff, KA requests are redundant
> > * Since we're symmetric, both peers would send requests when idle
> > * Thus a response is always redundant
> > 
> > And it boils down to sending NOOPs when otherwise idle, to ensure the
> > other party knows we're still alive (as a process).
> > 
> > Incidentally this simplified async KA is also much easier to
> > implement, requiring no state at the recipient.
> 
> Don't both sender and receiver need state? Each node needs to know:
> 
> - When it last sent an outgoing message (so it can send a noop when it's time)
> - When it last received an incoming message (so it knows when to treat the connection as "dead")

Yes.  But they don't need to remember if they've sent a "KA request"
or how many they have sent.

Request-response versus "async KA": Efficiency depends on timeouts
-----------------------------------------------------------------

Adding to the above: Request-response is three TCP packets (data,
data+ACK, ACK).  Mutual async keepalive is four packets in the same
time interval (data, data+ACK; data, data+ACK).

So it's more bandwidth efficient on any network when the keepalive
intervals are *symmetric*.

But you don't usually need symmetric keepalive times.  The client
often needs faster notification of lost connection than the server,
so that the client can reconnect to ensure it's up to date with
pending messages.  Whereas the server doesn't really care, except to
clean up unused connections (usually).

(That may even be true for symmetric peers once connection is
established: if it's always the same side which needs to know when to
reconnect and there is no other application state requirement.)

Let's use an example of client needing confirmation every 30 seconds
(to ensure client state is never >30 seconds out of date), and the
server needing 5 minutes (to clean up old connections).

In that situation, "async keepalive" is more efficient, because the
server sends KA messages every 30 seconds (when idle), and the client
sends a TCP ACK equally often, plus another exchange every 5 minutes -
that is more bandwidth efficient *and* more radio-power efficient than
request-response.

A smarter combination of both is for the slower KAs to be timed to
coincide with acknowledges from the faster KAs, to piggyback upon them.

Point is, there is an application-led aspect to keepalives (the 30
seconds in that example), a network-led aspect (whatever is needed for
NATs), and a WS-level aspect (whatever is needed by the server
cleaning up idle connections, 5 minutes in that example).

TCP ACK - no, sorry
-------------------

> The usage of keepalive as an ack was an interesting proposal,
> although I'm not sure if the websocket level can properly define an
> ack... It would only mean the message had been received, not
> necessarily processed, so I'm not sure what the benefit is. From a
> data integrity perspective, the interesting ack is "processing
> complete", not "delivery performed". The latter we have on some
> level via TCP already.

You can't rely on TCP ACK for this, because of unknown TCP relays and
HTTP proxies which might sit in between.

However, if you can confirm somehow that there is no TCP relay in the
way, then perhaps it can be used to be a little more bandwidth efficient.

Per-hop versus end-to-end
-------------------------

The most efficient place to put these keepalives is often per-hop (or
the shortest hops between active participants), not end-to-end - but
it does depend on what exactly is being confirmed.  For keeping NATs
open, per-hop is always good.  It lets you use fewer keepalive packets
for the same NAT timeout due to not needing to subtract end-to-end
latency, and it can be adapted with knowledge of each hop.  For
applications, per-hop isn't always suitable - it depends what semantic
the application associates with keepalive.

End-to-end adds more latency and jitter.  Depending on the various
requirements (including NAT), that may cause you to need a smaller
keepalive interval to achieve the same thing.

Connection aggregation
----------------------

For applications, sometimes keepalives can be shared over multiple
connections on the same hop (even between independent apps).  This is
because keepalive is a way of telling the app that not only is the
connection open, certain conditions have not changed since before.

For NATs, and for detecting specific connection failures, keepalives
cannot be shared.

If there are any multiplexing proxies which are aggregating
connections into fewer connections, then it is very good to defer
keepalives to the proxy (as in per-hop), so that it can use fewer
keepalives than one per original connection.  This can be quite a
significant saving.

Even if WebSocket never defines a multiplexing extension, this is a
particularly good reason to allow keepalive to be implemented per-hop,
so that a proxy using a different protocol to tunnel multiple WebSockets
can provide the keepalive function efficiently.

Radio power optimisation
------------------------

> Having said that, from a mobile perspective this is disastrous. In
> the worst-case scenario, if your keepalive interval is 5 minutes,
> you could end up sending a keepalive message every 2.5 minutes. This
> would wake up the radio twice as often, and, consequently, halve
> battery life. In a scenario where the server's keepalive is
> triggered by the client's, the message from the server would likely
> arrive within the timeframe that the radio is "awake" anyway (3-5
> seconds on smartphones) and thus avoid waking it up again.

Interesting point about the radio.  Now consider when you have 10
browser windows open on your phone (this is normal on my phone...),
each with 5 WebSocket-using applications...  The poor radio will be on
constantly if keepalives aren't synchronised.

Per-hop is a great place to put mobile radio optimisation, because you
can use a completely different mechanism.  Even a non-IP mechanism.
SMS has been mentioned; I don't know if there are other mobile network
signalling mechanisms available.

With multiple connections, if they need separate keepalives, then
there might be big power benefits to synchronising their timing, to
fit in the same power slot.

The request-response pattern has been noted as potentially good for
radio power consumption, provided it occurs fast enough to fit into a
single power slot.  In fact it's like TCP packet usage described
earlier: with symmetric timeouts, request-response is good for radio
power too.  With highly asymmetric timeouts, "async keepalive" is
better for radio power too.

Summary
-------

Keepalive optimisation is a fairly complicated area in general.  What
is good for one application can be expensive for another.  Their
aggregation and timing synchronisation are also potentially significant.

We should avoid over-designing it in WebSocket - we'd get it wrong
anyway - and we should avoid having a mandatory, over-simplistic
keepalive type because there are so many useful variations.

Unfortunately some aspects of efficient keepalive are impossible for
applications to implement on top of WebSocket.  For example
synchronisation among different applications to minimise radio power.

So, imho, we should make sure there is a place to hook in different
mechanisms as they are needed, by each application and link type, in a
future-extensible sort of way.

That probably means defining some API events (like heard-from or
not-heard-from), a generic way to request keepalive parameters
(strings & values go a long way!), and a place in the protocol for
transport-level negotiation of them, including per-hop.

-- Jamie