Re: [hybi] thewebsocketprotocol #28 (new): Fragmentation

[Jamie Lokier <[e-mail]>]

Zhong Yu wrote:
> On Wed, Nov 24, 2010 at 4:40 AM, Jamie Lokier <[e-mail]> wrote:
> > Zhong Yu wrote:
> >> The most important thing I think is the abstraction WS provides to app
> >> developers. Right now the abstraction is simple -- one message after
> >> another. The simplicity is a good thing, if it is compromised I don't
> >> know what websocket is supposed to be any more.
> >
> > Right now, WS provides *two* abstractions: The one to app developers,
> > and the one over TCP.
> >
> > Some of us believe they *should not be the same* due to network
> > engineering issues *because that makes the app developer's job easier*.
> >
> > Right now, the "simplicity" of WS means *robust* apps need to
> > implement half of TCP in Javascript on the client, which is not a good
> 
> That's where frameworks come in, which can provide degrees of
> robustness suitable for different applications. I don't think WS
> should or could address these application level concerns.

Certain concerns *cannot* be addressed at the Javascript level, or
they are hard to do.  Some of them are technical (connection/keepalive
aggregation, network error indication); some of them are social
because authors are motivated by self-interest, not aggregate
behaviour.  Frameworks don't make independently-written components on
the same page/server share connections/keepalives, and even *using* a
single framework, it's not clear if it's technically able to share
connections to the same server.

Ironically this makes HTTP way more efficient than WebSocket for some
kinds of small messaging when multiple pages are open, yet less
efficient for a single page and few components.  You can guess which
will be optimised for, by most authors.  The effect is not insignificant.

> The #1 feature needed for robustness is auto reconnect. Nobody
> proposed to build that feature in the WS protocol.

For a very good reason: No matter how much the system tries to auto
reconnect, at some point the application *must* be able to handle the
fact that connection is lost and can't be restored.  They must also
decide what to send on reconnection; queuing everything and resending
it all would be disastrous for many apps, after a long delay.

Since applications must handle that anyway (and it's easy to script),
there is little point *for the app* to have WS doing it automatically.

(There is a network reason though: Local congestion collapse, and
router NAT overflow, due to many pages/components repeatedly retrying
connections without backoff...)

If anything, #1 is a way to *detect when* a connection is dead and a
reconnect advisable.

> > sign.  (For example the "tic tac toe" example, which is supposed to be
> > simple, can lock up on real networks: No more moves when a TCP is
> > silently dropped.  A HTTP equivalent doesn't have this problem.)
> >
> > Why do you think the two abstractions should be the same?
> 
> They should not be. But no matter how complex the protocol must
> become, I wish it won't jeopardize the simple application abstraction.
> I don't want WebSocket to become WebService, and everybody needs to
> buy a 1000 page "The Definitive Guide".

I'm pretty sure everyone in the design discussions agrees on keeping
the application abstraction simple.

I'm dismayed that attempts to make the application abstraction *simpler*
by making the network layer work a bit harder are repeatedly
misunderstood as the opposite.

A glaringly obvious example is the way, as you've pointed out, that
the only robust way to use *current* WebSocket is with a "framework".
Many of the protocol proposals make it robust, and scale better, while
using the WebSocket object directly from scripts - without changing
the application abstraction *at all*.

> > That said, I'm not sure why we need to support messages with unknown
> > lengths in advance in the simplest protocol level anyway.  Is there a
> > use case for control frames such as graceful close or error indication
> > in the middle of a message?
> >
> > The use cases which come to my mind are: Keepalives - those are
> > desirable in th middle of a large message, if the message is being
> > generated slowly;
> 
> If it is so slow and it exceeds the timeout, who are we to say that
> the connection is "alive"? The ping-pong test certainly says nothing
> about application liveliness. I don't see the point.

It's not to detect application aliveness!!  That's a different
problem, which needs the app to explicitly participate.

You need regular keepalives to keep TCP connected through a NAT.  The
interval needed for that varies (a lot), but I have seen NATs drop
connections after as little as 30 seconds TCP inactivity, including
delays from TCP's own timers.

The problem of NAT is only going to get worse.

I see nothing wrong with applications caring about their own behaviour
(such as liveness / app reconnect/resync logic), while the silly NAT
behaviour is something they shouldn't have to deal with.

The simplest abstraction is probably a pipe that stays open as long as
the network can sustain it smoothly (i.e. without long interruptions
and unsustainable queueing).  Whether the pipe carries
messages/text/bytes is less important.

That implies NAT keepalives, that can always be sent, even in the
middle of a long message if they can be produced incrementally.

-- Jamie