Re: [hybi] Frame size

[Jamie Lokier <[e-mail]>]

Greg Wilkins wrote:
> Roberto Peon wrote:
> > 
> > 
> > On Fri, Apr 16, 2010 at 5:38 AM, Greg Wilkins <[e-mail]
> > <mailto:[e-mail]>> wrote:
> > 
> > 
> > 
> >     I thought one of the key arguments for a limited frame size,
> >     is that we could avoid the need to consider end points that
> >     could not store an entire frame in a buffer at once.
> >  
> > Why would you need to do that?
> 
> At the 77 meeting, I raised the point that currently error
> handling in the protocol is not sufficient to deal with
> implementations that impose a maximum frame size.  The spec
> currently says the end point should just close the socket
> if it receives a frame that is too large.
> 
> One option is to negotiate maximum frame size, but that
> appears to go against the no negotiation style of this
> protocol.

The current protocol does have negotation, between the application
endpoints.  They can choose not to send messages larger than what they
agreed, implicitly or explicitly.

The current protocol does not have frames and transport-level
fragmentation, so I'm not sure how the negotation in the current
protocol is relevant :-)

This is an argument for transport-level negotiation, even if just at
the beginning, independent of application negotiation.

> The suggestion was then made that we pick a small fixed
> maximum frame size, that all implementations would be
> required to handle.

Although maximum frame size may be an issue, setting a maximum (or
negotiating one) really doesn't help: Messages can be of arbitrary
length, and are assembled at the receiver from fragments by simply
appending the fragment data.

I can't imaging an implementation benefiting significantly from a
fragment size limit but no message size limit.  After all, it can just
append the bytes to the message as it receives them.

Now a maximum *message* size might be interesting.

That has the same problem at the next level: The WebSocket API,
anyway, requires received messages to be *queued* as events to be
processed.

So even if you limit the message size, you can receive any number of
messages into the queue and overflow memory that way.

It is not made clear, but I presume that web browsers in this
condition will either abort the connection or stop reading and thereby
use TCP flow control to block the sender.

That's quite an unfortunate condition to get into, because it breaks
"graceful close" and other kinds of control messages.

So even limiting fragment size and message size does not prevent
awkward "too much" conditions from occurring.

(I'll obliquely mention flow control as a possible answer...)

> Large messages would then be fragmented and assembled
> from 1 or more frames.    This does not solve the problem
> of end points that cannot handle large message, but it
> does make the problem not a transport problem.  Higher
> layers are better able to handle large messages, as
> they can stream to disk, reject with errors, parse and
> handle as a stream, etc etc.

I am confused: Since assembly of messages from frames is a transport
problem, what part of transport does limiting just the frame size (and
not the message size) solve?

The only component not participating in fragmentation and assembly may
be intermediaries, but there is no benefit to them *always* buffering
entire frames before forwarding; that would only add unwanted latency.

> The alternative approach is to not have fragmentation,
> but to allow the max  frame size be of arbitrary (or very large)
> length so it can always handle a message or arbitrary (or very
> large) length.  Then have reasonable error handling to cope with
> too large messages/frames.

Don't we need the error handling for too large messages anyway?

> The problem with this approach, is the sending of a too large
> to handle frame, breaks the ability to communicate bidirectionally.
> 
> Imagine if both client and server both start sending too large
> to handle frames at the same time.   Neither will be able to
> send too-large errors because they are sending large frames
> and to their knowledge the other side is handling them.
> 
> Fragmentation of large messages into frames gives an
> endpoint the ability to terminate the transmission of
> a too-large-message or even to interleave control frames
> containing errors.

That should really be about all types of error and also non-error
"abort this message" states, which includes graceful close.  Not just
too-large errors.

If end A sends a graceful close, and end B has just started sending a
large frame, end B cannot confirm the graceful close until it finishes
sending the frame.  That's not too bad because B still sees the
graceful close request and knows that the frame it is sending will be
(or should be!) ignored...

On that note, there should be a frame type which signals "the message
this is part of is aborted / has an error", quite distinct from
"message end".

An alternative encoding: Message end doesn't necessarily have to use
one of the length bits.  It's enough to say that a special frame type,
or a length field of 0xffff (for example) is used to mean "end of
message".  Since there ought to be a way to signal "abort message"
anyway, that'd be quite natural.  (HTTP chunking is like this).

> That is why my suggestion is to go for a 16k max frame size
> that all implementation would have to be able to handle.

16kb may seem like a good compromise now, but even today that's small
on some networks.  The lifetime of this protocol is the next 10-20
years: We can expect 10Tbit/s networks to be common by the end of
that.  A 16kb message will take just 10ns to transmit.  And then there
are virtual machines: Increasingly, browser applications talk to
corporate servers over virtual networks (no physical link), which are
very fast and like large packets.

Since I don't see any benefit in limiting the frame size for buffering
purposes (endpoint receivers can just append the bytes to the message
however it's received, and intermediaries shouldn't buffer whole large
frames), and there are performance and even memory-saving
optimisations (Google sendfile) to transmitting larger ones, I think
we shouldn't limit the future in that way.

A good place for frame headers for large messages is one per TCP/IP
packet, at the beginning.  Only sophisticated transmitters are likely
to be able to place them in that way, but if they can, why not.

Since the maximum TCP segment size is just under 64k, 64k works for that.

(I think we shouldn't design the protocol to only work well with
TCP/IP either.  Someone, somewhere will shoehorn it over a fast pipe
with bigger packets than IP.  But that's ok as long as framing is a
transport-level concern and fragmentation/reassembly are allowed at
each hop: non-TCP/IP components can modify the framing scheme.)

Anyway, premature optimisation and all.  We should aim to get the
broad principles and message/frames types right rather than worry
about the details of exactly how many bits go where and how the bytes
are laid out... before deciding those other things.

-- Jamie