Re: [hybi] hum #3: Message

[Pieter Hintjens <[e-mail]>]

On Thu, Aug 5, 2010 at 10:21 PM, Maciej Stachowiak <[e-mail]> wrote:

> Sending a file from disk to the network without processing it in any way is more efficient than cutting it into slices and interleaving with headers. The IP layer is already going to cut it into slices, in a way that almost certainly won't line up with WebSocket-level fragment boundaries.
>
> It's also more efficient for the receiver if they know the length up front. That way they can determine whether to allocate a large enough buffer, or just start spooling to persistent storage immediately.

Agree with both these points.  If a sender does fragment a large
message, they ALSO need to specify the full size up front, for the
same reasons we specify the frame size up front.  You want to avoid
sentinel-based reads.

Content framing sounds simple but is actually one of those things that
gets complex to implement.

There is a very simple solution that addresses both efficiency for
small messages, and easy of framing for very large messages.

Rather than a fixed two-byte length, use a 1-byte length which escapes
into an 8-byte length for frames longer than 254 bytes.  Small
messages, which are the most important for high latency and high
frequency scenarios, are optimally encoded.  Long messages are most
trivially encoded.

If you really want a 4-byte header, use a two-byte length that escapes
into an 8-byte length.

I'd strongly advise, based on experience, to avoid frame fragmentation
as a way of encoding long frames.  Multiple frames per message IS
valuable but for other reasons (multipart).

-Pieter