Re: [hybi] hum #3: Message

[Maciej Stachowiak <[e-mail]>]

On Aug 5, 2010, at 11:42 AM, John Tamplin wrote:

> On Thu, Aug 5, 2010 at 2:22 PM, Ian Hickson <[e-mail]> wrote:
> When transfering large blocks of data, chunking is quite inefficient. This
> means that we'd need to make sure that the fixed width field is at least
> long enough to handle large chunks of data.
> 
> At terabit Ethernet speeds, 32 bits is not enough since you could transfer
> that much in mere seconds, so presumably we'll use 64 bits.
> 
> Do we really want a length that long?
> 
> Let's assume the most recent proposal, which is a 4-byte header including a 2-byte length field.  So, if you need to send a message of 2^32 bytes, it will be sent as a 65538 frames, incurring an overhead of 262152 bytes for header overhead, or a total of 4295229448 bytes with .0006% overhead over the payload data.  It is true that this is higher overhead than a single large frame with a variable-length length field, but it hardly rises to "quite inefficient".  Also, other reasons have been given for why you don't want to send a single frame that long anyway, and for why you don't want to have to keep an arbitrarily large buffer in the sender just so you can count the bytes of the entire message before you send it.

In HTTP, it's common to send large entities with a length, even though a chunked encoding is available. In my experience, an explicit length is used much more often than chunked encoding, and I don't believe I've ever seen an HTTP message greater than 2^32 bytes in length sent with chunked transfer-encoding. Since both length up front and chunked encoding are available in HTTP, we can infer a revealed preference for explicit length for HTTP's use cases.

What reason is there to believe that WebSocket will be different? Why would we make the common and preferred way to send large messages over HTTP completely unavailable in WebSocket? That strikes me as hubris.


That being said, I think you're misunderstanding Ian's point. He expressed concern about the overhead for small messages to having a large fixed length, not the overhead to large messages of a chunked encoding.


Regards,
Maciej