Re: [hybi] WS ABNF

[Scott Ferguson <[e-mail]>]

Dave Cridland wrote:
> On Wed Feb 17 02:53:38 2010, Jamie Lokier wrote:
>> Then one day, someone else puts an 8GB file there and expects it to
>> work.  Why should it be fine for 3GB files but fail for an 8GB file?
>
> Because websocket messages would (then) be limited to 4G, just as IMAP 
> limits emails to 4G (yet the rest of the mailsystem doesn't). I'm not 
> aware of this *ever* causing a problem. I'm not aware of - amongst the 
> vast numbers of people who seem to enjoy picking holes in IMAP - 
> anyone complaining about that limit even theoretically, or complaining 
> that the literal octet count is represented in ASCII.
Chunked/fragment encoding is needed anyway because you don't want to 
force the sender to buffer an arbitrarily large message before sending 
it. And once you have chunking/fragmenting, the size of the fragment is 
irrelevant to the size of the full message.
>> Maybe for generality (no fixed limit), but it might have been an
>> attempt to use fewer bytes for small messages.  Considering most
>> messages are expected to be small(*), that's not entirely illogical.
>
> Not on the face of it, no. A naïve look would see that a whole 1-3 
> octets are shaved off the wire, and that can't be a bad thing in and 
> of itself.
>
> The trouble is that it's vanishingly rarely saving any TCP packets, 
> which would be interesting and useful, and nor is it saving any 
> messages, which is essentially beyond the scope of the protocol. Those 
> two *do* save appreciable amounts of bandwidth.
This isn't complicated.  Just use a chunk/fragment encoding that looks like

  len ::= [\x80-\xff]{0,3}[\x00-\x7f]

The 0-terminating chunk takes up a single byte, small messages (< 128) 
only need two bytes of overhead, chunk lengths are limited to 32-bits 
(actually 28-bits but it doesn't matter), the total message size is 
arbitrarily large, and it's trivial to implement.

Or something like utf-8 if you want the first byte to specify the number 
of bytes:

  len0 ::= [\x00-\x7f]
  len1 ::= [\x80-\x9f] [\x00-\xff]
  len2 ::= [\xa0-\xbf] [\x00-\xff]{2}
  len3 ::= [\xc0-\xdf] [\x00-\xff]{3}

Or just use a fixed 2-byte length prefix.

If it's important to add message types (and that would be cleaner as a 
different layer), that's just a single extra byte of overhead.

-- Scott
>
> Dave.