Re: [hybi] Framing take IV

[Ian Fette (イアンフェッティ) <[e-mail]>]

On Tue, Aug 3, 2010 at 5:38 PM, Greg Wilkins <[e-mail]> wrote:

>
> I'd like to refocus on what I believe we do have near-consensus on, so that
> we don't lose progress while we debate other possible features.
>
>
> I think that we have reasonable consensus on something like:
>
>   +--------------------------------------------------+
>   | frag(1) |unused(3) | opcode(4) |  Length(16)     |
>   +--------------------------------------------------+
>    |                      Data                        |
>   +--------------------------------------------------+
>
>     ws-frame            = frame-fragment
>                           frame-unused
>                           frame-opcode
>                           frame-length
>                           octet-data
>
>     frame-fragment      = %x0 ; 1 bit not a fragment
>                         | %x1 ; 1 bit fragment
>
>     frame-unused        = %x0 ; 3 bits
>
>     frame-opcode        = %x0 ; text data frame
>                         | %x1 ; binary data frame
>                         | %x2 ; ping
>                         | %x3 ; pong
>                         | %xf ; final frame
>
>     frame-length        = %x0000-FFFF
>
>     octet-data          = *( %x00-FF )
>
>
> I think there are no strong feelings either way on padding, but it is easy
> to add to any proposal, so let's sort the detail first and then consider
> padding at the very end.
>
> I do not think we have any consensus on compression, but the above framing
> can be easily extended at a later date (compressed opcodes, or add
> compressed bit, or just compress the stream).
>
> We are still a bit in the wilderness with regards to meta-data and
> multi-plexing.  However the above framing is at least extensible and would
> allow those and other extensions to be added by defining some new
> op-codes.
> But the problems with this style of extensibility are:
>
>    - The op-code space must be managed to avoid collisions, so that would
>    limit experimental and/or special purpose extensions
>    - Any extension data will be in it's own frame, and thus not directly
>    associated with a data frame, so extensions like multiplexing will be more
>    stateful and vulnerable to inserted frames.
>
> Neither of these problems are show stoppers, although they do combine
> badly.. ie multiplexing can be done with a channel switch and flow control
> frames, but only if op-codes are allocated to it and that will need
> centralized consensus on one true multiplexing algorithm.
>
>
>
> We have had many proposals about how to improve the extensibility of the
> framing, but so far I think only the proposal from roberto truly addresses
> the issue of non-centralized extensibility and extensions per frame.  His
> proposal was to optionally include meta-data with every frame - I refined
> that proposal to make the meta data easily ignorable:
>
>   +--------------------------------------------------------+
>   | frag(1) |opcode(3) | meta-length (12) |  Length(16)    |
>   +--------------------------------------------------------+
>   |                       Meta Data                        |
>   +--------------------------------------------------------+
>    |                          Data                          |
>   +--------------------------------------------------------+
>
>     ws-frame            = frame-fragment
>                           frame-opcode
>                           frame-meta-length
>                           frame-length
>                           +(meta-data)
>                           octet-data
>
>     frame-fragment      = %x0 ; not a fragment
>                         | %x1 ; fragment
>
>     frame-opcode        = %x0 ; text data frame
>                         | %x1 ; binary data frame
>                         | %x2 ; ping
>                         | %x3 ; pong
>                         | %xf ; final frame
>
>     frame-length        = %x0000-FFFF
>
>     meta-data           = ; length framed name value pairs
>
>     octet-data          = *( %x00-FF )
>
>
>
>
>
> The other proposal from Martin was that we add a  MIME frame op code:
>
>   +--------------------------------------------------+
>   | frag(1) |unused(3) | opcode(4) |  Length(16)     |
>   +--------------------------------------------------+
>    |                      Data                        |
>   +--------------------------------------------------+
>
>     ws-frame            = frame-fragment
>                           frame-unused
>                           frame-opcode
>                           frame-length
>                           octet-data
>
>     frame-fragment      = %x0 ; 1 bit not a fragment
>                         | %x1 ; 1 bit fragment
>
>     frame-unused        = %x0 ; 3 bits
>
>     frame-opcode        = %x0 ; text data frame
>                         | %x1 ; binary data frame
>                         | %x2 ; MIME data frame
>                         | %x4 ; ping
>                         | %x5 ; pong
>                         | %xf ; final frame
>
>     frame-length        = %x0000-FFFF
>
>     octet-data          = *( %x00-FF )
>
>
> I think Martin proposed it as standard CRLF separated Mime, but I think an
> argument could be made to put a length based encoding within that frame.
> This essentially would make the meta-data of robertos proposal available
> only for a single frame type.  An extension like multiplexing would be
> implemented by converting all binary/text frames to mime-frames.  (I'm not
> exactly sure mime is the right name for them, but let's go with it for now).
>
> Mime frames would allow arbitrary extensions to exchange arbitrary
> meta-data without centralized control.  It is a matter for either end
> how(if?) they would route any such messages to their APIs.
>
> There are many variations on these themes, but I think these are the 3
> basic approaches that have been identified, which I will call:
>
>    1. hixiesque - centralized extension via op-codes
>    2. robertoesque- per frame extension via meta-data
>    3. martinesque - extension via dedicated mime frame (meta-data plus
>    data)
>
> Any others?
>
> I think we really need to pick one of these approaches soon and get on with
> the detail.
>
> My own opinion is that my technical head is drawn to robertoesque solution,
> but my pragmatic head feels that martinesque is sufficient, simpler and more
> likely to get consensus.
>
>
> cheers
>
>
> PS. with a martinesque solution, who would ever use a %x1 binary data
> frame?
>
>
If I had to choose between Roberto's and Martin's, I would prefer Roberto's
simply because I worry that if you have metadata that changes between
messages, e.g. you are multiplexing a number of streams, you will end up
sending a control frame before each data frame, which sort of defeats the
purpose, makes it extra chatty, and opens us up to issues of frame injection
as well. So, I prefer to have whatever extension/meta-data associated with
the frame itself.

The one thing I don't like about Roberto's proposal is that I essentially
have to parse all the metadata to find the particular metadata I'm
interested in. I'm not sure if there are any ways to avoid this or make this
cheaper though if you have to support variable-length metadata, so maybe
it's just a price we pay.

I do think it would make a lot of sense to reserve a byte for stream ID
though. I know that for us (Google) to deploy, one way or another we will
need multiplexing, and I think it would be a lot more efficient if the
stream ID were at a known location than if we had to parse some metadata map
to get it out. I don't really buy the complexity arguments -- for us, the
main interest in WebSockets over JS hanging gets + posts is the ability to
pear down connections (two to one, better if we can multiplex) + more
efficiency. If we can't multiplex then there's not that much of a benefit to
the point where it becomes a lot less interesting to deploy (cost/benefit is
not so clear).