Re: [hybi] Framing Take VI (a compromise proposal)

[Greg Wilkins <[e-mail]>]

Ian,

thanks for the proposal.

It appears that the total message length is mandatory if fragmentation
is used, however I believe that one of the use-cases for fragmentation
is to send dynamic messages where the total length is not known in
advance.   Could we make message length optional somehow?

Alternately, message length (in bytes and/or in characters etc.) is
just one example of meta-data.  Would it not be possible to support
that in a standardized meta data extension?

If we are to have per frame extension data, then I think having the
extension length in the header so simple implementations can skip is a
good approach.  However, I think we probably would also need to goto
extra detail about how data from multiple extensions would be
separated within the extension data.

However, I remain a little concerned that having both op-code and per
frame extension data could result in some contradictory data.  For
example, if we have an op-code for final frame, then it is likely that
intermediaries will know that frame and possibly use it as part of
shutting down the connection (with half closes etc.).  If we then have
an extension that provides multiplexing and allows a channel ID to be
associated with every frame, what does it mean to have a final frame
op code extended with a channel ID?  The obvious interpretation for
the end points is that is the final frame for just that channel, but
intermediaries would not know that and could shutdown the whole
connection.

Thus until such potential contradictions can be worked out, I think
I'm still in favour of an op-code that indicates that the frame
contains extended data plus data.  This could still be used for
something like multiplexing as a total replacement for plain
text/binary frame opcodes. I'll put together a more detailed proposal
for this shortly.

cheers









2010/8/13 Ian Fette (イアンフェッティ) <[e-mail]>:
> There have been a lot of framing proposals coming out lately, that have
> generated a lot of good discussion and feedback. A group of us at Google
> have tried to sift through the proposals, requirements, and feedback and
> tried to come up with a proposal that I'm hoping meets people's needs. No
> one is ever going to be 100% happy with every last detail, but I really want
> to see us agree on something that we agree is solid and move on. At some
> point, we do need to decide.
> The below proposal is based largely on Dave's earlier proposal, to give
> credit where it's due. Takeshi Yoshino and John Tamplin are largely to
> credit for the modifications, my contribution is minimal :) We've tried to
> take into account what have come out of various consensus calls as well as
> where it seems people aren't likely to budge. We've given in on a number of
> points (e.g. multiplexing, giving up fixed 16-bit length, supporting a
> 64-bit length for large messages) in an attempt to make progress, so I would
> also ask that people really think about what they feel strongly about and
> what they can be willing to consider to support consensus.
> That said, here's the proposal:
>
>    0                   1                   2                   3
>
>    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>
>   +-+-+-+-+-------+-+-------------+-------------------------------+
>
>   |I|F|E|R| opcode|R| Payload len |    Extended payload length    |
>
>   |N|I|X|S|  (4)  |S|     (7)     |             (64)              |
>
>   |I|N|T|V|       |V|             |     (if payload len==127)     |
>
>   +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
>
>   |                                                               |
>
>   + - - - - - - - - - - - - - - - +-------------------------------+
>
>   |                               |                               |
>
>   +-------------------------------+ - - - - - - - - - - - - - - - +
>
>   |       Complete message length (64) (if INI==1 && FIN==0)      |
>
>   + - - - - - - - - - - - - - - - +-------------------------------+
>
>   |                               |     Ext len (8) (if EXT==1)   |
>
>   +-------------------------------+-------------------------------+
>
>   |   Extended extension length (64) (if EXT=1 && ext len == 0)   |
>
>   + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
>
>   |                                                               |
>
>   +---------------------------------------------------------------+
>
>   |                        Extension data                         |
>
>   |                          (if EXT==1)                          |
>
>   +---------------------------------------------------------------+
>
>   |                       Application data                        |
>
>   |                                                               |
>
>   +---------------------------------------------------------------+
>
>     ws-frame            = frame-type
>
>                           frame-reserved
>
>                           frame-payload-length
>
>                           frame-complete-message-length
>
>                           frame-extension-length
>
>                           extension-data
>
>                           application-data
>
>     frame-type          = type-initial
>
>                           type-final
>
>                           type-extension
>
>                           type-reserved
>
>                           type-opcode
>
>     type-initial        = %x0 ; not initial frame
>
>                         = %x1 ; initial frame
>
>     type-final          = %x0 ; not final frame
>
>                         = %x1 ; final frame
>
>     type-extension      = %x0 ; no extension
>
>                         = %x1 ; has extension
>
>     type-reserved       = %x0 ; 1 bit
>
>     type-opcode         = %x0 ; control frame
>
>                         / %x1 ; text data frame
>
>                         / %x2 ; binary data frame (blob)
>
>                         / %x3-%xB ; reserved
>
>                         / %xC-%xF ; user defined
>
>     frame-reserved      = %x0 ; 1 bit
>
>     frame-payload-length = %x00-7E
>
>                          / %x7F frame-extended-payload-length
>
>     frame-extended-payload-length = %x000000000000007F-FFFFFFFFFFFFFFFF ;
> payload length including all fields after this field
>
>     frame-complete-message-length = %x0000000000000000 ; complete message
> length is unknown (iff initial==1 && final==0)
>
>                                   / %x0000000000000001-FFFFFFFFFFFFFFFF ;
> complete message length. only application data part will be counted in (iff
> initial==1 && final==0)
>
>                                   / ; none (otherwise)
>
>     frame-extension-length = ; none (iff type-extension==0)
>
>                            / %x01-7F
>
>                            / %x00 frame-extended-extension-length
>
>     frame-extended-extension-length = %x000000000000007F-FFFFFFFFFFFFFFFF ;
> extension length
>
>     extension-data      = *( %x00-FF ) ; iff type-extension==1
>
>     application-data    = text-data
>
>                         / binary-data
>
>                         / control
>
>     text-data           = *( %x00-FF )
>
>     binary-data         = *( %x00-FF )
>
>     control-data        = control-type control-info
>
>     control-type        = %x00-FF ; type of control frame
>
>     control-info        = *( %x00-FF ) ; additional data of control frame
>
>
>
> * terminology
>
> - frame : WebSocket framing unit. if chunked, each chunk forms one frame
>
> - message : JavaScript level data unit. JavaScript tells WebSocket boundary
> between messages by some manner. may be chunked into multiple frames on wire
>
> - chunking : cutting single message into multiple frames
>
> * requirements and how this proposal addresses them
>
> bandwidth/memory consumption
>
> - small frame size for small message
>
> -- 3 octet frame for 1 octet payload with no extension
>
> - if payload is small (<127), we have only one octet length header field
>
> - turn off EXT flag to have no other overhead for extension
>
> - if not chunked, we don't have "complete message length field" as it's
> redundant
>
> processing efficiency
>
> - data type is available on initial chunk so that we can start UTF-8
> decoding immediately
>
> -- having a single opcode to start a fragmented message and separate opcodes
> to determine if it is a text or binary message means you can't start to
> decode UTF8 text until you receive the entire message, which means you add a
> buffering requirement of the undecoded message
>
> buffer management
>
> - complete message length is available on initial chunk so that we can
> allocate sufficient buffer. no realloc
>
> -- many people objected to not having an overall message length to ease
> receiver buffer management
>
> - Question: are endpoints likely enough to use UTF16 for internal
> representation of text that it would make sense to send the number of UTF16
> characters instead of bytes as the message length or as an additional field
> on text frames?
>
> - Note that there is still the option to not provide the message length for
> cases when it is not known, such as dynamically generated content or
> compressed content (though it is possible we would define the message length
> to be the uncompressed length in the case of compression).
>
> - Also, the message length is only required in the case the message is
> fragmented.
>
> extensibility
>
> - we'd include room for extension
>
> -- variable length (we cannot fix the size)
>
> simplicity/range of length header
>
> - variable length header
>
> -- we give up the use of fixed length header
>
> --- there should be small sized header for small data. 16 bit, 32 bit, 64
> bit, all not good
>
> --- 8 bit is also not good. We'll have too much overhead for chunking large
> message
>
> - variable length length header
>
> -- not so complicated. just two case. 1 octet for 0 - 126 and 9 octet for
> 127 - 2^64-1
>
> -- simpler enough than variable length length header defined in -76
>
> -- big range
>
> --- we can use extended payload length to send large message as 1 frame
> without any header stripe on it
> ---- various people objected to having to fragment messages at all if they
> are already in RAM (no more copy)
> -- small range
>
> --- small overhead as explained in "bandwidth/memory consumption" section
>
>
> simplicity/intermediates
>
> - extension transparent to intermediate
>
> -- payload length counts in extension. intermediates that don't recognize
> extension don't have to care about extension length
>
> - only 1 frame boundary indication scheme (length header based one. no
> sentinel based one) as many people prefer
>
> * extension plan
>
> - we included spaces for future extension
>
> - for new spec
>
> -- two bits per frame are available for future definition
>
> -- reserved opcodes
>
> - for implementors
>
> -- private use opcodes
>
> -- extension data based on extensions negotiated during the handshake
>
> _______________________________________________
> hybi mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/hybi
>
>