[hybi] Framing Take VI (a compromise proposal)

[Ian Fette (イアンフェッティ) <[e-mail]>]

>
> There have been a lot of framing proposals coming out lately, that have
generated a lot of good discussion and feedback. A group of us at Google
have tried to sift through the proposals, requirements, and feedback and
tried to come up with a proposal that I'm hoping meets people's needs. No
one is ever going to be 100% happy with every last detail, but I really want
to see us agree on something that we agree is solid and move on. At some
point, we do need to decide.

The below proposal is based largely on Dave's earlier proposal, to give
credit where it's due. Takeshi Yoshino and John Tamplin are largely to
credit for the modifications, my contribution is minimal :) We've tried to
take into account what have come out of various consensus calls as well as
where it seems people aren't likely to budge. We've given in on a number of
points (e.g. multiplexing, giving up fixed 16-bit length, supporting a
64-bit length for large messages) in an attempt to make progress, so I would
also ask that people really think about what they feel strongly about and
what they can be willing to consider to support consensus.

That said, here's the proposal:

>
   0                   1                   2                   3

>    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

>   +-+-+-+-+-------+-+-------------+-------------------------------+

>   |I|F|E|R| opcode|R| Payload len |    Extended payload length    |

>   |N|I|X|S|  (4)  |S|     (7)     |             (64)              |

>   |I|N|T|V|       |V|             |     (if payload len==127)     |

>   +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +

>   |                                                               |

>   + - - - - - - - - - - - - - - - +-------------------------------+

>   |                               |                               |

>   +-------------------------------+ - - - - - - - - - - - - - - - +

>   |       Complete message length (64) (if INI==1 && FIN==0)      |

>   + - - - - - - - - - - - - - - - +-------------------------------+

>   |                               |     Ext len (8) (if EXT==1)   |

>   +-------------------------------+-------------------------------+

>   |   Extended extension length (64) (if EXT=1 && ext len == 0)   |

>   + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +

>   |                                                               |

>   +---------------------------------------------------------------+

>   |                        Extension data                         |

>   |                          (if EXT==1)                          |

>   +---------------------------------------------------------------+

>   |                       Application data                        |

>   |                                                               |

>   +---------------------------------------------------------------+

>
    ws-frame            = frame-type

>                           frame-reserved

>                           frame-payload-length

>                           frame-complete-message-length

>                           frame-extension-length

>                           extension-data

>                           application-data

>
    frame-type          = type-initial

>                           type-final

>                           type-extension

>                           type-reserved

>                           type-opcode

>
    type-initial        = %x0 ; not initial frame

>                         = %x1 ; initial frame

>
    type-final          = %x0 ; not final frame

>                         = %x1 ; final frame

>
    type-extension      = %x0 ; no extension

>                         = %x1 ; has extension

>
    type-reserved       = %x0 ; 1 bit

>
    type-opcode         = %x0 ; control frame

>                         / %x1 ; text data frame

>                         / %x2 ; binary data frame (blob)

>                         / %x3-%xB ; reserved

>                         / %xC-%xF ; user defined

>
    frame-reserved      = %x0 ; 1 bit

>
    frame-payload-length = %x00-7E

>                          / %x7F frame-extended-payload-length

>
    frame-extended-payload-length = %x000000000000007F-FFFFFFFFFFFFFFFF ;
payload length including all fields after this field

>
    frame-complete-message-length = %x0000000000000000 ; complete message
length is unknown (iff initial==1 && final==0)

>                                   / %x0000000000000001-FFFFFFFFFFFFFFFF ;
complete message length. only application data part will be counted in (iff
initial==1 && final==0)

>                                   / ; none (otherwise)

>
    frame-extension-length = ; none (iff type-extension==0)

>                            / %x01-7F

>                            / %x00 frame-extended-extension-length

>
    frame-extended-extension-length = %x000000000000007F-FFFFFFFFFFFFFFFF ;
extension length

>
    extension-data      = *( %x00-FF ) ; iff type-extension==1

>
    application-data    = text-data

>                         / binary-data

>                         / control

>
    text-data           = *( %x00-FF )

>
    binary-data         = *( %x00-FF )

>
    control-data        = control-type control-info

>
    control-type        = %x00-FF ; type of control frame

>
    control-info        = *( %x00-FF ) ; additional data of control frame

>


* terminology

> - frame : WebSocket framing unit. if chunked, each chunk forms one frame

> - message : JavaScript level data unit. JavaScript tells WebSocket boundary
between messages by some manner. may be chunked into multiple frames on wire

> - chunking : cutting single message into multiple frames

>
* requirements and how this proposal addresses them

> bandwidth/memory consumption

> - small frame size for small message

> -- 3 octet frame for 1 octet payload with no extension

> - if payload is small (<127), we have only one octet length header field

> - turn off EXT flag to have no other overhead for extension

> - if not chunked, we don't have "complete message length field" as it's
redundant

>
processing efficiency

> - data type is available on initial chunk so that we can start UTF-8
decoding immediately

> -- having a single opcode to start a fragmented message and separate
opcodes to determine if it is a text or binary message means you can't start
to decode UTF8 text until you receive the entire message, which means you
add a buffering requirement of the undecoded message

>
buffer management

> - complete message length is available on initial chunk so that we can
allocate sufficient buffer. no realloc

> -- many people objected to not having an overall message length to ease
receiver buffer management

> - Question: are endpoints likely enough to use UTF16 for internal
representation of text that it would make sense to send the number of UTF16
characters instead of bytes as the message length or as an additional field
on text frames?

> - Note that there is still the option to not provide the message length for
cases when it is not known, such as dynamically generated content or
compressed content (though it is possible we would define the message length
to be the uncompressed length in the case of compression).

> - Also, the message length is only required in the case the message is
fragmented.

>
extensibility

> - we'd include room for extension

> -- variable length (we cannot fix the size)

>
simplicity/range of length header

> - variable length header

> -- we give up the use of fixed length header

> --- there should be small sized header for small data. 16 bit, 32 bit, 64
bit, all not good

> --- 8 bit is also not good. We'll have too much overhead for chunking large
message

> - variable length length header

> -- not so complicated. just two case. 1 octet for 0 - 126 and 9 octet for
127 - 2^64-1

> -- simpler enough than variable length length header defined in -76

> -- big range

> --- we can use extended payload length to send large message as 1 frame
without any header stripe on it
---- various people objected to having to fragment messages at all if they
are already in RAM (no more copy)
-- small range

>
--- small overhead as explained in "bandwidth/memory consumption" section


> simplicity/intermediates

> - extension transparent to intermediate

> -- payload length counts in extension. intermediates that don't recognize
extension don't have to care about extension length

> - only 1 frame boundary indication scheme (length header based one. no
sentinel based one) as many people prefer

>
* extension plan

> - we included spaces for future extension

> - for new spec

> -- two bits per frame are available for future definition

> -- reserved opcodes

> - for implementors

> -- private use opcodes

> -- extension data based on extensions negotiated during the handshake