[hybi] how do we move forward on agreeing on framing?

[John Tamplin <[e-mail]>]

There have been various discussions about various proposals, and to be sure
there will be no proposal that everyone is happy with.  What can we do to
come to some consensus on something that is acceptable for the base protocol
so we can move forward?

To me, I think the following things are critical:

   - fragmentation support - this has been accepted as a requirement without
   much dissension when it was declared that we achieved consensus on it, and
   if we don't have support for it in the base protocol it will greatly
   complicate efforts for certain extensions a number of people think will be
   important in the future, such as multiplexing
   - low overhead for small frames - as previously indicated, real-world
   WebSocket apps do make use of a substantial percentage of small messages,
   and when compression is added the percentage grows to an overwhelming
   majority.  Sure, there will be some WebSocket apps which won't get any
   benefit from small frames, but that shouldn't remove the capability from
   those that do.
   - we can't afford to decide on the extension format now - given the rate
   of building consensus, we won't ever finish the base protocol if we have to
   define that now.  That means anything that is critical to support in the
   base protocol, such as fragmentation or (if we decide it is important for
   implementations of the base protocol) the total message length, cannot be
   defined in an extension.
   - we need to leave our options open for extensibility - since we don't
   yet agree on exactly how the protocol should be extended or exactly what
   facilities such extensions may need, we need to have some flexibility for
   how those extensions can be represented

Does anyone disagree with these points?

So here is another attempt at incorporating feedback to try and get
something which could gain consensus as the framing protocol:

   - MORE - 1 bit - if set, this is not the final frame of the message
   - TML - 1 bit - if set, the total message length is included (which is
   not required to be on the first frame, and if repeated overrides earlier
   values so it may be used as an estimate for dynamic content)
   - Reserved - 2 bits - must be 0 except for private use
   - Opcode - 4 bits - 0=Continuation, 1=Control, 2=Text, 3=Binary,
   4-11=Reserved for future assignment, 12-15=Reserved for private use
   - Reserved - 1 bit - must be 0
   - Length - 7 bits - payload length in octets; if 127, then there is an
   extended length
   - Extended length - 8 bytes - payload length in octets (high bit must be
   0 for ease of use with languages without an unsigned data type), present
   only if Length=127
   - Total Message Length - 8 bytes - total message length in octets (high
   bit must be 0 as above), present only if TML=1
   - Payload - (Extended) Length bytes - payload, interpreted as defined by
   the Opcode

This addresses the concern about requiring the total message length to be in
the first frame or to be present at all if it is unknown.  It allows us to
use a bit to indicate extension data is present, or to define extensions by
new opcodes (and to grow the opcode field if necessary by using adjacent
reserve bits).  Some bits are available for direct assignment, such as if we
find frame-by-frame compression a useful future extension. It also cleans up
room for confusion in the previous proposal where you could change opcodes
between fragments.

Example messages:

   - MORE=0,TML=0,Opcode=Text, Length=12, Payload="Hello world!" -
   single-frame text message



   - MORE=1,TML=1,Opcode=Text, Length=5, MsgLength = 12, Payload="Hello" -
   first frame of fragmented message
   - MORE=0,TML=0,Opcode=Continuation, Length=7, Payload=" world!" - final
   frame of fragmented message



   - MORE=1,TML=1,Opcode=Text, Length=127, ExtLength=4096, MsgLength=16000,
   Payload=... - first frame of dynamically generated message, with estimated
   message length
   - MORE=1,TML=0,Opcode=Continuation, Length=127,
   ExtLength=4096,Payload=... - several frames delivering dynamic content
   - MORE=1,TML=1,Opcode=Continuation, Length=127, ExtLength=4096,
   MsgLength=14781, Payload=... - sender now knows length of dynamic content
   and sends it
   - MORE=0,TML=0,Opcode=Continuation, Length=127, ExtLength=2493,
   Payload=... - final frame of message

Extension Possibilities:

   - if we decide creating new opcodes for extensions, which then nest the
   next layer of extension within their payload data, we have room to allocate
   additional opcodes, and we can grow the opcode space if we need to
   - if we decide to include per-frame extension data, we can allocate a
   reserved bit to indicate the presence of an extension length which describes
   the length of extension data
   - if we decide to include per-frame compression, we can allocate a
   reserved bit to indicate that this frame is compressed
   - we can allocate new control frame types

Is this something we can all get behind, even if it doesn't do everything we
want, in the interest of getting something useful done soon?  If not, what
specific changes would be necessary for you to agree?

-- 
John A. Tamplin
Software Engineer (GWT), Google