Re: [hybi] how do we move forward on agreeing on framing?

[Francis Brosnan Blazquez <[e-mail]>]

Hi John,

Some thoughts...

> There have been various discussions about various proposals, and to be
> sure there will be no proposal that everyone is happy with.  What can
> we do to come to some consensus on something that is acceptable for
> the base protocol so we can move forward?

My impression is that the base protocol should by simple, as close as
possible to TCP. For example, your proposal, which is length limited
looks good.

> To me, I think the following things are critical:
>       * fragmentation support - this has been accepted as a
>         requirement without much dissension when it was declared that
>         we achieved consensus on it, and if we don't have support for
>         it in the base protocol it will greatly complicate efforts for
>         certain extensions a number of people think will be important
>         in the future, such as multiplexing

I think it is taking to much attention to include advanced features in
the base protocol, or to define extensions that allow them, especially
when those features can be implemented by protocols already defined like
BEEP [1]. 

I'm sure many people is expecting to access to a simple protocol that
allows direct connection and bidirectional content transfer from the
browser. Nothing else. 

>       * low overhead for small frames - as previously indicated,
>         real-world WebSocket apps do make use of a substantial
>         percentage of small messages, and when compression is added
>         the percentage grows to an overwhelming majority.  Sure, there
>         will be some WebSocket apps which won't get any benefit from
>         small frames, but that shouldn't remove the capability from
>         those that do.
>       * we can't afford to decide on the extension format now - given
>         the rate of building consensus, we won't ever finish the base
>         protocol if we have to define that now.  That means anything
>         that is critical to support in the base protocol, such as
>         fragmentation or (if we decide it is important for
>         implementations of the base protocol) the total message
>         length, cannot be defined in an extension.

+1

>       * we need to leave our options open for extensibility - since we
>         don't yet agree on exactly how the protocol should be extended
>         or exactly what facilities such extensions may need, we need
>         to have some flexibility for how those extensions can be
>         represented
> Does anyone disagree with these points?

+1 

It would be great to define base protocol as something simple, a common
base factor, where consensus is more easy. A protocol that allow to
build on instead of including features that you may not use.

> So here is another attempt at incorporating feedback to try and get
> something which could gain consensus as the framing protocol:
>       * MORE - 1 bit - if set, this is not the final frame of the
>         message
>       * TML - 1 bit - if set, the total message length is included
>         (which is not required to be on the first frame, and if
>         repeated overrides earlier values so it may be used as an
>         estimate for dynamic content)
>       * Reserved - 2 bits - must be 0 except for private use
>       * Opcode - 4 bits - 0=Continuation, 1=Control, 2=Text, 3=Binary,
>         4-11=Reserved for future assignment, 12-15=Reserved for
>         private use
>       * Reserved - 1 bit - must be 0
>       * Length - 7 bits - payload length in octets; if 127, then there
>         is an extended length
>       * Extended length - 8 bytes - payload length in octets (high bit
>         must be 0 for ease of use with languages without an unsigned
>         data type), present only if Length=127
>       * Total Message Length - 8 bytes - total message length in
>         octets (high bit must be 0 as above), present only if TML=1
>       * Payload - (Extended) Length bytes - payload, interpreted as
>         defined by the Opcode
> This addresses the concern about requiring the total message length to
> be in the first frame or to be present at all if it is unknown.  It
> allows us to use a bit to indicate extension data is present, or to
> define extensions by new opcodes (and to grow the opcode field if
> necessary by using adjacent reserve bits).  Some bits are available
> for direct assignment, such as if we find frame-by-frame compression a
> useful future extension. It also cleans up room for confusion in the
> previous proposal where you could change opcodes between fragments.
> 
> 
> Example messages:
>       * MORE=0,TML=0,Opcode=Text, Length=12, Payload="Hello world!" -
>         single-frame text message
> 
> 
>       * MORE=1,TML=1,Opcode=Text, Length=5, MsgLength = 12,
>         Payload="Hello" - first frame of fragmented message
>       * MORE=0,TML=0,Opcode=Continuation, Length=7, Payload=" world!"
>         - final frame of fragmented message
> 
> 
>       * MORE=1,TML=1,Opcode=Text, Length=127, ExtLength=4096,
>         MsgLength=16000, Payload=... - first frame of dynamically
>         generated message, with estimated message length
>       * MORE=1,TML=0,Opcode=Continuation, Length=127,
>         ExtLength=4096,Payload=... - several frames delivering dynamic
>         content
>       * MORE=1,TML=1,Opcode=Continuation, Length=127, ExtLength=4096,
>         MsgLength=14781, Payload=... - sender now knows length of
>         dynamic content and sends it
>       * MORE=0,TML=0,Opcode=Continuation, Length=127, ExtLength=2493,
>         Payload=... - final frame of message
> Extension Possibilities:
>       * if we decide creating new opcodes for extensions, which then
>         nest the next layer of extension within their payload data, we
>         have room to allocate additional opcodes, and we can grow the
>         opcode space if we need to
>       * if we decide to include per-frame extension data, we can
>         allocate a reserved bit to indicate the presence of an
>         extension length which describes the length of extension data
>       * if we decide to include per-frame compression, we can allocate
>         a reserved bit to indicate that this frame is compressed
>       * we can allocate new control frame types
> Is this something we can all get behind, even if it doesn't do
> everything we want, in the interest of getting something useful done
> soon?  If not, what specific changes would be necessary for you to
> agree?

Your proposal looks good to me. Cheers!


[1] http://www.beepcore.org
-- 
Francis Brosnan Blazquez <[e-mail]>
ASPL