Re: [hybi] Framing Take VI (a compromise proposal)

Ian Fette (イアンフェッティ) <ifette@google.com> Mon, 16 August 2010 04:02 UTC

Return-Path: <ifette@google.com>
X-Original-To: hybi@core3.amsl.com
Delivered-To: hybi@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id BE2CF3A6848 for <hybi@core3.amsl.com>; Sun, 15 Aug 2010 21:02:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -105.413
X-Spam-Level:
X-Spam-Status: No, score=-105.413 tagged_above=-999 required=5 tests=[AWL=0.263, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GCanZ6HRqwuh for <hybi@core3.amsl.com>; Sun, 15 Aug 2010 21:02:13 -0700 (PDT)
Received: from smtp-out.google.com (smtp-out.google.com [216.239.44.51]) by core3.amsl.com (Postfix) with ESMTP id 116703A6957 for <hybi@ietf.org>; Sun, 15 Aug 2010 21:02:12 -0700 (PDT)
Received: from hpaq1.eem.corp.google.com (hpaq1.eem.corp.google.com [172.25.149.1]) by smtp-out.google.com with ESMTP id o7G42mXl008428 for <hybi@ietf.org>; Sun, 15 Aug 2010 21:02:48 -0700
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=google.com; s=beta; t=1281931368; bh=gDuR+iPPQ/dfT0HqQHjLC8aC0/8=; h=MIME-Version:Reply-To:In-Reply-To:References:Date:Message-ID: Subject:From:To:Cc:Content-Type; b=AZG2bcfWEnwRh1IEmE3Z+HijfsbhBtuBAI8P0iQRwbipsVj3FyPLe6OhEm5GZPL7j JkxaCxYwko3gomTOe0g+w==
DomainKey-Signature: a=rsa-sha1; s=beta; d=google.com; c=nofws; q=dns; h=mime-version:reply-to:in-reply-to:references:date: message-id:subject:from:to:cc:content-type:x-system-of-record; b=Cwmr24vIQB7maDEBuUgrpr6/iTT7sAe3b9t1D16PbHCQSmt4KgCTXXb90d+WdKTHE MXLaYn9Rv+v8tlSEkZ6oQ==
Received: from yxm34 (yxm34.prod.google.com [10.190.4.34]) by hpaq1.eem.corp.google.com with ESMTP id o7G42kuD019204 for <hybi@ietf.org>; Sun, 15 Aug 2010 21:02:46 -0700
Received: by yxm34 with SMTP id 34so2515225yxm.35 for <hybi@ietf.org>; Sun, 15 Aug 2010 21:02:45 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.150.52.11 with SMTP id z11mr4909766ybz.15.1281931365572; Sun, 15 Aug 2010 21:02:45 -0700 (PDT)
Received: by 10.150.67.19 with HTTP; Sun, 15 Aug 2010 21:02:45 -0700 (PDT)
In-Reply-To: <AANLkTi=52ARa2v9oZ1tGosN7oL+kBBVH20XmtkXzPMM0@mail.gmail.com>
References: <AANLkTi=TBXO_Cbb+P+e2BVfx69shkf8E1-9ywDh_Y+Kz@mail.gmail.com> <AANLkTi=52ARa2v9oZ1tGosN7oL+kBBVH20XmtkXzPMM0@mail.gmail.com>
Date: Sun, 15 Aug 2010 21:02:45 -0700
Message-ID: <AANLkTimjt0YbXfLf83od5b2cSJaZUboBy3-03LFf=Kv8@mail.gmail.com>
From: "Ian Fette (イアンフェッティ)" <ifette@google.com>
To: Greg Wilkins <gregw@webtide.com>
Content-Type: multipart/alternative; boundary="000e0cd6a80eabe25e048de8e7c0"
X-System-Of-Record: true
Cc: hybi@ietf.org
Subject: Re: [hybi] Framing Take VI (a compromise proposal)
X-BeenThere: hybi@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
Reply-To: ifette@google.com
List-Id: Server-Initiated HTTP <hybi.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/hybi>, <mailto:hybi-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/hybi>
List-Post: <mailto:hybi@ietf.org>
List-Help: <mailto:hybi-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/hybi>, <mailto:hybi-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 16 Aug 2010 04:02:15 -0000

On Sun, Aug 15, 2010 at 8:44 PM, Greg Wilkins <gregw@webtide.com> wrote:

> Ian,
>
> thanks for the proposal.
>
> It appears that the total message length is mandatory if fragmentation
> is used, however I believe that one of the use-cases for fragmentation
> is to send dynamic messages where the total length is not known in
> advance.   Could we make message length optional somehow?
>
>
It is optional. %x0000000000000000 ; complete message length is unknown (iff
initial==1 && final==0) -- suggestions as to how to make that clearer most
welcome.


> Alternately, message length (in bytes and/or in characters etc.) is
> just one example of meta-data.  Would it not be possible to support
> that in a standardized meta data extension?
>
> If we are to have per frame extension data, then I think having the
> extension length in the header so simple implementations can skip is a
> good approach.  However, I think we probably would also need to goto
> extra detail about how data from multiple extensions would be
> separated within the extension data.
>
> However, I remain a little concerned that having both op-code and per
> frame extension data could result in some contradictory data.  For
> example, if we have an op-code for final frame, then it is likely that
> intermediaries will know that frame and possibly use it as part of
> shutting down the connection (with half closes etc.).  If we then have
> an extension that provides multiplexing and allows a channel ID to be
> associated with every frame, what does it mean to have a final frame
> op code extended with a channel ID?


I would assume it means that a complete message has been sent, but that the
channel remains open. E.g. I have two channels open, the fact that I've sent
a complete message on one channel shouldn't be taken as an indication that
I'm done sending messages on that channel. Even in the non-multiplexing
case, the fact that you've received a full message doesn't mean the
connection should be shut down. That's the point of "persistent" :)


>  The obvious interpretation for
> the end points is that is the final frame for just that channel,


No, it's not the final frame for the channel, it's the final frame for a
message. We said a message can be multiple frames, and multiple messages can
be sent on a websocket connection (or on a channel in the multiplex case).


> but
> intermediaries would not know that and could shutdown the whole
> connection.
>
> Thus until such potential contradictions can be worked out, I think
> I'm still in favour of an op-code that indicates that the frame
> contains extended data plus data.  This could still be used for
> something like multiplexing as a total replacement for plain
> text/binary frame opcodes. I'll put together a more detailed proposal
> for this shortly.
>
> cheers
>
>
>
>
>
>
>
>
>
> 2010/8/13 Ian Fette (イアンフェッティ) <ifette@google.com>:
> > There have been a lot of framing proposals coming out lately, that have
> > generated a lot of good discussion and feedback. A group of us at Google
> > have tried to sift through the proposals, requirements, and feedback and
> > tried to come up with a proposal that I'm hoping meets people's needs. No
> > one is ever going to be 100% happy with every last detail, but I really
> want
> > to see us agree on something that we agree is solid and move on. At some
> > point, we do need to decide.
> > The below proposal is based largely on Dave's earlier proposal, to give
> > credit where it's due. Takeshi Yoshino and John Tamplin are largely to
> > credit for the modifications, my contribution is minimal :) We've tried
> to
> > take into account what have come out of various consensus calls as well
> as
> > where it seems people aren't likely to budge. We've given in on a number
> of
> > points (e.g. multiplexing, giving up fixed 16-bit length, supporting a
> > 64-bit length for large messages) in an attempt to make progress, so I
> would
> > also ask that people really think about what they feel strongly about and
> > what they can be willing to consider to support consensus.
> > That said, here's the proposal:
> >
> >    0                   1                   2                   3
> >
> >    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> >
> >   +-+-+-+-+-------+-+-------------+-------------------------------+
> >
> >   |I|F|E|R| opcode|R| Payload len |    Extended payload length    |
> >
> >   |N|I|X|S|  (4)  |S|     (7)     |             (64)              |
> >
> >   |I|N|T|V|       |V|             |     (if payload len==127)     |
> >
> >   +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
> >
> >   |                                                               |
> >
> >   + - - - - - - - - - - - - - - - +-------------------------------+
> >
> >   |                               |                               |
> >
> >   +-------------------------------+ - - - - - - - - - - - - - - - +
> >
> >   |       Complete message length (64) (if INI==1 && FIN==0)      |
> >
> >   + - - - - - - - - - - - - - - - +-------------------------------+
> >
> >   |                               |     Ext len (8) (if EXT==1)   |
> >
> >   +-------------------------------+-------------------------------+
> >
> >   |   Extended extension length (64) (if EXT=1 && ext len == 0)   |
> >
> >   + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
> >
> >   |                                                               |
> >
> >   +---------------------------------------------------------------+
> >
> >   |                        Extension data                         |
> >
> >   |                          (if EXT==1)                          |
> >
> >   +---------------------------------------------------------------+
> >
> >   |                       Application data                        |
> >
> >   |                                                               |
> >
> >   +---------------------------------------------------------------+
> >
> >     ws-frame            = frame-type
> >
> >                           frame-reserved
> >
> >                           frame-payload-length
> >
> >                           frame-complete-message-length
> >
> >                           frame-extension-length
> >
> >                           extension-data
> >
> >                           application-data
> >
> >     frame-type          = type-initial
> >
> >                           type-final
> >
> >                           type-extension
> >
> >                           type-reserved
> >
> >                           type-opcode
> >
> >     type-initial        = %x0 ; not initial frame
> >
> >                         = %x1 ; initial frame
> >
> >     type-final          = %x0 ; not final frame
> >
> >                         = %x1 ; final frame
> >
> >     type-extension      = %x0 ; no extension
> >
> >                         = %x1 ; has extension
> >
> >     type-reserved       = %x0 ; 1 bit
> >
> >     type-opcode         = %x0 ; control frame
> >
> >                         / %x1 ; text data frame
> >
> >                         / %x2 ; binary data frame (blob)
> >
> >                         / %x3-%xB ; reserved
> >
> >                         / %xC-%xF ; user defined
> >
> >     frame-reserved      = %x0 ; 1 bit
> >
> >     frame-payload-length = %x00-7E
> >
> >                          / %x7F frame-extended-payload-length
> >
> >     frame-extended-payload-length = %x000000000000007F-FFFFFFFFFFFFFFFF ;
> > payload length including all fields after this field
> >
> >     frame-complete-message-length = %x0000000000000000 ; complete message
> > length is unknown (iff initial==1 && final==0)
> >
> >                                   / %x0000000000000001-FFFFFFFFFFFFFFFF ;
> > complete message length. only application data part will be counted in
> (iff
> > initial==1 && final==0)
> >
> >                                   / ; none (otherwise)
> >
> >     frame-extension-length = ; none (iff type-extension==0)
> >
> >                            / %x01-7F
> >
> >                            / %x00 frame-extended-extension-length
> >
> >     frame-extended-extension-length = %x000000000000007F-FFFFFFFFFFFFFFFF
> ;
> > extension length
> >
> >     extension-data      = *( %x00-FF ) ; iff type-extension==1
> >
> >     application-data    = text-data
> >
> >                         / binary-data
> >
> >                         / control
> >
> >     text-data           = *( %x00-FF )
> >
> >     binary-data         = *( %x00-FF )
> >
> >     control-data        = control-type control-info
> >
> >     control-type        = %x00-FF ; type of control frame
> >
> >     control-info        = *( %x00-FF ) ; additional data of control frame
> >
> >
> >
> > * terminology
> >
> > - frame : WebSocket framing unit. if chunked, each chunk forms one frame
> >
> > - message : JavaScript level data unit. JavaScript tells WebSocket
> boundary
> > between messages by some manner. may be chunked into multiple frames on
> wire
> >
> > - chunking : cutting single message into multiple frames
> >
> > * requirements and how this proposal addresses them
> >
> > bandwidth/memory consumption
> >
> > - small frame size for small message
> >
> > -- 3 octet frame for 1 octet payload with no extension
> >
> > - if payload is small (<127), we have only one octet length header field
> >
> > - turn off EXT flag to have no other overhead for extension
> >
> > - if not chunked, we don't have "complete message length field" as it's
> > redundant
> >
> > processing efficiency
> >
> > - data type is available on initial chunk so that we can start UTF-8
> > decoding immediately
> >
> > -- having a single opcode to start a fragmented message and separate
> opcodes
> > to determine if it is a text or binary message means you can't start to
> > decode UTF8 text until you receive the entire message, which means you
> add a
> > buffering requirement of the undecoded message
> >
> > buffer management
> >
> > - complete message length is available on initial chunk so that we can
> > allocate sufficient buffer. no realloc
> >
> > -- many people objected to not having an overall message length to ease
> > receiver buffer management
> >
> > - Question: are endpoints likely enough to use UTF16 for internal
> > representation of text that it would make sense to send the number of
> UTF16
> > characters instead of bytes as the message length or as an additional
> field
> > on text frames?
> >
> > - Note that there is still the option to not provide the message length
> for
> > cases when it is not known, such as dynamically generated content or
> > compressed content (though it is possible we would define the message
> length
> > to be the uncompressed length in the case of compression).
> >
> > - Also, the message length is only required in the case the message is
> > fragmented.
> >
> > extensibility
> >
> > - we'd include room for extension
> >
> > -- variable length (we cannot fix the size)
> >
> > simplicity/range of length header
> >
> > - variable length header
> >
> > -- we give up the use of fixed length header
> >
> > --- there should be small sized header for small data. 16 bit, 32 bit, 64
> > bit, all not good
> >
> > --- 8 bit is also not good. We'll have too much overhead for chunking
> large
> > message
> >
> > - variable length length header
> >
> > -- not so complicated. just two case. 1 octet for 0 - 126 and 9 octet for
> > 127 - 2^64-1
> >
> > -- simpler enough than variable length length header defined in -76
> >
> > -- big range
> >
> > --- we can use extended payload length to send large message as 1 frame
> > without any header stripe on it
> > ---- various people objected to having to fragment messages at all if
> they
> > are already in RAM (no more copy)
> > -- small range
> >
> > --- small overhead as explained in "bandwidth/memory consumption" section
> >
> >
> > simplicity/intermediates
> >
> > - extension transparent to intermediate
> >
> > -- payload length counts in extension. intermediates that don't recognize
> > extension don't have to care about extension length
> >
> > - only 1 frame boundary indication scheme (length header based one. no
> > sentinel based one) as many people prefer
> >
> > * extension plan
> >
> > - we included spaces for future extension
> >
> > - for new spec
> >
> > -- two bits per frame are available for future definition
> >
> > -- reserved opcodes
> >
> > - for implementors
> >
> > -- private use opcodes
> >
> > -- extension data based on extensions negotiated during the handshake
> >
> > _______________________________________________
> > hybi mailing list
> > hybi@ietf.org
> > https://www.ietf.org/mailman/listinfo/hybi
> >
> >
>