Re: [hybi] Framing Take VI (a compromise proposal)
Ian Fette (イアンフェッティ) <ifette@google.com> Mon, 16 August 2010 04:02 UTC
Return-Path: <ifette@google.com>
X-Original-To: hybi@core3.amsl.com
Delivered-To: hybi@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id BE2CF3A6848 for <hybi@core3.amsl.com>; Sun, 15 Aug 2010 21:02:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -105.413
X-Spam-Level:
X-Spam-Status: No, score=-105.413 tagged_above=-999 required=5 tests=[AWL=0.263, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GCanZ6HRqwuh for <hybi@core3.amsl.com>; Sun, 15 Aug 2010 21:02:13 -0700 (PDT)
Received: from smtp-out.google.com (smtp-out.google.com [216.239.44.51]) by core3.amsl.com (Postfix) with ESMTP id 116703A6957 for <hybi@ietf.org>; Sun, 15 Aug 2010 21:02:12 -0700 (PDT)
Received: from hpaq1.eem.corp.google.com (hpaq1.eem.corp.google.com [172.25.149.1]) by smtp-out.google.com with ESMTP id o7G42mXl008428 for <hybi@ietf.org>; Sun, 15 Aug 2010 21:02:48 -0700
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=google.com; s=beta; t=1281931368; bh=gDuR+iPPQ/dfT0HqQHjLC8aC0/8=; h=MIME-Version:Reply-To:In-Reply-To:References:Date:Message-ID: Subject:From:To:Cc:Content-Type; b=AZG2bcfWEnwRh1IEmE3Z+HijfsbhBtuBAI8P0iQRwbipsVj3FyPLe6OhEm5GZPL7j JkxaCxYwko3gomTOe0g+w==
DomainKey-Signature: a=rsa-sha1; s=beta; d=google.com; c=nofws; q=dns; h=mime-version:reply-to:in-reply-to:references:date: message-id:subject:from:to:cc:content-type:x-system-of-record; b=Cwmr24vIQB7maDEBuUgrpr6/iTT7sAe3b9t1D16PbHCQSmt4KgCTXXb90d+WdKTHE MXLaYn9Rv+v8tlSEkZ6oQ==
Received: from yxm34 (yxm34.prod.google.com [10.190.4.34]) by hpaq1.eem.corp.google.com with ESMTP id o7G42kuD019204 for <hybi@ietf.org>; Sun, 15 Aug 2010 21:02:46 -0700
Received: by yxm34 with SMTP id 34so2515225yxm.35 for <hybi@ietf.org>; Sun, 15 Aug 2010 21:02:45 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.150.52.11 with SMTP id z11mr4909766ybz.15.1281931365572; Sun, 15 Aug 2010 21:02:45 -0700 (PDT)
Received: by 10.150.67.19 with HTTP; Sun, 15 Aug 2010 21:02:45 -0700 (PDT)
In-Reply-To: <AANLkTi=52ARa2v9oZ1tGosN7oL+kBBVH20XmtkXzPMM0@mail.gmail.com>
References: <AANLkTi=TBXO_Cbb+P+e2BVfx69shkf8E1-9ywDh_Y+Kz@mail.gmail.com> <AANLkTi=52ARa2v9oZ1tGosN7oL+kBBVH20XmtkXzPMM0@mail.gmail.com>
Date: Sun, 15 Aug 2010 21:02:45 -0700
Message-ID: <AANLkTimjt0YbXfLf83od5b2cSJaZUboBy3-03LFf=Kv8@mail.gmail.com>
From: "Ian Fette (イアンフェッティ)" <ifette@google.com>
To: Greg Wilkins <gregw@webtide.com>
Content-Type: multipart/alternative; boundary="000e0cd6a80eabe25e048de8e7c0"
X-System-Of-Record: true
Cc: hybi@ietf.org
Subject: Re: [hybi] Framing Take VI (a compromise proposal)
X-BeenThere: hybi@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
Reply-To: ifette@google.com
List-Id: Server-Initiated HTTP <hybi.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/hybi>, <mailto:hybi-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/hybi>
List-Post: <mailto:hybi@ietf.org>
List-Help: <mailto:hybi-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/hybi>, <mailto:hybi-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 16 Aug 2010 04:02:15 -0000
On Sun, Aug 15, 2010 at 8:44 PM, Greg Wilkins <gregw@webtide.com> wrote: > Ian, > > thanks for the proposal. > > It appears that the total message length is mandatory if fragmentation > is used, however I believe that one of the use-cases for fragmentation > is to send dynamic messages where the total length is not known in > advance. Could we make message length optional somehow? > > It is optional. %x0000000000000000 ; complete message length is unknown (iff initial==1 && final==0) -- suggestions as to how to make that clearer most welcome. > Alternately, message length (in bytes and/or in characters etc.) is > just one example of meta-data. Would it not be possible to support > that in a standardized meta data extension? > > If we are to have per frame extension data, then I think having the > extension length in the header so simple implementations can skip is a > good approach. However, I think we probably would also need to goto > extra detail about how data from multiple extensions would be > separated within the extension data. > > However, I remain a little concerned that having both op-code and per > frame extension data could result in some contradictory data. For > example, if we have an op-code for final frame, then it is likely that > intermediaries will know that frame and possibly use it as part of > shutting down the connection (with half closes etc.). If we then have > an extension that provides multiplexing and allows a channel ID to be > associated with every frame, what does it mean to have a final frame > op code extended with a channel ID? I would assume it means that a complete message has been sent, but that the channel remains open. E.g. I have two channels open, the fact that I've sent a complete message on one channel shouldn't be taken as an indication that I'm done sending messages on that channel. Even in the non-multiplexing case, the fact that you've received a full message doesn't mean the connection should be shut down. That's the point of "persistent" :) > The obvious interpretation for > the end points is that is the final frame for just that channel, No, it's not the final frame for the channel, it's the final frame for a message. We said a message can be multiple frames, and multiple messages can be sent on a websocket connection (or on a channel in the multiplex case). > but > intermediaries would not know that and could shutdown the whole > connection. > > Thus until such potential contradictions can be worked out, I think > I'm still in favour of an op-code that indicates that the frame > contains extended data plus data. This could still be used for > something like multiplexing as a total replacement for plain > text/binary frame opcodes. I'll put together a more detailed proposal > for this shortly. > > cheers > > > > > > > > > > 2010/8/13 Ian Fette (イアンフェッティ) <ifette@google.com>: > > There have been a lot of framing proposals coming out lately, that have > > generated a lot of good discussion and feedback. A group of us at Google > > have tried to sift through the proposals, requirements, and feedback and > > tried to come up with a proposal that I'm hoping meets people's needs. No > > one is ever going to be 100% happy with every last detail, but I really > want > > to see us agree on something that we agree is solid and move on. At some > > point, we do need to decide. > > The below proposal is based largely on Dave's earlier proposal, to give > > credit where it's due. Takeshi Yoshino and John Tamplin are largely to > > credit for the modifications, my contribution is minimal :) We've tried > to > > take into account what have come out of various consensus calls as well > as > > where it seems people aren't likely to budge. We've given in on a number > of > > points (e.g. multiplexing, giving up fixed 16-bit length, supporting a > > 64-bit length for large messages) in an attempt to make progress, so I > would > > also ask that people really think about what they feel strongly about and > > what they can be willing to consider to support consensus. > > That said, here's the proposal: > > > > 0 1 2 3 > > > > 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 > > > > +-+-+-+-+-------+-+-------------+-------------------------------+ > > > > |I|F|E|R| opcode|R| Payload len | Extended payload length | > > > > |N|I|X|S| (4) |S| (7) | (64) | > > > > |I|N|T|V| |V| | (if payload len==127) | > > > > +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - + > > > > | | > > > > + - - - - - - - - - - - - - - - +-------------------------------+ > > > > | | | > > > > +-------------------------------+ - - - - - - - - - - - - - - - + > > > > | Complete message length (64) (if INI==1 && FIN==0) | > > > > + - - - - - - - - - - - - - - - +-------------------------------+ > > > > | | Ext len (8) (if EXT==1) | > > > > +-------------------------------+-------------------------------+ > > > > | Extended extension length (64) (if EXT=1 && ext len == 0) | > > > > + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + > > > > | | > > > > +---------------------------------------------------------------+ > > > > | Extension data | > > > > | (if EXT==1) | > > > > +---------------------------------------------------------------+ > > > > | Application data | > > > > | | > > > > +---------------------------------------------------------------+ > > > > ws-frame = frame-type > > > > frame-reserved > > > > frame-payload-length > > > > frame-complete-message-length > > > > frame-extension-length > > > > extension-data > > > > application-data > > > > frame-type = type-initial > > > > type-final > > > > type-extension > > > > type-reserved > > > > type-opcode > > > > type-initial = %x0 ; not initial frame > > > > = %x1 ; initial frame > > > > type-final = %x0 ; not final frame > > > > = %x1 ; final frame > > > > type-extension = %x0 ; no extension > > > > = %x1 ; has extension > > > > type-reserved = %x0 ; 1 bit > > > > type-opcode = %x0 ; control frame > > > > / %x1 ; text data frame > > > > / %x2 ; binary data frame (blob) > > > > / %x3-%xB ; reserved > > > > / %xC-%xF ; user defined > > > > frame-reserved = %x0 ; 1 bit > > > > frame-payload-length = %x00-7E > > > > / %x7F frame-extended-payload-length > > > > frame-extended-payload-length = %x000000000000007F-FFFFFFFFFFFFFFFF ; > > payload length including all fields after this field > > > > frame-complete-message-length = %x0000000000000000 ; complete message > > length is unknown (iff initial==1 && final==0) > > > > / %x0000000000000001-FFFFFFFFFFFFFFFF ; > > complete message length. only application data part will be counted in > (iff > > initial==1 && final==0) > > > > / ; none (otherwise) > > > > frame-extension-length = ; none (iff type-extension==0) > > > > / %x01-7F > > > > / %x00 frame-extended-extension-length > > > > frame-extended-extension-length = %x000000000000007F-FFFFFFFFFFFFFFFF > ; > > extension length > > > > extension-data = *( %x00-FF ) ; iff type-extension==1 > > > > application-data = text-data > > > > / binary-data > > > > / control > > > > text-data = *( %x00-FF ) > > > > binary-data = *( %x00-FF ) > > > > control-data = control-type control-info > > > > control-type = %x00-FF ; type of control frame > > > > control-info = *( %x00-FF ) ; additional data of control frame > > > > > > > > * terminology > > > > - frame : WebSocket framing unit. if chunked, each chunk forms one frame > > > > - message : JavaScript level data unit. JavaScript tells WebSocket > boundary > > between messages by some manner. may be chunked into multiple frames on > wire > > > > - chunking : cutting single message into multiple frames > > > > * requirements and how this proposal addresses them > > > > bandwidth/memory consumption > > > > - small frame size for small message > > > > -- 3 octet frame for 1 octet payload with no extension > > > > - if payload is small (<127), we have only one octet length header field > > > > - turn off EXT flag to have no other overhead for extension > > > > - if not chunked, we don't have "complete message length field" as it's > > redundant > > > > processing efficiency > > > > - data type is available on initial chunk so that we can start UTF-8 > > decoding immediately > > > > -- having a single opcode to start a fragmented message and separate > opcodes > > to determine if it is a text or binary message means you can't start to > > decode UTF8 text until you receive the entire message, which means you > add a > > buffering requirement of the undecoded message > > > > buffer management > > > > - complete message length is available on initial chunk so that we can > > allocate sufficient buffer. no realloc > > > > -- many people objected to not having an overall message length to ease > > receiver buffer management > > > > - Question: are endpoints likely enough to use UTF16 for internal > > representation of text that it would make sense to send the number of > UTF16 > > characters instead of bytes as the message length or as an additional > field > > on text frames? > > > > - Note that there is still the option to not provide the message length > for > > cases when it is not known, such as dynamically generated content or > > compressed content (though it is possible we would define the message > length > > to be the uncompressed length in the case of compression). > > > > - Also, the message length is only required in the case the message is > > fragmented. > > > > extensibility > > > > - we'd include room for extension > > > > -- variable length (we cannot fix the size) > > > > simplicity/range of length header > > > > - variable length header > > > > -- we give up the use of fixed length header > > > > --- there should be small sized header for small data. 16 bit, 32 bit, 64 > > bit, all not good > > > > --- 8 bit is also not good. We'll have too much overhead for chunking > large > > message > > > > - variable length length header > > > > -- not so complicated. just two case. 1 octet for 0 - 126 and 9 octet for > > 127 - 2^64-1 > > > > -- simpler enough than variable length length header defined in -76 > > > > -- big range > > > > --- we can use extended payload length to send large message as 1 frame > > without any header stripe on it > > ---- various people objected to having to fragment messages at all if > they > > are already in RAM (no more copy) > > -- small range > > > > --- small overhead as explained in "bandwidth/memory consumption" section > > > > > > simplicity/intermediates > > > > - extension transparent to intermediate > > > > -- payload length counts in extension. intermediates that don't recognize > > extension don't have to care about extension length > > > > - only 1 frame boundary indication scheme (length header based one. no > > sentinel based one) as many people prefer > > > > * extension plan > > > > - we included spaces for future extension > > > > - for new spec > > > > -- two bits per frame are available for future definition > > > > -- reserved opcodes > > > > - for implementors > > > > -- private use opcodes > > > > -- extension data based on extensions negotiated during the handshake > > > > _______________________________________________ > > hybi mailing list > > hybi@ietf.org > > https://www.ietf.org/mailman/listinfo/hybi > > > > >
- Re: [hybi] Framing Take VI (a compromise proposal) Patrick McManus
- Re: [hybi] Framing Take VI (a compromise proposal) Scott Ferguson
- [hybi] Framing Take VI (a compromise proposal) Ian Fette (イアンフェッティ)
- Re: [hybi] Framing Take VI (a compromise proposal) Ian Hickson
- Re: [hybi] Framing Take VI (a compromise proposal) Ian Fette (イアンフェッティ)
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Bjoern Hoehrmann
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Takeshi Yoshino
- Re: [hybi] Framing Take VI (a compromise proposal) Takeshi Yoshino
- Re: [hybi] Framing Take VI (a compromise proposal) Willy Tarreau
- Re: [hybi] Framing Take VI (a compromise proposal) Anne van Kesteren
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Greg Wilkins
- Re: [hybi] Framing Take VI (a compromise proposal) Ian Fette (イアンフェッティ)
- Re: [hybi] Framing Take VI (a compromise proposal) Greg Wilkins
- Re: [hybi] Framing Take VI (a compromise proposal) Takeshi Yoshino
- Re: [hybi] Framing Take VI (a compromise proposal) Greg Wilkins
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Dave Cridland
- Re: [hybi] Framing Take VI (a compromise proposal) Dave Cridland
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Greg Wilkins
- Re: [hybi] Framing Take VI (a compromise proposal) Patrick McManus
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Ian Fette (イアンフェッティ)
- Re: [hybi] Framing Take VI (a compromise proposal) Dave Cridland
- Re: [hybi] Framing Take VI (a compromise proposal) Douglas Otis
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Thomson, Martin
- Re: [hybi] Framing Take VI (a compromise proposal) gustav trede
- Re: [hybi] Framing Take VI (a compromise proposal) Ian Fette (イアンフェッティ)
- Re: [hybi] Framing Take VI (a compromise proposal) Thomson, Martin
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Thomson, Martin
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Scott Ferguson
- Re: [hybi] Framing Take VI (a compromise proposal) Scott Ferguson
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin
- Re: [hybi] Framing Take VI (a compromise proposal) Dave Cridland
- Re: [hybi] Framing Take VI (a compromise proposal) John Tamplin