Re: [hybi] Is it important to know frame length at the start of frame? (was: Re: Discontinuation of mux ...)

[Takeshi Yoshino <[e-mail]>]

Joakim,

In my post, I was doubting necessity of frame length header IN THE CONTEXT
OF WebSocket over HTTP/2.0 discussion as the HTTP/2.0 DATA frame also has a
frame length that is immediately decodable and you can choose to send a big
one DATA frame to send a big message to tell the peer that it's a big frame
at the start of arrival of the frame. Not doubting necessity of frame
length in general.

However, I noticed that I didn't consider that the size of the frame length
field of the current HTTP/2.0 spec is very small. It's 14 bit. Old SPDYs
used 24 bits (and the latest SPDY/4 uses 16 bits).

So, without help by WebSocket level length field, even by using an
un-fragmented WebSocket frame (at WebSocket semantics level), we cannot
tell the peer that "we're sending a message which is N (N > 16k) bytes
long" on the first DATA frame. Sorry that I didn't notice this point.


On Thu, Feb 13, 2014 at 1:19 AM, Joakim Erdfelt <[e-mail]> wrote:

> Eliminating the size from the frame also eliminates the ability to
> fast-fail on excessive messages.
>
> Scenario (with current RFC6455 / WS+HTTP/1.1):
>   Sending side starts to write out a single message of 100MB.
>   Receiving side, using a whole message delivery based API is configured
> for max messages of 20MB.
>   Receiving side sees the 100MB frame size and automatically closes the
> connection with 1009 and doesn't bother reading past the current network
> buffer.
>
> Scenario (without frame sizes):
>   Sending side starts to write out a single message of 100MB.
>   Receiving side, using a whole message delivery based API is configured
> for max messages of 20MB.
>   Receiving side reads till the 20MB mark, then fails the connection with
> 1009.  Wasting time, memory, and resources for no good reason.
>
> I really don't see how a discussion of a protocol without sizes is even
> going on.
> I mean, even the IP layer and TCP layer have sizes for their frames.
>
>
>
> --
> Joakim Erdfelt <[e-mail]>
> webtide.com <http://www.webtide.com/> - intalio.com/jetty
> Expert advice, services and support from from the Jetty & CometD experts
> eclipse.org/jetty - cometd.org
>
>
> On Wed, Feb 12, 2014 at 9:13 AM, Joakim Erdfelt <[e-mail]>wrote:
>
>> Scenario:
>>   Sending side uses a message stream based API to write out 1 websocket
>> message.
>>   The sending side's API is configured/implemented to buffer only 16k at
>> a time, writing out a frame of 16k for each buffer filled.
>>   The sending side writes out a 600k message.
>>   The receiving side is using a different API, one that is based on
>> complete messages.
>>   The receiving side is configured to not accept messages over 100k.
>>   The receiving side's API/implementation tosses a close code 1009 to the
>> sending side, per API.
>>   The receiving side's endpoint receives an error and close notification
>> indicating a protocol closure due to excessive message size from sending
>> side.
>>
>>
>>
>> --
>> Joakim Erdfelt <[e-mail]>
>>  webtide.com <http://www.webtide.com/> - intalio.com/jetty
>> Expert advice, services and support from from the Jetty & CometD experts
>> eclipse.org/jetty - cometd.org
>>
>>
>> On Wed, Feb 12, 2014 at 9:00 AM, Zhong Yu <[e-mail]> wrote:
>>
>>> Why not send the message in one frame? The frame length is then the
>>> message length.
>>>
>>> One argument for fragmentation is to insert control frames while
>>> transmitting a long message. However that reasoning is not very
>>> convincing:
>>>
>>> PONG - as long as we are writing data, any data, the peer should see
>>> that the connection is alive. It's unreasonable for the peer to think
>>> that our end is unresponsive while it is receiving data from us.
>>>
>>> PING - not necessary while we are writing data. If peer is
>>> unresponsive, we'll learn that from TCP layer.
>>>
>>> CLOSE - closing the connection in the middle of a message is abnormal
>>> anyway, it's probably not important to do the close handshake.
>>>
>>> Zhong Yu
>>>
>>>
>>> On Fri, Feb 7, 2014 at 1:30 AM, Takeshi Yoshino <[e-mail]>
>>> wrote:
>>> > I remember that in early days of the standardization, somebody was
>>> proposing
>>> > (*) that we should have a message length field in the first fragment to
>>> > allow for allocating buffer for full message at the beginning.
>>> >
>>> > On Thu, Jan 16, 2014 at 3:27 PM, "Martin J. Dürst" <
>>> [e-mail]>
>>> > wrote:
>>> >>
>>> >> On 2014/01/16 15:05, Yutaka Hirano wrote:
>>> >>>
>>> >>> Hi Joakim,
>>> >>>
>>> >>> Thanks for the comment.
>>> >>>
>>> >>> Knowing the size of a frame at the start of the frame, traditional
>>> >>>>
>>> >>>> websocket behavior, is desired, as it allows us to fast fail for the
>>> >>>> 1009
>>> >>>> (message too big) use case.
>>> >>>
>>> >>>
>>> >>> Do you know how effective it is?
>>> >>>
>>> >>> Having no prior knowledge of frame size is undesirable to us.
>>> >>>
>>> >>> On the other hand, the frame sender have more freedom because they
>>> don't
>>> >>> have to know the size of the frame in front.
>>> >>
>>> >>
>>> >> My understanding of why messages are split into frames was that this
>>> would
>>> >> allow to start a message without knowing its length. But if you don't
>>> know
>>> >> the size of a frame before sending it, then the right solution is to
>>> make it
>>> >> smaller. Frames are just arbitrary pieces of a message.
>>> >
>>> >
>>> > We should be clear if we want to make it able to
>>> > (a) know the length of WebSocket frame on receiving the header of each
>>> > fragment
>>> > (b) know the length of WebSocket message on receiving the first
>>> fragment
>>> >
>>> > As the proposal (*) has been rejected and not in the RFC6455 header,
>>> (b) is
>>> > not true even in RFC6455. What Joakim said should be to keep (a) which
>>> is
>>> > true now in RFC6455.
>>> >
>>> > But if a message is fragmented into smaller frames, we cannot reject
>>> (and
>>> > send 1009) the message on receiving the first frame. "Whether we can
>>> > fast-reject a big message on receiving some non-final frame" depends
>>> on how
>>> > the peer fragments a message.
>>> >
>>> > In some plans discussed in the main thread, it's planned to drop frame
>>> > length info for efficiency. But something else such as the length
>>> field of
>>> > HTTP/2.0 DATA frame tells how much data the fragment (when multiple
>>> DATAs
>>> > convey a WebSocket frame, each DATA would be called fragments of a
>>> frame)
>>> > contains. So, we won't be able to know the length of fragments
>>> > (original/encapsulated WebSocket frames) on receiving each of them, but
>>> > still be able to know the length of sub-fragments (e.g. HTTP/2.0 data
>>> > frames).
>>> >
>>> > I think this change is acceptable and shouldn't be considered as
>>> breaking of
>>> > RFC6455 semantics. Even in the mux extension I was editing
>>> > (http://tools.ietf.org/html/draft-ietf-hybi-websocket-multiplexing-11),
>>> I
>>> > made the same change. The length of the encapsulated frame is unknown
>>> until
>>> > receiving all the fragments of the encapsulating message. No one was
>>> > objecting to that when we were discussing the spec, IIRC.
>>> >
>>> >
>>> > _______________________________________________
>>> > hybi mailing list
>>> > [e-mail]
>>> > https://www.ietf.org/mailman/listinfo/hybi
>>> >
>>>
>>> _______________________________________________
>>> hybi mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/hybi
>>>
>>
>>
>