Re: WiSH: A General Purpose Message Framing over Byte-Stream Oriented Wire Protocols (HTTP)

[Mark Nottingham <[e-mail]>]

> On 25 Nov. 2016, at 1:32 pm, Andy Green <[e-mail]> wrote:
> 
> On Fri, 2016-11-25 at 13:12 +1100, Mark Nottingham wrote:
>> Andy,
>> 
>> The "people managing it" are the IETF. "Let the community do it" is
>> how the IETF works, with guidance from IETF leadership. I had a 
> 
> No... when ws for http/1 was hashed out, there were some adults in the
> room who could make judgments about what was or wasn't "a consensus" in
> a kinda-definitive way.  Several times these calls were important in
> stopping the definition going around in endless circles... even so you
> will recall there were like 80+ drafts of it.
> 
> That has gone away now.
> 
>> discussion with our Area Director about this topic in Seoul, and we
>> agreed that the hybi list was the best place to talk about
>> WebSockets.
> 
> That's not necessarily a problem, but is either of you guys who decided
> that going to restore the ML to have the kind of management it had when
> it spat out RFC6455?  Or if it's chicken-and-egg, since the list is
> moribund and no evident interest today to point to, some threshold for
> that happening?  If not as I said it feels like it's just getting
> punted over the wall.  It's going back to Hybi but the management of
> Hybi is gone and it's nobody's problem...

Adding Alexey, the Area Directory who's most relevant to this.

You seem to be looking for a Working Group. WGs are formed when the IESG (i.e., all of the ADs) are convinced that it's useful to do so. The first step is usually to put together a charter proposal, and start gathering a community.

In the meantime, the adult supervision you're looking for isn't over here either; our charter restricts us to HTTP.

Cheers,



> -Andy
> 
>> Cheers,
>> 
>> 
>>> On 25 Nov. 2016, at 11:51 am, Andy Green <[e-mail]> wrote:
>>> 
>>> On Fri, 2016-11-25 at 10:35 +1100, Mark Nottingham wrote:
>>>> To clarify, we said that the future of WebSockets really isn't in
>>>> scope for this WG; the proper venue for discussing that is:
>>>>   https://www.ietf.org/mailman/listinfo/hybi
>>> 
>>> They kind of shut that down, the people managing it have gone away
>>> 
>>> https://mailarchive.ietf.org/arch/msg/hybi/YVBJTzJcvzytIY46KfIS8bch
>>> f5E
>>> 
>>> the ML is still running but is very quiet.
>>> 
>>> WS itself seems to be in rude health out there.
>>> 
>>>> What *is* in-scope here is how (if at all) that protocol
>>>> interacts
>>>> with HTTP, including HTTP/2; there are several ways you could
>>>> implement WebSockets over HTTP/2, and a few pitfalls in doing so
>>>> that
>>>> the people on this list will be able to give you feedback on.
>>> 
>>> It's unfortunate that wasn't considered part of defining HTTP/2, so
>>> it
>>> could be baked in.   The subject was certainly raised.  But I can
>>> understand the desire to get the main business out of the door.
>>> 
>>>> However, it's hard to do that before there's agreement in the WS
>>>> community about what the requirements are. Ideally, that
>>>> community
>>>> would bring a single proposal that has broad support here for
>>>> review.
>>> 
>>> Any formal mechanism to manage that has gone away for hybi
>>> AFAICT.  So
>>> this "let the community do it" feels like a bit of a cop-out /
>>> bullet
>>> dodging.
>>> 
>>> At any rate I think the number of people interested in HTTP/2 WS is
>>> still very low compared to the number of people interested in WS
>>> client
>>> and server generally in my experience.  But at some point that will
>>> change, possibly suddenly.
>>> 
>>> -Andy
>>> 
>>>> Cheers,
>>>> 
>>>> 
>>>>> On 25 Nov. 2016, at 5:39 am, Van Catha <[e-mail]>
>>>>> wrote:
>>>>> 
>>>>> Thanks for clarification. Unfortunate that so little attention
>>>>> was
>>>>> paid to this.  Looks like some of us will be on HTTP1.1 for a
>>>>> long
>>>>> time.
>>>>> 
>>>>> On Mon, Nov 21, 2016 at 11:14 PM, Takeshi Yoshino <tyoshino@goo
>>>>> gle.
>>>>> com> wrote:
>>>>> Ah, no. Martin just warned us that we might face the same issue
>>>>> that SSE faced.
>>>>> 
>>>>> Mark's suggestion is a separate thing. The co-chairs (Mark and
>>>>> Patrick) said that this (WiSH) doesn't seems to be a topic that
>>>>> should be discussed in the HTTP WG given the charter of the WG,
>>>>> I
>>>>> think.
>>>>> 
>>>>> On Sun, Nov 20, 2016 at 12:26 PM, Van Catha <[e-mail]>
>>>>> wrote:
>>>>> I do not understand what this means.  Is the suggestion to
>>>>> ignore
>>>>> WiSH for now in favor of SSE?
>>>>> 
>>>>> On Fri, Nov 18, 2016 at 1:55 AM, Takeshi Yoshino <tyoshino@goog
>>>>> le.c
>>>>> om> wrote:
>>>>> I'd like to share the feedback on WiSH from IETF 97.
>>>>> 
>>>>> ----
>>>>> 
>>>>> Due to limited time, I got just one on-site comment from Martin
>>>>> about comparison with Server-sent event (EventSource).
>>>>> 
>>>>> As mentioned in the I-D, yes, this is kinda full-duplex SSE
>>>>> with
>>>>> the WS framing, and it might suffer from unexpected buffering
>>>>> by
>>>>> intermediaries if any as Martin said.
>>>>> 
>>>>> WiSH should work well for deployment with TLS only (possibly
>>>>> with
>>>>> some non-TLS part beyond server side front-end but are under
>>>>> control of the service providers). Given the wide trend of
>>>>> encouraging TLS and browser vendors' implementation status of
>>>>> H2, I
>>>>> think we should prioritize layering simplicity than taking care
>>>>> of
>>>>> gain of WiSH/H2/TCP + transparent proxy (with unexpected
>>>>> buffering)
>>>>> case. For H2-less TLS-less environment, we can just use the
>>>>> WebSocket protocol.
>>>>> 
>>>>> There can still be some risk of MITM (trusted) proxy and
>>>>> unexpected
>>>>> buffering with AntiVirus/Firewall for deployment with TLS, but
>>>>> other WebSocket/H2 mapping proposals also have issues of
>>>>> possible
>>>>> blocking, buffering, etc. WebSocket/TCP's handshake success
>>>>> rate
>>>>> for non-TLS port 80 was also not so good when it started
>>>>> getting
>>>>> deployed, and got improved gradually. I think the problems will
>>>>> get
>>>>> resolved once WiSH is accepted widely, and I believe the total
>>>>> pain
>>>>> and cost would be smaller.
>>>>> 
>>>>> ----
>>>>> 
>>>>> Mark suggested that we should find some other right place than
>>>>> HTTP
>>>>> WG. I'll discuss this with Mark. Maybe we'll consult the
>>>>> DISPATCH
>>>>> WG.
>>>>> 
>>>>> ----
>>>>> 
>>>>> Thanks everyone for the feedback.
>>>>> 
>>>>> Takeshi
>>>>> 
>>>>> On Thu, Nov 3, 2016 at 3:20 AM, Costin Manolache <costin@gmail.
>>>>> com>
>>>>> wrote:
>>>>> Good timing -  http://httpwg.org/http-extensions/encryption-pre
>>>>> view
>>>>> .html is addressing my concerns for
>>>>> webpush ( and general 'encrypted content' ), we're still
>>>>> discussing
>>>>> some details, but for this use
>>>>> case metadata won't be needed.
>>>>> 
>>>>> Costin
>>>>> 
>>>>> 
>>>>> On Tue, Nov 1, 2016 at 10:34 PM Takeshi Yoshino <tyoshino@googl
>>>>> e.co
>>>>> m> wrote:
>>>>> On Mon, Oct 31, 2016 at 5:57 AM, Wenbo Zhu <[e-mail]>
>>>>> wrote:
>>>>> 
>>>>> 
>>>>> On Sun, Oct 30, 2016 at 10:25 AM, Costin Manolache <costin@gmai
>>>>> l.co
>>>>> m> wrote:
>>>>> Thanks for the answer and pointers. From earlier responses, it
>>>>> seems possible to use GET
>>>>> or a non-web-stream request to would avoid the extra cost of
>>>>> the
>>>>> pre-flight.
>>>>> 
>>>>> 
>>>>> Yeah, at least the Content-Type in the HTTP request gets
>>>>> eliminated.
>>>>>  
>>>>> One more question/issue: in some cases it would be good to send
>>>>> some
>>>>> metadata (headers) along with binary frames. For example in
>>>>> webpush
>>>>> the content is an encrypted
>>>>> blob, and needs headers for the key/salt. I would assume a lot
>>>>> of
>>>>> other 'binary' messages would
>>>>> benefit if simple metadata could be sent along. Would it be
>>>>> possible to use one of the reserved
>>>>> bits for 'has metadata' and add some encoded headers ? I know
>>>>> in
>>>>> websocket they are intended 
>>>>> for 'extensions', but 'headers' seems a very common use case.
>>>>> Q about webpush: is the metadata different for each binary
>>>>> message? 
>>>>> 
>>>>> We discussed about metadata and how to use one of RSV bits etc.
>>>>> For
>>>>> the current version, let's make sure the WS compatibility is
>>>>> fully
>>>>> addressed (with minimum wire encoding like WiSH)
>>>>> 
>>>>> Agreed. Let's discuss the metadata needs separately. I agree
>>>>> it's
>>>>> important.
>>>>>  
>>>>>  
>>>>> 
>>>>> Having the binary frame use some MIME encoding to pass both
>>>>> text
>>>>> headers and the binary blob
>>>>> is possible - but has complexity and overhead.
>>>>> OTOH, if the binary blob relies on text headers (metata) to be
>>>>> useful, then you probably need define a dedicated MIME
>>>>> encoding.
>>>>> 
>>>>>   
>>>>> 
>>>>> Costin
>>>>> 
>>>>> On Sun, Oct 30, 2016 at 5:27 AM Takeshi Yoshino <tyoshino@googl
>>>>> e.co
>>>>> m> wrote:
>>>>> Thanks, Van, Costin.
>>>>> 
>>>>> On Sun, Oct 30, 2016 at 2:43 AM, Costin Manolache <costin@gmail
>>>>> .com
>>>>>> wrote:
>>>>> 
>>>>> Good point - websocket is widely deployed, including IoT - and
>>>>> the
>>>>> header is pretty easy to handle anyways.
>>>>> +1.
>>>>> 
>>>>> One question: is this intended to be handled by browsers, and
>>>>> exposed using the W3C websocket API ?
>>>>> Will a regular app be able to make WiSH requests and parse the
>>>>> stream by itself, without browser
>>>>> interference ? And if yes, any advice on how it interact with
>>>>> CORS
>>>>> ? 
>>>>> 
>>>>> The first step would be using Streams based upload/download via
>>>>> the
>>>>> Fetch API + protocol processing in JS.
>>>>> 
>>>>> The next step could be either introduction of an optimized
>>>>> native
>>>>> implementation of WiSH parser/framer in the form of the
>>>>> TransformStream which can be used as follows:
>>>>> 
>>>>> const responsePromise = fetch(url, init);
>>>>> responsePromise.then(response => {
>>>>>   const wishStream =
>>>>> response.body().pipeThrough(wishTransformStream);
>>>>>   function readAndProcessMessage() {
>>>>>     const readPromise = wishStream.read();
>>>>>     readPromise.then(result => {
>>>>>       if (result.done) {
>>>>>         // End of stream.
>>>>>         return;
>>>>>       }
>>>>> 
>>>>>       const message = result.value;
>>>>>       // Process the message
>>>>>       // E.g. access message.opcode for opcode, message.body
>>>>> for
>>>>> the body data
>>>>>       readAndProcessMessage();
>>>>>     });
>>>>>   }
>>>>>   readAndProcessMessage();
>>>>> });
>>>>> 
>>>>> and provide a polyfill that presents this as the WebSocket API,
>>>>> and
>>>>> (or skip the step and) go further i.e. native implementation
>>>>> for
>>>>> everything if it turns out optimization is critical.
>>>>> 
>>>>> We need to discuss this also in W3C/WHATWG.
>>>>> 
>>>>> Regarding CORS, if the request includes non CORS-safelisted
>>>>> headers, fetch() based JS polyfills will be basically subject
>>>>> to
>>>>> the CORS preflight requirement. We could try to exempt some of
>>>>> well
>>>>> defined headers if any for CORS like WebSocket handshake's
>>>>> headers
>>>>> and server-sent event's Last-Event-Id are exempted. Regarding
>>>>> the
>>>>> proposed subprotocol negotiation in the form of combination of
>>>>> the
>>>>> Accept header and the Content-Type header, the Accept header is
>>>>> one
>>>>> of the CORS-safelisted headers, so it's not a problem. The
>>>>> Content-
>>>>> Type header is considered to be non-CORS-safelisted if it's
>>>>> value
>>>>> is none of the CORS-safelisted media types. So, WiSH media type
>>>>> would trigger the preflight unless we exclude it.
>>>>> 
>>>>> Origin policy https://wicg.github.io/origin-policy/ might also
>>>>> help.
>>>>>  
>>>>> 
>>>>> Costin
>>>>> 
>>>>> On Fri, Oct 28, 2016 at 12:06 PM Takeshi Yoshino <tyoshino@goog
>>>>> le.c
>>>>> om> wrote:
>>>>> Sorry for being ambivalent.
>>>>> 
>>>>> We can of course revisit each design decision we made for RFC
>>>>> 6455
>>>>> framing and search for the optimal again. But as:
>>>>> - one of the main philosophies behind WiSH is compatibility
>>>>> with
>>>>> WebSocket in terms of both spec and implementation
>>>>> - the WebSocket is widely deployed and therefore we have a lot
>>>>> of
>>>>> implementations in various languages/platform
>>>>> - most browsers already have logic for the framing
>>>>> - the framing is not considered to be so big pain
>>>>> inheriting the WebSocket framing almost as-is is just good
>>>>> enough.
>>>>> Basically, I'm leaning toward this plan.
>>>>> 
>>>>> Takeshi
>>>>> 
>>>>> On Sat, Oct 29, 2016 at 3:12 AM, Takeshi Yoshino <tyoshino@goog
>>>>> le.c
>>>>> om> wrote:
>>>>> On Sat, Oct 29, 2016 at 2:55 AM, Loïc Hoguin <essen@ninenines.e
>>>>> u>
>>>>> wrote:
>>>>> On 10/28/2016 08:41 PM, Costin Manolache wrote:
>>>>> Current overhead is 2 bytes if frame is up to 125 bytes long -
>>>>> which I
>>>>> think it's not very common,
>>>>> 4 bytes for up to 64k, and 10 bytes for anything larger.
>>>>> IMHO adding one byte - i.e. making it fixed 5-byte, with first
>>>>> as
>>>>> is,
>>>>> and next 4 fixed length would
>>>>> be easiest to parse.
>>>>> 
>>>>> Is making it easy (or easier) to parse even a concern anymore?
>>>>> 
>>>>> Considering the number of agents and servers already supporting
>>>>> Websocket, the numerous libraries for nearly all languages and
>>>>> the
>>>>> great autobahntestsuite project validating it all, reusing the
>>>>> existing code is a very sensible solution.
>>>>> 
>>>>> 
>>>>> Yeah, I've been having similar feeling regarding cost for
>>>>> parser/encoder implementation though I might be biased.
>>>>>  
>>>>> There are obviously too many options to encode and each has
>>>>> benefits -
>>>>> my only concern was
>>>>> that the choice of 1, 2, 8 bytes for length may not match
>>>>> common
>>>>> sizes.
>>>>> 
>>>>> ( in webpush frames will be <4k ).
>>>>> 
>>>>> -- 
>>>>> Loïc Hoguin
>>>>> https://ninenines.eu
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> Mark Nottingham   https://www.mnot.net/
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> --
>> Mark Nottingham   https://www.mnot.net/
>> 
>> 
>> 
>> 
>> 

--
Mark Nottingham   https://www.mnot.net/