Re: WiSH: A General Purpose Message Framing over Byte-Stream Oriented Wire Protocols (HTTP)

[Mark Nottingham <[e-mail]>]

Andy,

The "people managing it" are the IETF. "Let the community do it" is how the IETF works, with guidance from IETF leadership. I had a discussion with our Area Director about this topic in Seoul, and we agreed that the hybi list was the best place to talk about WebSockets.

Cheers,


> On 25 Nov. 2016, at 11:51 am, Andy Green <[e-mail]> wrote:
> 
> On Fri, 2016-11-25 at 10:35 +1100, Mark Nottingham wrote:
>> To clarify, we said that the future of WebSockets really isn't in
>> scope for this WG; the proper venue for discussing that is:
>>   https://www.ietf.org/mailman/listinfo/hybi
> 
> They kind of shut that down, the people managing it have gone away
> 
> https://mailarchive.ietf.org/arch/msg/hybi/YVBJTzJcvzytIY46KfIS8bchf5E
> 
> the ML is still running but is very quiet.
> 
> WS itself seems to be in rude health out there.
> 
>> What *is* in-scope here is how (if at all) that protocol interacts
>> with HTTP, including HTTP/2; there are several ways you could
>> implement WebSockets over HTTP/2, and a few pitfalls in doing so that
>> the people on this list will be able to give you feedback on.
> 
> It's unfortunate that wasn't considered part of defining HTTP/2, so it
> could be baked in.   The subject was certainly raised.  But I can
> understand the desire to get the main business out of the door.
> 
>> However, it's hard to do that before there's agreement in the WS
>> community about what the requirements are. Ideally, that community
>> would bring a single proposal that has broad support here for review.
> 
> Any formal mechanism to manage that has gone away for hybi AFAICT.  So
> this "let the community do it" feels like a bit of a cop-out / bullet
> dodging.
> 
> At any rate I think the number of people interested in HTTP/2 WS is
> still very low compared to the number of people interested in WS client
> and server generally in my experience.  But at some point that will
> change, possibly suddenly.
> 
> -Andy
> 
>> Cheers,
>> 
>> 
>>> On 25 Nov. 2016, at 5:39 am, Van Catha <[e-mail]> wrote:
>>> 
>>> Thanks for clarification. Unfortunate that so little attention was
>>> paid to this.  Looks like some of us will be on HTTP1.1 for a long
>>> time.
>>> 
>>> On Mon, Nov 21, 2016 at 11:14 PM, Takeshi Yoshino <tyoshino@google.
>>> com> wrote:
>>> Ah, no. Martin just warned us that we might face the same issue
>>> that SSE faced.
>>> 
>>> Mark's suggestion is a separate thing. The co-chairs (Mark and
>>> Patrick) said that this (WiSH) doesn't seems to be a topic that
>>> should be discussed in the HTTP WG given the charter of the WG, I
>>> think.
>>> 
>>> On Sun, Nov 20, 2016 at 12:26 PM, Van Catha <[e-mail]>
>>> wrote:
>>> I do not understand what this means.  Is the suggestion to ignore
>>> WiSH for now in favor of SSE?
>>> 
>>> On Fri, Nov 18, 2016 at 1:55 AM, Takeshi Yoshino <tyoshino@google.c
>>> om> wrote:
>>> I'd like to share the feedback on WiSH from IETF 97.
>>> 
>>> ----
>>> 
>>> Due to limited time, I got just one on-site comment from Martin
>>> about comparison with Server-sent event (EventSource).
>>> 
>>> As mentioned in the I-D, yes, this is kinda full-duplex SSE with
>>> the WS framing, and it might suffer from unexpected buffering by
>>> intermediaries if any as Martin said.
>>> 
>>> WiSH should work well for deployment with TLS only (possibly with
>>> some non-TLS part beyond server side front-end but are under
>>> control of the service providers). Given the wide trend of
>>> encouraging TLS and browser vendors' implementation status of H2, I
>>> think we should prioritize layering simplicity than taking care of
>>> gain of WiSH/H2/TCP + transparent proxy (with unexpected buffering)
>>> case. For H2-less TLS-less environment, we can just use the
>>> WebSocket protocol.
>>> 
>>> There can still be some risk of MITM (trusted) proxy and unexpected
>>> buffering with AntiVirus/Firewall for deployment with TLS, but
>>> other WebSocket/H2 mapping proposals also have issues of possible
>>> blocking, buffering, etc. WebSocket/TCP's handshake success rate
>>> for non-TLS port 80 was also not so good when it started getting
>>> deployed, and got improved gradually. I think the problems will get
>>> resolved once WiSH is accepted widely, and I believe the total pain
>>> and cost would be smaller.
>>> 
>>> ----
>>> 
>>> Mark suggested that we should find some other right place than HTTP
>>> WG. I'll discuss this with Mark. Maybe we'll consult the DISPATCH
>>> WG.
>>> 
>>> ----
>>> 
>>> Thanks everyone for the feedback.
>>> 
>>> Takeshi
>>> 
>>> On Thu, Nov 3, 2016 at 3:20 AM, Costin Manolache <[e-mail]>
>>> wrote:
>>> Good timing -  http://httpwg.org/http-extensions/encryption-preview
>>> .html is addressing my concerns for
>>> webpush ( and general 'encrypted content' ), we're still discussing
>>> some details, but for this use
>>> case metadata won't be needed.
>>> 
>>> Costin
>>> 
>>> 
>>> On Tue, Nov 1, 2016 at 10:34 PM Takeshi Yoshino <[e-mail]
>>> m> wrote:
>>> On Mon, Oct 31, 2016 at 5:57 AM, Wenbo Zhu <[e-mail]>
>>> wrote:
>>> 
>>> 
>>> On Sun, Oct 30, 2016 at 10:25 AM, Costin Manolache <[e-mail]
>>> m> wrote:
>>> Thanks for the answer and pointers. From earlier responses, it
>>> seems possible to use GET
>>> or a non-web-stream request to would avoid the extra cost of the
>>> pre-flight.
>>> 
>>> 
>>> Yeah, at least the Content-Type in the HTTP request gets
>>> eliminated.
>>>  
>>> One more question/issue: in some cases it would be good to send
>>> some
>>> metadata (headers) along with binary frames. For example in webpush
>>> the content is an encrypted
>>> blob, and needs headers for the key/salt. I would assume a lot of
>>> other 'binary' messages would
>>> benefit if simple metadata could be sent along. Would it be
>>> possible to use one of the reserved
>>> bits for 'has metadata' and add some encoded headers ? I know in
>>> websocket they are intended 
>>> for 'extensions', but 'headers' seems a very common use case.
>>> Q about webpush: is the metadata different for each binary
>>> message? 
>>> 
>>> We discussed about metadata and how to use one of RSV bits etc. For
>>> the current version, let's make sure the WS compatibility is fully
>>> addressed (with minimum wire encoding like WiSH)
>>> 
>>> Agreed. Let's discuss the metadata needs separately. I agree it's
>>> important.
>>>  
>>>  
>>> 
>>> Having the binary frame use some MIME encoding to pass both text
>>> headers and the binary blob
>>> is possible - but has complexity and overhead.
>>> OTOH, if the binary blob relies on text headers (metata) to be
>>> useful, then you probably need define a dedicated MIME encoding.
>>> 
>>>   
>>> 
>>> Costin
>>> 
>>> On Sun, Oct 30, 2016 at 5:27 AM Takeshi Yoshino <[e-mail]
>>> m> wrote:
>>> Thanks, Van, Costin.
>>> 
>>> On Sun, Oct 30, 2016 at 2:43 AM, Costin Manolache <[e-mail]
>>>> wrote:
>>> Good point - websocket is widely deployed, including IoT - and the
>>> header is pretty easy to handle anyways.
>>> +1.
>>> 
>>> One question: is this intended to be handled by browsers, and
>>> exposed using the W3C websocket API ?
>>> Will a regular app be able to make WiSH requests and parse the
>>> stream by itself, without browser
>>> interference ? And if yes, any advice on how it interact with CORS
>>> ? 
>>> 
>>> The first step would be using Streams based upload/download via the
>>> Fetch API + protocol processing in JS.
>>> 
>>> The next step could be either introduction of an optimized native
>>> implementation of WiSH parser/framer in the form of the
>>> TransformStream which can be used as follows:
>>> 
>>> const responsePromise = fetch(url, init);
>>> responsePromise.then(response => {
>>>   const wishStream =
>>> response.body().pipeThrough(wishTransformStream);
>>>   function readAndProcessMessage() {
>>>     const readPromise = wishStream.read();
>>>     readPromise.then(result => {
>>>       if (result.done) {
>>>         // End of stream.
>>>         return;
>>>       }
>>> 
>>>       const message = result.value;
>>>       // Process the message
>>>       // E.g. access message.opcode for opcode, message.body for
>>> the body data
>>>       readAndProcessMessage();
>>>     });
>>>   }
>>>   readAndProcessMessage();
>>> });
>>> 
>>> and provide a polyfill that presents this as the WebSocket API, and
>>> (or skip the step and) go further i.e. native implementation for
>>> everything if it turns out optimization is critical.
>>> 
>>> We need to discuss this also in W3C/WHATWG.
>>> 
>>> Regarding CORS, if the request includes non CORS-safelisted
>>> headers, fetch() based JS polyfills will be basically subject to
>>> the CORS preflight requirement. We could try to exempt some of well
>>> defined headers if any for CORS like WebSocket handshake's headers
>>> and server-sent event's Last-Event-Id are exempted. Regarding the
>>> proposed subprotocol negotiation in the form of combination of the
>>> Accept header and the Content-Type header, the Accept header is one
>>> of the CORS-safelisted headers, so it's not a problem. The Content-
>>> Type header is considered to be non-CORS-safelisted if it's value
>>> is none of the CORS-safelisted media types. So, WiSH media type
>>> would trigger the preflight unless we exclude it.
>>> 
>>> Origin policy https://wicg.github.io/origin-policy/ might also
>>> help.
>>>  
>>> 
>>> Costin
>>> 
>>> On Fri, Oct 28, 2016 at 12:06 PM Takeshi Yoshino <tyoshino@google.c
>>> om> wrote:
>>> Sorry for being ambivalent.
>>> 
>>> We can of course revisit each design decision we made for RFC 6455
>>> framing and search for the optimal again. But as:
>>> - one of the main philosophies behind WiSH is compatibility with
>>> WebSocket in terms of both spec and implementation
>>> - the WebSocket is widely deployed and therefore we have a lot of
>>> implementations in various languages/platform
>>> - most browsers already have logic for the framing
>>> - the framing is not considered to be so big pain
>>> inheriting the WebSocket framing almost as-is is just good enough.
>>> Basically, I'm leaning toward this plan.
>>> 
>>> Takeshi
>>> 
>>> On Sat, Oct 29, 2016 at 3:12 AM, Takeshi Yoshino <tyoshino@google.c
>>> om> wrote:
>>> On Sat, Oct 29, 2016 at 2:55 AM, Loïc Hoguin <[e-mail]>
>>> wrote:
>>> On 10/28/2016 08:41 PM, Costin Manolache wrote:
>>> Current overhead is 2 bytes if frame is up to 125 bytes long -
>>> which I
>>> think it's not very common,
>>> 4 bytes for up to 64k, and 10 bytes for anything larger.
>>> IMHO adding one byte - i.e. making it fixed 5-byte, with first as
>>> is,
>>> and next 4 fixed length would
>>> be easiest to parse.
>>> 
>>> Is making it easy (or easier) to parse even a concern anymore?
>>> 
>>> Considering the number of agents and servers already supporting
>>> Websocket, the numerous libraries for nearly all languages and the
>>> great autobahntestsuite project validating it all, reusing the
>>> existing code is a very sensible solution.
>>> 
>>> 
>>> Yeah, I've been having similar feeling regarding cost for
>>> parser/encoder implementation though I might be biased.
>>>  
>>> There are obviously too many options to encode and each has
>>> benefits -
>>> my only concern was
>>> that the choice of 1, 2, 8 bytes for length may not match common
>>> sizes.
>>> 
>>> ( in webpush frames will be <4k ).
>>> 
>>> -- 
>>> Loïc Hoguin
>>> https://ninenines.eu
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> --
>> Mark Nottingham   https://www.mnot.net/
>> 
>> 
>> 
>> 
>> 

--
Mark Nottingham   https://www.mnot.net/