Re: [hybi] WS framing alternative

[Ian Hickson <[e-mail]>]

On Thu, 29 Oct 2009, SM wrote:
> At 21:22 29-10-2009, Ian Hickson wrote:
> > What is the process for proceeding in the IETF when people have 
> > fundamentally different and mutually exclusive opinions?
> 
> The group seeks rough consensus.  I guess that you could think of it as 
> the view that reconciles the various opinions.  That does not require 
> everyone to agree; although that's better.  It's not a hard rule; 
> there's no voting.  It's up to the Working Group Chair to evaluate the 
> consensus.

I guess we'd better get a chair. :-)


On Fri, 30 Oct 2009, Pieter Hintjens wrote:
> 
> Technical designs should not be opinion-driven at all.  There are
> optimal (provably optimal) solutions for any well-defined problem.  If
> two people do not agree on such solutions then either they are solving
> different problems (this is most common), or they are not aiming for
> optimal solutions but instead familiar and/or pre-existing solutions
> (this is also common).  Or they are just incompetent with delusions of
> competence (perhaps most common).

I believe the problem we have here is that we don't agree on what problem 
we are solving, though apparently Greg disagrees with me on that as well.


> A better way to resolve the conflicts of interest that many of us bring 
> to such discussions is to create competition between solutions, i.e. 
> multiple protocols that implementors can choose.  Over time that will 
> drive the solutions towards an optimum.

I agree that that is the best option. I thought that's what we were doing 
with WebSocket, BWTP, and other proposals, but advocates of solutions 
other than WebSocket have not been making much progress as far as I can 
tell. I personally would very much encourage such competition. I think 
it's very healthy and as you say, will drive towards an optimum solution.


On Fri, 30 Oct 2009, SM wrote:
> 
> Having multiple protocols for the same problem space doesn't favor 
> interoperability.  One or more of the protocols will take over which is 
> like "let the market decide".

Indeed, that's another way to put it.


On Sat, 31 Oct 2009, Greg Wilkins wrote:
>
> After initial failures to get my concerns addressed in WS, I did decide 
> to propose alternative solutions. BWTP mk I and BWTP mk II resulted.
> 
> But now it appears that a working group cannot be formed to consider 
> multiple solutions and that it will be free to consider any solution so 
> long as it is websocket.

I strongly object to us forming a working group that only considers one 
solution. So on this at least I strongly agree with Greg that we should 
support other efforts including BWTP. Why would we disallow other 
solutions than WebSocket?


On Fri, 30 Oct 2009, Thomson, Martin wrote:
> 
> Thankfully, we aren't arguing in a vacuum.  I'm happy to cede to the 
> will of a majority on the issue, just as I hope that you are.  And 
> deployment counts for a lot.  Until I feel that I'm arguing without 
> support, I will continue to argue for a text-based protocol with 
> (marginally) greater provision for extension than currently exists.

Then I shall similarly continue to argue for what I perceive to be the 
simpler and safer solution. :-)


On Fri, 30 Oct 2009, Greg Wilkins wrote:
> Ian Hickson wrote:
> > I'm proposing making the protocol so simple that dedicated frameworks 
> > are superfluous.
> 
> You are proposing a revolution in server infrastructure.

I don't think server infrastructure is anywhere near as homogeneous as 
that would suggest. There are some environments I'm familiar with -- both 
small (e.g. on a shared Web host) and large (e.g. in a dedicated 
datacenter) -- where deploying experimental WebSocket servers has not been 
a revolution at all, but has fit right in. I am sure you are right that in 
certain environments it would be a revolution, and I'm sure that there 
will be many cases in between.

The revolution, however, if any, is a one-time cost. We must ensure, 
therefore, that anything we do to defray this one-time cost is not going 
to cause long-term costs to be higher instead.

If you have specific proposals that would help with avoiding a revolution, 
then I'm happy to hear them, but we need to make sure they don't 
complicate the protocol, or make it harder to deploy WebSocket in 
environments where it's easy already, since if we do then we're just 
moving a one-time cost from one subset of today's authors to a permanent 
cost for all authors going forward.


> There is just no way that any of the organizations that I work with are 
> going to allow a server side application developer access to a raw 
> socket exposed on the internet!

WebSocket doesn't require that you do that. You don't need a MIME envelope 
mechanism to abstract out WebSocket so that libraries can be written that 
just receive an already-parsed packet. Nor do you need server-client 
negotiation for the app to inform the library what size packets to expect.


> No matter how much you think it a bad idea, there will be frameworks 
> that implement websocket on the server side - and they will have little 
> or no idea of the type of application that might be run on top of them.

That depends entirely on how the frameworks are written. There's no reason 
that the framework couldn't provide a framework to implement custom 
protocols such that the framework does all the heavy lifting and thus 
_does_ know what the protocol is. 


> >> If a ws client connects to a ws server, and neither knows if the 
> >> other supports some layered protocol - how do they discover that?
> > 
> > If a TCP client connects to a TCP server, and neither knows if the 
> > other supports some layered protocol - how do they discover that?
> > 
> > The answer is the same.
> 
> No it's not.
> 
> You run an *extensible* protocol like HTTP over TCP, so that new 
> protocols can be layered on top without breaking clients, servers or 
> intermediaries.

So where does this leave IMAP, SSH, Telnet, FTP, Gopher, and SMB?


> > That is, I do not see how our goals could ever be met by the same 
> > protocol, if I understand your goals correctly.
> 
> I don't see how allocating 1 bit of 7 unused can render the protocol 
> unsuitable for your goals.
> 
> You are being a tad melodramatic I think.

I really don't think adding a bit would do what you think it would. Unless 
we also complicate the API, there'd be no way to use this bit anyway. I 
really don't understand the scenario in which you think this would help.

My goal is to have the protocol be as simple as possible. Adding things to 
the protocol makes it less simple. That's how it renders the protocol 
unsuitable to my goals.


> >> I'm asking for 1 bit to be allocated to indicate that a frame 
> >> contains mime-encoded data.  That is mostly sufficient for layered 
> >> protocols to be built on.
>
> I would reuse the text and binary framing mechanism.

Why are frame types 0x00 and 0x80 not good enough then?


> but how do you know if the other end supports mime encoding at all. you 
> end up needing to do all sorts of discovery and other complexities.

You do it in the exact same way that you know that the other side supports 
IMAP. Or HTTP. Or SSH. Or SMB. Or Jabber.

You know because _that's why you're connecting_.

Why would you ever connect to a server that supported a protocol, if you 
weren't expecting to speak that protocol?

I really don't understand the use case that leads you to this line of 
argumentation.


> Browser vendors and server vendors might want to collaborate on a 
> multiplexing protocol built on top ws.  They could use the the mime 
> frames to do this - but they would be able to continue to work with 
> browsers and servers that did not implement the extension - as they 
> would ignore meta data they didn't understand that was used to try to 
> negotiate a multi-channel connection.

If a protocol layered on top of WebSocket wanted to support multiplexing, 
it could just go ahead and do that as part of its protocol.

If, on the other hand, there was a need for a protocol that was a layer 
over TCP like WebSocket but which had built-in multiplexing (for whatever 
reason), then you could do that just by inventing a new protocol, and 
using that. WebMultiplexingProtocol, or something. You would want to 
change the whole way the handshaking is done if you did this.


> The extra capabilities could be used by the browser, intermediaries 
> and/or servers.  They need not be exposed to application developers, you 
> could even argue that they shouldn't be exposed to application 
> developers.

The browsers aren't doing anything other than what the scripts want them 
to do. The servers have no reason to support anything the clients aren't 
going to use, which is what the scripts are going to do.

I really, truly do not understand how you expect this feature to be used. 
You need to explain a concrete use case, with actual examples of actual 
user applications and actual on-the-wire traffic explaining to me what you 
expect to have happen, because I have completely failed to understand the 
more abstract use cases you have given so far.


> > I think overgeneralisation is one of the biggest mistakes one can make 
> > in protocol and language design. My experience has led me to be a firm 
> > believer in picking specific problems and addressing them directly, 
> > rather than trying to solve everything at once (especially when one 
> > might not know all the problems one is trying to solve -- a common 
> > problem when discussing extensibility mechanisms).
> 
> You have already generalized the protocol by allowing for other frame 
> types.

How is that generalisation? It's addressing two very specific needs: 
sending text, and sending binary data (files) in v2.


> You obviously want to control how people extend WS and will only 
> allocate new frame byte values to what you think is worthy cause.

The extension mechanism is intended for providing new features in the 
future, specifically sending binary files, and sending binary streams 
(e.g. video). It's not a generic extension mechanism for protocol users.


> If there was a meta data frames, then anybody could extend the protocol 
> without the need to ask permission ( even if it was to support evil 
> monolithic servers and their unholy load balancing companions).

Extending the protocol in this manner is actively harmful, since it 
reduces the likelihood of interoperability between clients and servers.

If you want to implement new features, you do it at the application layer, 
like *all* features.

This is exactly like TCP. People don't extend TCP to add their features. 
We didn't extend TCP to add MIME encoding. We didn't extend TCP to add URI 
addressing. We didn't extend TCP to add caching or intermediary support. 
We layered a protocol on _top_ of TCP, namely HTTP.

The same applies to WebSocket.


> What authority do you plan to manage the allocation of frame types into 
> the future?  IANA?

The same authority as any other changes to the protocol. The IETF and 
WHATWG specification update processes.


> >> So browsers could add on their origin security model for example.
> > 
> > Security is not a feature you bolt on. It's an architecture that has 
> > to be designed in from the beginning.
> 
> RUBBISH!

...


> If this was the case we would have to put the same origin policy into 
> TCP... no wait into IP... no wait into ethernet..

TCP bakes its security in as part of its handshake. It's an intrinsic part 
of the TCP protocol. It wouldn't make sense to have origin security at the 
TCP level, since that's an application concern (relative to TCP).


> You can design a good protocol for the security needs of
> browser that is built on a more general transport protocol.

That is exactly what WebSocket is. It's a protocol for the security needs 
of browsers built on a more general transport protocol called TCP.


> > Could I ask you to more concretely describe your problem so that I 
> > could more concretely (maybe with actual code examples) describe the 
> > alternative solution that I am describing?
> 
> sticky load balancer with SSL offload that works with any ws app.
> 
> multiplexing protocol implemented in browser/server (not in 
> application).
> 
> orderly closing a ws connection implemented in an application 
> independent way.

If you could provide concrete solid examples, that would be very helpful. 
What would you need to do a "sticky load balancer with SSL offload" other 
than what TCP gives you already? What is the use case for the browser 
implementing multiplexing rather than a SharedWorker from script? (I don't 
buy that authors are going to intentionally DDOS themselves, so if they 
need multiplexing, it seems very likely that they will implement it.) Why 
does TCP's socket closing semantics not address your needs?



On Fri, 30 Oct 2009, Jamie Lokier wrote:
> > 
> > sticky load balancer with SSL offload that works with any ws app.
> 
> This is the key thing, I believe.
> 
> WebSockets forces all negotiation of extensions to be done by the 
> application.
>
> This prevents automatic extensions being done below the application, so 
> "works with any WS app" is prevented by the current proposal.
> 
> Here's an example: Just like HTTP, there could be value in an extension 
> which compresses some frames.  If it worked well, that'd be a good thing 
> to do _transparently_ in the browser+server, without awareness by the WS 
> application.

Sure. We could add compression in a future version of WebSocket.

We would do it as follows:

   Client->Server handshake would include a new header, 
   WebSocket-Compression: gzip

   Server-Client handshake would include the same new header, 
   WebSocket-Compression: gzip

   If the client and the server both sent the header, then instead of 
   using 0x00, the client would use 0x81, and send text data gzip 
   compressed, and instead of using 0x80, the client would use 0x82, and 
   send the binary data gzip compressed.

   The server could use either 0x00 or 0x81 to send text frames, and could 
   use either 0x80 or 0x82 to send binary frames; 0x81 and 0x82 would be 
   gzip compressed.

This would be a sensible thing to support in a future version if we decide 
that implementation experience shows compression to be highly desireable. 
(It's not obviously desireable, or I'd have added it already. Reasons why 
we might avoid it are that compression is not always useful for short 
strings, which are probably going to be most common; and that it's 
possible for compression to be introduced at the TLS level, thus making 
the protocol's use of compression unnecessary.)


> But you can't provide a transparent compression extension, because the 
> browser has no way to signal to the server "I handle this extension".

Sure it can. The handshake is completely extensible.


> Given that the only thing blocking this is the client's inability to 
> signal the server that it supports an extension, and that is so easy and 
> natural to do by allowing clients to add headers to the initial request, 
> I think that should be at least seriously considered.  Not dismissed 
> summarily.

We absolutely would not want clients and servers doing compression without 
a spec defining how it works. Said spec can allow sending headers to opt- 
in to compression, as well as defining which frames to use. It would be 
actively harmful for the WebSocket spec _without_ compression to say that 
clients could add arbitrary headers: interoperability would be a nightmare 
where implementations have to reverse-engineer each other to work out what 
they need to support.


> > multiplexing protocol implemented in browser/server (not in application).
> 
> That's similar to the compression example above.

If we want multiplexing that is transparent to scripts (as opposed to 
layered on top of WebSocket, which seems to make more sense to me), then 
we need a whole new protocol; WebSocket just isn't cut out to do built-in 
multiplexing. (It handles higher-level multiplexing fine.) We would quite 
easily be able to do this again by having a handshake that looks like 
WebSocket v1, but which the server and client can recognise as being just 
a dance to opt into v2.

For example, we could have the handshake be:

        GET /foo/bar HTTP/1.1
        Upgrade: WebSocket
        Connection: Upgrade
        Host: example.com
        Origin: http://example.com
        WebSocket-Upgrade: Multiplexing

Current-version servers would treat that as a regular v1 connection, and 
wouldn't opt in, but v2 servers could response with:

        HTTP/1.1 101 Web Socket Protocol Handshake
        Upgrade: WebSocket
        Connection: Upgrade
        WebSocket-Origin: http://example.com
        WebSocket-Location: ws://example.com
        WebSocket-Upgrade: Multiplexing

...where the Location is intentionally a mismatch so that new servers can 
be written that old UAs automatically drop connections with, and then the 
client could jump straight into our new multiplexing version of this, 
e.g.:

        Open Channel for /foo/bar

...or whatever the protocol is.

It's quite easy to make extensions to WebSocket that are backwards- and 
forwards-compatible. It just needs a revision of the protocol spec to 
define the new feature so that all UAs can be updated to support it.

We should, however, keep such features to a minimum, so as to keep the 
protocol from ballooning into something only experts can implement.


> > orderly closing a ws connection implemented in an application 
> > independent way.
> 
> Orderly close is more important than people realise.
> 
> A major problem with HTTP pipelining is you can't use it for
> non-idempotent requests because the server may close the connection
> any time, without implying an error condition.  That's one of the many
> reasons HTTP pipelining often cannot be used, and that makes HTTP in
> general slower and less network efficient than it could be.
> 
> The same applies to other request-response based protocols, such as
> many of those expected to run over WebSocket.

WebSocket handles this in the same way TCP does: application-layer 
acknowledgements. It guarantees ordering, but nothing beyond that. 
(There's not much more we _can_ do, in practice, given the realities of 
non-instantaneous communication and the unreliability of hardware links.)


> Another problem with closing is that thing not mentioned in the HTTP 
> RFCs, but essential: lingering half-closed sockets.  If a server sends a 
> response followed by closing the socket fully, some clients sometimes 
> don't receive the whole response because of the way TCP and pipelined 
> requests clash.
> 
> Both of these are why orderly close is important - more important than 
> it appears.  Because it makes it safe to do other things.
> 
> I expect "pipelining" in the meaning used by HTTP, that is sending 
> multiple messages before getting a response to each one, to be quite 
> prevalant with protocols running over WebSocket.
> 
> That makes orderly close important for WebSocket, to avoid fragile 
> applications.  Hopefully I've illustrated the kinds of problems to 
> expect.  Unfortunately it's all a bit arcane.
> 
> There's nothing wrong with doing it in the application.  But I don't 
> have high expectation of it being done correctly in those "we don't need 
> no steenkin' framework" 100 line Perl scripts by inexperienced authors.

Indeed. Do you have a suggestion for improving this at the protocol level 
without dramatically increasing the complexity of the protocol? i.e. is 
there something we can add to the spec that _would_ get this done 
correctly in the framework-less case you mention?

(There's no point adding something if it only helps the experienced 
programmers, since they can already solve it with a simple protocol.)


On Fri, 30 Oct 2009, Jamie Lokier wrote:
> 
> Inexperienced authors, especially those writing 100 lines of Perl, will 
> send ISO-8859-1 or other text which occasionally contains 0xff bytes in 
> the middle.

That's possible, but I think that will be rarer than one might expect, 
because naively copying the incoming bytes and then later outputting them 
verbatim will mean UTF-8 is used. Problems could occur when an author 
takes text from another source (e.g. a database) and outputs it in the 
wrong encoding. Is this likely to happen more or less often than not? If 
it's more often, then regardless of the length vs sentinel issue, the 
encoding problems are likely to be visible relatively quickly, which would 
take care of the problem. If we used a length marker in that case, though, 
the people correctly using UTF-8 would be more at risk of buffer overruns 
in incoming text, which would be harder to spot and would be a security 
error, which is worse than the aforementioned data corruption error.


> Even experience authors will make that mistake sometimes.  What do you
> think will happen when someone does something like this:
> 
>     - Read list of filenames in a directory.  They're UTF-8 (assumed),
>       or the author is unfamiliar with character encodings and everything
>       works fine in their ASCII development environment.
> 
>     - Concatenate the list with newlines, as people do.
> 
>     - Send the result as a frame.
> 
> Or this:
> 
>     - Read lines from a text file, which is in UTF-8 encoding.
> 
>     - Send each line as a frame.
> 
>     - (Oops, one of the text files you gave me had an 0xff byte in it.)
> 
> Result: Because of assumptions, 0xff bytes will be sent occasionally
> in the middle of a frame.  Everything afterwards will break, but it'll
> be rare enough that the author doesn't notice.  For the same reason
> you've explained authors get lengths wrong.

The difference is that this error will be immediately obvious (the 
connection will basically stop working, since most "frames" past this 
point would be dropped). The error the current spec is avoiding is much 
more insidious (buffer overrun risk on the server).


On Sat, 31 Oct 2009, Greg Wilkins wrote:
> 
> The sentinel approach also opens an easy attack vector.  If user data is 
> sent, then tricking a poor implementation into sending an OxFF will 
> allow packet insertion.

Yes, if a WebSocket server is taking data in one encoding (e.g. 1252) and 
outputting it verbatim on the wire, then one could inject any number of 
packets. I've added a paragraph about this to the Security Considerations 
section.


> This is similar to CRLFCRLF insertion attacks that can happen if user 
> data is set unfiltered into a HTTP header and/or cookie.

Indeed.


> length framing avoids this vulnerability.

Indeed.


> Note also that my proposal for a meta-data bit would allow headers to be 
> sent in one length frame and data in another, so the CRLFCRLF sentinel 
> would not be needed and that vulnerability would also be avoided.

Sure, but you would instead have the character/byte length discrepancy 
problem, which can become a code injection error.


On Fri, 30 Oct 2009, Jason Duell wrote:
>
> From my reading of the last few days' posts, it does seem like focusing 
> on use cases might be the best way to move the conversation forward.  
> The more concrete the use cases, the better.

Agreed.


On Fri, 30 Oct 2009, Ted Goddard wrote:
> 
> Without multiplexing, I expect we will see each Ajax component in each 
> page opening its own socket, and multiple pages open to the same server 
> is very common (users often forget that they already have a window open 
> in their browser) so a hundred connections to a single server would not 
> be unexpected.

This is only the case if the person hosting the widget doesn't mind this 
setup. If the server wants to reduce the number of connections to one per 
user, then SharedWorker support allows the client-side to share a single 
connection. A server doesn't have to accept connections from origins that 
are ignoring its policy -- just like it can drop connections from people 
trying to perform DOS attacks.


On Sat, 31 Oct 2009, Jamie Lokier wrote:
>
> (I'll add that deliberately opening multiple tabs to the same server is 
> quite common too.  Like, when you want to read 10 different articles 
> from the same news/blog site, many people open 10 tabs by right-clicking 
> from the headline page, let the pages load in the background, and work 
> their way through them)
> 
> Sharing WebSockets between components on different pages is difficult 
> for the simple reason that browser tabs are quite independent.  (But I 
> haven't seen anyone advocate that they should use separate HTTP 
> connections...)

SharedWorkers really let you share sockets pretty easily, assuming you 
have the server-side support for multiplexing over WebSocket.


> On a single page, Ajax component authors _could_ cooperate so their 
> components share connections, but how likely is that?

That depends on the site. How likely is it that they will cooperate to not 
overwrite each other's cookies, to not overwrite each other's global 
variables, to not use too much CPU, to not overload the server by not 
doing too much XHR, etc?

If we are talking about a server that is also serving the widgets, then 
that's a policy issue. If a site wants to DOS itself, there's plenty of 
ways to do it, regardless of what we do.

If we are talking about multiple widgets on a third-party site, then this 
is what Terms of Service are for. You require that the third-party 
components use a particular library, and you block origins that don't.


> Even with good intentions, it's technically *quite hard* to share 
> connections between independent components on a single page.

Nah, SharedWorkers makes it pretty easy.


> That's because one component sending a large message can block progress 
> of all other components on the page, if they share.

That's entirely up to the designer of the multiplexing part of the 
protocol. There's no reason the substrate couldn't define a maximum 
message size, such that larger messages have to be split up.


> And one component taking a long time to accept messages can block all 
> the others even if it's a short message.

That's the case even with multiple WebSocket connections, as far as I can 
tell.


> That is why I advocate a slightly more advanced multiplexing, akin to 
> interleaved HTTP chunks, where no component can block the progress of 
> any others, and chunks can be interleaved in a fair, orderly and 
> network-efficient manner.

Exactly how the multiplexing works is up to the author, but certainly I 
agree that a system like that is amongst the best ones. The great thing 
about leaving this to the application layer is that the application can 
pick the most appropriate scheme for the protocol being used.


On Sat, 31 Oct 2009, Greg Wilkins wrote:
> Jason Duell wrote:
> >
> > Is there any particular reason one couldn't do sticky load-balancing 
> > just by remembering the TCP IP/port four-tuple for a given web socket?
> 
> IP stickiness is a reasonable solution when cookies are not available, 
> but it is far from perfect.  Every user from AOL will look to be coming 
> from just a few IP addresses - which can and do frequently change for 
> any given user.

You can add cookies to a WebSocket connection; you just have to set it 
from HTTP first. Then, they will be included in the handshake.

You can also just add cookies to your application-layer protocol, so that 
the first message you send declares the machine you want to be connected 
to, if you have one, or that you want the server to give you a cookie.


> The primary problem that I would like to see addressed is the issue of 
> idle shutdown of connections.  Currently the client, server or 
> intermediary may decide a connection is idle too long and close it.  
> But there are no common timeouts, no way to discover these timeouts and 
> not way to negotiate these timeouts.

This is all stuff that can be added at the application layer relatively 
easily.


> Thus clients will often select and idle connection and try to use it 
> only to discover that it has been closed.  Lack of orderly shutdown 
> means that the client has no idea if the messages sent have been 
> delivered or not.

Even with orderly shutdown, you'll still have cases of the connection 
being killed unexpectedly.


> Websocket will make this a server problem as well, because servers can 
> now try to use a connection that has been idle closed by an intermediary 
> or client.

Only non-conforming clients would do this, but even without such bugs, 
connections might still be dropped by a network error.


> Having an orderly close will allow implementations to distinguish 
> between a failure (for which there is nothing much that can be done 
> short of a reliable protocol on top) and a routine idle shutdown, which 
> would now be able to be handled without any message loss.

Indeed. Protocols should add message acknowledgement on top of WebSocket 
to handle this if they need it.

Note that not all cases need this; sometimes the data is transient and 
missed messages are not a big deal. For example, a stock ticker system 
might not care about missed updates.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'