Re: [hybi] ws: and wss: schemes

[Ian Hickson <[e-mail]>]

On Fri, 14 Aug 2009, Julian Reschke wrote:
>
> [...] it now says:
> 
> >    URI scheme syntax.
> >       In ABNF terms using the terminals from the IRI specifications:
> >       [RFC5238] [RFC3987]
> > 
> >            "ws" ":" ihier-part [ "?" iquery ]
> 
> That is even worse than before, because it now uses productions from the 
> IRI spec defining *URI* syntax.

ws: and wss: URLs are i18n-aware; why would we want to limit them to 
ASCII?


> Furthermore, it still doesn't answer what the semantics of these parts 
> are. What do "ihier-part" and "iquery" represent in a ws URI?

This is defined by the RFC 3987, no? Surely we wouldn't want IRI 
components to have different meanings in different schemes?


> What's the effect? How are they used?

This is defined earlier in the Web Socket specification.


> PS: what does RFC5238 have to do with this?

Oops, typo. Fixed. (Meant 5234.)


On Fri, 14 Aug 2009, Julian Reschke wrote:
> Ian Hickson wrote:
> > > I assume you are using ABNF syntax (RFC5234) and terminology from the URI
> > > spec, but you really need to state that.
> > 
> > Thanks, fixed.
> > 
> > (I tried referencing STD68 instead of RFC5234, as we do in HTML5, but
> > apparently there's no index of STD references for xml2rfc?)
> 
> Just day "STD68" instead of "RFC5234" in the reference/@anchor element.

I have no <reference> elements, I'm using the <?rfc include=""?> feature 
and reference.RFC.xxxx.xml files. I couldn't find STD reference files.


> > > >    URI scheme semantics.
> > > >       The only operation for this scheme is to open a connection using
> > > >       the Web Socket protocol.
> > > > 
> > > >    Encoding considerations.
> > > >       UTF-8 only.
> > >
> > > What does this mean?
> > 
> > That the only encoding that can be used with this scheme is UTF-8. What is
> > unclear?
> 
> You can only have ASCII characters in a URI. I believe you're trying to do the
> right thing, but it really requires a few more words (...when non-URL
> characters are to be used in a ws URI, they need to be encoded using UTF-8 and
> then percent-escaped...)

I've deferred to RFC3987 to sidestep this issue.


> > I've refactored the Web Sockets API and Web Sockets Protocol specs so 
> > that the parsing of Web Socket URLs happens all within the Web Sockets 
> > Protocol spec. Let me know if that's not enough. I can include more 
> > details if you would like.
> 
> Maybe than that spec should carry the URI definition and registration.

Which spec? I'm confused. The protocol spec _does_ have the registrations.


> > (If you do want more, a reference to how another scheme registration 
> > defines the semantics of parts of the URL would be useful, so that I 
> > can use a similar style.)
> 
> Looking at relatively recent RFCs, and example would be
> <http://tools.ietf.org/html/rfc4516#section-2>.
> 
> If there is no defined semantics for the two parts then the definition 
> should just state that.

Ok, added a sentence describing the relevant components.


On Fri, 14 Aug 2009, Toby Inkster wrote:
> On Fri, 2009-08-14 at 10:47 +0000, Ian Hickson wrote:
> > I don't understand how one would reuse the http: scheme in this way in 
> > order to connect to 'example.com' as both an HTTP server and a Web 
> > Socket server.
> 
> The idea is not necessarily to reuse the HTTP protocol - just the URI 
> scheme. How to do this is pretty easy and would require very few changes 
> to the web sockets API.
> 
> 1. You, Ian Hickson, in your role as editor of the API spec. (Or a 
> person or organisation nominated by you) would register a domain or 
> subdomain such as websockets.net or sockets.w3.org. For the rest of this 
> message, I'll assume websockets.net.
> 
> 2. In part 4 of the draft, point #1 which starts "Parse a Web Socket
> URL's components..." describe an algorithm of how to parse a WebSocket
> URL like this (square brackets are used to indicate variables):
> 
> 	i) A web socket is parsed as a standard HTTP/HTTPS URL
> 	  obtaining [protocol], [host], [port] and [path].
> 	ii) If [host] is not websockets.net, throw SYNTAX_ERR.
> 	iii) If [port] is blank and [protocol] is "http" then
> 	  set [port] to 81.
> 	iv) If [port] is blank and [protocol] is "https" then
> 	  set [port] to 815.
> 	v) Match [path] against the following regular expression:
> 	  ^\/([^\/]+)(/.*)?$
> 	  Throw SYNTAX_ERR if the expression is not matched.
> 	  Set [host] to the first matched subexpression.
> 	  Set [path] to the second matched subexpression.
> 	vi) Set [secure] to true if [protocol] is "http".
> 	  Set [secure] to false if [protocol] is "https".
> 	  Throw SYNTAX_ERR if neither of the above.
> 	
> The above procedure will always end up with four variables:
> 
> 	* [host] representing the host the API should connect to
> 	* [port] representing the port the API should connect to
> 	* [path] representing the Web Socket resource name
> 	* [secure] indicating whether the Socket should be tunnelled
> 	  over TLS.
> 
> or will throw SYNTAX_ERR.
> 
> 3. ???
> 
> 4. Profit.

That seems like a wild layering violation.


On Fri, 14 Aug 2009, Jamie Lokier wrote:
> 
> However I can say for sure that I intend to deploy applications which 
> serve exactly the same resources over WebSockets and HTTP.
>
> The technical methods used have to be very different, and once you get 
> past the initial URL there's a certain amount of fudging.  You wouldn't 
> start 1000 WebSockets connections at different URLs to access different 
> resources on the same server; you'd create an application protocol on 
> top of it, and request all the resources through a shared endpoint.  
> (That's a consequence of WebSockets' design).
> 
> But the conceptual resources being accessed behind WebSockets and HTTP 
> will be identical in the applications I have in mind.  And the 
> conceptual "endpoint" resource at a single WebSockets address has a 
> corresponding endpoint resource over HTTP too.  So it seems natural to 
> use similar URLs for the two access methods.

If HTTP is applicable, then I really don't think you would want to use 
WebSockets. The two have completely different, non-overlapping use cases, 
as far as I can tell.


On Fri, 14 Aug 2009, Jamie Lokier wrote:
> Ian Hickson wrote:
> > The Web Socket API is here:
> > 
> >    http://dev.w3.org/html5/websockets/
> > 
> > I don't really see any sane way to have the API itself do some sort of 
> > fallback, especially not to HTTP.
> 
> Ah, that'll be why you didn't respond to discussions about that :-)

Actually I was just slow. :-)


> > On Sat, 8 Aug 2009, Jamie Lokier wrote:
> > > Tunnelling WebSocket two-way communications over standard HTTP 
> > > messages, using any of the methods used for that, would be natural 
> > > and probably useful behaviour.
> > 
> > That is a radically different system on the server-side.
> 
> Yes.
> 
> > While authors should naturally be allowed to do that using XHR if they 
> > desire,
> 
> All authors can just use the WebSocket API.
> 
> When running on older browsers they'll use a Javascript 
> backward-compatibility package which emulates WebSocket, using some 
> other mechanism underneath.
> 
> > forcing servers into having that kind of fallback behaviour radically 
> > increases the complexity of a server-side deployment.
> 
> Not much.  The art of good software development is to hide the 
> complexity where it has to exist.
> 
> "apt-get install ws2http".  Problem solved.

You have more faith in people developing libraries and software tools than 
I do.


> > A Web Socket protocol implementation can be written in 100 lines of 
> > Perl or so;
> 
> If using libraries, 5 lines of Perl.
> 
> If avoiding libraries, you can, barely, but:
> 
>    - Trust me there will be bugs, if only in mishandling UTF-8.
> 
>    - Anyone doing that values simplicity over performance.
>      Which is great, see next section.

What are the bugs?

   http://damowmow.com/playground/demos/websocket/blank-server.pl


> > having to add to that some sort of HTTP server or CGI scripting would 
> > make things significantly more complicated.
> 
> No, it's very simple to the person writing 100 lines of Perl.

Having written the 100 lines of Perl, I disagree. I would never want to 
have to add an HTTP server to that.


> They'll install a WebSocket <-> HTTP proxy which is written by 
> professionals in C.  "apt-get install ws2http".
>
> On the client side, they'll use a Javascript file which emulates 
> WebSocket (when necessary), using HTTP which works with ws2http.
> 
> Job done.  Johnny's interactive Tic-Tac-Toe (his very first web app!) 
> will work over the HTTP internet with *no changes at all* except minimal 
> configuration.

Well it'd be cool if such tools become available in the transition period, 
but I'm not making any assumptions that they will be.


> > On Sun, 9 Aug 2009, David Booth wrote:
> > > I can't see that as a significant issue, as there is only a trivial 
> > > difference between dispatching based on the string prefix 
> > > "http://wss.example/" and the string prefix "wss:".  Both are 
> > > simple, constant strings and both are equally "magic": they cause 
> > > agent to attempt the WSS protocol.
> > 
> > I don't understand how one would reuse the http: scheme in this way in 
> > order to connect to 'example.com' as both an HTTP server and a Web 
> > Socket server.
> 
> The implication is that it's context dependent - you know you're making 
> a WebSockets connection - and the prefix would serve much the same role 
> as URIs do for XML namespaces.
>
> But I must agree that http://www.example/my.domain.org/path/to/service 
> would be profoundly ugly and pointless.
> 
> On the other hand, http://my.domain.org/path/to/service would be very 
> nice if that Just Worked.
>
> That is, if WebSockets connected and did a standardised and safe 
> negotiation over HTTP to your own site's URL, and acquired a WebSockets 
> connection that way.  No need for the ws: prefix!
> 
> That would be nice because it leaves open the option to do drop down 
> automatically to something else if the WebSockets negotiation fails. 
> (Automatically in the WebSockets implementation, that is, without the 
> app author having to care).

We can drop down even with the ws: prefix.


> > I don't understand why we would even want a spider to attempt to 
> > connect to a Web Socket server. How would it know what 
> > application-level protocol to use? How would it expose this 
> > information to the user?
> 
> Spiders will follow any text that looks like a URL.  Even comments.

Then we definitely don't want to use http:/https:.


> > On Wed, 12 Aug 2009, Greg Wilkins wrote:
> > > The ws protocol has been designed in respect to a single proposed 
> > > javascript API to run within a browser.
> > > 
> > > There are many many uses for bidirectional web which go beyond that 
> > > simple restricted example.
> > 
> > For those purposes, we already have TCP/IP, no?
> 
> Because WebSockets isn't _that_ simple on the server, especially with 
> things layered on top, there will be frameworks to make it easier on the 
> server side.  Before you know it, that will be more familiar for a lot 
> of people than TCP/IP.  In the same way that people can write CGI/WSGI 
> now, but would find writing a TCP/IP server difficult.

Your faith in WebSockets' success is heart-warming, though I think it's a 
little optimistic. :-)


> With millions of programmers familiar with writing WebSockets code on 
> browser clients and servers, and good server-side frameworks for 
> handling it - of course it will be used in non-browser applications, 
> whether you think it's a good choice or not.

Fair enough, but we certainly shouldn't optimise for those cases.


On Fri, 14 Aug 2009, Jamie Lokier wrote:
> 
> I suspect WebSockets could benefit from a look at the mechanisms used by 
> cross-domain XHR to check for permission to proceed.

The handshake in Web Sockets is based on the CORS (XHR) mechanism.


> The only difference with WebSockets is that it (so far) seems to avoid 
> any descriptive metadata, which means there will still be applications 
> which ask for a WebSockets URL, but when the URL is for a different 
> protocol on top, it'll simply break with undefined behaviour instead of 
> a clean error message or fallback behaviour.

Web Sockets has an out-of-band protocol specifier to avoid this issue.


On Sun, 16 Aug 2009, Jamie Lokier wrote:
> 
> WebSockets philosophy seems to be to check for WebSockets itself, but 
> then avoid the problem at the next level, passing it on to the 
> application to do an initial handshake, which has the same problems as 
> raw TCP in that there is no _standard_ negotiation handshake.

The Web Sockets protocol does have an application-layer protocol handshake 
as part of the main Web Sockets handshake, for exactly this reason.


On Thu, 20 Aug 2009, Jamie Lokier wrote:

> Three notable things for which there is no consensus are:
> 
>   1. Does the WebSockets really have to use a fresh, new TCP
>      connection for every individual instance of the WebSockets object
>      created on every web page in a browser to the same site?  Or
>      could it upgrade an HTTP connection which has already been used
>      for something else?

I wouldn't expect more than one WebSocket connection per page; clients 
would presumably want to multiplex multiple services over one connection 
(typically making use of a single socket from a shared worker).

I also think that the case of an upgrade from HTTP to Web Socket will be 
relatively rare, such that optimising for that case (e.g. by reusing an 
HTTP connection if there is one) wouldn't necessarily make sense.


>   2. Since it begins with an HTTP request for upgrade to WebSockets,
>      is there any reason for that request not to be to an arbitrary
>      HTTP host+port+path by simply trying the upgrade method on the
>      appropriate HTTP path at the server?

I don't understand the question. Could you elaborate?


>   3. If a WebSockets connection cannot be established, clients
>      will(**) fall back to an equivalent (but bulkier/slower) protocol
>      which uses HTTP only.  What URI will they use in that case?  Do
>      we leave it to individual application/framework designers, or do
>      we suggest a common strategy?
> 
> ** - As in, even if the spec does not provide this and browsers do not
>      provide it, there will certainly be Javascript
>      WebSockets-emulation modules and corresponding server-sides which
>      do.  We can choose to ignore this and leave the mapping from
>      WebSockets URI to HTTP-fallback URI unspecified, or recommend
>      a mapping, or they can simply by the same URI given point 2.

Since this will only be relevant for a transition period, I think it makes 
more sense to just leave that up to the authors.


On Thu, 20 Aug 2009 [e-mail] wrote:
> 
> Some application is using these ws: or wss: URIs.  The URIs wind up in 
> documents or in other places where they can be discovered, perhaps 
> because that is how the application keeps track of or advertises various 
> web socket endpoints.  Now some other agent, perhaps a search crawler, 
> comes upon the documents and finds the references.  If they are 
> http-scheme URIs, then the crawler will know how to dereference them.  
> Of course, this agent will not go through the upgrade protocol, and will 
> attempt an ordinary HTTP GET.  If the server (I.e. the same software 
> that would have processed the Web socket upgrade) chooses to, it can 
> respond with metadata explaining that this is a Websocket resource, 
> perhaps providing information about its purpose and correct use, etc.  
> The means for returning that metadata might be along the lines of [1].  
> That discoverability seems to have some value.

From Google's perspective, I can't really see what we would do with this, 
to be honest. It's not like we'd ever surface a Web Socket URL in the 
results or anything like that -- and people can already expose HTML files 
easily, no need for a new way to discover them as per here.


> Websockets seem different in two ways, one of which is more or less a 
> happy accident:
> 
> 1) Websockets are new, so there's an awful lot of software out there 
> that will just not recognize the new schemes at all.

None of this software should ever get exposed to them, though.


> 2) (happy accident) I suspect the choice of an HTTP-compatible handshake 
> was made to get through firewalls, but I think it also facilitates the 
> use of HTTP for metadata discovery for web socket resources.

It was made only to allow port sharing; I do not believe anything will 
really get us through firewalls (other than tunnelling over TLS).

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'