Re: [hybi] thewebsocketprotocol #40 (new): Clarify binary/utf-8 mixed handling

Yes, if one accepts UTF-8, that must be validated, no question about it. The point is not to disallow UTF-8, but to be able to optimize on those websocket connections that use binary only, so this is not a fragmenting of the protocol any more than any other per-session negotiable feature is.

>From the comments, folks seem to be thinking about Javascript Websockets API, and how it does not allow binary nor support streaming. That's true, but as we've emphasized before: we're designing the websockets protocol to be usable with the JS Websockets API, but is not limited to it. Other APIs may offer both binary and streaming.

> -----Original Message-----
> From: hybi-bounces@ietf.org [mailto:hybi-bounces@ietf.org] On Behalf Of
> Bjoern Hoehrmann
> Sent: Wednesday, February 09, 2011 17:00
> To: g_e_montenegro@yahoo.com
> Cc: hybi@ietf.org
> Subject: Re: [hybi] thewebsocketprotocol #40 (new): Clarify binary/utf-8 mixed
> handling
> 
> * hybi issue tracker wrote:
> > Additionally, when only partial frames may be available, it is
> > expensive to verify that this is indeed a valid UTF-8 stream (protocol
> > implementation needs to take into account multi-byte characters and
> > end of current data payload).  If binary has been negotiated for this
> > session, processing can be optimized accordingly.
> 
> If you do accept UTF-8 encoded data then you have to validate it, other- wise
> you get strange and possibly dangerous failures if you receive mal- formed data,
> for instance, you can't trust that the length has been cal- culated correctly.
> Anyway, you would seem to have this problem due to fragmentation anyway if
> you accept text frames, and if you don't mean to accept text frames then you
> just don't, there would seem to be no need to negotiate for "binary-only". I also
> note that validating UTF-8 is not really expensive, it's just a matter of `state =
> table[state + byte]` for each byte. Work, sure, but not very "expensive". It's easy
> too if you use my http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ decoder for
> it.
> 
> > Proposal: allow negotiation to clarify if a stream will not mix binary
> > and text in order to enable optimizing for the binary-only case.
> 
> I do agree the protocol specification needs to discuss mixing frame types, like,
> what if you have a fragmented text message but one of the frames is not a text
> frame, but allowing to negotiate this complexity away will most likely lead to
> interoperability and security problems, as people will take shortcuts like not
> validating the frame type. We've in fact seen that already on this list. Essentially
> negotiation binary- only would be subsetting and fragmenting the protocol, and
> I don't think the benefit here warrants that.
> --
> Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de
> Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
> 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
> _______________________________________________
> hybi mailing list
> hybi@ietf.org
> https://www.ietf.org/mailman/listinfo/hybi