Re: [hybi] Fragmented text message

[Piotr Kulaga <[e-mail]>]

My understanding is that each websocket entity that uses UTF-8 must have a valid UTF-8 stream as a payload. This covers all UTF-8 frames including continuation frames.

I slightly more prefer approach where UTF-8 message must contain valid sequence rather than each continuation frame (simpler for intermediaries, endpoints that stream data to application still must handle partial UTF-8 code point encoding case). Fine with any approach as long as it is well defined.

From: [e-mail] [mailto:[e-mail]] On Behalf Of Takeshi Yoshino
Sent: Thursday, July 21, 2011 2:22 AM
To: Brian
Cc: [e-mail]
Subject: Re: [hybi] Fragmented text message

Agreed. Nothing in the spec disallows splitting UTF-8 byte sequence into separate frames.

Impatient receivers must use some intelligent UTF-8 decoder like Brian explained to get code points decoded. For example in python, codecs.getincrementaldecoder('utf-8')() does this. Maybe most of major platforms have library like it.

We may also add a constraint that the text message must be fragmented at UTF-8 byte sequence boundary, but it complicates fragmentation code. I'm not for that.

On Thu, Jul 21, 2011 at 17:47, Brian <[e-mail]<mailto:[e-mail]>> wrote:
Unless I'm mistaken, the fragmentation may occur in the middle of a multi-byte character sequence.  Your code should be aware of that when decoding.  My initial implementation buffers all fragments and then decodes the whole message into a string at once.  I imagine you could probably inspect the last four bytes of a fragment to determine whether there's a partial utf-8 character.  If there is, you could buffer just those few bytes and decode the rest of the fragment.  Then when the next fragment comes in, prepend those bytes to the new payload and continue. Depending on your use case and what you're optimizing for, it may be more efficient to just buffer the whole message and then decode.

Brian

On Thu, Jul 21, 2011 at 1:28 AM, Kukosa, Tomas <[e-mail]<mailto:[e-mail]>> wrote:
If the text message is fragmented must be each fragment a valid UTF-8 string or only complete defragmented message must be a valid UTF-8 string?
I.e. may I during receiving decode each fragment by UTF-8 and than join strings or do I need to receive all fragments and then decode only defragmented message?