Re: [hybi] Fragmented text message

Piotr Kulaga <> Thu, 21 July 2011 16:04 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 1F37821F874A for <>; Thu, 21 Jul 2011 09:04:51 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -10.598
X-Spam-Status: No, score=-10.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id LBxZU1BR7vHC for <>; Thu, 21 Jul 2011 09:04:49 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id AE2A221F8A55 for <>; Thu, 21 Jul 2011 09:04:49 -0700 (PDT)
Received: from ( by ( with Microsoft SMTP Server (TLS) id; Thu, 21 Jul 2011 09:04:48 -0700
Received: from ([]) by ([]) with mapi id 14.01.0323.002; Thu, 21 Jul 2011 09:01:04 -0700
From: Piotr Kulaga <>
To: Takeshi Yoshino <>, Brian <>
Thread-Topic: [hybi] Fragmented text message
Thread-Index: AcxHgDh/v4NQDoJkRvu0Mk4+OYBXmwAPTumAAAE7A4AAAVHvYA==
Date: Thu, 21 Jul 2011 16:01:02 +0000
Message-ID: <>
References: <> <> <>
In-Reply-To: <>
Accept-Language: en-US
Content-Language: en-US
x-originating-ip: []
Content-Type: multipart/alternative; boundary="_000_ED13A76FCE9E96498B049688227AEA29388ADF4DTK5EX14MBXC206r_"
MIME-Version: 1.0
Cc: "" <>
Subject: Re: [hybi] Fragmented text message
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Server-Initiated HTTP <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 21 Jul 2011 16:04:51 -0000

My understanding is that each websocket entity that uses UTF-8 must have a valid UTF-8 stream as a payload. This covers all UTF-8 frames including continuation frames.

I slightly more prefer approach where UTF-8 message must contain valid sequence rather than each continuation frame (simpler for intermediaries, endpoints that stream data to application still must handle partial UTF-8 code point encoding case). Fine with any approach as long as it is well defined.

From: [] On Behalf Of Takeshi Yoshino
Sent: Thursday, July 21, 2011 2:22 AM
To: Brian
Subject: Re: [hybi] Fragmented text message

Agreed. Nothing in the spec disallows splitting UTF-8 byte sequence into separate frames.

Impatient receivers must use some intelligent UTF-8 decoder like Brian explained to get code points decoded. For example in python, codecs.getincrementaldecoder('utf-8')() does this. Maybe most of major platforms have library like it.

We may also add a constraint that the text message must be fragmented at UTF-8 byte sequence boundary, but it complicates fragmentation code. I'm not for that.

On Thu, Jul 21, 2011 at 17:47, Brian <<>> wrote:
Unless I'm mistaken, the fragmentation may occur in the middle of a multi-byte character sequence.  Your code should be aware of that when decoding.  My initial implementation buffers all fragments and then decodes the whole message into a string at once.  I imagine you could probably inspect the last four bytes of a fragment to determine whether there's a partial utf-8 character.  If there is, you could buffer just those few bytes and decode the rest of the fragment.  Then when the next fragment comes in, prepend those bytes to the new payload and continue. Depending on your use case and what you're optimizing for, it may be more efficient to just buffer the whole message and then decode.


On Thu, Jul 21, 2011 at 1:28 AM, Kukosa, Tomas <<>> wrote:
If the text message is fragmented must be each fragment a valid UTF-8 string or only complete defragmented message must be a valid UTF-8 string?
I.e. may I during receiving decode each fragment by UTF-8 and than join strings or do I need to receive all fragments and then decode only defragmented message?