Re: [hybi] With deflate-stream, Close frame doesn't work as an end of data marker

[Takeshi Yoshino <[e-mail]>]

On Fri, Mar 11, 2011 at 01:31, John Tamplin <[e-mail]> wrote:

> On Thu, Mar 10, 2011 at 7:34 AM, Takeshi Yoshino <[e-mail]>wrote:
>
>> OK. So we can add an implementation note like this.
>>
>> Senders using this extension MUST apply RFC 1951 DEFLATE to all the bytes
>>> of the application data portion of data frames. Senders MAY include multiple
>>> blocks in a frame. Senders MAY use blocks of any type as defined in RFC
>>> 1951. Senders MUST use the algorithm described in Section 2.1 of RFC1979 to
>>> align compressed data to byte boundary. Senders MAY keep using the same LZ77
>>> sliding window for multiple frames.
>>
>>
>> If you're using zlib for applying DEFLATE compression, specify
>> Z_SYNC_FLUSH when you flush the last octets of application data for each
>> frame (in case the last block is non-compressed block (BTYPE=00), append one
>> empty compressed block (BTYPE=01)), and then strip the last 4 octets from
>> the compressed octets.
>>
>> Receivers MUST append 0x00 0x00 0xff 0xff to the application data portion
>>> and decompress it using DEFLATE to decode the original application data.
>>> Receivers MUST keep using the same LZ77 sliding window for all the frames of
>>> the same WebSocket connection. Receivers MUST assume the senders use 32768
>>> byte LZ77 sliding window.
>>
>>
>> If you're using zlib for applying DEFLATE compression, for each received
>> frame, concatenate 4 octets 0x00 0x00 0xff 0xff and the application data,
>> and then supply the resulting bytes to zlib and use the uncompressed data.
>>
>
> zlib is just one implementation, and I don't think the spec should be
> choosing one implementation instead of the algorithm.  Therefore, we
> shouldn't be using zlib-specific terminology in the spec.
>

So, the note including zlib terms should go somewhere else than the spec
text.

IMO at least we should explain what implementors actually should do to align
to octet boundary, and place that somewhere they can easily get to. The
current deflate-stream spec just says "Senders MUST NOT delay the
transmission of any portion of a WebSocket message because the deflate
encoding of the message does not end on a byte boundary". It's not so easy
for implementors to know what to do and be convinced what they're doing is
correct. The octet aligning method is NOT explained in RFC1951. If we could
add some text like "For example, you can do this by adding BTYPE=00 block"
(and at least add reference to zlib since it's not our invention),
implementors can understand what to do just by reading WebSocket spec and
RFC1951. No need to use Z_ stuff.


>
> I don't object to compressing only the payload (though it does make
> implementations more complicated, rather than just pushing a
> compressor/decompressor onto the stream having to switch back and forth),
> but couldn't the issue we are trying to solve here be done by just saying
> that if deflate-stream is negotiated and a close frame is read, you cannot
> initiate close processing until the final compressed block is read
> completely?
>
That's the quick fix to the deflate-stream extension I proposed above. I
want a short note in the spec saying that the sender should finalize the
stream after the close frame by BFINAL=1 block".

Some may immediately understand the deflate stream should finalized after
the close by just reading "Senders using this extension MUST apply RFC 1951
encodings to all bytes of the data stream following the handshake including
both data and control messages.", but expecting implementors to guess is not
good, I think.


>
> In any case, I think deflate-stream is only a stop-gap solution to get some
> form of compression in the spec, also serving as an example extension.  I
> think realistically we are going to have to have the ability to avoid
> compressing incompressible data in the stream.
>

I agree this point. I'm not objecting to having deflate-stream in the
initial spec.


>   Initially, where WebSockets implementations only support plain text, this
> won't be much of a problem, but eventually clients will be mixing binary
> data with their text and we would prefer to be able to use compression where
> it helps and not use it where it hurts.
>
> --
> John A. Tamplin
> Software Engineer (GWT), Google
>