Re: [hybi] With deflate-stream, Close frame doesn't work as an end of data marker

[[e-mail]]

Hi John,

[e-mail] wrote on 02/28/2011 06:32:03 AM:

> [BTW - here is the thread where I first posted results of an
> experiment 
- http://www.ietf.org/mail-archive/web/hybi/current/msg01810.html
> ]
> 
> On Mon, Feb 28, 2011 at 4:52 AM, Greg Wilkins <[e-mail]> wrote:
> > I'm +1 for compression within the frame.
> >
> > If we compress the framing, then intermediaries will have to
> > decompress in order to see framing and to know when to forward etc.
> > It also has significant debug issues as it makes it much harder to
> > decode a partial frame capture.
> >
> > I understand the argument that compression will not be as good, but I
> > think some compromise is necessary.
> 
> The framing is pretty small and it is hard to imagine much redundancy
> from the header (the opcode and reserved bits will be constant, but
> then you will interfere with the compression by all the constantly
> changing lengths), so I think little is lost by compressing only the
> payload.

Just to be clear, the compression does not have to be per payload of
a data frame, but can be entire stream of messages application passes in.
At the end of the message, though, F_SYNC_FLUSH is necessary to prevent 
receiver from waiting for the last compressed data to arrive from sender, 
but we should be aware that the compressor still maintains 32kB(or 
whatever 
it is configured for) of reference range in its context.

> 
> However, it does mean the implementation is more complicated, and I

It is actually very simple, in python for example:


        zlibObj = zlib.compressobj()

        def send(in_data):
                buf = zlibObj.compress(in_data)
                buf += zlibObj.flush(zilib.Z_SYNC_FLUSH)
                        :
                (construct frames/fragments and put into a queue)

Note: the 'zlibObj' has the same lifetime as that of WebSocket.


> don't believe this would be feasible using the stock inflate/deflate
> in Java for example.  I would not agree to just compressing each
> frame's payload independently -- my earlier experiments show this will
> frequently increase the size of the data and result in only modest
> savings compared to maintaining compression state over the life of the
> connection (while we are discussing it, 

Again, lifetime of the compression state/context can be the same. It 
just needs, Z_SYNC_FLUSH operation at the end of the 'WS-frame' to avoid 
unnecessary delay. Even compression was done for whole stream, we still 
need Z_SYNC_FLUSH per frame basis with the exact same reason - we cannot
delay the frame! (Unless we introduce nagle's algorithm like protocol
just for slightly better compression ratio, which I think would not be
feasible.) This, at least to me, looks more complicated.

I am aware that, if application passes in is very small message 
frequently, (Z_SYNC_FLUSH has to be done frequently), resulting 
compression ratio would not be that great. However, since we can not 
delay each frame, we need the Z_SYNC_FLUSH anyways. Therefore, there's 
no benefit in having compression layer under the framing layer.

As I mentioned earlier, the compression state can be shared between 
text and binary messages since they are not interleaved (even fragmented),
though it may be better to have dedicated compression state if we can
afford memory. Either way, we need to define the spec in the document.


- Yutaka


> we should talk about the
> parameters to use for deflate to minimize memory requirements on the
> server -- experiments done for compression in SPDY [mentioned in the
> linked thread] show you can get most of the benefit with more modest
> memory requirements using lower parameters).
> 
> It also means we have to agree on the details of how it should be
> specified, which is why the super-simple deflate-stream was added to
> the standard instead of trying to get agreement on something better.
> 
> --
> John A. Tamplin
> Software Engineer (GWT), Google
> _______________________________________________
> hybi mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/hybi