Re: [hybi] status of real world permessage-deflate implementations?

[Yutaka Hirano <[e-mail]>]

Sorry for the very late response.

I implemented permessage-deflate in Chrome so that it inserts
\x00\x00\xff\xff only at the end of the last frame of each message.
I have tested both the new and the old implementation in Chrome (ToT), but
it accepts your "by-spec-32k" data.
Please feel free to file a bug to http://crbug.com/.

Thanks,



On Fri, May 16, 2014 at 5:03 AM, Joakim Erdfelt <[e-mail]> wrote:

> Some more updates...
>
> This is looking like more and more like a *VERY SPECIFIC* chrome bug.
> Would be happy to migrate this to a chrome specific bug tracker, but I
> just wanted to update the rest of the group first on these results.
>
> My test data:
> http://joakim.erdfelt.com/jetty/websocket-intratail/
>
> These are all from server, not masked, destined for client that is being
> tested.
> Each file is a single, complete, frame.
> Each directory represents a single overall message.
> Every message is the same...
>  UTF8 e-book -> "A Connecticut Yankee in King Arthur's Court, Complete, by
> Mark Twain"
>  From Project Gutenberg: http://www.gutenberg.org/ebooks/86.txt.utf-8
>
> Extension Negotiation:
>
>   There were 2 overall client offer styles seen.
>    a) permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
>    b) permessage-deflate; client_max_window_bits
>
>   Server always negotiated as: permessage-deflate
>
> Once the connection is established, these raw frames are sent to the
> client, as-is, with no extra interpretation by the server.
> Connection is plain, http -> ws:// not https or wss:// or spdy.
>
>   /by-spec-*/        - intra-frame tail is present (per spec)
>   /by-spec-32k/      - fragmented, 20 frames,  32k max frame size
>   /by-spec-128k/     - fragmented,  5 frames, 128k max frame size
>   /by-spec-1-frame/  - not fragmented, 1 frame
>
>   /strip-intratails-*/        - intra-frame tail is removed (in violation
> of spec)
>   /strip-intratails-32k/      - fragmented, 20 frames,  32k max frame size
>   /strip-intratails-128k/     - fragmented,  5 frames, 128k max frame size
>   /strip-intratails-1-frame/  - not fragmented, 1 frame
>
> Tested Clients:
>
>   A) PyWebSocket Trunk (@revision 798) - echo_client.py - Python 2.7.5
>   B) Chrome 34.0.1847.132 - on Fedora 20 - Javascript/HTML
>   C) Chromium 34.0.1847.132-1.fc20 - on Fedora 20 - Javascript/HTML
>   D) Chrome 34.0.1847.131 - on OSX/10.9.2 (older MBP) - Javascript/HTML
>   E) Chrome Canary 37.0.1991.0 - on OSX/10.9.2 (older MBP) -
> Javascript/HTML
>   F) Chrome 34.0.1847.137 - on OSX/10.9.2 (first gen rMBP) -
> Javascript/HTML
>   G) Chrome Canary 37.0.1991.0 - on OSX/10.9.2 (first gen rMBP) -
> Javascript/HTML
>   H) Chrome Canary 37.0.1994.2 - on OSX/10.9.2 (first gen rMBP) -
> Javascript/HTML
>   I) Chrome 34.0.1847.137 m - Windows 7 SP1 (x64) - Javascript/HTML
>   J) Chrome 34.0.1847.131 m - Windows 8.1 (x64) - Javascript/HTML
>
> Results:
>
>   /by-spec-1-frame/          - all clients pass
>   /by-spec-32k/              - B fails, all of the rest are successful.
>   /by-spec-128k/             - all clients pass
>   /strip-intratails-1-frame/ - all clients pass
>   /strip-intratails-32k/     - only B passes, all other clients fail
>   /strip-intratails-128k/    - all clients fail.
>
> When a failure occurs, its always on the second frame sent.
>
> As these results are just mind-boggling, I went ahead and installed a
> fresh copy of fedora 20 in a vm (oracle virtualbox), updated all of the
> packages and tried again with Chrome 34.0.1847.132.  Same results.
> Important note! - When upgrading to stable version 34.0.1847.137-1 then
> chrome behaves properly again.
>
> As for the Fedora 20 main machine, it has a disabled firewall and selinux
> (so those can be ruled out as the cause).
>
>
> --
> Joakim Erdfelt <[e-mail]>
> webtide.com <http://www.webtide.com/> - intalio.com/jetty
> Expert advice, services and support from from the Jetty & CometD experts
> eclipse.org/jetty - cometd.org
>
>
> On Thu, May 15, 2014 at 6:15 AM, Takeshi Yoshino <[e-mail]>
> wrote:
>
>> Hi Joakim, Tobias,
>>
>>
>> On Wed, May 14, 2014 at 9:22 PM, Tobias Oberstein <
>> [e-mail]> wrote:
>>
>>> Hi Joakim,
>>>
>>>
>>>
>>> couple of thoughts ..
>>>
>>>
>>>
>>> 1)
>>>
>>> The latest Autobahn testsuite did include testing of various deflate
>>> parameter sets (section 12)
>>>
>>>
>>>
>>> http://autobahn.ws/testsuite/reports_20140314/clients/index.html
>>>
>>>
>>>
>>> but does NOT yet include testing permessage-deflate combined with
>>> fragmentation and/or multiple deflate blocks.
>>>
>>>
>>>
>>> That is missing. I guess we need it.
>>>
>>>
>>>
>>> 2)
>>>
>>> FWIW, AutobahnPython allows to set auto-fragmentation of outgoing
>>> messages via
>>>
>>>
>>>
>>> WebSocketClientFactory.setProtocolOptions(autoFragmentSize = 1024)
>>>
>>> WebSocketServerFactory.setProtocolOptions(autoFragmentSize = 1024)
>>>
>>>
>>>
>>> However, this works like this: first compress the complete message in
>>> one go and _then_ fragment.
>>>
>>>
>>>
>>> This means, there will only be a _single_ deflate block that is then
>>> splitted between mulitple frames (see (B) below)
>>>
>>>
>>>
>>> 3)
>>>
>>> In other words, there is another dimension: multiple deflate blocks in
>>> one or multiple frames  (see (C) below).
>>>
>>>
>>>
>>> (A)
>>>
>>> [FRAME 1 : DEFLATE BLOCK 1 - without tail bytes]
>>>
>>>
>>>
>>> (B)
>>>
>>> [FRAME 1 : DEFLATE BLOCK 1 (Part 1/2)] [FRAME 2 : DEFLATE BLOCK 1 (Part
>>> 2/2)- without tail bytes]
>>>
>>>
>>>
>>> (C)
>>>
>>> [FRAME 1 : DEFLATE BLOCK 1 with tail bytes] [FRAME 2 : DEFLATE BLOCK 2
>>> without tail bytes]
>>>
>>>
>>>
>>> Regarding the spec:
>>>
>>>
>>>
>>> """
>>>
>>> Note that for non-final fragments, the removal of 0x00 0x00 0xff 0xff must not be done.
>>>
>>> """
>>>
>>>
>>>
>>> I am now wondering which of the following would be valid:
>>>
>>>
>>>
>>> (D)
>>>
>>> [FRAME 1 : DEFLATE BLOCK 1 with tail bytes] [FRAME 2 : empty payload]
>>>
>>
>> This doesn't work since the concatenation of payloads ends with the tail
>> bytes. On the receiver, it adds the tail bytes and decodes and sees an
>> error. FRAME 2 must include 0x00. Please see
>> https://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-18#section-8.2.3.6
>>
>>
>>>
>>>
>>> (E)
>>>
>>> [FRAME 1 : DEFLATE BLOCK 1 (Part 1/2)] [FRAME 2 : DEFLATE BLOCK 1 (Part
>>> 2/2) with tail bytes | DEFLATE BLOCK 2 without tail bytes]
>>>
>>>
>>>
>>
>> This is fine.
>>
>>
>>> (F)
>>>
>>> [FRAME 1 : DEFLATE BLOCK 1 with tail bytes | DEFLATE BLOCK 2 without
>>> tail bytes]
>>>
>>>
>>>
>>
>> This is also fine.
>>
>>
>>> and whether the intention of the spec was more like
>>>
>>>
>>>
>>> """
>>>
>>> The tail bytes of the last (and only the last) deflate block must be
>>> stripped.
>>>
>>> """
>>>
>>
>> Correct.
>>
>>
>>>
>>>
>>> That is: no reference to frames at all, only to deflate blocks (last or
>>> non-last).
>>>
>>
>> Thanks for suggestion. I'll review my texts about this.
>>
>>
>>>
>>>
>>> Anyway: which of (A) - (F) is intended to be valid/invalid?
>>>
>>>
>>>
>>> My current understanding: all but (D) are valid.
>>>
>>
>> Right.
>>
>>
>>>
>>>
>>> /Tobias
>>>
>>>
>>>
>>>
>>>
>>> *Von:* hybi [mailto:[e-mail]] *Im Auftrag von *Joakim
>>> Erdfelt
>>> *Gesendet:* Mittwoch, 14. Mai 2014 20:12
>>> *An:* [e-mail]
>>> *Betreff:* [hybi] status of real world permessage-deflate
>>> implementations?
>>>
>>>
>>>
>>> So, I've been testing the server and client side implementations of
>>> various permessage-deflate implementations and come to realize that there
>>> is no consistency (yet).
>>>
>>>
>>>
>>> In Chrome 34, its client side decompress (received frames) is different
>>> than pywebsocket client decompress (received frames) which is different
>>> from autobahn testsuite's handling.
>>>
>>>
>>>
>>> Been testing our Jetty side implementation, and for large compressed
>>> messages (been using the Gutenberg project's text of "A Connecticut Yankee
>>> in King Arthur's Court, Complete, by Mark Twain" as my data).
>>>
>>> http://www.gutenberg.org/ebooks/86.txt.utf-8
>>>
>>>
>>>
>>> All tests are done with client upgrading to Jetty side using ..
>>>
>>>
>>>
>>> offer: permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
>>>
>>> negotiated: permessage-deflate
>>>
>>>
>>>
>>> I can make pywebsocket client happy, and it can send/receive single
>>> frame messages or fragmented messages (using either 16k and 32k max frame
>>> sizes) just fine.
>>>
>>> However, Chrome 34 errors out consistently if it receives a large
>>> fragmented message that is compressed.
>>>
>>> The inter-frame handling of the tail bytes (0x00, 0x00, 0xFF, 0xFF)
>>> appears to be different.
>>>
>>> In the last spec document,
>>> http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-18
>>> its clear that we must include the tail bytes.
>>>
>>> However, Chrome doesn't like that and fails the decompress w/abnormal
>>> closure.
>>>
>>> Inspection says:  failed to inflate a frame
>>>
>>> If I strip the tail bytes for Chrome 34, then its happy and pywebsocket
>>> is now broken.
>>>
>>> autobahn apparently doesn't care, one way or another, it accepts
>>> (happily) both with and without tail bytes between frames.
>>>
>>>
>>>
>>> --
>>>
>>> Joakim Erdfelt <[e-mail]>
>>>
>>> webtide.com <http://www.webtide.com/> - intalio.com/jetty
>>>
>>> Expert advice, services and support from from the Jetty & CometD experts
>>>
>>> eclipse.org/jetty - cometd.org
>>>
>>> _______________________________________________
>>> hybi mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/hybi
>>>
>>>
>>
>
> _______________________________________________
> hybi mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/hybi
>
>