Re: [hybi] websocket test suite

[Tobias Oberstein <[e-mail]>]

> Here is a first cut at a list of tests for section 4 of the spec.  Are
> these the kinds of tests people were thinking about?
> If so, then I'm happy to go the next step of detail and start
> providing more detail for each test.

I am interested in this, and I think a list of tightly specified,
executable test cases which read on and exhaust the spec is definitely
desirable.

Started a little project just days ago which includes a fuzzing server:
https://github.com/oberstet/Autobahn
Using this, I found some behavior in FF>=7 (which is on the new protocol 8)
in particular with respect to fragmentation and TCP cleanness (see below)
which I believe are issues.

> 
> Obviously some implementation are not going to be able to send some of
> the invalid frames or force fragmentation, so some suites will have to
> record some tests as skipped, or only testing in one direction etc.

Besides fragmentation, I would like to suggest "TCP clean" in addition
and in combination. By that I mean: depending on implementation of an
endpoint, there might be issues related in particular to how the octet
stream is chopped up on the wire / received by the implementation.
Independent of or only in combination with WebSockets message fragmenta-
tion. For an example, pls see
https://bugzilla.mozilla.org/show_bug.cgi?id=664344

Obviously, an implementation must be agnostic to the chops in which it
receives octets from an octet stream (TCP).

Currently, I am testing: a) force octets to wire after each frame and
b) force out each octet onto wire separately. I might add: c) force out
"random" numbers of octets. b) is also a good test to catch "naive"
implementations doing inefficient concatenations involving reallocations.

In general, I believe it might be worth looking not only into tests
catching blunt protocol violations, but memory/performance related issues
as well. I think of at least having test drives which torture the
implementation with regard to memory leaks.

> 4.2 Base Framing Tests
> ----------------------
> 
> * 7 bit length: send/receive frames of length 0-125

Suggestion: Control frames must have payload max. 125, but can have 0.

I wanted to ask this anyway: is it valid for data frames to have 0
payload? If yes, is a data message of payload 0 allowed?

I am asking, because current FF will happily consume messages
of payload 0 (not only individual fragments), and then generate
a JS onEvent() with message = empty string. I did not find anything
in the spec regarding this .. but I am not sure if its helpful for
a JS onEvent() to be fired on an empty string.

> * 16 bit length: send/receive frames of length 126-128, 65536
> * 63 bit length: send/receive frames of length 65537

Good idea to test for the integers around the cross-over points.
We could add testing for (MSB==1 with 8 octets len) => fail, to
be sure host languages which don't have unsigned longs don't
naively produce negative length when converting octets to their
signed long.

On the other hand: what do you do in an implementation upon
receiving the first frame of a fragmented message, and the frame
says it has length 2GB? I would just bail out with "frame too large"
anyway. I would find it unreasonable to allocated that amount.

Maybe we should test if, and when at which frame size the
implementation bails out with "frame too large".

> 
> * Non zero RSV: send frames with non zero reserved bits and no
> extension. Verify connection is failed.

Yep. The tests should verify each RSV bit combination .. 0-7.

> 
> * Known Opcodes: send/receive frames with text,binary,ping,pong
> opcodes (continuation & close tested separately).
> * Unknown Opcodes: send frames with unknown opcodes 3-7, B-F. Verify
> connection is failed.

Yep. And yes: the tests should verify each of the reserved opcodes.
For example, current FF correctly fails on reserved control frame
opcodes, but silently ignores reserved data frame opcodes.

> 4.3 Client to Server Masking
> ----------------------------
> 
> ???

Couple of ideas:
Obviously, all c2s frames must be masked.
Mask must be != 0.
Mask of frame N must be != mask for frame N-1 (,N-2, N-k).
Those 2 are just to catch lazy implementations, of course no
rigorous check of "the mask must use good entropy source".

What about testing server-to-client masking? The spec says both endpoints
must be capable of processing masked frames ..

> 4.4 Fragmentation
> -----------------
> 
> * 1 fragment message: send/receive message in 1 fragment
> * 2 fragment message: send/receive message in 2 fragments
> * 3 fragment message: send/receive message in 3 fragments
> * Injected control: send/receive 2 fragment message with unsolicited
> pong message between frames
> * Fragmented control: send fragmented ping. Verify that connection is failed.
> * Interleaved fragments: send a fragment with FIN set and Continuation
> opcode. Verify that connection is failed.

Not sure if I get you right here: I see 2 classes of violations:

a) Within fragmented message, a data frame with opcode == 1/2 and FIN == 1/0
=> continuation not closed (with opcode == 0 and FIN == 1)

b) Outside fragmented message, a data frame  with opcode == 0 and FIN == 1/0
=> nothing to continue (no previous opcode == 1/2 and FIN == 0)

6 cases in total. The valid OpCode:FIN sequences

unfragmented: 1/2:1
fragmented: 1/2:0, (0:0)*, 0:1

My implicit expectation was that an endpoint must fail immediately on any of
above 6 bogus cases, but I just can't find that explicit in the spec .. what
is intended?

> * Handled control: send/receive 2 fragment message with ping message
> between frames and last frame delayed. Verify pong is received before
> last frame is sent.
> * 7 bit fragment: send/receive message with initial/middle/final
> fragments of sizes 0-125
> * 16 bit fragment: send/receive message with initial/middle/final
> fragments of sizes 126-128,65536
> * 64 bit fragment: send/receive message with
> initial/middle/finalfragments of size 65537


> 4.5.1 Close Frames
> ------------------
> * Orderly Close: On an idle connection, send a close and receive a close.
> * Busy Close: send a close to an endpoint that has sent the first
> fragment of a message.
> * Simultaneous Close: send a close on both ends of a connection.
> * Message after close: attempt to send a message after a close. Verify
> message is not received.
> * Idle close: ??
> * Error Close: ??

A minor addition: the spec says, a close frame with payload > 0 must have
the first 2 octets being an status code, and may have additional content,
which then is UTF-8. So payload length = 1 is illegal, and if payload > 2,
then the rest must be UTF-8.

> 4.5.2 Ping
> ----------
> * Empty ping: Send an empty ping, verify pong is received.
> * Non empty ping: Send a non empty ping, verify pong is received.

Also something I wanted to ask anyway: the spec says

   Upon receipt of a Ping frame, an endpoint MUST send a Pong frame in
   response.  It SHOULD do so as soon as is practical.
...
   If an endpoint receives a Ping frame and has not yet sent Pong
   frame(s) in response to previous Ping frame(s), the endpoint MAY
   elect to send a Pong frame for only the most recently processed Ping
   frame.

Lets say: client has 1 large data message to send, is sending fragments
all the time. Receives Ping, but decides that it is "not practical" to
send Pong until _all_ fragments of big data message have been sent.
Is that valid?
What is the binding meaning of the words "SHOULD" and "practical"?
If there is none, how do you "verify" negatively? I mean: if there is
a Pong, fine. Conforming (at least under that load conditions). But
no Pong does not imply non-conforming, isn't it?

> 4.5.3 Pong
> ----------
> * Empty pong: Send an empty ping, verify pong is received and is empty.
> * Non empty pong: Send a non empty ping, verify pong is received and
> has matching content.
> * Outstanding ping: Send two pings with different content, reply with
> content for only the second. Verify connection stays open.
> * Unsolicited pong: Send/recv unsolicited pong.

Another point: if I've not overlooked something, the spec does not
constraint the payload (besides length). Maybe test for binary cleanness,
at least non-UTF-8 payload.

> 4.5.4 Data Frames
> -----------------
> * UTF-8 Text 0000-007f: Send/receive text message with characters in
> code point range  U+0000 to U+007F
> * UTF-8 Text 0080-07ff: Send/receive text message with characters in
> code point range  U+0080 to U+07FF
> * UTF-8 Text 0800-ffff: Send/receive text message with characters in
> code point range  U+0800 to U+FFFF
> * UTF-8 Text 010000-10ffff: Send/receive text message with characters
> in code point range  U+010000 to U+10FFFF
> * Illegal bytes UTF-8: Send message with invalid UTF-8 byte sequence.
> Verify message is not received.
> * Illegal codes UTF-8: Send message with invalid UTF-8 code points.
> Verify message is not received.
> * Binary frames: Send/receive message containing illegal UTF-8 bytes
> sequences and code points as binary message.

Maybe add tests:
- fragmented text message, fragmented within octet sequence for single code
point
- fail fast: fragmented text message with illegal UTF-8 fails already
upon receiving the frame containing the last octet of illegal sequence,
not only after receiving all message frames.