Re: [http-devops] HTTP/2 test suite is now available!

Tatsuhiro Tsujikawa <tatsuhiro.t@gmail.com> Sat, 01 February 2014 07:40 UTC

Return-Path: <tatsuhiro.t@gmail.com>
X-Original-To: http-devops@ietfa.amsl.com
Delivered-To: http-devops@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE5201A0536 for <http-devops@ietfa.amsl.com>; Fri, 31 Jan 2014 23:40:12 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.693
X-Spam-Level:
X-Spam-Status: No, score=-0.693 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001, TRACKER_ID=1.306] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L7NCKEN49E37 for <http-devops@ietfa.amsl.com>; Fri, 31 Jan 2014 23:40:07 -0800 (PST)
Received: from mail-oa0-x22b.google.com (mail-oa0-x22b.google.com [IPv6:2607:f8b0:4003:c02::22b]) by ietfa.amsl.com (Postfix) with ESMTP id 2925F1A0534 for <http-devops@ietf.org>; Fri, 31 Jan 2014 23:40:07 -0800 (PST)
Received: by mail-oa0-f43.google.com with SMTP id h16so6265682oag.16 for <http-devops@ietf.org>; Fri, 31 Jan 2014 23:40:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=KfnMQIpQD6bLnopu9RCHHFM5tpIxFmPxgc0vP5IrvDw=; b=JoYp/9iRjLwr758JLTWYUApW6EBHQ0snpIhp6Oh3IY+oo890vuFbYxy2UFuSmQZQSB K0ZvINMhvMWduaWnj1PYQJJrmuGP4vmyy1r69uEbnbQF2WwrwGIa/tZ0lLv0K8xmC7Ky jriub+1/kYdjA6fJJO3Mh+buM7ckVg658/qsO3UrSjkESVb82zLegQn6p00JJ1WoUJpO Pd1L7HfPFUeeO+ty1TF2hs5EznLJ/yBjO6K60XUf0tjoKcsgs+/27trqfyAe2ySd+7EO fNcMa5PkcysTldskwgeeKjJ0lFt5yWDZdT1O6E3GD5Wtr2gljM7WG+tbboKUmmKY7Cqg nAAg==
X-Received: by 10.60.165.72 with SMTP id yw8mr18419oeb.71.1391240403101; Fri, 31 Jan 2014 23:40:03 -0800 (PST)
MIME-Version: 1.0
Received: by 10.60.172.80 with HTTP; Fri, 31 Jan 2014 23:39:42 -0800 (PST)
In-Reply-To: <e8ca3c7f08364e0cb4ce252a80f93480@DM2PR03MB366.namprd03.prod.outlook.com>
References: <d167bbd5f75f4a97882c126cab095cdd@BL2PR03MB164.namprd03.prod.outlook.com> <CAPyZ6=KoAiFUWeZNm-JaTaZhJOxzgwymbMLfNpHd0FGCQEo_jw@mail.gmail.com> <e8ca3c7f08364e0cb4ce252a80f93480@DM2PR03MB366.namprd03.prod.outlook.com>
From: Tatsuhiro Tsujikawa <tatsuhiro.t@gmail.com>
Date: Sat, 01 Feb 2014 16:39:42 +0900
Message-ID: <CAPyZ6=J9o-Yq54HSPp8AMvF-BO-2k5biLFYexVqs3aX2BmJ7zQ@mail.gmail.com>
To: "Parashuram Narasimhan (MS OPEN TECH)" <panarasi@microsoft.com>
Content-Type: multipart/alternative; boundary="047d7b41904b062e8204f15366b5"
Cc: "Brian Raymor (MS OPEN TECH)" <Brian.Raymor@microsoft.com>, "Jeff Mendoza (MS OPEN TECH)" <jemendoz@microsoft.com>, "http-devops@ietf.org" <http-devops@ietf.org>, "Elancheziyan, Anbu" <aelanche@akamai.com>, Michelle Lai <miclai@microsoft.com>, Rob Trace <Rob.Trace@microsoft.com>
Subject: Re: [http-devops] HTTP/2 test suite is now available!
X-BeenThere: http-devops@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "Discussion about operation, deployment and testing of HTTP" <http-devops.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/http-devops>, <mailto:http-devops-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/http-devops/>
List-Post: <mailto:http-devops@ietf.org>
List-Help: <mailto:http-devops-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/http-devops>, <mailto:http-devops-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 01 Feb 2014 07:40:13 -0000

On Sat, Feb 1, 2014 at 7:31 AM, Parashuram Narasimhan (MS OPEN TECH) <
panarasi@microsoft.com> wrote:

>  Hi,
>
>
>
> I wanted to ping you again to see if you were able to get the test cases
> running for nghttp.
>
> We were also able to download your code and ran some initial test cases to
> ensure that we are doing it right on our end. The left column is the
> filenames and should give you an idea of how the http2 payload was sent to
> your server.
>
>
>

With HTTP2_BROWSER environment variable, I can run the test against nghttp
client. Thank you.


>  We did find a few bugs in our implementation, thanks to your code and
> have fixed the tests. Many of the tests passed. The ones labelled
> INCORRECT TEST were issues with the test framework and we have fixed most
> of the issues. We also found some issues other issues that may be in
> nghttp2. It would be great if you could take a look at it and confirm that
> the fails are not caused due to the test framework.
>
>
>
> It would be great if you can continue running your implementation against
> the test framework and let us know if you encounter issues with the test
> framework itself.
>
>
>

- invalid-level-goaway-test

I fixed this sometime ago in nghttp2

- *-when-closed-test

I'm thinking they are really necessary. The spec says it is RST_STREAM
situation, but responding each frame
with RST_STREAM is just wasting the processing and memory (e.g., many DATA
frames comes from the remote and responding each with
RST_STREAM). So "smart" client may remember when RST_STREAM is issued to
certain stream, but what gains if we do that using extra
memory. As a good compromise, I think it is best to ignore these frames.

- *-when-idle-test

I thought that these are the same categories of *-when-closed-test. So I
did not implement the check for this idle test.
It is not generally possible to detect that which stream ID is currently
idle without remembering all stream ID used so far,
especially those which are smaller than the last used/received stream ID.
But it is possible to detect idle stream which has larger stream ID than
the currently used. The current test case uses
stream ID 2, which is just the case. I have pending local commits to detect
this situation to pass the tests.


- push-promise-when-reserved-remote-test
With the latest nghttp2 and http2-test, this test passes.

- priority-when-reserved-remote
What does WRONG TEST mean?


I don't know why but the following tests fail on my PC:

- huge-index-test
  Error: done() invoked with non-Error: Not appropriate error code:
COMPRESSION_ERROR

- invalid-data-1-test
  Error: timeout of 10000ms exceeded

- invalid-data-2-test
  Error: timeout of 10000ms exceeded

- oversized-ping-test
  Error: done() invoked with non-Error: Not appropriate error code:
FRAME_SIZE_ERROR

Best regards,

Tatsuhiro Tsujikawa


>
> *Multiplexing*
>
> *NGHTTP2*
>
> invalid-level-data-test
>
> passed
>
> invalid-level-goaway-test
>
> FAIL - nghttp2 did not respond any error when receiving GOAWAY with
> specified StreamId - 6.8.  GOAWAY
>
> invalid-level-headers-test
>
> passed
>
> invalid-level-ping-test
>
> passed
>
> invalid-level-priority-test
>
> passed
>
> invalid-level-push-promise-test
>
> passed
>
> invalid-level-rst-stream-test
>
> passed
>
> invalid-level-settings-test
>
> passed
>
> *Framing*
>
> oversized-ping-test
>
> passed
>
> *Push*
>
> invalid-promised-id-1-test
>
> passed
>
> invalid-promised-id-2-test
>
> passed
>
> invalid-promised-id-3-test
>
> passed
>
> *Compression*
>
> huge-index-test
>
> passed
>
> invalid-data-1-test
>
> passed
>
> invalid-data-2-test
>
> passed
>
> *Stream*
>
> data-when-closed-test
>
> INCORRECT TEST  -waiting for connection error on closed stream - 5.1
> (closed state)
>
> priority-when-closed-test
>
> INCORRECT TEST  -waiting for connection error on closed stream - 5.1
> (closed state)
>
> push-promise-when-closed-test
>
> INCORRECT TEST  -waiting for connection error on closed stream - 5.1
> (closed state)
>
> reservation-when-closed-test
>
> INCORRECT TEST  -waiting for connection error on closed stream - 5.1
> (closed state)
>
> rst-stream-when-closed-test
>
> INCORRECT TEST  -waiting for connection error on closed stream - 5.1
> (closed state)
>
> window-update-when-closed-test
>
> INCORRECT TEST  -waiting for connection error on closed stream - 5.1
> (closed state)
>
> data-when-idle-test
>
> FAIL - nghttp2 did not fail (did not send GOAWAY) when sending data on
> idle stream  - 5.1 - last paragraph
>
> priority-when-idle-test
>
> FAIL - nghttp2 did not fail (did not send PRIORITY) when sending data on
> idle stream  - 5.1 - last paragraph
>
> push-promise-when-idle-test
>
> FAIL - RST_STREAM (stream error) instead of GOAWAY (connection error) when
> sending PUSH_PROMISE on stream in IDLE state - 5.1 - last paragraph
>
> rst-stream-when-idle-test
>
> FAIL - nghttp2 did not fail when sending RST_STREAM on stream in IDLE
> state - 6.4 last paragraph
>
> window-update-when-idle-test
>
> FAIL - nghttp2 did not fail when sending WINDOW_UPDATE on stream in IDLE
> state - 5.1 - last paragraph
>
> data-when-reserved-remote-test
>
> passed
>
> priority-when-reserved-remote
>
> WRONG TEST -      An endpoint MAY send a PRIORITY frame in this state to
> reprioritize the reserved stream. - 5.1 (reserved-remote)
>
> push-promise-when-reserved-remote-test
>
> FAIL - RST_STREAM instead of GOAWAY -    Receiving any other type of frame
> other than HEADERS or RST_STREAM MUST be treated as a connection error
> (Section 5.4.1) of type      PROTOCOL_ERROR. - 5.1 (reserved-remote)
>
> reservation-when-reserved-remote-test
>
> passed
>
> window-update-when-reserved-remote-test
>
> passed
>
>
>
>
>
> *From:* Parashuram Narasimhan (MS OPEN TECH)
> *Sent:* Saturday, January 18, 2014 9:13 AM
> *To:* 'Tatsuhiro Tsujikawa'; Michelle Lai
> *Cc:* http-devops@ietf.org; Brian Raymor (MS OPEN TECH); Elancheziyan,
> Anbu; Rob Trace
> *Subject:* RE: [http-devops] HTTP/2 test suite is now available!
>
>
>
> Hi,
>
>
>
> You can see the instructions to run the client test cases here -
> https://github.com/http2/http2-test/blob/master/README.md#testing-the-client.
> We do not have any server test cases and are working on adding more tests
> for both client and server. Note that the client is expected to talk HTTP2
> directly without upgrade.
>
>
>
> *From:* Tatsuhiro Tsujikawa [mailto:tatsuhiro.t@gmail.com<tatsuhiro.t@gmail.com>]
>
> *Sent:* Saturday, January 18, 2014 4:52 AM
> *To:* Michelle Lai
> *Cc:* http-devops@ietf.org; Parashuram Narasimhan (MS OPEN TECH); Brian
> Raymor (MS OPEN TECH); Elancheziyan, Anbu; Rob Trace
> *Subject:* Re: [http-devops] HTTP/2 test suite is now available!
>
>
>
> Hi,
>
>
>
> Thank you for this effort, testing platform is a great step forward for
> HTTP/2.0 development.
>
>
>
> I'd like to test nghttp2 client/server with this test suite, but I could
> not find the document how to hook the test to the ordinary command line
> tool.
>
> Could you tell me how to do it?
>
>
>
> Best regards,
>
> Tatsuhiro Tsujikawa
>
>
>
> On Fri, Jan 17, 2014 at 1:04 PM, Michelle Lai <miclai@microsoft.com>
> wrote:
>
>  Hi Everyone,
>
> I'd like to announce that HTTP/2 test suite is now available at
> https://github.com/http2/http2-test/
>
>
>
> The test suite is based on the principles of HTTP/2 Protocol test draft:
> http://www.ietf.org/id/draft-trace-httpbis-http2-test-00.txt to:
>
> ·         focus on HTTP/2 protocol compliance testing
>
> ·         contain client and server test cases
>
> ·         use platform-independent programming language - tests are
> written in JavaScript
>
> ·         provide cross-platform support - use NodeJS, grunt, mocha, and
> GÃbor's node-http2 module as the 09 implementation (will be updated as the
> draft moves forward)
>
>
>
> Usage and proposed test scenarios are listed in readme and wiki at GitHub.
>
>
>
> Currently we have few contributors: Akamai, Microsoft Open Tech, and GÃbor
> and are seeking more contributors to:
>
> ·         Implement more test cases
>
> ·         Improve framework robustness
>
> ·         Make framework as part of your build process, tweak it so that
> it works
>
>
>
> Thanks,
>
> Michelle
>
>
> _______________________________________________
> http-devops mailing list
> http-devops@ietf.org
> https://www.ietf.org/mailman/listinfo/http-devops
>
>
>