Re: Declarative HTTP Spec Test Suite

Mohammed Al Sahaf <mohammed@caffeinatedwonders.com> Sat, 01 June 2024 00:22 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=ietf.org@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CED92C180B5F for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Fri, 31 May 2024 17:22:50 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.847
X-Spam-Level:
X-Spam-Status: No, score=-2.847 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=w3.org header.b="I1nz9Bwq"; dkim=pass (2048-bit key) header.d=w3.org header.b="n9/mFG7P"; dkim=pass (2048-bit key) header.d=caffeinatedwonders.com header.b="a9xB+6+t"
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SwJZ0_FFGOCD for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Fri, 31 May 2024 17:22:45 -0700 (PDT)
Received: from mab.w3.org (mab.w3.org [IPv6:2600:1f18:7d7a:2700:d091:4b25:8566:8113]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 34EB9C151069 for <httpbisa-archive-bis2Juki@ietf.org>; Fri, 31 May 2024 17:22:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Subject:Content-Type:MIME-Version:References:In-Reply-To:Message-ID: Cc:From:To:Date:Reply-To; bh=+hfPTczk1OggiW4ndLhXgUiLwXKBt4qxricczNyD1kM=; b= I1nz9BwqB+GmcMOha5Snpl5qVuVPjmnCYn2abuXue548Hhv3B3lbd7+psbsM97y27Qy52+nrTEagS BPExI3K6+xOuG7S86RLPSHmQLBzYewtMqyJliosaW7e+fd6YTMimzpZtpJfDh6wNel7FVwtUJWFyk dMuWxLEgldjUgMFH/vV881X3uhWopmMuLhxb6p09IrtbR5IL4KEw1V+6l6ZWMxs8YGHYSChvepXH1 8P1csJKMVB7wToT5uWRVYWFdWH7QQTuxCK1mfgh7h5mer2Tu57E1kz8wfVkGvDxXH4urYWsV0pvi4 FCk8jKGBBibSKZL3ggzu0TgIVWwzFckHMA==;
Received: from lists by mab.w3.org with local (Exim 4.96) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1sDCVH-006eLw-0e for ietf-http-wg-dist@listhub.w3.org; Sat, 01 Jun 2024 00:21:47 +0000
Resent-Date: Sat, 01 Jun 2024 00:21:47 +0000
Resent-Message-Id: <E1sDCVH-006eLw-0e@mab.w3.org>
Received: from ip-10-0-0-144.ec2.internal ([10.0.0.144] helo=pan.w3.org) by mab.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <mohammed@caffeinatedwonders.com>) id 1sDCVD-006eKy-1o for ietf-http-wg@listhub.w3.internal; Sat, 01 Jun 2024 00:21:43 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Subject: Cc:From:To:Date:Reply-To; bh=+hfPTczk1OggiW4ndLhXgUiLwXKBt4qxricczNyD1kM=; t=1717201303; x=1718065303; b=n9/mFG7Ps8CAeuUAvKwwC9rth8xwuU3K9moFwkd5vKr2BFQ +5dvDxDqHPCerwNHTXY1eEKlWktTKrKSxmOdGJd15YkJY6LldjxS4WfeD9lDfuS/dE+YaIno5jv+w NNUn+o9tbvx2Fsieo1XWbnSxgJhclLTjnF4q4pCJ5H6UGqbfipoBbyDXJ5YYwFGn8G5jYG7jUcwRp 5AhwFh1oKq/HPbqTs2ZpP2CL4e87wYJ0VYMHY9PGsRvxmvPvpYgte6cJ5OLz4VKsW3Rnv2rUWo0a1 /ipr3aktwteAlSccCFsGLlZIUNaQBEwMXIIszLtM9Jf0wYCtWof9HmP9H4XxYhwQ==;
Received-SPF: pass (pan.w3.org: domain of caffeinatedwonders.com designates 185.70.43.17 as permitted sender) client-ip=185.70.43.17; envelope-from=mohammed@caffeinatedwonders.com; helo=mail-4317.proton.ch;
Received: from mail-4317.proton.ch ([185.70.43.17]) by pan.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <mohammed@caffeinatedwonders.com>) id 1sDCV9-00A6VY-1P for ietf-http-wg@w3.org; Sat, 01 Jun 2024 00:21:43 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=caffeinatedwonders.com; s=protonmail3; t=1717201293; x=1717460493; bh=+hfPTczk1OggiW4ndLhXgUiLwXKBt4qxricczNyD1kM=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=a9xB+6+tEhOAYlr35OzNGmaZ+YaEFRIuQnVS71bjXIL9uxOnnBXyLy/gH4HOUhtEV j8qxCS7hhXjUmGeLWWPYy3m8NVI/v4KsgqHJf4mS9usxU9gOOO1Yjp6DZut4FdolNp AzYArgT5udRTnW1eHk6xOpOzGMDT+ZjKk52xxxjfCxcH02qLkYDl+OCCBJydYd0Fis zX5zGwmQv/bnuG9g9EFB7nnECgQ6jCW2foq8v/Wg9beLED29C3H9kFA4XbrwrD/VHT tPYZrVRPQv1JxMUt9Zs0mTebi166avR1vp2yb1f1aQsEu23UkZow6w/wTlSTpliWcR QT4a0uQz0WBPQ==
Date: Sat, 01 Jun 2024 00:21:29 +0000
To: Lucas Pardue <lucas@lucaspardue.com>
From: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>
Cc: David Benjamin <davidben@chromium.org>, HTTP Working Group <ietf-http-wg@w3.org>
Message-ID: <4yFWJGIjwen-3mujsV5W2ymNwxAH_nE6-iAtvq7V2jVLSd9bByAqf7iGd4EI0-9aRMMQboOUVz1JSLhhjxs1e8aPCA9qlvH3v1kTIjlqoxE=@caffeinatedwonders.com>
In-Reply-To: <a21e2452-a122-4ad3-8278-e70980a6a112@app.fastmail.com>
References: <tPxvQGS7_bxKqrFish00m9LtTT_JQfE0RU1bthEcxYNTZR_uzbkEWKNOTg9c5azoZSYuDWt9pLCiS7Kd4nIWK8DPN1sOtcgbDGx4L7BeeU4=@caffeinatedwonders.com> <CAF8qwaD7j=_9JgVS8jfDMJhJeo6cYq0QT4q-QB_TJ8W7uYCokw@mail.gmail.com> <jVdymWwPiPmYqYprR2XiciBoG01cibifkCA_G_ph_SQN0JRw6OvLNOEvCs0GeL93bre-J2c2X3degy5JtZiWF5BrC_C5dXkszQM6wLwF1fI=@caffeinatedwonders.com> <a21e2452-a122-4ad3-8278-e70980a6a112@app.fastmail.com>
Feedback-ID: 74227028:user:proton
X-Pm-Message-ID: 21ca6b754cc4845c087adc83a2092003f98a7880
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="b1_dAbPRywk3aJEMXL3w6TmJL0FwWLan3pA6wGAN7kAA"
X-W3C-Hub-DKIM-Status: validation passed: (address=mohammed@caffeinatedwonders.com domain=caffeinatedwonders.com), signature is good
X-W3C-Hub-Spam-Status: No, score=-4.1
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, DMARC_PASS=-0.001, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_DBL_BLOCKED_OPENDNS=0.001, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: pan.w3.org 1sDCV9-00A6VY-1P e599d55d3e9333422dcd936ee16fa829
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Declarative HTTP Spec Test Suite
Archived-At: <https://www.w3.org/mid/4yFWJGIjwen-3mujsV5W2ymNwxAH_nE6-iAtvq7V2jVLSd9bByAqf7iGd4EI0-9aRMMQboOUVz1JSLhhjxs1e8aPCA9qlvH3v1kTIjlqoxE=@caffeinatedwonders.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/51981
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/email/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

Hey Lucas,

Your confirmation (and others on other branches) of the validity of the need and value of the proposal is reassuring! :) I didn't see the reports of the earlier HTTP Workshops, but the report you've linked shows Daniel saying:

> Massaged the right way, this could develop into a generic HTTP test/conformance tool that could be valuable for and appreciated by even more users going forward.

That's exactly what I'm talking about, be it vtest or any other tool we converge on. I'm looking forward to your tool being accessible!

I learned of the HTTP Workshop recently and already planning on attending. I'll submit the statement of interest form soon.

To answer the bigger question, should this draft proposal be turned into a formal proposal now (in this thread?)? Or is it better left until the HTTP workshop where the larger discussion could take place?

All the best,
Mohammed
[Blog](https://www.caffeinatedwonders.com/) | [LinkedIn](https://www.linkedin.com/in/mohammedalsahaf/)

On Thursday, May 30th, 2024 at 1:03 AM, Lucas Pardue <lucas@lucaspardue.com> wrote:

> Hi Mohammed,
>
> Thanks for bringing this topic up. It's something I care about too.
>
> Others have already expressed some of the complications or caveats I might have mentioned. But in spite of those things, I think there is room for improvement around testing, especially through the lens of making HTTP's dark corners more accessible to the community.
>
> There's definitely more we can do in terms of tooling and testing. I don't think there is a one size fits all approach. vtest looked good when it was presented at the HTTP Workshop in 2019 [1], good to see it still working well.
>
> Something that's always nagged me is writing test oriented around the HTTP/2 and HTTP/3 framing layer. Especially since the H2 Rapid Rest attacks last year. Since November, I've been working on a tool to address (in Rust, since that suits my preference and our internal dev env). The tool aims to make it easier to do interactive exploration of HTTP/3 framing and stream lifecycle management, and to capture the results of that as a reproducible test sequence. It can also be controlled programmatically to write test cases, similar to vtest. We already had plans to open source this tool soon, so it's a nice coincidence to see the topic come up. An open question is if we should provide a corpus of test case examples that could emulate some of the capabilities of h2spec*. I'll post again when open sourcing happens.
>
> Mohammed there's another HTTP workshop planned this year, so perhaps consider coming along to talk testing in person with some of the wider community.
>
> Cheers
> Lucas
>
> * Kazu Yamamoto maintains h3spec and its great. However, it is a bit more focused on QUIC than HTTP/3. I've no doubt more test cases could be added to h3spec but for my own needs, I want a Rust-focused set of tooling.
>
> [1] - https://daniel.haxx.se/blog/2019/04/02/the-http-workshop-2019-begins/
>
> [2] - https://github.com/HTTPWorkshop/workshop2024/blob/main/README.md
>
> On Wed, May 29, 2024, at 21:56, Mohammed Al Sahaf wrote:
>
>> Good catch! I did a retest with HTTP/1 and HTTP/2 split. I excluded h2c because setting it up is a hassle for me given my convenient tools; plus it's unsupported by browsers, so perhaps only curl will have results on that. The result appears more consistent now.
>>
>> +---------+---------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------+
>> |         |                                                           HTTP/1                                                          |                                      HTTP/2                                        |
>> |         +-------------------------------------------------------------+-------------------------------------------------------------+------------+-----------------------------------------------------------------------+
>> |         |                             HTTP                            |                            HTTPS                            | HTTP (h2c) |                                HTTPS                                  |
>> +=========+=============================================================+=============================================================+============+=======================================================================+
>> | curl    | `curl: (18) transfer closed with 2 bytes remaining to read` | `curl: (18) transfer closed with 2 bytes remaining to read` |     N/A    | `(92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)` |
>> |         |                                                             |                                                             |            |                                                                       |
>> |         | response payload displayed                                  | response payload displayed                                  |            | response payload displayed                                            |
>> +---------+-------------------------------------------------------------+-------------------------------------------------------------+------------+-----------------------------------------------------------------------+
>> | Firefox | `NS_ERROR_PARTIAL_TRANSFER`                                 | `NS_ERROR_PARTIAL_TRANSFER`                                 |     N/A    | displays the full payload without reporting any errors                |
>> |         |                                                             |                                                             |            |                                                                       |
>> |         | response payload displayed                                  | response payload displayed                                  |            |                                                                       |
>> +---------+-------------------------------------------------------------+-------------------------------------------------------------+------------+-----------------------------------------------------------------------+
>> | Chrome  | `(failed)net::ERR_CONTENT_LENGTH_MISMATCH`                  | `(failed)net::ERR_CONTENT_LENGTH_MISMATCH`                  |     N/A    | displays the full payload without reporting any errors                |
>> |         |                                                             |                                                             |            |                                                                       |
>> |         | nothing displayed                                           | nothing displayed                                           |            |                                                                       |
>> +---------+-------------------------------------------------------------+-------------------------------------------------------------+------------+-----------------------------------------------------------------------+
>>
>> Given the feedback on the other branches of this thread, I think it's best to scope my proposals to servers. As mentioned elsewhere, clients are harder to test. To quote Willi
>>
>> Tarreau
>>
>> :
>>
>>> testing clients is very difficult because contrary to
>>
>>> servers which just have to respond to sollicitations, someone has to act
>>
>>> on the client to run the desired tests, so the approach is different
>>
>>> (and different between various clients), and I'm not convinced that a
>>
>>> same test collection would work for all implementations due to this.
>>
>> All the best,
>> Mohammed
>>
>> On Tuesday, May 28th, 2024 at 4:47 PM, David Benjamin <davidben@chromium.org> wrote:
>>
>>> The results in scenario 2 sound off. Chrome shouldn't treat ERR_CONTENT_LENGTH_MISMATCH differently between HTTP and HTTPS. I'm not familiar with their implementation, but the Firefox results similarly don't make sense. Indeed it's quite important to enforce this over HTTPS, for HTTP/1.1, because that is what defends against truncation attacks. (In principle, TLS has the close_notify alert, but close_notify is, in practice, a fiction for HTTPS. Instead we must rely on in-protocol termination signals. For HTTP/1.1, one of those signals is Content-Length.) Also it's generally on HTTPS that one can be more strict, not less, because there are fewer intermediaries to worry about.
>>>
>>> Given some of your errors mention HTTP/2, I suspect you are comparing apples to oranges, and your HTTPS tests are testing HTTP/2. You mention the Go standard library, but keep in mind that Go automatically enables HTTP/2. The Content-Length header means very different things between HTTP/1.1 and HTTP/2. In HTTP/1.1, it is a critical part of framing and needs to be checked at that layer. (HTTP/1.1's framing is incredibly fragile. "Text" protocols are wonderful.) In HTTP/2, it has no impact on framing and was historically[0] considered advisory. The spec now considers it invalid, otherwise an h2-to-h1 intermediary will have problems. [1] discusses this. But that's where this mess with receivers enforcing dates to. Provided it doesn't cause you to turn around and send mis-framed HTTP/1.1, it is more-or-less safe, if sloppy, to accept it in HTTP/2.
>>>
>>> David
>>>
>>> [0] https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-http2-00#section-3.2.2
>>> [1] https://www.rfc-editor.org/rfc/rfc9110#section-8.6-11
>>>
>>> On Mon, May 27, 2024 at 7:30 AM Mohammed Al Sahaf <mohammed@caffeinatedwonders.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> This is a proposal that is triggered by some of my involvement with the [Caddy web server](https://caddyserver.com/) project. We (Caddy team) have been working towards developing a declarative test suite for the Caddy server. The discussions (particularly a [comment](https://github.com/caddyserver/caddy/pull/6255#issuecomment-2088632219) by a user) led me to believe it's best to bring up the HTTP spec compliance parts with the HTTP WG for better insight and to have a common well-being check for all members of the HTTP community.
>>>>
>>>> There are numerous RFCs governing HTTP and the behavior of its citizens. Compliance to the RFCs is only validated through interoperability or manual eyeing of the RFCs against the implementation. The RFCs, for good reasons, are walls of texts and are akin to legalese when it comes to interpretation. Consequently, nuanced sections are possibly missed without visible failures due to being an edge-case. Having a specification defined as a test suite in a declarative language removes much of ambiguity and enables validation of conformance by the HTTP citizens. I hope to turn this into an official proposal, but I'd like to put the draft forward for discussion to solidify the approach and the scope first.
>>>>
>>>> Motivation:
>>>>
>>>> Conformance is an assurance of compatibility across the various components of the web and gives confidence of breakage if any of them were to change behavior or if the HTTP semantics were to change. Conformity can also assist in optimization efforts. If the behavior is known for sure in advance, certain optimizations can be applied.
>>>>
>>>> Secondly, it unifies the expectations of the community. Let's take for example the HTTP semantics of the Content-Length header as defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-length). The RFC states when servers and user-agents SHOULD, SHOULD NOT, MAY, MAY NOT, AND MUST NOT send the Content-Length header, but it does not specify how should either of them (server and user-agent) handle cases of mismatch between content-length header value and actual content length of the payload. I have run an unscientific poll on Twitter about the assumed ideal behavior of a client if the Content-Length value does not match the actual content-length of the body.
>>>>
>>>> [First poll](https://twitter.com/MohammedSahaf/status/1792267681032253683): What's the ideal HTTP client (e.g. curl, browser) behavior when the server includes more bytes in response body than stated in the content-length header? e.g. "Content-Length: 2", actual body length: 3.
>>>>
>>>> Responses (19 responses):
>>>>
>>>> - Ignore header; read fully (4 votes; 21.1%)
>>>>
>>>> - Read till content-length value (6 votes; 31.6%)
>>>> - Abort/reject (9 votes; 47.4%)
>>>>
>>>> Reality:
>>>> When testing this scenario, I found the following:
>>>>
>>>> - curl aborts the connection, reporting "(18) transfer closed with 1 bytes remaining to read" for plaintext HTTP connection, and "(92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)" for HTTPS connections.
>>>>
>>>> - Firefox fails the transfer on plaintext HTTP with "NS_ERROR_NET_PARTIAL_TRANSFER"; but with HTTPS connection, it only reads and displays payload per the number of bytes stated in the header value.
>>>>
>>>> - Chrome fails the transfer on plaintext HTTP with "(failed)net::ERR_CONTENT_LENGTH_MISMATCH"; but with HTTPS connection, it ignores the header value and displays the full payload.
>>>>
>>>> [Second Poll](https://twitter.com/MohammedSahaf/status/1792267687411831063): What's the ideal HTTP client (e.g. curl, browser) behavior when the server includes fewer bytes in response body than stated in the content-length header? e.g. "Content-Length: 5", actual body length: 3
>>>>
>>>> Responses (18 responses):
>>>>
>>>> - Ignore header; read 3 (7 votes; 38.9%)
>>>>
>>>> - Pad; with what? (0 votes; 0%)
>>>>
>>>> - Reject/abort (9 votes; 50%)
>>>>
>>>> - Other; comment (2 votes; 11.1%, none of them elaborated)
>>>>
>>>> Reality:
>>>> When testing this scenario, I found the following:
>>>>
>>>> - curl aborts the connection, reporting "(18) transfer closed with 1 bytes remaining to read" for plaintext HTTP connection; for HTTPS, it prints the payload in full preceded by the message "(92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)".
>>>>
>>>> - Firefox displays the full payload for HTTPS connections without reporting any errors. For plaintext HTTP, it displays the full payload but reports an error "NS_ERROR_NET_PARTIAL_TRANSFER"
>>>>
>>>> - Chrome displays the full payload for HTTPS connections without reporting any errors. For plaintext HTTP, it fails to load the content and reports the error "(failed)net::ERR_CONTENT_LENGTH_MISMATCH".
>>>>
>>>> [Third Poll](https://twitter.com/MohammedSahaf/status/1792267692700827955): Assuming it's possible... What's the ideal HTTP client (e.g. curl, browser) behavior when the server includes negative value in the content-length header? e.g. "Content-Length: -2"
>>>>
>>>> Responses (21 responses):
>>>>
>>>> - Ignore value (7 votes; 33.3%)
>>>>
>>>> - Reject/abort (14 votes; 66.7%)
>>>>
>>>> Reality:
>>>> I couldn't in effect test the scenario. I'm using Caddy for all the scenarios, and the Go standard library doesn't set the Content-Length header if it's set to a value of less than 0 ([source](https://github.com/golang/go/blob/377646589d5fb0224014683e0d1f1db35e60c3ac/src/net/http/server.go#L1201))
>>>>
>>>> Conclusion: The variation observed in the user agents and the, albeit unscientific, poll responses show a lack of consensus on the expected behavior. Each agent (human or machine) apply their own interpretations and assumptions to the protocol. The disagreement makes the evolution of the protocol difficult because of the varying expectations.
>>>>
>>>> Method:
>>>>
>>>> The test suite should be defined in declarative format that can be easily composed by humans and read by machines. The declarative, programming-language-agnostic format allows developers (RFC developers and software developers) of all backgrounds to contribute without a programming-language-based gate. In my research, I have found the open-source tool [hurl](https://hurl.dev/) to be a suitable tool for defining HTTP server specification and to test it. It defines its own DSL for the request/response patterns and may be run with --test flag along with --report-{json,html,tap} to produce test results.
>>>>
>>>> The testing effort may be implemented in phases. The first phase is to author the test suite in a common, public repository. This makes it accessible for web server developers to clone it and run the test suite against their own software. The second phase is to provide an interface for automated testing and a UI to display the conformance summary of each web server submitted to the list.
>>>>
>>>> Challenges:
>>>>
>>>> Agnostic Tooling: The first challenge is to find an HTTP client that implements HTTP Semantics RFCs perfectly, otherwise its own idiosyncrasies will get in the way of the validation. One would be tempted to default to curl, especially that hurl uses curl under the hood. However, there is a chance curl may have its own set of HTTP idiosyncrasies that may affect the results of the test execution. Changes to curl are probably not desired unless the subject behavior is confirmed to be a defect. Involvement of curl is voluntary to curl, and the team may be looped and involved into this initiative for comments if desired.
>>>>
>>>> Suitable DSL (Domain Specific Language): The hurl DSL is decent. In my experiment for a proof-of-concept, I found it lacking a few functions or operations to be perfectly suitable, e.g. indicating optionality. hurl has the advantage of its DSL grammar being defined as a spec with deterministic parsing. Extensions and/or changes to hurl to accommodate this effort is up to the hurl developers.
>>>>
>>>> Prior Art:
>>>>
>>>> There's a wiki page title [HTTP Testing Resources](https://github.com/httpwg/wiki/wiki/HTTP-Testing-Resources) under the github.com/httpwg/wiki repository. The page contains the following note:
>>>>
>>>> -
>>>>
>>>> Note that there is no official conformance test suite. These tests have not been vetted for correctness by the HTTP Working Group, and the authority for conformance is always the relevant specification.
>>>>
>>>> Indeed some of the listed projects have not been updated for a while. Worthy of note on the page is the cache-tests.fyi project, REDBot, and httplint (which REDBot uses).
>>>>
>>>> The REDBot project (by Mark Nottingham) was used by one of Caddy users [to report a gap](https://github.com/caddyserver/caddy/issues/5849) in the conformance, which was subsequently fixed. Using REDBot requires pointing at a particular resource.
>>>>
>>>> The [cache-tests.fyi](https://cache-tests.fyi/) is of keen interest for some inspiration of design. The test suite is close in essence to this proposal, which is a declarative suite that can be run by cache system developers to validate their conformance to the cache related RFCs. The [Souin](https://github.com/darkweak/souin) caching system runs the cache-tests.fyi test suite on every pull request to note its conformance level and to watch for variations. The display of each system, the use case, and its conformance status allows the users and the developers to take appropriate actions. The website and the UI can be a phase-2 (long-term goal) of this proposal, but the details of how to set up the system and run the tests can be postponed until more information is known about the test suite itself.
>>>>
>>>> Humble Attempt:
>>>>
>>>> To test the idea and develop a proof-of-concept, I have managed to convert 4 tests from REDBot (procedural, Python-based) to Hurl (declarative format). The test suite contains a collection of test sets in nested directory structure. The suite declares the URL it will call for each test case so web servers can be configured accordingly for the subject URL. The GitHub repository for the PoC is here: github.com/mohammed90/http-semantics-test-suite. The repository currently does not have a license applied as it's only for display of PoC, though I am inclined to apply Apache-2 or any other open-source-compatible license once the approach is agreed and finalized.
>>>>
>>>> All the best,
>>>> Mohammed
>>>> [Blog](https://www.caffeinatedwonders.com/) | [LinkedIn](https://www.linkedin.com/in/mohammedalsahaf/)