From ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=ietf.org@listhub.w3.org Mon May 27 15:57:19 2024 Return-Path: X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1849CC14F6B0 for ; Mon, 27 May 2024 15:57:19 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.847 X-Spam-Level: X-Spam-Status: No, score=-2.847 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, MAILING_LIST_MULTI=-1, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=w3.org header.b="IZgdD8wd"; dkim=pass (2048-bit key) header.d=w3.org header.b="fk2M1v2I"; dkim=pass (2048-bit key) header.d=caffeinatedwonders.com header.b="Aoas5G5w" Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mNpUcq02p0PI for ; Mon, 27 May 2024 15:57:13 -0700 (PDT) Received: from mab.w3.org (mab.w3.org [IPv6:2600:1f18:7d7a:2700:d091:4b25:8566:8113]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A778CC14F6F2 for ; Mon, 27 May 2024 15:57:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Subject:Content-Type:MIME-Version:References:In-Reply-To:Message-ID: Cc:From:To:Date:Reply-To; bh=yP05LiXXGTubVHwRebUcNoGIAuGyjEsT1a2ZgI5wYJ4=; b= IZgdD8wdl/59koj3x3iou0LX7hzU/82FI2sFq/iXk64Zx3rNnenq4OtcSF5hydF2i3AuMzgoYyWb6 MqA9RSPm+m4ZoyEL+IlNzceIYgAkKNEBv0FkHiCe7wiiNCv+GS045XEpx7LPUn3Hic73yE1k5ZMjl pEHpVBhMOlWVlhecUoSziTGx7Wk08b7G5dyIu1Snpl5AbQsl8ooi8eTraat9zsN9oHBgkRF+2UtaL GV72fa7R3rcFezdia1jNhug5xqvaew7IQ7REeBq7FnFvA6cY6aVZln4vb9eBASJCnY1VXFQuqz6Xx Uj4q5M/0z+z535/1CcYUPx383RwGzl7c/Q==; Received: from lists by mab.w3.org with local (Exim 4.96) (envelope-from ) id 1sBjGM-00DQF4-2c for ietf-http-wg-dist@listhub.w3.org; Mon, 27 May 2024 22:56:18 +0000 Resent-Date: Mon, 27 May 2024 22:56:18 +0000 Resent-Message-Id: Received: from ip-10-0-0-144.ec2.internal ([10.0.0.144] helo=pan.w3.org) by mab.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1sBjGL-00DQE5-0s for ietf-http-wg@listhub.w3.internal; Mon, 27 May 2024 22:56:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Subject: Cc:From:To:Date:Reply-To; bh=yP05LiXXGTubVHwRebUcNoGIAuGyjEsT1a2ZgI5wYJ4=; t=1716850577; x=1717714577; b=fk2M1v2IdYKbS6VBEOylVBK09/4GTftZQklIaDnf3oyJ6Xl XWeFG6vmScV9ijNMiDwdpPt1F55tlsRYvL1fNGXPiVwzf/Lg/mE//6XlTyhCPddbiPXkBH2EniUBI K0xuD49n4dXxBDSgnq46GbM1ZjiXa2CNXJi6mKMJXSKdKLIdXa+xRjMAGIKGT+jeST6Hb3ZJSZVXh p6FudJshX9sGDzXQKHw2qrNY0Ujx2bRsKZFFSFtSechAt2GsGDkZfjTMjJs+0OF3QYvNeQFMaU7Au WZQtDl6MBbKhhb7A5RO18l7wcR4MTqpJJ9qrYmCIT8apCPv5QxlXZ936KxTHcCWg==; Received-SPF: pass (pan.w3.org: domain of caffeinatedwonders.com designates 185.70.40.22 as permitted sender) client-ip=185.70.40.22; envelope-from=mohammed@caffeinatedwonders.com; helo=mail-4022.proton.ch; Received: from mail-4022.proton.ch ([185.70.40.22]) by pan.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1sBjGJ-008Zak-2t for ietf-http-wg@w3.org; Mon, 27 May 2024 22:56:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=caffeinatedwonders.com; s=protonmail3; t=1716850571; x=1717109771; bh=yP05LiXXGTubVHwRebUcNoGIAuGyjEsT1a2ZgI5wYJ4=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=Aoas5G5wIsPhCNjA+hXTmau1mzQFXZ5j7/exMdFwm9/kpEGkxq2rpgRPc6Ms+LQVw G9VMy+utbSjikoFdOgC3ZAHG7ABk6VrZYJBR6MJWsZG6VVrXhYUEG6sWoNiaVNVcFD m9mrIo3I+4GCdqWmbmvqgAmN31YvprnWysyRwJMURJLxq2urYtKzrMyVohOcuNNFKJ JbGDRP+3g/xlHNLqfVgtzQAQcHog6KDMSxymdksu0DmkcF927tQRs+HBBHEcAvp0L4 oEAgNlB19UFUloUoE9gTZR7WukUVArPKzuJ+EsA2R9g0dlthfaSKytJ4B/4Nicf/jf N79KYi6coA+qg== Date: Mon, 27 May 2024 22:56:05 +0000 To: Willy Tarreau From: Mohammed Al Sahaf Cc: Poul-Henning Kamp , "ietf-http-wg@w3.org" Message-ID: <7c8w3fIP0cy2UKgXbcibDe3LsYUtM0se9Ew0QixqV7809W0PhMkNJUdH4cw4DDDaurLc1osZ_M-KTMjjrifUayN-3ElVCI5EWFyGwBZNcoY=@caffeinatedwonders.com> In-Reply-To: References: <202405271304.44RD4iQj027668@critter.freebsd.dk> Feedback-ID: 74227028:user:proton X-Pm-Message-ID: 9a792c0a6bcfdc5ba1ccc5891abd17d5f6a8b441 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-W3C-Hub-DKIM-Status: validation passed: (address=mohammed@caffeinatedwonders.com domain=caffeinatedwonders.com), signature is good X-W3C-Hub-Spam-Status: No, score=-4.1 X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, DMARC_PASS=-0.001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_DBL_BLOCKED_OPENDNS=0.001, W3C_AA=-1, W3C_WL=-1 X-W3C-Scan-Sig: pan.w3.org 1sBjGJ-008Zak-2t 25b99083c970c65be083d812e9717bae X-Original-To: ietf-http-wg@w3.org Subject: Re: Declarative HTTP Spec Test Suite Archived-At: Resent-From: ietf-http-wg@w3.org X-Mailing-List: archive/latest/51972 X-Loop: ietf-http-wg@w3.org Resent-Sender: ietf-http-wg-request@w3.org Precedence: list List-Id: List-Help: List-Post: List-Unsubscribe: On Monday, May 27th, 2024 at 4:23 PM, Willy Tarreau wrote: > On Mon, May 27, 2024 at 01:04:44PM +0000, Poul-Henning Kamp wrote: >=20 > > > This is a proposal that is triggered by some of my involvement with t= he Caddy > > > web server project. We (Caddy team) have been working towards develop= ing a > > > declarative test suite for the Caddy server. > > > [...] > > > Prior Art: > > > [...] > > > Indeed some of the listed projects have not been updated for a while. > >=20 > > Mohammed, > >=20 > > I may be misunderstanding you, but using repository activity as a > > screen will make you miss stuff which Just Works(TM). Fair enough! I stand corrected. > > I will argue that VTest (https://github.com/vtest/VTest) is one > > such: It chugs away, day in and day out, in both the Varnish Cache > > and HAproxy projects. > >=20 > > A specific reason for repository inactivity, which you might not > > have noticed, is that VTest is just the test program code and the > > test-cases do not live in the VTest respository, but in the "parent" > > projects which use VTest: > >=20 > > https://github.com/haproxy/haproxy/tree/master/reg-tests > >=20 > > and > >=20 > > https://github.com/varnishcache/varnish-cache/tree/master/bin/varnishte= st/tests > >=20 > > Full VTest documentation is under "Varnishtest" in the varnish project: > >=20 > > http://varnish-cache.org/docs/trunk/reference/index.html > >=20 > > I started writing VTest a dozen years ago for many of the same > > reasons you touch in your analysis, but one of my other reasons > > was that I wanted a good language to write test-cases in. > >=20 > > Here is a basic test from Varnish Cache, checking that chunked > > encoding works at all: > >=20 > > varnishtest "Check chunked encoding from backend works" > >=20 > > server s1 { > > rxreq > > expect req.url =3D=3D "/bar" > > send "HTTP/1.1 200 OK\r\n" > > send "Transfer-encoding: chunked\r\n" > > send "\r\n" > > send "00000004\r\n1234\r\n" > > send "00000000\r\n" > > send "\r\n" > >=20 > > rxreq > > expect req.url =3D=3D "/foo" > > send "HTTP/1.1 200 OK\r\n" > > send "Transfer-encoding: chunked\r\n" > > send "\r\n" > > send "00000004\r\n1234\r\n" > > chunked "1234" > > chunked "" > > } -start > >=20 > > varnish v1 -vcl+backend {} -start > >=20 > > client c1 { > > txreq -url "/bar" > > rxresp > > expect resp.status =3D=3D 200 > > expect resp.bodylen =3D=3D "4" > > txreq -url "/foo" > > rxresp > > expect resp.status =3D=3D 200 > > expect resp.bodylen =3D=3D "8" > > } -run > >=20 > > First, notice the lack of "boilerplate" code, getting a bog standard > > varnish instance is one line in the test-description and the VTest > > tool takes it from there. There is another module in VTest which > > does the same for HAproxy with the "haproxy" command. > >=20 > > Second, notice how VTest allows you to work at different levels of > > the HTTP protocol from one "instruction" to the next: The "s1" server > > receives an entire request with "rxreq" but in this particular test-cas= e > > it formulates the first response at the bytelevel, while the second > > response uses the "chunked" command which automatically generates the > > length. > >=20 > > As I read your long analysis, VTest is not a perfect fit for you, > > but I think it is more than half way there, and you are more than > > welcome to help us make it even better. Indeed, after reading the documentation, studying a few of the tests, and t= rying to draft a test by hand, vtest is close to the desired tool. I was al= so able to draft a simplistic test case pretty quickly: =09vtest "Test basic Caddy response" =09process p1 "caddy respond --listen 'localhost:8080' --body 'Hello'" -sta= rt =09delay 5 =09client c1 -connect "localhost:8080" { =09=09txreq -req GET -proto HTTP/1.1 -url / =09=09rxresp =09=09expect resp.proto =3D=3D HTTP/1.1 =09=09expect resp.status =3D=3D 200 =09 expect resp.body =3D=3D "Hello" =09} -run =09process p1 -stop =09process p1 -wait =20 The learning curve doesn't seem quite steep. The ability to pick the level = of HTTP protocol is quite neat. That makes it more apt than relying on curl= as it doesn't attempt to interpret the HTTP components rather merely struc= tures them and to be interpreted by the user's script. I noticed HTTP/2 req= uests aren't as approachable as HTTP/1. The documentation seems to imply th= e user has to manage the streams directly. Is that the case? It's also isn'= t clear how to make HTTPS requests. Another apparent gap is the inability t= o carry one value from one response of the server into the next. For instan= ce, to save the Etag value sent by the server from the initial request to b= e used with If-None-Match in subsequent requests. What am I missing? > I generally agree with the points made here. Is it possible to do better? > Certainly, just like with any tool. But it's already fairly complete and > it does something that few tools achieve right now: synchronizing both > sides, which is extremely convenient when testing reverse proxies. You > can for example have an http/1 server and verify that the output h2 > encoding out of your gateway matches what you expect. We (haproxy) do > use it routinely for various reasons ranging from regression testing to > spec test compliance (mostly detect breakage before our users), and since > it's fast (validates ~3600 expect rules from 200 tests in 8 seconds on my > laptop), we run it locally on every push and in the CI on ~20 platforms > for every push. >=20 > Another tool you might be interested in having a look at is h2spec. It's > h2-centric and tries to trigger edge cases from the implementation then > verifies if the behavior matches the spec. It requires a server however, > but that's no big deal. It's very close to the spec so it can be seen as > a catalogue of the checks to run on the traffic. I like vtest, and it seems to be a better tool for the goal presented in my= initial email than hurl. h2spec may also be helpful, but that'd depend on = how the test suite is orchestrated. However, I feel like my intent is misun= derstood. The intent isn't to just build a test suite for Caddy and call it= a day. The work on Caddy was the spark for the proposal, but not the end g= oal. I'm proposing building a test suite blessed by the HTTP WG for all rev= erse-proxies, web servers, and HTTP-speaking entities to check their confor= mance to HTTP.=20 All the best, Mohammed