Re: Declarative HTTP Spec Test Suite

Poul-Henning Kamp <phk@phk.freebsd.dk> Tue, 28 May 2024 08:27 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=ietf.org@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7EA21C1516EA for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Tue, 28 May 2024 01:27:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.751
X-Spam-Level:
X-Spam-Status: No, score=-2.751 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, MAILING_LIST_MULTI=-1, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=w3.org header.b="P11djQoQ"; dkim=pass (2048-bit key) header.d=w3.org header.b="jixkio5+"
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qVtKrjwwAWwH for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Tue, 28 May 2024 01:27:25 -0700 (PDT)
Received: from mab.w3.org (mab.w3.org [IPv6:2600:1f18:7d7a:2700:d091:4b25:8566:8113]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E2168C1519B3 for <httpbisa-archive-bis2Juki@ietf.org>; Tue, 28 May 2024 01:27:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Subject:Date:Content-Type:MIME-Version:References:From:In-reply-to:cc :To:Message-Id:Reply-To; bh=NUuXsFKa6Gel3JLyW9orllJZugz+xOzECWm3q/TCnqY=; b=P 11djQoQzvIIldf7Zy8OYbbAEM5q8InJxKt/J7jl35GTF6Z7bCwwvDjm5awjIup2TX3Y6reqg0fJAO 6O6/VonMOLib6d4E/Hy2GUNPlDrAfIgy53sG6697p9TAH6RodF/uH/ThkmA5PINv7hJmk7rCyDc1W NorN3J2DWpvUeCYRlyXysYLqe40pFXW2lPyOf1nol6dJAy3gS3+YLZ82cHamaGP+Ho/6W8ekp1bTl Iam6XjcesEEvG1ZHINrR349HXCI0fV8JYjT0mHBy6NtFtgcmjjCnEmnGe4S62Yxz60mXRxELScod7 RLaFjzMkDhrYqM2hrMB8XhBwTe01yOhzA==;
Received: from lists by mab.w3.org with local (Exim 4.96) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1sBsA8-00EMTP-0k for ietf-http-wg-dist@listhub.w3.org; Tue, 28 May 2024 08:26:28 +0000
Resent-Date: Tue, 28 May 2024 08:26:28 +0000
Resent-Message-Id: <E1sBsA8-00EMTP-0k@mab.w3.org>
Received: from ip-10-0-0-144.ec2.internal ([10.0.0.144] helo=pan.w3.org) by mab.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <phk@critter.freebsd.dk>) id 1sBsA5-00EMSU-2y for ietf-http-wg@listhub.w3.internal; Tue, 28 May 2024 08:26:25 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Date:Content-Type:MIME-Version:References:From:In-reply-to:Subject:cc :To:Message-Id:Reply-To; bh=NUuXsFKa6Gel3JLyW9orllJZugz+xOzECWm3q/TCnqY=; t=1716884785; x=1717748785; b=jixkio5+ahJ8cruI2fu5ecig06wy91O5WCX/qHv0gRYRzjg pQ3vUjWz3MohuEdz429Ab5jB+rd1x9JYxW1QYn9dtlxcNOW5udIY53ASK9+CerqPsNINEqt1FSsf0 AvSh7JIX7zvCJjHcAsmLc87JyoapkS9AoxOcikVAhtUec4k9mxLeloQwWObHFbSWT9KuRJ8VDr1xZ YxAf4yNIrxOxqSnUN647SSkrcQwZHSWBVoL6hV3Gni+wUxrY1SigKYCEXGpqqYycrfIbh1eYQ3Bp1 ZlMnaAsF2RfNi+reEo6k3cGFqVGl4PrWeCfZLarjFbvpB7ULQjSGaFktYKlfPwXw==;
Received-SPF: pass (pan.w3.org: domain of critter.freebsd.dk designates 130.225.244.222 as permitted sender) client-ip=130.225.244.222; envelope-from=phk@critter.freebsd.dk; helo=phk.freebsd.dk;
Received: from phk.freebsd.dk ([130.225.244.222]) by pan.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <phk@critter.freebsd.dk>) id 1sBsA4-008j7d-2S for ietf-http-wg@w3.org; Tue, 28 May 2024 08:26:25 +0000
Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by phk.freebsd.dk (Postfix) with ESMTPS id 85728892AA; Tue, 28 May 2024 08:26:19 +0000 (UTC)
Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.18.1/8.16.1) with ESMTPS id 44S8QJ7J053850 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 28 May 2024 08:26:19 GMT (envelope-from phk@critter.freebsd.dk)
Received: (from phk@localhost) by critter.freebsd.dk (8.18.1/8.16.1/Submit) id 44S8QHwK053849; Tue, 28 May 2024 08:26:17 GMT (envelope-from phk)
Message-Id: <202405280826.44S8QHwK053849@critter.freebsd.dk>
To: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>
cc: Willy Tarreau <w@1wt.eu>, "ietf-http-wg@w3.org" <ietf-http-wg@w3.org>
In-reply-to: <7c8w3fIP0cy2UKgXbcibDe3LsYUtM0se9Ew0QixqV7809W0PhMkNJUdH4cw4DDDaurLc1osZ_M-KTMjjrifUayN-3ElVCI5EWFyGwBZNcoY=@caffeinatedwonders.com>
From: Poul-Henning Kamp <phk@phk.freebsd.dk>
References: <tPxvQGS7_bxKqrFish00m9LtTT_JQfE0RU1bthEcxYNTZR_uzbkEWKNOTg9c5azoZSYuDWt9pLCiS7Kd4nIWK8DPN1sOtcgbDGx4L7BeeU4=@caffeinatedwonders.com> <202405271304.44RD4iQj027668@critter.freebsd.dk> <ZlSJQawTqEzLYVvg@1wt.eu> <7c8w3fIP0cy2UKgXbcibDe3LsYUtM0se9Ew0QixqV7809W0PhMkNJUdH4cw4DDDaurLc1osZ_M-KTMjjrifUayN-3ElVCI5EWFyGwBZNcoY=@caffeinatedwonders.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-ID: <53847.1716884777.1@critter.freebsd.dk>
Content-Transfer-Encoding: quoted-printable
Date: Tue, 28 May 2024 08:26:17 +0000
X-W3C-Hub-Spam-Status: No, score=-4.9
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DMARC_MISSING=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, W3C_AA=-1, W3C_IRA=-1, W3C_WL=-1
X-W3C-Scan-Sig: pan.w3.org 1sBsA4-008j7d-2S 01e3b581b649bde371837709e52c2f4a
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Declarative HTTP Spec Test Suite
Archived-At: <https://www.w3.org/mid/202405280826.44S8QHwK053849@critter.freebsd.dk>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/51976
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/email/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

--------
Mohammed Al Sahaf writes:

> > > As I read your long analysis, VTest is not a perfect fit for you,
> > > but I think it is more than half way there, and you are more than
> > > welcome to help us make it even better.
>
> Indeed, after reading the documentation, studying a few of the tests, and t
> rying to draft a test by hand, vtest is close to the desired tool. I was al
> so able to draft a simplistic test case pretty quickly:
>
> 	vtest "Test basic Caddy response"
>
> 	process p1 "caddy respond --listen 'localhost:8080' --body 'Hello'" -start
> 	delay 5
>
> 	client c1 -connect "localhost:8080" {
> 		txreq -req GET -proto HTTP/1.1 -url /
> 		rxresp
> 		expect resp.proto == HTTP/1.1
> 		expect resp.status == 200
> 	    expect resp.body == "Hello"
> 	} -run
>
> 	process p1 -stop
> 	process p1 -wait

Not bad :-)

Your example highlights one crucial detail I forgot to mention:

If you want to run multiple tests in parallel or have tests run in
parallel by different users on the same machine, something needs
to manage/assign unique TCP/UDP port numbers and temporary directories,
so the different test-runs do not interfere with each other.

For Varnish we added a debug CLI command so VTest can start varnishd
with "-a localhost:0" and then ask varnishd what port number it
actually got.  VTest support in HAproxy does something similar.

Likewise in the example I showed the "-vcl+backend {}" creates a
VCL program from varnishd, where all the "server s%d {}" instances
are already defined as backends, with whatever randomly assigned
IP# and port numbers they got.

> I noticed HTTP/2 requests aren't as approachable as HTTP/1.

You can generalize that statement by removing the word "requests" :-)

> The documentation seems to imply the user has to manage the streams
> directly. Is that the case?

Again:  We wanted VTest to be able to specify the test as generally
or specifically as required.

To be honest, I cannot remember if there is any automation that helps
with that, if not, it would be easy to add.

My own experience is that you want to manually assign the stream
numbers, so that you can correlate things further down the road.

> It's also isn't clear how to make HTTPS requests.

Right now VTest does not support SSL/TLS natively, but that is clearly
something we should add.

There are two major issues, the first being where does the certs come from?

A) VTest creates self-signed certs as needed on the fly.  That would
   probably be intolerably slow.

B) VTest creates a cache of self-signed certs, either with Y2K38
   like expiry or with logic to periodically refresh them.

C) The user of VTest have to provide the certs to be used, possibly
   from a helper script which does B)

The second major issue is:  Which SSL/TLS implementation, and here
there are only two options:

A) Pick one, and forget about using VTest to probe the edges of
   the SSL/TLS implementations.

B) Write one, so that VTest can test the edges of SSL and TLS traffic

I dont know what the situation is with respect with tools to test
SSL/TLS protocol implementations.  If they already have such, B)
would be surplus to requirements, if not, somebody should write it.

> Another apparent gap is the inability to carry one value from one response
> of the server into the next. For instance, to save the Etag value sent by
> the server from the initial request to be used with If-None-Match in subsequent
> requests. What am I missing?

Some code ? :-)

Right now macros are expanded when the {...} argument is parsed, so it is not
possible to change or define the value of a macro while it runs.

It would not be hard to provide some kind of "dictionary" where
things can be stashed and picked up again later.

I think we already talked about something like that, but I guess
nobody needed it enough to implement it.

> However, I feel like my intent is misunderstood. The intent isn't to just
> build a test suite for Caddy and call it a day. The work on Caddy was the
> spark for the proposal, but not the end goal. I'm proposing building a test
> suite blessed by the HTTP WG for all reverse-proxies, web servers, and
> HTTP-speaking entities to check their conformance to HTTP. 

I got that first time around, but did not comment on it, because I
did not want to discourage you.

Let me do so now.

Compliance Tests are a neat idea:  If you pass this pile of test-cases
you get to call your product "FOOBAR Compliant", and put the golden
"FOOBAR says: Will Just Work™" sticker on it.

Where the subject matter is one-way, for instance Nicolas Seriot's
JSON beastiarium (https://github.com/nst/JSONTestSuite) the success
of compliance testing is entirely up to the malicious creativity
of the person(s) writing the tests, and Nicolas did really well,
breaking (almost?) all JSON parsers in existence.

But where the subject matter is two way, like communication protocols,
I have never seen compliance testing deliver anywhere near promise.

The first problem is that even when everybody agree 100% on what
the protocol definition says, most protocols explode combinatorically
in very few exchanges.

A HTTP server can return 3xx to almost any request, that is not
a compliance failure, it can send "Connection: Close" or Keepalive,
that is not a compliance failure either.

Imagine this:

	compliance-client:
		HEAD /
	server-under-test:
		301 to /justkidding
	compliance-client:
		HEAD /justkidding
	server-under-test:
		301 to /

How many times is a compliant server allowed to yank a client
around like that, before it actually answers ?

(The above is surprisingly common patterns of identification)

The second problem is timing.  HTTP doesn't have rigidly specified
timeouts, but there is still stuff like "If-Modified-Since", which
the server is allowed to just ignore whenever it feels like it.

But the biggest problem of all, is that people simply do not agree
what standards say or mean.

I served time in meetings with angry IBM people, where we could not
agree what "have received" meant.  (Their mainframes crashed when
my code answered too fast because they (tacitly) assumed modems and
a telco would always be involved, where I used a 5 meter null-modem
cable.)

That is not to say that such testing is never valuable, it is, and
IETF have often arranged "bake-offs", where different implementations
would meet, and after first getting things to work, they would try
to break each other's implementations with hostile traffic.

So as I said: I do not want to discourage you, and as long as you dont
expect to hand out "HTTP-compliant" gold star stickers, I dont see
why it cannot or should not be done.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.