Re: Declarative HTTP Spec Test Suite

[Mohammed Al Sahaf <[e-mail]>]

On Monday, May 27th, 2024 at 4:23 PM, Willy Tarreau <[e-mail]> wrote:

> On Mon, May 27, 2024 at 01:04:44PM +0000, Poul-Henning Kamp wrote:
> 
> > > This is a proposal that is triggered by some of my involvement with the Caddy
> > > web server project. We (Caddy team) have been working towards developing a
> > > declarative test suite for the Caddy server.
> > > [...]
> > > Prior Art:
> > > [...]
> > > Indeed some of the listed projects have not been updated for a while.
> > 
> > Mohammed,
> > 
> > I may be misunderstanding you, but using repository activity as a
> > screen will make you miss stuff which Just Works(TM).

Fair enough! I stand corrected.

> > I will argue that VTest (https://github.com/vtest/VTest) is one
> > such: It chugs away, day in and day out, in both the Varnish Cache
> > and HAproxy projects.
> > 
> > A specific reason for repository inactivity, which you might not
> > have noticed, is that VTest is just the test program code and the
> > test-cases do not live in the VTest respository, but in the "parent"
> > projects which use VTest:
> > 
> > https://github.com/haproxy/haproxy/tree/master/reg-tests
> > 
> > and
> > 
> > https://github.com/varnishcache/varnish-cache/tree/master/bin/varnishtest/tests
> > 
> > Full VTest documentation is under "Varnishtest" in the varnish project:
> > 
> > http://varnish-cache.org/docs/trunk/reference/index.html
> > 
> > I started writing VTest a dozen years ago for many of the same
> > reasons you touch in your analysis, but one of my other reasons
> > was that I wanted a good language to write test-cases in.
> > 
> > Here is a basic test from Varnish Cache, checking that chunked
> > encoding works at all:
> > 
> > varnishtest "Check chunked encoding from backend works"
> > 
> > server s1 {
> > rxreq
> > expect req.url == "/bar"
> > send "HTTP/1.1 200 OK\r\n"
> > send "Transfer-encoding: chunked\r\n"
> > send "\r\n"
> > send "00000004\r\n1234\r\n"
> > send "00000000\r\n"
> > send "\r\n"
> > 
> > rxreq
> > expect req.url == "/foo"
> > send "HTTP/1.1 200 OK\r\n"
> > send "Transfer-encoding: chunked\r\n"
> > send "\r\n"
> > send "00000004\r\n1234\r\n"
> > chunked "1234"
> > chunked ""
> > } -start
> > 
> > varnish v1 -vcl+backend {} -start
> > 
> > client c1 {
> > txreq -url "/bar"
> > rxresp
> > expect resp.status == 200
> > expect resp.bodylen == "4"
> > txreq -url "/foo"
> > rxresp
> > expect resp.status == 200
> > expect resp.bodylen == "8"
> > } -run
> > 
> > First, notice the lack of "boilerplate" code, getting a bog standard
> > varnish instance is one line in the test-description and the VTest
> > tool takes it from there. There is another module in VTest which
> > does the same for HAproxy with the "haproxy" command.
> > 
> > Second, notice how VTest allows you to work at different levels of
> > the HTTP protocol from one "instruction" to the next: The "s1" server
> > receives an entire request with "rxreq" but in this particular test-case
> > it formulates the first response at the bytelevel, while the second
> > response uses the "chunked" command which automatically generates the
> > length.
> > 
> > As I read your long analysis, VTest is not a perfect fit for you,
> > but I think it is more than half way there, and you are more than
> > welcome to help us make it even better.

Indeed, after reading the documentation, studying a few of the tests, and trying to draft a test by hand, vtest is close to the desired tool. I was also able to draft a simplistic test case pretty quickly:

	vtest "Test basic Caddy response"

	process p1 "caddy respond --listen 'localhost:8080' --body 'Hello'" -start
	delay 5

	client c1 -connect "localhost:8080" {
		txreq -req GET -proto HTTP/1.1 -url /
		rxresp
		expect resp.proto == HTTP/1.1
		expect resp.status == 200
	    expect resp.body == "Hello"
	} -run

	process p1 -stop
	process p1 -wait
 
The learning curve doesn't seem quite steep. The ability to pick the level of HTTP protocol is quite neat. That makes it more apt than relying on curl as it doesn't attempt to interpret the HTTP components rather merely structures them and to be interpreted by the user's script. I noticed HTTP/2 requests aren't as approachable as HTTP/1. The documentation seems to imply the user has to manage the streams directly. Is that the case? It's also isn't clear how to make HTTPS requests. Another apparent gap is the inability to carry one value from one response of the server into the next. For instance, to save the Etag value sent by the server from the initial request to be used with If-None-Match in subsequent requests. What am I missing?

> I generally agree with the points made here. Is it possible to do better?
> Certainly, just like with any tool. But it's already fairly complete and
> it does something that few tools achieve right now: synchronizing both
> sides, which is extremely convenient when testing reverse proxies. You
> can for example have an http/1 server and verify that the output h2
> encoding out of your gateway matches what you expect. We (haproxy) do
> use it routinely for various reasons ranging from regression testing to
> spec test compliance (mostly detect breakage before our users), and since
> it's fast (validates ~3600 expect rules from 200 tests in 8 seconds on my
> laptop), we run it locally on every push and in the CI on ~20 platforms
> for every push.
> 
> Another tool you might be interested in having a look at is h2spec. It's
> h2-centric and tries to trigger edge cases from the implementation then
> verifies if the behavior matches the spec. It requires a server however,
> but that's no big deal. It's very close to the spec so it can be seen as
> a catalogue of the checks to run on the traffic.

I like vtest, and it seems to be a better tool for the goal presented in my initial email than hurl. h2spec may also be helpful, but that'd depend on how the test suite is orchestrated. However, I feel like my intent is misunderstood. The intent isn't to just build a test suite for Caddy and call it a day. The work on Caddy was the spark for the proposal, but not the end goal. I'm proposing building a test suite blessed by the HTTP WG for all reverse-proxies, web servers, and HTTP-speaking entities to check their conformance to HTTP. 


All the best,
Mohammed