Re: Declarative HTTP Spec Test Suite

[Poul-Henning Kamp <[e-mail]>]

> This is a proposal that is triggered by some of my involvement with the Caddy
> web server project. We (Caddy team) have been working towards developing a
> declarative test suite for the Caddy server.
> […]
> Prior Art:
> […]
> Indeed some of the listed projects have not been updated for a while.

Mohammed,

I may be misunderstanding you, but using repository activity as a
screen will make you miss stuff which Just Works™.

I will argue that VTest (https://github.com/vtest/VTest) is one
such:  It chugs away, day in and day out, in both the Varnish Cache
and HAproxy projects.

A specific reason for repository inactivity, which you might not
have noticed, is that VTest is just the test program code and the
test-cases do not live in the VTest respository, but in the "parent"
projects which use VTest:

	https://github.com/haproxy/haproxy/tree/master/reg-tests

and 

	https://github.com/varnishcache/varnish-cache/tree/master/bin/varnishtest/tests

Full VTest documentation is under "Varnishtest" in the varnish project:

	http://varnish-cache.org/docs/trunk/reference/index.html

I started writing VTest a dozen years ago for many of the same
reasons you touch in your analysis, but one of my other reasons
was that I wanted a good language to write test-cases in.

Here is a basic test from Varnish Cache, checking that chunked
encoding works at all:

	varnishtest "Check chunked encoding from backend works"
	
	server s1 {
        	rxreq
        	expect req.url == "/bar"
        	send "HTTP/1.1 200 OK\r\n"
        	send "Transfer-encoding: chunked\r\n"
        	send "\r\n"
        	send "00000004\r\n1234\r\n"
        	send "00000000\r\n"
        	send "\r\n"
	
        	rxreq
        	expect req.url == "/foo"
        	send "HTTP/1.1 200 OK\r\n"
        	send "Transfer-encoding: chunked\r\n"
        	send "\r\n"
        	send "00000004\r\n1234\r\n"
        	chunked "1234"
        	chunked ""
	} -start
	
	varnish v1 -vcl+backend {} -start
	
	client c1 {
        	txreq -url "/bar"
        	rxresp
        	expect resp.status == 200
        	expect resp.bodylen == "4"
        	txreq -url "/foo"
        	rxresp
        	expect resp.status == 200
        	expect resp.bodylen == "8"
	} -run

First, notice the lack of "boilerplate" code, getting a bog standard
varnish instance is one line in the test-description and the VTest
tool takes it from there.  There is another module in VTest which
does the same for HAproxy with the "haproxy" command.

Second, notice how VTest allows you to work at different levels of
the HTTP protocol from one "instruction" to the next: The "s1" server
receives an entire request with "rxreq" but in this particular test-case
it formulates the first response at the bytelevel, while the second
response uses the "chunked" command which automatically generates the
length.

As I read your long analysis, VTest is not a perfect fit for you,
but I think it is more than half way there, and you are more than
welcome to help us make it even better.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[e-mail]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.