Re: Declarative HTTP Spec Test Suite
Poul-Henning Kamp <phk@phk.freebsd.dk> Tue, 28 May 2024 08:27 UTC
Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=ietf.org@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7EA21C1516EA for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Tue, 28 May 2024 01:27:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.751
X-Spam-Level:
X-Spam-Status: No, score=-2.751 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, MAILING_LIST_MULTI=-1, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=w3.org header.b="P11djQoQ"; dkim=pass (2048-bit key) header.d=w3.org header.b="jixkio5+"
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qVtKrjwwAWwH for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Tue, 28 May 2024 01:27:25 -0700 (PDT)
Received: from mab.w3.org (mab.w3.org [IPv6:2600:1f18:7d7a:2700:d091:4b25:8566:8113]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E2168C1519B3 for <httpbisa-archive-bis2Juki@ietf.org>; Tue, 28 May 2024 01:27:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Subject:Date:Content-Type:MIME-Version:References:From:In-reply-to:cc :To:Message-Id:Reply-To; bh=NUuXsFKa6Gel3JLyW9orllJZugz+xOzECWm3q/TCnqY=; b=P 11djQoQzvIIldf7Zy8OYbbAEM5q8InJxKt/J7jl35GTF6Z7bCwwvDjm5awjIup2TX3Y6reqg0fJAO 6O6/VonMOLib6d4E/Hy2GUNPlDrAfIgy53sG6697p9TAH6RodF/uH/ThkmA5PINv7hJmk7rCyDc1W NorN3J2DWpvUeCYRlyXysYLqe40pFXW2lPyOf1nol6dJAy3gS3+YLZ82cHamaGP+Ho/6W8ekp1bTl Iam6XjcesEEvG1ZHINrR349HXCI0fV8JYjT0mHBy6NtFtgcmjjCnEmnGe4S62Yxz60mXRxELScod7 RLaFjzMkDhrYqM2hrMB8XhBwTe01yOhzA==;
Received: from lists by mab.w3.org with local (Exim 4.96) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1sBsA8-00EMTP-0k for ietf-http-wg-dist@listhub.w3.org; Tue, 28 May 2024 08:26:28 +0000
Resent-Date: Tue, 28 May 2024 08:26:28 +0000
Resent-Message-Id: <E1sBsA8-00EMTP-0k@mab.w3.org>
Received: from ip-10-0-0-144.ec2.internal ([10.0.0.144] helo=pan.w3.org) by mab.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <phk@critter.freebsd.dk>) id 1sBsA5-00EMSU-2y for ietf-http-wg@listhub.w3.internal; Tue, 28 May 2024 08:26:25 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Date:Content-Type:MIME-Version:References:From:In-reply-to:Subject:cc :To:Message-Id:Reply-To; bh=NUuXsFKa6Gel3JLyW9orllJZugz+xOzECWm3q/TCnqY=; t=1716884785; x=1717748785; b=jixkio5+ahJ8cruI2fu5ecig06wy91O5WCX/qHv0gRYRzjg pQ3vUjWz3MohuEdz429Ab5jB+rd1x9JYxW1QYn9dtlxcNOW5udIY53ASK9+CerqPsNINEqt1FSsf0 AvSh7JIX7zvCJjHcAsmLc87JyoapkS9AoxOcikVAhtUec4k9mxLeloQwWObHFbSWT9KuRJ8VDr1xZ YxAf4yNIrxOxqSnUN647SSkrcQwZHSWBVoL6hV3Gni+wUxrY1SigKYCEXGpqqYycrfIbh1eYQ3Bp1 ZlMnaAsF2RfNi+reEo6k3cGFqVGl4PrWeCfZLarjFbvpB7ULQjSGaFktYKlfPwXw==;
Received-SPF: pass (pan.w3.org: domain of critter.freebsd.dk designates 130.225.244.222 as permitted sender) client-ip=130.225.244.222; envelope-from=phk@critter.freebsd.dk; helo=phk.freebsd.dk;
Received: from phk.freebsd.dk ([130.225.244.222]) by pan.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <phk@critter.freebsd.dk>) id 1sBsA4-008j7d-2S for ietf-http-wg@w3.org; Tue, 28 May 2024 08:26:25 +0000
Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by phk.freebsd.dk (Postfix) with ESMTPS id 85728892AA; Tue, 28 May 2024 08:26:19 +0000 (UTC)
Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.18.1/8.16.1) with ESMTPS id 44S8QJ7J053850 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 28 May 2024 08:26:19 GMT (envelope-from phk@critter.freebsd.dk)
Received: (from phk@localhost) by critter.freebsd.dk (8.18.1/8.16.1/Submit) id 44S8QHwK053849; Tue, 28 May 2024 08:26:17 GMT (envelope-from phk)
Message-Id: <202405280826.44S8QHwK053849@critter.freebsd.dk>
To: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>
cc: Willy Tarreau <w@1wt.eu>, "ietf-http-wg@w3.org" <ietf-http-wg@w3.org>
In-reply-to: <7c8w3fIP0cy2UKgXbcibDe3LsYUtM0se9Ew0QixqV7809W0PhMkNJUdH4cw4DDDaurLc1osZ_M-KTMjjrifUayN-3ElVCI5EWFyGwBZNcoY=@caffeinatedwonders.com>
From: Poul-Henning Kamp <phk@phk.freebsd.dk>
References: <tPxvQGS7_bxKqrFish00m9LtTT_JQfE0RU1bthEcxYNTZR_uzbkEWKNOTg9c5azoZSYuDWt9pLCiS7Kd4nIWK8DPN1sOtcgbDGx4L7BeeU4=@caffeinatedwonders.com> <202405271304.44RD4iQj027668@critter.freebsd.dk> <ZlSJQawTqEzLYVvg@1wt.eu> <7c8w3fIP0cy2UKgXbcibDe3LsYUtM0se9Ew0QixqV7809W0PhMkNJUdH4cw4DDDaurLc1osZ_M-KTMjjrifUayN-3ElVCI5EWFyGwBZNcoY=@caffeinatedwonders.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-ID: <53847.1716884777.1@critter.freebsd.dk>
Content-Transfer-Encoding: quoted-printable
Date: Tue, 28 May 2024 08:26:17 +0000
X-W3C-Hub-Spam-Status: No, score=-4.9
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DMARC_MISSING=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, W3C_AA=-1, W3C_IRA=-1, W3C_WL=-1
X-W3C-Scan-Sig: pan.w3.org 1sBsA4-008j7d-2S 01e3b581b649bde371837709e52c2f4a
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Declarative HTTP Spec Test Suite
Archived-At: <https://www.w3.org/mid/202405280826.44S8QHwK053849@critter.freebsd.dk>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/51976
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/email/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>
-------- Mohammed Al Sahaf writes: > > > As I read your long analysis, VTest is not a perfect fit for you, > > > but I think it is more than half way there, and you are more than > > > welcome to help us make it even better. > > Indeed, after reading the documentation, studying a few of the tests, and t > rying to draft a test by hand, vtest is close to the desired tool. I was al > so able to draft a simplistic test case pretty quickly: > > vtest "Test basic Caddy response" > > process p1 "caddy respond --listen 'localhost:8080' --body 'Hello'" -start > delay 5 > > client c1 -connect "localhost:8080" { > txreq -req GET -proto HTTP/1.1 -url / > rxresp > expect resp.proto == HTTP/1.1 > expect resp.status == 200 > expect resp.body == "Hello" > } -run > > process p1 -stop > process p1 -wait Not bad :-) Your example highlights one crucial detail I forgot to mention: If you want to run multiple tests in parallel or have tests run in parallel by different users on the same machine, something needs to manage/assign unique TCP/UDP port numbers and temporary directories, so the different test-runs do not interfere with each other. For Varnish we added a debug CLI command so VTest can start varnishd with "-a localhost:0" and then ask varnishd what port number it actually got. VTest support in HAproxy does something similar. Likewise in the example I showed the "-vcl+backend {}" creates a VCL program from varnishd, where all the "server s%d {}" instances are already defined as backends, with whatever randomly assigned IP# and port numbers they got. > I noticed HTTP/2 requests aren't as approachable as HTTP/1. You can generalize that statement by removing the word "requests" :-) > The documentation seems to imply the user has to manage the streams > directly. Is that the case? Again: We wanted VTest to be able to specify the test as generally or specifically as required. To be honest, I cannot remember if there is any automation that helps with that, if not, it would be easy to add. My own experience is that you want to manually assign the stream numbers, so that you can correlate things further down the road. > It's also isn't clear how to make HTTPS requests. Right now VTest does not support SSL/TLS natively, but that is clearly something we should add. There are two major issues, the first being where does the certs come from? A) VTest creates self-signed certs as needed on the fly. That would probably be intolerably slow. B) VTest creates a cache of self-signed certs, either with Y2K38 like expiry or with logic to periodically refresh them. C) The user of VTest have to provide the certs to be used, possibly from a helper script which does B) The second major issue is: Which SSL/TLS implementation, and here there are only two options: A) Pick one, and forget about using VTest to probe the edges of the SSL/TLS implementations. B) Write one, so that VTest can test the edges of SSL and TLS traffic I dont know what the situation is with respect with tools to test SSL/TLS protocol implementations. If they already have such, B) would be surplus to requirements, if not, somebody should write it. > Another apparent gap is the inability to carry one value from one response > of the server into the next. For instance, to save the Etag value sent by > the server from the initial request to be used with If-None-Match in subsequent > requests. What am I missing? Some code ? :-) Right now macros are expanded when the {...} argument is parsed, so it is not possible to change or define the value of a macro while it runs. It would not be hard to provide some kind of "dictionary" where things can be stashed and picked up again later. I think we already talked about something like that, but I guess nobody needed it enough to implement it. > However, I feel like my intent is misunderstood. The intent isn't to just > build a test suite for Caddy and call it a day. The work on Caddy was the > spark for the proposal, but not the end goal. I'm proposing building a test > suite blessed by the HTTP WG for all reverse-proxies, web servers, and > HTTP-speaking entities to check their conformance to HTTP. I got that first time around, but did not comment on it, because I did not want to discourage you. Let me do so now. Compliance Tests are a neat idea: If you pass this pile of test-cases you get to call your product "FOOBAR Compliant", and put the golden "FOOBAR says: Will Just Workâ˘" sticker on it. Where the subject matter is one-way, for instance Nicolas Seriot's JSON beastiarium (https://github.com/nst/JSONTestSuite) the success of compliance testing is entirely up to the malicious creativity of the person(s) writing the tests, and Nicolas did really well, breaking (almost?) all JSON parsers in existence. But where the subject matter is two way, like communication protocols, I have never seen compliance testing deliver anywhere near promise. The first problem is that even when everybody agree 100% on what the protocol definition says, most protocols explode combinatorically in very few exchanges. A HTTP server can return 3xx to almost any request, that is not a compliance failure, it can send "Connection: Close" or Keepalive, that is not a compliance failure either. Imagine this: compliance-client: HEAD / server-under-test: 301 to /justkidding compliance-client: HEAD /justkidding server-under-test: 301 to / How many times is a compliant server allowed to yank a client around like that, before it actually answers ? (The above is surprisingly common patterns of identification) The second problem is timing. HTTP doesn't have rigidly specified timeouts, but there is still stuff like "If-Modified-Since", which the server is allowed to just ignore whenever it feels like it. But the biggest problem of all, is that people simply do not agree what standards say or mean. I served time in meetings with angry IBM people, where we could not agree what "have received" meant. (Their mainframes crashed when my code answered too fast because they (tacitly) assumed modems and a telco would always be involved, where I used a 5 meter null-modem cable.) That is not to say that such testing is never valuable, it is, and IETF have often arranged "bake-offs", where different implementations would meet, and after first getting things to work, they would try to break each other's implementations with hostile traffic. So as I said: I do not want to discourage you, and as long as you dont expect to hand out "HTTP-compliant" gold star stickers, I dont see why it cannot or should not be done. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.
- Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite Poul-Henning Kamp
- Re: Declarative HTTP Spec Test Suite Willy Tarreau
- Re: Declarative HTTP Spec Test Suite Daniel Stenberg
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite Daniel Stenberg
- Re: Declarative HTTP Spec Test Suite Poul-Henning Kamp
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite Willy Tarreau
- Re: Declarative HTTP Spec Test Suite Sawood Alam
- Re: Declarative HTTP Spec Test Suite Lucas Pardue
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite David Benjamin
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf