Re: Declarative HTTP Spec Test Suite
Mohammed Al Sahaf <mohammed@caffeinatedwonders.com> Sat, 01 June 2024 01:29 UTC
Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=ietf.org@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C47EEC169426 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Fri, 31 May 2024 18:29:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.848
X-Spam-Level:
X-Spam-Status: No, score=-2.848 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, MAILING_LIST_MULTI=-1, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=w3.org header.b="ABBZ92GB"; dkim=pass (2048-bit key) header.d=w3.org header.b="UqNGE4Hx"; dkim=pass (2048-bit key) header.d=caffeinatedwonders.com header.b="ymPR6o+t"
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c5pCCCsXfI_9 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Fri, 31 May 2024 18:29:25 -0700 (PDT)
Received: from mab.w3.org (mab.w3.org [IPv6:2600:1f18:7d7a:2700:d091:4b25:8566:8113]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A8A45C1519B6 for <httpbisa-archive-bis2Juki@ietf.org>; Fri, 31 May 2024 18:29:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Subject:Content-Type:MIME-Version:References:In-Reply-To:Message-ID: Cc:From:To:Date:Reply-To; bh=ZWyrRwR5fyC8bmbZnVks591fbdhX/bGYtcE99bPaGtk=; b= ABBZ92GB472TJqzKFJF3dG+sXmS+L6gZw6m4x5UUNv8c20emoJxbRgIxG4N3b6YDOlK9s6Im4bQFn OYKnxuTwZJ/SMPhdNfTncJtjvMambEndeGcebCCItArBMGvOj6diPBIdxRuqW9jgdx84QUyXE2L1i aweI+jXvESHWcNyQbuYd5srlUlUbssasgWLrjzv0rVFA4xFQSFhK3y6FnN8sjZMzIW2ZZyrFgzZjG 0hGGC6fFGvLFT1mml7i1FKWrAH0Sg/4DVg0VekWs61t/XzzICUD2OWnTdd1olRILPMKKMdKXI288t cBPYHADF6zanqyrhSOvw1pF6GeqGx7ZhYg==;
Received: from lists by mab.w3.org with local (Exim 4.96) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1sDDXo-006jh5-1a for ietf-http-wg-dist@listhub.w3.org; Sat, 01 Jun 2024 01:28:28 +0000
Resent-Date: Sat, 01 Jun 2024 01:28:28 +0000
Resent-Message-Id: <E1sDDXo-006jh5-1a@mab.w3.org>
Received: from ip-10-0-0-144.ec2.internal ([10.0.0.144] helo=pan.w3.org) by mab.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <mohammed@caffeinatedwonders.com>) id 1sDDXl-006jg6-1L for ietf-http-wg@listhub.w3.internal; Sat, 01 Jun 2024 01:28:25 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Subject: Cc:From:To:Date:Reply-To; bh=ZWyrRwR5fyC8bmbZnVks591fbdhX/bGYtcE99bPaGtk=; t=1717205305; x=1718069305; b=UqNGE4HxKub3UrNhNOkECiS4kZZxbvQQa+OmWOQQJAYmcJq x5YFORP71xcdyPSwAg93audYtbkmJgAl48PdJMxOa9JdA0oSV6uuit1y73Z73qPgJBjXUSbeT+iAw /tOCeeu6xWjsa1FIn/xuhnE3g1i95QMtFj2X8mpBUuefmtnAzHJOhbI/1zbPRDwLCQ81w/2OZk4h5 YGhScebUHHJjTKhY7qzBolTzUGrLW+IXoDPjFDa+yBsA/B8vsBEFu/Zzj2NJCsvmsO2RL/HA4mkGi ubP52ipLVZtOgV0/QoEyus0fap54iJhQOao+OYb66/qO366CfpYpnXrk60Cvlbug==;
Received-SPF: pass (pan.w3.org: domain of caffeinatedwonders.com designates 185.70.43.17 as permitted sender) client-ip=185.70.43.17; envelope-from=mohammed@caffeinatedwonders.com; helo=mail-4317.proton.ch;
Received: from mail-4317.proton.ch ([185.70.43.17]) by pan.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <mohammed@caffeinatedwonders.com>) id 1sDDXj-00A7LD-0x for ietf-http-wg@w3.org; Sat, 01 Jun 2024 01:28:25 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=caffeinatedwonders.com; s=protonmail3; t=1717205298; x=1717464498; bh=ZWyrRwR5fyC8bmbZnVks591fbdhX/bGYtcE99bPaGtk=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=ymPR6o+tauSpoQOdQQswjNhYEuWYakCFbVRoKw2+dVAHDccWi/xumask/eAPmtPwo 18NLiB5TcGAJRIlJi1DjUbpGxsoXNLw+ALtz5p9sZYA/JvqYZLQs8OK+h+DxE5ehpj bhsPlLVmcZWJ6WoGSkxK0C9PNfgeavw3PwuOOCGlGnrK6eCEIly68KoNmITht9uCTo vpe6zUWbShRRylzHLxVqOihTm+CLc09CW6iPhujsaLbfdXPZ++hpXvQNyD98l+3ilP y7EQmFaXlDSbX0+IUBY6hJaiLQ9gvt5UsfRgnSeqn3R8KIrHiDD8NfC2uJEZrUx82E pVchmAVxE3KNQ==
Date: Sat, 01 Jun 2024 01:28:13 +0000
To: Poul-Henning Kamp <phk@phk.freebsd.dk>
From: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>
Cc: Willy Tarreau <w@1wt.eu>, "ietf-http-wg@w3.org" <ietf-http-wg@w3.org>
Message-ID: <wP7LW-8K9zDV3YqwOqu1DomsuizO7c1OunRkG2OWBPGbsh_cWyac5OMlz-1uHFRITA8F37VFD7L8mMcl4Soy7O9d19pbgtfcJ9jxbmZZwbc=@caffeinatedwonders.com>
In-Reply-To: <202405280826.44S8QHwK053849@critter.freebsd.dk>
References: <tPxvQGS7_bxKqrFish00m9LtTT_JQfE0RU1bthEcxYNTZR_uzbkEWKNOTg9c5azoZSYuDWt9pLCiS7Kd4nIWK8DPN1sOtcgbDGx4L7BeeU4=@caffeinatedwonders.com> <202405271304.44RD4iQj027668@critter.freebsd.dk> <ZlSJQawTqEzLYVvg@1wt.eu> <7c8w3fIP0cy2UKgXbcibDe3LsYUtM0se9Ew0QixqV7809W0PhMkNJUdH4cw4DDDaurLc1osZ_M-KTMjjrifUayN-3ElVCI5EWFyGwBZNcoY=@caffeinatedwonders.com> <202405280826.44S8QHwK053849@critter.freebsd.dk>
Feedback-ID: 74227028:user:proton
X-Pm-Message-ID: 20404350c835649e423ab8d2f373347e9e2fe4a1
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
X-W3C-Hub-DKIM-Status: validation passed: (address=mohammed@caffeinatedwonders.com domain=caffeinatedwonders.com), signature is good
X-W3C-Hub-Spam-Status: No, score=-4.1
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, DMARC_PASS=-0.001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_DBL_BLOCKED_OPENDNS=0.001, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: pan.w3.org 1sDDXj-00A7LD-0x 947c0e3f998a68020d6fbe239e1737eb
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Declarative HTTP Spec Test Suite
Archived-At: <https://www.w3.org/mid/wP7LW-8K9zDV3YqwOqu1DomsuizO7c1OunRkG2OWBPGbsh_cWyac5OMlz-1uHFRITA8F37VFD7L8mMcl4Soy7O9d19pbgtfcJ9jxbmZZwbc=@caffeinatedwonders.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/51982
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/email/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>
On Tuesday, May 28th, 2024 at 11:26 AM, Poul-Henning Kamp <phk@phk.freebsd.dk> wrote: > -------- > > It's also isn't clear how to make HTTPS requests. > > > Right now VTest does not support SSL/TLS natively, but that is clearly > something we should add. > > There are two major issues, the first being where does the certs come from? > > A) VTest creates self-signed certs as needed on the fly. That would > probably be intolerably slow. > > B) VTest creates a cache of self-signed certs, either with Y2K38 > like expiry or with logic to periodically refresh them. > > C) The user of VTest have to provide the certs to be used, possibly > from a helper script which does B) > > The second major issue is: Which SSL/TLS implementation, and here > there are only two options: > > A) Pick one, and forget about using VTest to probe the edges of > the SSL/TLS implementations. > > B) Write one, so that VTest can test the edges of SSL and TLS traffic > > I dont know what the situation is with respect with tools to test > SSL/TLS protocol implementations. If they already have such, B) > would be surplus to requirements, if not, somebody should write it. The security team of Go have developed what they call BoGo test suite for their crypt/tls implementation to be run against BoringSSL test suite (https://go-review.googlesource.com/c/go/+/486495) The test suite (https://github.com/golang/go/tree/master/src/crypto/tls/testdata) is a set of plaintext files describing the (TLS) bytes flowing between the client and the server for a particular cipher suite. I don't remember (or don't know) the details of how they get their hands on the bytes recordings, but I learned about this from the GoTime interview with Filippo Valsorda, Roland Shoemaker, and Nicola Murino (Episode: https://changelog.com/gotime/313, transcript: https://github.com/thechangelog/transcripts/blob/master/gotime/go-time-313.md) They use the same test suite both ways, to test Go's implementation as well as BoringSSL testing their implementation and interoperability. Their approach solidified the idea of declarative tests for me. I got myself side-tracked, but the point I'm driving to is: perhaps starting with BoringSSL is a safe bet, and then using the bytes records in github.com/golang/go repository to build a minimal and compliant SSL/TLS implementation in vtest satisfy point B. > Where the subject matter is one-way, for instance Nicolas Seriot's > JSON beastiarium (https://github.com/nst/JSONTestSuite) the success > of compliance testing is entirely up to the malicious creativity > of the person(s) writing the tests, and Nicolas did really well, > breaking (almost?) all JSON parsers in existence. > > But where the subject matter is two way, like communication protocols, > I have never seen compliance testing deliver anywhere near promise. > > The first problem is that even when everybody agree 100% on what > the protocol definition says, most protocols explode combinatorically > in very few exchanges. > > A HTTP server can return 3xx to almost any request, that is not > a compliance failure, it can send "Connection: Close" or Keepalive, > that is not a compliance failure either. > > Imagine this: > > compliance-client: > HEAD / > server-under-test: > 301 to /justkidding > compliance-client: > HEAD /justkidding > server-under-test: > 301 to / > > How many times is a compliant server allowed to yank a client > around like that, before it actually answers ? > > (The above is surprisingly common patterns of identification) > > The second problem is timing. HTTP doesn't have rigidly specified > timeouts, but there is still stuff like "If-Modified-Since", which > the server is allowed to just ignore whenever it feels like it. Those are good catches. I think they can be resolved by stating requirements and expectations of the behavior of system under test. For instance, it can be a stated that only X redirections allowed, and optional values may have multiple test cases where in one of them the server isn't permitted to ignore. > But the biggest problem of all, is that people simply do not agree > what standards say or mean. > > I served time in meetings with angry IBM people, where we could not > agree what "have received" meant. (Their mainframes crashed when > my code answered too fast because they (tacitly) assumed modems and > a telco would always be involved, where I used a 5 meter null-modem > cable.) Isn't natural language lovely? Not to make this subject my Moby-Dick, but I honestly believe this is also a great reason to have the standards be translated into declarative test suite for clarity of intent. > That is not to say that such testing is never valuable, it is, and > IETF have often arranged "bake-offs", where different implementations > would meet, and after first getting things to work, they would try > to break each other's implementations with hostile traffic. > > So as I said: I do not want to discourage you, and as long as you dont > expect to hand out "HTTP-compliant" gold star stickers, I dont see > why it cannot or should not be done. > Let me put it in writing for posterity: The test suite is diagnostic and is not a certification for rubber-stamps. That wasn't very discouraging :) Concerns and contrary opinions are helpful in probing ideas and approaches to solidify the scope and implementation. I believe we see eye-to-eye on the value. I'm eager to learn the next steps. If the preference is have the majority of the discussion at the Workshop, that's also fine. I'm signing up shortly. All the best, Mohammed
- Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite Poul-Henning Kamp
- Re: Declarative HTTP Spec Test Suite Willy Tarreau
- Re: Declarative HTTP Spec Test Suite Daniel Stenberg
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite Daniel Stenberg
- Re: Declarative HTTP Spec Test Suite Poul-Henning Kamp
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite Willy Tarreau
- Re: Declarative HTTP Spec Test Suite Sawood Alam
- Re: Declarative HTTP Spec Test Suite Lucas Pardue
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite David Benjamin
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf
- Re: Declarative HTTP Spec Test Suite Mohammed Al Sahaf