Fuzzing Structured Fields from ABNF

Mark Nottingham <mnot@mnot.net> Thu, 09 July 2020 11:18 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=lists.ie@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5A3E03A0112 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Thu, 9 Jul 2020 04:18:36 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.749
X-Spam-Level:
X-Spam-Status: No, score=-2.749 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, MAILING_LIST_MULTI=-1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=mnot.net header.b=nEmGjxy2; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=TNH6jBio
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GTJgdbGSZUK2 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Thu, 9 Jul 2020 04:18:34 -0700 (PDT)
Received: from lyra.w3.org (lyra.w3.org [128.30.52.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6CE433A00E3 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Thu, 9 Jul 2020 04:18:34 -0700 (PDT)
Received: from lists by lyra.w3.org with local (Exim 4.92) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1jtUWn-0001tH-KX for ietf-http-wg-dist@listhub.w3.org; Thu, 09 Jul 2020 11:15:45 +0000
Resent-Date: Thu, 09 Jul 2020 11:15:45 +0000
Resent-Message-Id: <E1jtUWn-0001tH-KX@lyra.w3.org>
Received: from mimas.w3.org ([128.30.52.79]) by lyra.w3.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <mnot@mnot.net>) id 1jtUWl-0001sV-Ka for ietf-http-wg@listhub.w3.org; Thu, 09 Jul 2020 11:15:43 +0000
Received: from out4-smtp.messagingengine.com ([66.111.4.28]) by mimas.w3.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <mnot@mnot.net>) id 1jtUWj-0001io-R6 for ietf-http-wg@w3.org; Thu, 09 Jul 2020 11:15:43 +0000
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 0B13C5C0134 for <ietf-http-wg@w3.org>; Thu, 9 Jul 2020 07:15:29 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Thu, 09 Jul 2020 07:15:29 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mnot.net; h=from :content-type:content-transfer-encoding:mime-version:subject :message-id:date:to; s=fm3; bh=nel3gJe+kg/6dO497seabHnqGLPa9qE5r FbF2WO7HZQ=; b=nEmGjxy2KYXarGrbEruaILw7IS8Yw1fuUtCpMbx/7SZXUYCnr rqdThcTzwQgIf+MpkDZzTJVahcMF3jJo9HowFaYn1dYHdekq/1d/G9f5DIS2wBBL FeUiNnA+rVuyVla6OFt/7NedM137sbZRzxj3sgdlyO9pWD4LVeu8I2gAf1hEs8fs QaTibMqkgr+EWgWmsIdeCHz2yxq21KU9wlKKQupe+3KuuUMICEdBuHYc/wZHs1cs xAhrg2fE22oDdj5bJRlEPMqLZvdwKe+oaGYJLhCN30wxTv3xce+aKWvkrNhY9xQH 8bWz78k/mAbisjrMTMgMjS8FnAzo2VU0gPZcw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=nel3gJ e+kg/6dO497seabHnqGLPa9qE5rFbF2WO7HZQ=; b=TNH6jBiooHa+K+ZxZCTSsh I7cSbN2fFVDFmMP6w4E2g1C4U2M/ukBkWbvCKG2FbCw3nZ+POSck6dMoMfrF2hM0 TGNUAFu2bUIsp4V5ESt6si2GkKca3gl8Lh3rSlweq/IfjG4h6+XK6TtLV7/bSFTK bsROq1DtEAblKa87eQQE0b/U8KsF8twbxEva/uVXMevpfWPFodJrAT+Yo8iQNtHI vIDBevAW/SiR3WRttAo/Wxr8L0s5bv0XbtJnhGTRRHw+p7mVO1nP22gIgDjxZVWc SYD/ZZ2DQ/0iPBJDyxPpBBFJZWXQXRoZYYy2TCseWFC/fWNbYSX2Grx211hmR7ig ==
X-ME-Sender: <xms:UPwGX5FLMF4cy4M98fFDjutAFNRa0h3hef7mERBbtI3wztwz3BtJQg>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrudelgdeflecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhephfgtgfgguffkfffvofesthhqmhdthh dtvdenucfhrhhomhepofgrrhhkucfpohhtthhinhhghhgrmhcuoehmnhhothesmhhnohht rdhnvghtqeenucggtffrrghtthgvrhhnpefggeetheeggffhjeehheffgfdvvedvvdefje fhudetgeefveduiedujefgfeffveenucffohhmrghinhepghhithhhuhgsrdgtohhmpdhg ihhtlhgrsgdrtghomhdpphihphhirdhorhhgpdhmnhhothdrnhgvthenucfkphepuddule drudejrdduheekrddvhedunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehm rghilhhfrhhomhepmhhnohhtsehmnhhothdrnhgvth
X-ME-Proxy: <xmx:UPwGX-WOFi6BxFFsRLlaGRIGOjHoj3EM_ON0TKTHGyFrqEBKdJniuw> <xmx:UPwGX7LteCzBG6LAFiKOo2L4xFIh2-RTfMdYTa-zhhkW9GXOZ5N8XQ> <xmx:UPwGX_GMBlYhk0sI6F0RtP5KGyifNd2UdlbRrhERZWQFYgoB37DuAQ> <xmx:UfwGXwdPP-gAY5QwKLirgzPRxkRdehCjuFrbwtI24j7wMzDkTPwkEw>
Received: from macbook-air.mnot.net (119-17-158-251.77119e.mel.static.aussiebb.net [119.17.158.251]) by mail.messagingengine.com (Postfix) with ESMTPA id 5164A328005A for <ietf-http-wg@w3.org>; Thu, 9 Jul 2020 07:15:27 -0400 (EDT)
From: Mark Nottingham <mnot@mnot.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Message-Id: <F7691BD3-8329-48C2-91DF-26F585D092D4@mnot.net>
Date: Thu, 9 Jul 2020 21:15:21 +1000
To: HTTP Working Group <ietf-http-wg@w3.org>
X-Mailer: Apple Mail (2.3608.80.23.2.2)
Received-SPF: pass client-ip=66.111.4.28; envelope-from=mnot@mnot.net; helo=out4-smtp.messagingengine.com
X-W3C-Hub-Spam-Status: No, score=-9.8
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, W3C_AA=-1, W3C_DB=-1, W3C_IRA=-1, W3C_IRR=-3, W3C_WL=-1
X-W3C-Scan-Sig: mimas.w3.org 1jtUWj-0001io-R6 8c7140c1d796e71719d052e51f1af270
X-Original-To: ietf-http-wg@w3.org
Subject: Fuzzing Structured Fields from ABNF
Archived-At: <https://www.w3.org/mid/F7691BD3-8329-48C2-91DF-26F585D092D4@mnot.net>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/37863
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

Kate, one of my colleagues, pointed out that one of the wonderful tools she's created:
  https://github.com/katef/kgt
... can be used to create input for blab from ABNF:
  https://gitlab.com/akihe/blab
... which can in turn be used to create values that conform to that ABNF.

I've put up a slightly modified blab file (with a couple of small tweaks, including one to remove generated base64, since it's so rarely actual base64):
  https://gist.github.com/mnot/06deaeadab6616929a59d687ca7dff25

This can be used to check your parser to see if it can handle the (hopefully valid) input. In the case of my parser:
  https://pypi.org/project/http-sfv/

It looks like this:

blab -l . -e "structured-fields.sf-dictionary" | python3 -m http_sfv --dictionary --stdin
blab -l . -e "structured-fields.sf-list" | python3 -m http_sfv --list --stdin
blab -l . -e "structured-fields.sf-item" | python3 -m http_sfv --item --stdin

(with http_sfv installed from source; I'll release a new version that does --stdin shortly).

Just thought this might be helpful for some implementers. I'll sleep a bit better knowing that the ABNF does indeed appear to produce correct output...

Cheers,


P.S. Should we do the same with http-core ABNF?

--
Mark Nottingham   https://www.mnot.net/