[sidr] RPKI validator testing summary

Andrew Chi <achi@bbn.com> Wed, 30 November 2011 15:38 UTC

Return-Path: <achi@bbn.com>
X-Original-To: sidr@ietfa.amsl.com
Delivered-To: sidr@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id ADC7A21F8B4F for <sidr@ietfa.amsl.com>; Wed, 30 Nov 2011 07:38:48 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.299
X-Spam-Status: No, score=-4.299 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MANGLED_DOSE=2.3, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id uK6nguJ7dGxB for <sidr@ietfa.amsl.com>; Wed, 30 Nov 2011 07:38:48 -0800 (PST)
Received: from smtp.bbn.com (smtp.bbn.com []) by ietfa.amsl.com (Postfix) with ESMTP id D964321F8B4E for <sidr@ietf.org>; Wed, 30 Nov 2011 07:38:47 -0800 (PST)
Received: from dhcp89-089-139.bbn.com ([]:51013 helo=[]) by smtp.bbn.com with esmtp (Exim 4.74 (FreeBSD)) (envelope-from <achi@bbn.com>) id 1RVmFD-000JKz-4G for sidr@ietf.org; Wed, 30 Nov 2011 10:38:47 -0500
Message-ID: <4ED64E04.7030408@bbn.com>
Date: Wed, 30 Nov 2011 10:38:44 -0500
From: Andrew Chi <achi@bbn.com>
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20111105 Thunderbird/8.0
MIME-Version: 1.0
To: sidr wg <sidr@ietf.org>
Content-Type: text/plain; charset="UTF-8"; format="flowed"
Content-Transfer-Encoding: 7bit
Subject: [sidr] RPKI validator testing summary
X-BeenThere: sidr@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Secure Interdomain Routing <sidr.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/sidr>, <mailto:sidr-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/sidr>
List-Post: <mailto:sidr@ietf.org>
List-Help: <mailto:sidr-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sidr>, <mailto:sidr-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 30 Nov 2011 15:38:48 -0000

Several people have expressed a desire to be kept informed about RPKI 
validator testing, so I'm sending a brief summary to the list.  At IETF 
82, Rob Austein, Tim Bruijnzeels, and I did some more RPKI validator 
testing.  If there are other validator implementers out there, please 
let us know so we can include you.

Data Sets

Good Data: The well-known trust anchors list that Rob assembled

Bad Data: BBN's RPKI Syntax Conformance repository
TA: rsync://rpki.bbn.com/conformance/root.cer

Previously we compared validators on good data.  This time we added 
specially crafted bad data, where each of 161 input files was intended 
designed to violate a single item in the spec.  The violations range 
from mundane (negative serial number) to serious (SKI != hash of public 
key), but we intended each case to be detectable using the standalone 
file, without referencing other files.

Overall Results

The "Good Data" testing confirmed that all three validators currently 
agree, modulo some known minor issues/differences:
- rcynic: key usage checking
- RIPE: stale CRL strictness
- BBN: bottom-up processing, scope of subdir chasing

The "Bad Data" testing is a work-in-progress, but the BBN conformance 
test cases have already proven useful in revealing corner cases in the 
validators.  All three validators (rcynic, RIPE, BBN) will benefit from 
more robust error handling due to this test set.  Sunday's session was 
also useful for debugging the syntax conformance cases themselves. 
Credit where it's due: various Cert/CRL AKI issues (thanks Rob), various 
CMS encoding issues + "good" CMS case for contrast (thanks Tim), missing 
top-level MFT/CRL (thanks Tim).

Another useful side-effect of the testing was that it raised some 
questions about relying party gray areas.

Relying party gray-areas

The specs leave room for the relying party to decide what to do with 
imperfect but not completely invalid objects.  This is for good reason. 
  But I believe implementers could benefit from an informational doc or 
BCP to guide them in these gray areas.  Here are two examples we came 

1. An invalid CRL casts doubt on, and depending on the strictness of the 
validator, potentially invalidates the manifest in the same directory. 
This is because the EE cert in the manifest is no longer perfect -- it 
*could* have been revoked by a CRL.  In a sense, this is the strictest 
interpretation of the spec.  But one could also imagine wanting more 
resilience against an attacker who can remove a CRL.  This is left up to 
the RP right now and the current implementations differ.

2. AIA correctness.  Does res-certs require validators to reject a 
certificate with a messed up AIA URI, even if top-down traversal is ok? 
  Having clean AIAs obviously helps bottom-up validators.  But 
validators capable of bottom-up traversal must already defend against 
AIA-wild-goose-chase DoS, e.g. by limiting chase depth.  Should we 
encourage validators to enforce AIA correctness?

Matt Lepinski and I are currently thinking about how to structure a 
document that will help RP implementers make informed decisions in these 
types of cases.