Re: [Sidrops] what to do when the CRL is hosed?

Christopher Morrow <> Wed, 01 April 2020 18:33 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 11FB13A15FE for <>; Wed, 1 Apr 2020 11:33:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id Rm0t6w5Sxy7j for <>; Wed, 1 Apr 2020 11:33:10 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::f2f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id DBCE83A1614 for <>; Wed, 1 Apr 2020 11:33:09 -0700 (PDT)
Received: by with SMTP id t4so254136qvz.8 for <>; Wed, 01 Apr 2020 11:33:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Wt8mqvQ31xemRJI69e3FVdj4gZtM2HTb+vLFgmUo/GI=; b=vfEGkIavkADSuhe/d/0JFhqnf7UvG7Ie63eECRiNP1E0tFcMQnPCoFUQAfwcazDo4S 2YXi5leSftdKr8E+qyGvooHXiEJCKHWJ+2bIDtRCChtUZbr8pI56vzuATqAaq96Er6F1 BCLXEtrUXeuyng71S8yvhwmwYioNPMm1+pSdg31e2nTyvKUr3EUePbCm5zEJT/IOX7wJ pLoJ1gYyAq4CWwxYbH+EU3hpAdKuJ6t9vyfeZ+4NFLcHhvhkmOAKSpcG4X1mYVj0ZITe FKHP9gFTBFgfb111N7SEtWU7rL6MuM1G8ahzwfpEaVgwERixcMzYkWkpG26k106G/jyO 5AuA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Wt8mqvQ31xemRJI69e3FVdj4gZtM2HTb+vLFgmUo/GI=; b=MbqmSfBjC/n5+asgYXfd/3m5oXDU4zOf6Zg6XqVqzhz5KvYO3hbPvw5RYJQ6sKPLGq oBumonqWVLGoyCjFQ41juH+NaCc08TTpX+X9v/sHZbMwG9XhI35gUDXmcppn03SvpchI JqhbyR/mil1nPKwV5Na7RkI1YTaCq4syYPXbC7BqVV27qqSTK44qe0HP+4nMf4ZkGVVy g5kJoJV0sOQaa2kDwP5q78KdDdcGUyKunVZ+enSd+R/cdF8qwGvI1gn+qm9K8x0y5CJ/ Kt8WUgGhE/o0J70lvFDOjze1614atNDtKH5c7+bIxijL6W7KfaZAqF3ZqQgz+AQCEkje mHVQ==
X-Gm-Message-State: ANhLgQ2OYbAqTYfcgTbrlfGDr/4hJh1Y6dzeJZWoDCUcd0K1nldBAd6G iZgQsCSCJf7v0wL29iirK2WFHobkbYD6sgiROibgDU4n
X-Google-Smtp-Source: ADFU+vvLoxPwi+oz9kLxhgQ8Hc8NaBK7/5CpKGLdV+0OXBMVAzDy8BTLNTcOspXPsVY6IDLNwYcUCYq0LzJWmjCIbvQ=
X-Received: by 2002:a0c:eb4a:: with SMTP id c10mr16564211qvq.70.1585765988468; Wed, 01 Apr 2020 11:33:08 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <>
In-Reply-To: <>
From: Christopher Morrow <>
Date: Wed, 01 Apr 2020 14:32:57 -0400
Message-ID: <>
To: George Michaelson <>
Cc: SIDR Operations WG <>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <>
Subject: Re: [Sidrops] what to do when the CRL is hosed?
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: A list for the SIDR Operations WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 01 Apr 2020 18:33:12 -0000

On Tue, Mar 31, 2020 at 6:21 PM George Michaelson <> wrote:
> I think you did capture the spirit.
> I would remind people that *all* the products in a repository are
> signed objects. To arrive at a state where a manifest is "wrong"

this was the meat of the first part of my message:
  "We planned from the get-go that transport security wasn't available,
    and instead object security would be required."

This brought along some, perhaps implicit, requirements about the
state of the repository and objects contained therein.

> demands two things: it doesn't reflect some real-world situation
> regarding contents of the repository *and* its signed by keys you can
> validate back to a trust anchor. If you cannot validate the manifest,
> then its not "wrong" its forged, or invalid cryptographically.  To me,

I agree with this,

> a  "wrong" manifest checks out cryptographically but somehow is not
> coherent with the state of the repository. If objects are on the
> manifest but cannot be fetched, you are probably suffering either from
> hiding of objects, or a transitional state of publication. If objects
> can be found which are not on the manifest, you may be seeing
> artifacts which are not defined as critical (must be found) or, the
> manifest has not yet been updated.

This part isn't really what we started talking about, but bears some
discussion I think. The manifest is 'must be there' objects in the repository.
If there are manifest listed objects which are not in the fetch... that
repository is in a bad state and probably operations should decide
what to do about that.
  o Do we have guidance?
  o Does that guidance provide a list of actions/tradeoffs?
  o will resolution come in time (for your particular problem)

I think most of the docs so far are less definitive ;) about this.
This, is one of the things that I think the discussion/wg should provide
in a more clear manner.

As an ops group it seems reasonable for us to do this even,
we're pointing out a difference between theory and practice and
documenting a fix.

> There are potentially transitional states in the production of a large
> repository under eventually consistent production (e.g. multiple
> asynchronous cooperating processes, dual-redundant signing models)
> where what can be fetched and what is on manifest do not align.  If we

Part of the agreement between repository and RP is really, I think:
  "This repository once published is correct"
  (note, not: 'eventually consistent')

This isn't really said in the docs either, and since we let all the flowers
bloom here, and kept saying:
  "the repositories are going to be inconsistent for a bit"

I think we didn't help repository operators pick 'sane' models for
management of their repository content over time.

> wish to demand that the manifest and a given state of the repository
> *always* align, we need to formalise that in a way which makes it
> clear we publish repository state in as close to atomic-update as
> possible. (copying a repo to a new dir, modifying the contents of the
> new dir, and (re)publishing the change through the rename() call in
> UNIX is one such model)

This was, actually, part of the point I made 'years back' about management
of repositories... that lead to the model I (admittedly 'puppet for
dummies' style)
put together for managing the test repository work setup.

I think it'd be worth document (as part of the outcome of the proposed
conversation) what 'contract' we expect the respository operations and
relying parties to have.. 'contract' in a software expectations sense, not
in a legal-beagle sense.

The next bits step into the 'given case X guidance from the WG for
operators says 'do Y'.

> A missing manifest is not a "wrong" manifest -its a sign of data being
> hidden from you either because of transitional effect in publishing
> the repository, or for persisting state a problem in the repository
> (diskfull?) or, production systems, or a MITM attack. Which do you
> think is actually most likely at this current time? This is no
> different to a missing CRL in that regard. Which do you think is the
> most likely reason, because you cannot a-priori know that its a MITM
> attack, or a failure in networks, or storage systems, or production
> systems.
> During the early design phases of the system, we determined that since
> the products were all cryptographically signed, there was no strict
> requirement for cryptographic protections on transport. If that
> decision needs to be revised I think it should be done in standards
> work, here, as a discussion. I do not yet see a document which says
> that. I don't see formalisms which go to a normative change in SHOULD
> to MUST on this.
> So these comments aside, my plea is that people clearly state when
> they say "wrong" or "bad" if they mean that it is cryptographically

yes, loose lips (phrasing) sinks ships (discussions).
I'm trying to be more clear, if I'm not please say:
  "You are unclear here, more words pls!"

> valid, but does not align with some external reality, or some other
> meaning. (Which btw, is complex because the part which none of this
> can adequately capture is *intentionality* -Just because you think a
> publication of RPKI state is nonsensical does not mean its not
> intentional)
> Attack models need to state clearly how they acheve the state. An
> attack on the integrity of the manifest needs to explain coherently
> how it creates a manifest which checks out cryptographically, and yet
> hides or legitemates something which should not exist. They also need
> to explain how the specific attack on the manifest in these situation
> outweighs other considerations: If they have the keys, then surely the
> attack vector is to make validly signed things which cannot be
> detected as an attack?
> -George
> On Wed, Apr 1, 2020 at 1:42 AM Christopher Morrow
> <> wrote:
> >
> > first, apologies for getting back around to this so late :(
> >
> > On Thu, Mar 26, 2020 at 10:57 AM Stephen Kent
> > <> wrote:
> >
> > > So, I think discussing MITM attacks is a distraction, unless we have examples of how
> > > such attacks can affect RPs in ways different from attacks on repositories. Maybe re-reading
> > > RFC 8211 would be useful, as it tries to analyze a range of possible "adverse actions"  by CAs or
> > > repository managers in the RPKI context, and discusses how RPKI mechanisms are intended to
> > > detect/counter these actions.
> >
> > In the case of the incident which started this thread we ended up
> > publishing part of the content a
> > repository needs to publish such that the relying parties can verify
> > properly that the content in the
> > repository is correct/valid/usable. The discussion then went along a path like:
> >    1) "well... maybe we shouldn't have belt and suspenders?" (manifest AND crl)
> >    2) "what happens if we don't publish this pesky CRL? and rely only
> > on the manifest?"
> >    3) "what if we don't publish the manifest and only rely on the CRL?"
> >    4) "CRL + Manifest has made 'rp software' hard/buggy"
> >
> > In the world where the protections specified for RPKI exist:
> >   1) self contained content protection (roa / ee-certs / etc are
> > packaged securely)
> >   2) crl signed and available in the repository for revocation actions
> > on objects in the repository
> >   3) manifest signed and listing all objects of interest in the repository
> >
> > Steve (kent!) is right mitm is harder to see as a threat.
> >   "All objects you get are signed by a ca-cert which is signed by the
> > root.. which is in the list of TAL you have. You can't have missing
> > objects and you cant' remove objects without affecting the signed
> > manifest"
> >
> > In a world where we remove one/some of the protections:
> >    A) no more manifest
> >    B) no more crl
> >    C) both
> >
> > I think mitm problems are much harder to detect/deal with :(
> > It sounds like WG folk (RP users and RP/CA software authors) are
> > asking for guidance on handling the problem(s) discussed here.
> > It sounds, to me, like a chat at the upcoming interim meeting would be
> > a great place to start that with some slideware and a proposal to use
> > as kindling.
> >
> > I think the shape of the conversation is roughly:
> >   "What would be the effect (on the routing system) for RelyingParties
> > if we decided to be less strict about CRL existence?"
> >   "What would be the effect (on the routing system) for Relying
> > Parties if we decided to be less strict about repository contents vs
> > Manifest contents?"
> >   "What happens to the routing system if the manifest and crl are
> > either/both 'broken' in a repository?"
> >
> > I don't think it matters much to the routing system where the breakage
> > occurs (my repository or RIPE/ARIN/etc) certainly there's more fallout
> > from ARIN/RIPE/etc, but... you can't get to me either way :)
> > (possibly) until repair and propogation.
> >
> > Thoughts on some slideware and discussion?
> > Did I about capture the meat of the sandwich here?
> >
> > -chris
> > co-chair but asking as a regular chemical engineer at this party.
> >
> > _______________________________________________
> > Sidrops mailing list
> >
> >