[Sidrops] 6486bis: referenced object validation

[Ben Maddison <[e-mail]>]

Hi all,

As a result of the incident last night that caused the number of VRPs
output by some RPs (certainly routinator, possibly others too) to drop
dramatically, a couple of us have been going over the observed behaviour
in comparison with the current text of 6486-bis.

It appears from Job's analysis [1] that the incident was triggered when
several resource certs that were listed on an APNIC issued manifest
became invalid due to their 'notAfter' time expiring, which in turn
caused the affected RPs to consider the manifest invalid and fail the
fetch.

It should be perfectly legitimate for a manifest to list objects which
will become invalid during the lifetime of the manifest, or which are
not yet valid when the manifest is generated (for pre-staging).
The presence of such objects should not invalidate the manifest.

I think there are at least two places that need an update in light of
this:

Section 6.4 states:
    """
    If there are files listed in
    the manifest that cannot be retrieved from the publication point, or
    if they fail the validity tests specified in [RFC6488], the fetch has
    failed and the RP MUST proceed to Section 6.7
    """

The reference should probably be to both RFC6487 and RFC6488, as the
former provides the validation details for .cer and .crl objects, which
are presumably intended to be included in this step.

Additionally, due to the values in the 'notBefore' and 'notAfter' in a
Resource of EE cert, an object that was legitimately included in a
manifest may become invalid when processed later in the manifests
lifetime by an RP.

I would therefore suggest the following revision:
    """
    ... if they fail the validity tests specified in [RFC6487] and [RFC6488],
    with the exception of the checks against certificate Validity From
    and To values described in Section 7.2 of [RFC6487], the fetch has
    failed ...
    """

Although not directly related to the current problem, the list of
objects in section 6.4 for which the ability to process is mandatory 
should perhaps be reduced to those in general circulation today, and not
include BGPsec certs or ghostbuster records, since those add
approximately zero utility to an operator who wishes to run an RP today,
and are therefore unlikely to be priorities for implementation.

Next, section 6.6 states:
    """
    If a current manifest contains entries for objects that are not
    within the scope of the manifest (Section 6.2), the fetch has failed
    and the RP SHOULD proceed to Section 6.7
    """
I believe that the reference to 6.2 is a typo, and should point to
section 2:
    """
    A manifest associated with a CA's repository publication point
    contains a list of:
    o  the set of (non-expired, non-revoked) certificates issued and
       published by this CA,
    o  the most recent CRL issued by this CA, and
    o  all published signed objects that are verifiable using EE
       certificates [RFC6487] issued by this CA.
    """

The term "non-expired" begs the question of whether a certificate should
be non-expired at the time of manifest generation or RP processing to be
in scope.

On the one hand, insisting that a listed certificate be valid throughout the manifest
lifetime is what gave rise to the failure described above, and is
clearly, IMO, the wrong choice.

On the other, insisting that a certificate be valid at the time of
manifest generation precludes certificates which have not yet reached
their notBefore time. Validating as if "at manifest generation" also
appears to contravene the guidance in section 4.6.1 of RFC6487:
    """
    Relying Parties SHOULD NOT attempt to infer from this
    time information that a certificate was valid at a time in the past,
    or that it will be valid at a time in the future, as the scope of an
    RP's test of validity of a certificate refers specifically to
    validity at the current time.
    """

As above, therefore, I would suggest that rather than choosing either
of the above options, the question of temporal expiry should be left 
out of manifest validation altogether.
Thus point one of section 2 becomes:
    """
    A manifest associated with a CA's repository publication point
    contains a list of:
    o  the set of non-revoked certificates issued and published by
       this CA,
    o  the most recent CRL issued by this CA, and
    o  all published signed objects that are verifiable using EE
       certificates [RFC6487] issued by this CA.

    The list MUST include any objects that have a validity period that
    overlaps the period between the thisUpdate and nextUpdate times of
    the manifest.

    ...
    """

There may be other parts of the text that are similarly in need of
update or clarification on this point: I'll update if I find any.

Hopefully the above makes sense. Please let me know if anything is
unclear.

Cheers,

Ben

[1]: https://lists.nlnetlabs.nl/pipermail/rpki/2020-December/000245.html