[Acme] Revisiting Proactive Issuance & new-order CSR

[Daniel McCarney <[e-mail]>]

Hi folks,

Just over a year ago in a thread titled "Proactive vs On-Finalization
Certificate Issuance"[0] we reached the consensus on the question of
whether to
issue a certificate "on-finalization" or "proactively" with the conclusion
to
standardize on proactive issuance.

Implementation experience shows that proactive issuance is slow in
practice. An
order can be associated with many authorizations and an authorization can be
associated with many orders. This interacts poorly with optimizations Let's
Encrypt has developed from in-the-field experience with ACME and it is
difficult
to rate-limit effectively. I propose that we return to the on-finalization
style
approach and that we provide the CSR during finalization..

## Issues with proactive issuance

### Authorization Reuse

Proactive issuance requires that when an authorization is switched to a
valid
state by the result of a challenge update that we find the associated
orders and
iterate each order's authorizations to see if all of the authorizations are
valid and issuance can occur.

Let's Encrypt reuses existing authorizations[1][2] whenever possible, as
encouraged by ACME. This has been of huge value in reducing resource
consumption
(both in terms of database storage and challenge/API requests). Many clients
create pending authorizations they do not finalize and similarly re-create
pending authorizations for identifiers that the account already has valid
authorizations for. Unchecked this leads to significant database growth and
resource consumption, even with aggressive rate-limits[3].

The combination of authorization reuse and proactive issuance means a given
authorization can be associated with many orders. This necessitates an
expensive
many-to-many query at each authorization update to see if any associated
orders are now complete.

One possible workaround is a rate-limit that would constrain the number of
orders an authorization can be associated with. However establishing the
value
of the limit to both reduce server-side load & also effectively support
Let's
Encrypt's large-volume integrators and their issuance patterns is
challenging.

### CSR-first new-order flow

To support proactive issuance the CA also needs to have the CSR for the
order
at-hand when checking each authorization in case issuance can proceed. The
`new-order` request that creates the order therefore carries the full CSR
that
was previously delivered by the "certificate finalize" message.

We see users create many more authorizations than they are able to finalize
successfully. With the legacy "new-cert" flow our authorization reuse
dramatically helped reduce the DB space used up by authorizations that will
never lead to issuance because no CSR is stored until all identifiers are
validated.

With the new-order flow we have to accept and store a full CSR before
handing
back authorizations to the client. Reuse of authorizations does not prevent
the
DB bloat that will occur from users submitting new order requests and not
fulfilling them. The CSR contains a full public key and is one of the single
largest contributors to DB disk space usage. Addressing this will require
another strict rate limit on outstanding new-orders and is again difficult
to
“dial-in” in to both prevent excessive resource consumption and also
maintain
the ability for big integrators to process large-scale issuance using our
API.

# Proposed Change

I believe we should change the `new-order` endpoint to not accept a full
CSR but
to instead accept a list of identifiers that the ACME client wishes to
authenticate & issue for. Authorizations can be created and returned for
those
domains as happens today.

The order resource should be changed to have a “finalizeURL” field which
provides the client a URL to POST a CSR to indicate that the CA should
issue the
certificate. Proactive issuance should be removed outright.

This will address both the problem of scanning existing many-to-many orders
& their authorizations as well as the db bloat that will occur from
front-loading the delivery of the CSR.

It may be tempting to leave proactive issuance as an optional feature but
this
will require maintaining the CSR in the new-order and I also believe that
having
two very distinct methods of issuance will complicate the client ecosystem
and
returns us to the question of how to determine which will occur that
spawned the
original "Proactive vs On-finalization Certificate Issuance"
thread[0].

[0] https://mailarchive.ietf.org/arch/msg/acme/0lVmGl8e-rmSH0x7ycDW5dj6GAY
[1] https://community.letsencrypt.org/t/upcoming-api-changes/17947
[2]
https://community.letsencrypt.org/t/automatic-recycling-of-pending-authorizations/41321
[3] https://letsencrypt.org/docs/rate-limits/