[openpgp] Re: Fwd: [pqc-forum] Question regarding pure vs. pre-hash ML-DSA

[Daniel Huigens <[e-mail]>]

Hi David (& all, but minus the PQC forum since my message is very
OpenPGP-specific),

On Friday, August 30th, 2024 at 23:08, David A. Cooper wrote:
> OpenPGP is effectively taking two
> different approaches to signing, depending on whether RSA, ECDSA, or
> EdDSA is being used. With RSA and ECDSA the message that is signed is
> the trailer. With EdDSA the message that is signed is the hash of the
> trailer.

The way it's framed in RFC 9580 (and the way I usually think of it),
OpenPGP always computes the hash (of the message+trailer, or salt+
message+trailer in the case of v6 signatures), and then passes that to
the signature algorithm:

   All signatures are formed by producing a hash over the signature data
   and then using the resulting hash in the signature algorithm.

However, it's true that both RFC 8017 (for RSA) and FIPS 186 (for ECDSA)
contain a hashing step, and it's not the case that the data gets hashed
twice. So, you could indeed also frame it as saying that OpenPGP passes
the (salt+)message+trailer to those algorithms.

But, given the above quote, I'd frame it more as saying that the hash
computed in OpenPGP is used in the RSA and ECDSA signing operations
directly (essentially monkey-patching the algorithms to accept a hash
digest instead of a message). Though, it's a bit unfortunate that this
is not more clearly spelled out.

Nevertheless, I would still prefer to keep the invariant that OpenPGP
computes the hash as stated in the spec. So, to me, that would imply
either using pure ML-DSA, or monkey-patching HashML-DSA to accept a
hash instead of a message as well (like we've arguably done with RSA
and ECDSA - but doing the same with HashML-DSA seems more like
extending a mess than cleaning one up).

If instead we want to retcon OpenPGP to the framing you've used, i.e.
for some algorithms we pass the message and for some algorithms we pass
the hash, that would open the door to "cleanly" using HashML-DSA without
monkey-patching it - but it would require monkey-patching OpenPGP in the
"PQC in OpenPGP" spec, causing a different mess instead.

Hence, I think the pure variant seems like a better fit for OpenPGP.
But, let us know if you disagree :)

Best,
Daniel