[openpgp] text vs. binary in an OpenPGP "Signed Message"

[Daniel Kahn Gillmor <[e-mail]>]

I've been looking into interoperability for the OpenPGP "Signed Message"
construct, and found myself in a corner of unspecified behavior due to
the distinct ways that OpenPGP differentiates between "text" and
"binary" data.

The simplest modern OpenPGP "Signed Message" looks like this:

- One-Pass Signature Packet (OPS)
- Literal Data Packet (LIT)
- Signature Packet (SIG)

Compactly, we can represent that as "OPS LIT SIG"

Each Literal Data Packet can be tagged with a "format" octet [0].  The
two well-specified values for this format octet are either 'b' (binary)
or 'u' (Unicode text).  A 'u'-formatted packet…

> MUST be encoded with UTF-8 (see [RFC3629]) and stored with <CR><LF>
> text endings (that is, network-normal line endings).

[0] https://www.rfc-editor.org/rfc/rfc9580.html#section-5.9-3.1.1

I'll denote a Literal Data Packet with format 'b' as LITb, and with 'u'
as LITu.

The Signature packet has a Signature Type octet [1], which for the
OpenPGP Message construct is probably constrained to one of two values:

- 0x00 - Binary Document Signature
- 0x01 - Text Document Signature

Signature type 0x01 is expected to be made over …

> the text data with its line endings converted to <CR><LF>.

[1] https://www.rfc-editor.org/rfc/rfc9580.html#sigtype-text

I'll denote a Binary Document Signature Packet as SIG0 and a Text
Document Signature Packet as SIG1.

The OPS has a corresponding Signature Type octet [2], which I believe is
expected match the Signature Type octet of the corresponding signature.
I'll denote these as OPS0 and OPS1.

[2] https://www.rfc-editor.org/rfc/rfc9580.html#section-5.4-3.2.1

Let's start by assuming that the OPS matches the SIG (though
i don't see where the spec mandates that), and that we shouldn't expect
any implementation to validate such a Signed Message.

So, we still have four different valid kinds of "Signed Message" with a
single signature:

- (a) OPS0 LITb SIG0
- (b) OPS1 LITt SIG1
- (c) OPS0 LITt SIG0
- (d) OPS1 LITb SIG1

Every implementation i've tested appears to agree how to verify (a) and
(b).  I've yet to find an implementation that generates (c).  But i've
found multiple implementations that produce (d):

    https://gitlab.com/sequoia-pgp/sequoia-sop/-/issues/46 and
    https://github.com/pgpainless/pgpainless/issues/465, so far…

There are at least two different interpretations about what to do with
(d): GnuPG attempts to verify the unmodified bytestream within the LITb,
without converting the line-endings to CRLF, while every other
implementation i've tried (including at least RNP, Sequoia, PGPainless,
rpgp, and gosop) appears to try to convert the bytestream to CRLF before
verification.

So i guess my questions for the WG are:

- Is it a bug if a signer produces (c) or (d)?

- Should a verifier reject (c) or (d) automatically as malformed?

- If not, should a verifier that encounters (d) attempt to apply CRLF
  line endings to the LITb?

What do you think?

     --dkg

PS For now, maybe we can set aside the uglier scenarios, like mismatched
signature types in the OPS+SIG combo, multiple SIGs of different 
types in the same message, or these constructs within an encrypted or
compressed wrapping. 😬