[openpgp] Re: OpenPGP grammar concerns (fwd: [Peter Gutmann] Re: [oss-security] Retrospective by 'gpg.fail' authors)

[Andrew Gallagher <[e-mail]>]

Hi, Peter. 

On 4 Oct 2026, at 06:32, Peter Gutmann <[e-mail]> wrote:
> 
> 
> I was just quoting Thomas Ptacek there,

Do you have a reference? I’m not familiar enough with his works to find the specific quote.

> but I assume it's things like
> signatures where you hash the message, then you hash in bits of the signature
> subpacket, then you hash in some more stuff just for luck (0x04 0xFF etc), and
> that's the signature.

The trailer structure is definitely underdocumented, but none of it is there “just for luck”, it’s carefully constructed to protect against malleability attacks. If it had been better documented we might not have had the Obeneur downgrade attack against v5 sigs [1], so to some extent I agree. But that’s not a wire format issue as such.

> It's particularly bad for library implementations because you have no control
> over the data quantities you're seeing, the caller can feed you data and pull
> processed data back out in arbitrary quantities with breaks at arbitrary
> locations, for which the jumble-of-packets is a gift that keeps on giving in
> terms of all the corner cases and boundary conditions that need to be handled.

Pipelining and buffered IO should handle any such issues, surely?

> 1. Proper encapsulation rather than the jumble-of-packets, so some form of:
> 
>    KEYEX_SET( KEYEX, KEYEX, KEYEX, ... ) || ENCRYPT( data ) || MAC
>    ESKS*{ pkesk_area[ PKESK, ...], skesk_area[ SKESK, ...] } || SEIPD
> 
> and the same for other situations where multiple packets are valid like
> signatures, so:
> 
>    SIGNATURE_SET( SIG, SIG, SIG, ... )
> 
> rather than:
> 
>    SIG || surprise! || more_surprise! || mystery_stuff

I agree that more explicit details in the framing would simplify the receiver’s state machine, but I think “surprise” is over-egging it - at any point in the data stream there are only a small number of packet types that can reasonably follow. We definitely need to tighten up the grammar definition to better align “legal” with “reasonable”, and we can do so without defining new wire fomats. We could then consider more explicit framing as a subsequent step.

> 2. Removal of the pointless literal data packet, which only exists to
>   communicate DOS file metadata but adds an extra unnecessary step of
>   processing to every single PGP operation to handle first the encapsulating
>   packet's length encoding and then an inner layer of literal-data length
>   encoding.

I disagree, the primary function of the literal data packet is to store the length of the cleartext. In a one-pass message construction, you need to know the cleartext length, because the signature(s) follow. And I don’t see any way in principle of constructing a sign-then-encrypt message without double encapsulation. The length of the encrypted data stream must be outside, and the structure of the (usually signed) cleartext message, including multiple field lengths, is contained inside. Storing the cleartext structure outside the encrypted blob leaks the internal structure to eavesdroppers unnecessarily and also violates layer separation. So there will always be a double length encoding in sign-then-encrypt. 

Unless you’re suggesting that pgp inverts its model and does encrypt-then-sign instead? This would allow the cleartext data length to be inferred from the encrypted data length, however the encrypted data must be padded, either to the next block boundary or to the anonymity cohort’s bucket width. So there’s still an “inner length” that has to be separately encoded. I’ll grant that this could be part of the definition of the “encrypted data” structure and not the cleartext, but it would still have to be inside the encrypted blob for bucketing to be effective (there’s no point padding for anonymity if you leak the size of the padding in the external metadata!). And this would represent a downgrade in pgp’s security model. You could argue that in an email context the sender’s identity is already visible on the wire, but pgp is not just for email.

>   This would require adding an encapsulated-content-type indicator to the
>   preceding KEYEX_SET() / ESKS{}, which also means you know what's coming
>   next without having to do a lookahead to find out.  

Why a lookahead? I’ll defer to other implementers here but I’m not aware of anyone else doing so. OpenPGP is optimised for streaming (hence the OPS construction) so lookaheads would surely be a common pain point if they were truly necessary.

[1] https://blog.pgpkeys.eu/security-issues-librepgp-2024-08.html