Re: [openpgp] AEAD Chunk Size

[Bart Butler <[e-mail]>]

Hi Neal,

Maybe a reasonable compromise is, as we discussed a while ago, simply an upper bound on the chunk size rather than fixing it to a single value. What's a reasonable upper bound for constrained systems? 4MB?

And for others: what IS the use case for really large chunks? In the limit of the message being one giant chunk, AEAD becomes essentially our current MDC and loses all additional value it has in terms of streaming, which is detecting errors/modifications early rather than at the last possible moment. Why allow producers to undermine this, given its the reason for its existence?

I agree with Jon that it's difficult to foresee future use cases, which is why I don't favor removing the size entirely--if some compelling use case for larger sizes appears, or for having a variety of small sizes (more likely IMHO), then great.

But I have yet to see a compelling use case for really large AEAD chunks, and two compelling reasons not to have them (embedded systems and general poor UX in terms of not being able to error out early). So do we really not want to do some kind of reasonable upper limit in the RFC? It would be one sentence in the RFC and minimal work for existing implementations to cap the AEAD chunk size to reasonable value.

-Bart

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, March 28, 2019 3:04 PM, Neal H. Walfield <[e-mail]> wrote:

> At Thu, 28 Mar 2019 14:27:27 -0700,
> Jon Callas wrote:
> 

> > > For me, using an unbounded amount of memory is not an option for a
> > > component processing OpenPGP data if we want to build robust systems
> > > on top.
> > 

> > Okay, prior to this working group, when there was running code
> > without a consensus rough or not, this problem existed. Even with
> > compression, PGP 2 ran on DOS machines with a max of 640K of
> > RAM. There are many similarly constrained systems that run OpenPGP
> > implementations.
> 

> Until now, OpenPGP didn't require buffering data. A decrypted AEAD
> chunk MUST only be released when it has been authenticated. In the
> current proposal, AEAD chunks are potentially unbounded (well, up to 4
> exabytes...) in size. No one can decrypt such chunks without
> cheating, i.e., releasing unauthenticated plaintext.
> 

> > > Therefore, we need to process OpenPGP data in bounded space. Since
> > > there can be no relation between encrypted and decrypted message size
> > > due to compression, the only option I see is to provide a streaming
> > > API, which let's us process data in constant space.
> > 

> > I’m not quite sure what you mean when you say “bounded space”
> > because one interpretation of that is obviously false. OpenPGP has
> > always supported being able to process messages where the encryptor
> > does not know the size ahead of time. That’s why we have
> > indeterminate lengths and chunking.
> 

> [snip]
> 

> > For example, RFC 4880 allows a partial body length (a chunk) to be
> > 2^30 octets, and that could be irritating to handle.
> 

> indeterminate lengths and partial body chunking don't prohibit the
> implementation from releasing the text early; there is no requirement
> to buffer the whole chunk.
> 

> > One of (perhaps unstated) goals of OpenPGP is that it allow for
> > highly constrained implementations. This was a huge consideration in
> > both 2440 and 4880.
> 

> We are arguing within this tradition. We want highly constrained
> systems to be able to process (nearly) all OpenPGP messages in a
> conforming manner. AEAD mandates buffering whole chunks, because
> unauthenticated plaintext MUST NOT be released. By limiting chunks to
> some kilobytes in size, highly constrained systems will be able to
> process these messages without releasing unauthenticated plaintext.
> 

> > Efail is primarily a problem with MIME encoding and layering violations. It works just as much with S/MIME as OpenPGP. Perhaps I’m missing something, but I don’t see how Efail is relevant to resource bounds.
> 

> EFAIL is also about ciphertext modification. If an implementation
> never releases unauthenticated plaintext, then ciphertext modification
> is not possible. Highly constrained systems can do this if the chunk
> size is limited to something they can handle.
> 

> > > Therefore, we need to use chunking and authenticate message prefixes.
> > > We need to use chunks that are reasonably small, and this size should
> > > preferably not be configurable. We should consider performance
> > > constraints and pick one suitable size. Configurable chunk sizes
> > > bring complexity and increase the attack surface, as was pointed out
> > > in this thread.
> > 

> > I’m with you on a lot of this, but I don’t know what you mean by “configurable”? Do you mean that there should be one chunk size only? If so, what size do you propose? 32 Meg or thereabouts (2^25 is in that ball park)? If so, would that mean that all messages smaller than your chunk size would be a single chunk?
> 

> Configurable means that the chunk size is not fixed by the standard.
> There are two proposals right now: we either fix the chunk size to
> 16kb (support by Justus, Sebastian and I) or to 256kb (preferred by
> Bart from Proton Mail, since that is what they are using in practice).
> 

> > > I appreciate the desire to protect against truncation. But,
> > > truncation is pretty common when we transmit data, so I'd argue that
> > > application developers are more likely to expect and gracefully deal
> > > with truncated data than with ciphertext being manipulated or the PGP
> > > implementation consuming unbounded amounts of memory.
> > 

> > Does this mean that you think that message truncation is an error that OpenPGP doesn’t need to guard against?
> > That’s the way I interpret the first line in the paragraph above (“I appreciate … But,…”). If so, that’s counter to the long-standing consensus of the working group. It’s the whole reason we have MDCs and the reason why they were aggressively pushed in the implementations and non-MDC packets browbeaten into doing MDCs. See the non-normative discussion in section 5.13 of 4880.
> 

> Either you stream or you protect against truncation. I don't see how
> you can do both. That doesn't mean you can't detect truncation and
> signal that it occurred. Indeed, that is a property that MDC and
> signatures provide. But, then it is up to the application to recover.
> (Perhaps the application, with or without the help of the OpenPGP
> implementation, simply buffers the whole message or a sufficiently
> large prefix. But, that is built on top. We want OpenPGP to remain
> streamable.)
> 

> > > I have implemented AEAD in Sequoia, and I have evaluated the
> > > implementations in GnuPG and RNP. Every implementation is either
> > > unsafe, not robust, or does not implement the proposal.
> > 

> > Tell us more. What problems did you find?
> 

> Sequoia is written in Rust, and in Rust out of memory errors are
> fatal. Handling a 4 exabyte chunk would result in an out of memory
> error, and termination of the process.
> 

> Both GnuPG and RNP process unauthenticated plaintext. They release
> the plaintext for further processing before it is authenticated by the
> AEAD tag.
> 

> > In my reading of this, I think I have identified two points you’re making.
> > (1) It’s possible for a chunk to be larger than reasonable processing resources.
> > (2) It’s possible for a long stream to have an error in the last chunk that signals an error wayyyyyy in the past.
> > Handling (1) is reasonably easy. Return an error. This situation exists today. It’s possible to make partial bodies of a gigabyte each, and an implementation may not be able to handle that. Return an error.
> 

> I don't understand why partial bodies are relevant here. They are
> quite easy to implement using a small buffer even given large partial
> body chunks.
> 

> The issue with returning an error in the face of large AEAD chunks is
> that it is completely gratuitous. If the sender had only used a small
> chunk size, it would have been possible to authenticate the
> ciphertext!
> 

> What is the advantage of large AEAD chunks (serious question!)? It
> seems their only effect is to encourage the release of unauthenticated
> plaintext.
> 

> > Handling (2) is also easy, you return an error. This might be unsatisfying, because the error might be in the past, and lots of stuff already handled. Is this your objection?
> 

> Supporting stream means allowing for truncation. We want streaming,
> and we are prepared to handle truncation should it occur.
> 

> Thanks,
> 

> :) Neal
> 

> openpgp mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/openpgp