Re: [openpgp] AEAD Chunk Size

"Neal H. Walfield" <> Fri, 29 March 2019 09:08 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id E279C120261 for <>; Fri, 29 Mar 2019 02:08:02 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 9D0KdLXS6tLT for <>; Fri, 29 Mar 2019 02:07:58 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id BCB35120257 for <>; Fri, 29 Mar 2019 02:07:58 -0700 (PDT)
Received: from ([] by with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.86_2) (envelope-from <>) id 1h9nUR-0003fo-I0; Fri, 29 Mar 2019 09:07:55 +0000
Date: Fri, 29 Mar 2019 10:07:54 +0100
Message-ID: <>
From: "Neal H. Walfield" <>
To: Jon Callas <>
Cc:, Justus Winter <>, Jon Callas <>
In-Reply-To: <>
References: <> <> <> <> <>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL/10.8 EasyPG/1.0.0 Emacs/24.5 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=ISO-2022-JP
Archived-At: <>
Subject: Re: [openpgp] AEAD Chunk Size
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Ongoing discussion of OpenPGP issues." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 29 Mar 2019 09:08:03 -0000

On Fri, 29 Mar 2019 03:19:39 +0100,
Jon Callas wrote:
> Like some interim replies, particularly Bart Butler, I thought we had a rough consensus and that it was approximately:
> * MUST support 16KB chunks.
> * SHOULD support 256K chunks, as these are common (Protonmail).
> * MAY support larger up to the present very large size.
> * MAY reject or error out on chunks larger than 16KB, but repeating ourselves, SHOULD support 256K.

I still think this is a bad idea.  And my discussions offline with
Justus and Marcus suggest that they also don't agree with this.  As I
understand Sebastian's position, he also doesn't agree:

Bart and Sunny are also for reducing the range of allowable chunk sizes:

The proposal that I put forward removed the chunk size byte and
standardized a single, fixed chunk size.  I suggested 16kb, but given
Bart's arguments, I was willing to compromise on 256kb:

I also suggested using a different packet tag, and marking 20 as
reserved to avoid breaking users of the current AEAD packet:

> I could also get behind a hard limit of 2^30 on the grounds that that’s what we had for partial body lengths, but I understand the comment that there are things like multi-terabyte S3 buckets and out and out forbidding those to be single-chunk is a bit churlish, but only a bit.

This is the bit that I don't understand.  Clearly you see some
advantage to having large chunk sizes.  I've asked several times on
this list, but no one can tell me what concrete advantages large chunk
sizes have.  (Tobias has suggested completely removing chunking, which
does protect against truncation attacks, but it sacrifices the ability
to work with O(1)-sized buffers, i.e., streaming.)  Can you please
tell me why large chunk sizes are helpful?

> If we really want to tie a bow around everything, then define some notation or other marker so that an implementation can mark in a key what the max chunk size is.
> How’s that?

Please don't add even more complexity!  Again, what are the advantages
of choosing one chunk size over another?

Ferguson, Schneier and Kohno write in "Cryptography Engineering":

  "There are already enough insecure fast systems; we don't need
  another one" (2.9)


  "The more complex a system, the more likely it has security
  problems...  complexity is the worst enemy of security" (2.10)

I feel like making the chunk size configurable adds complexity whose
only justification is some hand wavy "well, in the future it might
help...".  Please help me understand why large/variable chunk sizes
are useful.

> Now with some other points, I found myself going especially Socratic
> because the missives that you and Justus wrote weren’t
> *actionable*. When I was editor, a thing I would say from time to
> time is at about 90% of the time someone would say, “Please change X
> to Y for reason Z” it would be a no-brainer to do so. Most of the
> remaining 10% would end up with some reasonable debate than then an
> answer. But if someone said, “I think X is wrong” then 90% of the
> time nothing would happen because there’s nothing actionable there,
> meaning that there’s no clear point for the editor to be doing. All
> of these standards are at some level compromises and there’s an old
> aphorism that the best compromise is one where everyone is equally
> unhappy.

I don't understand why you think I haven't been concrete :/.  In my
original email, I provided a concrete patch to 4880bis:

But, let me try to use your framing:

I propose that we remove the chunk size parameter from the AEAD header
and fix it to a small value (e.g., 64 KB or 256 KB), because 1.) no
one has shown that a large chunk size is useful, 2.) large chunk sizes
encourage implementations to release unauthenticated plaintext, which
is a serious security concern [1], 3.) if an implementation releases
unauthenticated plaintext for large chunks, then an attacker can
always convince it to release unauthenticated plaintext for the first
chunk in a message [2], 4.) making the chunk size variable increases
complexity, which is a security concern.

  but RNP appears to do the same.
  starting at "Let's assume..."

> In my particular case, I thought I mostly agreed with where you wanted to go, but I completely disagreed with some of your premises, and disagreed with some of the reasoning that got us to the conclusion I agreed with. That’s why I was being so Socratic. I thought before the missives today that we had a rough consensus and that I agreed with y’all at some basic level. Then I thought we really disagreed, and I gradually realized that we agree on the destination, but not on the path of how to get there.

As I understand your position, you want to allow implementations a
broad degree of flexibility in choosing the chunk size.  Can you
please explain why this is useful?  I've reread your messages, but
beyond what appears to me to be some hand wavy performance argument,
and an apparent misunderstand (chunk size != message size) [3],
neither of which are convincing, I can't find any other arguments.
I'm sorry if you did and I didn't understand.

  [3] I believe that if we limit it to 16K, we *will* regret that, for
  performance or other reasons. And while some of the upper sizes are
  presently absurdly large, one never knows what happens a couple
  decades from now.

  Moreover, forbidding it says that we state now that no one could
  ever have a reasonable use; my experience is that categorical
  statements like that are just asking the fates to bite us in an
  uncomfortable place. Amazon S3 increased their max object size to
  5TB a few years ago. I’m not saying it should be that large, but I
  think this is a pretty convincing argument that 16K is too small.

> If I may give some advice, it is better to make sure that your requests are actionable. If they’re not, then you’re less likely to get what you want, if for no other reason than the rest of us don’t know what you want, we’re having to guess. The basic argument that you were making ― that implementations such as yours need tight constraints ― is one we all can get behind. The other side here ― they want to do something kinda wacky (like terabyte chunks) ― can be pushed off into the land of MAYs. 

I'm a bit confused.  In my original mail, I included a PR:

Was that not actionable enough?  Should I link to the PR more often?

I feel that I need answers to two questions regarding the "allow large
chunk size" position.  I recently asked them, but I have gotten a
response.  Here they are again:

  I've spent some time thinking about use cases for different chunk
  sizes, and I can't come up with any modulo some, IMHO, insignificant
  performance tweaks.  Can you please give some examples of use cases
  that would profit from different chunk sizes?

  What should / would you recommend an implementation do if it
  encounters a chunk that it can't buffer?  I see two choices: report
  an error, or release unauthenticated plaintext.

Regarding our implementation: it doesn't actually have tight resource
constraints.  Our primary goal is to write a secure implementation.
From that goal, we derived the constraint that we must always work in
a bounded amount of space.  The current version of AEAD doesn't allow
this without potentially rejecting messages that other implementations
can process by being insecure (releasing unauthenticated plaintext),
which we don't want to do.

But, my concerns are not only about my implementation.  I'm concerned
about the ecosystem.  And, the current proposal encourages insecure
implementations as demonstrated by GnuPG and RNP processing
unauthenticated plaintext.  I think the standard should not
proactively make it harder to write a secure implementation.  And that
is what I see the AEAD chunk size doing.



P.S. I hope it is clear that I'm trying to engage in a constructive
manner.  I sincerely haven't understand your position, and I really
want to.