Re: [openpgp] AEAD Chunk Size

Wyllys Ingersoll <> Fri, 29 March 2019 20:19 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id C895512032A for <>; Fri, 29 Mar 2019 13:19:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.998
X-Spam-Status: No, score=-1.998 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 6VaAfIwv5LKT for <>; Fri, 29 Mar 2019 13:19:38 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::135]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 5379812033B for <>; Fri, 29 Mar 2019 13:19:19 -0700 (PDT)
Received: by with SMTP id f22so5874205ita.3 for <>; Fri, 29 Mar 2019 13:19:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=sShOZcrAZSzYl8s4fpZuw2R6rttzb2LcpxtdU8tZS7M=; b=ijLqC/hYzkxU65azGNk3qKDamC/Y1DwZHX5wNSXm09SOz9aE7gpdmzEa4eq8NP0Q8g mEhW58lORelwebsbp15plQ5/kEcYDyc8Hq2qk2NRcAyYbwEi7oOZJkC0KEmIVjeduJeD kxvQ4XJLQn9dzSZanjBVeOf1mAaG8LXgQxxRx2wcZbl70KZEMyiXSrwigvpQ2f3hMUnN DEZd6QqH+61x4ePrByrOYb83nWLQj9UlL/tZVicydnPTYabO3FPHIM/c/WfoRfJcBW0T khm5Fo/ueyIHZYi6Vzm/g26gEI7A0QllWcYNUbnVWGP0/ZIm3FCP3IggzEHifI2V8zS3 IFcg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=sShOZcrAZSzYl8s4fpZuw2R6rttzb2LcpxtdU8tZS7M=; b=fQkVPNB7ZWF67V12bw7M038KggHfHl5AzM/wpaOVS6JFDARv889grpyohyqp/mFB/v rjEbGao1DXGv6QEMvB4zlJ2Nze+5qW9xfo+jKwW6rBDLE1yJadjIyQM2Nt8jdTJ4Kh8z Ykr5JIVtRdEbTp3NyZuRiTP9LjXVwI8PIdOXTR6NynoLhshi2wpissaY5LLSpTeIk5a8 /cf+hYTmVYzxx5NnEm1TfwwZtkN9PwIU4AVvzVjofT7gxIMhUJYCLbeYaCSWj3QvP97W vsA2YMqMtMZW0dw7yoBP/UxTEucuwpFRJW2pIwT12wAb0UIqpVuRGBEHjksEu4XJHbsF wD+w==
X-Gm-Message-State: APjAAAUsDb3S8je4NXv/rgP6lN5QUDmSkKXBLQCQr1w6vNVnLAcuOH72 +RKY5+iXkAsze5MpOu0DaxTQQsMZw0WZ5PwvfOg=
X-Google-Smtp-Source: APXvYqyvrw5JNLfdkhLosMsJvktKyWHkXk9xKNEMSc0/+KhU5qw252JGCqCxVu1McbHX6zPxrosHVBLWXpLNtgIlRcw=
X-Received: by 2002:a24:e346:: with SMTP id d67mr6381778ith.42.1553890758461; Fri, 29 Mar 2019 13:19:18 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <> <> <> <> <>
In-Reply-To: <>
From: Wyllys Ingersoll <>
Date: Fri, 29 Mar 2019 16:19:07 -0400
Message-ID: <>
To: Jon Callas <>
Cc: "Neal H. Walfield" <>, " OpenPGP" <>, Justus Winter <>, Jon Callas <>
Content-Type: multipart/alternative; boundary="000000000000ad8e910585416205"
Archived-At: <>
Subject: Re: [openpgp] AEAD Chunk Size
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Ongoing discussion of OpenPGP issues." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 29 Mar 2019 20:19:42 -0000

As an implementor (iPGMail on iOS), I strongly support Jon's suggestion
(16K "MUST", 256K "SHOULD", everything else "MAY").  I dont want to deal
with lots of optional sizes that just introduce complexity with little
practical benefit beyond some outliers that want to consume huge files in
one big bite.  Mobile devices are somewhat memory limited (though that is
every growing), but 64K or 256K is certainly not a problem.

-Wyllys Ingersoll

On Fri, Mar 29, 2019 at 3:59 PM Jon Callas <joncallas=> wrote:

> > On Mar 29, 2019, at 2:07 AM, Neal H. Walfield <> wrote:
> >
> > On Fri, 29 Mar 2019 03:19:39 +0100,
> > Jon Callas wrote:
> >> Like some interim replies, particularly Bart Butler, I thought we had a
> rough consensus and that it was approximately:
> >>
> >> * MUST support 16KB chunks.
> >> * SHOULD support 256K chunks, as these are common (Protonmail).
> >> * MAY support larger up to the present very large size.
> >> * MAY reject or error out on chunks larger than 16KB, but repeating
> ourselves, SHOULD support 256K.
> >
> > I still think this is a bad idea.
> Noted. It was a position that I threw out because I thought it encompassed
> a lot of things that people said and also allows you, the implementer to
> say, “screw it, we’re doing 256K only and to heck with all of you” and it
> would meet the standard. You could even implement 64K, thus generating a
> brinksmanship debate with you and Protonmail and I don’t have to listen to
> it.
> I think that my proposal above is flawed because there’s this squishy
> space between 64K and 256K, and really hoped someone would either say,
> “Fine. I’ll give up on 256K” or “Fine. 256K is fine.” And then we’d modify
> the proposal and probably be done. It was, however a position I thought we
> might reason from.
> >> I could also get behind a hard limit of 2^30 on the grounds that that’s
> what we had for partial body lengths, but I understand the comment that
> there are things like multi-terabyte S3 buckets and out and out forbidding
> those to be single-chunk is a bit churlish, but only a bit.
> >
> > This is the bit that I don't understand.  Clearly you see some
> > advantage to having large chunk sizes.
> Well, actually what I was doing there was offering a number lower than
> 2^whatever for the max max max oh-dear-god-why-do-you-want-to-do-this size
> at a (to me) very, very generous gigabyte to the other side on the max size
> that you can then safely ignore.
> >
> > But, let me try to use your framing:
> >
> > I propose that we remove the chunk size parameter from the AEAD header
> > and fix it to a small value (e.g., 64 KB or 256 KB), because 1.) no
> > one has shown that a large chunk size is useful, 2.) large chunk sizes
> > encourage implementations to release unauthenticated plaintext, which
> > is a serious security concern [1], 3.) if an implementation releases
> > unauthenticated plaintext for large chunks, then an attacker can
> > always convince it to release unauthenticated plaintext for the first
> > chunk in a message [2], 4.) making the chunk size variable increases
> > complexity, which is a security concern.
> This isn’t actionable.
> There’s a lot of rationale here, but not an actual proposal.
> My proposal (left at the top of this missive for easy referral) was a
> concrete proposal. The above paragraph has a lot of justification in it and
> is interesting polemic but it is not an actual proposal.
> If you had said:
> * Chunk size is 256K. Always. If you have less than 256K, pad it with
> zeroes.
> Then that would be a proposal.
> > As I understand your position, you want to allow implementations a
> > broad degree of flexibility in choosing the chunk size.  Can you
> > please explain why this is useful?  I've reread your messages, but
> > beyond what appears to me to be some hand wavy performance argument,
> > and an apparent misunderstand (chunk size != message size) [3],
> > neither of which are convincing, I can't find any other arguments.
> > I'm sorry if you did and I didn't understand.
> You don’t understand my position.
> My position is that there are a bunch of people (like you, at least I
> think you) who want to build a general-purpose implementation that you hope
> will be used all over the place. You have a legitimate need for small
> chunks.
> There are other people who observe that there’s also a legitimate need for
> huge chunks because they want to do storage encryption on very large things
> and — well, they have considerations that I’m not paying a lot of attention
> to.
> My proposal above lets them do their thing and lets you ignore them.
> That’s my position. Making as many people happy as possible.
> >
> > I'm a bit confused.  In my original mail, I included a PR:
> >
> >
> >
> > Was that not actionable enough?  Should I link to the PR more often?
> It would help, but even better would be to send what the proposal is. I
> really did read it and it’s a patch file. Patch files are hard for humans
> to read. It’s really nice to have a redline, but it’s also nice to know
> what the full thing is without having to mentally emulate git.
> >
> >  I've spent some time thinking about use cases for different chunk
> >  sizes, and I can't come up with any modulo some, IMHO, insignificant
> >  performance tweaks.  Can you please give some examples of use cases
> >  that would profit from different chunk sizes?
> I’m not the one proposing the large chunk sizes. And as I understand the
> people who are doing it, performance isn’t their issue, data integrity is.
> There’s a dilemma in here.
> One is that there’s a belief that
> >
> >  What should / would you recommend an implementation do if it
> >  encounters a chunk that it can't buffer?  I see two choices: report
> >  an error, or release unauthenticated plaintext.
> Report an error. I’ve said that many times.
> > Regarding our implementation: it doesn't actually have tight resource
> > constraints.  Our primary goal is to write a secure implementation.
> Okay, so you’re arguing resources when you don’t really mean it. Gotcha. I
> thought that was the case that the things you were saying didn’t jibe
> together.
> > From that goal, we derived the constraint that we must always work in
> > a bounded amount of space.  The current version of AEAD doesn't allow
> > this without potentially rejecting messages that other implementations
> > can process by being insecure (releasing unauthenticated plaintext),
> > which we don't want to do.
> That constraint is a long-standing meta-requirement in OpenPGP, so cool.
> However, I detect that there is an impossible-to-solve problem here, and
> I’ll outline it in another note later.
> >
> > But, my concerns are not only about my implementation.  I'm concerned
> > about the ecosystem.  And, the current proposal encourages insecure
> > implementations as demonstrated by GnuPG and RNP processing
> > unauthenticated plaintext.  I think the standard should not
> > proactively make it harder to write a secure implementation.  And that
> > is what I see the AEAD chunk size doing.
> I think I understand where you’re coming from, but this is a standards
> organization. Standards are concerned with interoperability between
> implementors. I learned that from Jeff Schiller ages ago. Think of a
> standard as being like a dictionary and grammar. When you send me a
> message, I use that grammar/dictionary to determine what it means.
> Similtharly, if I want to encode a message that I want you to understand,
> then I encode it according to that grammar as well.
> My proposal above says to an encoder that the safest thing to do is to use
> a chunk size of 64K. Everyone can decode that, and you’re fine. Many, many,
> many of them are going to be okay with 256K, and odds are that anyone of
> any import will do fine with it. (That’s what the SHOULD means). It also
> says there are people out in Lala-land who might be playing with chunks
> that are out to 2^whatever, and that’s kinda cool, but ignore them.
> That’s why I made it. It gives a sandbox for people who want to go out on
> the edge that the mainstream can ignore.
> >
> >
> > Thanks,
> >
> > Neal
> >
> > P.S. I hope it is clear that I'm trying to engage in a constructive
> > manner.  I sincerely haven't understand your position, and I really
> > want to.
> I think that’s because I’m trying to support a community of people who
> have reasonable, differing needs, and you want to mandate your
> implementation ideas on the universe. I apologize if that’s harsh, but it’s
> what I perceive. If I’m mistaken, let me know.
>         Jon
> _______________________________________________
> openpgp mailing list