Re: [openpgp] Deprecating compression support

Andrey Jivsov <crypto@brainhub.org> Mon, 18 March 2019 22:04 UTC

Return-Path: <andrey@brainhub.org>
X-Original-To: openpgp@ietfa.amsl.com
Delivered-To: openpgp@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A0D3C1311AB for <openpgp@ietfa.amsl.com>; Mon, 18 Mar 2019 15:04:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level:
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=unavailable autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8KMDhk-VbZeL for <openpgp@ietfa.amsl.com>; Mon, 18 Mar 2019 15:04:10 -0700 (PDT)
Received: from mail-yw1-f54.google.com (mail-yw1-f54.google.com [209.85.161.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E152B1311A7 for <openpgp@ietf.org>; Mon, 18 Mar 2019 15:04:09 -0700 (PDT)
Received: by mail-yw1-f54.google.com with SMTP id e76so14230091ywa.9 for <openpgp@ietf.org>; Mon, 18 Mar 2019 15:04:09 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=F/7YWqmW+Hzqz2azhiuQ2a39DJ3e5agUcMUS7c68EOw=; b=nE46sTieT6NdF/HA8se8ztQdMeqrCb+5ALVtNQEblLHrMSpdlyko0CSpT+7d4kyYjB K956UnoSYH5whV0l+kAbGM/Lc6dujEsC7cbiA4iQPBvGH5Tpwwh7YDM0lfCAmuuvSV18 B1wTcvXrWZzRZynd521kPV1RJUR8UEfG2y/ZDnNgqPLdziz0PLoREMC6m//j9z5WPjcv hHrVqTu8JjvbKxxJKjKgxO7tuGD4dw6yXV+l+5OhCkFvAkD3K5Zx8xGDRT8uzAyGhdXN NO7qyQnAYAZNb235xhj82pYTVsUx1RAA1CQmq9WAFsRX7aPHFnZ8IUjPmSiWIh5mUvK3 XKzw==
X-Gm-Message-State: APjAAAXwxFsKn75KcfOqow2v4bKO0xEfdmm7iZwGEPVh4pPczG0kS5MY 8KeE8TIRHXqVkI+m8XYrP5McEr/hd7VIxIxDYPXuCw==
X-Google-Smtp-Source: APXvYqzjkQsps9R0nSeSZkWMukZMWqPpKyhpPlch/iHmEEPYoCDMdoy8LzIv8awwXRRKyg7TF5g4OUPxa+VVOxkXotk=
X-Received: by 2002:a25:9945:: with SMTP id n5mr16172514ybo.453.1552946648550; Mon, 18 Mar 2019 15:04:08 -0700 (PDT)
MIME-Version: 1.0
References: <871s3475dy.fsf@europa.jade-hamburg.de> <96055353-B0EB-4E25-95CC-B25D9C5A0BA8@icloud.com>
In-Reply-To: <96055353-B0EB-4E25-95CC-B25D9C5A0BA8@icloud.com>
From: Andrey Jivsov <crypto@brainhub.org>
Date: Mon, 18 Mar 2019 15:03:57 -0700
Message-ID: <CAKUk3bvBWoh9jz+T6t5yGs-P-P4cSg8AnSo_md3OFnzqVN-3=A@mail.gmail.com>
To: Jon Callas <joncallas=40icloud.com@dmarc.ietf.org>
Cc: openpgp@ietf.org, Jon Callas <joncallas@icloud.com>
Content-Type: multipart/alternative; boundary="00000000000057a90105846591c7"
Archived-At: <https://mailarchive.ietf.org/arch/msg/openpgp/JyZcywIM00LsxoGaT7r0ZAORVdY>
Subject: Re: [openpgp] Deprecating compression support
X-BeenThere: openpgp@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Ongoing discussion of OpenPGP issues." <openpgp.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/openpgp>, <mailto:openpgp-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/openpgp/>
List-Post: <mailto:openpgp@ietf.org>
List-Help: <mailto:openpgp-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/openpgp>, <mailto:openpgp-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Mar 2019 22:04:15 -0000

I agree with deprecation at a reasonable pace. SHOULD looks right.

I would primarily like to correct what PGP Zip is. It is a TAR file with
standard OpenPGP compression. The compression layer was not provided by
gzip (that would not have worked in Windows environment). PGP Zip doesn't
need a license  / is free. Therefore, we should expect that OpenPGP
compression is likely used by users of commercial PGP on Windows. In
business world, many users have PGP installed because their employer wants
every machine to use PGP disk, and so they get PGP Zip available to them.

On Mon, Mar 18, 2019, 1:46 PM Jon Callas <joncallas=
40icloud.com@dmarc.ietf.org> wrote:

> I like the basic proposal. I think that deprecation is better than
> banning, and consequently we ought to be doing it with SHOULDs and SHOULD
> NOTs rather than MUSTs, as that’s banning rather than deprecating.
>
> However, I want to add that implementations today can deprecate it on
> their own and no changes in the standard need not be there. If an
> implementation creates a key with compression preferences set to no
> compression then any other implementation is bound by the standard not to
> compress. I advocate this as well. As OpenPGP sits today, any implementer
> can unilaterally not only deprecate, but eliminate compression. In fact,
> I’ll go so far as to advocate that all implementers, even GnuPG ought to
> just start making keys by default that eschew compression. More on this
> below.
>
> My rationale for getting rid of compression is different. I think that the
> security-based reasons in this thread are largely unconvincing and often
> just not quite correct. I don’t want to discuss that in this thread,
> because I agree with the destination — let’s move away from compression,
> and am afraid that if we debate the reasons for it we distract from the
> result we mutually want. We should deprecate compression, but not because
> of security, but because of simplicity, and in short a case of “that was
> then, this is now."
>
> OpenPGP has many things in it that are needlessly complex, and compression
> is one of them. (Other needless complexities include the way that packet
> sizes are computed, and others.) Much of this comes from its heritage as a
> utility, and a utility that was designed for DOS and BBSes. The
> considerations of networking and communications in 1991 are not at all what
> we should have for 2021.
>
> One of these differences is that zip-style compression is just not as
> useful now. Obviously, the wins for compression are on things that are the
> largest. Most of the large things we transfer today are already compressed,
> and thus compressing them again is mostly a waste of cpu cycles. There are
> some file formats that have zip-style compression built in. For example,
> GIF and PNG graphics are explicitly compressed. Other formats like JPEG are
> a different sort of compression but have the same characteristic that it is
> reasonably mathematically pseudo-random and not readily compressed further.
> Ditto for audio formats like MP3 and AAC, while FLAC and ALAC are like GIF
> and PNG in that they’re losslessly compressed.
>
> Today, the large things we send are media, and therefore there’s no need
> to compress them. The things we send that are easily compressible tend to
> be smaller.
>
> In the cases where large files are sent that can be compressed, the best
> solution is for that system to compress the file itself. Remember,
> OpenPGP’s history comes from DOS programs on small computers and they
> didn’t even have a way to pipe a tar or zip command to an encryption
> program. That’s not the case today.
>
> Historically, compression has been the largest share of the time it takes
> to process a file with an OpenPGP implementation, and thus we are doing
> something by default that burns CPU for little gain. We should stop doing
> it by default. Again, the best way to do this is to create keys that have a
> compression preference of none. We can do this now.
>
> Over a decade ago, the PGP program had a feature of the “PGP Zip” file.
> This was literally nothing more than a tar file that was run through gzip,
> and then encrypted with OpenPGP without compression. PGP itself had some
> nice viewers that let someone manipulate the container just as other
> directory compression systems did, but any unix system could handle the
> file by just decrypting with gnupg and piping that to tar. Later on, as I
> remember (correct me if I’m wrong, Derek, or someone), we shifted to bz2.
> The same strategy can be improved as compression tools improve.
>
> The compression inside of OpenPGP is also hard to implement correctly. The
> default compression, the “DEFLATE” option is a modified implementation of
> ZIP-style programs from the era of the late 1980s. As those programs
> coalesced into default (not precisely standard, merely ubiquitous)
> implementations, they went in one direction and that era’s PGP stayed with
> the variant. When we did RFC 2440, we added in ZLIB (RFC 1950) encryption
> so that an implementation wouldn’t have to hack the compression software.
> This is another reason to move away from compression inside OpenPGP. RFC
> 4880 added in an option for BZ2 (it was the new hotness at the time), and
> these days the cool kids are using 7-Zip as it's even better than bzip. The
> best way to keep up with advances in compression is piping from a
> compressor to some OpenPGP implementation, rather than continuing to chase
> advancements.
>
> The underlying reality is even worse. Go look at section 5.6 of 2440, and
> there are interoperability hints for working with PGP2 because it had
> further limitations in it for internal table sizes. This is another reason
> to get rid of it for simplicity. It flat isn’t needed.
>
> Simply not having compression aids implementation, as well. A number of
> years ago, I was putting together a Javascript implementation and there
> were no good compression libraries at the time, so we just created keys
> that said “no compression, please” and ran with it. Every implementation
> can handle decoding an OpenPGP object that is not compressed, so there are
> fewer bits of backwards compatibility to transition away from it.
>
> Lastly, I’ll note that everything I’ve said here could be applied to
> banning encryption in 4880-bis, as well. I am not opposed to a ban, but I
> think deprecation is better, particularly since every implementation can
> just stop doing compression by default and everything will work just fine.
> Repeating myself, I even advocate this. Everyone who makes some OpenPGP
> program can stop today, and likely should. I’ll even promote that to
> SHOULD.
>
> (The biggest reason against an outright ban is that there are a lot of
> systems that use OpenPGP in the midst of some internal process, like moving
> around large, sensitive files. There are thus likely a whole lot of shell
> scripts somewhere that someone’s got to find and change, and I wouldn’t be
> surprised if something breaks if there’s a sudden shift. If we simply start
> creating keys with preference of no compression, it gets maximum quick
> uptake, and can even be improved with some options in gnupg.conf or
> equivalent.)
>
> So to sum up, I completely support deprecating compression, because it
> improves and simplifies the standard. Today, compression by default is of
> limited benefit, and it simplifies implementation and understanding of
> OpenPGP to phase it out. I am an enthusiastic supporter of implementations
> just making the changes now, without a document change. I’m also in favor
> of deprecating. I’m not opposed to a ban, but I think it’s unneeded.
>
>         Jon
>
>
> _______________________________________________
> openpgp mailing list
> openpgp@ietf.org
> https://www.ietf.org/mailman/listinfo/openpgp
>