[Cellar] compression libraries in IETF specifications (was Re: [cellar-wg/matroska-specification] deprecate LZO compression (#376))

Michael Richardson <mcr@sandelman.ca> Sun, 03 May 2020 16:10 UTC

Return-Path: <mcr@sandelman.ca>
X-Original-To: cellar@ietfa.amsl.com
Delivered-To: cellar@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id EAD483A0ECC for <cellar@ietfa.amsl.com>; Sun, 3 May 2020 09:10:31 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id iSUBHQUb1MrZ for <cellar@ietfa.amsl.com>; Sun, 3 May 2020 09:10:29 -0700 (PDT)
Received: from tuna.sandelman.ca (tuna.sandelman.ca []) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 687D43A0ECE for <cellar@ietf.org>; Sun, 3 May 2020 09:10:28 -0700 (PDT)
Received: from sandelman.ca (obiwan.sandelman.ca [IPv6:2607:f0b0:f:2::247]) by tuna.sandelman.ca (Postfix) with ESMTP id 6238E3897B for <cellar@ietf.org>; Sun, 3 May 2020 12:08:30 -0400 (EDT)
Received: from localhost (localhost [IPv6:::1]) by sandelman.ca (Postfix) with ESMTP id 10789135 for <cellar@ietf.org>; Sun, 3 May 2020 12:10:27 -0400 (EDT)
From: Michael Richardson <mcr@sandelman.ca>
To: cellar@ietf.org
In-Reply-To: <cellar-wg/matroska-specification/issues/376@github.com>
References: <cellar-wg/matroska-specification/issues/376@github.com>
X-Mailer: MH-E 8.6; nmh 1.7+dev; GNU Emacs 25.1.1
X-Face: $\n1pF)h^`}$H>Hk{L"x@)JS7<%Az}5RyS@k9X%29-lHB$Ti.V>2bi.~ehC0; <'$9xN5Ub# z!G,p`nR&p7Fz@^UXIn156S8.~^@MJ*mMsD7=QFeq%AL4m<nPbLgmtKK-5dC@#:k
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature"
Date: Sun, 03 May 2020 12:10:27 -0400
Message-ID: <24830.1588522227@localhost>
Archived-At: <https://mailarchive.ietf.org/arch/msg/cellar/z4M7h3Ok_JkNNnIopU4zlFloMfA>
Subject: [Cellar] compression libraries in IETF specifications (was Re: [cellar-wg/matroska-specification] deprecate LZO compression (#376))
X-BeenThere: cellar@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Codec Encoding for LossLess Archiving and Realtime transmission <cellar.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/cellar>, <mailto:cellar-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cellar/>
List-Post: <mailto:cellar@ietf.org>
List-Help: <mailto:cellar-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/cellar>, <mailto:cellar-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 03 May 2020 16:10:32 -0000

Hi, I wanted to bring this item to the list's attention.

Spencer: I seem to remember that the IETF has had compression IPR issues in
specifications in the past.  IPCOMP comes to mind, but I am guessing that
other RTP/SIP groups have gone through this.. Is there possibly an IESG
statement that might help us here?

But, first, the software license on the libraries is not entirely germane.
An IETF specification can well contain algorithms which are currently only
available in software licensed in ways that might not be useable by all
Someone just will have to write a new library with a license that suits them!

What matters if that the specification (and any related patents!) is
available to all.  The IETF's minimum IPR requirement is generally RAND,
which is often not good enough for open source, which requires RF.
The decision as to which IPR claims to tolerate is a WG-level decision, with
the proviso that the WG (and IETF) is informed about the issue.

So even in the absense of the jackoalan and zivillian LZO implementations,
the WG should be free to standardize LZO, if it the IPR claims on the
algorithm itself are well understood.

{In the compression space, my understanding is that there have been formats
where there were no IPR claims on decompressors, but there was significant
IPR-encumbered innovation on doing compression.  An encoder might need a
specific license in order to get 70% compression rather than just 40%, but
this had no affect on the decoder.  But, I don't recall the specific
situations where this occured.  Maybe back in the old ZOO/ARC/ZIP days...}

Steve Lhomme <notifications@github.com> wrote:
    > Matroska Blocks can be compressed with zlib, bz2 and lzo compression.

    > zlib and bz2 have decompression libraries available under permisssive
    > licenses (BSD like for bz2 and OSI approved, GPL compatible, non
    > copyleft for zlib).

    > lzo doesn't have such a permissive library. The [official
    > library](https://www.oberhumer.com/opensource/lzo/) is GPLv2, there is
    > also decompression code that doesn't rely on this library [in
    > FFmpeg](https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/lzo.c)
    > which is LGPLv2 which means grabbing the whole libavutil to compile.

    > This license is not permissive enough to be used in many closed source
    > products where they don't want to touch any GPL-based license. That
    > means a file that is created with LZO in a GPL/LGPL software may not be
    > read universally because of licensing issues. I think we should avoid
    > it and mark LZO as deprecated.

    > The LZO compression advantage over others is that it is [very fast to
    > compress/decompress](https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO). But
    > the compression ratio is not good. That is good for realtime
    > transmission but bad for long term storage. In the end automatic tools
    > that test each compression (like mkclean) will never pick it anyway.

    > I found 2 other options with better licenses:

    > - a [C++ version in MIT license](https://github.com/jackoalan/lzokay) with compression
    > - a [C# version in MIT license](https://github.com/zivillian/lzo.net) without decompression

    > They are both based on the [Linux Kernel documentation for
    > LZO](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/lzo.txt).

    > In libmatroska2 I've been using minilzo GPLv2 but I may try to port one
    > of these in C so I can use it (or simply write a C wrapper around the
    > C++ one). That means there would be an alternative that is MIT license,
    > so freely used by anyone.