Re: [Cellar] compression libraries in IETF specifications (was Re: [cellar-wg/matroska-specification] deprecate LZO compression (#376))

Spencer Dawkins at IETF <spencerdawkins.ietf@gmail.com> Tue, 05 May 2020 20:43 UTC

Return-Path: <spencerdawkins.ietf@gmail.com>
X-Original-To: cellar@ietfa.amsl.com
Delivered-To: cellar@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3B7B93A0A64 for <cellar@ietfa.amsl.com>; Tue, 5 May 2020 13:43:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level:
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ab4sHSAZcCPq for <cellar@ietfa.amsl.com>; Tue, 5 May 2020 13:43:46 -0700 (PDT)
Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6FB4B3A0A65 for <cellar@ietf.org>; Tue, 5 May 2020 13:43:46 -0700 (PDT)
Received: by mail-lf1-x12e.google.com with SMTP id 188so2503450lfa.10 for <cellar@ietf.org>; Tue, 05 May 2020 13:43:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VkENzJPazxnlSzPxtFKZ15ZSMnGdD9+Z0G3bn/pylmo=; b=u+vWPO/w0O6R/wWCI4CWApAzsB9+SJgMuAKtQlXKYsDFiArCUQXgfEwtlP3yeF3C5+ OKXuy5/AOT+Cf9poTP4D8sfdb4AYIMdHQDY8oQZn7KQ2wKu7rwd4rEO9bOf4N2puuq9C pxz1eaxwrUzDxneYckFP9TPIY0kYzlgb+4uzJhFrRhXB0pdIzuwwjJtPi3G04B3Nt0bY ZTLtNSgLsBF5iS7NMbcNaIkWfB02tlqQL2IiRrUAOO5hNa849tWBgGeqC7gfxsG5cPQB iomgx6D8LOOMXPOHMzns5Q131F3bVTB7PZ86pfO9W7rhxeAewavZ1K+6AtLO+iMFgj4t 6KZw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VkENzJPazxnlSzPxtFKZ15ZSMnGdD9+Z0G3bn/pylmo=; b=uL+PKLSVNBMEAk0u7lkX8Wt2srkNrlL9mgyAHYdMA8augZ2baKaym9sTvUH90sc4z7 A6WKWGzrD8bAB9Z+AKs54TVzdroP4QPdWZt1gRdYzBFegV7enkv3xN07RbeQvS43GMcl +04lmekapyfsS/ItYP7yJacg9GLTRKQe7eKRoGZIM88izRDsdOCprw8VTkhjq6Gc65bg y38jKCmxH9cYmvKxVWxIJON4X2mVa5dHb1zAph+GIVe9lewEfjsG5b10k4MNynCYolR1 u21jZ0dPaBnozk+ByEBnwerDce6JEvaTmLxVxNS70Xzc6pcnhkHXESRNuKwIQ5NKHGPM 6+ZA==
X-Gm-Message-State: AGi0PubdqvSUMwf9kbF6SdUND91MKqOH+hgc7fTlNVl8RiVSomPtoPaB Yn3oUM9B2iG2VMn44c2O6ATxIAocBZyc24zT+PZWEzQQcDM=
X-Google-Smtp-Source: APiQypLvcHzrdEQVdsvORWek+IsCWXgtvU05EpvmvmTYb8GUbOGTfiH5sWDHhlEoEafkpZdJc4cmrD25r5CFBAwUeTs=
X-Received: by 2002:a19:f70f:: with SMTP id z15mr2791412lfe.53.1588711424698; Tue, 05 May 2020 13:43:44 -0700 (PDT)
MIME-Version: 1.0
References: <cellar-wg/matroska-specification/issues/376@github.com> <24830.1588522227@localhost>
In-Reply-To: <24830.1588522227@localhost>
From: Spencer Dawkins at IETF <spencerdawkins.ietf@gmail.com>
Date: Tue, 5 May 2020 15:43:17 -0500
Message-ID: <CAKKJt-dxwwWpOHweAFDOEnAvpOi5dnT1c8rtd-NwZYp-AU+Obg@mail.gmail.com>
To: Michael Richardson <mcr@sandelman.ca>
Cc: Codec Encoding for LossLess Archiving and Realtime transmission <cellar@ietf.org>
Content-Type: multipart/alternative; boundary="0000000000001eb07c05a4ecb498"
Archived-At: <https://mailarchive.ietf.org/arch/msg/cellar/YUhSc0CjQ-_-tv71wQtTmNZrs38>
Subject: Re: [Cellar] compression libraries in IETF specifications (was Re: [cellar-wg/matroska-specification] deprecate LZO compression (#376))
X-BeenThere: cellar@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Codec Encoding for LossLess Archiving and Realtime transmission <cellar.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/cellar>, <mailto:cellar-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cellar/>
List-Post: <mailto:cellar@ietf.org>
List-Help: <mailto:cellar-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/cellar>, <mailto:cellar-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 05 May 2020 20:43:49 -0000

Hi, Michael,

On Sun, May 3, 2020 at 11:10 AM Michael Richardson <mcr@sandelman.ca> wrote:

>
> Hi, I wanted to bring this item to the list's attention.
>
> Spencer: I seem to remember that the IETF has had compression IPR issues in
> specifications in the past.  IPCOMP comes to mind, but I am guessing that
> other RTP/SIP groups have gone through this..


I had forgotten IPCOMP, but was remembering RoHC and RoHCv2. I think you're
remembering SIGCOMP.

The complete list of disclosures that mention "compression" is at
https://datatracker.ietf.org/ipr/search/?draft=&rfc=&doctitle=compression&submit=doctitle&group=&holder=&iprtitle=&patent=


> Is there possibly an IESG
> statement that might help us here?
>

I believe the only relevant IESG statement on licensing for repositories is
to make it clear that they need to include standard boilerplate (
https://www.ietf.org/about/groups/iesg/statements/open-source-repositories-license/
).

>
> But, first, the software license on the libraries is not entirely germane.
> An IETF specification can well contain algorithms which are currently only
> available in software licensed in ways that might not be useable by all
> implementers.
> Someone just will have to write a new library with a license that suits
> them!
>
> What matters if that the specification (and any related patents!) is
> available to all.  The IETF's minimum IPR requirement is generally RAND,
> which is often not good enough for open source, which requires RF.
> The decision as to which IPR claims to tolerate is a WG-level decision,
> with
> the proviso that the WG (and IETF) is informed about the issue.
>
> So even in the absense of the jackoalan and zivillian LZO implementations,
> the WG should be free to standardize LZO, if it the IPR claims on the
> algorithm itself are well understood.
>

This is exactly right. The IESG cares that the working group made the
decision to request publication on a document without being "submarined" by
undisclosed patents or patients with vague or confusing licensing
provisions. If we have our eyes open, we'll be fine.


> {In the compression space, my understanding is that there have been formats
> where there were no IPR claims on decompressors, but there was significant
> IPR-encumbered innovation on doing compression.  An encoder might need a
> specific license in order to get 70% compression rather than just 40%, but
> this had no affect on the decoder.  But, I don't recall the specific
> situations where this occured.  Maybe back in the old ZOO/ARC/ZIP days...}
>

I have the same vague memories (perhaps I'm remembering that licensing was
different, rather than decompressors hat no known IPR, but that's pretty
close).

I hope this helps!

Spencer


> Steve Lhomme <notifications@github.com> wrote:
>     > Matroska Blocks can be compressed with zlib, bz2 and lzo compression.
>
>     > zlib and bz2 have decompression libraries available under permisssive
>     > licenses (BSD like for bz2 and OSI approved, GPL compatible, non
>     > copyleft for zlib).
>
>     > lzo doesn't have such a permissive library. The [official
>     > library](https://www.oberhumer.com/opensource/lzo/) is GPLv2, there
> is
>     > also decompression code that doesn't rely on this library [in
>     > FFmpeg](https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/lzo.c
> )
>     > which is LGPLv2 which means grabbing the whole libavutil to compile.
>
>     > This license is not permissive enough to be used in many closed
> source
>     > products where they don't want to touch any GPL-based license. That
>     > means a file that is created with LZO in a GPL/LGPL software may not
> be
>     > read universally because of licensing issues. I think we should avoid
>     > it and mark LZO as deprecated.
>
>     > The LZO compression advantage over others is that it is [very fast to
>     > compress/decompress](
> https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO).
> But
>     > the compression ratio is not good. That is good for realtime
>     > transmission but bad for long term storage. In the end automatic
> tools
>     > that test each compression (like mkclean) will never pick it anyway.
>
> ....
>     > I found 2 other options with better licenses:
>
>     > - a [C++ version in MIT license](https://github.com/jackoalan/lzokay)
> with compression
>     > - a [C# version in MIT license](https://github.com/zivillian/lzo.net)
> without decompression
>
>     > They are both based on the [Linux Kernel documentation for
>     > LZO](
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/lzo.txt
> ).
>
>     > In libmatroska2 I've been using minilzo GPLv2 but I may try to port
> one
>     > of these in C so I can use it (or simply write a C wrapper around the
>     > C++ one). That means there would be an alternative that is MIT
> license,
>     > so freely used by anyone.
>