[codec] Non-bitstream impacting encoder changes and testing

Gregory Maxwell <gmaxwell@juniper.net> Thu, 14 April 2011 07:25 UTC

Return-Path: <gmaxwell@juniper.net>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost []) by ietfc.amsl.com (Postfix) with ESMTP id 3EF54E0872 for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 00:25:59 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.599
X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([]) by localhost (ietfc.amsl.com []) (amavisd-new, port 10024) with ESMTP id fvDpgHeFOcOk for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 00:25:58 -0700 (PDT)
Received: from exprod7og112.obsmtp.com (exprod7og112.obsmtp.com []) by ietfc.amsl.com (Postfix) with ESMTP id 6263BE0871 for <codec@ietf.org>; Thu, 14 Apr 2011 00:25:58 -0700 (PDT)
Received: from P-EMHUB02-HQ.jnpr.net ([]) (using TLSv1) by exprod7ob112.postini.com ([]) with SMTP ID DSNKTaahhRrysi9td82vzPF8wsoCSCtZ0WAE@postini.com; Thu, 14 Apr 2011 00:25:58 PDT
Received: from EMBX01-HQ.jnpr.net ([fe80::c821:7c81:f21f:8bc7]) by P-EMHUB02-HQ.jnpr.net ([fe80::88f9:77fd:dfc:4d51%11]) with mapi; Thu, 14 Apr 2011 00:21:20 -0700
From: Gregory Maxwell <gmaxwell@juniper.net>
To: "codec@ietf.org" <codec@ietf.org>
Date: Thu, 14 Apr 2011 00:21:19 -0700
Thread-Topic: Non-bitstream impacting encoder changes and testing
Thread-Index: AQHL+nMplLVKOYZizkO3qwmcViO+tA==
Message-ID: <BCB3F026FAC4C145A4A3330806FEFDA93BA8B64640@EMBX01-HQ.jnpr.net>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: [codec] Non-bitstream impacting encoder changes and testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 07:25:59 -0000

I have some fairly minor patches queued up for which improve some
of the encoder signal analysis and should produce some minor quality
improvements.  I have more development testing to do before I consider
them done, but I've been dragging my feet because I'm not quite sure
how this sort of change will interact with testing.

In Opus the encoder behavior is non-normative— precisely so the
format could continue to enjoy encoder-side improvements (like we've
seen with Speex, Vorbis, Theora, and MPEG codecs). But I'm concerned
that some people might consider the testing which has already been
performed to be invalidated by this sort of improvement.

Rather than fall into a pit over this after the fact, I thought it would
be better to ask if WG participants have any views on this subject?

My view is that changes which do not change the bitstream/decoder should
not be considered to invalidate any testing. Even if a change is later
discovered to degrade the performance of the codec, an implementer is
free to use an older encoder, manually back out that change, etc. So,
basically, every test is at least evidence of what can be achieved with
the format even if the encoder is changed later. Good testing practice
should identify the versions used in any case.

The obvious alternatives I see are to simply not improve the encoder
right now, or to work on a separate encoder implementation (a fork of
the reference encoder). I don't either of these are especially good.