Re: [codec] Non-bitstream impacting encoder changes and testing

Anisse Taleb <anisse.taleb@huawei.com> Mon, 18 April 2011 22:46 UTC

Return-Path: <anisse.taleb@huawei.com>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 2DDD8E0846 for <codec@ietfc.amsl.com>; Mon, 18 Apr 2011 15:46:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.235
X-Spam-Level:
X-Spam-Status: No, score=-6.235 tagged_above=-999 required=5 tests=[AWL=0.364, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m8u5TysvLe-Z for <codec@ietfc.amsl.com>; Mon, 18 Apr 2011 15:46:17 -0700 (PDT)
Received: from lhrga02-in.huawei.com (lhrga02-in.huawei.com [195.33.106.143]) by ietfc.amsl.com (Postfix) with ESMTP id 73C17E083C for <codec@ietf.org>; Mon, 18 Apr 2011 15:46:17 -0700 (PDT)
Received: from huawei.com (lhrga02-in [172.18.7.45]) by lhrga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LJV003GHDX4QA@lhrga02-in.huawei.com> for codec@ietf.org; Mon, 18 Apr 2011 23:46:16 +0100 (BST)
Received: from LHREML202-EDG.china.huawei.com ([172.18.7.118]) by lhrga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPS id <0LJV00EI5DX2Z8@lhrga02-in.huawei.com> for codec@ietf.org; Mon, 18 Apr 2011 23:46:14 +0100 (BST)
Received: from LHREML402-HUB.china.huawei.com (10.201.5.31) by LHREML202-EDG.china.huawei.com (172.18.7.189) with Microsoft SMTP Server (TLS) id 14.1.270.1; Mon, 18 Apr 2011 23:46:10 +0100
Received: from LHREML503-MBX.china.huawei.com ([fe80::f93f:958b:5b06:4f36]) by LHREML402-HUB.china.huawei.com ([::1]) with mapi id 14.01.0270.001; Mon, 18 Apr 2011 23:46:13 +0100
Date: Mon, 18 Apr 2011 22:46:12 +0000
From: Anisse Taleb <anisse.taleb@huawei.com>
In-reply-to: <BCB3F026FAC4C145A4A3330806FEFDA93BA8B64640@EMBX01-HQ.jnpr.net>
X-Originating-IP: [10.200.217.213]
To: Gregory Maxwell <gmaxwell@juniper.net>, "codec@ietf.org" <codec@ietf.org>
Message-id: <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC8B36@LHREML503-MBX.china.huawei.com>
MIME-version: 1.0
Content-type: text/plain; charset="us-ascii"
Content-language: en-US
Content-transfer-encoding: 7bit
Accept-Language: en-GB, en-US
Thread-topic: Non-bitstream impacting encoder changes and testing
Thread-index: AQHL+nMplLVKOYZizkO3qwmcViO+tJRkPlsQ
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
References: <BCB3F026FAC4C145A4A3330806FEFDA93BA8B64640@EMBX01-HQ.jnpr.net>
Subject: Re: [codec] Non-bitstream impacting encoder changes and testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 22:46:18 -0000

Dear Greg,

I personally have no problems with this, any improvement to the encoder is welcome. When It comes to adopting the codec as a standard, the code has to be frozen at some point such that formal testing can start on an agreed version of the codec. Any other tests performed with intermediate version are of course valuable, but will not reflect the quality of the codec as published by the IETF. 

Kind regards,
/Anisse 

> 
> 
> I have some fairly minor patches queued up for which improve some
> of the encoder signal analysis and should produce some minor quality
> improvements.  I have more development testing to do before I consider
> them done, but I've been dragging my feet because I'm not quite sure
> how this sort of change will interact with testing.
> 
> In Opus the encoder behavior is non-normative- precisely so the
> format could continue to enjoy encoder-side improvements (like we've
> seen with Speex, Vorbis, Theora, and MPEG codecs). But I'm concerned
> that some people might consider the testing which has already been
> performed to be invalidated by this sort of improvement.
> 
> Rather than fall into a pit over this after the fact, I thought it would
> be better to ask if WG participants have any views on this subject?
> 
> My view is that changes which do not change the bitstream/decoder should
> not be considered to invalidate any testing. Even if a change is later
> discovered to degrade the performance of the codec, an implementer is
> free to use an older encoder, manually back out that change, etc. So,
> basically, every test is at least evidence of what can be achieved with
> the format even if the encoder is changed later. Good testing practice
> should identify the versions used in any case.
> 
> The obvious alternatives I see are to simply not improve the encoder
> right now, or to work on a separate encoder implementation (a fork of
> the reference encoder). I don't either of these are especially good.
> 
> 
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec