Re: [codec] draft test and processing plan for the IETF Codec

Anisse Taleb <anisse.taleb@huawei.com> Mon, 18 April 2011 23:29 UTC

Return-Path: <anisse.taleb@huawei.com>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id DF76BE069A for <codec@ietfc.amsl.com>; Mon, 18 Apr 2011 16:29:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.266
X-Spam-Level:
X-Spam-Status: No, score=-6.266 tagged_above=-999 required=5 tests=[AWL=0.333, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4bx6pVaPfLDY for <codec@ietfc.amsl.com>; Mon, 18 Apr 2011 16:29:41 -0700 (PDT)
Received: from lhrga04-in.huawei.com (lhrga04-in.huawei.com [195.33.106.149]) by ietfc.amsl.com (Postfix) with ESMTP id 23AF4E067C for <codec@ietf.org>; Mon, 18 Apr 2011 16:29:41 -0700 (PDT)
Received: from huawei.com (localhost [127.0.0.1]) by lhrga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LJV009QLFXDP0@lhrga04-in.huawei.com> for codec@ietf.org; Tue, 19 Apr 2011 00:29:38 +0100 (BST)
Received: from LHREML202-EDG.china.huawei.com ([172.18.7.118]) by lhrga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPS id <0LJV00M13FXDWY@lhrga04-in.huawei.com> for codec@ietf.org; Tue, 19 Apr 2011 00:29:37 +0100 (BST)
Received: from LHREML402-HUB.china.huawei.com (10.201.5.31) by LHREML202-EDG.china.huawei.com (172.18.7.189) with Microsoft SMTP Server (TLS) id 14.1.270.1; Tue, 19 Apr 2011 00:29:33 +0100
Received: from LHREML503-MBX.china.huawei.com ([fe80::f93f:958b:5b06:4f36]) by LHREML402-HUB.china.huawei.com ([::1]) with mapi id 14.01.0270.001; Tue, 19 Apr 2011 00:29:36 +0100
Date: Mon, 18 Apr 2011 23:29:35 +0000
From: Anisse Taleb <anisse.taleb@huawei.com>
In-reply-to: <683635837.116272.1302766059230.JavaMail.root@lu2-zimbra>
X-Originating-IP: [10.200.217.213]
To: Koen Vos <koen.vos@skype.net>
Message-id: <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC8B8E@LHREML503-MBX.china.huawei.com>
MIME-version: 1.0
Content-type: text/plain; charset="utf-8"
Content-language: en-US
Content-transfer-encoding: 7bit
Accept-Language: en-GB, en-US
Thread-topic: [codec] draft test and processing plan for the IETF Codec
Thread-index: AQHL+jZvkJ8ngzsGfkWZVsxSDmWuIZRci0kAgABaZICAB1rFAA==
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
References: <4DA65618.9060302@octasic.com> <683635837.116272.1302766059230.JavaMail.root@lu2-zimbra>
Cc: "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 23:29:42 -0000

Dear Koen,
Answers inline.

> But my problem with your 24 "new" requirements is not just procedural.
> If you read the charter of this WG, you'll see that "beating most existing
> codecs" was never a goal.  
> The strongest wording about performance is that
> the codec shall enable "high-quality audio services on the Internet".

The goal of this working group is to ensure the existence of a single
high-quality audio codec that is optimized for use over the Internet and
that can be widely implemented and easily distributed among application
developers, service operators, and end users. 

It all depends on where you put the bar for high-quality audio, and how you evaluate that a codec is optimized for internet applications. High-quality is very subjective, if I am used to the quality of "existing codecs" I would like to understand how Opus provides high-quality audio relative to these codecs. 

> Instead of spending countless hours on testing against arbitrary
> requirements, what better way to verify that we've reached that goal than
> deploying it in actual Internet applications?

Because in Engineering (as well as Business), goals have to be measurable and measured. If your goal is to deploy a codec for an internet application, why do you need IETF? go ahead, compile, link and deploy! 

Most of the codecs cited in the document are "real" and widely deployed. I do apologize for the inconsistencies of some of the test points, I did not claim the test plan to be perfect especially given the time it took to be produced, the goal was to initiate the discussion, if there are codecs you do not wish to see compared to Opus feel free to voice your disagreement.

> 
> Early adopters will realize there is always a risk that the bitstream will
> have to change, no matter how much we test.  They'll also know that the
> likelihood of such changes decreases fairly quickly with time and adoption.
> 

I find it distressing that you mention that the bitstream would change after the codec has been adopted and published by the IETF. While this is something you can do with a proprietary codec, I see no point in standardizing a codec with the a priori knowledge that it will change in the future...

Please define Standard?


> Stephen Botzko recently pointed out that codecs implemented in hardware are
> difficult to upgrade(*).  While true, it's no argument against deploying
> sooner rather than later, because no sensible hardware manufacturer is
> going
> to put OPUS in hardware before it has proven itself in the market place.

If that is true, then why are we here? Why not deploy the codec, gain acceptance in the market place and reach a stable version that is ready for hardware implementation, and then let's standardize it. 
> 
> The testing done to date by developers and independent parties shows we do
> indeed have "running code."  But the proof of the pudding is in the eating,
> and voluntary deployment in real-world (software) applications is the right
> next step.
>

Not when we all have to eat the same pudding and we know it contains radioactive lead.

Kind regards,
/Anisse