IETF Logo Mail Archive

Re: [codec] draft test and processing plan for the IETF Codec

Anisse Taleb <anisse.taleb@huawei.com> Mon, 18 April 2011 23:37 UTC

Return-Path: <anisse.taleb@huawei.com>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 53F7CE07F4 for <codec@ietfc.amsl.com>; Mon, 18 Apr 2011 16:37:26 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.291
X-Spam-Level:
X-Spam-Status: No, score=-6.291 tagged_above=-999 required=5 tests=[AWL=0.308, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l1Fot4mK6bqW for <codec@ietfc.amsl.com>; Mon, 18 Apr 2011 16:37:25 -0700 (PDT)
Received: from lhrga04-in.huawei.com (lhrga04-in.huawei.com [195.33.106.149]) by ietfc.amsl.com (Postfix) with ESMTP id 00CA9E077D for <codec@ietf.org>; Mon, 18 Apr 2011 16:37:25 -0700 (PDT)
Received: from huawei.com (localhost [127.0.0.1]) by lhrga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LJV009Y4GABP0@lhrga04-in.huawei.com> for codec@ietf.org; Tue, 19 Apr 2011 00:37:24 +0100 (BST)
Received: from LHREML201-EDG.china.huawei.com ([172.18.7.118]) by lhrga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPS id <0LJV00MBKGABWY@lhrga04-in.huawei.com> for codec@ietf.org; Tue, 19 Apr 2011 00:37:23 +0100 (BST)
Received: from LHREML402-HUB.china.huawei.com (10.201.5.31) by LHREML201-EDG.china.huawei.com (172.18.7.188) with Microsoft SMTP Server (TLS) id 14.1.270.1; Tue, 19 Apr 2011 00:37:19 +0100
Received: from LHREML503-MBX.china.huawei.com ([fe80::f93f:958b:5b06:4f36]) by LHREML402-HUB.china.huawei.com ([::1]) with mapi id 14.01.0270.001; Tue, 19 Apr 2011 00:37:22 +0100
Date: Mon, 18 Apr 2011 23:37:22 +0000
From: Anisse Taleb <anisse.taleb@huawei.com>
In-reply-to: <4DA6F158.7070103@octasic.com>
X-Originating-IP: [10.200.217.213]
To: Jean-Marc Valin <jean-marc.valin@octasic.com>, Paul Coverdale <coverdale@sympatico.ca>
Message-id: <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC8B9A@LHREML503-MBX.china.huawei.com>
MIME-version: 1.0
Content-type: text/plain; charset="us-ascii"
Content-language: en-US
Content-transfer-encoding: 7bit
Accept-Language: en-GB, en-US
Thread-topic: [codec] draft test and processing plan for the IETF Codec
Thread-index: AQHL/iGQx3IjQEUIRkKBz9iltnmafA==
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
References: <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC684E@LHREML503-MBX.china.huawei.com> <BCB3F026FAC4C145A4A3330806FEFDA93BA8B6463D@EMBX01-HQ.jnpr.net> <BLU0-SMTP463B56C50578E4BB6938BBD0AD0@phx.gbl> <4DA6F158.7070103@octasic.com>
Cc: "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 23:37:26 -0000

JM, Greg, Paul,
[taking emails in chronological order was ill advised :-)]

I do not disagree with the statistical pitfalls you mention. As Paul stated and also what I wrote in a direct reply to this, there is no single uber-requirement to be passed by the codec, rather a vector of requirements that summarize the performance of the codec compared to other codecs. These have to be analyzed and discussed one by one.

Kind regards,
/Anisse  

> -----Original Message-----
> From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] On Behalf Of
> Jean-Marc Valin
> Sent: Thursday, April 14, 2011 3:07 PM
> To: Paul Coverdale
> Cc: codec@ietf.org
> Subject: Re: [codec] draft test and processing plan for the IETF Codec
> 
> > I don't think the situation is as dire as you make out. Your analysis
> > assumes that all requirements are completely independent. This is not the
> > case, in many cases if you meet one requirement you are likely to meet
> > others of the same kind (eg performance as a function of bit rate).
> >
> > But in any case, the statistical analysis procedure outlined in the test
> > plan doesn't assume that every requirement must be met with absolute
> > certainty, it allows for a confidence interval.
> 
> This is exactly what Greg is considering in his analysis. He's starting
> from the assumption that the codec really meets *all* 162 requirements.
> Consider just the NWT requirements: if we were truly no worse than the
> reference codec, then with 87 tests against a 95% confidence interval, we
> would be expected to fail about 4 tests just by random chance. Considering
> both NWT and BT requirements, the odds of passing Anisse's proposed test
> plan given the assumptions above are 4.1483e-33. See http://xkcd.com/882/
> for a more rigorous analysis.
> 
> Cheers,
> 
> 	Jean-Marc
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec

v2.14.0 | Report a Bug | By Email