Re: [codec] comparitive quality testing

Cullen Jennings <fluffy@cisco.com> Fri, 15 April 2011 00:55 UTC

Return-Path: <fluffy@cisco.com>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 6E201E0877 for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 17:55:35 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.581
X-Spam-Level:
X-Spam-Status: No, score=-110.581 tagged_above=-999 required=5 tests=[AWL=0.018, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JtzkqmQ9aL+m for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 17:55:34 -0700 (PDT)
Received: from sj-iport-2.cisco.com (sj-iport-2.cisco.com [171.71.176.71]) by ietfc.amsl.com (Postfix) with ESMTP id EC4DDE0857 for <codec@ietf.org>; Thu, 14 Apr 2011 17:55:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=fluffy@cisco.com; l=1465; q=dns/txt; s=iport; t=1302828934; x=1304038534; h=subject:mime-version:from:in-reply-to:date:cc: content-transfer-encoding:message-id:references:to; bh=revZNkZQPe1G0hduFyTsMDEITYXZ0NHOf0EHXpUfBoc=; b=P25bNINdnI7P808t6JvwAmRhhRHtnTQcm3kypTjsBrWRNBHcVoD3gbaY l2rGOk8BD2wJIuU7119efrzMZ0fATA8i385ep2BWcaoRFxrvFUO0++t9s 65ruZGzQ4rYmwTA+VAQCTF0XbxieoNXand81h/pKieVgBs49m5puMY93B M=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AvsEAEGXp02rRDoI/2dsb2JhbAClf3eIb55CnQ6FbgSFWogVg3M
X-IronPort-AV: E=Sophos;i="4.64,214,1301875200"; d="scan'208";a="337825253"
Received: from mtv-core-3.cisco.com ([171.68.58.8]) by sj-iport-2.cisco.com with ESMTP; 15 Apr 2011 00:55:33 +0000
Received: from [192.168.4.100] (rcdn-fluffy-8712.cisco.com [10.99.9.19]) by mtv-core-3.cisco.com (8.14.3/8.14.3) with ESMTP id p3F0sEwl016196; Fri, 15 Apr 2011 00:55:32 GMT
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset="us-ascii"
From: Cullen Jennings <fluffy@cisco.com>
In-Reply-To: <BANLkTimE6EzGY76Lm+-wtWtRTQgOjqhAEw@mail.gmail.com>
Date: Thu, 14 Apr 2011 18:55:32 -0600
Content-Transfer-Encoding: quoted-printable
Message-Id: <DB4DD197-97D2-492C-B896-A720347D3533@cisco.com>
References: <BCB3F026FAC4C145A4A3330806FEFDA93BA8B64643@EMBX01-HQ.jnpr.net> <BANLkTimE6EzGY76Lm+-wtWtRTQgOjqhAEw@mail.gmail.com>
To: Roman Shpount <roman@telurix.com>
X-Mailer: Apple Mail (2.1084)
Cc: codec@ietf.org
Subject: Re: [codec] comparitive quality testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 00:55:35 -0000

On Apr 14, 2011, at 12:21 PM, Roman Shpount wrote:

> I think part of the confusion comes from the fact that there are two purposes for the comparative testing. One is to validate that the codec meets the WG requirements. Another is to show how new codec compares to the industry dominant codecs. For me, the second goal is more important then the first one. I think if we care about the adoption of Opus, we should consider making the comparative test results a deliverable for the working group. It is very hard for a real company in the open market to justify doing something, like adapting a new codec without a compelling reason. Knowing how this codec compares to other existing codecs is a big part of providing such a reason. If we look at the tests from this point of view, we need to see how Opus compares to G.729 and AMR in narrow band, and AMR-WB and G.722 in wideband, Since there are no existing deployments of a meaningful size (apart from a closed proprietary systems, like Skype) for UWB and FB, we can compare Opus with industry leaders, such as G.719. 
> 
> One can argue that we should also compare Opus with patent free codecs, which adds iLBC and Speex to the list, but I personally see this as less of a requirement. iLBC never managed to get market traction outside of the open source world,

Sort of curios how you came to that conclusion about iLBC's market traction?