Re: [codec] comparitive quality testing

Cullen Jennings <fluffy@cisco.com> Fri, 15 April 2011 21:13 UTC

Return-Path: <fluffy@cisco.com>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 38079E07D2 for <codec@ietfc.amsl.com>; Fri, 15 Apr 2011 14:13:51 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.583
X-Spam-Level:
X-Spam-Status: No, score=-110.583 tagged_above=-999 required=5 tests=[AWL=0.016, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id clozpJTB5Jbv for <codec@ietfc.amsl.com>; Fri, 15 Apr 2011 14:13:47 -0700 (PDT)
Received: from sj-iport-6.cisco.com (sj-iport-6.cisco.com [171.71.176.117]) by ietfc.amsl.com (Postfix) with ESMTP id D1E99E0801 for <codec@ietf.org>; Fri, 15 Apr 2011 14:13:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=fluffy@cisco.com; l=3414; q=dns/txt; s=iport; t=1302902026; x=1304111626; h=subject:mime-version:from:in-reply-to:date:cc: content-transfer-encoding:message-id:references:to; bh=RVLC18E3JkuxgCftFMu5kWzXtFyapRf5w/EUmdX0YFc=; b=Sm893xB8j0kWbyx4CJsAHLABdL/9BpY9ZyxpCaUNbUDTUER7q95fp9b2 tS3b5HHjjttxz7MUjnaveT5tZGvlm+ldomN7RPmIbvHodzmVmI1i/kDo6 sov56OakhBURxPPz3QTGJB+Df5Onji/m3YcKBLiYz2G0sH2pZhkXVTMsp o=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AvsEAEu0qE2rRDoJ/2dsb2JhbACmCXeIb58UnQSFbgSFYIgYg3Q
X-IronPort-AV: E=Sophos;i="4.64,221,1301875200"; d="scan'208";a="682507306"
Received: from mtv-core-4.cisco.com ([171.68.58.9]) by sj-iport-6.cisco.com with ESMTP; 15 Apr 2011 21:13:46 +0000
Received: from [192.168.4.100] (rcdn-fluffy-8712.cisco.com [10.99.9.19]) by mtv-core-4.cisco.com (8.14.3/8.14.3) with ESMTP id p3FLDjs4007021; Fri, 15 Apr 2011 21:13:45 GMT
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset="us-ascii"
From: Cullen Jennings <fluffy@cisco.com>
In-Reply-To: <BANLkTin53-nu9XzNi76KRV6npcWHUx4ptA@mail.gmail.com>
Date: Fri, 15 Apr 2011 15:13:44 -0600
Content-Transfer-Encoding: quoted-printable
Message-Id: <F195167E-3542-439B-A7F9-01C27580B207@cisco.com>
References: <BCB3F026FAC4C145A4A3330806FEFDA93BA8B64643@EMBX01-HQ.jnpr.net><BANLkTimE6EzGY76Lm+-wtWtRTQgOjqhAEw@mail.gmail.com><DB4DD197-97D2-492C-B896-A720347D3533@cisco.com> <BANLkTin53-nu9XzNi76KRV6npcWHUx4ptA@mail.gmail.com>
To: Roman Shpount <roman@telurix.com>
X-Mailer: Apple Mail (2.1084)
Cc: codec@ietf.org
Subject: Re: [codec] comparitive quality testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 21:13:51 -0000

Just a side note about iLBC... Cisco ships iLBC on most the voice platform it supports - this covers a wide range of optimized fixed point implementations. It is a used by many of customers. Some service provider terminate far more PSTN traffic with iLBC than the do with G.711 and G.729 combined.  Many other vendors also have widespread support for iLBC. Much of the data about how many minutes of each codec is considered confidential but if you have access to some of that data, have a look. 

Cullen <with my cisco employee hat on >


On Apr 14, 2011, at 7:20 PM, Roman Shpount wrote:

> I know that iLBC was picked by PacketCable and is supported by quite a few gateways. On the other hand, iLBC standard itself looks as if the codec was contributed by one company with almost nobody investing any time in its actual standardization. Quality comparisons and resource requirements seemed to be coming from the same company and accepted on their say so. Most of the deployed versions of iLBC are a floating point reference implementation, which is coded in a manner as inefficient as a codec implementation can possible be. There are no open source fixed-point implementations, even though creating one is trivial. Reference vectors consist of a single audio file recorded by a single voice. Encoder, despite being a bit exact reference, produces different results based on optimization options or compiler. To summarize, I assumed that there is no interest in this code since there are a lot of problems with it that would have been easy to fix if anybody bothered. It looks like a proof of concept and not a finished product. Finally, amount of voice traffic carried by iLBC is only a small fraction of G.729.
> _____________
> Roman Shpount
> 
> 
> On Thu, Apr 14, 2011 at 8:55 PM, Cullen Jennings <fluffy@cisco.com> wrote:
> 
> On Apr 14, 2011, at 12:21 PM, Roman Shpount wrote:
> 
> > I think part of the confusion comes from the fact that there are two purposes for the comparative testing. One is to validate that the codec meets the WG requirements. Another is to show how new codec compares to the industry dominant codecs. For me, the second goal is more important then the first one. I think if we care about the adoption of Opus, we should consider making the comparative test results a deliverable for the working group. It is very hard for a real company in the open market to justify doing something, like adapting a new codec without a compelling reason. Knowing how this codec compares to other existing codecs is a big part of providing such a reason. If we look at the tests from this point of view, we need to see how Opus compares to G.729 and AMR in narrow band, and AMR-WB and G.722 in wideband, Since there are no existing deployments of a meaningful size (apart from a closed proprietary systems, like Skype) for UWB and FB, we can compare Opus with industry leaders, such as G.719.
> >
> > One can argue that we should also compare Opus with patent free codecs, which adds iLBC and Speex to the list, but I personally see this as less of a requirement. iLBC never managed to get market traction outside of the open source world,
> 
> Sort of curios how you came to that conclusion about iLBC's market traction?
> 
>