Re: [codec] comparitive quality testing

Roman Shpount <roman@telurix.com> Fri, 15 April 2011 01:20 UTC

Return-Path: <roman@telurix.com>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 3D538E0679 for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 18:20:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.976
X-Spam-Level:
X-Spam-Status: No, score=-2.976 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bIwMJe3K1ABC for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 18:20:40 -0700 (PDT)
Received: from mail-ew0-f44.google.com (mail-ew0-f44.google.com [209.85.215.44]) by ietfc.amsl.com (Postfix) with ESMTP id F390BE065C for <codec@ietf.org>; Thu, 14 Apr 2011 18:20:39 -0700 (PDT)
Received: by ewy19 with SMTP id 19so790058ewy.31 for <codec@ietf.org>; Thu, 14 Apr 2011 18:20:39 -0700 (PDT)
Received: by 10.14.123.137 with SMTP id v9mr481704eeh.216.1302830439039; Thu, 14 Apr 2011 18:20:39 -0700 (PDT)
Received: from mail-ey0-f172.google.com (mail-ey0-f172.google.com [209.85.215.172]) by mx.google.com with ESMTPS id u1sm1554139eeh.20.2011.04.14.18.20.37 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 14 Apr 2011 18:20:38 -0700 (PDT)
Received: by eye13 with SMTP id 13so793456eye.31 for <codec@ietf.org>; Thu, 14 Apr 2011 18:20:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.213.104.133 with SMTP id p5mr2730133ebo.109.1302830437395; Thu, 14 Apr 2011 18:20:37 -0700 (PDT)
Received: by 10.213.4.14 with HTTP; Thu, 14 Apr 2011 18:20:37 -0700 (PDT)
In-Reply-To: <DB4DD197-97D2-492C-B896-A720347D3533@cisco.com>
References: <BCB3F026FAC4C145A4A3330806FEFDA93BA8B64643@EMBX01-HQ.jnpr.net> <BANLkTimE6EzGY76Lm+-wtWtRTQgOjqhAEw@mail.gmail.com> <DB4DD197-97D2-492C-B896-A720347D3533@cisco.com>
Date: Thu, 14 Apr 2011 21:20:37 -0400
Message-ID: <BANLkTin53-nu9XzNi76KRV6npcWHUx4ptA@mail.gmail.com>
From: Roman Shpount <roman@telurix.com>
To: Cullen Jennings <fluffy@cisco.com>
Content-Type: multipart/alternative; boundary="00151748dd7c6c702104a0ead923"
Cc: codec@ietf.org
Subject: Re: [codec] comparitive quality testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 01:20:41 -0000

I know that iLBC was picked by PacketCable and is supported by quite a few
gateways. On the other hand, iLBC standard itself looks as if the codec was
contributed by one company with almost nobody investing any time in its
actual standardization. Quality comparisons and resource requirements seemed
to be coming from the same company and accepted on their say so. Most of the
deployed versions of iLBC are a floating point reference implementation,
which is coded in a manner as inefficient as a codec implementation can
possible be. There are no open source fixed-point implementations, even
though creating one is trivial. Reference vectors consist of a single audio
file recorded by a single voice. Encoder, despite being a bit exact
reference, produces different results based on optimization options or
compiler. To summarize, I assumed that there is no interest in this code
since there are a lot of problems with it that would have been easy to fix
if anybody bothered. It looks like a proof of concept and not a finished
product. Finally, amount of voice traffic carried by iLBC is only a small
fraction of G.729.
_____________
Roman Shpount


On Thu, Apr 14, 2011 at 8:55 PM, Cullen Jennings <fluffy@cisco.com> wrote:

>
> On Apr 14, 2011, at 12:21 PM, Roman Shpount wrote:
>
> > I think part of the confusion comes from the fact that there are two
> purposes for the comparative testing. One is to validate that the codec
> meets the WG requirements. Another is to show how new codec compares to the
> industry dominant codecs. For me, the second goal is more important then the
> first one. I think if we care about the adoption of Opus, we should consider
> making the comparative test results a deliverable for the working group. It
> is very hard for a real company in the open market to justify doing
> something, like adapting a new codec without a compelling reason. Knowing
> how this codec compares to other existing codecs is a big part of providing
> such a reason. If we look at the tests from this point of view, we need to
> see how Opus compares to G.729 and AMR in narrow band, and AMR-WB and G.722
> in wideband, Since there are no existing deployments of a meaningful size
> (apart from a closed proprietary systems, like Skype) for UWB and FB, we can
> compare Opus with industry leaders, such as G.719.
> >
> > One can argue that we should also compare Opus with patent free codecs,
> which adds iLBC and Speex to the list, but I personally see this as less of
> a requirement. iLBC never managed to get market traction outside of the open
> source world,
>
> Sort of curios how you came to that conclusion about iLBC's market
> traction?
>
>