Re: [codec] comparitive quality testing

Roman Shpount <roman@telurix.com> Tue, 19 April 2011 23:08 UTC

Return-Path: <roman@telurix.com>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id DCB51E0822 for <codec@ietfc.amsl.com>; Tue, 19 Apr 2011 16:08:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.976
X-Spam-Level:
X-Spam-Status: No, score=-2.976 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AFKZ0gKA3y8z for <codec@ietfc.amsl.com>; Tue, 19 Apr 2011 16:08:37 -0700 (PDT)
Received: from mail-ey0-f172.google.com (mail-ey0-f172.google.com [209.85.215.172]) by ietfc.amsl.com (Postfix) with ESMTP id CD451E06F9 for <codec@ietf.org>; Tue, 19 Apr 2011 16:08:36 -0700 (PDT)
Received: by eye13 with SMTP id 13so62827eye.31 for <codec@ietf.org>; Tue, 19 Apr 2011 16:08:36 -0700 (PDT)
Received: by 10.14.137.201 with SMTP id y49mr2289983eei.244.1303254515399; Tue, 19 Apr 2011 16:08:35 -0700 (PDT)
Received: from mail-ew0-f44.google.com (mail-ew0-f44.google.com [209.85.215.44]) by mx.google.com with ESMTPS id m55sm243819eei.15.2011.04.19.16.08.34 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 19 Apr 2011 16:08:34 -0700 (PDT)
Received: by ewy19 with SMTP id 19so63294ewy.31 for <codec@ietf.org>; Tue, 19 Apr 2011 16:08:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.213.103.203 with SMTP id l11mr24032ebo.55.1303254513911; Tue, 19 Apr 2011 16:08:33 -0700 (PDT)
Received: by 10.213.98.83 with HTTP; Tue, 19 Apr 2011 16:08:33 -0700 (PDT)
In-Reply-To: <6261291B-0CB8-4719-910E-2ED7A7767D82@telio.no>
References: <BCB3F026FAC4C145A4A3330806FEFDA93BA8B64643@EMBX01-HQ.jnpr.net> <BANLkTimE6EzGY76Lm+-wtWtRTQgOjqhAEw@mail.gmail.com> <DB4DD197-97D2-492C-B896-A720347D3533@cisco.com> <BANLkTin53-nu9XzNi76KRV6npcWHUx4ptA@mail.gmail.com> <6261291B-0CB8-4719-910E-2ED7A7767D82@telio.no>
Date: Tue, 19 Apr 2011 19:08:33 -0400
Message-ID: <BANLkTik1DhJjUPYFtNL08gbSjqUB0KkLbQ@mail.gmail.com>
From: Roman Shpount <roman@telurix.com>
To: Alan Duric <Alan.Duric@telio.no>
Content-Type: multipart/alternative; boundary="001636d346405a86eb04a14d9637"
Cc: "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] comparitive quality testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 23:08:40 -0000

Dear Alan,

This discussion is a bit unrelated to the overall discussion in the group,
but I would still like to clarify some of my points.

As far as iLBC market penetration is concerned, what I know (and please
correct me if I am wrong), none of the major US VoIP carriers (AT&T,
Verizon, Qwest, Level 3, XO, GlobalCorssing) offer iLBC termination to their
customers. It is a choice of G.711 or G.729 A or B only. This is either
caused by the luck of market demand, or by them using Sonus gateways which,
as far as I know, do not support iLBC.

Once again, as far as I know Skype is not using iLBC for anything. It is
using Silk, G.711. G.729A, and ISAC.

As far as I know none of the cable VoIP operators in US use iLBC, it is all
G.711.

As far as hardware vendors are concerned, Cisco does support iLBC on its
gateways a quite a few (but I do not think all) of its phones. Sonus only
supports iLBC in its SBC, but not in gateways. Polycom, Snom, Aastra do not
seem to support iLBC in their phones.

Where iLBC seemed to thrive is in soft-phones and various types of desktop
based software VoIP products. In such products the benefits of not having a
per instance license are probably the greatest. iLBC is also universally
supported by open source projects such as Asterisk, Freeswitch, PJ-SIP,
OpenH323 descendents, and so on. iLBC is also present in Android and iOS.
What is interesting about all of those products (and I did check pretty much
all the open source projects I can find) that the iLBC implementation in
these products are essentially the reference code from the RFC. This code is
extremely inefficient. What I mean by this it consumes more then 10 times
the CPU resources of Voiceage or IPP optimized G.729A. With Freeswitch or
Asterisk you cannot get more the 50-100 channels of iLBC per modern dual CPU
server. On the other hand running over 500 channels of G.729A is fairly
normal. What was interesting for me that no one spent any time optimizing
any of the open source versions of the codec even though this is fairly
trivial. On the other hand, such high CPU requirements on server and mobile
platforms make iLBC almost unusable. If anyone on this list knows the open
source optimized or fixed point implementation, please point it out to me.

There is also a question of ILBC Freeware license. Based on IETF rules, all
the code released as a part of IETF RFC should be covered by BSD license.
The reasoning and applicability of iLBC Freeware license in this context is
unclear.

IPR disclosure for iLBC RFC was not updated when RFC was finalized and only
has provisional patent applications, not the final granted patent numbers.

Finally, I would like to switch to something that has direct applicability
to the current working group. iLBC claims to be the same computation
complexity as G.729A. From what I know, ILBC reference implementation is
about 3-4 times the complexity of G.729A. It is possible to create an iLBC
implementation of the same complexity as G.729A, but this will, from what
I've seen, cause the quality degradation similar to degradation between
G.729 and G.729A. I did not measure this quality degradation, but I know
that, for instance, reducing the number of computation points in LPC to LSF
conversion, to bring it inline with G.729A, will produce different encoded
audio. All the quality tests for iLBC, from what I've seen, were done using
the reference code. I assume (I have no good way of testing this), that the
optimized code in Cisco and other production gateways will produce different
encoded audio. I have never seen the quality test for this optimized code.
In case of testing codecs where implementation varies for the same
bitstream, it is very important to compare quality and computation
complexity of the same implementation. Otherwise you will get results that
are misleading.

Furthermore, reference code in the RFC, based on the compiler used and
optimization options will produce different encoded bitstream on the same
input audio. In some cases encoded bitsream will match the reference vector
provided on iLBCfreeware.org, but will produce different results on other
audio files. Some differences are expected in decoder, since it is a
floating point implementation, but I am not sure encoded audio differences
are expected according even to iLBC Freeware site.

If we plan to run comparison test with iLBC we need to specify which
implementation we are planning to use. Additionally we need to specify
platform, compiler version, and compilation settings. Finally, we need to
specify if enhancer should be used in decoder. If we will not do this, the
test input will not be identical across the test sites.
_____________
Roman Shpount


On Sat, Apr 16, 2011 at 6:07 AM, Alan Duric <Alan.Duric@telio.no> wrote:

> Dear Roman,
>
> despite the fact that I was not able to follow work of the group for quite
> some time, I was informed about Your post form colleagues and I needed to
> react, since there is quite a number of disinformations that You are placing
> (though not sure if by purpose or just by not knowing better)!
>
> Comments are inline
>
>
> On Apr 15, 2011, at 3:20 AM, Roman Shpount wrote:
>
> I know that iLBC was picked by PacketCable and is supported by quite a few
> gateways. On the other hand, iLBC standard itself looks as if the codec was
> contributed by one company with almost nobody investing any time in its
> actual standardization.
>
>
> A number of people that were involved in the verification were and are
> unarguably among the authorites in the speech coding world and have been
> heavily involved in the previous standardization efforts by number of
> standardization bodies. Their reputation is a way to big to endorse
> something what they do not fully stand behind. In the past a number of
> standards brought up by ITU, ETSI and TIA have been primarily or solely
> contributed by one company and it does not make those standards less
> valuable.
>
> Quality comparisons and resource requirements seemed to be coming from the
> same company and accepted on their say so.
>
>
> Please see above.As well, Dynastat (at that point of time one of the best
> independent labs) have done assessment of the codec in comparison to other
> existing standards. Results are available through IETF archive.
>
>
> Most of the deployed versions of iLBC are a floating point reference
> implementation, which is coded in a manner as inefficient as a codec
> implementation can possible be.
>
>
> Where do You get this information??? This is prime example of  "say so"!
> Information like this are confusing those that are following mailing list
> and I find it: either malicious or - You do not know much of VOIP
> deployments in general and thus should not be raising Your voice in front of
> this many people! CalbeLabs and Skype are terminating with iLBC more minutes
> towards PSTN then with any other codec combined. My company also happens to
> terminate billions of voice minutes traffic since 2004 and through very
> close cooperation with voice (or media) gateway manufacturers and by being
> advisor to a number of major carriers, I trust I have pretty good insight on
> implementations of various codecs and minutes that are carried with them.
>
> There are no open source fixed-point implementations, even though creating
> one is trivial. Reference vectors consist of a single audio file recorded by
> a single voice. Encoder, despite being a bit exact reference, produces
> different results based on optimization options or compiler. To summarize, I
> assumed that there is no interest in this code since there are a lot of
> problems with it that would have been easy to fix if anybody bothered. It
> looks like a proof of concept and not a finished product. Finally, amount of
> voice traffic carried by iLBC is only a small fraction of G.729.
>
>
> Please do not post stuff like this to the mailing list before doing any due
> diligence in advance, since people may not take seriously Your other
> comments that are valuable. G.729 is not much deployed at all, I suppose You
> are referring to G.729A implementations, which by the number of carried VoIP
> minutes is dwarfed by iLBC at this point of time. My company carries over 60
> pct of all residential international traffic from Norway and cooperates with
> world leading carriers and we all know the issues of terminating G.729A
> carried calls on GSM based systems. That is the reason why You will find
> "premium" quality minutes from number of carriers conveyed by iLBC, but
> never with G.729A!
>
> Let's rather use (deployment) facts then "say so" when bringing something
> as important to communications world as OPUS is!
>
>
>
> Best regards,
> Alan
>
>
> Alan Duric
> CTO and Cofounder
> Telio Holding ASA | Oslo exchange: TELIO
>
> twitter:  alanduric
> linkedin.com/in/alanduric
>
>
>
>
> On Thu, Apr 14, 2011 at 8:55 PM, Cullen Jennings <fluffy@cisco.com> wrote:
>
>>
>> On Apr 14, 2011, at 12:21 PM, Roman Shpount wrote:
>>
>> > I think part of the confusion comes from the fact that there are two
>> purposes for the comparative testing. One is to validate that the codec
>> meets the WG requirements. Another is to show how new codec compares to the
>> industry dominant codecs. For me, the second goal is more important then the
>> first one. I think if we care about the adoption of Opus, we should consider
>> making the comparative test results a deliverable for the working group. It
>> is very hard for a real company in the open market to justify doing
>> something, like adapting a new codec without a compelling reason. Knowing
>> how this codec compares to other existing codecs is a big part of providing
>> such a reason. If we look at the tests from this point of view, we need to
>> see how Opus compares to G.729 and AMR in narrow band, and AMR-WB and G.722
>> in wideband, Since there are no existing deployments of a meaningful size
>> (apart from a closed proprietary systems, like Skype) for UWB and FB, we can
>> compare Opus with industry leaders, such as G.719.
>> >
>> > One can argue that we should also compare Opus with patent free codecs,
>> which adds iLBC and Speex to the list, but I personally see this as less of
>> a requirement. iLBC never managed to get market traction outside of the open
>> source world,
>>
>> Sort of curios how you came to that conclusion about iLBC's market
>> traction?
>>
>>
> <ATT00001..txt>
>
>
>