Re: [codec] comparitive quality testing

[Alan Duric <[e-mail]>]

Dear Roman,

despite the fact that I was not able to follow work of the group for quite some time, I was informed about Your post form colleagues and I needed to react, since there is quite a number of disinformations that You are placing (though not sure if by purpose or just by not knowing better)!

Comments are inline


On Apr 15, 2011, at 3:20 AM, Roman Shpount wrote:

I know that iLBC was picked by PacketCable and is supported by quite a few gateways. On the other hand, iLBC standard itself looks as if the codec was contributed by one company with almost nobody investing any time in its actual standardization.

A number of people that were involved in the verification were and are unarguably among the authorites in the speech coding world and have been heavily involved in the previous standardization efforts by number of standardization bodies. Their reputation is a way to big to endorse something what they do not fully stand behind. In the past a number of standards brought up by ITU, ETSI and TIA have been primarily or solely contributed by one company and it does not make those standards less valuable.

Quality comparisons and resource requirements seemed to be coming from the same company and accepted on their say so.

Please see above.As well, Dynastat (at that point of time one of the best independent labs) have done assessment of the codec in comparison to other existing standards. Results are available through IETF archive.


Most of the deployed versions of iLBC are a floating point reference implementation, which is coded in a manner as inefficient as a codec implementation can possible be.

Where do You get this information??? This is prime example of  "say so"! Information like this are confusing those that are following mailing list and I find it: either malicious or - You do not know much of VOIP deployments in general and thus should not be raising Your voice in front of this many people! CalbeLabs and Skype are terminating with iLBC more minutes towards PSTN then with any other codec combined. My company also happens to terminate billions of voice minutes traffic since 2004 and through very close cooperation with voice (or media) gateway manufacturers and by being advisor to a number of major carriers, I trust I have pretty good insight on implementations of various codecs and minutes that are carried with them.

There are no open source fixed-point implementations, even though creating one is trivial. Reference vectors consist of a single audio file recorded by a single voice. Encoder, despite being a bit exact reference, produces different results based on optimization options or compiler. To summarize, I assumed that there is no interest in this code since there are a lot of problems with it that would have been easy to fix if anybody bothered. It looks like a proof of concept and not a finished product. Finally, amount of voice traffic carried by iLBC is only a small fraction of G.729.

Please do not post stuff like this to the mailing list before doing any due diligence in advance, since people may not take seriously Your other comments that are valuable. G.729 is not much deployed at all, I suppose You are referring to G.729A implementations, which by the number of carried VoIP minutes is dwarfed by iLBC at this point of time. My company carries over 60 pct of all residential international traffic from Norway and cooperates with world leading carriers and we all know the issues of terminating G.729A carried calls on GSM based systems. That is the reason why You will find "premium" quality minutes from number of carriers conveyed by iLBC, but never with G.729A!

Let's rather use (deployment) facts then "say so" when bringing something as important to communications world as OPUS is!



Best regards,
Alan


Alan Duric
CTO and Cofounder
Telio Holding ASA | Oslo exchange: TELIO

twitter:  alanduric
linkedin.com/in/alanduric<http://linkedin.com/in/alanduric>




On Thu, Apr 14, 2011 at 8:55 PM, Cullen Jennings <[e-mail]<mailto:[e-mail]>> wrote:

On Apr 14, 2011, at 12:21 PM, Roman Shpount wrote:

> I think part of the confusion comes from the fact that there are two purposes for the comparative testing. One is to validate that the codec meets the WG requirements. Another is to show how new codec compares to the industry dominant codecs. For me, the second goal is more important then the first one. I think if we care about the adoption of Opus, we should consider making the comparative test results a deliverable for the working group. It is very hard for a real company in the open market to justify doing something, like adapting a new codec without a compelling reason. Knowing how this codec compares to other existing codecs is a big part of providing such a reason. If we look at the tests from this point of view, we need to see how Opus compares to G.729 and AMR in narrow band, and AMR-WB and G.722 in wideband, Since there are no existing deployments of a meaningful size (apart from a closed proprietary systems, like Skype) for UWB and FB, we can compare Opus with industry leaders, such as G.719.
>
> One can argue that we should also compare Opus with patent free codecs, which adds iLBC and Speex to the list, but I personally see this as less of a requirement. iLBC never managed to get market traction outside of the open source world,

Sort of curios how you came to that conclusion about iLBC's market traction?


<ATT00001..txt>