Re: [codec] comparitive quality testing

[Cullen Jennings <[e-mail]>]

On Apr 19, 2011, at 5:08 PM, Roman Shpount wrote:

> Dear Alan,
> 
> This discussion is a bit unrelated to the overall discussion in the group, but I would still like to clarify some of my points.
> 
> As far as iLBC market penetration is concerned, what I know (and please correct me if I am wrong), none of the major US VoIP carriers (AT&T, Verizon, Qwest, Level 3, XO, GlobalCorssing) offer iLBC termination to their customers.

Hmm, at least some of the above are willing to sell iLBC termination to PSTN.

> It is a choice of G.711 or G.729 A or B only. This is either caused by the luck of market demand, or by them using Sonus gateways which, as far as I know, do not support iLBC.

Most the call flows I see to a Sonus box have an ACME Packet SBC in front of them. ACME fixes everything including the codec :-) I may be mistaken but I was under the impression that some Sonus equipment does support iLBC. 

> 
> Once again, as far as I know Skype is not using iLBC for anything. It is using Silk, G.711. G.729A, and ISAC.
> 
> As far as I know none of the cable VoIP operators in US use iLBC, it is all G.711.
> 
> As far as hardware vendors are concerned, Cisco does support iLBC on its gateways a quite a few (but I do not think all) of its phones.

> Sonus only supports iLBC in its SBC, but not in gateways. Polycom, Snom, Aastra do not seem to support iLBC in their phones.

Cisco has it on most of it's phones. I just did a grep of some call flows that I had from various phones.  At least some phones from snom support iLBC, and some phones from aastra, and some from polycom, and a boat load over other vendors that matched my grep.  I can't tell you how prevalent it is across other vendors product lines but the three you mention do support it in at least some of the phones. 

> 
> Where iLBC seemed to thrive is in soft-phones and various types of desktop based software VoIP products. In such products the benefits of not having a per instance license are probably the greatest. iLBC is also universally supported by open source projects such as Asterisk, Freeswitch, PJ-SIP, OpenH323 descendents, and so on. iLBC is also present in Android and iOS. What is interesting about all of those products (and I did check pretty much all the open source projects I can find) that the iLBC implementation in these products are essentially the reference code from the RFC. This code is extremely inefficient. What I mean by this it consumes more then 10 times the CPU resources of Voiceage or IPP optimized G.729A. With Freeswitch or Asterisk you cannot get more the 50-100 channels of iLBC per modern dual CPU server. On the other hand running over 500 channels of G.729A is fairly normal. What was interesting for me that no one spent any time optimizing any of the open source versions of the codec even though this is fairly trivial. On the other hand, such high CPU requirements on server and mobile platforms make iLBC almost unusable.


> If anyone on this list knows the open source optimized or fixed point implementation, please point it out to me.
> 
> There is also a question of ILBC Freeware license. Based on IETF rules, all the code released as a part of IETF RFC should be covered by BSD license. The reasoning and applicability of iLBC Freeware license in this context is unclear.
> 
> IPR disclosure for iLBC RFC was not updated when RFC was finalized and only has provisional patent applications, not the final granted patent numbers.
> 
> Finally, I would like to switch to something that has direct applicability to the current working group. iLBC claims to be the same computation complexity as G.729A. From what I know, ILBC reference implementation is about 3-4 times the complexity of G.729A. It is possible to create an iLBC implementation of the same complexity as G.729A, but this will, from what I've seen, cause the quality degradation similar to degradation between G.729 and G.729A. I did not measure this quality degradation, but I know that, for instance, reducing the number of computation points in LPC to LSF conversion, to bring it inline with G.729A, will produce different encoded audio. All the quality tests for iLBC, from what I've seen, were done using the reference code. I assume (I have no good way of testing this), that the optimized code in Cisco and other production gateways will produce different encoded audio. I have never seen the quality test for this optimized code. In case of testing codecs where implementation varies for the same bitstream, it is very important to compare quality and computation complexity of the same implementation. Otherwise you will get results that are misleading.
> 
> Furthermore, reference code in the RFC, based on the compiler used and optimization options will produce different encoded bitstream on the same input audio. In some cases encoded bitsream will match the reference vector provided on iLBCfreeware.org, but will produce different results on other audio files. Some differences are expected in decoder, since it is a floating point implementation, but I am not sure encoded audio differences are expected according even to iLBC Freeware site.
> 
> If we plan to run comparison test with iLBC we need to specify which implementation we are planning to use. Additionally we need to specify platform, compiler version, and compilation settings. Finally, we need to specify if enhancer should be used in decoder. If we will not do this, the test input will not be identical across the test sites.
> _____________
> Roman Shpount
> 
> 
> On Sat, Apr 16, 2011 at 6:07 AM, Alan Duric <[e-mail]> wrote:
> Dear Roman,
> 
> despite the fact that I was not able to follow work of the group for quite some time, I was informed about Your post form colleagues and I needed to react, since there is quite a number of disinformations that You are placing (though not sure if by purpose or just by not knowing better)!
> 
> Comments are inline
> 
> 
> On Apr 15, 2011, at 3:20 AM, Roman Shpount wrote:
> 
>> I know that iLBC was picked by PacketCable and is supported by quite a few gateways. On the other hand, iLBC standard itself looks as if the codec was contributed by one company with almost nobody investing any time in its actual standardization.
> 
> A number of people that were involved in the verification were and are unarguably among the authorites in the speech coding world and have been heavily involved in the previous standardization efforts by number of standardization bodies. Their reputation is a way to big to endorse something what they do not fully stand behind. In the past a number of standards brought up by ITU, ETSI and TIA have been primarily or solely contributed by one company and it does not make those standards less valuable.
> 
>> Quality comparisons and resource requirements seemed to be coming from the same company and accepted on their say so.
> 
> Please see above.As well, Dynastat (at that point of time one of the best independent labs) have done assessment of the codec in comparison to other existing standards. Results are available through IETF archive.
> 
> 
>> Most of the deployed versions of iLBC are a floating point reference implementation, which is coded in a manner as inefficient as a codec implementation can possible be.
> 
> Where do You get this information??? This is prime example of  "say so"! Information like this are confusing those that are following mailing list and I find it: either malicious or - You do not know much of VOIP deployments in general and thus should not be raising Your voice in front of this many people! CalbeLabs and Skype are terminating with iLBC more minutes towards PSTN then with any other codec combined. My company also happens to terminate billions of voice minutes traffic since 2004 and through very close cooperation with voice (or media) gateway manufacturers and by being advisor to a number of major carriers, I trust I have pretty good insight on implementations of various codecs and minutes that are carried with them.
> 
>> There are no open source fixed-point implementations, even though creating one is trivial. Reference vectors consist of a single audio file recorded by a single voice. Encoder, despite being a bit exact reference, produces different results based on optimization options or compiler. To summarize, I assumed that there is no interest in this code since there are a lot of problems with it that would have been easy to fix if anybody bothered. It looks like a proof of concept and not a finished product. Finally, amount of voice traffic carried by iLBC is only a small fraction of G.729.
> 
> Please do not post stuff like this to the mailing list before doing any due diligence in advance, since people may not take seriously Your other comments that are valuable. G.729 is not much deployed at all, I suppose You are referring to G.729A implementations, which by the number of carried VoIP minutes is dwarfed by iLBC at this point of time. My company carries over 60 pct of all residential international traffic from Norway and cooperates with world leading carriers and we all know the issues of terminating G.729A carried calls on GSM based systems. That is the reason why You will find "premium" quality minutes from number of carriers conveyed by iLBC, but never with G.729A!
> 
> Let's rather use (deployment) facts then "say so" when bringing something as important to communications world as OPUS is!
> 
> 
> 
> Best regards,
> Alan
> 
> 
> Alan Duric
> CTO and Cofounder
> Telio Holding ASA | Oslo exchange: TELIO
> 
> twitter: 	alanduric  
> linkedin.com/in/alanduric
> 
> 
> 
>> 
>> On Thu, Apr 14, 2011 at 8:55 PM, Cullen Jennings <[e-mail]> wrote:
>> 
>> On Apr 14, 2011, at 12:21 PM, Roman Shpount wrote:
>> 
>> > I think part of the confusion comes from the fact that there are two purposes for the comparative testing. One is to validate that the codec meets the WG requirements. Another is to show how new codec compares to the industry dominant codecs. For me, the second goal is more important then the first one. I think if we care about the adoption of Opus, we should consider making the comparative test results a deliverable for the working group. It is very hard for a real company in the open market to justify doing something, like adapting a new codec without a compelling reason. Knowing how this codec compares to other existing codecs is a big part of providing such a reason. If we look at the tests from this point of view, we need to see how Opus compares to G.729 and AMR in narrow band, and AMR-WB and G.722 in wideband, Since there are no existing deployments of a meaningful size (apart from a closed proprietary systems, like Skype) for UWB and FB, we can compare Opus with industry leaders, such as G.719.
>> >
>> > One can argue that we should also compare Opus with patent free codecs, which adds iLBC and Speex to the list, but I personally see this as less of a requirement. iLBC never managed to get market traction outside of the open source world,
>> 
>> Sort of curios how you came to that conclusion about iLBC's market traction?
>> 
>> 
>> <ATT00001..txt>
> 
>