Re: [rtcweb] Opus over the Internet?

["Eggert, Lars" <[e-mail]>]

That's all great, but is there any qualitative comparison data of the different codecs available?

-- 
Sent from a mobile device; please excuse typos.

On Aug 30, 2012, at 21:07, "Jean-Marc Valin" <[e-mail]> wrote:

> On 12-08-30 09:14 AM, [e-mail] wrote:
>> Are there any tests that actually compare different codecs, say Opus
>> vs. G.722 vs. AMR-WB, in typical Internet use, meaning some loss and
>> jitter? I suppose the performance is only partially related to the
>> codec, and partially to other implementation decisions, so it might
>> be difficult to compare apples-with-apples. But since people are
>> arguing that Opus is an "Internet codec", it would be actually nice
>> to see some results in this sense. Or does "Internet codec" mean
>> something else?
> 
> The term "Internet codec" means different things to different people,
> but to me this means more than "having good packet loss concealment".
> Sure we have PLC (though it was never compared to G.722 or AMR-WB) and
> we also have (optional) low bit-rate embedded redundancy (also called
> FEC) and (also optional) independent frames, but that's just one aspect.
> 
> One of the first things we've been asked when designing Opus was to make
> the rate *really* adaptable because we never know what kind of rates
> will be available. This not only meant having a wide range of bitrates,
> but also being able to vary in small increments. This is why Opus scales
> from about 6 kb/s to 512 kb/s, in increments of 0.4 kb/s (one byte with
> 20 ms frames). Compare that to AMR-WB, which scales from 6.6 to 23.85 in
> (on average) ~2.5 kb/s increments. The reason Opus can have more than
> 1200 possible bitrates is because on the Internet, other layers in the
> protocol stack provide octet granularity framing/sizing. We don't need
> to spend 11 bits signalling the bitrate because UDP already encodes the
> packet size.
> 
> There's also practical aspects. If you look at the rates supported by
> AMR-WB, you see that *none* of them represents an integer number of
> bytes per frame. For example, 23.85 kb/s is 59 bytes plus 5 bits per
> frame. It's definitely not the only cause, but the bottom line is that
> the payload format for AMR-WB (rfc4867) is quite complex. Now compare
> this with the Opus RTP payload
> (http://tools.ietf.org/html/draft-spittka-payload-rtp-opus). Although
> it's not complete, it's not only much simpler, but it also makes it
> possible to decode RTP packets without having even seen the SDP or any
> out-of-band signalling.
> 
> People may have other definitions, but that's what *I* mean by Opus
> being an "Internet codec".
> 
> Cheers,
> 
>    Jean-Marc
> 
>>> -----Original Message----- From: [e-mail]
>>> [mailto:[e-mail]] On Behalf Of ext Stefan Hakansson
>>> LK Sent: 30 August, 2012 14:58 To: Jean-Marc Valin Cc:
>>> [e-mail] Subject: Re: [rtcweb] Confirmation of consensus on
>>> audio codecs
>>> 
>>> On 08/29/2012 03:10 PM, Jean-Marc Valin wrote:
>> On 08/29/2012 05:51 AM, Stefan Hakansson LK wrote:
>>>> ...
>> 
>>>>>> That is great, but sort of underlines that there would be no
>>>>>> harm in delaying the decision until there are experiences
>>>>>> made from real world use - 'cause it would not be that long
>>>>>> till that experience has been made (Markus also brought up
>>>>>> the IPR status as a reason for waiting - I have no idea how
>>>>>> long it would take to know more about that).
>> 
>> As Harald is pointing out, rtcweb implementations are going to ship 
>> pretty soon.
>>>> 
>>>> If that is the case (and I think and hope so), why would we need
>>>> to make it MTI before seeing it in action?
>>>> 
>>>> [...]
>>>> 
>> Are you expecting *another* single, standardized, royalty-free codec 
>> that operates over vast ranges of bitrates and operating conditions, 
>> from narrowband speech to stereo music, all with low delay, coming
>> out in the next year? If not, what are you really waiting for?
>>>> 
>>>> No, I'm not expecting that. I would just prefer us to see that
>>>> Opus does indeed deliver as promised before making it MTI. If it
>>>> does, fine. If not, we'd have to discuss what to do then.
>>>> 
>>>> To me it is like if you're going to place a bet, for an upcoming
>>>> big race, on a horse that has never been in an actual race, but
>>>> shows great promise. If you know that the horse is going to
>>>> participate in a few practice races soon, would you not prefer to
>>>> wait and see how it fares in those before placing the bet (given
>>>> that the odds would not change)?
>>>> 
>>>> Anyway, that's my view. Let's see what the chairs say.
>> 
>>>>>> As Paul suggested, I was referring to the lack of formal,
>>>>>> controlled, characterization tests. That is how other SDOs do
>>>>>> it. I don't think that is the only way to do it, but I think
>>>>>> we should at least have either such tests conducted or
>>>>>> experience from deployment and use (in a wide range of
>>>>>> conditions and device types) before making it MTI.
>> 
>> Opus has had "ITU-style" testing on English ( 
>> http://www.opus-codec.org/comparison/GoogleTest1.pdf ) and Mandarin
>> ( http://www.opus-codec.org/comparison/GoogleTest2.pdf ). And if you 
>> don't trust Google on the tests I linked to, it's also been tested
>> by Nokia:
>> 
>>>> http://research.nokia.com/files/public/%5B16%5D_InterSpeech2011_Voice_
>> 
>>>> 
> Quality_Characterization_of_IETF_Opus_Codec.pdf
>>>> Come on, those tests are very limited compared to a formal
>>>> characterization test. (Example: 
>>>> www.itu.int/dms_pub/itu-t/opb/tut/T-TUT-ASC-2010-MSW-E.docx).
>>>> There is very little info on the material used, the environment,
>>>> processing and scripts and so on. And there seems to be no tests
>>>> whatsoever (at least not involving humans) with actual channels
>>>> introducing jitter and losses.
>>>> 
>>>> Note: I am in no way proposing that a formal characterization
>>>> test is needed, or even the right thing to do. It is a costly and
>>>> time consuming process, and alternative approaches could prove to
>>>> be more efficient. What I am saying is that I think we should not
>>>> mandate a codec that has neither gone through that kind of formal
>>>> characterization testing nor has any field experience from actual
>>>> use on a reasonable scale (covering different conditions and
>>>> device types). It just seems wrong, especially given that we
>>>> will soon have field experience.
>>>> 
>> 
>> Now, unlike other SDOs, the testing did not stop there. ITU-T codecs 
>> generally end up being testing with something in the order of tens
>> of minutes worth of audio. In *addition* to that kind of testing,
>> Opus also had automated testing with hundreds of years worth of
>> audio. If anything, I think other SDOs should learn something here.
>>>> 
>>>> This may very well be true. I guess this comes down to how much
>>>> you trust that the quality assessment models (e.g. PEAQ, POLQA)
>>>> give the same result as human test subjects would. But I think
>>>> this sounds like a really good thing.
>>>> 
>> 
>> Jean-Marc
>> 
>>>> 
>>> 
>>> _______________________________________________ rtcweb mailing
>>> list [e-mail] https://www.ietf.org/mailman/listinfo/rtcweb
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb