Re: [rtcweb] Opus over the Internet?

[Jean-Marc Valin <[e-mail]>]

On 12-08-30 09:14 AM, [e-mail] wrote:
> Are there any tests that actually compare different codecs, say Opus
> vs. G.722 vs. AMR-WB, in typical Internet use, meaning some loss and
> jitter? I suppose the performance is only partially related to the
> codec, and partially to other implementation decisions, so it might
> be difficult to compare apples-with-apples. But since people are
> arguing that Opus is an "Internet codec", it would be actually nice
> to see some results in this sense. Or does "Internet codec" mean
> something else?

The term "Internet codec" means different things to different people,
but to me this means more than "having good packet loss concealment".
Sure we have PLC (though it was never compared to G.722 or AMR-WB) and
we also have (optional) low bit-rate embedded redundancy (also called
FEC) and (also optional) independent frames, but that's just one aspect.

One of the first things we've been asked when designing Opus was to make
the rate *really* adaptable because we never know what kind of rates
will be available. This not only meant having a wide range of bitrates,
but also being able to vary in small increments. This is why Opus scales
from about 6 kb/s to 512 kb/s, in increments of 0.4 kb/s (one byte with
20 ms frames). Compare that to AMR-WB, which scales from 6.6 to 23.85 in
(on average) ~2.5 kb/s increments. The reason Opus can have more than
1200 possible bitrates is because on the Internet, other layers in the
protocol stack provide octet granularity framing/sizing. We don't need
to spend 11 bits signalling the bitrate because UDP already encodes the
packet size.

There's also practical aspects. If you look at the rates supported by
AMR-WB, you see that *none* of them represents an integer number of
bytes per frame. For example, 23.85 kb/s is 59 bytes plus 5 bits per
frame. It's definitely not the only cause, but the bottom line is that
the payload format for AMR-WB (rfc4867) is quite complex. Now compare
this with the Opus RTP payload
(http://tools.ietf.org/html/draft-spittka-payload-rtp-opus). Although
it's not complete, it's not only much simpler, but it also makes it
possible to decode RTP packets without having even seen the SDP or any
out-of-band signalling.

People may have other definitions, but that's what *I* mean by Opus
being an "Internet codec".

Cheers,

	Jean-Marc

>> -----Original Message----- From: [e-mail]
>> [mailto:[e-mail]] On Behalf Of ext Stefan Hakansson
>> LK Sent: 30 August, 2012 14:58 To: Jean-Marc Valin Cc:
>> [e-mail] Subject: Re: [rtcweb] Confirmation of consensus on
>> audio codecs
>> 
>> On 08/29/2012 03:10 PM, Jean-Marc Valin wrote:
> On 08/29/2012 05:51 AM, Stefan Hakansson LK wrote:
>>> ...
> 
>>>>> That is great, but sort of underlines that there would be no
>>>>> harm in delaying the decision until there are experiences
>>>>> made from real world use - 'cause it would not be that long
>>>>> till that experience has been made (Markus also brought up
>>>>> the IPR status as a reason for waiting - I have no idea how
>>>>> long it would take to know more about that).
> 
> As Harald is pointing out, rtcweb implementations are going to ship 
> pretty soon.
>>> 
>>> If that is the case (and I think and hope so), why would we need
>>> to make it MTI before seeing it in action?
>>> 
>>> [...]
>>> 
> Are you expecting *another* single, standardized, royalty-free codec 
> that operates over vast ranges of bitrates and operating conditions, 
> from narrowband speech to stereo music, all with low delay, coming
> out in the next year? If not, what are you really waiting for?
>>> 
>>> No, I'm not expecting that. I would just prefer us to see that
>>> Opus does indeed deliver as promised before making it MTI. If it
>>> does, fine. If not, we'd have to discuss what to do then.
>>> 
>>> To me it is like if you're going to place a bet, for an upcoming
>>> big race, on a horse that has never been in an actual race, but
>>> shows great promise. If you know that the horse is going to
>>> participate in a few practice races soon, would you not prefer to
>>> wait and see how it fares in those before placing the bet (given
>>> that the odds would not change)?
>>> 
>>> Anyway, that's my view. Let's see what the chairs say.
> 
>>>>> As Paul suggested, I was referring to the lack of formal,
>>>>> controlled, characterization tests. That is how other SDOs do
>>>>> it. I don't think that is the only way to do it, but I think
>>>>> we should at least have either such tests conducted or
>>>>> experience from deployment and use (in a wide range of
>>>>> conditions and device types) before making it MTI.
> 
> Opus has had "ITU-style" testing on English ( 
> http://www.opus-codec.org/comparison/GoogleTest1.pdf ) and Mandarin
> ( http://www.opus-codec.org/comparison/GoogleTest2.pdf ). And if you 
> don't trust Google on the tests I linked to, it's also been tested
> by Nokia:
> 
>>> http://research.nokia.com/files/public/%5B16%5D_InterSpeech2011_Voice_
>
>>> 
Quality_Characterization_of_IETF_Opus_Codec.pdf
>>> Come on, those tests are very limited compared to a formal
>>> characterization test. (Example: 
>>> www.itu.int/dms_pub/itu-t/opb/tut/T-TUT-ASC-2010-MSW-E.docx).
>>> There is very little info on the material used, the environment,
>>> processing and scripts and so on. And there seems to be no tests
>>> whatsoever (at least not involving humans) with actual channels
>>> introducing jitter and losses.
>>> 
>>> Note: I am in no way proposing that a formal characterization
>>> test is needed, or even the right thing to do. It is a costly and
>>> time consuming process, and alternative approaches could prove to
>>> be more efficient. What I am saying is that I think we should not
>>> mandate a codec that has neither gone through that kind of formal
>>> characterization testing nor has any field experience from actual
>>> use on a reasonable scale (covering different conditions and
>>> device types). It just seems wrong, especially given that we
>>> will soon have field experience.
>>> 
> 
> Now, unlike other SDOs, the testing did not stop there. ITU-T codecs 
> generally end up being testing with something in the order of tens
> of minutes worth of audio. In *addition* to that kind of testing,
> Opus also had automated testing with hundreds of years worth of
> audio. If anything, I think other SDOs should learn something here.
>>> 
>>> This may very well be true. I guess this comes down to how much
>>> you trust that the quality assessment models (e.g. PEAQ, POLQA)
>>> give the same result as human test subjects would. But I think
>>> this sounds like a really good thing.
>>> 
> 
> Jean-Marc
> 
>>> 
>> 
>> _______________________________________________ rtcweb mailing
>> list [e-mail] https://www.ietf.org/mailman/listinfo/rtcweb