Re: [rtcweb] Opus vs AMR-WB for packet loss

[Jean-Marc Valin <[e-mail]>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

OK, here's something concrete for comparing Opus to AMR-WB in packet
loss conditions. I've simulated 30% bursty packet loss in the
following conditions:
1) AMR-WB mode 8 (23.85 kb/s)
2) Opus at 24 kb/s (no FEC)
3) Opus at 24 kb/s with FEC (24 kb/s is the total including FEC)

I've uploaded the results at: http://jmvalin.ca/plc/

I think everyone will agree that even without FEC, Opus sounds much
better than AMR-WB. And when you turn FEC on, then you barely even
notice the losses.

	Jean-Marc

On 08/31/2012 11:16 AM, Mandyam, Giridhar wrote:
> Thank you for pointing out these results.
> 
> 
> 
> I think these results represent a very good start, but the results
> as provided may not be sufficient to characterize Opus.  I’ve
> already cited the Dynastat study in 3GPP2 for EVRC-B
> characterization in a separate email.  In addition to packet loss
> (in the 3GPP2 study, the analog would be frame error rate),
> background noise conditions were varied (based on MNRU, street
> noise, car noise).  This is the kind of testing that is typical for
> codecs standardized in 3GPP and 3GPP2.
> 
> 
> 
> There are several details missing.  For instance,
> 
> 
> 
> Was only office noise tested? If so, why?
> 
> How many listeners were used?
> 
> What was the mix of talkers (e.g. no. of female, no. of male)?
> 
> 
> 
> If you can point to more details of the testing, that would be
> very helpful for members of the group.
> 
> 
> 
> -Giri Mandyam
> 
> 
> 
> *From:*[e-mail] [mailto:[e-mail]]
> *On Behalf Of *Koen Vos *Sent:* Friday, August 31, 2012 7:38 AM 
> *To:* [e-mail] *Subject:* Re: [rtcweb] Opus over the
> Internet?
> 
> 
> 
> Lars Eggert wrote:
> 
>> That's all great, but is there any qualitative comparison data of
>> the different codecs available?
> 
> Please have a look at the test Dynastat did with SILK: 
> http://developer.skype.com/resources/SILKDataSheet.pdf
> 
> Note that SILK in Opus is quite similar to the old SILK tested by 
> Dynastat.  All changes were thoroughly tested to not degrade 
> performance, and in most cases improved it.  So the SILK Dynastat 
> results give a lower bound on the performance of a subset of the
> Opus modes.
> 
> While these plots don't show G.722, it was in fact included in the
> test for clean input signals without packet loss.  According to the
> test results: - SILK at 8.85 kbps is outperformed by G.722 at 64
> kbps with statistical significance - SILK at 12.65 kbps is a
> statistical tie with G.722 at 64 kbps - SILK at 18.85 kbps and
> above outperforms G.722 at 64 kbps with statistical significance.
> 
> The packet loss in that test was maximally random (ie not bursty).
>  While that may not perfectly mimic real networks, I can think of
> no reason why the results would be reversed or very different in
> practice.
> 
> To give one more measure of suitability for the Internet: Skype has
> used SILK to deliver 100s of billions of Skype-to-Skype voice
> minutes over the Internet, and our users are very satisfied with
> it.
> 
> Hope this helps, koen.
> 
> Lars Eggert wrote: That's all great, but is there any qualitative
> comparison data of the different codecs available?
> 
> 
> 
> --
> 
> Sent from a mobile device; please excuse typos.
> 
> 
> 
> On Aug 30, 2012, at 21:07, "Jean-Marc Valin" <jmvalin at
> mozilla.com> wrote:
> 
> 
> 
>> On 12-08-30 09:14 AM, Markus.Isomaki at nokia.com wrote:
> 
>>> Are there any tests that actually compare different codecs, say
>>> Opus
> 
>>> vs. G.722 vs. AMR-WB, in typical Internet use, meaning some
>>> loss and
> 
>>> jitter? I suppose the performance is only partially related to
>>> the
> 
>>> codec, and partially to other implementation decisions, so it
>>> might
> 
>>> be difficult to compare apples-with-apples. But since people
>>> are
> 
>>> arguing that Opus is an "Internet codec", it would be actually
>>> nice
> 
>>> to see some results in this sense. Or does "Internet codec"
>>> mean
> 
>>> something else?
> 
>> 
> 
>> The term "Internet codec" means different things to different
>> people,
> 
>> but to me this means more than "having good packet loss
>> concealment".
> 
>> Sure we have PLC (though it was never compared to G.722 or
>> AMR-WB) and
> 
>> we also have (optional) low bit-rate embedded redundancy (also
>> called
> 
>> FEC) and (also optional) independent frames, but that's just one
>> aspect.
> 
>> 
> 
>> One of the first things we've been asked when designing Opus was
>> to make
> 
>> the rate *really* adaptable because we never know what kind of
>> rates
> 
>> will be available. This not only meant having a wide range of
>> bitrates,
> 
>> but also being able to vary in small increments. This is why Opus
>> scales
> 
>> from about 6 kb/s to 512 kb/s, in increments of 0.4 kb/s (one
>> byte with
> 
>> 20 ms frames). Compare that to AMR-WB, which scales from 6.6 to
>> 23.85 in
> 
>> (on average) ~2.5 kb/s increments. The reason Opus can have more
>> than
> 
>> 1200 possible bitrates is because on the Internet, other layers
>> in the
> 
>> protocol stack provide octet granularity framing/sizing. We don't
>> need
> 
>> to spend 11 bits signalling the bitrate because UDP already
>> encodes the
> 
>> packet size.
> 
>> 
> 
>> There's also practical aspects. If you look at the rates
>> supported by
> 
>> AMR-WB, you see that *none* of them represents an integer number
>> of
> 
>> bytes per frame. For example, 23.85 kb/s is 59 bytes plus 5 bits
>> per
> 
>> frame. It's definitely not the only cause, but the bottom line is
>> that
> 
>> the payload format for AMR-WB (rfc4867) is quite complex. Now
>> compare
> 
>> this with the Opus RTP payload
> 
>> (http://tools.ietf.org/html/draft-spittka-payload-rtp-opus).
>> Although
> 
>> it's not complete, it's not only much simpler, but it also makes
>> it
> 
>> possible to decode RTP packets without having even seen the SDP
>> or any
> 
>> out-of-band signalling.
> 
>> 
> 
>> People may have other definitions, but that's what *I* mean by
>> Opus
> 
>> being an "Internet codec".
> 
>> 
> 
>> Cheers,
> 
>> 
> 
>> Jean-Marc
> 
>> 
> 
>>>> -----Original Message----- From: rtcweb-bounces at ietf.org
> 
>>>> [mailto:rtcweb-bounces at ietf.org] On Behalf Of ext Stefan
>>>> Hakansson
> 
>>>> LK Sent: 30 August, 2012 14:58 To: Jean-Marc Valin Cc:
> 
>>>> rtcweb at ietf.org Subject: Re: [rtcweb] Confirmation of
>>>> consensus on
> 
>>>> audio codecs
> 
>>>> 
> 
>>>> On 08/29/2012 03:10 PM, Jean-Marc Valin wrote:
> 
>>> On 08/29/2012 05:51 AM, Stefan Hakansson LK wrote:
> 
>>>>> ...
> 
>>> 
> 
>>>>>>> That is great, but sort of underlines that there would
>>>>>>> be no
> 
>>>>>>> harm in delaying the decision until there are
>>>>>>> experiences
> 
>>>>>>> made from real world use - 'cause it would not be that
>>>>>>> long
> 
>>>>>>> till that experience has been made (Markus also brought
>>>>>>> up
> 
>>>>>>> the IPR status as a reason for waiting - I have no idea
>>>>>>> how
> 
>>>>>>> long it would take to know more about that).
> 
>>> 
> 
>>> As Harald is pointing out, rtcweb implementations are going to
>>> ship
> 
>>> pretty soon.
> 
>>>>> 
> 
>>>>> If that is the case (and I think and hope so), why would we
>>>>> need
> 
>>>>> to make it MTI before seeing it in action?
> 
>>>>> 
> 
>>>>> [...]
> 
>>>>> 
> 
>>> Are you expecting *another* single, standardized, royalty-free
>>> codec
> 
>>> that operates over vast ranges of bitrates and operating
>>> conditions,
> 
>>> from narrowband speech to stereo music, all with low delay,
>>> coming
> 
>>> out in the next year? If not, what are you really waiting for?
> 
>>>>> 
> 
>>>>> No, I'm not expecting that. I would just prefer us to see
>>>>> that
> 
>>>>> Opus does indeed deliver as promised before making it MTI.
>>>>> If it
> 
>>>>> does, fine. If not, we'd have to discuss what to do then.
> 
>>>>> 
> 
>>>>> To me it is like if you're going to place a bet, for an
>>>>> upcoming
> 
>>>>> big race, on a horse that has never been in an actual race,
>>>>> but
> 
>>>>> shows great promise. If you know that the horse is going
>>>>> to
> 
>>>>> participate in a few practice races soon, would you not
>>>>> prefer to
> 
>>>>> wait and see how it fares in those before placing the bet
>>>>> (given
> 
>>>>> that the odds would not change)?
> 
>>>>> 
> 
>>>>> Anyway, that's my view. Let's see what the chairs say.
> 
>>> 
> 
>>>>>>> As Paul suggested, I was referring to the lack of
>>>>>>> formal,
> 
>>>>>>> controlled, characterization tests. That is how other
>>>>>>> SDOs do
> 
>>>>>>> it. I don't think that is the only way to do it, but I
>>>>>>> think
> 
>>>>>>> we should at least have either such tests conducted or
> 
>>>>>>> experience from deployment and use (in a wide range of
> 
>>>>>>> conditions and device types) before making it MTI.
> 
>>> 
> 
>>> Opus has had "ITU-style" testing on English (
> 
>>> http://www.opus-codec.org/comparison/GoogleTest1.pdf ) and
>>> Mandarin
> 
>>> ( http://www.opus-codec.org/comparison/GoogleTest2.pdf ). And
>>> if you
> 
>>> don't trust Google on the tests I linked to, it's also been
>>> tested
> 
>>> by Nokia:
> 
>>> 
> 
>>>>> http://research.nokia.com/files/public/%5B16%5D_InterSpeech2011_Voice_
>
>>>>> 
>>> 
> 
>>>>> 
> 
>> Quality_Characterization_of_IETF_Opus_Codec.pdf
> 
>>>>> Come on, those tests are very limited compared to a formal
> 
>>>>> characterization test. (Example:
> 
>>>>> www.itu.int/dms_pub/itu-t/opb/tut/T-TUT-ASC-2010-MSW-E.docx
>>>>> <http://www.itu.int/dms_pub/itu-t/opb/tut/T-TUT-ASC-2010-MSW-E.docx>).
>
>>>>>  There is very little info on the material used, the
>>>>> environment,
> 
>>>>> processing and scripts and so on. And there seems to be no
>>>>> tests
> 
>>>>> whatsoever (at least not involving humans) with actual
>>>>> channels
> 
>>>>> introducing jitter and losses.
> 
>>>>> 
> 
>>>>> Note: I am in no way proposing that a formal
>>>>> characterization
> 
>>>>> test is needed, or even the right thing to do. It is a
>>>>> costly and
> 
>>>>> time consuming process, and alternative approaches could
>>>>> prove to
> 
>>>>> be more efficient. What I am saying is that I think we
>>>>> should not
> 
>>>>> mandate a codec that has neither gone through that kind of
>>>>> formal
> 
>>>>> characterization testing nor has any field experience from
>>>>> actual
> 
>>>>> use on a reasonable scale (covering different conditions
>>>>> and
> 
>>>>> device types). It just seems wrong, especially given that
>>>>> we
> 
>>>>> will soon have field experience.
> 
>>>>> 
> 
>>> 
> 
>>> Now, unlike other SDOs, the testing did not stop there. ITU-T
>>> codecs
> 
>>> generally end up being testing with something in the order of
>>> tens
> 
>>> of minutes worth of audio. In *addition* to that kind of
>>> testing,
> 
>>> Opus also had automated testing with hundreds of years worth
>>> of
> 
>>> audio. If anything, I think other SDOs should learn something
>>> here.
> 
>>>>> 
> 
>>>>> This may very well be true. I guess this comes down to how
>>>>> much
> 
>>>>> you trust that the quality assessment models (e.g. PEAQ,
>>>>> POLQA)
> 
>>>>> give the same result as human test subjects would. But I
>>>>> think
> 
>>>>> this sounds like a really good thing.
> 
>>>>> 
> 
>>> 
> 
>>> Jean-Marc
> 
>>> 
> 
>>>>> 
> 
>>>> 
> 
>>>> _______________________________________________ rtcweb
>>>> mailing
> 
>>>> list rtcweb at ietf.org
>>>> https://www.ietf.org/mailman/listinfo/rtcweb
> 
>> _______________________________________________
> 
>> rtcweb mailing list
> 
>> rtcweb at ietf.org
> 
>> https://www.ietf.org/mailman/listinfo/rtcweb
> 
> 
> 
> 
> 
> _______________________________________________ rtcweb mailing
> list [e-mail] https://www.ietf.org/mailman/listinfo/rtcweb
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQQSOXAAoJEJ6/8sItn9q9hpEIAL0XKKrLJBb4AfUH7700lkYH
JS60S/XFmwBOOZ55q+AUHEprvtIcsTD3XE1Yrqn0O5xWvpIA8Rhb93g5kT+zaUNl
VcXZ4OsB9Yt8ahqy0yVdtXHxo//nIffiqNnQlyAschiIXONGxBCkwvyU21XnrPL0
oEtcnx3Y9sG5eFJMuJJt3k9NBpvL1TzmHQOVuYNMNvCgCNHYQIhS9RHplJmAJnp4
XIaz63JrKpmw+jgsQeqJNfyf8G3+JrvWYouA5aoHMkFhzg3dKatjaN2bctp/fNNE
4jzjPlegTS4eSR3cqyqkv1lW+StIxEcwWEhKYqc2vK3S4lUpbWKFAC2t83L/wSQ=
=4iDk
-----END PGP SIGNATURE-----