[rtcweb] Opus over the Internet?

[<[e-mail]>]

Hi,

Stefan Hakansson wrote:
>
>And there seems to be no tests whatsoever (at least not involving
>humans) with actual channels introducing jitter and losses.
>

Are there any tests that actually compare different codecs, say Opus vs. G.722 vs. AMR-WB, in typical Internet use, meaning some loss and jitter? I suppose the performance is only partially related to the codec, and partially to other implementation decisions, so it might be difficult to compare apples-with-apples. But since people are arguing that Opus is an "Internet codec", it would be actually nice to see some results in this sense. Or does "Internet codec" mean something else? 

Markus

>-----Original Message-----
>From: [e-mail] [mailto:[e-mail]] On Behalf
>Of ext Stefan Hakansson LK
>Sent: 30 August, 2012 14:58
>To: Jean-Marc Valin
>Cc: [e-mail]
>Subject: Re: [rtcweb] Confirmation of consensus on audio codecs
>
>On 08/29/2012 03:10 PM, Jean-Marc Valin wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 08/29/2012 05:51 AM, Stefan Hakansson LK wrote:
>...
>>
>>> That is great, but sort of underlines that there would be no harm in
>>> delaying the decision until there are experiences made from real
>>> world use - 'cause it would not be that long till that experience has
>>> been made (Markus also brought up the IPR status as a reason for
>>> waiting - I have no idea how long it would take to know more about
>>> that).
>>
>> As Harald is pointing out, rtcweb implementations are going to ship
>> pretty soon.
>
>If that is the case (and I think and hope so), why would we need to make it
>MTI before seeing it in action?
>
>[...]
>
>> Are you expecting *another* single, standardized, royalty-free codec
>> that operates over vast ranges of bitrates and operating conditions,
>> from narrowband speech to stereo music, all with low delay, coming out
>> in the next year? If not, what are you really waiting for?
>
>No, I'm not expecting that. I would just prefer us to see that Opus does
>indeed deliver as promised before making it MTI. If it does, fine. If not, we'd
>have to discuss what to do then.
>
>To me it is like if you're going to place a bet, for an upcoming big race, on a
>horse that has never been in an actual race, but shows great promise. If you
>know that the horse is going to participate in a few practice races soon, would
>you not prefer to wait and see how it fares in those before placing the bet
>(given that the odds would not change)?
>
>Anyway, that's my view. Let's see what the chairs say.
>>
>>> As Paul suggested, I was referring to the lack of formal, controlled,
>>> characterization tests. That is how other SDOs do it. I don't think
>>> that is the only way to do it, but I think we should at least have
>>> either such tests conducted or experience from deployment and use (in
>>> a wide range of conditions and device types) before making it MTI.
>>
>> Opus has had "ITU-style" testing on English (
>> http://www.opus-codec.org/comparison/GoogleTest1.pdf ) and Mandarin (
>> http://www.opus-codec.org/comparison/GoogleTest2.pdf ). And if you
>> don't trust Google on the tests I linked to, it's also been tested by
>> Nokia:
>>
>http://research.nokia.com/files/public/%5B16%5D_InterSpeech2011_Voice_
>> Quality_Characterization_of_IETF_Opus_Codec.pdf
>Come on, those tests are very limited compared to a formal characterization
>test. (Example:
>www.itu.int/dms_pub/itu-t/opb/tut/T-TUT-ASC-2010-MSW-E.docx). There is
>very little info on the material used, the environment, processing and scripts
>and so on. And there seems to be no tests whatsoever (at least not involving
>humans) with actual channels introducing jitter and losses.
>
>Note: I am in no way proposing that a formal characterization test is needed,
>or even the right thing to do. It is a costly and time consuming process, and
>alternative approaches could prove to be more efficient.
>What I am saying is that I think we should not mandate a codec that has
>neither gone through that kind of formal characterization testing nor has any
>field experience from actual use on a reasonable scale (covering different
>conditions and device types). It just seems wrong, especially given that we will
>soon have field experience.
>
>>
>> Now, unlike other SDOs, the testing did not stop there. ITU-T codecs
>> generally end up being testing with something in the order of tens of
>> minutes worth of audio. In *addition* to that kind of testing, Opus
>> also had automated testing with hundreds of years worth of audio. If
>> anything, I think other SDOs should learn something here.
>
>This may very well be true. I guess this comes down to how much you trust
>that the quality assessment models (e.g. PEAQ, POLQA) give the same result
>as human test subjects would. But I think this sounds like a really good thing.
>
>>
>> 	Jean-Marc
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.12 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>>
>iQEcBAEBAgAGBQJQPhThAAoJEJ6/8sItn9q9nAcH/0BsXl/FBgMhbYWVALpNmI
>Yi
>> Yj1ezyW8JVSv4io1YIozucl6KiDVi3LwYFteYgl78PlQ7o2muv/lJP9VswRHooH5
>>
>Gax+aJUZkz7SlKxn+25wUlLgKQw6GAcUID5sJ0ewWdGOrwZu+SlZgYF3egYmm
>wyL
>>
>pEl8BFj7wu3uakRm/BE9LsUDPFdCoZS6uXqXRJOy9P1dZ25edvgaymQvOwhEcX
>u9
>>
>xMq82YPNU2CZpfb88ew/l+U9FjW8c4iGkZYiu+VzmBL8wYOFP5pBhBhWc/bEb
>0WX
>>
>8IUC5PJ0MMp49kb7/kfQj65/Py7u6ytYaO4PA0rjMHJ2V1PzH4EuqtNweYXVzNI
>=
>> =nwlD
>> -----END PGP SIGNATURE-----
>>
>
>_______________________________________________
>rtcweb mailing list
>[e-mail]
>https://www.ietf.org/mailman/listinfo/rtcweb