Re: [codec] Opus comparision test plan

Hi Christian,

For the test it should be ensured that the used codec implementation fulfills  high quality requirements. I don't know the AAC-ELD fdk-aac library.  The link in your document did not work for me. It does not make sense to test an AAC-ELD encoder that does not provide state-of-the-art encoding quality.
Concerning SBR: AFAIK know AAC-ELD standard does not have SBR specified as a tool.

I think, 20 % Frame erasure rate makes no sense as it is unrealistic. I would prefer frame erase rates like 1%, 3% or  6%. These rates should be used both in speech and music experiments. 

For music testing I would recommend  the bitrates 24, 32, 48 kbit/s for mono at 32 kHz sample rate. For stereo 32, 48, 64 and 96 kbit/s could be used. I think these working points would be used most likely in applications where low delay is a major requirement.
It is more important to test a higher number of contents than to tests the very high bitrates.

Mono and stereo should not be mixed in one experiment to avoid the influence of this factor on the assessment of the coding quality. 

What is a MOS 4 anchor in the context of a MUSHRA test?

Best regards,
Bernhard

-----Original Message-----
From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] On Behalf Of Paul Coverdale
Sent: Mittwoch, 17. April 2013 01:28
To: 'Jean-Marc Valin'; 'Christian Hoene'
Cc: alfons.martin@symonics.com; codec@ietf.org; patrick.schreiner@symonics.com
Subject: Re: [codec] Opus comparision test plan

Hi Christian,

Yes, I think there is still some work to be done on the plan, I noticed for example that input level and listening level still need to be defined. And why are you using MUSHRA? It may be quite time-consuming with a large number of conditions.

But just to comment on some of the points raised by Jean-Marc:
- we need to be consistent about the input filtering. We can't use different bandwidths for Opus and AMR-WB.
- I don't understand the point about making sure to tell the Opus encoder what the percentage of loss is so it can optimize the encoding for it. What happens in practice when the loss percentage isn't known?

Cheers,

...Paul

>-----Original Message-----
>From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] On Behalf 
>Of Jean-Marc Valin
>Sent: Tuesday, April 16, 2013 2:36 PM
>To: Christian Hoene
>Cc: alfons.martin@symonics.com; codec@ietf.org; 
>patrick.schreiner@symonics.com
>Subject: Re: [codec] Opus comparision test plan
>
>Christian,
>
>I was only able to look at your plan quickly but still found many 
>issues with it. I think you should take your time to fix those and also 
>all the ones I'm sure I missed. So far...
>
>== AMR-WB test ==
>
>- You should use 8 kHz bandwidth for Opus. Opus it not restricted to 7 
>kHz and low-pass filtering will hurt quality.
>- For AMR-WB the 8.85 mode is "intended to be used only temporarily 
>during severe radio channel conditions or during network congestion", 
>so you should probably test the 18.25 kbps mode instead.
>- It's not clear from your document, but Opus should use VBR
>- If you're going to test packet loss, there should be at least one 
>test with FEC -- probably the 23.85 one
>- Make sure to tell the Opus encoder what the percentage of loss is so 
>it can optimize the encoding for it.
>
>== AAC-ELD test ==
>
>- Some of the rates in the test are in the bitrates where both codecs 
>are expected to be "falling apart", so we will not learn much there 
>(really bad quality vs absolutely terrible doesn't help because people 
>don't want either).
>- Some of the rates are way too high for listeners to be able to 
>reliably judge. These are guaranteed to end up with everything tied 
>unless there's a problem with the test.
>- Using SBR at 128 kb/s for ELD is useless and probably harming quality
>- There is no reason to test music at 24/32 kHz because music is almost 
>always available at 48 kHz, so it would hurt to artificially limit the 
>signal.
>- When considering CBR vs VBR you have to consider the delay, but also 
>the different meanings these have between codecs. I'm not familiar with 
>the details, but what most codecs like AAC-LC and MP3 call "CBR"
>actually uses a bit reservoir and is equivalent to Opus "Constrained 
>VBR".
>- The choice of music samples is critical. Just that choice can 
>sometimes determine the outcome of a music test. You need a wide 
>variety of samples to get something statistically significant. It's 
>also probably a good idea to use "normal" samples at low rates and "hard"
>samples at high rate.
>- I'd need to recheck the MUSHRA spec, but I'm pretty sure mixing mono 
>and stereo in the same session is highly discouraged.
>- Not sure in what form, but it might be worth testing the pre-1.1 
>encoder of Opus.
>
>In general, for mono music, I'd recommend restricting yourself to rates 
>between 40 and 64 kb/s. For stereo music, I'd stay within 56-96 kb/s.
>Outside of this you're either in the falling apart region or in the 
>"nearly transparent for most listeners".
>
>Also, I hope you will fix your methodology from the previous test to 
>actually tell the listeners about the hidden reference.
>
>In any case, this is only what I've been able to find with a quick look.
>It's probably good to have wider review once you fix everything.
>
>	Jean-Marc
>
>On 04/16/2013 03:43 AM, Christian Hoene wrote:
>> Hello,
>>
>> as for next week, we place to compare Opus against AMR-WB (with noisy
>> signals) and AAC-eLD. Attached our preliminary test plan. Feel free 
>> to criticize it.
>>
>> In addition to the comparison, we will do some Opus characterization 
>> (Opus vs. Opus). More to about this test later.
>>
>> With best regards,
>>
>> Christian Hoene
>>
>>
>>
>>
>> Symonics GmbH
>> Sand 13
>> 72076 Tübingen
>> Tel +49 7071 5681302
>> Fax +49 7071 5681309
>> Email: christian.hoene@symonics.com
>>
>> Geschäftsführer/Presidents: Michael Haun, Dr. Christian Hoene, 
>> Patrick Schreiner Sitz der Gesellschaft/Place of Business: Tübingen 
>> Registereintrag/Commercial Register: Amtsgericht Stuttgart, HRB 
>> 739918
>>
>>
>>
>>
>> _______________________________________________
>> codec mailing list
>> codec@ietf.org
>> https://www.ietf.org/mailman/listinfo/codec
>>
>
>_______________________________________________
>codec mailing list
>codec@ietf.org
>https://www.ietf.org/mailman/listinfo/codec

_______________________________________________
codec mailing list
codec@ietf.org
https://www.ietf.org/mailman/listinfo/codec