Re: [codec] Opus comparision test plan

[Koen Vos <[e-mail]>]

Hi Paul,

> - we need to be consistent about the input filtering. We can't use different
> bandwidths for Opus and AMR-WB.

Nobody uses a low-pass filter to remove the frequencies between 7 and 8 kHz in practice.  So applying such a filter to test material reveals little about real-life performance.

Philosophically, filtering with the smaller frequency response makes little sense either.  Let's say someone built a codec that encodes only frequencies from 1000 to 1100 Hz.  When comparing that codec, should we apply a similar narrow bandpass filter to the inputs of other codecs?  How about if two codecs encoded mutually-non-overlapping parts of the spectrum?

> - I don't understand the point about making sure to tell the Opus encoder
> what the percentage of loss is so it can optimize the encoding for it. What
> happens in practice when the loss percentage isn't known?

Here I agree more with you.  There may be poor implementations that neglect to report packet loss back to the encoder.  For characterization purposes it would be interesting to measure the performance hit from such implementations as compared to ideal implementations.

best,
koen.


________________________________________
From: [e-mail] on behalf of Paul Coverdale
Sent: Tuesday, April 16, 2013 4:27 PM
To: 'Jean-Marc Valin'; 'Christian Hoene'
Cc: [e-mail]; [e-mail]; [e-mail]
Subject: Re: [codec] Opus comparision test plan

Hi Christian,

Yes, I think there is still some work to be done on the plan, I noticed for
example that input level and listening level still need to be defined. And
why are you using MUSHRA? It may be quite time-consuming with a large number
of conditions.

But just to comment on some of the points raised by Jean-Marc:
- we need to be consistent about the input filtering. We can't use different
bandwidths for Opus and AMR-WB.
- I don't understand the point about making sure to tell the Opus encoder
what the percentage of loss is so it can optimize the encoding for it. What
happens in practice when the loss percentage isn't known?


Cheers,

...Paul


>-----Original Message-----
>From: [e-mail] [mailto:[e-mail]] On Behalf
>Of Jean-Marc Valin
>Sent: Tuesday, April 16, 2013 2:36 PM
>To: Christian Hoene
>Cc: [e-mail]; [e-mail];
>[e-mail]
>Subject: Re: [codec] Opus comparision test plan
>
>Christian,
>
>I was only able to look at your plan quickly but still found many issues
>with it. I think you should take your time to fix those and also all the
>ones I'm sure I missed. So far...
>
>== AMR-WB test ==
>
>- You should use 8 kHz bandwidth for Opus. Opus it not restricted to 7
>kHz and low-pass filtering will hurt quality.
>- For AMR-WB the 8.85 mode is "intended to be used only temporarily
>during severe radio channel conditions or during network congestion", so
>you should probably test the 18.25 kbps mode instead.
>- It's not clear from your document, but Opus should use VBR
>- If you're going to test packet loss, there should be at least one test
>with FEC -- probably the 23.85 one
>- Make sure to tell the Opus encoder what the percentage of loss is so
>it can optimize the encoding for it.
>
>== AAC-ELD test ==
>
>- Some of the rates in the test are in the bitrates where both codecs
>are expected to be "falling apart", so we will not learn much there
>(really bad quality vs absolutely terrible doesn't help because people
>don't want either).
>- Some of the rates are way too high for listeners to be able to
>reliably judge. These are guaranteed to end up with everything tied
>unless there's a problem with the test.
>- Using SBR at 128 kb/s for ELD is useless and probably harming quality
>- There is no reason to test music at 24/32 kHz because music is almost
>always available at 48 kHz, so it would hurt to artificially limit the
>signal.
>- When considering CBR vs VBR you have to consider the delay, but also
>the different meanings these have between codecs. I'm not familiar with
>the details, but what most codecs like AAC-LC and MP3 call "CBR"
>actually uses a bit reservoir and is equivalent to Opus "Constrained
>VBR".
>- The choice of music samples is critical. Just that choice can
>sometimes determine the outcome of a music test. You need a wide variety
>of samples to get something statistically significant. It's also
>probably a good idea to use "normal" samples at low rates and "hard"
>samples at high rate.
>- I'd need to recheck the MUSHRA spec, but I'm pretty sure mixing mono
>and stereo in the same session is highly discouraged.
>- Not sure in what form, but it might be worth testing the pre-1.1
>encoder of Opus.
>
>In general, for mono music, I'd recommend restricting yourself to rates
>between 40 and 64 kb/s. For stereo music, I'd stay within 56-96 kb/s.
>Outside of this you're either in the falling apart region or in the
>"nearly transparent for most listeners".
>
>Also, I hope you will fix your methodology from the previous test to
>actually tell the listeners about the hidden reference.
>
>In any case, this is only what I've been able to find with a quick look.
>It's probably good to have wider review once you fix everything.
>
>       Jean-Marc
>
>On 04/16/2013 03:43 AM, Christian Hoene wrote:
>> Hello,
>>
>> as for next week, we place to compare Opus against AMR-WB (with noisy
>> signals) and AAC-eLD. Attached our preliminary test plan. Feel free to
>> criticize it.
>>
>> In addition to the comparison, we will do some Opus characterization
>> (Opus vs. Opus). More to about this test later.
>>
>> With best regards,
>>
>> Christian Hoene
>>
>>
>>
>>
>> Symonics GmbH
>> Sand 13
>> 72076 Tübingen
>> Tel [phone]
>> Fax [phone]
>> Email: [e-mail]
>>
>> Geschäftsführer/Presidents: Michael Haun, Dr. Christian Hoene, Patrick
>> Schreiner Sitz der Gesellschaft/Place of Business: Tübingen
>> Registereintrag/Commercial Register: Amtsgericht Stuttgart, HRB 739918
>>
>>
>>
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>>
>
>_______________________________________________
>codec mailing list
>[e-mail]
>https://www.ietf.org/mailman/listinfo/codec

_______________________________________________
codec mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/codec