Re: [codec] draft test and processing plan for the IETF Codec

Dear Jean-Marc,
Thanks for the information on available bitrates for speex and iLBC, it will
help updating the test plan.

Regarding the super-wideband definition, it is really common to have
different bandwidths in a MUSHRA test, and it is not the only factor which
guides the listeners to score the codecs. If you look at the test plan, we
proposed to test the super-wideband and full-band in the same test using
some different band-limited signals as anchors. From my experience, it is
less than obvious that this difference between 12 and 14 kHz will really
affect the ranking of codec for speech content.
Your proposed bitrate for super-wideband and full band are also noted.
Best regards,
David

David Virette
HUAWEI TECHNOLOGIES CO.,LTD. 

Building C
Riesstrasse 25
80992 Munich, Germany
Tel: +49 89 158834 4148
Fax: +49 89 158834 4447
Mobile: +49 1622047469
E-mail: david.virette@huawei.com
www.huawei.com
----------------------------------------------------------------------------
---------------------------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which 
is intended only for the person or entity whose address is listed above. Any
use of the 
information contained herein in any way (including, but not limited to,
total or partial 
disclosure, reproduction, or dissemination) by persons other than the
intended 
recipient(s) is prohibited. If you receive this e-mail in error, please
notify the sender by 
phone or email immediately and delete it!

-----Original Message-----
From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] On Behalf Of
Jean-Marc Valin
Sent: vendredi 15 avril 2011 06:50
To: Jean-Marc Valin
Cc: codec@ietf.org
Subject: Re: [codec] draft test and processing plan for the IETF Codec

So here's some more specific comments on actual bitrates:

1) For narrowband Speex, the rates currently listed are 8, 12, 16 kb/s. 
Those should be changed to 8, 11, 15 kb/s to match the actual Speex 
bitrates.

2) For iLBC, the rates currently listed are 8, 12, 16 kb/s. I think we 
should only use 15.2 kb/s for iLBC. There's another rate, which is 13.33 
kb/s but that's for 30 ms frames so it's not very interesting.

3) For Speex wideband, the rates currently listed are 12, 24, 32 kb/s. I 
think Speex wideband around 12 kb/s is just crap. Worth testing would be 
20.6 and 27.8 kb/s.

4) For super-wideband Speex, I recommend just dumping that. This Speex 
mode was a mistake right from the start and usually has worse quality 
than wideband Speex.

Regarding super-wideband, one thing to keep in mind is that Opus defines 
super-wideband as having a 12 kHz audio bandwidth (24 kHz sampling 
rate). This makes comparisons with other codecs more difficult. The 
rates currently listed for super-wideband are 24, 32, 64 kb/s. I 
recommend running 24 kb/s in super-wideband and running 32 and 64 kb/s 
in fullband mode (even if the input is a 32 kHz signal).

For the very low delay tests (10 ms frame size), I think all the listed 
rates should be using fullband mode except the 32 kb/s.

That's it for now. Any thoughts?

	Jean-Marc

On 11-04-14 11:16 PM, Jean-Marc Valin wrote:
> Hi Anisse,
>
> I gave some more thought on your proposed test plan and as Cullen
> suggested, I think the main cause of disagreement is not that much on
> the testing, but on the conditions for publishing (large number of BT,
> NWT). Considering that ultimately, the decision to publish a spec is
> always based on WG consensus, then I think that problem can be
> completely bypassed. Once we make it up to the individuals to decide,
> then we can focus on "simply" designing a good test.
>
> Overall I thought the conditions you were proposing in section 2 were
> pretty reasonable. There's a few details like selecting existing rates
> for codecs like Speex and iLBC, but that should be easy to solve. Once
> these are sorted out, interested parties (we had several hands raised in
> the last meeting) can start testing and we then let each individual
> decide on whether the codec is any good based on the results of the tests.
>
> Sounds like a plan?
>
> Jean-Marc
>
>
> On 11-04-13 03:32 AM, Anisse Taleb wrote:
>> Hi,
>> Please find attached a first draft of a test plan of the IETF codec
>> (Opus).
>> The proposal does not claim to be complete, there are still many
>> missing things, e.g. tandeming cases, tests with delay jitter, dtx
>> etc. Consider it as a starting point for discussion where everyone is
>> welcome to contribute in a constructive manner. Further updates are
>> planned, but let's see first some initial comments.
>>
>> The attachment is a pdf version, please let me know if you would like
>> to see another format and I would be glad to oblige.
>>
>> Comments and additions are welcome!
>>
>> Kind regards,
>> /Anisse
>> (From La Jolla - San Diego).
>>
>>
>>
>> _______________________________________________
>> codec mailing list
>> codec@ietf.org
>> https://www.ietf.org/mailman/listinfo/codec
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
>
>
_______________________________________________
codec mailing list
codec@ietf.org
https://www.ietf.org/mailman/listinfo/codec