Re: [codec] draft test and processing plan for the IETF Codec

[David Virette <[e-mail]>]

Hi Jean-Marc,
The BT and NWT are just intended to gather the meaningful information. Of
course we can ask the listening test sites to report the raw data and the
interested parties can compute all the statistics they feel interesting. But
I think that some people will only want to see the comparison with the
reference codec coming from the requirement document and with the additional
codecs (G.729, G.722,...).
So it is good to list all meaningful comparisons that should be reported by
listening labs. This list of comparison is missing in the current version of
the document, but we will add it. Moreover, I think we should ask the labs
to report the NWT and BT results for all the comparisons, then we will have
the full picture. People will then be able to check the requirements based
on the NWT and everybody should be even more than happy if it finally
appears to be BT.
This kind of comparison for information is also common in ITU-T as it gives
some useful information on the actual codec quality on top of the necessary
results to check the initial requirements.
Best regards,
David

-----Original Message-----
From: [e-mail] [mailto:[e-mail]] On Behalf Of
Jean-Marc Valin
Sent: vendredi 15 avril 2011 12:42
To: Koen Vos
Cc: [e-mail]
Subject: Re: [codec] draft test and processing plan for the IETF Codec

Koen,

The point I was making in the earlier email and that Cullen has stated 
earlier is that we don't even need *any* BT or NWT requirements in this 
document. It will be up to the WG individual participants to decide for 
themselves whether they think it's good to publish Opus, based on the 
information available at the time they make the decision. So Anisse's 
test plan proposal is meant to "gather useful data", not decide on 
whether to publish.

	Jean-Marc

On 11-04-15 04:44 AM, Koen Vos wrote:
> I would also suggest replacing all BT (better than) requirements by NWT
(no worse than).
>
> My reasoning is that:
> - The WG never had the goal to be better than other codecs (see charter).
> - Proving to be better can be very hard, especially when several codecs
are close to transparent.  To show significance in that case you'd need a
vast number of listeners, which makes a test more cumbersome to perform.
>
> best,
> koen.
>
>
> ----- Original Message -----
> From: "Jean-Marc Valin"<[e-mail]>
> To: "Jean-Marc Valin"<[e-mail]>
> Cc: [e-mail]
> Sent: Thursday, April 14, 2011 9:49:58 PM
> Subject: Re: [codec] draft test and processing plan for the IETF Codec
>
> So here's some more specific comments on actual bitrates:
>
> 1) For narrowband Speex, the rates currently listed are 8, 12, 16 kb/s.
> Those should be changed to 8, 11, 15 kb/s to match the actual Speex
> bitrates.
>
> 2) For iLBC, the rates currently listed are 8, 12, 16 kb/s. I think we
> should only use 15.2 kb/s for iLBC. There's another rate, which is 13.33
> kb/s but that's for 30 ms frames so it's not very interesting.
>
> 3) For Speex wideband, the rates currently listed are 12, 24, 32 kb/s. I
> think Speex wideband around 12 kb/s is just crap. Worth testing would be
> 20.6 and 27.8 kb/s.
>
> 4) For super-wideband Speex, I recommend just dumping that. This Speex
> mode was a mistake right from the start and usually has worse quality
> than wideband Speex.
>
> Regarding super-wideband, one thing to keep in mind is that Opus defines
> super-wideband as having a 12 kHz audio bandwidth (24 kHz sampling
> rate). This makes comparisons with other codecs more difficult. The
> rates currently listed for super-wideband are 24, 32, 64 kb/s. I
> recommend running 24 kb/s in super-wideband and running 32 and 64 kb/s
> in fullband mode (even if the input is a 32 kHz signal).
>
> For the very low delay tests (10 ms frame size), I think all the listed
> rates should be using fullband mode except the 32 kb/s.
>
> That's it for now. Any thoughts?
>
> 	Jean-Marc
>
>
> On 11-04-14 11:16 PM, Jean-Marc Valin wrote:
>> Hi Anisse,
>>
>> I gave some more thought on your proposed test plan and as Cullen
>> suggested, I think the main cause of disagreement is not that much on
>> the testing, but on the conditions for publishing (large number of BT,
>> NWT). Considering that ultimately, the decision to publish a spec is
>> always based on WG consensus, then I think that problem can be
>> completely bypassed. Once we make it up to the individuals to decide,
>> then we can focus on "simply" designing a good test.
>>
>> Overall I thought the conditions you were proposing in section 2 were
>> pretty reasonable. There's a few details like selecting existing rates
>> for codecs like Speex and iLBC, but that should be easy to solve. Once
>> these are sorted out, interested parties (we had several hands raised in
>> the last meeting) can start testing and we then let each individual
>> decide on whether the codec is any good based on the results of the
tests.
>>
>> Sounds like a plan?
>>
>> Jean-Marc
>>
>>
>> On 11-04-13 03:32 AM, Anisse Taleb wrote:
>>> Hi,
>>> Please find attached a first draft of a test plan of the IETF codec
>>> (Opus).
>>> The proposal does not claim to be complete, there are still many
>>> missing things, e.g. tandeming cases, tests with delay jitter, dtx
>>> etc. Consider it as a starting point for discussion where everyone is
>>> welcome to contribute in a constructive manner. Further updates are
>>> planned, but let's see first some initial comments.
>>>
>>> The attachment is a pdf version, please let me know if you would like
>>> to see another format and I would be glad to oblige.
>>>
>>> Comments and additions are welcome!
>>>
>>> Kind regards,
>>> /Anisse
>>> (From La Jolla - San Diego).
>>>
>>>
>>>
>>> _______________________________________________
>>> codec mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/codec
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>>
>>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
>
>
_______________________________________________
codec mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/codec