Re: [codec] A concrete proposal for requirements and testing

[Stephen Botzko <[e-mail]>]

in-line
Stephen Botzko

On Tue, Apr 12, 2011 at 1:35 PM, Jean-Marc Valin <[e-mail]> wrote:

> Hi Stephen,
>
>
> On 11-04-12 07:16 AM, Stephen Botzko wrote:
>
>> On the whole, the thread started reasonably, with good discussions on
>> adding some speech conditions, choosing the number of subjects, etc. The
>> thread diverged when AMR was proposed as an anchor in section 4.2.
>> Jean-Marc "strongly disagreed" to testing requirements against AMR-NB
>> and AMR-WB, saying it was "not fair",
>>
>
> Correct so far.
>
>
>  and suggesting that he thought
>> Opus would fail in that comparison (though he thought it would be
>> close).
>>
>
> Not quite. I actually believe Opus sounds better than AMR-WB since that's
> the conclusion from both the Google test and the earlier Dynastat test on
> SILK. I have no opinion on AMR-NB because I haven't listened to that codec
> enough to have an opinion on how good it sounds. It still remains that
> there's no point in having "no worse than AMR-*" as a strict requirement,
> not only because of the IPR, but also because the codec we're working on
> does much more than each of these codecs do. I assume that's the same reason
> why G.729.1 didn't compare with AMR-NB at all and used a *much* lower rate
> for its AMR-WB comparison.


Sorry if I misunderstood your original comment, it was not intentional.
>>>
     Requiring Opus to also beat all codecs out there in all circumstances
is thus totally unreasonable (despite the fact that the tests we did so far
indicate we may be close to that).
>>>
BTW, I do agree that the WG was not chartered to create a codec that beats
all codecs out there at all operating points.  As I recall, we wanted a good
codec, but were prepared to lose some quality to [hopefully] avoid
encumbrances.

ITU-T test plans don't cover all possible anchors (it simply costs too
much).   And the bit rates of the anchors can be lower than the codec under
test.  Quality is only one of the requirements, complexity is also a
requirement - so a codec with lower quality but also lower complexity can be
proposed and developed there.  It is possible to apply that same idea here -
agreeing on an anchor that may out-perform Opus for a narrow range of
operating points, but also agreeing to give Opus a reasonable bit-rate
advantage for the "not worse" test condition.


>
>
>  He suggested that only unencumbered codecs be used as anchors
>> for strict requirements. [He did _not_ object to the use of encumbered
>> codecs in testing objectives, just in testing "strict requirements"].
>>
>
> Correct.
>
>        Jean-Marc
>