Re: [codec] A concrete proposal for requirements and testing

[Jean-Marc Valin <[e-mail]>]

Hi Roni,

On 11-04-09 07:39 AM, Roni Even wrote:
> As for telepresence HE-AAC is not the codec currently in use so comparing to
> it does not qualify as getting better quality

Thanks very much for pointing out that HE-AAC is not a telepresence 
codec. As you know, when we design a codec for interactive applications, 
we have to make sacrifices (e.g. in quality) to achieve a low delay and 
Opus is no exception. On the other hand HE-AAC makes no such sacrifices 
and is considered by many (including the people who ran that test) as 
the best non-interactive codec for music at 64 kb/s.

The people who ran the HA listening test are only interested in highest 
quality at a given bitrate, which is why they selected HE-AAC and 
Vorbis. We're just glad they accepted to include Opus in their 
comparison, despite the test not being fair to Opus (other codecs have a 
much higher delay). Despite all that, it turned out that Opus 
out-performed all HE-AAC encoders on a wide variety of audio samples. 
Considering that low-delay flavors of AAC encoders (which we have never 
been able to obtain to test against) have to make the quality/delay 
trade-offs I mentioned, I'm pretty confident that Opus would do well 
against them -- despite this not being a requirement.

Cheers,

	Jean-Marc

> Roni Even
>
>> -----Original Message-----
>> From: [e-mail] [mailto:[e-mail]] On Behalf
>> Of Ron
>> Sent: Saturday, April 09, 2011 12:57 PM
>> To: [e-mail]
>> Subject: Re: [codec] A concrete proposal for requirements and testing
>>
>>
>> Hi Roni,
>>
>> On Sat, Apr 09, 2011 at 10:14:06AM +0300, Roni Even wrote:
>>> Hi,
>>>
>>> According to the proponents there is nothing preventing you from
>> using OPUS
>>> for testing or in your product. What is being discussed is if the
>> current
>>> version of the codec answers the current charter of the CODEC WG.
>>
>> Well the RF IPR grants only permit us to do that once we've reached the
>> stage of publishing a definitive RFC.  The "internal" testing of course
>> has already taken place, within the bounds that Koen and others
>> clarified
>> for interim use of their technology.  That we aren't seeing any more
>> real
>> feedback from that testing says to me that it has achieved what it is
>> likely to in the short term.  And now we should put out a request for
>> comments to a broader audience, to solicit feedback from newer eyes.
>>
>> But yes, the question now is indeed, have we fallen short of some part
>> of the charter, that so far has been missed.  I'm no real expert on
>> IETF procedure, but I see the "last call" as exactly that.  Time to
>> point out whatever you think others have so far missed, while there is
>> still time to fix the draft documents.
>>
>>> I see in the email discussion two issues which may need to be
>> resolved.
>>>
>>> The first is if the codec meets the requirements, but the
>> requirements are
>>> not finalized yet. I can see that agreement can be reached if there
>> is a
>>> focus to finish the requirement document.
>>
>> We do have a draft requirements document.  So I guess it would be
>> helpful
>> if people can point out exactly what sections they don't like, and what
>> they'd like to replace it with, if anything.
>>
>> I'm not aware of the editors being behind with any proposed changes
>> that there is consensus for, are they?
>>
>>> Note that the objectives are "
>>> Designing for use in interactive applications (examples include, but
>> are not
>>> limited to, point-to-point voice calls, multi-party voice
>> conferencing,
>>> telepresence, teleoperation, in-game voice chat, and live music
>>> performance)" so there are claims that you need to compare OPUS to
>> codecs
>>> used in this application spaces.
>>
>> Are you aware of, and/or proposing an alternate codec, that you think
>> meets the group charter better than our candidate codec?  I'd have
>> thought the time to mention that would now be long past, but if you
>> have one that we've missed, then please, I'd love to know, what is
>> its name, and where do I download its reference implementation?
>>
>> That would certainly be an important piece of new information that
>> the group should consider.
>>
>> But since I'm not aware of one, I can only read that as saying that
>> the candidate must indeed be suitable "for use in interactive etc."
>>
>> I think the charter also says that if it isn't, we may need to make
>> another codec.  But I'm not aware of anyone else seeing the need for
>> that just yet, are you?
>>
>>
>>> Another way to address it is to limit the applications in the
>> charter.
>>
>> Given that we're now rumbling toward last call, could you please be
>> explicit about which of the applications you think we have failed at,
>> and what evidence you have to support that.  It will be much easier
>> to address them that way.
>>
>>> One example may be to remove the telepresence application case.
>>
>> For instance I'm not sure I follow how you think we may have failed
>> at that use case?
>>
>> requirements 2.3 says:
>>
>>   ...
>>   In addition, telepresence applications require super-wideband and
>>   full-band audio capability with useful bit-rates in the 32 - 80 kb/s
>>   range.  While voice is still the most important signal to be encoded,
>>   it must be possible to obtain good quality (even if not transparent)
>>   music.
>>   ...
>>
>> Given that Opus just trumped HE-AAC in an unconstrained blind test
>> at 64kb/s (stereo), got mistaken for the reference, and can do 2.5ms
>> latency, which bit do you think we need to improve for telepresence?
>>
>>
>>
>>> The second issue is the objective that  "The goal of this working
>> group is
>>> to ensure the existence of a single high-quality audio codec that is
>>> optimized for use over the Internet and that can be widely
>> implemented and
>>> easily distributed among application developers, service operators,
>> and end
>>> users." Here again we may need to address the new IPR claims.
>>
>> I think Stephan said he'd prefer people to talk about that elsewhere,
>> and afaik they are.  That's still a goal, and to the best of my
>> knowledge
>> we are still on target to achieve it by the time we've dotted the t's
>> and
>> crossed the i's on the rest of this paperwork.
>>
>>> It can be done
>>> by re-charter or by changing the codec around the IPR. In the mean
>> time some
>>> view that in the current state, the OPUS codec should be compared to
>>> existing royalty baring codecs.
>>
>> I think it's on the record under note well that xiph will pull its IPR
>> if
>> RF status cannot be achieved, so I don't think we need to worry about
>> that.
>> We can't guarantee there will not be further vexatious claims, but we
>> can
>> be sure that isn't a possibility this group will need to consider in
>> the
>> context of finalising the Opus candidate we have.
>>
>> Opus only need be compared to other codecs that this group could
>> endorse
>> as being superior to it at meeting the charter objectives.  If you
>> really
>> have one, then maybe we do have a contest after all, but otherwise, the
>> only question remaining is did we fail at a goal, and what can we do
>> about fixing that.
>>
>> It doesn't matter if moonrocks are prettier than the ones I have in my
>> garden, they're still never going to get used there.  So please, if
>> we've failed at the charter, I *really* want to know.  Starting with
>> paragraph and verse.  But there's no point comparing apples and
>> oranges.
>>
>> The answer to that is simple.  Oranges always win. :)
>>
>> Cheers,
>> Ron
>>
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
>
>