Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

[Jean-Marc Valin <[e-mail]>]

On 11-01-25 10:59 PM, Stephan Wenger wrote:
> But please add language in the requirement doc to the extent that there
> MUST be a conformance spec for each candidate codec that allows to
> automatically determine the conformance of a codec implementation.  Absent
> this spec, the IPR statements received are meaningless.

IIRC, the guidelines already say that we need a conformance test.

> My preference would be to have that conformance spec in the same document
> as the codec, so to make citations in IPR statements easy and to avoid a
> charter-update.

Agreed. We really want the conformance tool in the same document as the 
codec itself.

Cheers,

	Jean-Marc

> Stephan
>
>
> On 1.25.2011 18:24 , "Raymond (Juin-Hwey) Chen"<[e-mail]>  wrote:
>
>> My opinion is that we should not require bit-exactness.  There are many
>> examples of previous codec
>> standards that did not require bit-exactness, including both
>> royalty-bearing codecs (such as MPEG)
>> and royalty-free codecs (such as our BroadVoice/BV16/BV32), so I don't
>> see any real reason why we
>> can't follow that same route.
>>
>> I can see at least two important advantages for not requiring
>> bit-exactness:
>> (1) it allows future innovations to improve the performance of the codec
>> without breaking the bit-
>> stream compatibility, and
>> (2) it allows the codec implementers the greatest flexibility in
>> implementing the codecs in ways
>> that are most efficient for the particular hardware or processor
>> architecture they have.
>> Having a bit-exact standard will kill both.
>>
>> The main drawback of a non-bit-exact standard is that the verification of
>> the conformance of codec
>> implementations is not as straightforward as with a bit-exact standard.
>> However, I don't think
>> that is a show stopper or even a difficult problem.  There are many
>> possible ways to do software-
>> based objective measurements to verify that a candidate codec
>> implementation is likely to sound
>> essentially indistinguishable from the output of the reference codec.  We
>> had previously measured
>> the SNRs and PESQ scores of a large number of test files using the
>> reference codec and candidate
>> codec and examined the statistical distribution of the differences in the
>> scores.  Jean-Marc just
>> posted a nice little Matlab code that built upon psychoacoustic modeling,
>> NMR, averages, outliers
>> checking, etc.  Koen also mentioned another tool.  Even if these tools
>> and methodologies are not
>> perfect yet, with some additional tuning of their algorithms and
>> parameters, it is really not that
>> difficult to come up with objective, software-based verification methods
>> that will give you a high
>> level of confidence whether a candidate codec will sound at least as good
>> as the reference codec
>> (or perhaps better if improvements are made).
>>
>> I like Jean-Marc's idea of having both a floating-point reference codec
>> and a fixed-point
>> reference codec. For people who don't want to run the risk of degrading
>> audio quality through
>> their own fixed-point algorithm conversion, they can just implement the
>> fixed-point reference
>> codec in a bit-exact manner.  For people who want to implement the codec
>> in floating-point (e.g.
>> on computers), the easiest way is just to use the floating-point
>> reference code as is.  In both
>> cases, the conformance verification will be quite easy.
>>
>> Raymond
>>
>> -----Original Message-----
>> From: [e-mail] [mailto:[e-mail]] On Behalf Of
>> Jean-Marc Valin
>> Sent: Tuesday, January 25, 2011 12:58 PM
>> To: Koen Vos
>> Cc: [e-mail]; Stephen Botzko
>> Subject: Re: [codec] requirements #12 (closed): bit-exact vs.
>> bit-compatible?
>>
>> My preference actually goes with using both fixed-point and floating
>> point.
>> That way, implementers of floating-point decoder can compare with the
>> float
>> version and implementers of the fixed-point decoder can compare with the
>> fixed-point version.
>>
>> This is not unheard of either, if you look at the reference implementation
>> of the 3GPP(2?) EVRC codec, the distribution includes two sets of output
>> vectors. One is for a fixed-point implementation that uses 31-bit
>> precision
>> in the multiplications and the other one is for 32-bit precision. You just
>> choose whatever is more efficient for your DSP and use the appropriate
>> testvectors for testing.
>>
>>         Jean-Marc
>>
>> On 11-01-25 03:37 PM, Koen Vos wrote:
>>> A similar tool is in the code in silk\test\signalCompare.c.  It's based
>>> on LPC, and measures the SNR in a partially-whitened domain.  For music
>>> the critical-band approach of Jean-Marc's tool may work better, not sure.
>>>
>>> One question is about what implementation should be the reference for
>>> compliance testing: floating-point or fixed-point?
>>>
>>> A fixed-point reference allows for bit-exact implementations, which is
>>> nice.  And fixed-point reference implementations are the norm in speech
>>> coding.
>>> However, CELT's floating-point implementation has a larger dynamic
>>> range than the fixed-point version, and is thus arguable better.  A
>>> fixed-point reference could not test this extra dynamic range.
>>>
>>> Any suggestions?
>>> koen.
>>>
>>>
>>> ----- Original Message -----
>>> From: "Jean-Marc Valin"<[e-mail]>
>>> To: "Stephen Botzko"<[e-mail]>
>>> Cc: [e-mail]
>>> Sent: Tuesday, January 25, 2011 8:29:25 AM
>>> Subject: Re: [codec] requirements #12 (closed): bit-exact vs.
>>> bit-compatible?
>>>
>>> Just to give an idea of what I'm talking about in terms of compliance
>>> test,
>>> here's a tool I just wrote that could be used for testing. There may
>>> need
>>> to be a bit of refining and playing with the thresholds, but the main
>>> idea
>>> is there:
>>>
>>> http://jmvalin.ca/misc_stuff/compare.m
>>>
>>> It works with Octave, and is likely very easy to get to work with
>>> Matlab.
>>> Any comments?
>>>
>>>        Jean-Marc
>>>
>>> On 11-01-25 10:57 AM, Jean-Marc Valin wrote:
>>>> Hi Stephen,
>>>>
>>>> The reference implementation includes both floating-point and
>>>> fixed-point,
>>>> so I'm not sure it's a good idea to say that one is "better" than the
>>>> other. As for IEEE 754-2008, as far as I know it is not a "bit-exact"
>>>> standard either when it comes to some operations (e.g. transcendental
>>>> function) and even for other operations, most compilers I know of are
>>>> not
>>>> strict IEEE-754 (at least by default) because of issues such as the x86
>>>> extended precision operations.
>>>>
>>>> Regarding MPEG bit-streams, I'm not sure where to get confirmation, but
>>>> this is what Wikipedia has to say about MPEG-1 Layer 3:
>>>>
>>>>
>>>> "Decoding, on the other hand, is carefully defined in the standard.
>>>> Most
>>>> decoders are "bitstream compliant", which means that the decompressed
>>>> output – that they produce from a given MP3 file – will be the
>>>> same, within
>>>> a specified degree of rounding tolerance, as the output specified
>>>> mathematically in the ISO/IEC standard document (ISO/IEC 11172-3).
>>>> Therefore, comparison of decoders is usually based on how
>>>> computationally
>>>> efficient they are (i.e., how much memory or CPU time they use in the
>>>> decoding process)."
>>>>
>>>>
>>>> That implies a standard based on "infinite precision", with a degree of
>>>> tolerance for different implementations.
>>>>
>>>> Cheers,
>>>>
>>>> Jean-Marc
>>>>
>>>> On 11-01-25 10:42 AM, Stephen Botzko wrote:
>>>>> It seems to me that the reference encoder and decoder will be bit
>>>>> exact for
>>>>> a given floating point format, correct? So one could specify IEEE
>>>>> 754-2008
>>>>> when bit exactness is needed in the reference (for instance regression
>>>>> testing), but not require it for compliance.
>>>>>
>>>>> Folks who are modifying the reference encoder or decoder algorithms
>>>>> are "on
>>>>> their own" as far as audio quality is concerned. Though I agree with
>>>>> Stephan that we need to address compliance (when exactly does a
>>>>> ported or
>>>>> otherwise modified encoder/decoder become non-compliant?).
>>>>>
>>>>> Stephen Botzko
>>>>>
>>>>> On Tue, Jan 25, 2011 at 9:34 AM, Roman Shpount<[e-mail]
>>>>> <mailto:[e-mail]>>   wrote:
>>>>>
>>>>> One more concern that I have related to bit exactness is codec
>>>>> regression testing. One can produce a non-bit-exact encoder that
>>>>> produces reasonable results for a reference decoder, but if paired
>>>>> with
>>>>> a modified, non-bit-exact decoder will produce significant audio
>>>>> artifacts. Since neither decoder or encoder are bit exact we will need
>>>>> to have a test procedure that will validate that both encoder or
>>>>> decoder will not break any standard compliant and non-bit-exact
>>>>> encoders or decoders. Not really sure how this can be done.
>>>>>
>>>>> I might be wrong, but I think MPEG decoders are bit exact and encoders
>>>>> are not.
>>>>> _____________
>>>>> Roman Shpount
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jan 25, 2011 at 12:03 AM, Jean-Marc Valin<[e-mail]
>>>>> <mailto:[e-mail]>>   wrote:
>>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> I understand your concern and I'd be interested if you have
>>>>> alternative ways of handling the licensing to avoid any issue. It's
>>>>> not like this is a unique situation. As far as I know, most (all?)
>>>>> MPEG codecs have similar non-bit exact definitions. I have also
>>>>> heard that they also require some IPR licensing...
>>>>>
>>>>> In general the issue of bit-exactness has been discussed and so far
>>>>> I don't recall many arguing in favor of a bit-exact definition.
>>>>> Most of the concerns that have been expressed are solved by
>>>>> considering that non-bitexact does not mean you cannot be bit-exact
>>>>> with the reference encoder. It merely means that you don't *have*
>>>>> to. So regardless of how conformance is defined exactly, one always
>>>>> has the option of being bit-exact with the reference
>>>>> implementation, which obviously guarantees compliance.
>>>>>
>>>>> As for language mentioning compliance, I believe it belongs more to
>>>>> the guidelines (it's not a requirement of the codec itself), which
>>>>> includes the following text:
>>>>>
>>>>> 4. To reduce the risk of bias towards certain CPU/DSP architectures,
>>>>> ideally the decoder specification should not require "bit-exact"
>>>>> conformance with the reference implementation. The output of a
>>>>> decoder implementation should only be "close enough" to the
>>>>> output of the reference decoder. A comparison tool should be
>>>>> provided along with the codec to verify objectively that the
>>>>> output of a decoder is likely to be perceptually
>>>>> indistinguishable from that of the reference decoder. However,
>>>>> an implementation may still wish to produce an output that is
>>>>> bit-exact with the reference implementation to simplify the
>>>>> testing procedure.
>>>>>
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Jean-Marc
>>>>>
>>>>>
>>>>> On 11-01-24 11:02 PM, Stephan Wenger wrote:
>>>>>
>>>>> Hi all:
>>>>>
>>>>> Let me speak once more against this decision (if such a
>>>>> decision were
>>>>> really made; see the p.s.).
>>>>>
>>>>> There are currently three IPR disclosures against the codec
>>>>> draft and/or
>>>>> its predecessors. The Xiph disclosure is at this point a
>>>>> placeholder
>>>>> (Xiph folks: it's time to fix that!). However, the two other
>>>>> disclosures
>>>>> on file provide a patent grant only for necessary patent claims
>>>>> and only
>>>>> when the standard is practiced in full compliance. These terms (in
>>>>> various formulations) are quite common.
>>>>>
>>>>> In order to ensure one has a license (or can rely on a non-assert
>>>>> covenant), one has to ensure one meets the conditions set by the
>>>>> rightholder. On stuff such as reciprocity clauses this is
>>>>> simple. On
>>>>> compliance, it's not always easy.
>>>>>
>>>>> The traditional compliance test for a media codec is a
>>>>> stimulus-response
>>>>> test: you feed test vectors into the codec, and you get
>>>>> results. If the
>>>>> results match, you are in compliance, if not, you are not. Simple.
>>>>>
>>>>> Without bit exactness, the compliance criteria have to be defined
>>>>> differently. We can do so, and, indeed, I recall that this has been
>>>>> mentioned as one plan forward. However, I have seen zero
>>>>> activity in this
>>>>> direction, and I have also not seen any language that mentions
>>>>> this in the
>>>>> requirements draft. I think that the subject of compliance
>>>>> tests, at
>>>>> least in its most basic outline, needs to be documented in the
>>>>> requirements draft. The details can be taken care of elsewhere
>>>>> and later,
>>>>> but not too much later. It should be clear that a codec
>>>>> candidate (if
>>>>> there were more than one) needs to have compliance criteria
>>>>> defined before
>>>>> that codec candidate can become an RFC. Without that, the key
>>>>> goal of the
>>>>> WG, a reasonably freely practicable codec, is just not
>>>>> achievable in the
>>>>> current legal environment (which includes, in this case, the IPR
>>>>> disclosures on file).
>>>>>
>>>>> Of course, it would be sooooo much simpler if we would mandate
>>>>> a bit exact
>>>>> decoder... Is it really that restricting to require that?
>>>>>
>>>>> Stephan
>>>>>
>>>>> P.s.: for the IETF procedures newcomers: humms taken at
>>>>> meetings need to
>>>>> be confirmed on a mailing list, and consensus needs to be
>>>>> declared by the
>>>>> chairs. On this subject, I do recall mailing list discussions after
>>>>> Maastricht, but I do not recall that consensus was reached, yet
>>>>> alone
>>>>> declared. (Unfortunately, I currently don't have the time to go
>>>>> through
>>>>> the mailing list archives to verify my recollection; Sorry.)
>>>>>
>>>>>
>>>>>
>>>>> On 1.24.2011 16:45 , "codec issue tracker"<[e-mail]
>>>>> <mailto:[e-mail]>>   wrote:
>>>>>
>>>>> #12: bit-exact vs. bit-compatible?
>>>>>
>>>>> Changes (by gmaxwell@Ã… ):
>>>>>
>>>>> * status: new =>   closed
>>>>> * resolution: =>   worksforme
>>>>>
>>>>>
>>>>> Comment:
>>>>>
>>>>> http://www.ietf.org/proceedings/78/minutes/codec.txt
>>>>>
>>>>> "On the topic of bit exact. Consensus was bit exactness is
>>>>> not required."
>>>>>
>>>>> I believe this issue is already closed.
>>>>>
>>>>> --
>>>>>
>>>>> ------------------------------------+----------------------------------
>>>>> ---
>>>>> --
>>>>> Reporter: hoene@Ã… | Owner:
>>>>> Type: enhancement | Status: closed
>>>>> Priority: minor | Milestone:
>>>>> Component: requirements | Version:
>>>>> Severity: Active WG Document | Resolution: worksforme
>>>>> Keywords: |
>>>>>
>>>>> ------------------------------------+----------------------------------
>>>>> ---
>>>>> --
>>>>>
>>>>> Ticket
>>>>> URL:<http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1>
>>>>> codec<http://tools.ietf.org/codec/>
>>>>>
>>>>> _______________________________________________
>>>>> codec mailing list
>>>>> [e-mail]<mailto:[e-mail]>
>>>>> https://www.ietf.org/mailman/listinfo/codec
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> codec mailing list
>>>>> [e-mail]<mailto:[e-mail]>
>>>>> https://www.ietf.org/mailman/listinfo/codec
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> codec mailing list
>>>>> [e-mail]<mailto:[e-mail]>
>>>>> https://www.ietf.org/mailman/listinfo/codec
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> codec mailing list
>>>>> [e-mail]<mailto:[e-mail]>
>>>>> https://www.ietf.org/mailman/listinfo/codec
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> codec mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/codec
>>>>
>>>> _______________________________________________
>>>> codec mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/codec
>>>
>>> _______________________________________________
>>> codec mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/codec
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec