Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

[Jean-Marc Valin <[e-mail]>]

My preference actually goes with using both fixed-point and floating point. 
That way, implementers of floating-point decoder can compare with the float 
version and implementers of the fixed-point decoder can compare with the 
fixed-point version.

This is not unheard of either, if you look at the reference implementation 
of the 3GPP(2?) EVRC codec, the distribution includes two sets of output 
vectors. One is for a fixed-point implementation that uses 31-bit precision 
in the multiplications and the other one is for 32-bit precision. You just 
choose whatever is more efficient for your DSP and use the appropriate 
testvectors for testing.

	Jean-Marc

On 11-01-25 03:37 PM, Koen Vos wrote:
> A similar tool is in the code in silk\test\signalCompare.c.  It's based on LPC, and measures the SNR in a partially-whitened domain.  For music the critical-band approach of Jean-Marc's tool may work better, not sure.
>
> One question is about what implementation should be the reference for compliance testing: floating-point or fixed-point?
>
> A fixed-point reference allows for bit-exact implementations, which is nice.  And fixed-point reference implementations are the norm in speech coding.
> However, CELT's floating-point implementation has a larger dynamic range than the fixed-point version, and is thus arguable better.  A fixed-point reference could not test this extra dynamic range.
>
> Any suggestions?
> koen.
>
>
> ----- Original Message -----
> From: "Jean-Marc Valin"<[e-mail]>
> To: "Stephen Botzko"<[e-mail]>
> Cc: [e-mail]
> Sent: Tuesday, January 25, 2011 8:29:25 AM
> Subject: Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?
>
> Just to give an idea of what I'm talking about in terms of compliance test,
> here's a tool I just wrote that could be used for testing. There may need
> to be a bit of refining and playing with the thresholds, but the main idea
> is there:
>
> http://jmvalin.ca/misc_stuff/compare.m
>
> It works with Octave, and is likely very easy to get to work with Matlab.
> Any comments?
>
> 	Jean-Marc
>
> On 11-01-25 10:57 AM, Jean-Marc Valin wrote:
>> Hi Stephen,
>>
>> The reference implementation includes both floating-point and fixed-point,
>> so I'm not sure it's a good idea to say that one is "better" than the
>> other. As for IEEE 754-2008, as far as I know it is not a "bit-exact"
>> standard either when it comes to some operations (e.g. transcendental
>> function) and even for other operations, most compilers I know of are not
>> strict IEEE-754 (at least by default) because of issues such as the x86
>> extended precision operations.
>>
>> Regarding MPEG bit-streams, I'm not sure where to get confirmation, but
>> this is what Wikipedia has to say about MPEG-1 Layer 3:
>>
>>
>> "Decoding, on the other hand, is carefully defined in the standard. Most
>> decoders are "bitstream compliant", which means that the decompressed
>> output – that they produce from a given MP3 file – will be the same, within
>> a specified degree of rounding tolerance, as the output specified
>> mathematically in the ISO/IEC standard document (ISO/IEC 11172-3).
>> Therefore, comparison of decoders is usually based on how computationally
>> efficient they are (i.e., how much memory or CPU time they use in the
>> decoding process)."
>>
>>
>> That implies a standard based on "infinite precision", with a degree of
>> tolerance for different implementations.
>>
>> Cheers,
>>
>> Jean-Marc
>>
>> On 11-01-25 10:42 AM, Stephen Botzko wrote:
>>> It seems to me that the reference encoder and decoder will be bit exact for
>>> a given floating point format, correct? So one could specify IEEE 754-2008
>>> when bit exactness is needed in the reference (for instance regression
>>> testing), but not require it for compliance.
>>>
>>> Folks who are modifying the reference encoder or decoder algorithms are "on
>>> their own" as far as audio quality is concerned. Though I agree with
>>> Stephan that we need to address compliance (when exactly does a ported or
>>> otherwise modified encoder/decoder become non-compliant?).
>>>
>>> Stephen Botzko
>>>
>>> On Tue, Jan 25, 2011 at 9:34 AM, Roman Shpount<[e-mail]
>>> <mailto:[e-mail]>>  wrote:
>>>
>>> One more concern that I have related to bit exactness is codec
>>> regression testing. One can produce a non-bit-exact encoder that
>>> produces reasonable results for a reference decoder, but if paired with
>>> a modified, non-bit-exact decoder will produce significant audio
>>> artifacts. Since neither decoder or encoder are bit exact we will need
>>> to have a test procedure that will validate that both encoder or
>>> decoder will not break any standard compliant and non-bit-exact
>>> encoders or decoders. Not really sure how this can be done.
>>>
>>> I might be wrong, but I think MPEG decoders are bit exact and encoders
>>> are not.
>>> _____________
>>> Roman Shpount
>>>
>>>
>>>
>>> On Tue, Jan 25, 2011 at 12:03 AM, Jean-Marc Valin<[e-mail]
>>> <mailto:[e-mail]>>  wrote:
>>>
>>> Hi Stephan,
>>>
>>> I understand your concern and I'd be interested if you have
>>> alternative ways of handling the licensing to avoid any issue. It's
>>> not like this is a unique situation. As far as I know, most (all?)
>>> MPEG codecs have similar non-bit exact definitions. I have also
>>> heard that they also require some IPR licensing...
>>>
>>> In general the issue of bit-exactness has been discussed and so far
>>> I don't recall many arguing in favor of a bit-exact definition.
>>> Most of the concerns that have been expressed are solved by
>>> considering that non-bitexact does not mean you cannot be bit-exact
>>> with the reference encoder. It merely means that you don't *have*
>>> to. So regardless of how conformance is defined exactly, one always
>>> has the option of being bit-exact with the reference
>>> implementation, which obviously guarantees compliance.
>>>
>>> As for language mentioning compliance, I believe it belongs more to
>>> the guidelines (it's not a requirement of the codec itself), which
>>> includes the following text:
>>>
>>> 4. To reduce the risk of bias towards certain CPU/DSP architectures,
>>> ideally the decoder specification should not require "bit-exact"
>>> conformance with the reference implementation. The output of a
>>> decoder implementation should only be "close enough" to the
>>> output of the reference decoder. A comparison tool should be
>>> provided along with the codec to verify objectively that the
>>> output of a decoder is likely to be perceptually
>>> indistinguishable from that of the reference decoder. However,
>>> an implementation may still wish to produce an output that is
>>> bit-exact with the reference implementation to simplify the
>>> testing procedure.
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Jean-Marc
>>>
>>>
>>> On 11-01-24 11:02 PM, Stephan Wenger wrote:
>>>
>>> Hi all:
>>>
>>> Let me speak once more against this decision (if such a
>>> decision were
>>> really made; see the p.s.).
>>>
>>> There are currently three IPR disclosures against the codec
>>> draft and/or
>>> its predecessors. The Xiph disclosure is at this point a
>>> placeholder
>>> (Xiph folks: it's time to fix that!). However, the two other
>>> disclosures
>>> on file provide a patent grant only for necessary patent claims
>>> and only
>>> when the standard is practiced in full compliance. These terms (in
>>> various formulations) are quite common.
>>>
>>> In order to ensure one has a license (or can rely on a non-assert
>>> covenant), one has to ensure one meets the conditions set by the
>>> rightholder. On stuff such as reciprocity clauses this is
>>> simple. On
>>> compliance, it's not always easy.
>>>
>>> The traditional compliance test for a media codec is a
>>> stimulus-response
>>> test: you feed test vectors into the codec, and you get
>>> results. If the
>>> results match, you are in compliance, if not, you are not. Simple.
>>>
>>> Without bit exactness, the compliance criteria have to be defined
>>> differently. We can do so, and, indeed, I recall that this has been
>>> mentioned as one plan forward. However, I have seen zero
>>> activity in this
>>> direction, and I have also not seen any language that mentions
>>> this in the
>>> requirements draft. I think that the subject of compliance
>>> tests, at
>>> least in its most basic outline, needs to be documented in the
>>> requirements draft. The details can be taken care of elsewhere
>>> and later,
>>> but not too much later. It should be clear that a codec
>>> candidate (if
>>> there were more than one) needs to have compliance criteria
>>> defined before
>>> that codec candidate can become an RFC. Without that, the key
>>> goal of the
>>> WG, a reasonably freely practicable codec, is just not
>>> achievable in the
>>> current legal environment (which includes, in this case, the IPR
>>> disclosures on file).
>>>
>>> Of course, it would be sooooo much simpler if we would mandate
>>> a bit exact
>>> decoder... Is it really that restricting to require that?
>>>
>>> Stephan
>>>
>>> P.s.: for the IETF procedures newcomers: humms taken at
>>> meetings need to
>>> be confirmed on a mailing list, and consensus needs to be
>>> declared by the
>>> chairs. On this subject, I do recall mailing list discussions after
>>> Maastricht, but I do not recall that consensus was reached, yet
>>> alone
>>> declared. (Unfortunately, I currently don't have the time to go
>>> through
>>> the mailing list archives to verify my recollection; Sorry.)
>>>
>>>
>>>
>>> On 1.24.2011 16:45 , "codec issue tracker"<[e-mail]
>>> <mailto:[e-mail]>>  wrote:
>>>
>>> #12: bit-exact vs. bit-compatible?
>>>
>>> Changes (by gmaxwell@Ã… ):
>>>
>>> * status: new =>  closed
>>> * resolution: =>  worksforme
>>>
>>>
>>> Comment:
>>>
>>> http://www.ietf.org/proceedings/78/minutes/codec.txt
>>>
>>> "On the topic of bit exact. Consensus was bit exactness is
>>> not required."
>>>
>>> I believe this issue is already closed.
>>>
>>> --
>>> ------------------------------------+-------------------------------------
>>> --
>>> Reporter: hoene@Ã… | Owner:
>>> Type: enhancement | Status: closed
>>> Priority: minor | Milestone:
>>> Component: requirements | Version:
>>> Severity: Active WG Document | Resolution: worksforme
>>> Keywords: |
>>> ------------------------------------+-------------------------------------
>>> --
>>>
>>> Ticket
>>> URL:<http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1>
>>> codec<http://tools.ietf.org/codec/>
>>>
>>> _______________________________________________
>>> codec mailing list
>>> [e-mail]<mailto:[e-mail]>
>>> https://www.ietf.org/mailman/listinfo/codec
>>>
>>>
>>>
>>> _______________________________________________
>>> codec mailing list
>>> [e-mail]<mailto:[e-mail]>
>>> https://www.ietf.org/mailman/listinfo/codec
>>>
>>>
>>> _______________________________________________
>>> codec mailing list
>>> [e-mail]<mailto:[e-mail]>
>>> https://www.ietf.org/mailman/listinfo/codec
>>>
>>>
>>>
>>> _______________________________________________
>>> codec mailing list
>>> [e-mail]<mailto:[e-mail]>
>>> https://www.ietf.org/mailman/listinfo/codec
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> codec mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/codec
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec