Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

[Jean-Marc Valin <[e-mail]>]

As far as I know, MPEG decoders are specified, but not bit exact. With a 
bit-exact specification, there simply could not be any floating-point 
implementation of an MP3 or AAC decoder.

As long as the decoder is correctly specified with a tolerance on the 
deviation compared to the reference decoder, I do not see how an encoder 
could work with one decoder and not with another. As an example of (very 
simple) tolerance, we could say that a decoder implementation has to be 
within 80 dB SNR of the reference implementation on some test vectors. In 
practice we can use slightly more perceptually relevant metrics.

	Jean-Marc

On 11-01-25 09:34 AM, Roman Shpount wrote:
> One more concern that I have related to bit exactness is codec regression
> testing. One can produce a non-bit-exact encoder that produces reasonable
> results for a reference decoder, but if paired with a modified,
> non-bit-exact decoder will produce significant audio artifacts. Since
> neither decoder or encoder are bit exact we will need to have a test
> procedure that will validate that both encoder or decoder will not break
> any standard compliant and non-bit-exact encoders or decoders. Not really
> sure how this can be done.
>
> I might be wrong, but I think MPEG decoders are bit exact and encoders are not.
> _____________
> Roman Shpount
>
>
> On Tue, Jan 25, 2011 at 12:03 AM, Jean-Marc Valin <[e-mail]
> <mailto:[e-mail]>> wrote:
>
>     Hi Stephan,
>
>     I understand your concern and I'd be interested if you have alternative
>     ways of handling the licensing to avoid any issue. It's not like this
>     is a unique situation. As far as I know, most (all?) MPEG codecs have
>     similar non-bit exact definitions. I have also heard that they also
>     require some IPR licensing...
>
>     In general the issue of bit-exactness has been discussed and so far I
>     don't recall many arguing in favor of a bit-exact definition. Most of
>     the concerns that have been expressed are solved by considering that
>     non-bitexact does not mean you cannot be bit-exact with the reference
>     encoder. It merely means that you don't *have* to. So regardless of how
>     conformance is defined exactly, one always has the option of being
>     bit-exact with the reference implementation, which obviously guarantees
>     compliance.
>
>     As for language mentioning compliance, I believe it belongs more to the
>     guidelines (it's not a requirement of the codec itself), which includes
>     the following text:
>
>     4. To reduce the risk of bias towards certain CPU/DSP architectures,
>     ideally the decoder specification should not require "bit-exact"
>     conformance with the reference implementation. The output of a
>     decoder implementation should only be "close enough" to the
>     output of the reference decoder. A comparison tool should be
>     provided along with the codec to verify objectively that the
>     output of a decoder is likely to be perceptually
>     indistinguishable from that of the reference decoder. However,
>     an implementation may still wish to produce an output that is
>     bit-exact with the reference implementation to simplify the
>     testing procedure.
>
>
>
>     Cheers,
>
>     Jean-Marc
>
>
>     On 11-01-24 11:02 PM, Stephan Wenger wrote:
>
>         Hi all:
>
>         Let me speak once more against this decision (if such a decision were
>         really made; see the p.s.).
>
>         There are currently three IPR disclosures against the codec draft
>         and/or
>         its predecessors. The Xiph disclosure is at this point a placeholder
>         (Xiph folks: it's time to fix that!). However, the two other
>         disclosures
>         on file provide a patent grant only for necessary patent claims and
>         only
>         when the standard is practiced in full compliance. These terms (in
>         various formulations) are quite common.
>
>         In order to ensure one has a license (or can rely on a non-assert
>         covenant), one has to ensure one meets the conditions set by the
>         rightholder. On stuff such as reciprocity clauses this is simple. On
>         compliance, it's not always easy.
>
>         The traditional compliance test for a media codec is a
>         stimulus-response
>         test: you feed test vectors into the codec, and you get results. If the
>         results match, you are in compliance, if not, you are not. Simple.
>
>         Without bit exactness, the compliance criteria have to be defined
>         differently. We can do so, and, indeed, I recall that this has been
>         mentioned as one plan forward. However, I have seen zero activity
>         in this
>         direction, and I have also not seen any language that mentions this
>         in the
>         requirements draft. I think that the subject of compliance tests, at
>         least in its most basic outline, needs to be documented in the
>         requirements draft. The details can be taken care of elsewhere and
>         later,
>         but not too much later. It should be clear that a codec candidate (if
>         there were more than one) needs to have compliance criteria defined
>         before
>         that codec candidate can become an RFC. Without that, the key goal
>         of the
>         WG, a reasonably freely practicable codec, is just not achievable
>         in the
>         current legal environment (which includes, in this case, the IPR
>         disclosures on file).
>
>         Of course, it would be sooooo much simpler if we would mandate a
>         bit exact
>         decoder... Is it really that restricting to require that?
>
>         Stephan
>
>         P.s.: for the IETF procedures newcomers: humms taken at meetings
>         need to
>         be confirmed on a mailing list, and consensus needs to be declared
>         by the
>         chairs. On this subject, I do recall mailing list discussions after
>         Maastricht, but I do not recall that consensus was reached, yet alone
>         declared. (Unfortunately, I currently don't have the time to go through
>         the mailing list archives to verify my recollection; Sorry.)
>
>
>
>         On 1.24.2011 16:45 , "codec issue tracker"<[e-mail]
>         <mailto:[e-mail]>> wrote:
>
>             #12: bit-exact vs. bit-compatible?
>
>             Changes (by gmaxwell@Å ):
>
>             * status: new => closed
>             * resolution: => worksforme
>
>
>             Comment:
>
>             http://www.ietf.org/proceedings/78/minutes/codec.txt
>
>             "On the topic of bit exact. Consensus was bit exactness is not
>             required."
>
>             I believe this issue is already closed.
>
>             --
>             ------------------------------------+-------------------------------------
>             --
>             Reporter: hoene@Å | Owner:
>             Type: enhancement | Status: closed
>             Priority: minor | Milestone:
>             Component: requirements | Version:
>             Severity: Active WG Document | Resolution: worksforme
>             Keywords: |
>             ------------------------------------+-------------------------------------
>             --
>
>             Ticket
>             URL:<http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1>
>             codec<http://tools.ietf.org/codec/>
>
>             _______________________________________________
>             codec mailing list
>             [e-mail] <mailto:[e-mail]>
>             https://www.ietf.org/mailman/listinfo/codec
>
>
>
>         _______________________________________________
>         codec mailing list
>         [e-mail] <mailto:[e-mail]>
>         https://www.ietf.org/mailman/listinfo/codec
>
>
>     _______________________________________________
>     codec mailing list
>     [e-mail] <mailto:[e-mail]>
>     https://www.ietf.org/mailman/listinfo/codec
>
>
>
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec