Re: [codec] draft test and processing plan for the IETF Codec

[Stephen Botzko <[e-mail]>]

in-line

On Mon, Apr 18, 2011 at 12:48 PM, Koen Vos <[e-mail]> wrote:

> Stephen Botzko wrote:
> > If you simply want to know which "sounds better" to the user,
>
> That's probably the best you can hope for yes.
>
> > then perhaps bandpass filtering gets in the way.
>
> Correct.
>
>
> > If you want to see if there are there is an underlying difference in
> intelligibility
> > or user tolerance for the coding artifacts,, then the bandpass filtering
> might be
> > useful, since it controls for the known preference that users have for
> wider
> > frequency response.
>
> Sounds like an interesting academic study.  You should also look into any
> long-term health effects (so you can argue for a 5 year test plan!).
>
> One thing we know for sure though: pre-distoring test signals creates a
> bias in the
> results and thus invalidates any conclusion from the test.
>

I don't think this is particularly academic, such filtering seems to show up
in most test plans I've seen. I don't see how it "invalidates the
conclusion", as the input signal is the same for all codecs in any event.

Also, bandpass filtering is not really "pre-distorting".

Best,
Stephen Botzko

>
> best,
> koen.
>
>
> ------------------------------
> *From: *"Stephen Botzko" <[e-mail]>
>
> *To: *"Koen Vos" <[e-mail]>
> *Cc: *"Paul Coverdale" <[e-mail]>, [e-mail]
> *Sent: *Monday, April 18, 2011 4:18:05 AM
>
> *Subject: *Re: [codec] draft test and processing plan for the IETF Codec
>
> in-line
> Stephen Botzko
>
> On Mon, Apr 18, 2011 at 3:27 AM, Koen Vos <[e-mail]> wrote:
>
>> Hi Paul,
>>
>> > I think where this discussion is going is that we need to be more
>> > precise in defining what we mean by "NB", "WB", "SWB" and "FB" if
>> > we want to make meaningful comparisons between codecs.
>>
>> The discussion so far was about whether to pre-distort test signals by
>> bandpass filtering.
>>
>
> I think this might depend on what you want to learn from the test.
>
> If you simply want to know which "sounds better" to the user, then perhaps
> bandpass filtering gets in the way.
>
> If you want to see if there are there is an underlying difference in
> intelligibility or user tolerance for the coding artifacts,, then the
> bandpass filtering might be useful, since it controls for the known
> preference that users have for wider frequency response.
>
>
>
>>
>> I don't see what the name of a codec's mode has to do with meaningful
>> comparisons.  It's the sampling rate that matters: what happens when a
>> VoIP application swaps one codec for another while leaving all else the
>> same.  So where possible you want to compare codecs running at equal
>> sampling rates.  That gives a clear grouping of codecs for 8, 16 and
>> 48 kHz (some call these NB, WB and FB).
>>
>> The open question is what to do in between 16 and 48 kHz.  Opus accepts
>> 24 kHz signals, other codecs use 32 kHz (and they all call it SWB).
>> Here you could either compare directly, which puts the 32 kHz codecs at
>> an advantage.  Or you could run Opus in FB mode by upsampling the 32
>> kHz signal to 48 kHz, as Jean-Marc suggested for 32 and 64 kbps.
>>
>> best,
>> koen.
>>
>>
>> ----- Original Message -----
>> From: "Paul Coverdale" <[e-mail]>
>> To: "Koen Vos" <[e-mail]>
>> Cc: [e-mail], "Anisse Taleb" <[e-mail]>
>> Sent: Sunday, April 17, 2011 5:40:33 PM
>> Subject: RE: [codec] draft test and processing plan for the IETF Codec
>>
>> Hi Koen,
>>
>> There's no doubt that increased audio bandwidth, other things being equal,
>> enhances the perception of quality (well, up to the point where the input
>> signal spectrum itself runs out of steam). I think where this discussion is
>> going is that we need to be more precise in defining what we mean by "NB",
>> "WB", "SWB" and "FB" if we want to make meaningful comparisons between
>> codecs. In fact, the nominal -3 dB passband bandwidth of G.722 is actually a
>> minimum of 50 to 7000 Hz, you can go up to 8000 Hz and still meet the
>> anti-aliassing requirement.
>>
>> Regards,
>>
>> ...Paul
>>
>> >-----Original Message-----
>> >From: Koen Vos [mailto:[e-mail]]
>> >Sent: Sunday, April 17, 2011 1:44 AM
>> >To: Paul Coverdale
>> >Cc: [e-mail]; Anisse Taleb
>> >Subject: Re: [codec] draft test and processing plan for the IETF Codec
>> >
>> >Hi Paul,
>> >
>> >> The filtering described in the test plan [..] is there to establish
>> >> a common bandwidth (and equalization characteristic in some cases)
>> >> for the audio chain (be it NB, WB, SWB) so that subjects can focus
>> >> on comparing the distortion introduced by each of the codecs in the
>> >> test, without confounding it with bandwidth effects.
>> >
>> >I believe it would be a mistake to test with band-limited signals, for
>> >these reasons:
>> >
>> >1. Band-limited test signals are atypical of real-world usage.  People
>> >in this WG have always emphasized that we should test with realistic
>> >scenarios (like network traces for packet loss), and the proposal goes
>> >against that philosophy.
>> >
>> >2. Band limiting the input hurts a codec's performance.  In the Google
>> >test for instance, Opus-WB@20 kbps outperformed the LP7 anchor --
>> >surely that wouldn't happen if Opus ran on an LP7 signal.  That makes
>> >the proposed testing procedure less relevant for deciding whether this
>> >codec will be of value on the Internet.
>> >
>> >3. Audio bandwidth matters to end users.  Real-life experiments show
>> >that codecs with more bandwidth boost user ratings and call durations.
>> >(E.g. see slides 2, 3 of
>> >http://www.ietf.org/proceedings/77/slides/codec-3.pdf)
>> >So if a codec scores higher "just" because it encodes more bandwidth,
>> >that's still a real benefit to users.  And the testing procedure
>> >proposed already reduces the impact of differing bandwidths, by using
>> >MOS scores without pairwise comparisons.
>> >
>> >4. Testing with band-limited signals risks perpetuating crippled codec
>> >design.  In order to do well in the tests, a codec designer would be
>> >"wise" to downsample the input or otherwise optimize towards the
>> >artificial test signals.  This actually lowers the performance for
>> >real-world signals, and usually adds complexity.  And as long as
>> >people design codecs with a band-limited response, they'll argue to
>> >test with one as well.  Let's break this circle.
>> >
>> >I also found it interesting how the chosen bandwidths magically match
>> >those of ITU standards, while potentially hurting Opus.  For instance,
>> >Opus-SWB has only 12 kHz bandwidth, but would still be tested with a
>> >14 kHz signal.
>> >
>> >best,
>> >koen.
>> >
>> >
>> >----- Original Message -----
>> >From: "Paul Coverdale" <[e-mail]>
>> >To: "Koen Vos" <[e-mail]>
>> >Cc: [e-mail], "Anisse Taleb" <[e-mail]>
>> >Sent: Saturday, April 16, 2011 6:25:04 PM
>> >Subject: RE: [codec] draft test and processing plan for the IETF Codec
>> >
>> >Hi Koen and Jean-Marc,
>> >
>> >The filtering described in the test plan is not meant to be for anti-
>> >aliassing, it is there to establish a common bandwidth (and equalization
>> >characteristic in some cases) for the audio chain (be it NB, WB, SWB) so
>> >that subjects can focus on comparing the distortion introduced by each
>> >of the codecs in the test, without confounding it with bandwidth
>> >effects.
>> >
>> >Regards,
>> >
>> >...Paul
>> >
>> >>-----Original Message-----
>> >>From: Koen Vos [mailto:[e-mail]]
>> >>Sent: Saturday, April 16, 2011 4:07 PM
>> >>To: Paul Coverdale
>> >>Cc: [e-mail]; Anisse Taleb
>> >>Subject: Re: [codec] draft test and processing plan for the IETF Codec
>> >>
>> >>Paul Coverdale wrote:
>> >>> You mean that VoIP applications have no filtering at all, not even
>> >>> anti-aliassing?
>> >>
>> >>The bandpass filter in the test plan runs on the downsampled signal,
>> >>so it's not an anti-aliasing filter.
>> >>
>> >>Also, the plan's bandpass for narrowband goes all the way up to Nyquist
>> >>(4000 Hz), whereas for wideband it goes only to 7000 Hz.  So if the
>> >>bandpass filters were to somehow deal with aliasing, they are not being
>> >>used consistently.
>> >>
>> >>I presume the resamplers in the plan use proper anti-aliasing filters
>> >>representative of those in VoIP applications (and described in
>> >>Jean-Marc's post).
>> >>
>> >>best,
>> >>koen.
>> >>
>> >>
>> >>----- Original Message -----
>> >>From: "Paul Coverdale" <[e-mail]>
>> >>To: "Koen Vos" <[e-mail]>, "Anisse Taleb"
>> >><[e-mail]>
>> >>Cc: [e-mail]
>> >>Sent: Saturday, April 16, 2011 4:42:06 AM
>> >>Subject: RE: [codec] draft test and processing plan for the IETF Codec
>> >>
>> >>Hi Koen,
>> >>
>> >>You mean that VoIP applications have no filtering at all, not even
>> >>anti-aliassing?
>> >>
>> >>...Paul
>> >>
>> >>>-----Original Message-----
>> >>>From: [e-mail] [mailto:[e-mail]] On Behalf
>> >>>Of Koen Vos
>> >>>Sent: Saturday, April 16, 2011 1:04 AM
>> >>>To: Anisse Taleb
>> >>>Cc: [e-mail]
>> >>>Subject: Re: [codec] draft test and processing plan for the IETF Codec
>> >>>
>> >>>Hi Anisse,
>> >>>
>> >>>I noticed your plan tests with band-limited signals: Narrowband
>> >signals
>> >>>are
>> >>>filtered from 300-4000 Hz, Wideband from 50-7000 Hz, Superwideband
>> >from
>> >>>50-14000 Hz.
>> >>>
>> >>>However, VoIP applications have no such band-pass filters (which
>> >>degrade
>> >>>quality and add complexity).  So results will be more informative to
>> >>the
>> >>>WG
>> >>>and potential adopters of the codec if the testing avoids band-pass
>> >>>filtering as well.  We want test conditions to mimic the real world as
>> >>>closely as possible.
>> >>>
>> >>>Instead of band-pass filtering, tests on speech could use a simple
>> >>high-
>> >>>pass
>> >>>filter with a cutoff around 50 Hz, as many VoIP applications do indeed
>> >>>have
>> >>>such a filter.
>> >>>
>> >>>best,
>> >>>koen.
>> >>>
>> >>>
>> >>
>> >
>>
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>>
>
>