Re: [codec] draft test and processing plan for the IETF Codec

[Anisse Taleb <[e-mail]>]

Dear Koen,
Thanks for the explanation. When it comes to the second part of my question, i.e. interaction with other effects, such as channel quality, call drops, bandwidth, I was curious about the statistical methodology you used. Any info on that?

Kind regards,
/Anisse 

> -----Original Message-----
> From: Koen Vos [mailto:[e-mail]]
> Sent: Tuesday, April 19, 2011 11:40 AM
> To: Anisse Taleb
> Cc: [e-mail]; Paul Coverdale
> Subject: Re: [codec] draft test and processing plan for the IETF Codec
> 
> Hi Anisse,
> 
> The reason that SILK-SWB has a much smaller confidence interval is simple:
> we had many more data points for that case (about a million, versus just a
> couple thousand for each of the other ones).  This difference in numbers
> was caused by the fact that in the randomized testing we only allowed a
> small fraction of calls to switch to anything else than the default.
> 
> best,
> koen.
> 
> 
> ----- Original Message -----
> From: "Anisse Taleb" <[e-mail]>
> To: "Koen Vos" <[e-mail]>, "Paul Coverdale"
> <[e-mail]>
> Cc: [e-mail]
> Sent: Monday, April 18, 2011 6:03:09 PM
> Subject: RE: [codec] draft test and processing plan for the IETF Codec
> 
> Dear Koen,
> 
> Regarding point 3. This is quite interesting results, just for my
> understanding, I was wondering why the confidence intervals for SILK-SWB
> were small in comparison with the other alternatives? My understanding of
> this experiment is that the audio bandwidth is not the only factor
> affecting quality and call time. How did you isolate the other effect?
> 
> Do you have any more information about the experimental setup and
> statistical analysis conducted to derive these results.
> 
> Kind regards,
> /Anisse
> 
> 
> > -----Original Message-----
> > From: Koen Vos [mailto:[e-mail]]
> > Sent: Sunday, April 17, 2011 7:44 AM
> > To: Paul Coverdale
> > Cc: [e-mail]; Anisse Taleb
> > Subject: Re: [codec] draft test and processing plan for the IETF Codec
> >
> > Hi Paul,
> >
> > > The filtering described in the test plan [..] is there to establish
> > > a common bandwidth (and equalization characteristic in some cases)
> > > for the audio chain (be it NB, WB, SWB) so that subjects can focus
> > > on comparing the distortion introduced by each of the codecs in the
> > > test, without confounding it with bandwidth effects.
> >
> > I believe it would be a mistake to test with band-limited signals, for
> > these reasons:
> >
> > 1. Band-limited test signals are atypical of real-world usage.  People
> > in this WG have always emphasized that we should test with realistic
> > scenarios (like network traces for packet loss), and the proposal goes
> > against that philosophy.
> >
> > 2. Band limiting the input hurts a codec's performance.  In the Google
> > test for instance, Opus-WB@20 kbps outperformed the LP7 anchor --
> > surely that wouldn't happen if Opus ran on an LP7 signal.  That makes
> > the proposed testing procedure less relevant for deciding whether this
> > codec will be of value on the Internet.
> >
> > 3. Audio bandwidth matters to end users.  Real-life experiments show
> > that codecs with more bandwidth boost user ratings and call durations.
> > (E.g. see slides 2, 3 of http://www.ietf.org/proceedings/77/slides/codec-
> > 3.pdf)
> > So if a codec scores higher "just" because it encodes more bandwidth,
> > that's still a real benefit to users.  And the testing procedure
> > proposed already reduces the impact of differing bandwidths, by using
> > MOS scores without pairwise comparisons.
> >
> > 4. Testing with band-limited signals risks perpetuating crippled codec
> > design.  In order to do well in the tests, a codec designer would be
> > "wise" to downsample the input or otherwise optimize towards the
> > artificial test signals.  This actually lowers the performance for
> > real-world signals, and usually adds complexity.  And as long as
> > people design codecs with a band-limited response, they'll argue to
> > test with one as well.  Let's break this circle.
> >
> > I also found it interesting how the chosen bandwidths magically match
> > those of ITU standards, while potentially hurting Opus.  For instance,
> > Opus-SWB has only 12 kHz bandwidth, but would still be tested with a
> > 14 kHz signal.
> >
> > best,
> > koen.
> >
> >
> > ----- Original Message -----
> > From: "Paul Coverdale" <[e-mail]>
> > To: "Koen Vos" <[e-mail]>
> > Cc: [e-mail], "Anisse Taleb" <[e-mail]>
> > Sent: Saturday, April 16, 2011 6:25:04 PM
> > Subject: RE: [codec] draft test and processing plan for the IETF Codec
> >
> > Hi Koen and Jean-Marc,
> >
> > The filtering described in the test plan is not meant to be for anti-
> > aliassing, it is there to establish a common bandwidth (and equalization
> > characteristic in some cases) for the audio chain (be it NB, WB, SWB) so
> > that subjects can focus on comparing the distortion introduced by each of
> > the codecs in the test, without confounding it with bandwidth effects.
> >
> > Regards,
> >
> > ...Paul
> >
> > >-----Original Message-----
> > >From: Koen Vos [mailto:[e-mail]]
> > >Sent: Saturday, April 16, 2011 4:07 PM
> > >To: Paul Coverdale
> > >Cc: [e-mail]; Anisse Taleb
> > >Subject: Re: [codec] draft test and processing plan for the IETF Codec
> > >
> > >Paul Coverdale wrote:
> > >> You mean that VoIP applications have no filtering at all, not even
> > >> anti-aliassing?
> > >
> > >The bandpass filter in the test plan runs on the downsampled signal,
> > >so it's not an anti-aliasing filter.
> > >
> > >Also, the plan's bandpass for narrowband goes all the way up to Nyquist
> > >(4000 Hz), whereas for wideband it goes only to 7000 Hz.  So if the
> > >bandpass filters were to somehow deal with aliasing, they are not being
> > >used consistently.
> > >
> > >I presume the resamplers in the plan use proper anti-aliasing filters
> > >representative of those in VoIP applications (and described in
> > >Jean-Marc's post).
> > >
> > >best,
> > >koen.
> > >
> > >
> > >----- Original Message -----
> > >From: "Paul Coverdale" <[e-mail]>
> > >To: "Koen Vos" <[e-mail]>, "Anisse Taleb"
> > ><[e-mail]>
> > >Cc: [e-mail]
> > >Sent: Saturday, April 16, 2011 4:42:06 AM
> > >Subject: RE: [codec] draft test and processing plan for the IETF Codec
> > >
> > >Hi Koen,
> > >
> > >You mean that VoIP applications have no filtering at all, not even
> > >anti-aliassing?
> > >
> > >...Paul
> > >
> > >>-----Original Message-----
> > >>From: [e-mail] [mailto:[e-mail]] On Behalf
> > >>Of Koen Vos
> > >>Sent: Saturday, April 16, 2011 1:04 AM
> > >>To: Anisse Taleb
> > >>Cc: [e-mail]
> > >>Subject: Re: [codec] draft test and processing plan for the IETF Codec
> > >>
> > >>Hi Anisse,
> > >>
> > >>I noticed your plan tests with band-limited signals: Narrowband signals
> > >>are
> > >>filtered from 300-4000 Hz, Wideband from 50-7000 Hz, Superwideband from
> > >>50-14000 Hz.
> > >>
> > >>However, VoIP applications have no such band-pass filters (which
> > >degrade
> > >>quality and add complexity).  So results will be more informative to
> > >the
> > >>WG
> > >>and potential adopters of the codec if the testing avoids band-pass
> > >>filtering as well.  We want test conditions to mimic the real world as
> > >>closely as possible.
> > >>
> > >>Instead of band-pass filtering, tests on speech could use a simple
> > >high-
> > >>pass
> > >>filter with a cutoff around 50 Hz, as many VoIP applications do indeed
> > >>have
> > >>such a filter.
> > >>
> > >>best,
> > >>koen.
> > >>
> > >>
> > >
> >