Re: [codec] draft test and processing plan for the IETF Codec

[Anisse Taleb <[e-mail]>]

Hi Koen,

The reason for me asking is that the ACD numbers (regardless of which codec is used ) you report seem to be quite high compared to what is reported in (*). It could be, as you mention, that by filtering the bad channel conditions the overall average goes up. Are your results published somewhere else? With the detailed experimental setup and filtering procedure? 

(*) “An Experimental Study of the Skype Peer-to-Peer VoIP System,” by S. Guha, N. Daswani and R. Jain. 

Kind regards,
/Anisse

> -----Original Message-----
> From: Koen Vos [mailto:[e-mail]]
> Sent: Tuesday, April 19, 2011 11:19 PM
> To: Anisse Taleb
> Cc: [e-mail]; Paul Coverdale
> Subject: Re: [codec] draft test and processing plan for the IETF Codec
> 
> Hi Anisse,
> 
> Yes for sure, we carefully filtered out anything that could interfere with
> the test.  So we kept only good network channels and so on.  You know,
> regular rigorous testing, but without the intentional biases :P.
> 
> The most amazing thing was: coinciding with the randomized testing we saw
> an increase in bug reports about audio quality..  People complaining about
> muffled audio etc.
> 
> best,
> koen.
> 
> 
> ----- Original Message -----
> From: "Anisse Taleb" <[e-mail]>
> To: "Koen Vos" <[e-mail]>
> Cc: [e-mail], "Paul Coverdale" <[e-mail]>
> Sent: Tuesday, April 19, 2011 11:58:31 AM
> Subject: RE: [codec] draft test and processing plan for the IETF Codec
> 
> Dear Koen,
> Thanks for the explanation. When it comes to the second part of my
> question, i.e. interaction with other effects, such as channel quality,
> call drops, bandwidth, I was curious about the statistical methodology you
> used. Any info on that?
> 
> Kind regards,
> /Anisse
> 
> > -----Original Message-----
> > From: Koen Vos [mailto:[e-mail]]
> > Sent: Tuesday, April 19, 2011 11:40 AM
> > To: Anisse Taleb
> > Cc: [e-mail]; Paul Coverdale
> > Subject: Re: [codec] draft test and processing plan for the IETF Codec
> >
> > Hi Anisse,
> >
> > The reason that SILK-SWB has a much smaller confidence interval is
> simple:
> > we had many more data points for that case (about a million, versus just
> a
> > couple thousand for each of the other ones).  This difference in numbers
> > was caused by the fact that in the randomized testing we only allowed a
> > small fraction of calls to switch to anything else than the default.
> >
> > best,
> > koen.
> >
> >
> > ----- Original Message -----
> > From: "Anisse Taleb" <[e-mail]>
> > To: "Koen Vos" <[e-mail]>, "Paul Coverdale"
> > <[e-mail]>
> > Cc: [e-mail]
> > Sent: Monday, April 18, 2011 6:03:09 PM
> > Subject: RE: [codec] draft test and processing plan for the IETF Codec
> >
> > Dear Koen,
> >
> > Regarding point 3. This is quite interesting results, just for my
> > understanding, I was wondering why the confidence intervals for SILK-SWB
> > were small in comparison with the other alternatives? My understanding of
> > this experiment is that the audio bandwidth is not the only factor
> > affecting quality and call time. How did you isolate the other effect?
> >
> > Do you have any more information about the experimental setup and
> > statistical analysis conducted to derive these results.
> >
> > Kind regards,
> > /Anisse
> >
> >
> > > -----Original Message-----
> > > From: Koen Vos [mailto:[e-mail]]
> > > Sent: Sunday, April 17, 2011 7:44 AM
> > > To: Paul Coverdale
> > > Cc: [e-mail]; Anisse Taleb
> > > Subject: Re: [codec] draft test and processing plan for the IETF Codec
> > >
> > > Hi Paul,
> > >
> > > > The filtering described in the test plan [..] is there to establish
> > > > a common bandwidth (and equalization characteristic in some cases)
> > > > for the audio chain (be it NB, WB, SWB) so that subjects can focus
> > > > on comparing the distortion introduced by each of the codecs in the
> > > > test, without confounding it with bandwidth effects.
> > >
> > > I believe it would be a mistake to test with band-limited signals, for
> > > these reasons:
> > >
> > > 1. Band-limited test signals are atypical of real-world usage.  People
> > > in this WG have always emphasized that we should test with realistic
> > > scenarios (like network traces for packet loss), and the proposal goes
> > > against that philosophy.
> > >
> > > 2. Band limiting the input hurts a codec's performance.  In the Google
> > > test for instance, Opus-WB@20 kbps outperformed the LP7 anchor --
> > > surely that wouldn't happen if Opus ran on an LP7 signal.  That makes
> > > the proposed testing procedure less relevant for deciding whether this
> > > codec will be of value on the Internet.
> > >
> > > 3. Audio bandwidth matters to end users.  Real-life experiments show
> > > that codecs with more bandwidth boost user ratings and call durations.
> > > (E.g. see slides 2, 3 of
> http://www.ietf.org/proceedings/77/slides/codec-
> > > 3.pdf)
> > > So if a codec scores higher "just" because it encodes more bandwidth,
> > > that's still a real benefit to users.  And the testing procedure
> > > proposed already reduces the impact of differing bandwidths, by using
> > > MOS scores without pairwise comparisons.
> > >
> > > 4. Testing with band-limited signals risks perpetuating crippled codec
> > > design.  In order to do well in the tests, a codec designer would be
> > > "wise" to downsample the input or otherwise optimize towards the
> > > artificial test signals.  This actually lowers the performance for
> > > real-world signals, and usually adds complexity.  And as long as
> > > people design codecs with a band-limited response, they'll argue to
> > > test with one as well.  Let's break this circle.
> > >
> > > I also found it interesting how the chosen bandwidths magically match
> > > those of ITU standards, while potentially hurting Opus.  For instance,
> > > Opus-SWB has only 12 kHz bandwidth, but would still be tested with a
> > > 14 kHz signal.
> > >
> > > best,
> > > koen.
> > >
> > >
> > > ----- Original Message -----
> > > From: "Paul Coverdale" <[e-mail]>
> > > To: "Koen Vos" <[e-mail]>
> > > Cc: [e-mail], "Anisse Taleb" <[e-mail]>
> > > Sent: Saturday, April 16, 2011 6:25:04 PM
> > > Subject: RE: [codec] draft test and processing plan for the IETF Codec
> > >
> > > Hi Koen and Jean-Marc,
> > >
> > > The filtering described in the test plan is not meant to be for anti-
> > > aliassing, it is there to establish a common bandwidth (and
> equalization
> > > characteristic in some cases) for the audio chain (be it NB, WB, SWB)
> so
> > > that subjects can focus on comparing the distortion introduced by each
> of
> > > the codecs in the test, without confounding it with bandwidth effects.
> > >
> > > Regards,
> > >
> > > ...Paul
> > >
> > > >-----Original Message-----
> > > >From: Koen Vos [mailto:[e-mail]]
> > > >Sent: Saturday, April 16, 2011 4:07 PM
> > > >To: Paul Coverdale
> > > >Cc: [e-mail]; Anisse Taleb
> > > >Subject: Re: [codec] draft test and processing plan for the IETF Codec
> > > >
> > > >Paul Coverdale wrote:
> > > >> You mean that VoIP applications have no filtering at all, not even
> > > >> anti-aliassing?
> > > >
> > > >The bandpass filter in the test plan runs on the downsampled signal,
> > > >so it's not an anti-aliasing filter.
> > > >
> > > >Also, the plan's bandpass for narrowband goes all the way up to
> Nyquist
> > > >(4000 Hz), whereas for wideband it goes only to 7000 Hz.  So if the
> > > >bandpass filters were to somehow deal with aliasing, they are not
> being
> > > >used consistently.
> > > >
> > > >I presume the resamplers in the plan use proper anti-aliasing filters
> > > >representative of those in VoIP applications (and described in
> > > >Jean-Marc's post).
> > > >
> > > >best,
> > > >koen.
> > > >
> > > >
> > > >----- Original Message -----
> > > >From: "Paul Coverdale" <[e-mail]>
> > > >To: "Koen Vos" <[e-mail]>, "Anisse Taleb"
> > > ><[e-mail]>
> > > >Cc: [e-mail]
> > > >Sent: Saturday, April 16, 2011 4:42:06 AM
> > > >Subject: RE: [codec] draft test and processing plan for the IETF Codec
> > > >
> > > >Hi Koen,
> > > >
> > > >You mean that VoIP applications have no filtering at all, not even
> > > >anti-aliassing?
> > > >
> > > >...Paul
> > > >
> > > >>-----Original Message-----
> > > >>From: [e-mail] [mailto:[e-mail]] On
> Behalf
> > > >>Of Koen Vos
> > > >>Sent: Saturday, April 16, 2011 1:04 AM
> > > >>To: Anisse Taleb
> > > >>Cc: [e-mail]
> > > >>Subject: Re: [codec] draft test and processing plan for the IETF
> Codec
> > > >>
> > > >>Hi Anisse,
> > > >>
> > > >>I noticed your plan tests with band-limited signals: Narrowband
> signals
> > > >>are
> > > >>filtered from 300-4000 Hz, Wideband from 50-7000 Hz, Superwideband
> from
> > > >>50-14000 Hz.
> > > >>
> > > >>However, VoIP applications have no such band-pass filters (which
> > > >degrade
> > > >>quality and add complexity).  So results will be more informative to
> > > >the
> > > >>WG
> > > >>and potential adopters of the codec if the testing avoids band-pass
> > > >>filtering as well.  We want test conditions to mimic the real world
> as
> > > >>closely as possible.
> > > >>
> > > >>Instead of band-pass filtering, tests on speech could use a simple
> > > >high-
> > > >>pass
> > > >>filter with a cutoff around 50 Hz, as many VoIP applications do
> indeed
> > > >>have
> > > >>such a filter.
> > > >>
> > > >>best,
> > > >>koen.
> > > >>
> > > >>
> > > >
> > >