Re: [Mlcodec] draft-valin-opus-extension-01

[Jean-Marc Valin <[e-mail]>]

Hi Roman,

Indeed, maybe the DRED draft should more clearly emphasize its benefits. 
There are several reasons why I think the approach we're proposing would 
work better than using a separate ML codec through RED. Those can be 
divided in two categories:

1) There are benefits to having the redundancy integrated within Opus. 
In case of loss, we need to quickly switch from Opus, to the redundancy, 
and then back to Opus. Since codecs are stateful, you cannot directly 
switch back and forth without introducing discontinuities. Switching 
cleanly would either require deep integration of the two codecs (about 
as close as what we're already proposing) to update each other's states, 
or else figure out some cross-fading, which would create all kinds of 
undesirable side effects. I also believe that integration within Opus 
makes synchronization easier and is likely easier to deploy in general.

2) There are also benefits coming from the fact that DRED is not 
designed to be a general-purpose ML speech codec, but is rather 
specifically optimized for redundancy. For example, while each DRED 
packet is independent from the others (no prediction across packet), the 
many frames within a redundancy packet (up to a second right now) are 
still coded with prediction to make it more efficient. Our scheme also 
makes it possible to use a different bitrate as a function of how old 
each frame is within the redundancy. For example, the audio between 
t=0ms and t=40ms can be coded at 1000 b/s when part of the redundancy 
for the t=40ms packet, but it might be coded at just 400 b/s for the 
redundancy of the t=900ms packet. That can be done without having to 
re-encode the audio multiple times. You can read more about the DRED 
design principles in Section 2 of
https://arxiv.org/pdf/2212.04453.pdf
or in this blog post:
https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets

Cheers,

	Jean-Marc


On 2023-08-06 20:01, Roman Shpount wrote:
> Hi All,
> 
> One question that I have about Opus extensions is what are the benefits 
> of extending Opus vs using RED and combining Opus with other codecs, 
> such as ML-based codecs, which would provide redundant audio?
> 
> Thank You,
> _____________
> Roman Shpount
> 
> 
> On Sun, Aug 6, 2023 at 1:39 PM Stephan Wenger <[e-mail] 
> <mailto:[e-mail]>> wrote:
> 
>     Hi,____
> 
>     The constituting meeting of an IETF WG is an unusually early point
>     for an IETF working group draft adoption.  I’m not objecting at this
>     point, but can I ask to make this a two-week call at the minimum? 
>     It’s summer, and some relevant people I know of are on vacation.____
> 
>     Stephan____
> 
>     __ __
> 
>     __ __
> 
>     *From: *Mlcodec <[e-mail]
>     <mailto:[e-mail]>> on behalf of Greg Maxwell
>     <[e-mail] <mailto:[e-mail]>>
>     *Date: *Saturday, August 5, 2023 at 11:34
>     *To: *[e-mail] <mailto:[e-mail]> <[e-mail]
>     <mailto:[e-mail]>>
>     *Subject: *[Mlcodec] draft-valin-opus-extension-01____
> 
>     At IETF 117 there was consensus in the room to adopt
>     draft-valin-opus-extension-01
>     ( https://datatracker.ietf.org/doc/draft-valin-opus-extension/
>     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/> )
>     as a working group document and a starting point for further
>     development.
> 
>     I'm raising this for the benefit of those who could not make the
>     meeting,
>     and to request that this consensus be confirmed on the list.
> 
>     So, if you have anything further to say about the adoption, positive or
>     negative, please speak up.
> 
>     -- 
>     Mlcodec mailing list
>     [e-mail] <mailto:[e-mail]>
>     https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>____
> 
>     -- 
>     Mlcodec mailing list
>     [e-mail] <mailto:[e-mail]>
>     https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>
> 
>