Re: [Mlcodec] draft-valin-opus-extension-01

[Roman Shpount <[e-mail]>]

Hi Jean-Marc,

I am sold on DRED (your point #2). It is extremely interesting.

What I need clarification about is whether DRED should be integrated into
Opus or if it should be defined as a standalone CODEC, which can be used
with other codecs using RED. I do see that there is some bitrate saving by
making it an Opus extension. On the other hand, if DRED is defined as an
independent CODEC, it can be used with other codecs, such as AMR or Lyra.
Using RED with two different CODECs is not entirely a new idea.
Synchronizing state and making smooth transitions is also something that
can be addressed. It is not that different from PLC transitions.

Using RED for packaging also makes a lot of SDP negotiation issues simpler.
It also lets you expose parameters for DRED as SDP parameters, which would
be complicated with an Opus extension.

Best Regards,
_____________
Roman Shpount


On Mon, Aug 7, 2023 at 1:47 AM Jean-Marc Valin <[e-mail]> wrote:

> Hi Roman,
>
> Indeed, maybe the DRED draft should more clearly emphasize its benefits.
> There are several reasons why I think the approach we're proposing would
> work better than using a separate ML codec through RED. Those can be
> divided in two categories:
>
> 1) There are benefits to having the redundancy integrated within Opus.
> In case of loss, we need to quickly switch from Opus, to the redundancy,
> and then back to Opus. Since codecs are stateful, you cannot directly
> switch back and forth without introducing discontinuities. Switching
> cleanly would either require deep integration of the two codecs (about
> as close as what we're already proposing) to update each other's states,
> or else figure out some cross-fading, which would create all kinds of
> undesirable side effects. I also believe that integration within Opus
> makes synchronization easier and is likely easier to deploy in general.
>
> 2) There are also benefits coming from the fact that DRED is not
> designed to be a general-purpose ML speech codec, but is rather
> specifically optimized for redundancy. For example, while each DRED
> packet is independent from the others (no prediction across packet), the
> many frames within a redundancy packet (up to a second right now) are
> still coded with prediction to make it more efficient. Our scheme also
> makes it possible to use a different bitrate as a function of how old
> each frame is within the redundancy. For example, the audio between
> t=0ms and t=40ms can be coded at 1000 b/s when part of the redundancy
> for the t=40ms packet, but it might be coded at just 400 b/s for the
> redundancy of the t=900ms packet. That can be done without having to
> re-encode the audio multiple times. You can read more about the DRED
> design principles in Section 2 of
> https://arxiv.org/pdf/2212.04453.pdf
> or in this blog post:
>
> https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets
>
> Cheers,
>
>         Jean-Marc
>
>
> On 2023-08-06 20:01, Roman Shpount wrote:
> > Hi All,
> >
> > One question that I have about Opus extensions is what are the benefits
> > of extending Opus vs using RED and combining Opus with other codecs,
> > such as ML-based codecs, which would provide redundant audio?
> >
> > Thank You,
> > _____________
> > Roman Shpount
> >
> >
> > On Sun, Aug 6, 2023 at 1:39 PM Stephan Wenger <[e-mail]
> > <mailto:[e-mail]>> wrote:
> >
> >     Hi,____
> >
> >     The constituting meeting of an IETF WG is an unusually early point
> >     for an IETF working group draft adoption.  I’m not objecting at this
> >     point, but can I ask to make this a two-week call at the minimum?
> >     It’s summer, and some relevant people I know of are on vacation.____
> >
> >     Stephan____
> >
> >     __ __
> >
> >     __ __
> >
> >     *From: *Mlcodec <[e-mail]
> >     <mailto:[e-mail]>> on behalf of Greg Maxwell
> >     <[e-mail] <mailto:[e-mail]>>
> >     *Date: *Saturday, August 5, 2023 at 11:34
> >     *To: *[e-mail] <mailto:[e-mail]> <[e-mail]
> >     <mailto:[e-mail]>>
> >     *Subject: *[Mlcodec] draft-valin-opus-extension-01____
> >
> >     At IETF 117 there was consensus in the room to adopt
> >     draft-valin-opus-extension-01
> >     ( https://datatracker.ietf.org/doc/draft-valin-opus-extension/
> >     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/> )
> >     as a working group document and a starting point for further
> >     development.
> >
> >     I'm raising this for the benefit of those who could not make the
> >     meeting,
> >     and to request that this consensus be confirmed on the list.
> >
> >     So, if you have anything further to say about the adoption, positive
> or
> >     negative, please speak up.
> >
> >     --
> >     Mlcodec mailing list
> >     [e-mail] <mailto:[e-mail]>
> >     https://www.ietf.org/mailman/listinfo/mlcodec
> >     <https://www.ietf.org/mailman/listinfo/mlcodec>____
> >
> >     --
> >     Mlcodec mailing list
> >     [e-mail] <mailto:[e-mail]>
> >     https://www.ietf.org/mailman/listinfo/mlcodec
> >     <https://www.ietf.org/mailman/listinfo/mlcodec>
> >
> >
>