Re: [Mlcodec] draft-valin-opus-extension-01

[Jean-Marc Valin <[e-mail]>]

Hi Roman,

The interactions between DRED and the rest of Opus aren't that simple. 
DRED actually does quiet a bit of back-and-forth directly within the 
underlying SILK and CELT decoders. While it's probably possible to do 
all that while treating DRED as a separate codec, it would make things 
quite complicated -- and I don't actually know how I would do that.

When looking at Opus+DRED, I don't see any upside (but many downsides) 
to defining DRED as a separate codec. Now as for using DRED with other 
codecs, it may still be feasible to define a way to combine codec X with 
DRED. But even in that case, it's likely you'd need to implement the 
same kind of tight coupling with the decoder.

Regarding your comment about SDP, I'm not sure I understand what would 
be made more complicated with DRED being an Opus extension.

Cheers,

	Jean-Marc

On 2023-08-07 23:45, Roman Shpount wrote:
> Hi Jean-Marc,
> 
> I am sold on DRED (your point #2). It is extremely interesting.
> 
> What I need clarification about is whether DRED should be integrated 
> into Opus or if it should be defined as a standalone CODEC, which can be 
> used with other codecs using RED. I do see that there is some bitrate 
> saving by making it an Opus extension. On the other hand, if DRED is 
> defined as an independent CODEC, it can be used with other codecs, such 
> as AMR or Lyra. Using RED with two different CODECs is not entirely a 
> new idea. Synchronizing state and making smooth transitions is also 
> something that can be addressed. It is not that different from PLC 
> transitions.
> 
> Using RED for packaging also makes a lot of SDP negotiation issues 
> simpler. It also lets you expose parameters for DRED as SDP parameters, 
> which would be complicated with an Opus extension.
> 
> Best Regards,
> _____________
> Roman Shpount
> 
> 
> On Mon, Aug 7, 2023 at 1:47 AM Jean-Marc Valin <[e-mail] 
> <mailto:[e-mail]>> wrote:
> 
>     Hi Roman,
> 
>     Indeed, maybe the DRED draft should more clearly emphasize its
>     benefits.
>     There are several reasons why I think the approach we're proposing
>     would
>     work better than using a separate ML codec through RED. Those can be
>     divided in two categories:
> 
>     1) There are benefits to having the redundancy integrated within Opus.
>     In case of loss, we need to quickly switch from Opus, to the
>     redundancy,
>     and then back to Opus. Since codecs are stateful, you cannot directly
>     switch back and forth without introducing discontinuities. Switching
>     cleanly would either require deep integration of the two codecs (about
>     as close as what we're already proposing) to update each other's
>     states,
>     or else figure out some cross-fading, which would create all kinds of
>     undesirable side effects. I also believe that integration within Opus
>     makes synchronization easier and is likely easier to deploy in general.
> 
>     2) There are also benefits coming from the fact that DRED is not
>     designed to be a general-purpose ML speech codec, but is rather
>     specifically optimized for redundancy. For example, while each DRED
>     packet is independent from the others (no prediction across packet),
>     the
>     many frames within a redundancy packet (up to a second right now) are
>     still coded with prediction to make it more efficient. Our scheme also
>     makes it possible to use a different bitrate as a function of how old
>     each frame is within the redundancy. For example, the audio between
>     t=0ms and t=40ms can be coded at 1000 b/s when part of the redundancy
>     for the t=40ms packet, but it might be coded at just 400 b/s for the
>     redundancy of the t=900ms packet. That can be done without having to
>     re-encode the audio multiple times. You can read more about the DRED
>     design principles in Section 2 of
>     https://arxiv.org/pdf/2212.04453.pdf
>     <https://arxiv.org/pdf/2212.04453.pdf>
>     or in this blog post:
>     https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets <https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets>
> 
>     Cheers,
> 
>              Jean-Marc
> 
> 
>     On 2023-08-06 20:01, Roman Shpount wrote:
>      > Hi All,
>      >
>      > One question that I have about Opus extensions is what are the
>     benefits
>      > of extending Opus vs using RED and combining Opus with other codecs,
>      > such as ML-based codecs, which would provide redundant audio?
>      >
>      > Thank You,
>      > _____________
>      > Roman Shpount
>      >
>      >
>      > On Sun, Aug 6, 2023 at 1:39 PM Stephan Wenger <[e-mail]
>     <mailto:[e-mail]>
>      > <mailto:[e-mail] <mailto:[e-mail]>>> wrote:
>      >
>      >     Hi,____
>      >
>      >     The constituting meeting of an IETF WG is an unusually early
>     point
>      >     for an IETF working group draft adoption.  I’m not objecting
>     at this
>      >     point, but can I ask to make this a two-week call at the
>     minimum?
>      >     It’s summer, and some relevant people I know of are on
>     vacation.____
>      >
>      >     Stephan____
>      >
>      >     __ __
>      >
>      >     __ __
>      >
>      >     *From: *Mlcodec <[e-mail]
>     <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>     <mailto:[e-mail]>>> on behalf of Greg Maxwell
>      >     <[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>>
>      >     *Date: *Saturday, August 5, 2023 at 11:34
>      >     *To: *[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>
>     <[e-mail] <mailto:[e-mail]>
>      >     <mailto:[e-mail] <mailto:[e-mail]>>>
>      >     *Subject: *[Mlcodec] draft-valin-opus-extension-01____
>      >
>      >     At IETF 117 there was consensus in the room to adopt
>      >     draft-valin-opus-extension-01
>      >     (
>     https://datatracker.ietf.org/doc/draft-valin-opus-extension/
>     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/>
>      >     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/
>     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/>> )
>      >     as a working group document and a starting point for further
>      >     development.
>      >
>      >     I'm raising this for the benefit of those who could not make the
>      >     meeting,
>      >     and to request that this consensus be confirmed on the list.
>      >
>      >     So, if you have anything further to say about the adoption,
>     positive or
>      >     negative, please speak up.
>      >
>      >     --
>      >     Mlcodec mailing list
>      > [e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>
>      > https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>
>      >     <https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>>____
>      >
>      >     --
>      >     Mlcodec mailing list
>      > [e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>
>      > https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>
>      >     <https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>>
>      >
>      >
>