Re: [Mlcodec] draft-valin-opus-extension-01

[Roman Shpount <[e-mail]>]

Hi Jean-Marc,

I did not know how tightly DRED was integrated with Opus. What you are
saying essentially implies that DRED is an integral part of the Opus codec,
which might justify it being an extension.

That said, sending Opus and DRED payload using a RED packet does not mean
they should be independently decoded. All that changes is just payload
packaging. You can define how DRED is decoded in combination with Opus.
Someone else can adapt the same procedures to AMR-WB or EVS. The DRED
encoder, payload definition, and SDP negotiation procedures can be re-used.

Finally, you need to define Opus extension negotiation in SDP. If the DRED
payload is negotiated as a part of RED, all the SDP negotiation procedures
are already defined. All you need to do is to define the payload format and
mime-type parameters.
_____________
Roman Shpount


On Tue, Aug 8, 2023 at 3:42 AM Jean-Marc Valin <[e-mail]> wrote:

> Hi Roman,
>
> The interactions between DRED and the rest of Opus aren't that simple.
> DRED actually does quiet a bit of back-and-forth directly within the
> underlying SILK and CELT decoders. While it's probably possible to do
> all that while treating DRED as a separate codec, it would make things
> quite complicated -- and I don't actually know how I would do that.
>
> When looking at Opus+DRED, I don't see any upside (but many downsides)
> to defining DRED as a separate codec. Now as for using DRED with other
> codecs, it may still be feasible to define a way to combine codec X with
> DRED. But even in that case, it's likely you'd need to implement the
> same kind of tight coupling with the decoder.
>
> Regarding your comment about SDP, I'm not sure I understand what would
> be made more complicated with DRED being an Opus extension.
>
> Cheers,
>
>         Jean-Marc
>
> On 2023-08-07 23:45, Roman Shpount wrote:
> > Hi Jean-Marc,
> >
> > I am sold on DRED (your point #2). It is extremely interesting.
> >
> > What I need clarification about is whether DRED should be integrated
> > into Opus or if it should be defined as a standalone CODEC, which can be
> > used with other codecs using RED. I do see that there is some bitrate
> > saving by making it an Opus extension. On the other hand, if DRED is
> > defined as an independent CODEC, it can be used with other codecs, such
> > as AMR or Lyra. Using RED with two different CODECs is not entirely a
> > new idea. Synchronizing state and making smooth transitions is also
> > something that can be addressed. It is not that different from PLC
> > transitions.
> >
> > Using RED for packaging also makes a lot of SDP negotiation issues
> > simpler. It also lets you expose parameters for DRED as SDP parameters,
> > which would be complicated with an Opus extension.
> >
> > Best Regards,
> > _____________
> > Roman Shpount
> >
> >
> > On Mon, Aug 7, 2023 at 1:47 AM Jean-Marc Valin <[e-mail]
> > <mailto:[e-mail]>> wrote:
> >
> >     Hi Roman,
> >
> >     Indeed, maybe the DRED draft should more clearly emphasize its
> >     benefits.
> >     There are several reasons why I think the approach we're proposing
> >     would
> >     work better than using a separate ML codec through RED. Those can be
> >     divided in two categories:
> >
> >     1) There are benefits to having the redundancy integrated within
> Opus.
> >     In case of loss, we need to quickly switch from Opus, to the
> >     redundancy,
> >     and then back to Opus. Since codecs are stateful, you cannot directly
> >     switch back and forth without introducing discontinuities. Switching
> >     cleanly would either require deep integration of the two codecs
> (about
> >     as close as what we're already proposing) to update each other's
> >     states,
> >     or else figure out some cross-fading, which would create all kinds of
> >     undesirable side effects. I also believe that integration within Opus
> >     makes synchronization easier and is likely easier to deploy in
> general.
> >
> >     2) There are also benefits coming from the fact that DRED is not
> >     designed to be a general-purpose ML speech codec, but is rather
> >     specifically optimized for redundancy. For example, while each DRED
> >     packet is independent from the others (no prediction across packet),
> >     the
> >     many frames within a redundancy packet (up to a second right now) are
> >     still coded with prediction to make it more efficient. Our scheme
> also
> >     makes it possible to use a different bitrate as a function of how old
> >     each frame is within the redundancy. For example, the audio between
> >     t=0ms and t=40ms can be coded at 1000 b/s when part of the redundancy
> >     for the t=40ms packet, but it might be coded at just 400 b/s for the
> >     redundancy of the t=900ms packet. That can be done without having to
> >     re-encode the audio multiple times. You can read more about the DRED
> >     design principles in Section 2 of
> >     https://arxiv.org/pdf/2212.04453.pdf
> >     <https://arxiv.org/pdf/2212.04453.pdf>
> >     or in this blog post:
> >
> https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets
> <
> https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets
> >
> >
> >     Cheers,
> >
> >              Jean-Marc
> >
> >
> >     On 2023-08-06 20:01, Roman Shpount wrote:
> >      > Hi All,
> >      >
> >      > One question that I have about Opus extensions is what are the
> >     benefits
> >      > of extending Opus vs using RED and combining Opus with other
> codecs,
> >      > such as ML-based codecs, which would provide redundant audio?
> >      >
> >      > Thank You,
> >      > _____________
> >      > Roman Shpount
> >      >
> >      >
> >      > On Sun, Aug 6, 2023 at 1:39 PM Stephan Wenger <[e-mail]
> >     <mailto:[e-mail]>
> >      > <mailto:[e-mail] <mailto:[e-mail]>>> wrote:
> >      >
> >      >     Hi,____
> >      >
> >      >     The constituting meeting of an IETF WG is an unusually early
> >     point
> >      >     for an IETF working group draft adoption.  I’m not objecting
> >     at this
> >      >     point, but can I ask to make this a two-week call at the
> >     minimum?
> >      >     It’s summer, and some relevant people I know of are on
> >     vacation.____
> >      >
> >      >     Stephan____
> >      >
> >      >     __ __
> >      >
> >      >     __ __
> >      >
> >      >     *From: *Mlcodec <[e-mail]
> >     <mailto:[e-mail]>
> >      >     <mailto:[e-mail]
> >     <mailto:[e-mail]>>> on behalf of Greg Maxwell
> >      >     <[e-mail] <mailto:[e-mail]>
> >     <mailto:[e-mail] <mailto:[e-mail]>>>
> >      >     *Date: *Saturday, August 5, 2023 at 11:34
> >      >     *To: *[e-mail] <mailto:[e-mail]>
> >     <mailto:[e-mail] <mailto:[e-mail]>>
> >     <[e-mail] <mailto:[e-mail]>
> >      >     <mailto:[e-mail] <mailto:[e-mail]>>>
> >      >     *Subject: *[Mlcodec] draft-valin-opus-extension-01____
> >      >
> >      >     At IETF 117 there was consensus in the room to adopt
> >      >     draft-valin-opus-extension-01
> >      >     (
> >     https://datatracker.ietf.org/doc/draft-valin-opus-extension/
> >     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/>
> >      >     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/
> >     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/>> )
> >      >     as a working group document and a starting point for further
> >      >     development.
> >      >
> >      >     I'm raising this for the benefit of those who could not make
> the
> >      >     meeting,
> >      >     and to request that this consensus be confirmed on the list.
> >      >
> >      >     So, if you have anything further to say about the adoption,
> >     positive or
> >      >     negative, please speak up.
> >      >
> >      >     --
> >      >     Mlcodec mailing list
> >      > [e-mail] <mailto:[e-mail]>
> >     <mailto:[e-mail] <mailto:[e-mail]>>
> >      > https://www.ietf.org/mailman/listinfo/mlcodec
> >     <https://www.ietf.org/mailman/listinfo/mlcodec>
> >      >     <https://www.ietf.org/mailman/listinfo/mlcodec
> >     <https://www.ietf.org/mailman/listinfo/mlcodec>>____
> >      >
> >      >     --
> >      >     Mlcodec mailing list
> >      > [e-mail] <mailto:[e-mail]>
> >     <mailto:[e-mail] <mailto:[e-mail]>>
> >      > https://www.ietf.org/mailman/listinfo/mlcodec
> >     <https://www.ietf.org/mailman/listinfo/mlcodec>
> >      >     <https://www.ietf.org/mailman/listinfo/mlcodec
> >     <https://www.ietf.org/mailman/listinfo/mlcodec>>
> >      >
> >      >
> >
>