Re: [Mlcodec] draft-valin-opus-extension-01

[Jean-Marc Valin <[e-mail]>]

Hi Roman,

In terms of SDP for DRED, right now the idea is just for the receiver to 
tell the sender "I'm able to use up to X seconds of DRED". From there, 
the sender can figure out how to distribute its total bitrate budget 
between the "base" Opus and DRED, depending on its estimate of the 
network conditions. If there's no loss, the encoder could even stop 
sending DRED completely. Or if most of the previous frames are silence, 
the encoder will encode less redundancy and spend more on the base Opus. 
That's another reason why I don't think we'd want DRED to be 
controlled/signaled independently from the rest of Opus.

	Jean-Marc


On 2023-08-08 23:16, Roman Shpount wrote:
> Hi Jean-Marc,
> 
> I did not know how tightly DRED was integrated with Opus. What you are 
> saying essentially implies that DRED is an integral part of the Opus 
> codec, which might justify it being an extension.
> 
> That said, sending Opus and DRED payload using a RED packet does not 
> mean they should be independently decoded. All that changes is just 
> payload packaging. You can define how DRED is decoded in combination 
> with Opus. Someone else can adapt the same procedures to AMR-WB or EVS. 
> The DRED encoder, payload definition, and SDP negotiation procedures can 
> be re-used.
> 
> Finally, you need to define Opus extension negotiation in SDP. If the 
> DRED payload is negotiated as a part of RED, all the SDP negotiation 
> procedures are already defined. All you need to do is to define the 
> payload format and mime-type parameters.
> _____________
> Roman Shpount
> 
> 
> On Tue, Aug 8, 2023 at 3:42 AM Jean-Marc Valin <[e-mail] 
> <mailto:[e-mail]>> wrote:
> 
>     Hi Roman,
> 
>     The interactions between DRED and the rest of Opus aren't that simple.
>     DRED actually does quiet a bit of back-and-forth directly within the
>     underlying SILK and CELT decoders. While it's probably possible to do
>     all that while treating DRED as a separate codec, it would make things
>     quite complicated -- and I don't actually know how I would do that.
> 
>     When looking at Opus+DRED, I don't see any upside (but many downsides)
>     to defining DRED as a separate codec. Now as for using DRED with other
>     codecs, it may still be feasible to define a way to combine codec X
>     with
>     DRED. But even in that case, it's likely you'd need to implement the
>     same kind of tight coupling with the decoder.
> 
>     Regarding your comment about SDP, I'm not sure I understand what would
>     be made more complicated with DRED being an Opus extension.
> 
>     Cheers,
> 
>              Jean-Marc
> 
>     On 2023-08-07 23:45, Roman Shpount wrote:
>      > Hi Jean-Marc,
>      >
>      > I am sold on DRED (your point #2). It is extremely interesting.
>      >
>      > What I need clarification about is whether DRED should be integrated
>      > into Opus or if it should be defined as a standalone CODEC, which
>     can be
>      > used with other codecs using RED. I do see that there is some
>     bitrate
>      > saving by making it an Opus extension. On the other hand, if DRED is
>      > defined as an independent CODEC, it can be used with other
>     codecs, such
>      > as AMR or Lyra. Using RED with two different CODECs is not
>     entirely a
>      > new idea. Synchronizing state and making smooth transitions is also
>      > something that can be addressed. It is not that different from PLC
>      > transitions.
>      >
>      > Using RED for packaging also makes a lot of SDP negotiation issues
>      > simpler. It also lets you expose parameters for DRED as SDP
>     parameters,
>      > which would be complicated with an Opus extension.
>      >
>      > Best Regards,
>      > _____________
>      > Roman Shpount
>      >
>      >
>      > On Mon, Aug 7, 2023 at 1:47 AM Jean-Marc Valin
>     <[e-mail] <mailto:[e-mail]>
>      > <mailto:[e-mail] <mailto:[e-mail]>>> wrote:
>      >
>      >     Hi Roman,
>      >
>      >     Indeed, maybe the DRED draft should more clearly emphasize its
>      >     benefits.
>      >     There are several reasons why I think the approach we're
>     proposing
>      >     would
>      >     work better than using a separate ML codec through RED. Those
>     can be
>      >     divided in two categories:
>      >
>      >     1) There are benefits to having the redundancy integrated
>     within Opus.
>      >     In case of loss, we need to quickly switch from Opus, to the
>      >     redundancy,
>      >     and then back to Opus. Since codecs are stateful, you cannot
>     directly
>      >     switch back and forth without introducing discontinuities.
>     Switching
>      >     cleanly would either require deep integration of the two
>     codecs (about
>      >     as close as what we're already proposing) to update each other's
>      >     states,
>      >     or else figure out some cross-fading, which would create all
>     kinds of
>      >     undesirable side effects. I also believe that integration
>     within Opus
>      >     makes synchronization easier and is likely easier to deploy
>     in general.
>      >
>      >     2) There are also benefits coming from the fact that DRED is not
>      >     designed to be a general-purpose ML speech codec, but is rather
>      >     specifically optimized for redundancy. For example, while
>     each DRED
>      >     packet is independent from the others (no prediction across
>     packet),
>      >     the
>      >     many frames within a redundancy packet (up to a second right
>     now) are
>      >     still coded with prediction to make it more efficient. Our
>     scheme also
>      >     makes it possible to use a different bitrate as a function of
>     how old
>      >     each frame is within the redundancy. For example, the audio
>     between
>      >     t=0ms and t=40ms can be coded at 1000 b/s when part of the
>     redundancy
>      >     for the t=40ms packet, but it might be coded at just 400 b/s
>     for the
>      >     redundancy of the t=900ms packet. That can be done without
>     having to
>      >     re-encode the audio multiple times. You can read more about
>     the DRED
>      >     design principles in Section 2 of
>      > https://arxiv.org/pdf/2212.04453.pdf
>     <https://arxiv.org/pdf/2212.04453.pdf>
>      >     <https://arxiv.org/pdf/2212.04453.pdf
>     <https://arxiv.org/pdf/2212.04453.pdf>>
>      >     or in this blog post:
>      >
>     https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets <https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets> <https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets <https://www.amazon.science/blog/neural-encoding-enables-more-efficient-recovery-of-lost-audio-packets>>
>      >
>      >     Cheers,
>      >
>      >              Jean-Marc
>      >
>      >
>      >     On 2023-08-06 20:01, Roman Shpount wrote:
>      >      > Hi All,
>      >      >
>      >      > One question that I have about Opus extensions is what are the
>      >     benefits
>      >      > of extending Opus vs using RED and combining Opus with
>     other codecs,
>      >      > such as ML-based codecs, which would provide redundant audio?
>      >      >
>      >      > Thank You,
>      >      > _____________
>      >      > Roman Shpount
>      >      >
>      >      >
>      >      > On Sun, Aug 6, 2023 at 1:39 PM Stephan Wenger
>     <[e-mail] <mailto:[e-mail]>
>      >     <mailto:[e-mail] <mailto:[e-mail]>>
>      >      > <mailto:[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>>> wrote:
>      >      >
>      >      >     Hi,____
>      >      >
>      >      >     The constituting meeting of an IETF WG is an unusually
>     early
>      >     point
>      >      >     for an IETF working group draft adoption.  I’m not
>     objecting
>      >     at this
>      >      >     point, but can I ask to make this a two-week call at the
>      >     minimum?
>      >      >     It’s summer, and some relevant people I know of are on
>      >     vacation.____
>      >      >
>      >      >     Stephan____
>      >      >
>      >      >     __ __
>      >      >
>      >      >     __ __
>      >      >
>      >      >     *From: *Mlcodec <[e-mail]
>     <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>     <mailto:[e-mail]>>
>      >      >     <mailto:[e-mail]
>     <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>     <mailto:[e-mail]>>>> on behalf of Greg Maxwell
>      >      >     <[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>
>      >     <mailto:[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>>>
>      >      >     *Date: *Saturday, August 5, 2023 at 11:34
>      >      >     *To: *[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>
>      >     <mailto:[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>>
>      >     <[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>
>      >      >     <mailto:[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>>>
>      >      >     *Subject: *[Mlcodec] draft-valin-opus-extension-01____
>      >      >
>      >      >     At IETF 117 there was consensus in the room to adopt
>      >      >     draft-valin-opus-extension-01
>      >      >     (
>      > https://datatracker.ietf.org/doc/draft-valin-opus-extension/
>     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/>
>      >     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/
>     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/>>
>      >      >   
>       <https://datatracker.ietf.org/doc/draft-valin-opus-extension/
>     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/>
>      >     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/
>     <https://datatracker.ietf.org/doc/draft-valin-opus-extension/>>> )
>      >      >     as a working group document and a starting point for
>     further
>      >      >     development.
>      >      >
>      >      >     I'm raising this for the benefit of those who could
>     not make the
>      >      >     meeting,
>      >      >     and to request that this consensus be confirmed on the
>     list.
>      >      >
>      >      >     So, if you have anything further to say about the
>     adoption,
>      >     positive or
>      >      >     negative, please speak up.
>      >      >
>      >      >     --
>      >      >     Mlcodec mailing list
>      >      > [e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>
>      >     <mailto:[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>>
>      >      > https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>
>      >     <https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>>
>      >      >     <https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>
>      >     <https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>>>____
>      >      >
>      >      >     --
>      >      >     Mlcodec mailing list
>      >      > [e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>
>      >     <mailto:[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>>
>      >      > https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>
>      >     <https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>>
>      >      >     <https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>
>      >     <https://www.ietf.org/mailman/listinfo/mlcodec
>     <https://www.ietf.org/mailman/listinfo/mlcodec>>>
>      >      >
>      >      >
>      >
>