Re: [Moq] Maybe a good building block for Media over QUIC -- Deadline-aware Transport Protocol

[Chuan Ma <[e-mail]>]

Dear Sharabayko:

Thank you for your response! Sorry for the late reply. We were updating our
draft of DTP, and your questions helped a lot.

It is clear how the sender would drop the block. I wonder how would the
> receiver know the origin time of the block to know if it is already too
> late or not?


This is a great question. Initially, the receiver side of DTP is not
designed to know the beginning time of the block and make any decision. But
in the latest version of our draft, we defined an optional parameter 'start
time stamp' in the BLOCK_INFO frame. DTP will add a starting time stamp in
the frame which can carry the information of the creation time of the
block, and then the receiver can get the origin time of the block.

The receiving application does not get a partially received but canceled
> block, right? If so, this might be a limitation, ...


DTP may indeed drop a block entirely without sending any data in the block,
and we do encounter some demands for sending the data in the block as much
as possible before the deadline. We are considering making it a new feature
of DTP.

So that DTP looks more like a protocol on top of QUIC, and simplifying
> implementation. Although just a though, not a call to action. :)


This is a good idea. If possible, we would like to design a thin-layer
version of DTP on top of QUIC. It would be great to add new features to our
layer without worrying about the update of QUIC.

Thanks again for your excellent question. We are looking forward to hearing
from you again!

Best Regards,

Chuan Ma

Maxim Sharabayko <[e-mail]> 于2022年7月21日周四 18:33写道：

> Hi Chuan Ma,
>
>
>
> Thank you for your detailed explanations!
>
>
>
> *> **  In 'Block Deadline', the 'deadline' is a time duration in
> milliseconds, representing the requirement of the time limit from the
> application (…)*
>
>
>
> It is clear how the sender would drop the block. I wonder how would the
> receiver know the origin time of the block to know if it is already too
> late or not?
>
>
>
> *> **We implement the action 'drop block' by canceling the corresponding
> stream in QUIC. So both sender and receiver can drop the block due to the
> application's command.(…)*
>
>
>
> Looks good, thank you for clarifying!
>
> The receiving application does not get a partially received but canceled
> block, right? If so, this might be a limitation, as sometimes having even a
> part of a frame you would show still provides a better quality of
> experience compared to just a frozen picture.
>
>
>
> *> **The scheduler of DTP is directly implemented inside QUIC because it
> can fully use the internal mechanism of QUIC, obtain the detected network
> conditions and handle the scheduling and discarding of data blocks more
> flexibly. (…)*
>
>
>
> It might make sense to specify some additional API requirement for a QUIC
> library to provide the most of the data needed for DTP to work best.
>
> So that DTP looks more like a protocol on top of QUIC, and simplifying
> implementation.
>
> Although just a though, not a call to action. :)
>
>
>
> Regards,
>
> Maxim
>
>
>
>
>
>
>
>
>
> Maxim Sharabayko
>
> Principal Research Engineer
>
> [image: Logo] <https://www.haivision.com/>
>
>
>
> *From: *Chuan Ma <[e-mail]>
> *Sent: *Monday, July 18, 2022 3:15 PM
> *To: *Maxim Sharabayko <[e-mail]>; [e-mail]
> *Subject: *Re: [Moq] Maybe a good building block for Media over QUIC --
> Deadline-aware Transport Protocol
>
>
>
>  Dear Sharabayko:
>
>
>
>  Thank you for your questions! We are glad to know that you are interested
> in our work. Our response is as follows.
>
>
>
> 1. I think you have not specified the format of the ‘Block Deadline’ field
> of the BLOCK_INFO frame, and the ‘Timestamp’ field of the Timestamped ACK
> Frame. Is it a wall clock time of some format, microseconds elapsed since
> some event or something else?
>
>
>
>   In our implementation, the timestamp in the 'Timestamped ACK Frame'
> is 'Unix timestamp' (Unix time <https://en.wikipedia.org/wiki/Unix_time>).
> Using the field may require time synchronization between the sender and
> receiver.
>
>
>   In 'Block Deadline', the 'deadline' is a time duration in milliseconds,
> representing the requirement of the time limit from the application.
> 'Deadline' indicates the upper bound of allowed end-to-end transmission
> delay of the application data. For example, if a video streaming
> application wishes to have a 200ms buffer, it can set the parameter
> 'deadline' to 200. This parameter also provides a certain degree of
> flexibility for data scheduling and other processing.
>
>
>
> 2. If I understand correctly, only sender can drop a block not meeting the
> deadline, right?
>
>
>
>   This is a good question, and we have considered many situations.
>
>   We implement the action 'drop block' by canceling the corresponding
> stream in QUIC. So both sender and receiver can drop the block due to the
> application's command. The two endpoints can cancel the transmission of
> certain blocks according to a mutually agreed application layer
> regulation.  For example, the video streaming client may cancel the data
> blocks before a specific timestamp if the user fast forward.
>
>
>
>   In addition, the sender side can drop a block when it realizes the block
> will miss the deadline by an evaluation function. When we design DTP, we
> assume that if the application has specific requirements for the delay of a
> data block, then these data blocks cannot be used by the application when
> they arrive after the required delay. So when the sender detects a block
> will miss the deadline, instead of transmitting useless data, it can choose
> to drop the block actively to save the bandwidth. We didn't implement
> active dropping on the receiver side because the receiver had to
> synchronize the clock with the server to use the 'deadline' parameter,
> which introduced overhead.
>
>   All in all, both the sender and receiver can discard data blocks. Both
> ends allow canceling a block by the application, and the sender side can
> drop the data block actively by deciding whether it misses the deadline.
> How do you like these solutions? If you think there is a better strategy,
> we can implement and test it on DTP's current design framework.
>
>
>
> 3. Does the scheduler work on top of QUIC transport layer, submitting the
> next block once the previous one has been acknowledged by the receiver? Or
> should it be working within QUIC to schedule sending of already submitted
> blocks as individual QUIC streams?
>
>
>
>   This is also an excellent question. The scheduler of DTP is directly
> implemented inside QUIC because it can fully use the internal mechanism of
> QUIC, obtain the detected network conditions and handle the scheduling and
> discarding of data blocks more flexibly.
>
>   However, we believe it is feasible to implement an upper layer of QUIC
> using the idea of DTP. This design is somewhat similar to designing DTP as
> a jitter buffer layer. On the sender side, it can adjust the sending order
> by applying the given priority and deadline requirements from the
> application and using the abstraction of dividing data into blocks and
> matching each of them to a QUIC stream to send. On the receiver side, after
> receiving the data blocks, they can submit them in the order they arrive,
> or they can be cached and then submitted in order. The receiver side can
> also drop the data block according to whether the received block meets the
> deadline requirements or request the sender to send a specific block. It
> can also directly submit the received data to the application layer without
> additional processing at the receiver side. This way, DTP can be used as an
> intermediate layer between applications and QUIC to perform some decisions
> and optimizations. However, this way, DTP may not be able to learn details
> about data transmission from QUIC, such as which parts of a block arrived
> and which did not. Using the running status information from QUIC's
> internal components like the congestion control algorithm or the network
> status requires new API from QUIC.
>
>   In conclusion, DTP is currently implemented inside QUIC to access the
> network status information in QUIC but can be implemented on top of QUIC.
> Not sure if this can answer your question?
>
>
>
> ===
>
>
>
>   Thank you again for your great questions. According to your feedback, we
> will update our DTP draft to fix some uncertain expressions. If you have
> any further questions, we are always happy to respond.
>
>
>
> Best Regards.
>
>
>
> Chuan Ma
>
>
>
> Maxim Sharabayko <[e-mail]> 于2022年7月13日周三 23:02写道：
>
> Hi Chuan Ma,
>
>
>
> Thank you for sharing!
>
>
>
> I have a couple of questions after reading the draft.
>
>
>
>
>
> 1. I think you have not specified the format of the ‘Block Deadline’ field
> of the BLOCK_INFO frame, and the ‘Timestamp’ field of the Timestamped ACK
> Frame. Is it a wall clock time of some format, microseconds elapsed since
> some event or something else?
>
>
>
> 2. If I understand correctly, only sender can drop a block not meeting the
> deadline, right?
>
>
>
> 3. Does the scheduler work on top of QUIC transport layer, submitting the
> next block once the previous one has been acknowledged by the receiver? Or
> should it be working within QUIC to schedule sending of already submitted
> blocks as individual QUIC streams?
>
>
>
> Kind regards,
>
>
>
> - Maxim
>
>
>
>
>
> *From: *Moq <[e-mail]> on behalf of 马川 <
> [e-mail]>
> *Date: *Tuesday, 12. July 2022 at 17:13
> *To: *[e-mail] <[e-mail]>
> *Subject: *[Moq] Maybe a good building block for Media over QUIC --
> Deadline-aware Transport Protocol
>
> Dear all:
>
> Hello everyone, I'm one of the maintainers of the Deadline-ware Transport
> Protocol (DTP)[https://www.ietf.org/archive/id/draft-shi-quic-dtp-05.html].
> We suppose that DTP would be a good building block for creating protocols
> for Media over QUIC. In this email, I'd like to introduce our protocol to
> you, and we'd like to know your ideas about how to leverage our protocol to
> meet the requirement of this working group.
>
> ----
> Introduction of DTP
> ----
>
> Deadline-aware Transport Protocol (DTP) is an extension of QUIC to enable
> half-unreliable transmission and to provide block-based
> deliver-before-deadline transmission. It has the following features:
>
> 1. Using 'Block'(or segment) as the primary data unit to transport.
>
>     The concept of 'block' means a consistent and complete data segment
> like the video or audio frame in media streaming. The application can use
> a' block' as soon as it has reached the receiver side of the transmission.
>
> 2. Each 'Block' has attributes like 'priority' and 'deadline'.
>
>     The 'priority' means the importance of the data like audio blocks
> usually have a higher priority than video blocks in online conferences. DTP
> introduces 'deadline' to each block as the delay requirement from the
> application layer.
>
> 3. DTP uses plenty of methods to make blocks reach their destination in
> time.
>
>     DTP tries to meet the 'deadline' requirements by scheduling the order
> to send each 'block',  canceling some blocks when they have missed the
> 'deadline', and using other methods like enabling FEC for each block.
>
> In conclusion, DTP focuses on the deadline requirement of the applications
> and tries to create a loose time-sensitive network on the transport layer
> by using the abstraction of data blocks.
>
> ----
> Use cases
> ----
>
> The use cases of DTP mainly contain applications that generate blocks of
> data in a routine and might require feedback or response. Such applications
> include:
>
> 1. 360-degree video streaming
> 2. cloud VR gaming
> 3. cooperative augmented vehicular reality
>
> 4. online conference
>
> 5. live streaming
>
> These applications all require ultra-low latency and live media streaming.
> Media streaming with a latency of 2~5 RTT is the optimize-range of DTP and
> is also one of the targets of Media over QUIC. As a result, We think DTP
> might help handle some media transport scenarios in MOQ.
>
> ---
> Useful functions
> ---
>
> DTP offers a significant number of functions that might be helpful for
> live media transmission in Media over QUIC:
>
>
> 1. DTP is an extension of QUIC, so it inherits many good features from
> QUIC like stream multiplexing, unreliable datagram, and intrinsic security.
> 2. DTP integrates some standard designs of some frameworks of MOQ and
> exposes extensible interfaces like stream(block) scheduler, FEC, and
> segment mapping.
> 3. It is easy to add features to DTP to try your ideas like enabling
> multi-path transmission or overlaying.
>
>
> So it is a good idea to have a look at DTP and give it a try.
>
> ---
> Compare DTP with Warp
> ---
>
> The idea of DTP is similar to Warp [
> https://www.ietf.org/archive/id/draft-lcurley-warp-01.html]. They both
> have the concept of breaking media files into segments and adding
> attributes like a priority to the data segments. They both try to use the
> idea of scheduling and stream canceling to decrease the delay. The main
> difference between them is that Warp focuses more on transmitting control
> data of media, and DTP focuses more on time-sensitive transmission using
> data blocks. Compared to Warp, DTP tries to meet more transport scenarios
> aside from media streaming and tries to be more extensible.
>
> ---
> Implementations
> ---
>
> - DTP main page:  https://github.com/STAR-Tsinghua/DTP
>
> - Live stream prototype:
> https://github.com/STAR-Tsinghua/LiveEvaluationSystem
>
> - Other implementations
>
>   - TCP-like socket API: currently private
>
>   - Proxy: currently private
>
>
>
> We have our protocol available for test on Github [
> https://github.com/STAR-Tsinghua/DTP]. We tried to build some systems
> based on our protocol. You can find a prototype of a live stream system in
> our lab's other project [
> https://github.com/STAR-Tsinghua/LiveEvaluationSystem]. We also tried
> several ideas to integrate our protocol with some already deployed systems.
> We've met a lot of challenges in this process as the users don't want to
> change their protocol stack. We tried to simplify the process of using DTP
> by building a TCP-like socket API and developing a DTP proxy that can make
> a DTP tunnel forwarding TCP flows. If you are interested in any of our
> implementations, please get in touch with me to discuss more details.
>
>
>
> ---
> Conclusion
> ---
>
> To sum up, DTP is a low-level extension based on QUIC. It has the
> following features to make it a suitable building block for MOQ.
>
>
>
> 1. Media-oriented functions: It offers valuable functions to transmit live
> media like FEC and stream scheduler
>
> 2. Delay optimized: It is especially good at satisfying the delay
> requirement from the application
>
> 3. Flexible and Extensible: DTP is highly extensible and can support
> multiple protocols to handle media transmission like carrying encoding
> information and caching data in the middle.
>
>
> Thank you for the time to read this email. Please let me know if you have
> any questions. Any suggestion or idea is welcome.
>
> We are looking forward to your feedback. Your reply means a lot to us.
>
> Best regards.
>
> Chuan Ma
>
>
>
> *Maxim Sharabayko *
>
> Principal Research Engineer
>
> [image: Logo] <https://www.haivision.com/>
>
>
>
>
>
>
>
>
>