Re: [ipwave] Adaptive RESTful Real-time Live Streaming for Things (A-REaLiST)'

[Abhijan Bhattacharyya <[e-mail]>]

Hello Alex and Michael,
Thank you for your comments. May I please furnish my responses to both of you in this single mail.

<Alex>

>I have seen another drone from DLR that communicates with a van, but
>not 
>using CoAP. It uses TCP/IP family of protocols.
>
>I understand the need to transfer video (First-Person View, FPV), but
>I 
>do not understand how CoAP helps with video.
>

I agree to your observation. Most of the FPV applications from drones are using TCP/IP as the transport.  Before we designed the solution (A-REaLiST) we had analysed the performance of some popular off-the-shelve drone for applications which needed real-time navigation decisions to be delivered to the drone from a controller base station sitting remotely. The controller decision engines used to work based on the FPV inputs from the drone. The drone uses some proprietary variation of H.264 encoded video that is delivered over TCP/IP. There had been several instances of long video freezes. We investigated. It seemed that TCP's state machine became a bottleneck. Also, the drone needs to manage the process memory (which is not too high) in order to buffer the contents in case of a lossy transmission so that TCP's reliable delivery mechanism can function properly.

It would not be really worthwhile to compare CoAP, which is a RESTful application layer protocol, with TCP/IP transport. Rather, our approach was a bit different as will be explained below.

 >I understand the need to transfer video (First-Person View, FPV), but
>I 
>do not understand how CoAP helps with video.    

CoAP has been traditionally designed for lightweight transfer of small sensor data for constrained devices in LLN. So video may sound a bit wiered at first in the context of CoAP. However, video is nothing but a time-series information which needs to be rendered in real-time (and in strict real-time for FPV based remote decision engines). We found that CoAP has a unique feature that allows seamless switching of protocol semantics (incurring no additional cost on resources) between reliable and best-effort delivery while transmitting the information segments of the stream. So, depending on certain application driven criteria, you can classify some parts of your information as critical and some as non-critical. The critical segments are delivered reliably and the non-critical ones with best-effort. The "criticality" of an information segment is determined by the fact, whether that segment is essential to render the video at the receiver. A naive example: In a temporally compressed video stream, the I-frames may be considered as critical, while the p-frames and b-frames may be sent with best effort because the rendering engine essentially syncs with the I-frames. 

Moreover, our approach follows the tradition set by HTTP where video is considered just like any other web-traffic and HTTP-streaming  on TCP/IP (the de facto mechanism for OTT video delivery at this point) streams video over RESTful API calls. Same way, considering the requirements of LLN and constrained devices, we tried to extend CoAP so that we have a unified RESTful mechanism which is equally good in transferring small sensor information, control, telemetry and actuation information, as well as stream real-time time-series information like video. Being RESTful helps to seamlessly connect to other services on the web as well. 

>Video over IPv6 over OCB works fine between vehicles.
>
>In the future, with the hopeful development of 802.11-NGV (Next 
>Generation Vehicular), I expect latencies in the order of hundreds of
>
>micro-seconds, and 1Gbps bandwidths. I think video over IP, and
>lidar 
>streams over IP, would take advantage of that.
 
As you may notice from the above, the proposed solution is agnostic of what you use at the lower layers. One thing that we kept in mind while conceiving this solution is that, you may have a radio link with a promised high bandwidth, but you may have unpredictable intermittent connectivity too due to random impairment / obstructions . As per our experience, drones flying within WiFi zone in an indoor environment also suffer from such unpredictable link behaviour due to sporadic zero-radio zones. This overpowers the promise of bandwidth. We tried to design the solution keeping those practical problems in mind.

>But such a multi-feature protocol is also claimed by Google with QUIC
>
>protocol. Why not QUIC?

I am not sure whether QUIC can cater the kind of flexibility we wanted at the application layer (please forgive my limited knowledge on this). Also, we wanted to create a solution which should be based on something that already has a strong foundation through standardization and wide adoption (may not be particularly in drones right now - but who knows!). Furthermore, while performing the state-of-the-art studies we encountered this paper: https://dl.acm.org/citation.cfm?id=3083175 ( Not so QUIC: A Performance Study of DASH over QUIC ). But, I do agree that further investigations is worth carrying out to check the performance over QUIC. But, perhaps that should not disqualify the proposed solution leveraging CoAP at this point.

>I can look at the draft, but I need to understand why CoAP?

It becomes a bit of a race-condition. :) We have explained this "why" in section 2 of the draft (https://tools.ietf.org/html/draft-bhattacharyya-core-a-realist-00#page-5) . But to look into the draft you keep the explanation as a pre-requisite. To address this deadlock, I tried to briefly explain the idea above in this mail. If you find it satisfactory, then may I please request you have a look at the link and scroll through the entire draft subsequently. You may also please go through page# 70-81 of this deck: https://datatracker.ietf.org/meeting/103/materials/slides-103-core-consolidated-slides-07 . We presented this in Bangkok.

>(and yes, I do agree there are strong IP requirements for video in 
>drones, like multi-cameras, multiple laser, infrared, and others)

Thank you. Probably there is an extensive scope to contribute.

</Alex>

<Michael>

>And SCTP should be easy to deploy too if there is losing frames is
>preferred
>to winding up non-realtime.

First of all, we did not find much standard practices using SCTP for this kind of applications in recent implementations. Again, we are addressing the problem from the application-layer with a view of a generalized web infrastructure. Furthermore, if you may have noticed in the previous texts explaining the idea, losing frame is not the design concern. The concern is: That loss has to be an informed decision controlled by the application-layer. Also, if you may please have the opportunity to go through the draft, we are essentially proposing a rudimentary mechanism to marry application intelligence with protocol semantics.  

</Michael>

More views and comments will be more than welcome.

With Best Regards
Abhijan Bhattacharyya
Consultant / Scientist,
{Internet Protocols | 5G | Standardization}, 
TCS Research,
Tata Consultancy Services
Building 1B,Ecospace
Plot -  IIF/12 ,New Town, Rajarhat,
Kolkata - 700160,West Bengal
India
Ph:- [phone]
Cell:- [phone] | [phone]
Mailto: [e-mail]
Website: http://www.tcs.com
____________________________________________
Experience certainty.	IT Services
Business Solutions
Consulting
____________________________________________


-----"its" <[e-mail]> wrote: -----

>To: [e-mail]
>From: Alexandre Petrescu 
>Sent by: "its" 
>Date: 01/24/2019 04:48PM
>Subject: Re: [ipwave] Adaptive RESTful Real-time Live Streaming for
>Things (A-REaLiST)'
>
>"External email. Open with Caution"
>
>
>Le 24/01/2019 à 05:07, Abhijan Bhattacharyya a écrit :
>> Dear list,
>> We submitted a draft in CoRE with a technical proposal for
>efficient 
>> transfer of time-series stream and presented in IETF Bangkok. The 
>> immediate use case that we considered was to transfer First Person
>View 
>> (FPV) video from a remote drone to a control unit over IP based 
>> infrastructure. We built it on CoAP leveraging some of CoAP's
>unique 
>> features.
>
>Hi,
>
>Thank you for the message.
>
>I have seen another drone from DLR that communicates with a van, but
>not 
>using CoAP. It uses TCP/IP family of protocols.
>
>I understand the need to transfer video (First-Person View, FPV), but
>I 
>do not understand how CoAP helps with video.
>
>Video over IPv6 over OCB works fine between vehicles.
>
>In the future, with the hopeful development of 802.11-NGV (Next 
>Generation Vehicular), I expect latencies in the order of hundreds of
>
>micro-seconds, and 1Gbps bandwidths. I think video over IP, and
>lidar 
>streams over IP, would take advantage of that.
>
>Why CoAP?
>
>> The ambition is to create a protocol stack that would be 
>> equally good in sending small sporadic bytes (sensor data) as well
>as 
>> streaming infinite time-series (like live video) while ensuring
>good QoE 
>> despite being low on resource consumption.
>
>But such a multi-feature protocol is also claimed by Google with QUIC
>
>protocol. Why not QUIC?
>
>> The draft is available here: 
>> https://tools.ietf.org/html/draft-bhattacharyya-core-a-realist-00
>> 
>> Probably, the use case that we had in mind and the technology we 
>> proposed, both can be extended to the purview of this WG.
>> 
>> Requesting your views on this draft please.. If relevant then we
>can 
>> present this draft in Prague.
>
>I can look at the draft, but I need to understand why CoAP?
>
>Video per se is not a sufficient reason to use CoAP, I think.
>
>(and yes, I do agree there are strong IP requirements for video in 
>drones, like multi-cameras, multiple laser, infrared, and others)
>
>Alex
>
>> 
>> Here is the abstract for your ready reference:
>> 
>> This draft presents extensions to Constrained Application
>Protocol
>> (CoAP) to enable RESTful Real-time Live Streaming for improving
>the
>> Quality of Experience (QoE) for delay-sensitive Internet of
>Things
>> (IoT) applications. The overall architecture is termed
>''Adaptive
>> RESTful Real-time Live Streaming for Things (A-REaLiST)''. It
>is
>> particularly designed for applications which rely on real-time
>> augmented vision through live First Person View (FPV) feed from
>> constrained remote agents like Unmanned Aerial Vehicle (UAV),
>etc.
>> These extensions provide the necessary hooks to help solution
>> designers ensure low-latency transfer of streams and, for
>contents
>> like video, a quick recovery from freeze and corruption without
>> incurring undue lag. A-REaLiST is an attempt to provide an
>> integrated approach to maintain the balance amongst QoE,
>resource-
>> efficiency and loss resilience. It provides the necessary hooks
>to
>> optimize system performance by leveraging contextual
>intelligence
>> inferred from instantaneous information segments in flight.
>These
>> extensions equip CoAP with a standard for efficient RESTful
>> streaming for Internet of Things (IoT) contrary to
>HTTP-streaming in
>> conventional Internet.
>> 
>> 
>> 
>> 
>> Thank you. Looking forward.
>> 
>> P.S. The work has been accepted and presented in Wi-UAV workshop in
>
>> Globecom, Dec,  2018. The public link to the paper is yet to be
>available.
>> 
>> With Best Regards
>> Abhijan Bhattacharyya
>> Consultant / Scientist,
>> {Internet Protocols | 5G | Standardization},
>> TCS Research,
>> Tata Consultancy Services
>> Building 1B,Ecospace
>> Plot -  IIF/12 ,New Town, Rajarhat,
>> Kolkata - 700160,West Bengal
>> India
>> Ph:- [phone]
>> Cell:- [phone] | [phone]
>> Mailto: [e-mail]
><mailto:[e-mail]>
>> Website: http://www.tcs.com
>> ____________________________________________
>> Experience certainty. IT Services
>> Business Solutions
>> Consulting
>> ____________________________________________
>> 
>> =====-----=====-----=====
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>> and any attachments. Thank you
>> 
>> 
>> _______________________________________________
>> its mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/its
>> 
>
>_______________________________________________
>its mailing list
>[e-mail]
>https://www.ietf.org/mailman/listinfo/its
>
>