RE: QUIC for robotics and other time-sensitive applications
"Lubashev, Igor" <ilubashe@akamai.com> Wed, 01 May 2019 17:39 UTC
Return-Path: <ilubashe@akamai.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8F3F21200A2 for <quic@ietfa.amsl.com>; Wed, 1 May 2019 10:39:10 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.651
X-Spam-Level:
X-Spam-Status: No, score=0.651 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, KHOP_DYNAMIC=1.363, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=akamai.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id px6MGEVbu0ye for <quic@ietfa.amsl.com>; Wed, 1 May 2019 10:39:08 -0700 (PDT)
Received: from mx0a-00190b01.pphosted.com (mx0a-00190b01.pphosted.com [IPv6:2620:100:9001:583::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B55EB1200B6 for <quic@ietf.org>; Wed, 1 May 2019 10:39:07 -0700 (PDT)
Received: from pps.filterd (m0122332.ppops.net [127.0.0.1]) by mx0a-00190b01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x41HWBPg023509; Wed, 1 May 2019 18:39:06 +0100
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=akamai.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : mime-version; s=jan2016.eng; bh=3ZWmgKEAaJWykyXQxfFgE5Vg04jAcT3502uMUVwSn5U=; b=ixk+GxZz28fkGtjVS+Dc7izNou5sjrr3sVZT1AcYiAAVsV+ThzD5/8ipDcAYC4isYbpG wR0IHAgvq3hQuY3g2bpnmBRRGKrQhwysuAUiXW7JMLBgoPXCsNfsEMVLICNMnbDCpAq2 4Eo5wCx9a8TRQaGjsx5lS/5e111R822hd+sL11YBXskjotULJrR0Isnfrwcgobwn8zi2 EsMv7EeZ55IaLYFI3zqLNlPyEuUEKavFcy2XWtaVP1SV7uV9ZxpE1dh52BVw7jUVbh+t ldcBgAv/Ho+VQUSuDCeHlyfdttetEInItgV0CL8JUMOk4SnC2zCC8ZEUtwnGsrExILLj mg==
Received: from prod-mail-ppoint2 (prod-mail-ppoint2.akamai.com [184.51.33.19] (may be forged)) by mx0a-00190b01.pphosted.com with ESMTP id 2s6gsbxkru-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 May 2019 18:39:06 +0100
Received: from pps.filterd (prod-mail-ppoint2.akamai.com [127.0.0.1]) by prod-mail-ppoint2.akamai.com (8.16.0.27/8.16.0.27) with SMTP id x41HWCxw010000; Wed, 1 May 2019 13:39:04 -0400
Received: from email.msg.corp.akamai.com ([172.27.123.53]) by prod-mail-ppoint2.akamai.com with ESMTP id 2s4jdvhgtk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 01 May 2019 13:39:00 -0400
Received: from USMA1EX-DAG1MB5.msg.corp.akamai.com (172.27.123.105) by usma1ex-dag3mb5.msg.corp.akamai.com (172.27.123.55) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 1 May 2019 13:33:09 -0400
Received: from USMA1EX-DAG1MB5.msg.corp.akamai.com (172.27.123.105) by usma1ex-dag1mb5.msg.corp.akamai.com (172.27.123.105) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 1 May 2019 13:33:08 -0400
Received: from USMA1EX-DAG1MB5.msg.corp.akamai.com ([172.27.123.105]) by usma1ex-dag1mb5.msg.corp.akamai.com ([172.27.123.105]) with mapi id 15.00.1473.003; Wed, 1 May 2019 13:33:08 -0400
From: "Lubashev, Igor" <ilubashe@akamai.com>
To: Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com>, Ted Hardie <ted.ietf@gmail.com>, John Wason <johnwason1@gmail.com>
CC: IETF QUIC WG <quic@ietf.org>
Subject: RE: QUIC for robotics and other time-sensitive applications
Thread-Topic: QUIC for robotics and other time-sensitive applications
Thread-Index: AQHU/4kFmHgV+zSU3UK4cEKX71ffXaZVfeQAgAAD3YCAAM7+UIAAZ0EAgAAEioD//8f0EA==
Date: Wed, 01 May 2019 17:33:08 +0000
Message-ID: <fc0a9e35393a435a9b6f5662aef866e2@usma1ex-dag1mb5.msg.corp.akamai.com>
References: <542b6445-de16-8b13-4ffc-f1ca46c535da@gmail.com> <CA+9kkMDD-DyReh-mCYgF==t8Lcy9tVOrABiWhyMx2VgcwfbOtw@mail.gmail.com> <ae90d734-43f2-412d-123c-03421b44d6c8@gmail.com> <df610cdb89614e459f64292ca02c2041@usma1ex-dag1mb5.msg.corp.akamai.com> <3c34ef0d-40ae-3ec0-6a60-2da4dab1b339@gmail.com> <CAN1APdeq8-tTOki4-KHPLBDxJNupJZK8jePjsfT7XEeJhR7fUA@mail.gmail.com>
In-Reply-To: <CAN1APdeq8-tTOki4-KHPLBDxJNupJZK8jePjsfT7XEeJhR7fUA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: [172.19.36.42]
Content-Type: multipart/alternative; boundary="_000_fc0a9e35393a435a9b6f5662aef866e2usma1exdag1mb5msgcorpak_"
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-05-01_08:, , signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905010109
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-05-01_08:, , signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905010109
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/PP_NBczlw1jnmTvQLlMp7JFgkx8>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 01 May 2019 17:39:10 -0000
The QUIC library API may include a facility to set/update stream priorities for the sender of data. Webrtc-quic may include the same w/o any change to the underlying QUIC protocol. However, if a request for a priority change of a stream is originating from the receiver of the data, QUIC transport has no facility to relay this information to the sender. An application protocol would need to handle this. I think it is a useful extension for QUIC v2 to push stream priority to the transport, since the transport can manage this control signal more efficiently. From: Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com> Sent: Wednesday, May 01, 2019 12:38 PM To: Lubashev, Igor <ilubashe@akamai.com>; Ted Hardie <ted.ietf@gmail.com>; John Wason <johnwason1@gmail.com> Cc: IETF QUIC WG <quic@ietf.org> Subject: Re: QUIC for robotics and other time-sensitive applications QUIC does not prioritise streams at the transport layer, that is an implementation detail although pacing is recommended. If you bring you own application layer protocol you can decide how to schedule data across streams within the flow control constraints. You could add priorities to your application level custom API. If you layer a protocol on top of H3 you are more constrained since you must assume other implementations that won’t necessarily do what you want them to do. On 1 May 2019 at 18.22.22, John Wason (johnwason1@gmail.com<mailto:johnwason1@gmail.com>) wrote: Igor, It appears I misunderstood the distinction between per-stream partial reliability and intra-stream partial reliability. If I send one partial-reliability message per stream, using RESET frames should be sufficient. How this is exposed as an API is important, since I am not sure the webrtc-quic team has considered this yet. In response to your questions 1) Each message contains a header that should be sufficient for the receiver to understand the incoming stream 2) Does QUIC provide the type of priority that you suggest? My understanding is that QUIC multiplexes streams using a priority based ordering. If I can emulate partial-reliability using RESET frames to cancel entire streams, that should be sufficient for my needs. -John On 5/1/2019 10:33 AM, Lubashev, Igor wrote: John, I am happy to resurrect that draft-lubashev-quic-partial-reliability for QUIC v2 (some people preferred version 02, which actually gets you a bit more features at the cost of a bit more complexity). I would like to first understand what partially reliable streams give you that is not available with regular streams. I.e. what are you missing if you use STREAM (with a possible subsequent RESET) for messages? (You cannot use DATAGRAM for messages, since you would like the transport to facilitate retransmission until expiration.) My draft-lubashev-quic-partial-reliability lists a few things that stream-per-message does not get you: 1. If you have multiple application message streams, you would need a header for each QUIC stream to identify the application stream. This seems like a small burden, except for very short messages. * If the order of messages in an application stream matters, you would need to reorder them yourself. This cannot be very complicated for the application. 1. QUIC per-application-stream flow control is rendered ineffective. I am actually not a huge fan of QUIC per-stream flow control, as I strongly prefer stream priorities as a signal. But I now know of a case when per-stream flow control is, indeed useful (when there is only one slow draining stream that uses up the entire connection buffer before a higher-priority stream shows up). Are there any other benefits you see in partially reliable streams? * Igor From: John Wason <johnwason1@gmail.com><mailto:johnwason1@gmail.com> Sent: Tuesday, April 30, 2019 5:52 PM To: Ted Hardie <ted.ietf@gmail.com><mailto:ted.ietf@gmail.com> Cc: IETF QUIC WG <quic@ietf.org><mailto:quic@ietf.org> Subject: Re: QUIC for robotics and other time-sensitive applications Thanks Ted, this looks very interesting. Based on these documents the DETNET sits below the level of QUIC that we are currently discussing and would mainly affect the IP header format of the resulting UDP packets. The QUIC protocol will still require some form of partial reliability to function on a DETNET network. -John Wason On 4/30/2019 5:37 PM, Ted Hardie wrote: Hi John, Thanks for the message. If you have not already reviewed the work of DETNET (https://datatracker.ietf.org/wg/detnet/about/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_wg_detnet_about_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=xLXCzRykwXzX7bQ_EfAh4Rea2sEzel9wgnprQAuwPwc&e=>) here in the IETF, it might be useful to do so, as there are a number of elements of your use case which appeared to be shared with their work. In particular, it might be useful to consider whether the integration of their proposed IP data plane (https://datatracker.ietf.org/doc/draft-ietf-detnet-dp-sol-ip/?include_text=1<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dietf-2Ddetnet-2Ddp-2Dsol-2Dip_-3Finclude-5Ftext-3D1&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=TIzKr-IO44yjBfPhhPp5mldz8QENRQGE-4WD7bbdn5k&e=>) with QUIC would be valuable in achieving your timing goals. best regard, Ted Hardie On Tue, Apr 30, 2019 at 12:15 PM John Wason <johnwason1@gmail.com<mailto:johnwason1@gmail.com>> wrote: Greetings QUIC WG, The discussion on Issue #92 of webrtc-quic suggested that I post my requirements to this list. I am a robotics engineer and the developer of Robot Raconteur, a communication framework for robotics and automation systems (https://github.com/robotraconteur/robotraconteur<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_robotraconteur_robotraconteur&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=VIzDVXjr63UwHbT0rvYCzS91VAZarySeWftFNCsC2hE&e=>). Robotics applications require different data types and QoS when compared to business or web applications. Robotics applications need real-time communication of isochronous data, for instance constant updates on the position of the joints of the robot, alongside generic non-time-sensitive data. Currently most robotics applications are using TCP (Robot Raconteur, ROS*, OPC/UA), UDP (DDS, ROS2, ROS*), or some form of custom Ethernet protocol (Ethernet/IP, EtherCAT, PowerLink). To add more confusion to the situation, there is the recent introduction of IEEE 802.1Q Time Sensitive Networking (TSN) which claims to allow TCP and UDP to achieve hard real-time performance. While there are many robotics protocols out there, only Robot Raconteur and OPC/UA are compatible with existing web and cloud infrastructure. (Some limited support for ROS/ROS2 has become available on AWS, but the design of ROS does not play very well outside of local networks.*) Robot Raconteur implements web browser and web server support by allowing the websocket protocol to be used. Clients can use a URL that specifies that the connection is a websocket, and incoming TCP connections can detect if the connection is the Robot Raconteur protocol or an incoming websocket connection. Robot Raconteur running inside an ASP.NET<https://urldefense.proofpoint.com/v2/url?u=http-3A__ASP.NET&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=9uchBdRzcuqtIB6lZuT0wlGoelFWgIzMVT4hTBGEVxc&e=> web server has been demonstrated, and there is no technical reason that other server technologies cannot accept Robot Raconteur connections through websockets. (OPC/UA uses SOAP for its web based transport, but support is very poor and only available in a few implementations.) The main issue limiting Robot Raconteur for more demanding applications is the use of TCP for data transport. TCP suffers from "head-of-line" blocking, which means that if stale data encounters an error it can stop fresh data from reaching the other peer. There is also the problem of large messages blocking the transmission of smaller, more time sensitive messages on the connection. QUIC appears to solve most of these problems, and the webrtc-quic support under development provides a lot of new and interesting opportunities for integrating robotics applications with web technology. By adopting QUIC, the resulting software will remain completely compatible with web infrastructure. Transmitting time-sensitive data requires the ability to multiplex (and prioritize) messages and drop messages if the data contained becomes stale. The multiplexing issue is solved using the "streams as messages" concept in QUIC, with built in priority handling. This design looks very promising, and considerably simpler to use compared to SCTP streams. The current issue that is somewhat questionable is the implementation of partial reliability. The discussion right now is focused on DATAGRAM extensions (draft-pauly-quic-datagram-02), which allow for "raw" access to the socket so protocols such as RTP can tunnel through QUIC. This requirement is quite a bit different than what is required for an application like Robot Raconteur. Robot Raconteur is based on message passing, where either the entire message is received or the entire message is cancelled because the data contained is stale. The message may be considerably larger than MTU, and the QoS may be "partial," meaning that a set number of attempts may be made to resend, and/or there is an expiration time on the message. This capability is discussed in draft-lubashev-quic-partial-reliability-03, and is similar to how SCTP handles partial reliability. If only a raw DATAGRAM capability is provided, then partial reliability will need to be built on top of QUIC, defeating the purpose of using QUIC for this application. Other applications such as online gaming have very similar performance requirements, so I think that the partial reliability capability will be important to many future users. The complexity and required effort to add the partial reliability capability to QUIC is low, so I hope that this capability is not abandoned. Robot Raconteur has received a grant from the United States ARM Institute and New York State to begin commercializing the technology. This project will be done in partnership with Wason Technology, LLC, Rensselaer Polytechnic Institute, Southwest Research Institute, and United Technologies Research Center. The project starts soon, and considerable development effort will be invested in the Robot Raconteur technology starting early summer. Integrating QUIC is one of the tasks that I hope to undertake during this project. Please let me know if you have any questions about Robot Raconteur or the requirements for QUIC in robotics applications. -John Wason (* I am aware that there are some caveats to how ROS/ROS2 interact with the wider world, but the technologies are not built on or compatible with web standards so an in depth discussion is not relevant for the current topic. ROS does have a UDP transport, but it is not commonly used.) -- John Wason, Ph.D. Wason Technology, LLC 16 Sterling Lake Rd Tuxedo, NY 10987 (518) 279-6234 wason@wasontech.com<mailto:wason@wasontech.com> -- John Wason, Ph.D. Wason Technology, LLC 16 Sterling Lake Rd Tuxedo, NY 10987 (518) 279-6234 wason@wasontech.com<mailto:wason@wasontech.com> -- John Wason, Ph.D. Wason Technology, LLC 16 Sterling Lake Rd Tuxedo, NY 10987 (518) 279-6234 wason@wasontech.com<mailto:wason@wasontech.com>
- QUIC for robotics and other time-sensitive applic… John Wason
- Re: QUIC for robotics and other time-sensitive ap… Ted Hardie
- Re: QUIC for robotics and other time-sensitive ap… John Wason
- RE: QUIC for robotics and other time-sensitive ap… Lubashev, Igor
- Re: QUIC for robotics and other time-sensitive ap… John Wason
- Re: QUIC for robotics and other time-sensitive ap… Mikkel Fahnøe Jørgensen
- Re: QUIC for robotics and other time-sensitive ap… John Wason
- RE: QUIC for robotics and other time-sensitive ap… Lubashev, Igor
- Re: QUIC for robotics and other time-sensitive ap… John Wason
- Re: QUIC for robotics and other time-sensitive ap… Mikkel Fahnøe Jørgensen
- Re: QUIC for robotics and other time-sensitive ap… Lucas Pardue
- Re: QUIC for robotics and other time-sensitive ap… Phillip Hallam-Baker