Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn

Praveen Balasubramanian <pravb@microsoft.com> Thu, 12 July 2018 16:25 UTC

Return-Path: <pravb@microsoft.com>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A520E130E9A; Thu, 12 Jul 2018 09:25:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.01
X-Spam-Level:
X-Spam-Status: No, score=-2.01 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_HIGH=-0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3baXLfK5qTup; Thu, 12 Jul 2018 09:25:44 -0700 (PDT)
Received: from NAM05-DM3-obe.outbound.protection.outlook.com (mail-eopbgr730118.outbound.protection.outlook.com [40.107.73.118]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3D957130E89; Thu, 12 Jul 2018 09:25:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XuNv93+y1pH3k8cnuNP1QmtjSgyDIy5cBSp6mzmHZOs=; b=FL3qaepGCZJHO8b1WmiZaYXTgCeaHMRQ3U9xlVapnhtj/HLiQqHHEJxYXdOZzt7MjlPLJTZc50E2yAgWMeG638Hs9u9YUeDI+KE5zSYq3xfaIENkbHtUEXy+YxdAZroVXrZZv9BTQAJk+HYRwRvXGKHNdQ9f6qK+yp28sc2UsVI=
Received: from MWHPR21MB0191.namprd21.prod.outlook.com (10.173.52.137) by MWHPR21MB0829.namprd21.prod.outlook.com (10.173.51.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.973.12; Thu, 12 Jul 2018 16:25:41 +0000
Received: from MWHPR21MB0191.namprd21.prod.outlook.com ([fe80::60a2:45d4:b972:fca4]) by MWHPR21MB0191.namprd21.prod.outlook.com ([fe80::60a2:45d4:b972:fca4%6]) with mapi id 15.20.0973.013; Thu, 12 Jul 2018 16:25:41 +0000
From: Praveen Balasubramanian <pravb@microsoft.com>
To: Mirja Kühlewind <mirja.kuehlewind@tik.ee.ethz.ch>, Yuchung Cheng <ycheng@google.com>
CC: "tcpm@ietf.org" <tcpm@ietf.org>, "draft-ietf-tcpm-accurate-ecn@ietf.org" <draft-ietf-tcpm-accurate-ecn@ietf.org>
Thread-Topic: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn
Thread-Index: AdNsZwbKZl7dQfJhQeSI1pXOpUN55hIGf9OAAUAqXJAWJtFgAAHJBo4AAAE8zAAAIRU6AAALHRSAAADIbAAAAHfNAA==
Date: Thu, 12 Jul 2018 16:25:41 +0000
Message-ID: <MWHPR21MB01916BC2DD236143D171DC2EB6590@MWHPR21MB0191.namprd21.prod.outlook.com>
References: <AM5PR0701MB25477BD5BEB403A98AA2B983933F0@AM5PR0701MB2547.eurprd07.prod.outlook.com> <44FDECF5-A031-4343-BA1A-AE0D9C2C078C@tik.ee.ethz.ch> <VI1PR0701MB2558F5DE5FCE5CDC6A43F94793D30@VI1PR0701MB2558.eurprd07.prod.outlook.com> <E729457B-96C5-493D-9B14-70663C24DFB4@tik.ee.ethz.ch> <db66271d-3654-6066-fecc-a405bb88b7f5@bobbriscoe.net> <CAK6E8=dkuyD+PJv9+4iwdXNu0pEv8n59acHx1Q-yBeCBQ=CcEg@mail.gmail.com> <646D10B9-FED7-4E2D-9A9F-0C052F1C908D@tik.ee.ethz.ch> <CAK6E8=evQwrEgYpmbu7GW1oTAkz-xG5HzyRW5e=uBsmJfdjfAQ@mail.gmail.com> <B0B81087-B740-43D5-BB79-FBF8DA9A2FD9@tik.ee.ethz.ch>
In-Reply-To: <B0B81087-B740-43D5-BB79-FBF8DA9A2FD9@tik.ee.ethz.ch>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [2001:4898:80e8:b:f1c6:cdc0:5b39:ca82]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; MWHPR21MB0829; 6:lEVAW1U3GljQx5jjzoddeUM69baXSTSQ4Y52kf4tTb1bH/QK/kfvwjP5ByEl9TrndvFxYhh6NtAqyMQxglF9T1tt1C5hIPaZYslqu3xNb7vKbnrBMaKOAtoUjHFzbOv4T6UVSx79TVKxOgkIIgQgdrjkEd2x3PV6OfJrngsnjjhhLgwk/nfxZJKhNZKYMd5boyb6cs63uChsEg0fpdDmxW541BxlfgKpbDOO7bkCkQs+N60rPRo9k+NacI3I4p9pfN6IVMEo8onGu/kWkFUQSuQXvdmJ2ThGeopYQf8uvet0kBbbJ8EBKKetAZpftJsdMVaSxcG6LTyGbQ637JkfLb7NRhb5mWweydA5NsGmhxarPPBYnXGXC62jrLrZWEEV82Ah02Iwa2GXvErHoY6gdTBlDfCch1Ul43g8hdfm919zIWx5ujMgEH/BLpdawMG2zKwAFaCqMNU/lMCwsQatvQ==; 5:W113PJod2BEfG9+/bYDNYE4tIfp1OKJy5n7ZQ+4RtWm9x67Zo2FSXKrJDOtC5sfJ/BwBfaRcKQ0NrjR12N7iRAANCsKLfEf2Dq9WU3clJsKXKfLEws6VQZmwxwtOR8UF4sVVLHMi7uC6h8LmNW730YqUp+S6uuNvj9broUscpRs=; 7:A1nsf7pK/aRFoPt56fdGxj50H5VsoZuYu3NeoLGzNOUcuU4dvo+WhVcVNeCVbRiuVZ6o8AygkIn2Y5+KAUSofPWpIgS/1Gzu7NpDQb/h8URGPG2qSR0VG++wqCeczJomqUbF7SQyEfOE8DdqnT0L6UhRDb63BsUWDBPKVwwygRLbVuJliwYJo1/nriG7SCkYARQg2iL2dQpSHSFozQMoXO4ZtkWld98uexrEdTMxID+CyTAyCxtL/9z7C8XPyWAy
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-ms-office365-filtering-correlation-id: aafaaa25-e685-483a-f2b8-08d5e8141c73
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: UriScan:(109105607167333); BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989117)(5600053)(711020)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(48565401081)(2017052603328)(7193020); SRVR:MWHPR21MB0829;
x-ms-traffictypediagnostic: MWHPR21MB0829:
authentication-results: spf=none (sender IP is ) smtp.mailfrom=pravb@microsoft.com;
x-microsoft-antispam-prvs: <MWHPR21MB0829F783BB9220D362E0DE1DB6590@MWHPR21MB0829.namprd21.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(28532068793085)(189930954265078)(82608151540597)(109105607167333)(788757137089)(211936372134217)(100405760836317)(153496737603132)(219752817060721);
x-ms-exchange-senderadcheck: 1
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3002001)(10201501046)(93006095)(93001095)(3231311)(944501410)(52105095)(2018427008)(6055026)(149027)(150027)(6041310)(20161123560045)(20161123558120)(20161123564045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011)(7699016); SRVR:MWHPR21MB0829; BCL:0; PCL:0; RULEID:; SRVR:MWHPR21MB0829;
x-forefront-prvs: 0731AA2DE6
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(136003)(366004)(346002)(376002)(39860400002)(396003)(53754006)(189003)(199004)(52314003)(13464003)(446003)(97736004)(7736002)(8936002)(256004)(14444005)(33656002)(8990500004)(6436002)(186003)(14454004)(93886005)(55016002)(561944003)(8676002)(81166006)(81156014)(966005)(305945005)(16200700003)(53946003)(46003)(9686003)(10090500001)(478600001)(5660300001)(6306002)(74316002)(2900100001)(476003)(53936002)(105586002)(68736007)(99286004)(11346002)(7696005)(86612001)(22452003)(106356001)(25786009)(10290500003)(110136005)(316002)(54906003)(6246003)(5250100002)(6116002)(2906002)(229853002)(486006)(575784001)(76176011)(102836004)(4326008)(86362001)(53546011)(6506007)(559001)(569006); DIR:OUT; SFP:1102; SCL:1; SRVR:MWHPR21MB0829; H:MWHPR21MB0191.namprd21.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts)
x-microsoft-antispam-message-info: zYauDOo6uyucYhsrpCOiZb6TrZweGmq4tvkkf3nk+6BRKemuSdVFYQpukS1h6xqlnX4F6ikL0gSOLaMHfB8TtSNJRSfT/yHFHblSAU26zuxk8mECbwl3H7w7WlMvEslCJyrWlDDiJRoRWMzuwsy+63sYhTW9WJM9qkIS6AmPZ2d9bp7WgNdHTlBUW0NnZ0+EuI+60ta78iEb2pOizj6e490J0m2Qedq5+aT3+aMwS0uYbAB33x8c25y2i/RYBsRZ886/wIHyrTRGI6j42cxm2CzSJIf4KdjNDQnswoIxEkZhwqz888x3SGHK1aUk55hVPsnC5rb4vi3q/ON15w/0bqmTfgQwWeGZAI229pYYINE=
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-Network-Message-Id: aafaaa25-e685-483a-f2b8-08d5e8141c73
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Jul 2018 16:25:41.5702 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR21MB0829
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/18imQLfVLXSu_jiRMGNMb6lu754>
Subject: Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.27
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 12 Jul 2018 16:25:50 -0000

I had similar concerns with the new TCP option at the last tcpm meeting. I was requesting that we make the option truly optional but I got some feedback that implementations should honor and process the option on receive even if they don’t generate it. I'd prefer if we make the accurate measurements part optional. The biggest advantage of Accurate ECN is the negotiation mechanism that DCTCP lacks and the same negotiation can be used on the Internet for any scalable TCP that wants to rely on accurate ECN.

Re. GRO/LRO/RSC: https://docs.microsoft.com/en-us/windows-hardware/drivers/network/exception-conditions-that-terminate-coalescing see condition 4 for an exception that terminates coalescing: "The segment contains one or more TCP options other than the TCP timestamp option". So while software implementations of coalescing can be updated, existing hardware will cause poor performance with new TCP options that are present in a lot of data packets. 

>> because you want to have feedback for control packets as well
I don’t fully understand this. ECN++ allows marking of control packets and defines a response. Do we really need the TCP option for processing the feedback?

-----Original Message-----
From: tcpm [mailto:tcpm-bounces@ietf.org] On Behalf Of Mirja Kühlewind
Sent: Thursday, July 12, 2018 9:04 AM
To: Yuchung Cheng <ycheng@google.com>
Cc: tcpm@ietf.org; draft-ietf-tcpm-accurate-ecn@ietf.org
Subject: Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn

Hi Yuchung,

please see below.

> Am 12.07.2018 um 11:41 schrieb Yuchung Cheng <ycheng@google.com>:
> 
> Yes packets with ACE options and (different) ACE-counter header would 
> break GRO. There're many legacy h/w and s/w.

I’m not the expert here but my understanding is that is would not break GRO but could not make actual use or it if the ACE counter changed or the option is present. However, usually your will only have a small number of CE marks every couple of RTTs, and the counter only changes if you CE marks/the option is only present a few times per RTT. So the impact should be rather low. But this clear something to evaluate for the experiment.

> 
> Even without option, ACE header only allows reflecting up to 8 packets 
> for receiver segmentation offload and ACK suppression?

Same as above, it can only up to 8 CE marked packets per ACK, however, CE marking rater are expected to be rather low. ACK suppression should probably not be used if the ACE counter has changes, however, usually the ACE counter stays stable for multiple RTTs and then ACK suppression is not a problem. However, not sure I understood you question here correctly…?

> 
> The appendix on ack loss causes ambiguity is good: but the pattern 
> drops specifically the state-switch (CE<->noCE) ACK which only happens 
> on delayed ACK. Is that pattern common? - or can we address most of 
> that by delaying less ACKs?

I guess you have a 50% chance today to hit a delayed ACK. I guess delaying less (where you delay every second ACK today) would mean ACK very packet, and thus double ACK load on the network. I guess on today networks that actually in most cases not a problem, however, there might be specially cases where it is. However, that’s problem an independent question to evaluate.

> 
> I am all for making ECN more accurate for wide area beyond DCTCP, but 
> am evaluating the pros/cons. What if we just 1. negotiate 'better' ECN 
> via new SYN option

Not sure I understand your proposal correctly, but I assume you mean negotiate via SYN option and then just use the ACE counter in the header? If so, I don’t understand why you see the negotiation part in the TCP header as the problem?

> 2. mark all packets like ECN++

That would be nice but here you actually need AccECN because you want to have feedback for control packets as well. AccECN is providing this feedback. AccECN does not change the „use of ECN“; that’s what we have ECN++ for; both thing ideally would be deployed together however. Was that your questions?

Mirja


> 
> 
> 
> 
> 
> 
> On Thu, Jul 12, 2018 at 3:23 AM, Mirja Kühlewind 
> <mirja.kuehlewind@tik.ee.ethz.ch> wrote:
>> Hi Yuchung,
>> 
>> the question if you need this „more“ on accuracy really depends on the use case. If you use DCTCP as today, the ACE counter in the TCP is probably sufficient and you might not want to pay the additional overhead in your data center (that why the option is actually optional).
>> 
>> If you however, e.g., have very differently sized packets, then the byte counter in the option could give you a more accurate signal. Or if you are also interested in the ECT(1) counter, you need the option. Further the ACE counter also give your feedback on control packet and the option enables you to distinguish between CE-amrked payload and control packets, which can also become important when experimenting with making all packets ECN-capable.
>> 
>> Given accECN is a general feedback mechanism that in fact is designed to enable new future uses of the ECN signal, we wanted to keep all these option available while making is still as simple as possible and as flexible as possible.
>> 
>> Mirja
>> 
>> 
>> 
>>> Am 11.07.2018 um 14:35 schrieb Yuchung Cheng <ycheng@google.com>:
>>> 
>>> Hi Bob,
>>> 
>>> Neal and I evaluated the earlier draft. It is well-thought out but 
>>> we're concerned about the options. Option is not mandatory but the 
>>> lack of it also reduces accuracy. Option runs into space issues w/ 
>>> SACK and offload issues w/ TSO/GRO. They can be addressed for sure 
>>> but aren't easy.
>>> 
>>> We're curious how much more "accuracy" it buys over current 
>>> DCTCP-style ECN. Is there any study to show trade-offs of 
>>> full-ACE-w-options vs ACE-wo-options vs current DCTCP-ECN?
>>> 
>>> 
>>> On Wed, Jul 11, 2018 at 11:00 AM, Bob Briscoe <ietf@bobbriscoe.net> wrote:
>>>> Michael, tcpm list,
>>>> 
>>>> As well as addressing your points, as Mirja has already mentioned 
>>>> below, we added a whole new appendix giving the rationale for the 
>>>> bits and codepoints that AccECN has proposed to use on 1) the SYN 
>>>> and 2) SYN/ACK. A 3rd subsection also identifies space for future 
>>>> evolution. It also points to where rationale was already given in the body of the draft.
>>>> 
>>>> The appendix is in the draft submitted last week, available here:
>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
>>>> ols.ietf.org%2Fhtml%2Fdraft-ietf-tcpm-accurate-ecn-07%23appendix-B&
>>>> amp;data=02%7C01%7Cpravb%40microsoft.com%7C2e4980f91d89410fa58708d5
>>>> e8111fbb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6366700826104
>>>> 53620&amp;sdata=J5qwrZ8H%2BbfriTF1jxJXZldX6%2BA0Uy%2FTV0lGdH2UbWY%3
>>>> D&amp;reserved=0
>>>> 
>>>> We'd be interested to hear whether this allays your concerns.
>>>> 
>>>> We have asked to present this in Montreal as well.
>>>> 
>>>> Cheers
>>>> 
>>>> 
>>>> Bob
>>>> 
>>>> 
>>>> On 02/07/18 16:54, Mirja Kühlewind wrote:
>>>>> 
>>>>> Hi Micheal,
>>>>> 
>>>>> I addressed a couple of your comments below.
>>>>> 
>>>>> For the other, bigger comments regarding extensibility, that I did 
>>>>> not yet address below, we plan to add a new section to the 
>>>>> appendix to explain extensibility options as previously discussed 
>>>>> by mail. We will probably send a separate email on that part.
>>>>> 
>>>>> Mirja
>>>>> 
>>>>> 
>>>>>> Am 12.03.2018 um 01:59 schrieb Scharf, Michael (Nokia - 
>>>>>> DE/Stuttgart)
>>>>>> <michael.scharf@nokia.com>:
>>>>>> 
>>>>>> Hi Mirja,
>>>>>> 
>>>>>> Thanks a lot for the explanation. I won't follow-up on some of 
>>>>>> the editorial suggestions.
>>>>>> 
>>>>>> Yet, I continue to believe that some formal wording in the 
>>>>>> document needs to change, as explained below.
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>> Michael (with no hat on)
>>>>>> 
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Mirja Kühlewind [mailto:mirja.kuehlewind@tik.ee.ethz.ch]
>>>>>>> Sent: Monday, March 05, 2018 1:54 PM
>>>>>>> To: Scharf, Michael (Nokia - DE/Stuttgart) 
>>>>>>> <michael.scharf@nokia.com>
>>>>>>> Cc: draft-ietf-tcpm-accurate-ecn@ietf.org; tcpm@ietf.org
>>>>>>> Subject: Re: Comments on draft-ietf-tcpm-accurate-ecn
>>>>>>> 
>>>>>>> Hi Micheal,
>>>>>>> 
>>>>>>> thanks for your feedback and sorry for my late reply.
>>>>>>> 
>>>>>>> Please see inline.
>>>>>>> 
>>>>>>>> Am 03.12.2017 um 20:17 schrieb Scharf, Michael (Nokia - 
>>>>>>>> DE/Stuttgart)
>>>>>>> 
>>>>>>> <michael.scharf@nokia.com>:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> I have read draft-ietf-tcpm-accurate-ecn-05 (without the 
>>>>>>>> appendix). I
>>>>>>> 
>>>>>>> believe this document needs further work before moving forward.
>>>>>>>> 
>>>>>>>> Please find below my comments marked as [ms]. I have read the
>>>>>>> 
>>>>>>> document independent of the review from Gorry. I apologize if 
>>>>>>> there is duplication.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> Michael (with no hat on)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ******************************
>>>>>>>> 
>>>>>>>> * Abstract:
>>>>>>>> 
>>>>>>>>  Recently, new TCP mechanisms like Congestion Exposure (ConEx) 
>>>>>>>> or Data
>>>>>>> 
>>>>>>> Center TCP
>>>>>>>> 
>>>>>>>>  (DCTCP) need more accurate ECN feedback information whenever 
>>>>>>>> more  than one marking is received in one RTT.
>>>>>>>> 
>>>>>>>> [ms] I don't think this statement is fully backed by RFC 8257. 
>>>>>>>> I suggest to
>>>>>>> 
>>>>>>> remove this, or replace it by a more generic statement that more 
>>>>>>> accurate information can be useful for several TCP extensions.
>>>>>>> 
>>>>>>> I disagree. Both ConEx and DCTCP need more accurate information. 
>>>>>>> They do not need the mechanism that is specified in this draft, 
>>>>>>> however, this is not what the sentences is saying.
>>>>>> 
>>>>>> In my understanding (as a non-native speaker), the use of the word "need"
>>>>>> is not correct here. DCTCP as specified in RFC 8257 can be 
>>>>>> implemented without any such mechanism.
>>>>>> 
>>>>>> What would work for me is something of the form "... Data Center 
>>>>>> TCP cannot get precise ECN feedback whenever more than one 
>>>>>> marking is received in one RTT“.
>>>>> 
>>>>> This is not correct. DCTP need more than one feedback signal per 
>>>>> RTT and therefore cannot use RFC3168; instead it implement it’s 
>>>>> own feedback mechanism. However, to avoid confusion such that 
>>>>> people could assume DCTP would not work without the accECN scheme 
>>>>> as specified in this doc, I rephrased to:
>>>>> 
>>>>> "Recently, proposed
>>>>>      mechanisms like Congestion Exposure (ConEx <xref 
>>>>> target="RFC7713"/>),
>>>>>      DCTCP <xref target="RFC8257"/> or L4S <xref
>>>>>      target="I-D.ietf-tsvwg-l4s-arch"/> need to know when more than one
>>>>>      marking is received in one RTT which is
>>>>>      information that cannot be provided by the feedback scheme as 
>>>>> specified in
>>>>>      <xref target="RFC3168"/>."
>>>>> 
>>>>> 
>>>>>>>>  This document specifies an
>>>>>>>>  experimental scheme to provide more than one feedback signal 
>>>>>>>> per RTT  in the TCP header.  Given TCP header space is scarce, 
>>>>>>>> it overloads  the three existing ECN-related flags in the TCP 
>>>>>>>> header and provides  additional information in a new TCP option.
>>>>>>>> 
>>>>>>>> [ms] This statement needs to be rewritten to correctly reflect 
>>>>>>>> what is
>>>>>>> 
>>>>>>> requested from IANA. My understanding is that this experimental 
>>>>>>> document asks for allocation of a reserved TCP header flag. This 
>>>>>>> needs to be called out prominently, IMHO. In addition, since 
>>>>>>> this is not a standard, the suggested experimentation with the 
>>>>>>> main TCP header must IMHO be explicitly mentioned. I also 
>>>>>>> suggest to have later in a document a section that explicitly 
>>>>>>> explains why it is appropriate to modify the main TCP header in 
>>>>>>> an experiment.
>>>>>>> 
>>>>>>> I don’t know if any requirement that IANA assignment need to be 
>>>>>>> called out in the abstract but we can do that. However, I 
>>>>>>> believe the question if this document should or should not 
>>>>>>> assign the bit is still not completely solved, or is it?
>>>>>> 
>>>>>> I believe this question will have to be reviewed during WGLC and, 
>>>>>> more importantly, IETF last call. For the moment, my concern is 
>>>>>> that the document correctly describes the IANA allocation.
>>>>>> 
>>>>>> I would like to see here a statement such as : "Given TCP header 
>>>>>> space is scarce, this specification allocates a reserved header 
>>>>>> bit and overloads the two ECN flags in the TCP header ...“.
>>>>> 
>>>>> A bit lengthy but now:
>>>>> 
>>>>> "Given TCP header space is
>>>>>      scarce, it allocates a reserved header bit, that was 
>>>>> previously used for
>>>>>      ECN-Nonce which was recently declared historic, and overloads the
>>>>>      two existing ECN flags in the TCP header. Further, additional
>>>>>      information can be provided in a new TCP option that however 
>>>>> is not used
>>>>>      on the TCP SYN."
>>>>> 
>>>>>>>> * 1.  Introduction
>>>>>>>> 
>>>>>>>>  Recently, proposed mechanisms like Congestion Exposure (ConEx  
>>>>>>>> [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] 
>>>>>>>> need  more accurate ECN feedback information whenever more than 
>>>>>>>> one
>>>>>>> 
>>>>>>> marking
>>>>>>>> 
>>>>>>>>  is received in one RTT.
>>>>>>>> 
>>>>>>>> [ms] At least for RFC 8257 seems to be implementable withoit this.
>>>>>>>> Instead
>>>>>>> 
>>>>>>> of stating a "need", it would IMHO make more sense to discuss 
>>>>>>> the benefits of the suggested mechanism in this document of its 
>>>>>>> own, independent of other proposals. To me, this document should 
>>>>>>> be independent of other documents and specifically other 
>>>>>>> experiments. We have to think about cases where not all 
>>>>>>> experiments are successful. Then independent documents will be 
>>>>>>> more future-proof in future.
>>>>>>> 
>>>>>>> This is a naming collision… The sentence was meant to say that 
>>>>>>> these mechanisms new more accurate ECN feedback than provided 
>>>>>>> today by
>>>>>>> RFC3168 but it was not meant to say that these mechanism have to 
>>>>>>> use the scheme as specified in this document.
>>>>>>> 
>>>>>>> I added the following part sentence:
>>>>>>> 
>>>>>>> „Recently, proposed mechanisms like Congestion Exposure (ConEx 
>>>>>>> [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] 
>>>>>>> need more accurate ECN feedback information than provided by the 
>>>>>>> feedback scheme as specified in [RFC3168] whenever more than one 
>>>>>>> marking is received in one RTT. This document specifies an 
>>>>>>> alternative feedback scheme that provides more accurate 
>>>>>>> information and could be used by these new TCP extensions.“
>>>>>>> 
>>>>>>> Does this help?
>>>>>> 
>>>>>> See my proposal for the abstract. I continue to disagree with the 
>>>>>> term "need" but I think this can be sorted out by another term.
>>>>>> 
>>>>>>>>  If AccECN progresses from experimental to the standards  
>>>>>>>> track, it is intended to be a complete replacement for classic 
>>>>>>>> TCP/  ECN feedback, not a fork in the design of TCP.
>>>>>>>> 
>>>>>>>> [ms] This sentence should be removed, as this is speculation.
>>>>>>> 
>>>>>>> Why? It states an intent… and that’s the intent that we have.
>>>>>>> 
>>>>>>>>  Until the AccECN experiment succeeds, [RFC3168] will remain as 
>>>>>>>> the  standards track specification for adding ECN to TCP.
>>>>>>>> 
>>>>>>>> [ms] This sentence should be removed (or reworded)
>>>>>>> 
>>>>>>> Why? Does it help to add an only here:
>>>>>>> 
>>>>>>> "Until the AccECN experiment succeeds, [RFC3168] will remain as 
>>>>>>> the only standards track specification for adding ECN to TCP.“
>>>>>> 
>>>>>> This wording is better.
>>>>>> 
>>>>>>>>  AccECN feedback overloads flags and fields in the main TCP 
>>>>>>>> header  with new definitions, so both ends have to support the 
>>>>>>>> new wire  protocol before it can be used.
>>>>>>>> 
>>>>>>>> [ms] In my reading this experimental document asks for *new* 
>>>>>>>> allocation
>>>>>>> 
>>>>>>> of a reserved TCP header flag.
>>>>>>> 
>>>>>>> Is this better?
>>>>>>> 
>>>>>>> "AccECN feedback overloads the two existing ECN flags as well as the
>>>>>>>     currently reserved and previously called NS flag in the main 
>>>>>>> TCP header
>>>>>>>     with new definitions, so both ends have to support the new 
>>>>>>> wire protocol
>>>>>>>     before it can be used.“
>>>>>>> 
>>>>>>> I understand that you are not happy with the word „overload“ 
>>>>>>> here but the point of this sentence really is that the flags 
>>>>>>> can/could be used differently and therefore we need a new 
>>>>>>> negotiation before we can use them.
>>>>>> 
>>>>>> For me the following would work: "AccECN feedback overloads the 
>>>>>> two existing ECN flags and allocates the currently reserved and 
>>>>>> previously called NS flag in the main TCP header.
>>>>>> Given the new definitions, both ends have to support the new wire 
>>>>>> protocol before it can be used."
>>>>>> 
>>>>>> I believe the wording has to be crystal clear on the reservation 
>>>>>> of bit 7 when it is discussed the first time in the text. In 
>>>>>> follow-up sections, maybe shorter terms could be used.
>>>>> 
>>>>> Okay, now:
>>>>> 
>>>>> "AccECN feedback overloads the two existing ECN flags and
>>>>>    allocates the currently reserved and previously called NS flag in the
>>>>>    TCP header, to be used as one field indicating the number of 
>>>>> congestion
>>>>>    experienced marked packets. Given the new definitions of these 
>>>>> three bits,
>>>>>    both ends     have to support the new wire protocol before it can be
>>>>> used.
>>>>>    Therefore during the TCP handshake the two ends use these three 
>>>>> bit in
>>>>>    the TCP header to negotiate the most advanced feedback protocol
>>>>>    that they can both support in a backward compatible way to
>>>>>    <xref target="RFC3168"/>."
>>>>>>> 
>>>>>>> If you prefer, we can also remove the NS flag in this list, as 
>>>>>>> ECN Nonce was anyway never deployed.
>>>>>>> 
>>>>>>>>  For that we refer to [RFC3168] or any RFC that  specifies a 
>>>>>>>> different response to TCP ECN feedback, for example:
>>>>>>>>  [RFC8257]; or the ECN experiments referred to in  
>>>>>>>> [I-D.ietf-tsvwg-ecn-experimentation], namely: a TCP-based Low 
>>>>>>>> Latency  Low Loss Scalable (L4S) congestion control 
>>>>>>>> [I-D.ietf-tsvwg-l4s-arch];  ECN-capable TCP control packets 
>>>>>>>> [I-D.ietf-tcpm-generalized-ecn], or  Alternative Backoff with 
>>>>>>>> ECN (ABE)  [I-D.ietf-tcpm-alternativebackoff-ecn].
>>>>>>>> 
>>>>>>>> [ms] At least ABE seems orthogonal. Anyway, I think this 
>>>>>>>> paragraph can just
>>>>>>> 
>>>>>>> be deleted. If other experiments need more accurate feedback, it 
>>>>>>> is up to them to explain how they would use this mechanism. This 
>>>>>>> document should focus on how to signal the feedback, not how to 
>>>>>>> use that.
>>>>>>> 
>>>>>>> Yes, that is what the paragraph says. Isn’t it better to be 
>>>>>>> explicit about this?
>>>>>>> 
>>>>>>>>  It is likely (but not required) that the AccECN protocol will 
>>>>>>>> be  implemented along with the following experimental additions 
>>>>>>>> to the  TCP-ECN protocol: ECN-capable TCP control packets and 
>>>>>>>> retransmissions  [I-D.ietf-tcpm-generalized-ecn], which 
>>>>>>>> includes the ECN-capable SYN/  ACK experiment [RFC5562]; and 
>>>>>>>> testing receiver non-compliance  
>>>>>>>> [I-D.moncaster-tcpm-rcv-cheat].
>>>>>>>> 
>>>>>>>> [ms] I am a big fan of simple, standalone documents. In my 
>>>>>>>> view, the TCPM
>>>>>>> 
>>>>>>> working group should publish draft-ietf-tcpm-accurate-ecn and
>>>>>>> draft-ietf-
>>>>>>> tcpm-generalized-ecn independent documents, which probably 
>>>>>>> implies that draft-ietf-tcpm-generalized-ecn does not use 
>>>>>>> AccECN. If experimentation with ECT in SYN requires a 
>>>>>>> combination, this could be done in a new, third document. Apart 
>>>>>>> from having simpler focused documents, this could significantly 
>>>>>>> help later with moving forward documents to standards track.
>>>>>>> 
>>>>>>> I disagree, however, this is a discussion to have on 
>>>>>>> draft-ietf-tcpm- generalized-ecn. I don’t see a problem in  
>>>>>>> providing a reference here that says „it is likely…“ and nothing 
>>>>>>> more.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> * 1.1.  Document Roadmap
>>>>>>>> 
>>>>>>>> [ms] A macroscopic comment is that this document has a lot of 
>>>>>>>> introduction
>>>>>>> 
>>>>>>> and tutorial text with lot's of redundancy towards other 
>>>>>>> documents. I think the document can be made much easier to read 
>>>>>>> by shorten it. In many cases this is just an editorial change as 
>>>>>>> there is redundancy. As one such example, just remove this 
>>>>>>> section.
>>>>>>> 
>>>>>>> I guess this a matter of taste. As an AD, I’m a big fan of short 
>>>>>>> and concise documents, however, some redundancy can also help 
>>>>>>> understanding, especially if you explain things multiple times 
>>>>>>> but with a different level of detail. I personally would not 
>>>>>>> need the roadmap but I know many people who find these things 
>>>>>>> helpful and to be honest I don’t see how removing this part 
>>>>>>> makes the doc any better. If you don’t want it, don’t read it.
>>>>>>> 
>>>>>>>> 
>>>>>>>> * 1.2.  Goals
>>>>>>>> 
>>>>>>>> [ms] I think this section can also just be removed.
>>>>>>> 
>>>>>>> I have to say I also don’t see the point of removing this part. Given
>>>>>>> we’ve
>>>>>>> done the work on requirements, I think we should also link to this doc
>>>>>>> somewhere.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> * 1.3.  Experiment Goals
>>>>>>>> 
>>>>>>>>  TCP is critical to the robust functioning of the Internet, therefore
>>>>>>>>  any proposed modifications to TCP need to be thoroughly tested. The
>>>>>>>>  present specification describes an experimental protocol that adds
>>>>>>>>  more accurate ECN feedback to the TCP protocol.  The intention is to
>>>>>>>>  specify the protocol sufficiently so that more than one
>>>>>>>>  implementation can be built in order to test its function,
>>>>>>>> robustness
>>>>>>>>  and interoperability (with itself and with previous version of ECN
>>>>>>>>  and TCP).
>>>>>>>> 
>>>>>>>> [ms] I think all what is written in this paragraph is obvious, no?
>>>>>>>> Can't we just
>>>>>>> 
>>>>>>> delete this?
>>>>>>> 
>>>>>>> Sure, however, I don’t think it hurts to spell it out. For me both is
>>>>>>> fine, keep it
>>>>>>> or remove it.
>>>>>>> 
>>>>>>>>  The experimental protocol will be considered successful if it is
>>>>>>>>  deployed and if it satisfies the requirements of [RFC7560] in the
>>>>>>>>  consensus opinion of the IETF tcpm working group.  In short, this
>>>>>>>>  requires that it improves the accuracy and timeliness of TCP's ECN
>>>>>>>>  feedback, as claimed in Section 5, while striking a balance between
>>>>>>>>  the conflicting requirements of resilience, integrity and
>>>>>>>>  minimisation of overhead.  It also requires that it is not unduly
>>>>>>>>  complex, and that it is compatible with prevalent equipment
>>>>>>>>  behaviours in the current Internet (e.g. hardware offloading and
>>>>>>>>  middleboxes), whether or not they comply with standards.
>>>>>>>> 
>>>>>>>>  Testing will mostly focus on fall-back strategies in case of
>>>>>>>>  middlebox interference.  Current recommended strategies are
>>>>>>>> specified
>>>>>>>>  in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
>>>>>>>>  these strategies depends on the actual deployment situation of
>>>>>>>>  middleboxes.  Therefore experimental verification to confirm large-
>>>>>>>>  scale path traversal in the Internet is needed before finalizing
>>>>>>>> this
>>>>>>>>  specification on the Standards Track.
>>>>>>>> 
>>>>>>>> [ms] These two paragraphs must be entirely rewritten. As I have
>>>>>>> 
>>>>>>> mentioned before, I don't think an RFC should speculate about TCPM and
>>>>>>> its
>>>>>>> consensus opinion. I would suggest a wording along the lines of:
>>>>>>>> 
>>>>>>>> <ms>
>>>>>>>>  The experimental protocol will be considered successful if
>>>>>>>>  testing confirms that the proposed mechanism can be deployed at
>>>>>>>> large
>>>>>>> 
>>>>>>> scale.
>>>>>>>> 
>>>>>>>>  Testing will mostly focus on fall-back strategies in case of
>>>>>>>>  middlebox interference.  Current recommended strategies are
>>>>>>>> specified
>>>>>>>>  in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
>>>>>>>>  these strategies depends on the actual deployment situation of
>>>>>>>>  middleboxes.  Therefore experimental verification to confirm large-
>>>>>>>>  scale path traversal in the Internet is needed, e.g., by support in
>>>>>>>>  major TCP stacks.
>>>>>>>> </ms>
>>>>>>>> 
>>>>>>> I don’t understand your point here. I don’t think that the paraphrase
>>>>>>> speculates about the consensus of tcpm, in contrast it say tcpm has to
>>>>>>> decided if the requirements previously specified by tcpm are
>>>>>>> sufficiently
>>>>>>> fulfilled. I don’t see a reason to not mention the requirement draft as
>>>>>>> this
>>>>>>> draft as tcpm consensus and was written for this purpose.
>>>>>> 
>>>>>> My suggested wording uses the expression "can be deployed at large scale"
>>>>>> and I believe this is relevant.
>>>>>> 
>>>>>> The document already describes in Section 5 how the protocol satisfies
>>>>>> the agreed requirements for a more accurate ECN feedback protocol [RFC7560].
>>>>>> So, if the TCPM working group publishes this document with the content of
>>>>>> Section 5, I believe the TCPM working group already has reached consensus
>>>>>> that the protocol meets requirements. In addition, it is possible that new
>>>>>> requirements would be identified in future, e.g., as an outcome of the
>>>>>> experiment, and that would obviously have to be considered by TCPM. In that
>>>>>> case, for the success of the experiment not only RFC 7560 would matter, but
>>>>>> also further requirements. My proposed wording does not have all these
>>>>>> problems.
>>>>>> 
>>>>>> In a nutshell, I continue to believe that this section has to change.
>>>>> 
>>>>> 
>>>>> Okay, used your proposed wording. You have a point about the requirement
>>>>> and I mis-read you proposal earlier as „has to be deployed large-scale“.
>>>>> 
>>>>>>>> * 1.5.  Recap of Existing ECN feedback in IP/TCP
>>>>>>>> 
>>>>>>>> [ms] This section could probably be shortened as well.
>>>>>>>> 
>>>>>>>>  The last bit in byte 13 of the TCP header was defined as the Nonce
>>>>>>>>  Sum (NS) for the ECN Nonce [RFC3540].  RFC 3540 was never deployed
>>>>>>>> so
>>>>>>>>  it is being reclassified as historic, making this TCP flag available
>>>>>>>>  for use by the AccECN experiment instead.
>>>>>>>> 
>>>>>>>> [ms] This wording, as well as Figure 1, needs to take into account the
>>>>>>>> IANA
>>>>>>> 
>>>>>>> status when draft-ietf-tsvwg-ecn-experimentation is published.
>>>>>>> 
>>>>>>> Is does. However, I can explicitly say that is has be re-clssified as
>>>>>>> reserved.
>>>>>>> 
>>>>>>> "RFC 3540 was never deployed so it is being reclassified as historic
>>>>>>> [I-D.ietf-
>>>>>>> tsvwg-ecn-experimentation] and the respective flag has been marked as
>>>>>>> „reserved“ in the IANA TCP Header Flags registry, making this TCP flag
>>>>>>> available for use by the AccECN experiment instead.“
>>>>>>> 
>>>>>>> Better?
>>>>>>> 
>>>>>>>> In my understanding, this experimental document asks for new assignment
>>>>>>> 
>>>>>>> of a reserved TCP header flag.
>>>>>>> 
>>>>>>> As I said I’m not sure if we have fully concluded this discussion yet.
>>>>>>> However,
>>>>>>> what we really would want to is mention somewhere that this experiment
>>>>>>> with this flags is running. I guess there are three options:
>>>>>>> 1) keep it in the registry as reserved and conserve the knowledge in
>>>>>>> tcpm
>>>>>>> that this experiment is running and no other experimental RFC such use
>>>>>>> this
>>>>>>> flags as long as this experiment is running.
>>>>>>> 2) Keep is marked as reserved but add a note about this experiment in
>>>>>>> the
>>>>>>> IANA registry
>>>>>>> 3) Or assign it right away with IESG approval. I guess in this case tcpm
>>>>>>> could
>>>>>>> also consider to change the registration policy to „IETF Review“.
>>>>>> 
>>>>>> The current registration policy for the TCP header flags is "standards
>>>>>> action". I understand that the IESG could approve exceptions. But given the
>>>>>> policy, I believe the document has to be very precise on the request
>>>>>> regarding bit 7.
>>>>> 
>>>>> Okay, it now says:
>>>>> 
>>>>> "[TO BE REMOVED: IANA is requested to update the existing entry in the
>>>>> Transmission Control Protocol (TCP) Header Flags registration
>>>>> (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.iana.org%2Fassignments%2Ftcp-header-flags%2Ftcp-header-flags.xhtml%23tcp-header-flags-1&amp;data=02%7C01%7Cpravb%40microsoft.com%7C2e4980f91d89410fa58708d5e8111fbb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636670082610453620&amp;sdata=KXOFPQ%2FfWW0dekv5%2BdYM8stbLl80hLtlJVnM9s5G9lQ%3D&amp;reserved=0)
>>>>> for Bit 7 to "AE (Accurate ECN), previously used by Historic as NS (Nonce
>>>>> Sum) [RFC3540, RFC8311]" and change the reference to this RFC-to-be instead
>>>>> of RFC8311.]“
>>>>> 
>>>>> I guess we could also ask IANA to add an additional comment column instead
>>>>> (but not sure if we then have to update RFC3168, which I think we really
>>>>> don’t want. Should be fine now.
>>>>> 
>>>>>>>> * 2.  AccECN Protocol Overview and Rationale
>>>>>>>> 
>>>>>>>>  o  an essential part that re-uses ECN TCP header bits to feed back
>>>>>>>>     the number of arriving CE marked packets.  This provides more
>>>>>>>>     accuracy than classic ECN feedback, but limited resilience
>>>>>>>> against
>>>>>>>>     ACK loss;
>>>>>>>> 
>>>>>>>> [ms] The word "re-use" is IMHO not correct.
>>>>>>> 
>>>>>>> I think this is nit picking. Using a different phrasing here makes the
>>>>>>> sentence
>>>>>>> unnecessary complicated. We don’t try to some how get a round the fact
>>>>>>> that we need to handle the flag registration correctly. However, here
>>>>>>> the
>>>>>>> point really is to explain how the protocol word. The main point of
>>>>>>> using the
>>>>>>> work „re-use“ here is really that we say that these flags are or have
>>>>>>> been
>>>>>>> used different by other TCP extension (and we therefore need a proper
>>>>>>> negotiation scheme).
>>>>>> 
>>>>>> If the allocation of a reserved flag is correctly explained in the
>>>>>> abstract and introduction, I think these sentences can use a bit relaxed
>>>>>> terminology.
>>>>>> 
>>>>>>>>  The two part design was necessary, given limitations on the space
>>>>>>>>  available for TCP options and given the possibility that certain
>>>>>>>>  incorrectly designed middleboxes prevent TCP using any new options
>> 

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40microsoft.com%7C2e4980f91d89410fa58708d5e8111fbb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636670082610453620&amp;sdata=BzLdW3YTISACG5pakJhqTGE%2BWg5mRX3T514Z7woG2oI%3D&amp;reserved=0