Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn

["Scharf, Michael (Nokia - DE/Stuttgart)" <[e-mail]>]

Hi Bob,

Sorry for the delayed response. Honestly speaking, this material is hard to parse and I don’t fully understand what the proposed change of text in a version -07 would be.

And I may get somehow lost in the details here, but the rational for the change as compared to -03 is still not clear to me.

In any case, to me as individual, future extensibility continues to be more relevant for the TCP header than “nice-to-have” features in an experiment, and I would prefer to have wording on the particular design choices in the document (e.g., in an appendix) instead of elsewhere.

Thanks

Michael





From: Bob Briscoe [mailto:[e-mail]]
Sent: Saturday, April 07, 2018 1:16 AM
To: Scharf, Michael (Nokia - DE/Stuttgart) <[e-mail]>
Cc: Richard Scheffenegger <[e-mail]>; Mirja Kühlewind <[e-mail]>; [e-mail]; [e-mail]
Subject: Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn

Michael,

I realize now that I misunderstood your question during the TCPM session in London. (Sorry for the delay - I tuned out for a while).

As well as the 2 codepoints that Richard has highlighted, the draft is careful to require that implementations of the AccECN experiment will silently accept future use of other unexpected codepoints ('forward compatibility'). For instance, at the end of S.3.2,3<https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-06#section-3.2.3>:

   Note that the Data Sender MUST NOT test whether the arriving counter

   in the initial ACE field has been initialized to a specific valid

   value - the above check solely tests whether the ACE fields have been

   incorrectly zeroed.  This allows hosts to use different initial

   values as an additional signalling channel in future.
We have only used codepoints 6&7 here. Assuming we avoid zero, because it might be due to bleaching, this leaves values 1-5 unused.

The analysis I did of the remaining combinations of codepoints when taken across the whole handshake is here:
    http://bobbriscoe.net/projects/latency/ecn-modes.pdf
Page 1 covers the TCP ECN flags.

The 1st row marked A-CU (short for Accurate ECN Currently Unassigned) shows the 5 values that the above text keeps for future use.
The 2nd two rows marked A-CU are now used by AccECN (since draft-04) to deal with the bleaching we discovered during testing.
All the combinations tagged with 'N' (for nonce) are likely to be the most useful if needed for any additional negotiation space.

The other one Richard tagged (labelled 'broken') is still polluted by a non-negligible number of bugged servers that reflect the 6 most significant TCP flags in the SYN back in the SYN/ACK. From a 2014 measurement study that Richard and Mirja co-authored, there were still about 0.3% of the Alexa 1M exhibiting the symptoms of this bug. I'm in the process of checking our more recent data, where we specifically checked if 111 used on the AccECN SYN was reflected.

Taken as a whole, I admit this doesn't leave much flexibility for future variation, but it's the best we could do with the available space constraints.
Does this sufficiently address your concern?


Cheers



Bob

PS. Similarly, although we have not allowed any space in the AccECN TCP Option for flag bits, we have allowed for different initial values of the counters to be used to signal new versions of the protocol in future. See the end of S.3.2.7.4<https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-06#section-3.2.7.4>:

   Note that the Data Sender MUST NOT test whether the arriving byte

   counters in the initial AccECN Option have been initialized to

   specific valid values - the above checks solely test whether these

   fields have been incorrectly zeroed.  This allows hosts to use

   different initial values as an additional signalling channel in

   future.

On 19/03/18 16:04, Scharf, Michael (Nokia - DE/Stuttgart) wrote:

For what it is worth, this is exactly the sort of discussion I believe we need to have.



I would be much more comfortable if the document did foresee some way negotiate an "enhanced" version of Accurate ECN in a later phase - most notably a PS version of the protocol. If that would cost some functionality right now, as we may have to reserve some codepoints in the EXP version, that would be perfectly fine for me, as long as we could add that functionality later in a PS version (under our current understanding).



I have not thought about the details. But we have only three reserved TCP header bits left after publication of this document and I am worried about that.



Michael





-----Original Message-----

From: Richard Scheffenegger [mailto:[e-mail]]

Sent: Monday, March 19, 2018 4:27 PM

To: Scharf, Michael (Nokia - DE/Stuttgart) <[e-mail]><mailto:[e-mail]>;

Mirja Kühlewind <[e-mail]><mailto:[e-mail]>

Cc: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>

Subject: Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn



Hi Michael,



regarding your comment about the lack of future extensibility of the ECN /

AccECN header handshake without further bits...



I wanted to note, that the logic table we have in the accecn draft currently

has two entries, that are reserved because of observed, but legacy, stacks

misbehaving when reflecting these TCP header bits.



So there is a possibility to eventually use those entries - once the broken

legacy stacks have been confirmeed to have left the internet - as some

additional negotiation capability.



Also, the feedback of the byte counters vs. CE packet counter are split into

using a tcp option and the tcp header bits.

An extention of AccECN could take the form of using a differen TCP option

together with the header counter negotiation...



Earlier work by us determined that packing more information into the tcp

header bits alone (as an example) would likely need an additional bit (that

can also be used in a negotiation scheme)...





   +--------+--------+------------+-------------+----------------------+

   | A      | B      |  SYN A->B  |   SYN/ACK   | Feedback Mode        |

   |        |        |            |     B->A    |                      |

   +--------+--------+------------+-------------+----------------------+

   |        |        | AE CWR ECE |  AE CWR ECE |                      |

   | AccECN | AccECN | 1   1   1  |  0   1   0  | AccECN (Not-ECT on   |

   |        |        |            |             | SYN)                 |

   | AccECN | AccECN | 1   1   1  |  0   1   1  | AccECN (ECT1 on SYN) |

   | AccECN | AccECN | 1   1   1  |  1   0   0  | AccECN (ECT0 on SYN) |

   | AccECN | AccECN | 1   1   1  |  1   1   0  | AccECN (CE on SYN)   |

   |        |        |            |             |                      |

#  | AccECN | Nonce  | 1   1   1  |  1   0   1  | classic ECN          |

   | AccECN | ECN    | 1   1   1  |  0   0   1  | classic ECN          |

   | AccECN | No ECN | 1   1   1  |  0   0   0  | Not ECN              |

   |        |        |            |             |                      |

   | Nonce  | AccECN | 0   1   1  |  0   0   1  | classic ECN          |

   | ECN    | AccECN | 0   1   1  |  0   0   1  | classic ECN          |

   | No ECN | AccECN | 0   0   0  |  0   0   0  | Not ECN              |

   |        |        |            |             |                      |

#  | AccECN | Broken | 1   1   1  |  1   1   1  | Not ECN              |

   +--------+--------+------------+-------------+----------------------+

Best regards,

   Richard





----- Original Message -----

From: "Scharf, Michael (Nokia - DE/Stuttgart)" <[e-mail]><mailto:[e-mail]>

To: "Mirja Kühlewind" <[e-mail]><mailto:[e-mail]>

Cc: <[e-mail]><mailto:[e-mail]>; <[e-mail]><mailto:[e-mail]>

Sent: Monday, March 12, 2018 1:59 AM

Subject: Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn





Hi Mirja,



Thanks a lot for the explanation. I won't follow-up on some of the

editorial suggestions.



Yet, I continue to believe that some formal wording in the document needs

to change, as explained below.



Thanks



Michael (with no hat on)





-----Original Message-----

From: Mirja Kühlewind [mailto:[e-mail]]

Sent: Monday, March 05, 2018 1:54 PM

To: Scharf, Michael (Nokia - DE/Stuttgart) <[e-mail]><mailto:[e-mail]>

Cc: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>

Subject: Re: Comments on draft-ietf-tcpm-accurate-ecn



Hi Micheal,



thanks for your feedback and sorry for my late reply.



Please see inline.



Am 03.12.2017 um 20:17 schrieb Scharf, Michael (Nokia - DE/Stuttgart)

<[e-mail]><mailto:[e-mail]>:



Hi all,



I have read draft-ietf-tcpm-accurate-ecn-05 (without the appendix). I

believe this document needs further work before moving forward.



Please find below my comments marked as [ms]. I have read the

document independent of the review from Gorry. I apologize if there is

duplication.



Thanks



Michael (with no hat on)





******************************



* Abstract:



   Recently, new TCP mechanisms like Congestion Exposure (ConEx) or

Data

Center TCP

   (DCTCP) need more accurate ECN feedback information whenever

more

   than one marking is received in one RTT.



[ms] I don't think this statement is fully backed by RFC 8257. I

suggest to

remove this, or replace it by a more generic statement that more accurate

information can be useful for several TCP extensions.



I disagree. Both ConEx and DCTCP need more accurate information. They

do

not need the mechanism that is specified in this draft, however, this is

not

what the sentences is saying.



In my understanding (as a non-native speaker), the use of the word "need"

is not correct here. DCTCP as specified in RFC 8257 can be implemented

without any such mechanism.



What would work for me is something of the form "... Data Center TCP

cannot get precise ECN feedback whenever more than one marking is

received

in one RTT".



   This document specifies an

   experimental scheme to provide more than one feedback signal per

RTT

   in the TCP header.  Given TCP header space is scarce, it overloads

   the three existing ECN-related flags in the TCP header and provides

   additional information in a new TCP option.



[ms] This statement needs to be rewritten to correctly reflect what is

requested from IANA. My understanding is that this experimental

document

asks for allocation of a reserved TCP header flag. This needs to be

called out

prominently, IMHO. In addition, since this is not a standard, the

suggested

experimentation with the main TCP header must IMHO be explicitly

mentioned. I also suggest to have later in a document a section that

explicitly

explains why it is appropriate to modify the main TCP header in an

experiment.



I don’t know if any requirement that IANA assignment need to be called

out

in the abstract but we can do that. However, I believe the question if

this

document should or should not assign the bit is still not completely

solved, or

is it?



I believe this question will have to be reviewed during WGLC and, more

importantly, IETF last call. For the moment, my concern is that the

document correctly describes the IANA allocation.



I would like to see here a statement such as : "Given TCP header space is

scarce, this specification allocates a reserved header bit and overloads

the two ECN flags in the TCP header ...".



* 1.  Introduction



   Recently, proposed mechanisms like Congestion Exposure (ConEx

   [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] need

   more accurate ECN feedback information whenever more than one

marking

   is received in one RTT.



[ms] At least for RFC 8257 seems to be implementable withoit this.

Instead

of stating a "need", it would IMHO make more sense to discuss the

benefits

of the suggested mechanism in this document of its own, independent of

other proposals. To me, this document should be independent of other

documents and specifically other experiments. We have to think about

cases

where not all experiments are successful. Then independent documents

will

be more future-proof in future.



This is a naming collision… The sentence was meant to say that these

mechanisms new more accurate ECN feedback than provided today by

RFC3168 but it was not meant to say that these mechanism have to use

the

scheme as specified in this document.



I added the following part sentence:



„Recently, proposed mechanisms like Congestion Exposure (ConEx

[RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] need more

accurate ECN feedback information than provided by the feedback

scheme

as specified in [RFC3168] whenever more than one marking is received in

one

RTT. This document specifies an alternative feedback scheme that

provides

more accurate information and could be used by these new TCP

extensions.“



Does this help?



See my proposal for the abstract. I continue to disagree with the term

"need" but I think this can be sorted out by another term.



   If AccECN progresses from experimental to the standards

   track, it is intended to be a complete replacement for classic TCP/

   ECN feedback, not a fork in the design of TCP.



[ms] This sentence should be removed, as this is speculation.



Why? It states an intent… and that’s the intent that we have.





   Until the AccECN experiment succeeds, [RFC3168] will remain as the

   standards track specification for adding ECN to TCP.



[ms] This sentence should be removed (or reworded)



Why? Does it help to add an only here:



"Until the AccECN experiment succeeds, [RFC3168] will remain as the only

standards track specification for adding ECN to TCP.“



This wording is better.



   AccECN feedback overloads flags and fields in the main TCP header

   with new definitions, so both ends have to support the new wire

   protocol before it can be used.



[ms] In my reading this experimental document asks for *new*

allocation

of a reserved TCP header flag.



Is this better?



"AccECN feedback overloads the two existing ECN flags as well as the

      currently reserved and previously called NS flag in the main TCP

header

      with new definitions, so both ends have to support the new wire

protocol

      before it can be used.“



I understand that you are not happy with the word „overload“ here but

the

point of this sentence really is that the flags can/could be used

differently

and therefore we need a new negotiation before we can use them.



For me the following would work: "AccECN feedback overloads the two

existing ECN flags and

allocates the currently reserved and previously called NS flag in the main

TCP header.

Given the new definitions, both ends have to support the new wire

protocol

before it can be used."



I believe the wording has to be crystal clear on the reservation of bit 7

when it is discussed the first time in the text. In follow-up sections,

maybe shorter terms could be used.



If you prefer, we can also remove the NS flag in this list, as ECN Nonce

was

anyway never deployed.





   For that we refer to [RFC3168] or any RFC that

   specifies a different response to TCP ECN feedback, for example:

   [RFC8257]; or the ECN experiments referred to in

   [I-D.ietf-tsvwg-ecn-experimentation], namely: a TCP-based Low

Latency

   Low Loss Scalable (L4S) congestion control

[I-D.ietf-tsvwg-l4s-arch];

   ECN-capable TCP control packets [I-D.ietf-tcpm-generalized-ecn], or

   Alternative Backoff with ECN (ABE)

   [I-D.ietf-tcpm-alternativebackoff-ecn].



[ms] At least ABE seems orthogonal. Anyway, I think this paragraph can

just

be deleted. If other experiments need more accurate feedback, it is up to

them to explain how they would use this mechanism. This document

should

focus on how to signal the feedback, not how to use that.



Yes, that is what the paragraph says. Isn’t it better to be explicit

about this?





   It is likely (but not required) that the AccECN protocol will be

   implemented along with the following experimental additions to the

   TCP-ECN protocol: ECN-capable TCP control packets and

retransmissions

   [I-D.ietf-tcpm-generalized-ecn], which includes the ECN-capable SYN/

   ACK experiment [RFC5562]; and testing receiver non-compliance

   [I-D.moncaster-tcpm-rcv-cheat].



[ms] I am a big fan of simple, standalone documents. In my view, the

TCPM

working group should publish draft-ietf-tcpm-accurate-ecn and draft-ietf-

tcpm-generalized-ecn independent documents, which probably implies

that

draft-ietf-tcpm-generalized-ecn does not use AccECN. If experimentation

with ECT in SYN requires a combination, this could be done in a new,

third

document. Apart from having simpler focused documents, this could

significantly help later with moving forward documents to standards

track.



I disagree, however, this is a discussion to have on draft-ietf-tcpm-

generalized-ecn. I don’t see a problem in  providing a reference here

that

says „it is likely…“ and nothing more.







* 1.1.  Document Roadmap



[ms] A macroscopic comment is that this document has a lot of

introduction

and tutorial text with lot's of redundancy towards other documents. I

think

the document can be made much easier to read by shorten it. In many

cases

this is just an editorial change as there is redundancy. As one such

example,

just remove this section.



I guess this a matter of taste. As an AD, I’m a big fan of short and

concise

documents, however, some redundancy can also help understanding,

especially if you explain things multiple times but with a different

level of

detail. I personally would not need the roadmap but I know many people

who find these things helpful and to be honest I don’t see how removing

this

part makes the doc any better. If you don’t want it, don’t read it.







* 1.2.  Goals



[ms] I think this section can also just be removed.



I have to say I also don’t see the point of removing this part. Given we’ve

done the work on requirements, I think we should also link to this doc

somewhere.





* 1.3.  Experiment Goals



   TCP is critical to the robust functioning of the Internet, therefore

   any proposed modifications to TCP need to be thoroughly tested. The

   present specification describes an experimental protocol that adds

   more accurate ECN feedback to the TCP protocol.  The intention is to

   specify the protocol sufficiently so that more than one

   implementation can be built in order to test its function,

robustness

   and interoperability (with itself and with previous version of ECN

   and TCP).



[ms] I think all what is written in this paragraph is obvious, no?

Can't we just

delete this?



Sure, however, I don’t think it hurts to spell it out. For me both is

fine, keep it

or remove it.





   The experimental protocol will be considered successful if it is

   deployed and if it satisfies the requirements of [RFC7560] in the

   consensus opinion of the IETF tcpm working group.  In short, this

   requires that it improves the accuracy and timeliness of TCP's ECN

   feedback, as claimed in Section 5, while striking a balance between

   the conflicting requirements of resilience, integrity and

   minimisation of overhead.  It also requires that it is not unduly

   complex, and that it is compatible with prevalent equipment

   behaviours in the current Internet (e.g. hardware offloading and

   middleboxes), whether or not they comply with standards.



   Testing will mostly focus on fall-back strategies in case of

   middlebox interference.  Current recommended strategies are

specified

   in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of

   these strategies depends on the actual deployment situation of

   middleboxes.  Therefore experimental verification to confirm large-

   scale path traversal in the Internet is needed before finalizing

this

   specification on the Standards Track.



[ms] These two paragraphs must be entirely rewritten. As I have

mentioned before, I don't think an RFC should speculate about TCPM and

its

consensus opinion. I would suggest a wording along the lines of:



<ms>

   The experimental protocol will be considered successful if

   testing confirms that the proposed mechanism can be deployed at

large

scale.

   Testing will mostly focus on fall-back strategies in case of

   middlebox interference.  Current recommended strategies are

specified

   in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of

   these strategies depends on the actual deployment situation of

   middleboxes.  Therefore experimental verification to confirm large-

   scale path traversal in the Internet is needed, e.g., by support in

   major TCP stacks.

</ms>



I don’t understand your point here. I don’t think that the paraphrase

speculates about the consensus of tcpm, in contrast it say tcpm has to

decided if the requirements previously specified by tcpm are sufficiently

fulfilled. I don’t see a reason to not mention the requirement draft as

this

draft as tcpm consensus and was written for this purpose.



My suggested wording uses the expression "can be deployed at large

scale"

and I believe this is relevant.



The document already describes in Section 5 how the protocol satisfies the

agreed requirements for a more accurate ECN feedback protocol

[RFC7560].

So, if the TCPM working group publishes this document with the content of

Section 5, I believe the TCPM working group already has reached

consensus

that the protocol meets requirements. In addition, it is possible that new

requirements would be identified in future, e.g., as an outcome of the

experiment, and that would obviously have to be considered by TCPM. In

that case, for the success of the experiment not only RFC 7560 would

matter, but also further requirements. My proposed wording does not

have

all these problems.



In a nutshell, I continue to believe that this section has to change.



* 1.5.  Recap of Existing ECN feedback in IP/TCP



[ms] This section could probably be shortened as well.



   The last bit in byte 13 of the TCP header was defined as the Nonce

   Sum (NS) for the ECN Nonce [RFC3540].  RFC 3540 was never deployed

so

   it is being reclassified as historic, making this TCP flag available

   for use by the AccECN experiment instead.



[ms] This wording, as well as Figure 1, needs to take into account the

IANA

status when draft-ietf-tsvwg-ecn-experimentation is published.



Is does. However, I can explicitly say that is has be re-clssified as

reserved.



"RFC 3540 was never deployed so it is being reclassified as historic

[I-D.ietf-

tsvwg-ecn-experimentation] and the respective flag has been marked as

„reserved“ in the IANA TCP Header Flags registry, making this TCP flag

available for use by the AccECN experiment instead.“



Better?



In my understanding, this experimental document asks for new

assignment

of a reserved TCP header flag.



As I said I’m not sure if we have fully concluded this discussion yet.

However,

what we really would want to is mention somewhere that this

experiment

with this flags is running. I guess there are three options:

1) keep it in the registry as reserved and conserve the knowledge in tcpm

that this experiment is running and no other experimental RFC such use

this

flags as long as this experiment is running.

2) Keep is marked as reserved but add a note about this experiment in

the

IANA registry

3) Or assign it right away with IESG approval. I guess in this case tcpm

could

also consider to change the registration policy to „IETF Review“.



The current registration policy for the TCP header flags is "standards

action". I understand that the IESG could approve exceptions. But given

the policy, I believe the document has to be very precise on the request

regarding bit 7.



* 2.  AccECN Protocol Overview and Rationale



   o  an essential part that re-uses ECN TCP header bits to feed back

      the number of arriving CE marked packets.  This provides more

      accuracy than classic ECN feedback, but limited resilience

against

      ACK loss;



[ms] The word "re-use" is IMHO not correct.



I think this is nit picking. Using a different phrasing here makes the

sentence

unnecessary complicated. We don’t try to some how get a round the fact

that we need to handle the flag registration correctly. However, here the

point really is to explain how the protocol word. The main point of using

the

work „re-use“ here is really that we say that these flags are or have

been

used different by other TCP extension (and we therefore need a proper

negotiation scheme).



If the allocation of a reserved flag is correctly explained in the

abstract and introduction, I think these sentences can use a bit relaxed

terminology.



   The two part design was necessary, given limitations on the space

   available for TCP options and given the possibility that certain

   incorrectly designed middleboxes prevent TCP using any new options.



[ms] IMHO it would make sense to more explicitly mention the

downsides

of only specifying an option and not allocating a TCP header flag, in

this

experimental document.



We need to use the flags (all three of them) for the negotiation.



I think that an explanation why negotiation by a TCP option would not

solve all use cases (or requirements) would help here.



The obvious  alternative would be to postpone the header flag

allocation to

a follow-up standards track document and just keep it reserved.



We can still do that. We should discuss and make a final decision.





   The essential part overloads the previous definition of the three

   flags in the TCP header that had been assigned for use by ECN.  This

   design choice deliberately replaces the classic ECN feedback

   protocol, rather than leaving classic ECN feedback intact and adding

   more accurate feedback separately because:



[ms] Similar like previous comments, in my reading there are only _two_

ECN header flags.



I think there are three flags that "had been assigned for use by ECN“ as

ECN

Nonce is also an ECN mechanism. The fact that one of the flags is now

marked as reserved instead, it not that important for me here.



And, in addition, I think care is needed with wording such "replaces

the

classic ECN feedback". I don't think this experiment replaces the ECN

standards. That would be up to a follow-up PS.



This sentence is not meant to say that RFC3168 is replaced. Actually we

don’t.

You can still use RFC3168 even if AccECN is implemented and deploy

(however, we do intent that AccECN will be used as the default scheme in

future and RFC3168 is hopefully simply not needed anymore at some

point,

even though you probably still need to have it implemented as the

negotiation specified in this draft covers that as well, anyway...). The

sentence says that if AccECN is negotiation, the header flags as used by

RFC3168 and previously ECN Nonce are used differently (aka re-used).

That’s

all.





2.1.  Capability Negotiation



   AccECN is a change to the wire protocol of the main TCP header,

   therefore it can only be used if both endpoints have been upgraded

to

   understand it.  The TCP client signals support for AccECN on the

   initial SYN of a connection and the TCP server signals whether it

   supports AccECN on the SYN/ACK.  The TCP flags on the SYN that the

   client uses to signal AccECN support have been carefully chosen so

   that a TCP server will interpret them as a request to support the

   most recent variant of ECN feedback that it supports.  Then the

   client falls back to the same variant of ECN feedback.



[ms] As this is an experimental specification, I would really like to

see a

discussion how a future standards track version of more accurate ECN

could

be negotiated.



As described in this draft. There will be no different. AccECN IS and

will

always be backward compatible with RFC3168.



That is not the problem I think about. I wonder about a PS version of

accurate ECN feedback that would possibly include changes as compared to

this experiment, e.g., because the experiment may have some lessons

learnt. What options would we have to negotiate the PS version, and how

could a stack implementing the PS version figure out whether the remote

end uses the experimental or the PS version of the protocol?



The reason why I ask is because I don't see any easy solution to this but

I may miss something. Maybe this would not be a concern if there were

some

codepoints left.



How could both endpoints detect whether the other one implements

the

future standards track version?



If the initiator implements AccECN it will request it’s use. If the

receiver also

implement it, it will/can negotiate it, if not it will look like an

RFC3168 request

for the receiver (as the NS flags will be ignored in the SYN) and it will

negotiate RFC3168 ECN feedback if implemented. There is no additional

detection needed.



My concern is the migration strategy for a future version, given that we

only experiment right now. For instance, if the initiator implements the

experimental version but the recipient implements the PS version of

AccECN, how would that work? Under the assumption that there are

changes,

would there be a way to know? Maybe the answer to this question is no,

given the small number of bits we have. But not having any room for

extensions is not necessarily good protocol design.



For instance, would the only safe variant be that we allocate yet

another

reserved TCP header flag in a proposed standard to negotiate the

standards

track version, thus investing another reserved bit in the TCP header?



No, that’s exactly what we use the NS flags for in the handshake.



If the PS version of AccECN was different to the experimental version of

AccECN, and if there was a deployed base of both, I still believe we could

end up in a situation in which we had to allocate yet another TCP header

flag to distinguish the different versions of AccECN. The NS flag would

not work for that if it is used by the experimental version. The risk of

having to spend further TCP header flags somehow concerns me. Of

course,

that problem would only matter if the AccECN experiment succeeds

somehow,

but lessons learnt would require protocol changes, which is speculation.



I may be wrong, but to me it is too early to speculate how the PS

version

would look like, and whether it would have to be different to the

experimental version, due to lessons learnt.



Of course you can always be wrong. However, the handshake negotiation

is

not the part we need experimentation for. That part is straight forward

and

works. If we really happen to detect a problem in that part, we would

need

to end the experiment declare failure and start over new.



My concern is ending the experiment when the experiment got (partly)

deployed in the Internet. In that case neither a new RFC nor a change of

the IANA registry will solve the migration issue.



I believe in the IETF we typically design protocols that allow future

extension, and it is not exactly clear to be how AccECN could be extended

later.



This is an TCP extension. If we want future extension we use the usually

TCP

mechanism (by defining a new TCP option I guess).



The root cause of my concern is that this proposal does *not* use the

usual way to experiment with TCP by options. It experiments with a header

flag, including in the SYN, and it seems to consume all codepoints. So, I

see the risk of a protocol design "not ready for future improvements".



I cannot easily propose text. Maybe "lack of extensibility" is just one of

the short-comings of the protocol design that cannot be avoided but that

short-coming would have to be noted.



   An AccECN TCP client does not send the new AccECN Option on the

SYN

   as SYN option space is limited and successful negotiation using the

   flags in the main header is taken as sufficient evidence that both

   ends also support the AccECN Option.  The TCP server sends the

AccECN

   Option on the SYN/ACK and the client sends it on the first ACK to

   test whether the network path forwards the option correctly.



[ms] For what it is worth, I would personally be quite fine with

allowing (or

even mandating) an option in the SYN in the experimental version of this

protocol. For instance, saving the SYN option space would then an

excellent

reason for moving towards the PS specification. I am also fine with being

in

the rough part of the consensus here.



The point is that we really don’t need the option in the SYN as we don’t

use it

for negotiation purposes as we use the header bits instead. So why

should

we waste the space?



For instance, mandating the option in the SYN would be away for the

receiver to distinguish the experiment from a follow-up PS version of the

spec, as the PS version may not mandate the option, to save header space.



Maybe that proposal does not make any sense, and it may only have

downsides. But the document already speculates about a PS-follow-up. So

it

seems a valid question to ask if the EXP and the PS version of the spec

have to be identical. This all comes down to the SYN negotiation.



* 2.3.  Delayed ACKs and Resilience Against ACK Loss



   If the AccECN Option is not available, e.g. it is being stripped by

a

   middlebox, the AccECN protocol will only feed back information on CE

   markings (using the ACE field).  Although not ideal, this will be

   sufficient, because it is envisaged that neither ECT(0) nor ECT(1)

   will ever indicate more severe congestion than CE, even though

future

   uses for ECT(0) or ECT(1) are still unclear

   [I-D.ietf-tsvwg-ecn-experimentation].



[ms] This needs to be reworded



Why?









* 2.4.  Feedback Metrics



   The CE packet counter in the ACE field and the CE byte counter in

the

   AccECN Option both provide feedback on received CE-marks.  The CE

   packet counter includes control packets that do not have payload

   data, while the CE byte counter solely includes marked payload

bytes.

   If both are present, the byte counter in the option will provide the

   more accurate information needed for modern congestion control and

   policing schemes, such as DCTCP or ConEx.



[ms] I suggest to write in the last sentence only "... the option will

provide

the more accurate information needed for congestion control". In

general,

I

would prefer to have references to other mechanisms at only few (ideally

a

*single*) places in the document, instead of mixing them together.



Sorry, I don’t see your point here. ConEx has been mentioned previously,

so

why not also mention it here.



As written earlier, in my understanding DCTCP does not "need" this. I

would suggest to have one place where to define exactly how this

mechanism

can be used by other TCP extensions. Having said this, in this specific

paragraph I am less concerned about that than elsewhere.



   Feedback in bytes is recommended in order to protect against the

   receiver using attacks similar to 'ACK-Division' to artificially

   inflate the congestion window, which is why [RFC5681] now

recommends

   that TCP counts acknowledged bytes not packets.



[ms] At least the last part and the reference to RFC 5681 is IMHO not

needed here.



Why? RFC5681 explains/refers the ACK division attack, so I think it is a

very

good reference to have here.







* 2.5.  Generic (Dumb) Reflector



   The ACE field provides information about CE markings on both data

and

   control packets.  According to [RFC3168] the Data Sender is meant to

   set control packets to Not-ECT.  However, mechanisms in certain

   private networks (e.g. data centres) set control packets to be ECN

   capable because they are precisely the packets that performance

   depends on most.



   For this reason, AccECN is designed to be a generic reflector of

   whatever ECN markings it sees, whether or not they are compliant

with

   a current standard.  Then as standards evolve, Data Senders can

   upgrade unilaterally without any need for receivers to upgrade too.

   It is also useful to be able to rely on generic reflection behaviour

   when senders need to test for unexpected interference with markings

   (for instance [I-D.kuehlewind-tcpm-ecn-fallback] and

   [I-D.moncaster-tcpm-rcv-cheat]).



   The initial SYN is the most critical control packet, so AccECN

   provides feedback on whether it is CE marked.  Although RFC 3168

   prohibits an ECN-capable SYN, providing feedback of CE marking on

the

   SYN supports future scenarios in which SYNs might be ECN-enabled

   (without prejudging whether they ought to be).  For instance,

   [I-D.ietf-tsvwg-ecn-experimentation] updates this aspect of RFC 3168

   to allow experimentation with ECN-capable TCP control packets.



[ms] To me, the only thing that matters in this document that AccECN

can

provide feedback on whether the SYN is CE marked. The discussion on

how

to experiment with ECT e.g. in SYNs IMHO does not belong into this

document. So it seems sufficient here to note that one of the benefits of

AccECN is that CE marks in SYNs can be fed back.



I disagree. Explicitly saying that AccECN is only an feedback scheme and

DOES

NOT define how the information is used is VERY important because

people

come back to me over and over again and mix these things up.





* 3.1.1.  Negotiation during the TCP handshake



   Given the ECN Nonce [RFC3540] is being reclassified as historic, the

   present specification renames the TCP flag at bit 7 of the TCP

header

   flags from NS (Nonce Sum) to AE (Accurate ECN) (see IANA

   Considerations in Section 6).



[ms] As mentioned before, this needs to be rewritten to ask for new

IANA

allocation of bit 7 in the TCP header flags.



I really don’t understand this comment. That is what the IANA section

does

as referred here correctly.



In my reading of

https://www.iana.org/assignments/tcp-header-flags/tcp-header-

flags.xhtml,

this flag currently has no name, i.e., it does not have the name NS. So

this statement on "renaming" is formally incorrect IMHO.



I'd suggest something like: "This specification assigns the name AE

(Accurate ECN) to the TCP flag at bit 7; this flag has previously been

known as NS (Nonce Sum) ...".



Again, yes, we can discuss if this document should do this, but if

published as

it is, section 6 says everything that is needs to say. (I does not put

any

request on IANA, instead it is written as it would show up

post-publication,

implicitly providing a request to IANA at approval time. That is fine.)





   Table 2: ECN capability negotiation between Client (A) and Server

(B)



[ms] As far as I can see, in -05 this table allocates all existing

codepoints,

while -03 had two currently unused codepoints. Not having any

codepoints

left seems to me not really future proof, e.g., regarding future proposed

standards in this space (and I personally believe that TCP header flags

must

be allocated in a PS). And I don't fully see a need of feeding back ECT0

and

specifically ECT1 in the TCP header flags as part of the experiment. Do

we

know for sure that this is the only possible use case of these two

unallocated

header bits? And why can't e.g. this be done in a TCP option instead? Or

do I

miss something?



The point is really, if we don’t assign them now and start deployment we

effetely we not be able to every assign them again because don’t have a

different negotiation mechanisms. Realizing this, it is just the right

think to

define the space completely that is negotiated as use for AccECN in the

handshake.



I disagree that consuming all codepoints and not having room for future

extensions is the "right thing" to do. It is in fact the wrong thing. But

possibly there is no alternative with the few bits, so maybe we end up

doing it.



Yet, at minimum, using all codepoints needs to be reasoned in the

document. Specifically since in -03 a different protocol design seemed

possible, which looked to me more future-proof.



* 3.1.2.  Retransmission of the SYN



   However,

   current measurements imply that a drop is less likely to be due to

   middlebox interference than other intermittent causes of loss, e.g.

   congestion, wireless interference, etc.



[ms] Such wording IMHO doesn't belong into normative text. This may

actually also apply to other heuristics discussed in this section, which

are not

really important for interoperability.



I don’t really understand your point. This sentence is solely meant to

reason

the design decision to not say that a sender SHOULD attempt to re-

negotiation after a loss.



One could split the reasoning from the normative design. The normative

specification may still be relevant in 20 years from now. Middleboxes will

almost likely be different in 20 years. Having said this, this is more of

an editorial remark.



3.2.7.  Path Traversal of the AccECN Option



3.2.7.1.  Testing the AccECN Option during the Handshake



   The TCP client MUST NOT include the AccECN TCP Option on the SYN.



[ms] I am not sure if I really understand the motivation for not

allowing a

option in the SYN. If the sender has space in the SYN left, what is the

harm in

an experimental version of the protocol? And I may miss something, but

what would prevent the use of 2-byte option to negotiate the use of

AccECN, e.g., to avoid experimental allocation of bit 7 in the initial

SYN?



I did have a draft on that as a proposal for an alternative design.

However,

the gourd was more supportive of this design as it is proof of middlebox

SYN

option mangling which is a know problem.



Therefore we simply don’t need option in the SYN and there is no reason

to

waste the space.



While I think many tutorial text in this document could be shortened, I

believe the use of a reserved TCP header flag should be reasoned.



I’m actually uncertain what you expect here.



For instance, this reply with the explanation of SYN option mangling seems

useful explanation.



* 3.2.8.  Usage of the AccECN TCP Option



   The following rules determine when a Data Receiver in AccECN mode

   sends the AccECN TCP Option, and which fields to include:



   Change-Triggered ACKs:  If an arriving packet increments a different

      byte counter to that incremented by the previous packet, the Data

      Receiver MUST immediately send an ACK with an AccECN Option,

      without waiting for the next delayed ACK (this is in addition to

      the safety recommendation in Section 3.2.5 against ambiguity of

      the ACE field).



      This is stated as a "MUST" so that the data sender can rely on

      change-triggered ACKs to detect transitions right from the very

      start of a flow, without first having to detect whether the

      receiver complies.  A concern has been raised that certain

offload

      hardware needed for high performance might not be able to support

      change-triggered ACKs, although high performance protocols such

as

      DCTCP successfully use change-triggered ACKs.



[ms] To me this sounds like a perfect example for a SHOULD with

additional

guidance why implementing this SHOULD is really important.



This is one of the most discussed point from the author and we really

tried to

get additional guidance here of what to do also from outside the IETF but

did

no clear feedback.



As explained this MUST enables additional functionality. However, this is

an

experiment document. If we detect that this MUST does actually hinder

implementation or has just never been implemented, we should

reconsider

this in the final PS RFC.



I was on the SHOULD side of the discussion, but can say that the

implementation in Linux was way more simple then expected. Offloading

might be a different topic but that is where I could not get a clear

feedback

and offloading could probably be anyway optimized for use with AccECN

(if

it

gets deploy widely).



If a MUST requires changes in hardware, I think there must be a clear

reason.



As individual contributor, with the current explanation in the text, I

believe this has to be a SHOULD.



I though we added this as something to mention in the exp goals section,

but

obviously we didn’t. I added the following text now:



"Another experimentation focus is the implementation feasibiliy of

change-

triggered ACKs as described in section 3.2.8. While on average this

should not

lead to a higher ACK rate, it changes the ACK patter which especially can

have

an impact on hardware offload. Further experimentation is needed to

advise

if this should a hard requirement or just prefer behavior.“



If it is unclear if a MUST can actually be implemented, having a MUST is

in my opinion the wrong approach.



One could equally state here that further experimentation is needed to

determine whether the SHOULD can be upgraded to a MUST.



   For the avoidance of doubt, the change-triggered ACK mechanism is

   deliberately worded to ignore the arrival of a control packet with

no

   payload, which therefore does not alter any byte counters, because

it

   is important that TCP does not acknowledge pure ACKs.  The change-

   triggered ACK approach will lead to some additional ACKs but it

feeds

   back the timing and the order in which ECN marks are received with

   minimal additional complexity.



[ms] The additional acks create network load. I think some wording is

needed on the tradeoff between information accuracy and network load.

There are network environments in which any additional packet is very

expensive (e.g., energy) and it is not clear to me how the protocol

design

takes into account the potential overhead of additional ACKs. Maybe this

could be another reason for a SHOULD.



The above. However, this is not really an additional ACK because you do

delay the next one. Further experimentation needed.



The document states "lead to some additional ACKs". If that does not

increase network load, I think it has to be explicitly explained why the

ACK load is at most equal to a current TCP stack, in all potential cases.

If it can increases network load, it has to be reasoned why increasing

load (and risk of reverse congestion and the like) is worth the effort.



I agree that this may be an area of experimentation, but I believe then it

has to be explained to implementers what the tradeoffs are.



* 4.2.  Compatibility with Other TCP Options and Experiments



   AccECN is compatible (at least on paper) with the most commonly

used

   TCP options: MSS, time-stamp, window scaling, SACK and TCP-AO.  It

is

   also compatible with the recent promising experimental TCP options

   TCP Fast Open (TFO [RFC7413]) and Multipath TCP (MPTCP [RFC6824]).



[ms] I would suggest the wording "... compatible with the experimental

TCP options ..." or even "... compatible with the TCP options …".



These option are to experimental..?



"It is also compatible with the experimental TCP options TCP Fast Open

(TFO [RFC7413]) and Multipath TCP (MPTCP [RFC6824])." would have same

technical meaning. Having said this, this is editorial only.



The point of using „commonly used“ was to say that we checked on those

as

they seem important. Just because your favorite experiment option is not

listed here it doesn’t means it incompatible, we just didn’t check. I’m

okay to

removed „commonly used“ but I don’t think it makes anything better.





* 4.3.  Compatibility with Feedback Integrity Mechanisms



[ms] Quite a bit in this section is experimental work, which IMHO

should be clearly emphasized. The one exception is…



I would really like to keep this section because integrity is usually the

fist

question that come up when I present AccECN. Effectively these are two

independent topics, however, I really think it help people to understand

the

whole picture if this is also discussed in this document.





      However, TCP-AO is often too brittle to use on many end-to-end

      paths, where middleboxes can make verification fail in their

      attempts to improve performance or security, e.g. by

      resegmentation or shifting the sequence space.



[ms] I am not sure if deployment challenges of other options need to be

discussed in this document.



If we keep the discussion, I guess we should mention this as well. As the

doc

clearly stated section 4 is not meant to be normative.



As far as I can see, there are use cases for TCP-AO where middleboxes are

simply not a problem, but it is exactly this sort of discussion that may

not be needed in this document. But I won't rat-hole on this comment

here.



   Originally the ECN Nonce [RFC3540] was proposed to ensure integrity

   of congestion feedback.  With minor changes AccECN could be

optimised

   for the possibility that the ECT(1) codepoint might be used as an

ECN

   Nonce . However, given RFC 3540 is being reclassified as historic,

   the AccECN design has been generalised so that it ought to be able

to

   support other possible uses of the ECT(1) codepoint, such as a lower

   severity or a more instant congestion signal than CE.



[ms] The discussion of RFC 3540 can probably be removed to a large

extent.



Unfortunately, please still think that ECN Nonce, even though is was

never

deployed and doesn’t really work, is the only was to provide integrity

protection and we need it as a prerequisite to deploy ECN at all… i would

really prefer to keep this in this non-normative part of the doc.







* 6.  IANA Considerations



[ms] I think this section needs to be rewritten to request a new

allocation

of bit 7 of the TCP header flags. At least for the process I think it

would make

sense to have somewhere in the document a comprehensive explanation

of

why an experimental document requests a change of the main TCP

header,

and why this cannot be avoided (most notably in the initial SYN) by an

alternative protocol design.



As I said previously this section is written as it would look like after

the

allocation has happened with publication approval of the IESG. This is

fine.



Having a discussion about an experiment doc assigning a flag (or not) is

a

question for tcpm as a whole and not specifically this document. How do

we

envision to every use any further flags? We go to PS right away? Or

should

we change the registration policy? For me the latter makes actually more

sense. However, if we don’t want/can to decide this now, we also could

go

forward as it is with IESG approval. However, is this case it is also not

needed

to explain this in the document. The responsible AD has to explain this

to the

other IESG probably in the ballot or even better the shepherd could

provide

these information in the write-up.







* 9.  Comments Solicited



   Comments and questions are encouraged and very welcome.  They

can

be

   addressed to the IETF TCP maintenance and minor modifications

working

   group mailing list <[e-mail]><mailto:[e-mail]>, and/or to the authors.



[ms] This section is not needed IMHO



Yes, it will be removed before publication.





10.  References



   [I-D.ietf-tsvwg-ecn-experimentation]

              Black, D., "Relaxing Restrictions on Explicit Congestion

              Notification (ECN) Experimentation",

draft-ietf-tsvwg-ecn-

              experimentation-07 (work in progress), October 2017.



[ms] Normative reference?



Don’t see why. No need to read ietf-tsvwg-ecn-experimentation to

understand the spec in this doc.



Thanks!

Mirja





_______________________________________________

tcpm mailing list

[e-mail]<mailto:[e-mail]>

https://www.ietf.org/mailman/listinfo/tcpm





---

This email has been checked for viruses by AVG.

http://www.avg.com





_______________________________________________

tcpm mailing list

[e-mail]<mailto:[e-mail]>

https://www.ietf.org/mailman/listinfo/tcpm



--

________________________________________________________________

Bob Briscoe                               http://bobbriscoe.net/