Re: [Ecn-in-quic] ECN in QUIC, quick review input wanted

[Ingemar Johansson S <[e-mail]>]

Hi
The wiki is now updated (and simplified, based on the last few days discussion)
https://github.com/quicwg/base-drafts/wiki/ECN-in-QUIC

To all of you who will take a few days off.. have nice holiday.. and to all of you who will be working.. have a good few days anyway!

/Ingemar

From: Ecn-in-quic [mailto:[e-mail]] On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
Sent: den 21 december 2017 14:34
To: Ingemar Johansson S <[e-mail]>; Magnus Westerlund <[e-mail]>; [e-mail]
Cc: Jana Iyengar ([e-mail]) <[e-mail]>; Ian Swett <[e-mail]>
Subject: Re: [Ecn-in-quic] ECN in QUIC, quick review input wanted

Hi Ingemar,

In that case, and nobody else objects we better focus our description to the running number (not referring to deltas anymore). We have the arguments as below if someone re-proposes the deltas. It is true that it becomes very complex to try to cope with all the side effects of deltas.

Koen.

From: Ingemar Johansson S [mailto:[e-mail]]
Sent: Thursday, December 21, 2017 12:13 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[e-mail]<mailto:[e-mail]>>; Magnus Westerlund <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: Jana Iyengar ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>; Ian Swett <[e-mail]<mailto:[e-mail]>>; Ingemar Johansson S <[e-mail]<mailto:[e-mail]>>
Subject: RE: [Ecn-in-quic] ECN in QUIC, quick review input wanted

Hi Koen

You have a very strong point, and even though one can possibly sort out ambiguities in one way or the other, I begin to wonder if it is worth the extra pain to potentially save a few bytes with the delta encoding alternative.

And I believe that we can compress also the running counters quite well. So I am starting to think that running counters is the best alternative.

PS: I removed the alternative (bytes) options from the Wiki. We can always put it back if there is somehow found out that we really need byte counters.

/Ingemar

From: De Schepper, Koen (Nokia - BE/Antwerp) [mailto:[e-mail]]
Sent: den 21 december 2017 11:01
To: Ingemar Johansson S <[e-mail]<mailto:[e-mail]>>; Magnus Westerlund <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: Jana Iyengar ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>; Ian Swett <[e-mail]<mailto:[e-mail]>>
Subject: RE: [Ecn-in-quic] ECN in QUIC, quick review input wanted

Hi Ingemar,

Related to the duplicates, I'm not referring to duplicate packets (with equal sequence numbers), but duplicate recomposed frames due to assumed lost previous packets. If you do not get an ACK for a previously sent packet, at a certain time you need to decide to retransmit the still relevant contents of that packet in a newly recomposed packet with a new sequence number. I assume you need to still retransmit all unACKed sequence ranges, but usually this is covered by other overlapping ACK ranges in ACK frames in former and later packets which got ACKed. For delta counters, there is no overlap possible, so every delta should carefully be sent only once. The full running counter "does" have an overlap behavior, because a newer value overwrites an older one (so sequence numbers in packets determines the "newest" packet). So counters could even go down (not that we need this).

So the problem case is:
Counter X in receiver is increased by 10 (from 54 to 64).
A packet with sequence number 104 with an ACK frame is sent back to the sender with a delta of 10 for X.
This packet is delayed too long and assumed lost.
The receiver creates a new packet with sequence number 117 with a new ACK frame with again the same delta 10 for X (as X in sender is still assumed 54 and in receiver still 64). He probably will consume the info in 104 and forgets it?
The sender is receiving the packet with sequence number 104 so increments the counter X with 10 -> 64.
The sender is receiving the packet with sequence number 117 so increments the counter X with 10 -> 74.

The receiver is getting ACK frames for his packets 104 and 117 and can see the problem only if he kept the info in 104.
As we cannot send negative deltas the receiver can use this info to not sent further deltas until his counter X goes also above 74.

For congestion control specifically it is not always needed to retransmit deltas I think. As it is a closed loop, the network will further mark packets as long as the end-systems don't respond, so these counters will anyway increase if needed. For high marking probabilities and a high loss rate in the back path this might cause slow response problems though. For classic CC's this might also mean that a loss burst which is normally within one RTT might be spread over several RTTs. Also for more deterministic protocols like our ECN capability check which should detect a single marked packet, this might be more problematic (not retransmitting may cause the sender to conclude that the path does not support ECN; sending also a counter for 00 packets might solve this; detecting a missing ACK packet might delay the decision for one more RTT, ...).

Am I correct that this is the best we can get from the QUIC packet sequence numbers, or are there other mechanisms that I'm missing and resolve this issue?

Is this problematic enough to go for running counters instead of deltas, or can we live with these sporadic imperfections?

What do you think? I'm still open for both approaches, as long as we all are aware of this issue (if it is one).

Koen.

From: Ingemar Johansson S [mailto:[e-mail]]
Sent: Wednesday, December 20, 2017 8:26 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[e-mail]<mailto:[e-mail]>>; Magnus Westerlund <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: Jana Iyengar ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>; Ian Swett <[e-mail]<mailto:[e-mail]>>
Subject: RE: [Ecn-in-quic] ECN in QUIC, quick review input wanted

Hi
A few comments inline, marked [IJ]

/Ingemar

From: De Schepper, Koen (Nokia - BE/Antwerp) [mailto:[e-mail]]
Sent: den 20 december 2017 16:02
To: Ingemar Johansson S <[e-mail]<mailto:[e-mail]>>; Magnus Westerlund <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: Jana Iyengar ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>; Ian Swett <[e-mail]<mailto:[e-mail]>>
Subject: RE: [Ecn-in-quic] ECN in QUIC, quick review input wanted

Hi Ingemar, all,

Some comments from my side:

  *   Maybe a bit early to start discussing doc structure, but I would split the structure in receiver and sender functionality. That means we first describe (after an introduction) the ECN-echo/feedback wire-protocol functionality from the receiver. It is the only generic functionality that is required from the receiver. The rest is sender behavior. This draft is making sure that deployed QUIC stacks have this ECN-echo capability build in. For the sender we describe one mandatory sender behavior: ECN-Capability-Check, and one optional: ECN-based congestion control functionality (so below, also +1 from my side).
[IJ] Yes, this makes sense



  *   In ECN Capability check section: "[ED note, should we say 'frame'?]"
No, as we want to couple ECN marking to packets. I assume the max packet sequence number in the ACK is the right total-packet counter (minus sender skipped sequence ranges, etc...).
[IJ] Yes, it should be packets, not frames



  *   The ECN-Capability check figure better shows "ACK + ECN-counters" instead of "ACK + ECN field" Boxes A and B could then show the functions inside of them (B contains [ECN-feedback] and A [ECN capability check]). To avoid confusion, I would also put: "1st packet (ECT set)".
[IJ] Yes, ACK + ECN counters is a better term, the term field was a more generic one from the days when we discussed the use of per packet indication of the ECN bits.



  *   The code example is using the TOS byte. We should avoid the same problem as with DiffServ including the ECN bits, ECN should not include/bleach the DiffServ bits. So better explicitly:

int iptos = ... (0 or optionally initialized with diffserv value, not using the 2 least significant bits)

int ect = ... (2 bit value between 0 and 3)

iptos = (iptos & 0xFC) | (ect & 0x03);

res = setsockopt(fd_out_rtp, IPPROTO_IP, IP_TOS,  &iptos, sizeof(iptos));

Maybe better to also copy the relevant ecn-getter lines in the draft:

ret = getsockopt(sock_fd, IPPROTO_IP, IP_TOS, (void*)&iptos, &optlen);

if(ret == -1) {

// getsockopt/socket failure, clean-up socket state (and/or exit);

...

}

ecn = iptos & 0x03;

diffserv = iptos & 0xFC; // if needed

Also, I'm not sure if we don't need to use a 1 byte type (unsigned char) instead of int. Using int might have a side effect that it might use the 8 MSBits (first byte) on a big indean platform, with all zeroes as a consequence. We better give a CPU-independent, well-tested and bug-free example.

[IJ] You are right, the example is a bit sloppy, should be corrected, even though this part will perhaps not end up in the transport draft, it is nonetheless good to get it right.



  *   Do we still need to describe the 3 alternatives? If we agree, I think it is better to clean-up and don't offer alternatives, to avoid repeating discussions afterwards? If no one objects, we go for the packet counters only?
[IJ] I believe that it is OK to remove the other byte related alternatives. I have so far not heard any arguments against packet counters.


  *   I prefer also having ACK and ECN in the same frame, as they need to be handled at the same level, are both depending on each other and to avoid an extra frame-type header byte. Additionally, section 8.16.2 would be too complex too, otherwise. ECN-frames would have to be considered as equal to ACK-frames, and the spec should say that "Implementations MUST NOT generate packets that only contain ACK and/or ECN frames in response to packets which only contain ACK and/or ECN frames" ... Actually, everywhere in the spec where ACK-frame is mentioned, we would have to extent what is needed for ECN frames.
[IJ] I too believe more in an ACK + ECN counter frame



  *   I am also concerned that delta-based values could cause ambiguous duplication/retransmission cases. As mentioned in the previous mail, I don't think we need to couple the counts to the ACKed ranges in the ACK blocks. This will lead to a master-mind-game-like solver to reproduce the counters as every ACK might have different partly overlapping blocks. By trying to simulate the ACK behavior with reporting deltas on the running counters at the receiver end to those counter values at the previous ACK sent, I saw these 2 possible problems: 1) if assumed lost but reordered ACK packets trigger re-sending a delta, how do we undo them when we realize they have been received anyway? 2) if the ACK for the first packet is lost, how will we detect ECN-capability in a delta based feedback if only the first packet was marked ECT by the sender? As Bob mentioned, it will take another RTT to resend it and detect ECN capability. Maybe not such a problem for detection only, and when used for CC with all packets marked ECT, it will at least increment one counter (ECT or CE, unless it was bleached). Further to be thought about.
[IJ] As I understand from earlier discussion, duplicates should not be a problem, Ian´s argument is that duplicates should be filtered out early in the packet process, and neat thing is that duplicates do not need to be recorded in QUIC,  like is for instance the case with DSACK in TCP.
As regards to packet reordering. The way I see things is that
  ECT(0) packet is received--> increment the ECT(0) counter
  ECT(1) packet is received--> increment the ECT(1) counter
  CE packet is received--> increment the CE counter
When a new ACK+ECN counter frame is generated, the deltas are computed from the difference in how much respective counter has incremented since the previous ACK + ECN counter frame.

Now there are a few possible tricky cases with the delta values that you mention above, and I believe that it is not desired to end up with complex state machines just to save a few bytes overhead.
The best alternative can after all report the counter values as is, without the additional complication with deltas, and it is not strictly necessary to report all bits in the counters, it should be sufficient to expand to 2 or 4 or 8 bytes integers only when the when 6 LSB, 14 LSB and 30LSB counters wrap around.






Koen.

From: Ecn-in-quic [mailto:[e-mail]] On Behalf Of Ingemar Johansson S
Sent: Monday, December 18, 2017 8:40 PM
To: Magnus Westerlund <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: Jana Iyengar ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>; Ian Swett <[e-mail]<mailto:[e-mail]>>
Subject: Re: [Ecn-in-quic] ECN in QUIC, quick review input wanted

Hi

Comments on most of the comments, marked [IJ].
/Ingemar

From: Magnus Westerlund
Sent: den 18 december 2017 15:19
To: Ingemar Johansson S <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: Jana Iyengar ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>; Ian Swett <[e-mail]<mailto:[e-mail]>>
Subject: Re: [Ecn-in-quic] ECN in QUIC, quick review input wanted


Hi,

I have done a review. I did edit in some editorial corrections. My comments are:

1. Requirement R.5 Editorial

I did change this requirement to what I assumed it meant to say. "must not be able" didn't make sense in the below.

* R.5 A QUIC implementation may not be able to verify ECN support in the OS network stack. The capability check should cover this case similar as if the network does not support ECN (see R6).

[IJ] OK, makes sense.

2. Comment on below note

"[In case duplicates are asked in QUIC: Once peer A receives the ACK frame it verifies that the number of ECT marked bytes(or packets) is equal to or greater than the number of transmitted ditto [ED note, "equal to or larger" in case duplicates are counted]"

This appear to be an issue we need to work through more. Using byte or packet counter where duplication sneaks in without making clear the level of duplication creates uncertainties in interpretation. At the same time suppressing a CE mark in a duplicated packet is also not without issues. The correct congestion response is knowing the actual number of delivered bytes and how much was congestion marked.

[IJ] Based on an earlier discussion on the tracker (see Ian´s comment in https://github.com/quicwg/base-drafts/issues/805 ) it is possible that this is not an issue. But it needs to be verified in the transport draft that this is indeed the case.

3. Clarification of IETF QUIC 1.0 minimal requirements.

## QUIC v1.0 limitation
Subsequent packets (after the 1st) should set the ECN bits to Not-ECT. This because v1.0 will only verify that the ECN capability exchange and the ACK+ECN frame operates correctly.

Should this be clarified, that initial capability exchange and acking with ECN information is proposed to be MUST be implemented. Where after initial exchange the setting of ECT is OPTIONAL. If a sender sets the ECN field to ECT after the initial one the sender MUST implement a congestion response to ACKs indicating ECN-CE marked packets. It is expected that future QUIC profiles will define if ECN sender capability is mandated.

[IJ] Yes, definitely a +1 one on this, unless somebody objects, I can add this

4. ECN Feedback.

Frankly I don't quite understand why ACK and ECN must be in the same Frame. Yes, they need to be in the same IP packet, and have corresponding information coverage to make them useful. But, that I don't see require combined frames. What am I missing? Is it the case of retransmitting ACK frames, where one may have multiple ACK Frames from different times in the same frame alternatively, need to re-encode the corresponding ACK interval into one new frame?

[IJ] I leave this as a separate discussion topic.

5. ECN Feedback:

I think the text should clarify that the ECN fields covers exactly the same packets that are included in the ACK. Thus these field will normally report on a relative small number of received packets between each ACK. Retransmission of an ACK frame requires corresponding ECN information to be retransmitted to, or is this case simply re-encoded to cover previous non ackwlodeged ACKs + new received packets that are ACKed?

[IJ] The idea is that the ECT(0),ECT(1) an CE deltas are stored for each transmitted ECN feedback. If one ECN feedback is indicated as lost, then the deltas are added to the next transmitted ECN feedback.

6. Editorial inconsistency in ECN Feedback.

"The wire specification below only address the ACK+ECN frame and assumes bytes marked as report metric."

Then alternative 1 is assumed and counts in packets.

[IJ] Right, I will correct

7. ECN Feedback alt 3.

"All in all this alternative would mostly encode the ECN block as 3 octects and sometimes as 4 or 6 octets."

Shouldn't this alternative result in 4 bytes or more. 2 bytes for ECT, 1 byte for the non used ECT codepoint, and 1 byte for CE when nothing is marked. Then larger ACK blocks results in additional bytes for CE and/or ECT(Value Used).

[IJ] Yes, you are right, will correct this

8. Reduction of overhead

Isn't decoupling the ECN reporting from the actual ACK blocks problematic. Then you don't know over which packets the ECN information maps to.

[IJ] The rationale is that it is only strictly necessary with a prompt transmission of ECN feedback when the CE counter increases. It is less urgent to transmit ECN feedback  if the ECT(0) and ECT(1) counters increase as these are only there to detect faults like bleaching. The proposal here is to transmit ECN feedback at least once per RTT if either ECT(0) or ECT(1) increases. In any case this is just an idea and it is possible that it complicates the design more that its worth in overhead reduction.

The reasonable method to reduce for this initial phase is that zero values for the counter corresponding to the ACK block, i.e. only Non-ECT marked packets received are using regular ACK, not including the ECN frame.

[IJ] Yes, that is a possibility



Cheers

Magnus



Den 2017-12-17 kl. 14:43, skrev Ingemar Johansson S:
Hi
I hope that we will be able to write up a draft that describes ECN in QUIC before the Melbourne Interim, but it is getting short on time and the xmas holiday is soon here.

I have now updated the ECN in QUIC wiki, based on my current best understanding. Koen, Stuart and  Praveen have also worked on the wiki + some extra input from experiments by Lars, many thanks!

Please read at your earliest convenience through the sections
https://github.com/quicwg/base-drafts/wiki/ECN-in-QUIC#ecn-capability-check

https://github.com/quicwg/base-drafts/wiki/ECN-in-QUIC#ecn-feedback-wire-format (it is sufficient to just read about alternative 1)

https://github.com/quicwg/base-drafts/wiki/ECN-in-QUIC#handling-of-lost-acks

and

https://github.com/quicwg/base-drafts/wiki/ECN-in-QUIC#reduction-of-overhead

Does this make sense ?

/Ingemar
==================================
Ingemar Johansson  M.Sc.
Master Researcher

Ericsson Research
Network Protocols & E2E Performance
Labratoriegränd 11
971 28, Luleå, Sweden
Phone [phone]
SMS/MMS [phone]
[e-mail]<mailto:[e-mail]>
www.ericsson.com

The world is full of magical things patiently
    waiting for our wits to grow sharper
               Bertrand Russell
==================================



_______________________________________________

Ecn-in-quic mailing list

[e-mail]<mailto:[e-mail]>

https://www.ietf.org/mailman/listinfo/ecn-in-quic


--



Magnus Westerlund



----------------------------------------------------------------------

Media Technologies, Ericsson Research

----------------------------------------------------------------------

Ericsson AB                 | Phone  [phone]

Torshamnsgatan 23           | Mobile [phone]

SE-164 80 Stockholm, Sweden | mailto: [e-mail]<mailto:[e-mail]>

----------------------------------------------------------------------