Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt - ACK Thining

[Gorry Fairhurst <[e-mail]>]

See in-line, marked [GF], and I will try to unpick why I don't agree 
with some of this text.

On 06/11/2020 21:51, Bob Briscoe wrote:
> Gorry,
>
> Thanks as always for checking all this... inline tagged [BB]
>
> On 06/11/2020 14:01, Gorry Fairhurst wrote:
>> I see this rev introduced a section on ACK Filtering, which I have 
>> specific comments on. We need to consider the cost of sending ACKs - 
>> especially anything that results in them being large - however, this 
>> needs to be done by choosing a default at the receiver - not as an 
>> adaptation, the adaption if necessary needs to be introduce more ACKs 
>> when this is beneficial, as in some cases for DAAS, not in trying to 
>> figure something from delayed or "missing" ACKs.
>>
>> 1) So let me say first that I think the suggestion for updating the 
>> receiver to thin, and every 2-6 packets seems reasonable.
>>
>>     “MUST immediately send an ACK once 'n' CE marks have arrived since
>>       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>>       than 6.”
>>
>> I do agree that endpoints can do much more to reduce the return path 
>> capacity that they consume - and that the endpoints have the 
>> opportunity to encode the feedback information more efficiently than 
>> can be done by a device (or queue) in the network. From our 
>> experiments, n=10 would also often be likely OK.
>
> NB: n is the no. of *CE marks*, which is only the ACK thinning ratio 
> in the worst case if there's 100% marking.
> IOW, with n=6 the receiver could ACK less than 1 in 10 data packets 
> whenever marking probability < 60%.
>
> Altho the receiver's ACK ratio will have to fall to 6 to 1 for the 
> duration of a 100% marking sequence, whenever some data packets are 
> not marked, the receiver aACK ratio can rise. This *could* allow any 
> build-up of ACKs queued on the reverse path to relax, which could 
> improve the chances of not overflowing into ACK loss.
>
> It's hard to predict when 100% marking in the forward direction will 
> coincide with reverse path congestion, and I'm not saying it won't.
> Just pointing out that n does not equal the ACK thinning ratio, it's 
> the floor of it.
>
> n=6 is a consequence of only having a 3-bit field to count CE marks. 
> We could push the envelope and make the max n=7. But not n=8, 
> otherwise wrap after 8 marks would be indistinguishable from zero marks.
>
[GF] Right, so the 3-bit field, limits the ACK-Ratio tobe 6 only when 
there is 100% marking - less with more likely patterns, that seems 
reasonable.

[GF] I also would agree that it is quite OK to increase the ACK-Ratio 
(send more ACKs) in some specific conditions, known to endpoints.

[GF] What concerns me, is that to avoid the need for ACK-Thining and 
related methods within the network, the "normal" condition needs to be 
for a rec eievr to send (far) fewer ACKs.

>>
>> 2. Although in some cases a receiver can predict a good "n", I do not 
>> believe a receiver can typically find out how to choose “n” by 
>> observing the ACKs on a connection.
>> 2. I do not believe a receiver can find out how top to choose “n” by 
>> observing the ACKs received on a connection.
>
> [BB] I think you intended to delete one of these sentences. Whatever, 
> both of them seem to contain a typo - otherwise they are circular? 
> Surely the receiver is sending the ACKs, so either sentence as it 
> stands ends with "...I do not believe a receiver can typically decide 
> how often to send an ACK by observing how often it is sending ACKs 
> itself." ???
>
[GF] So, the topic and solution are a little complex ... let me try to 
write a simpler statement: The receiver alone cannot predict a good ACK 
ratio based on the ACKs the remote endpoint receives. ECN-marking of an 
ACK stream does not provide useful informatiopn, because in this 
specific case, the control packets are small, and we enter the realm of 
packet-congestible links (as in RFC7141), where the overhead of 
transmission time slot scheduling and fitting packets to burst 
transmission plans can be more sigmnificant than the number of bits sent.

>>
>> Sure, if the path is modelled as a simple bit-congestive path that 
>> has a queue, then you could monitoring timing/loss of packets and 
>> determine a rate that tries not to fill a queue. I would not know how 
>> common such a bottleneck link is.
>>
>> I do know that many bottleneck links are more complex and needs to 
>> consider the scheduling transmission of small ACK packets on the 
>> return path performance. In many radio technologies, the overhead 
>> from transmission of ACKs is proportionally much higher than for data 
>> segments -  These layer 2s are constrained in terms of packets (e.g. 
>> where lower layer framing and transmission scheduling is an important 
>> consideration to the efficiency/cost of using the service). This has 
>> often been “fixed” using ACK Thinning or PEPs. Protocols that send 
>> fewer ACKs/data packet are not impacted.
>
> [BB] I agree that the receiver alone cannot predict a good 'n'.
> AccECN is a not the complete solution, but we've grasped the 
> opportunity to take a step in the right direction (within the 
> chartered scope of AccECN). AccECN at least allows the data sender to 
> measure the extent of congestion marking on the arriving ACK stream 
> (if there's ECN support at the bottleneck). Whereas before AccECN, a 
> random ECN mark on one pure ACK in a round couldn't be distinguished 
> from 100% of ACKs marked.
>
[GF] We've relied for decades on ACK-Thinning (or equivalent) in the 
network, where the link KNOWS what the sub-IP layer is doing, and hence 
knows whether an ACK is eating into resources that are scarce; but this 
solution is already problematic for encrypted traffic, and when ACKs 
carry more than ACK information. So, back to my point: To fix this, I 
think we need to reduce the level of feedback at the receiver.

[GF] So ... I see two problems with using ECN marks on the return path 
to vary the ACK Ratio:

(i) I'm not so sure the Layer 2 knows how to mark ACK packets - because 
they are not bit-congestible signals, and  I'd prefer to see ECN marking 
based on queue capacity (bit congestibale). I could see methods that let 
the sub-IP resource allocation ballance the use of forward and return 
transmssion opportunities and use this to CE-mark both directions - and 
see if flows adapt their return path resource consumption to optimise 
their forward data path transmission. It could perform better if it were 
to seperate ACKs (or any short packets? - also sometimes done) from 
"actual" data and then pro-actively CE-mark ACK packets to free-up 
transmission resource for other packets in the uplink or downlink. I'm 
not sure this cross-layer coupling is wise.

(ii) Before a new method is deployed, and the ACK Ratio remains 1:2, 
then the network nodes still do either ACK-Thinning or queue the ACKs.

>>
>> Scheduling (at the IP layer,can also often result in variations in 
>> timing - something that is hard to predict and I would suggests make 
>> automated selection of “n” very hard, and the need to thin is often 
>> not directly visible to a user of the path. Traffic multiplexed with 
>> protocols that use different ACK policies (perhaps QUIC?) also make 
>> this sort of detection less reliable.
>
> [BB] Being hard shouldn't prevent us trying.
> The remaining piece would be to advance a TCP ACK congestion control 
> protocol on the experimental track (RFC5690 
> <https://tools.ietf.org/html/rfc5690> is informational).
>

[GF] To me, this is a complicated way to approach the question of how 
much feedback is needed. I actually don't see the justification for 
adding an extra control loop, that will interact with other control 
loops in the network.

>>
>> 3) A queue might be competing for scheduling opportunities against 
>> other IP flows (e.g., WFQ), or competing for transmission 
>> opportunities with a shared resource (common in radio links).
>>
>> ACKs can and do take transmission opportunities away from the 
>> resource pools that would be used by other services that share the 
>> same resource pool. This can be reduced by some form of per-flow 
>> scheduling with deeper per-flow queues - but that in itself can 
>> induce unwanted coupling between flows - limiting the peak rate of 
>> one flow, to achieve a higher ACK ratio for another flow.
>>
>> Consider a QUIC flow sharing a return path with a TCP flow, the 
>> higher rate of larger QUIC ACKs reduces the transmission 
>> opportunities for TCP ACKs, which in turn suppresses the TCP flow, 
>> limiting the overall system performance. If the ACKs use the same 
>> transmission resource, they could take directly away from 
>> transmission opportunities in the forward direction, but even if they 
>> do not - the rate of ACKs retuned impacts the sending rate end to end.
>
> [BB] This says two things to me:
> * if TCP is going to introduce control over the ACK ratio, QUIC and 
> other transports ought to as well {Note 1}.
> * and all ACK CC algorithms ought to be designed to ensure some degree 
> of equitable capacity sharing {Note 2}, including between the ACK 
> stream of flow A and the data stream of flow B transferring in the 
> opposite direction.
>
> {Note 1}: I don't think this needs any formal "all or nothing" rule 
> where the transport ADs co-ordinate multiple WGs. Enough of the people 
> involved are the same for the right thing to happen (I think). But 
> some AD oversight wouldn't do any harm.
> {Note 2}: There are numerous challenges to face, but I think it's 
> paramount to first establish which node is considered responsible for 
> solving the problem (the bottleneck or one of the transport 
> endpoints). This doesn't imply that the IETF will ever ensure that 
> other nodes don't 'interfere'. For instance, if the sender is deemed 
> responsible for determining the level of ACK filtering, it will still 
> have to make allowance for bottlenecks attempting to ACK thin as well.
>
>
>>
>> 4) The new section adds some cross-layer requirements on sub-IP 
>> layers, which seem to imply more complexity. I’m not comfortable with 
>> such a proposal.
>>
>> It seems that the text suggests not implement thinning and that these 
>> systems implement logic to detect the presence of the option by 
>> observing SYNs and then configure the link to do something special. 
>
> This was the proposal that I was most interested to hear your views 
> on. We can remove it if the WG doesn't want it, but first I want to 
> make sure the proposal is fully understood:
> * The aim is for the RFC to tell bottleneck implementers (via this 
> text) that *only in cases where AccECN has been negotiated* the 
> preferred way to improve flow performance is for the link to signal 
> ECN to the transport, not to drop the ACKs itself.

[GF] In principle could be interesting - if this were a separate signal 
for "ACKs" - we don't have this signal. We have a signal for all IP traffic.

> * If the link doesn't support ECN, it still does the ACK filtering itself
> * Links aren't going to delegate their control unless transports 
> actually act on the ECN signals that links emit - and improve 
> performance better than links have been. So, it's important that 
> transports take on these responsibilities - the data sender takes 
> responsibility for determining the ACK rate, asking its correspondent 
> receiver does actually thin its own ACKs.
> Here's who signals what, and who does what
>
>                link-.
>                      \
>                       '-[ECN CE]-.
>                                   \
>                                    '-> receiver
>                                     .-
>                                    /
>             .-[(Acc)ECN feedback]-'
>            /
>  sender <-'
>         -.
>           \
>            '---[ACK CC ratio]---.
>                                  \
>                                   '-> receiver(treats as polite 
> request, not mandatory)
>
>
[GF] That's a useful diagram to help understand!!! The control loops for 
layer 2 channel sharing oftem operate on a similar timescale.

[GF] Let's be sure we start from the same problem:

* Current TCP Specs suggest the ACK Ratio is 1:2. I could say that means 
that 33% of transmission opportunities are used solely for feedback. 
Before we had sophisticated transports, tha was fine, but nowadays this 
is a very high proportion by modern standards, and given the way the 
ACK-Clock no longer has to be for each segment or pair of segments ... 
reality moved away from that when stretch-ACKs became common.

* Many senders (e.g, using offload) might have an ACK Ratio of 1:4, but 
not all.

* Some paths "Thin" ... by how much depends.  In many cases, an ACK 
Ratio 1:2, causes excessive resource usage that predjudices performance, 
an ACK Ratio of 1:4 is still 20% of transmission opportunities. An 
ACK-Ratio of 1:10, would mean 10% of transmission opportunities are used 
for feedback (that's an order of magnitude less control than data 
segments, and is a reasonable operating point). My suggestion is that 
MOST of the time, transports can live with an ACK Ratio 1:10, and in 
many cases this is enough to avoid ACK Thinning measures being triggered.

* A transport can adapt feedback, for sure,  and there will be cases 
where lower rate (e.g. 1:1 or 1:2 initially) or higher rates (1:10,1:20, 
etc as rates increase and rate-based methods are used) will be benefit 
the transport - but changing the default operating point from 1:2 to 
1:10 is likely already significantly reducing the need (and triggering) 
of ACK thinning.

- So, maybe we should do this, rather than encourage a new generation of 
smart ECN-based ACK modification schemes. If we mitigate the feedback 
traffic, we can then focus on deploying ECN for capacity detection.

>> I don’t think think this is the correct approach - the result will be 
>> throttling of the connection speed. Is it necessary that the sub-IP 
>> layer does not implement thinning for connections that use AccECN 
>> feedback?
>
> Why do you say the connection speed would be throttled?
[GF] For example: A sudden increasse in reverse path traffic will need 
to be buffered until the radio transmission schedule changes. That will 
delay feedback to the sender (advertised rwnd/cwnd for TCP), and hence 
the forward rate will then throttle determined by the bottleneck ACK rate.
> Did the text not explain the proposal well?

Well, I think the problem is complex, which is why this email is already 
long.

> Or did you understand the proposal but disagree with it?
>
>
>>
>> When the text says: “SHOULD preserve the timing of each ACK”
>> - does that imply that nodes need to discard rather than delay 
>> packets when link scheduling or transmission scheduling is used. This 
>> seems a very odd requirement, it might be better too simply strip-out 
>> the AccECN option or discard all packets that set this.
>
> Sorry, that may be clumsy wording. It just means "the link SHOULD NOT 
> filter ACKs" (if it supports ECN marking).
>>
>> I can’t see how queuing the ACKs, or deleting ACKs rather than 
>> queuing them would improve the performance of AccECN TCP connections.
>
> May be the piece where the link's preferred role is to signal to the 
> transport to slow down the ACK rate using ECN wasn't clear.
>
>
> Bob
>
>>
>> Gorry
>>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

Summary ... because this thread has lots of "discussion" or a 
complicated interaction, and you asked about the AccECN draft text:

+ I agree that AccECN can be used with a sender that uses a more 
appropiate target ACK Ratio - that is good! I agree this is worth saying.

+ ACK Thining, and variants - including uses of compression - is a 
deployed mitigation, and that changing the ACK Ratio can eliminate this 
practice and the need to implement this.

+ I agree that if the path is a simple bottleneck, then ECN-marking 
should be encouraged. However, not appropiate without much more 
consideration for a more complex L2 where the marking is 
packet-congestible. A link predomonantly carrying small ACKs seems to 
lead into the research issues in RFC7141.

+ I would disagree with any encouragement to use cross-layer methods 
with use ECN-marking to seek to adjust ACK feedback traffic and I am 
concerned that using ECN to modify the ACK Ratio might create new layers 
of feedback loop that will aggrevate the problem.

I think the current text goes to far.

Gorry