Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt - ACK Thining

[Bob Briscoe <[e-mail]>]

Gorry, [this thread is getting even longer, but I think we might soon 
start to violently agree.]
Inline tagged [BB2]...

On 10/11/2020 09:41, Gorry Fairhurst wrote:
>
> See in-line, marked [GF], and I will try to unpick why I don't agree 
> with some of this text.
>
> On 06/11/2020 21:51, Bob Briscoe wrote:
>> Gorry,
>>
>> Thanks as always for checking all this... inline tagged [BB]
>>
>> On 06/11/2020 14:01, Gorry Fairhurst wrote:
>>> I see this rev introduced a section on ACK Filtering, which I have 
>>> specific comments on. We need to consider the cost of sending ACKs - 
>>> especially anything that results in them being large - however, this 
>>> needs to be done by choosing a default at the receiver - not as an 
>>> adaptation, the adaption if necessary needs to be introduce more 
>>> ACKs when this is beneficial, as in some cases for DAAS, not in 
>>> trying to figure something from delayed or "missing" ACKs.
>>>
>>> 1) So let me say first that I think the suggestion for updating the 
>>> receiver to thin, and every 2-6 packets seems reasonable.
>>>
>>>     “MUST immediately send an ACK once 'n' CE marks have arrived since
>>>       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>>>       than 6.”
>>>
>>> I do agree that endpoints can do much more to reduce the return path 
>>> capacity that they consume - and that the endpoints have the 
>>> opportunity to encode the feedback information more efficiently than 
>>> can be done by a device (or queue) in the network. From our 
>>> experiments, n=10 would also often be likely OK.
>>
>> NB: n is the no. of *CE marks*, which is only the ACK thinning ratio 
>> in the worst case if there's 100% marking.
>> IOW, with n=6 the receiver could ACK less than 1 in 10 data packets 
>> whenever marking probability < 60%.
>>
>> Altho the receiver's ACK ratio will have to fall to 6 to 1 for the 
>> duration of a 100% marking sequence, whenever some data packets are 
>> not marked, the receiver aACK ratio can rise. This *could* allow any 
>> build-up of ACKs queued on the reverse path to relax, which could 
>> improve the chances of not overflowing into ACK loss.
>>
>> It's hard to predict when 100% marking in the forward direction will 
>> coincide with reverse path congestion, and I'm not saying it won't.
>> Just pointing out that n does not equal the ACK thinning ratio, it's 
>> the floor of it.
>>
>> n=6 is a consequence of only having a 3-bit field to count CE marks. 
>> We could push the envelope and make the max n=7. But not n=8, 
>> otherwise wrap after 8 marks would be indistinguishable from zero marks.
>>
> [GF] Right, so the 3-bit field, limits the ACK-Ratio tobe 6 only when 
> there is 100% marking - less with more likely patterns, that seems 
> reasonable.
>
> [GF] I also would agree that it is quite OK to increase the ACK-Ratio 
> (send more ACKs) in some specific conditions, known to endpoints.
>
> [GF] What concerns me, is that to avoid the need for ACK-Thining and 
> related methods within the network, the "normal" condition needs to be 
> for a rec eievr to send (far) fewer ACKs.
>

[BB2] "What concerns me is that, ... the normal condition needs to be 
for a receiver to send (far) fewer ACKs".
That's ambiguous to me... it might mean:
a) your concern is that the implication of what I'm saying will need far 
fewer ACKs (which you don't want).

But given what you say later, I'm pretty sure you meant:
b) your concern is that you want the normal condition to be fewer ACKs 
(i.e. you want the opposite of (a))


>>>
>>> 2. Although in some cases a receiver can predict a good "n", I do 
>>> not believe a receiver can typically find out how to choose “n” by 
>>> observing the ACKs on a connection.
>>> 2. I do not believe a receiver can find out how top to choose “n” by 
>>> observing the ACKs received on a connection.
>>
>> [BB] I think you intended to delete one of these sentences. Whatever, 
>> both of them seem to contain a typo - otherwise they are circular? 
>> Surely the receiver is sending the ACKs, so either sentence as it 
>> stands ends with "...I do not believe a receiver can typically decide 
>> how often to send an ACK by observing how often it is sending ACKs 
>> itself." ???
>>
> [GF] So, the topic and solution are a little complex ... let me try to 
> write a simpler statement: The receiver alone cannot predict a good 
> ACK ratio based on the ACKs the remote endpoint receives. ECN-marking 
> of an ACK stream does not provide useful informatiopn, because in this 
> specific case, the control packets are small, and we enter the realm 
> of packet-congestible links (as in RFC7141), where the overhead of 
> transmission time slot scheduling and fitting packets to burst 
> transmission plans can be more sigmnificant than the number of bits sent.
>

[BB2] If a link becomes packet-congested, ECN can (and should) still be 
used to indicate such congestion. The implementation has to be designed 
so that the ECN-marking process isn't compromised by packet processing 
overload. But that's perfectly possible for a good implementer.


>>>
>>> Sure, if the path is modelled as a simple bit-congestive path that 
>>> has a queue, then you could monitoring timing/loss of packets and 
>>> determine a rate that tries not to fill a queue. I would not know 
>>> how common such a bottleneck link is.
>>>
>>> I do know that many bottleneck links are more complex and needs to 
>>> consider the scheduling transmission of small ACK packets on the 
>>> return path performance. In many radio technologies, the overhead 
>>> from transmission of ACKs is proportionally much higher than for 
>>> data segments -  These layer 2s are constrained in terms of packets 
>>> (e.g. where lower layer framing and transmission scheduling is an 
>>> important consideration to the efficiency/cost of using the 
>>> service). This has often been “fixed” using ACK Thinning or PEPs. 
>>> Protocols that send fewer ACKs/data packet are not impacted.
>>
>> [BB] I agree that the receiver alone cannot predict a good 'n'.
>> AccECN is a not the complete solution, but we've grasped the 
>> opportunity to take a step in the right direction (within the 
>> chartered scope of AccECN). AccECN at least allows the data sender to 
>> measure the extent of congestion marking on the arriving ACK stream 
>> (if there's ECN support at the bottleneck). Whereas before AccECN, a 
>> random ECN mark on one pure ACK in a round couldn't be distinguished 
>> from 100% of ACKs marked.
>>
> [GF] We've relied for decades on ACK-Thinning (or equivalent) in the 
> network, where the link KNOWS what the sub-IP layer is doing, and 
> hence knows whether an ACK is eating into resources that are scarce; 
> but this solution is already problematic for encrypted traffic, and 
> when ACKs carry more than ACK information. So, back to my point: To 
> fix this, I think we need to reduce the level of feedback at the receiver.
>

[BB2] Certainly as we get more packets per RTT, we shouldn't need to 
have more ACKs per RTT. Better still, we ought to pack more information 
into each ACK.

However, do you have enough evidence to say reducing the absolute level 
of feedback is /all/ that is necessary? That's quite a strong assertion. 
It presumes that there is not a trade-off. It says, if we achieve some 
lower level of feedback then, in all the cases when the reverse path is 
uncongested, it would never be advantageous to increase the ACK frequency.


> [GF] So ... I see two problems with using ECN marks on the return path 
> to vary the ACK Ratio:
>
> (i) I'm not so sure the Layer 2 knows how to mark ACK packets - 
> because they are not bit-congestible signals, and  I'd prefer to see 
> ECN marking based on queue capacity (bit congestibale).
>

[BB2] I don't see why ECN is hard or inappropriate for 
packet-congestion. Is there any experience that says that? And I don't 
know why you'd prefer to see ECN for bit congestion. Is there some 
reasoning behind that?

> I could see methods that let the sub-IP resource allocation ballance 
> the use of forward and return transmssion opportunities and use this 
> to CE-mark both directions - and see if flows adapt their return path 
> resource consumption to optimise their forward data path transmission. 
> It could perform better if it were to seperate ACKs (or any short 
> packets? - also sometimes done) from "actual" data and then 
> pro-actively CE-mark ACK packets to free-up transmission resource for 
> other packets in the uplink or downlink. I'm not sure this cross-layer 
> coupling is wise.
>

[BB2] Why complicate matters? A link doesn't know which flow in one 
direction relates to which flow in the other (the reverse path of some 
flows might not even go through this link). And a link doesn't know the 
relative importance of ACKs and data, which is specific to different 
congestion controllers. So how can a link balance both directions well? 
What's wrong with the link blindly signalling congestion on packets 
(irrespective of whether data or ACKs), in each direction independently. 
And let the end systems find which degree of self-controlled 
ACK-thinning maximizes their performance.

What is important is to come to consensus on whether the link is or is 
not the best place to control reverse path congestion.  As above, there 
is scope for transports to just reduce the regular ACK load, full stop. 
Beyond that, to find a good trade-off between too much and too few ACKs, 
I certainly think we can agree that neither the transport nor the link 
can do that well on its own. Communication between them seems essential. 
Attempting to communicate from host to network seems like a non-starter, 
given the information the transport would have to convey would be quite 
complex. Congestion signalling from the link to the transport seems 
sufficient (to me). And using ECN seems likely to be significantly 
better than using loss.


> (ii) Before a new method is deployed, and the ACK Ratio remains 1:2, 
> then the network nodes still do either ACK-Thinning or queue the ACKs.
>

[BB2] As above, I'm sure there's scope for reducing ACK frequency 
without (significantly?) affecting performance. And you've been doing 
this research, so you should know. And I guess, maybe, that might make 
most link-based ACK filtering unnecessary. But I don't think we need to 
decide now that this will be enough, and ACK CC will not be necessary 
(see later).

>>>
>>> Scheduling (at the IP layer,can also often result in variations in 
>>> timing - something that is hard to predict and I would suggests make 
>>> automated selection of “n” very hard, and the need to thin is often 
>>> not directly visible to a user of the path. Traffic multiplexed with 
>>> protocols that use different ACK policies (perhaps QUIC?) also make 
>>> this sort of detection less reliable.
>>
>> [BB] Being hard shouldn't prevent us trying.
>> The remaining piece would be to advance a TCP ACK congestion control 
>> protocol on the experimental track (RFC5690 
>> <https://tools.ietf.org/html/rfc5690> is informational).
>>
>
> [GF] To me, this is a complicated way to approach the question of how 
> much feedback is needed. I actually don't see the justification for 
> adding an extra control loop, that will interact with other control 
> loops in the network.
>

[BB2] Yes, I understand that you favour the transport picking a higher 
ACK ratio for all paths. But surely the empirical evidence from 
link-based ACK filtering shows that it adapts the ACK ratio to 
conditions, and doesn't just apply a higher ACK ratio all the time. But 
I guess you would argue that's because links are fighting with 
transports that weren't specifically designed to cope with a higher ACK 
ratio, so not filtering at all is better when it's not necessary.

>>>
>>> 3) A queue might be competing for scheduling opportunities against 
>>> other IP flows (e.g., WFQ), or competing for transmission 
>>> opportunities with a shared resource (common in radio links).
>>>
>>> ACKs can and do take transmission opportunities away from the 
>>> resource pools that would be used by other services that share the 
>>> same resource pool. This can be reduced by some form of per-flow 
>>> scheduling with deeper per-flow queues - but that in itself can 
>>> induce unwanted coupling between flows - limiting the peak rate of 
>>> one flow, to achieve a higher ACK ratio for another flow.
>>>
>>> Consider a QUIC flow sharing a return path with a TCP flow, the 
>>> higher rate of larger QUIC ACKs reduces the transmission 
>>> opportunities for TCP ACKs, which in turn suppresses the TCP flow, 
>>> limiting the overall system performance. If the ACKs use the same 
>>> transmission resource, they could take directly away from 
>>> transmission opportunities in the forward direction, but even if 
>>> they do not - the rate of ACKs retuned impacts the sending rate end 
>>> to end.

[BB2] A question:
Has the approach ever been tried where the link just does dumb 
congestion notification (without regard to type or size of packets), and 
the end system congestion controls optimize data transfer performance 
and the tradeoff with ACK ratio? Whether analytically, or empirically?

>>
>> [BB] This says two things to me:
>> * if TCP is going to introduce control over the ACK ratio, QUIC and 
>> other transports ought to as well {Note 1}.
>> * and all ACK CC algorithms ought to be designed to ensure some 
>> degree of equitable capacity sharing {Note 2}, including between the 
>> ACK stream of flow A and the data stream of flow B transferring in 
>> the opposite direction.
>>
>> {Note 1}: I don't think this needs any formal "all or nothing" rule 
>> where the transport ADs co-ordinate multiple WGs. Enough of the 
>> people involved are the same for the right thing to happen (I think). 
>> But some AD oversight wouldn't do any harm.
>> {Note 2}: There are numerous challenges to face, but I think it's 
>> paramount to first establish which node is considered responsible for 
>> solving the problem (the bottleneck or one of the transport 
>> endpoints). This doesn't imply that the IETF will ever ensure that 
>> other nodes don't 'interfere'. For instance, if the sender is deemed 
>> responsible for determining the level of ACK filtering, it will still 
>> have to make allowance for bottlenecks attempting to ACK thin as well.
>>
>>
>>>
>>> 4) The new section adds some cross-layer requirements on sub-IP 
>>> layers, which seem to imply more complexity. I’m not comfortable 
>>> with such a proposal.
>>>
>>> It seems that the text suggests not implement thinning and that 
>>> these systems implement logic to detect the presence of the option 
>>> by observing SYNs and then configure the link to do something special. 
>>
>> This was the proposal that I was most interested to hear your views 
>> on. We can remove it if the WG doesn't want it, but first I want to 
>> make sure the proposal is fully understood:
>> * The aim is for the RFC to tell bottleneck implementers (via this 
>> text) that *only in cases where AccECN has been negotiated* the 
>> preferred way to improve flow performance is for the link to signal 
>> ECN to the transport, not to drop the ACKs itself.
>
> [GF] In principle could be interesting - if this were a separate 
> signal for "ACKs" - we don't have this signal. We have a signal for 
> all IP traffic.
>

[BB2] I don't think the signal should be different for ACKs and data. 
That's unnecessarily complicated for the link, and makes the transport's 
job harder if different links each choose a different balance (it's hard 
enough even if all links use the same balance).

>> * If the link doesn't support ECN, it still does the ACK filtering itself
>> * Links aren't going to delegate their control unless transports 
>> actually act on the ECN signals that links emit - and improve 
>> performance better than links have been. So, it's important that 
>> transports take on these responsibilities - the data sender takes 
>> responsibility for determining the ACK rate, asking [I meant 
>> 'assuming'] its correspondent receiver does actually thin its own ACKs.
>> Here's who signals what, and who does what
>>
>>                link-.
>>                      \
>>                       '-[ECN CE]-.
>>                                   \
>>                                    '-> receiver
>>                                     .-
>>                                    /
>>             .-[(Acc)ECN feedback]-'
>>            /
>>  sender <-'
>>         -.
>>           \
>>            '---[ACK CC ratio]---.
>>                                  \
>>                                   '-> receiver(treats as polite 
>> request, not mandatory)
>>
>>
> [GF] That's a useful diagram to help understand!!! The control loops 
> for layer 2 channel sharing oftem operate on a similar timescale.
>
> [GF] Let's be sure we start from the same problem:
>
> * Current TCP Specs suggest the ACK Ratio is 1:2. I could say that 
> means that 33% of transmission opportunities are used solely for 
> feedback.
>

[BB2] Well, that depends how much aggregation is used to pack packets 
into each xmt opportunity. But your figures here and below are still 
meaningful relative to each other, even if aggregation shifts them all 
downwards in absolute terms.

> Before we had sophisticated transports, tha was fine, but nowadays 
> this is a very high proportion by modern standards, and given the way 
> the ACK-Clock no longer has to be for each segment or pair of segments 
> ... reality moved away from that when stretch-ACKs became common.
>
> * Many senders (e.g, using offload) might have an ACK Ratio of 1:4, 
> but not all.
>
> * Some paths "Thin" ... by how much depends.  In many cases, an ACK 
> Ratio 1:2, causes excessive resource usage that predjudices 
> performance, an ACK Ratio of 1:4 is still 20% of transmission 
> opportunities. An ACK-Ratio of 1:10, would mean 10% of transmission 
> opportunities are used for feedback (that's an order of magnitude less 
> control than data segments, and is a reasonable operating point). My 
> suggestion is that MOST of the time, transports can live with an ACK 
> Ratio 1:10, and in many cases this is enough to avoid ACK Thinning 
> measures being triggered.
>
> * A transport can adapt feedback, for sure,  and there will be cases 
> where lower rate (e.g. 1:1 or 1:2 initially) or higher rates 
> (1:10,1:20, etc as rates increase and rate-based methods are used) 
> will be benefit the transport - but changing the default operating 
> point from 1:2 to 1:10 is likely already significantly reducing the 
> need (and triggering) of ACK thinning.
>

[BB2] The point tho' is that the opportunity for ACK thinning has arisen 
as packet rates have increased, because you only really need a certain 
number of ACKs per RTT to maintain control. E.g if you need 8 ACKs per 
RTT then if the window is 16 you need ACK ratio = 2, but if the window 
is 320, ACK ratio = 40 should work OK. However, just because the average 
flow rate is continually increasing, doesn't mean flows don't still need 
to go slow before they go fast.

If a link enforces an ACK ratio, it doesn't know which flows are 
currently getting started, which flows have small RTTs, etc. etc. Flows 
know that.

> - So, maybe we should do this, rather than encourage a new generation 
> of smart ECN-based ACK modification schemes. If we mitigate the 
> feedback traffic, we can then focus on deploying ECN for capacity 
> detection.
>

[BB2] I'm fine starting out with transports doing more ACK thinning 
themselves.
But it's no use if links still filter the ACKs of these 'good' flows on 
top of the flows filtering themselves.

QUIC has encryption to prevent that (at least until networks find 
heuristics to do QUIC ACK thinning). TCP does not.

So it does not harm to require links not to filter AccECN-capable 
packets (perhaps we can widen that to all ECN-capable packets).
And it also does no harm to recommend that links ECN mark instead. If 
they did, they would of course ECN-mark all ECN-capable packets, of course.

Then:
* if your strategy of cutting down the absolute ACK ratio is sufficient, 
we don't need to introduce ACK CC, but we still have more ECN deployment 
anyway.
* And if your strategy isn't enough, we have the signals for the basis 
of robust ACK CC.

>>> I don’t think think this is the correct approach - the result will 
>>> be throttling of the connection speed. Is it necessary that the 
>>> sub-IP layer does not implement thinning for connections that use 
>>> AccECN feedback?
>>
>> Why do you say the connection speed would be throttled?
> [GF] For example: A sudden increasse in reverse path traffic will need 
> to be buffered until the radio transmission schedule changes. That 
> will delay feedback to the sender (advertised rwnd/cwnd for TCP), and 
> hence the forward rate will then throttle determined by the bottleneck 
> ACK rate.

[BB2] Well, actually the ACKs leaving the head of the buffer should be 
congestion-marked for the backlog behind them when they dequeue, as well 
as for their sojourn time (due to the buffer in front of them when they 
enqueued). I can point you at a paper on this if interested.
But, whatever, this is a better way to get the receiver to thin the ACK 
stream than the network discarding packets from it (better = less 
damaging to performance).

And, bear in mind that the first step is for transports to reduce ACK 
ratio overall, so we need a way for the link to distinguish the old 
transports that probably haven't reduced their ACK ratio, from newer 
ones that probably have.

Encouraging the link to ECN mark (all traffic incl ACKs) is worthwhile 
whether or not it's used for ACK CC.

And we only resort to ACK CC if necessary - but the signals are there if 
we need to develop the algorithms.

>> Did the text not explain the proposal well?
>
> Well, I think the problem is complex, which is why this email is 
> already long.
>
>> Or did you understand the proposal but disagree with it?
>>
>>
>>>
>>> When the text says: “SHOULD preserve the timing of each ACK”
>>> - does that imply that nodes need to discard rather than delay 
>>> packets when link scheduling or transmission scheduling is used. 
>>> This seems a very odd requirement, it might be better too simply 
>>> strip-out the AccECN option or discard all packets that set this.
>>
>> Sorry, that may be clumsy wording. It just means "the link SHOULD NOT 
>> filter ACKs" (if it supports ECN marking).
>>>
>>> I can’t see how queuing the ACKs, or deleting ACKs rather than 
>>> queuing them would improve the performance of AccECN TCP connections.
>>
>> May be the piece where the link's preferred role is to signal to the 
>> transport to slow down the ACK rate using ECN wasn't clear.
>>

Bob


>>
>> Bob
>>
>>>
>>> Gorry
>>>
>>
>> -- 
>> ________________________________________________________________
>> Bob Briscoehttp://bobbriscoe.net/
>
> Summary ... because this thread has lots of "discussion" or a 
> complicated interaction, and you asked about the AccECN draft text:
>
> + I agree that AccECN can be used with a sender that uses a more 
> appropiate target ACK Ratio - that is good! I agree this is worth saying.
>
> + ACK Thining, and variants - including uses of compression - is a 
> deployed mitigation, and that changing the ACK Ratio can eliminate 
> this practice and the need to implement this.
>
> + I agree that if the path is a simple bottleneck, then ECN-marking 
> should be encouraged. However, not appropiate without much more 
> consideration for a more complex L2 where the marking is 
> packet-congestible. A link predomonantly carrying small ACKs seems to 
> lead into the research issues in RFC7141.
>
> + I would disagree with any encouragement to use cross-layer methods 
> with use ECN-marking to seek to adjust ACK feedback traffic and I am 
> concerned that using ECN to modify the ACK Ratio might create new 
> layers of feedback loop that will aggrevate the problem.
>
> I think the current text goes to far.
>
> Gorry
>

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/