Re: [tcpm] More motivating scenarios for tcpm-ack-pull

["Carles Gomez Montenegro" <[e-mail]>]

Hi Richard,

Thanks a lot for your message!

It is very interesting to witness how the patch you mention offers a way
to trigger immediate ACKs, and the related context and benefits.

Perhaps a combination or (re)use of existing TCP header fields (e.g.
CWR/PSH as per your suggestion below, or the Urgent pointer as suggested
earlier by Bob) could be a way to enable triggering immediate ACKs.

A concern with these techniques is that they are limited to some extent.
For example, some applications produce small segments with the PSH flag
set and benefit from Delayed ACKs (e.g. to piggyback ACKs and replies). In
those cases, even regardless of the CWR flag, triggering immediate ACKs
will avoid achieving performance benefits of Delayed ACKs.

In contrast, a dedicated resource (e.g. a TCP header reserved bit, as we
propose in the draft) will not suffer such limitations. However, that
resource needs to be allocated.

Summarizing, there may be two approaches to enable ACK Pull:

1.- Overloading semantics of existing TCP header fields.
2.- Dedicating a resource.

Then the question is, considering pros and cons of both approaches: which
one do we prefer?

It would be great to hear the WG feedback on the above question!

Thanks,

Carles




> Carles,
>
> I just found, that as a performance improvement measure, linux has
> recently added a patch to trigger immediate ACKs when a packet with CWR
> is received...
>
> While the underlying issue was found in the dctcp context, the patch
> appears to be against the generic rfc3168 ECN handling code.
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fd2123a3d7527d4c7092633d55e877c0cc1d84a3
>
> While that performance improvement has obvious benefits, RFC3168 is
> silent on this particular issue.
>
> But there are use cases where it is certainly valuable for the sender to
> be able to elicit an immediate ACK from the receiver.
>
>
> I don't know if it warrants a dedicated bit though - or if, for example,
> the combination of CWR/PSH could perhaps obtain the ACK PULL semantics
> as an overload of of their normal semantics...
>
>
> Best regards,
>     Richard
>
>
> Am 18.11.2019 um 08:21 schrieb Jon Crowcroft:
>> Mark Handley said a lot of the middlebox corner case avoidance stuff is
>> in the tcpmp RFC...have got time to look just yet...
>>
>> On Mon, 18 Nov 2019, 04:46 Carles Gomez Montenegro,
>> <[e-mail] <mailto:[e-mail]>> wrote:
>>
>>     Hi Bob,
>>
>>     Once again, thanks a lot for all the very useful feedback!
>>
>>     Please find below some inline responses.
>>
>>      > Carles, Jon, (re-sending, this time without accidentally omitting
>>     tcpm,
>>      > also see couple of addenda tagged [Bob adds:])
>>      >
>>      > I'm generally supportive of a mechanism to suppress delayed ACKs
>> from
>>      > the receiver (in any transport protocol including TCP). So thank
>> you.
>>      >
>>      > You might want to borrow at least some of the text for additional
>>      > motivating scenarios for such a facility from the first two posts
>>     on a
>>      > thread for a similar facility in QuiC:
>>      > https://github.com/quicwg/base-drafts/issues/1978
>>      >
>>      > Also pasted at the end.
>>      >
>>      > Some is not relevant to your specific ack-pull scheme, 'cos it's
>>      > proposing a more general facility in QUIC for the sender to the
>>     receiver
>>      > alters its ACK ratio. (theoretically there are more bits
>> available in
>>      > QUIC, but this particular request has been put on hold while
>> someone
>>      > goes and finds them ;)
>>
>>     Thanks a lot. This is indeed very useful!
>>
>>     As you explain, the more general facility for QUIC allows to
>> suppress
>>     Delayed ACKs and also allows to modify the ACK ratio in a more
>> general
>>     way.
>>
>>     Regarding the Delayed ACK suppression discussion, the additional
>>     motivation is definitely important.
>>
>>      >       Another motivating example (rather niche)
>>      >
>>      > Coincidentally, Just this morning I published a tech report that
>>     had to
>>      > use the hack of overlapping a byte from the previous segment to
>>     force a
>>      > quick ack:
>>      > TCP Prague Fall-back on Detection of a Classic ECN AQM
>>      > <https://arxiv.org/pdf/1911.00710.pdf#page9> (the link takes you
>>     to Fig
>>      > 2, which illustrates the hack)
>>      > I'd rather not have to do these sorts of hacks - it really messes
>>     with
>>      > the segmentation code.
>>
>>     This is very useful feedback as well, and a nice coincidence!
>>
>>     Regarding the hack, sending an old byte may be even worse when there
>> is
>>     not a next data segment readily available for sending, which would
>>     lead to
>>     sending a whole packet just to carry an old byte.
>>
>>      >       Middlebox traversal of bit 6
>>      >
>>      > I recall that someone did a study of traversal of each of the
>>     reserved
>>      > flags (Marcelo Bagnulo maybe?). I don't think it was that good
>> [Bob
>>      > adds: I mean traversal - not the study!].
>>      > A good search engine might find it ;)
>>
>>     I wasn't able to find the study, but this is an important
>>     consideration..
>>
>>     One might think that if new functionality is defined and
>> standardized,
>>     related middlebox traversal of it should improve over time. Perhaps
>> that
>>     might be optimistic...
>>
>>      >       More alternative approaches:
>>      >
>>      > 1/ At one stage, we tried to include a bit for Delayed Ack
>> control in
>>      > another protocol called Accurate ECN. We were persuaded to take
>>     it out,
>>      > because it was mission creep (outside the scope of the protocol
>>     we were
>>      > working on):
>>      >
>>     https://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-03#appendix-B.4
>>
>>     I was unaware of this, thanks!
>>
>>     This is actually very aligned with what we are proposing...
>>
>>      > 2/ Have you considered using the Urgent Pointer in a novel way?
>>     I.e. a
>>      > non-zero Urgent Pointer field, even tho the URG flag is zero.
>>      > [Bob adds:] From memory, traversal was pretty good - much better
>> than
>>      > bits 4-6. But I think Windows treated it as a potential attack. I
>>      > believe Fernando Gont did a study on how 'invalid' TCP header
>>     fields are
>>      > handled.
>>      >
>>      > You could assign, say, 3 bits of the urgent pointer for a
>> variable
>>      > ack_exp that is log base 2 of the ack ratio the sender would
>>     like. Then
>>      > the sender can request the receiver uses ack_ratio` = 2^ack_exp,
>>     as in
>>      > the QUIC proposal below,
>>      > With ack_exp = 0, you get ack_ratio = 2^0 = 1, which has the same
>>     effect
>>      > as ack-pull. But you also have a more general facility for high
>> speed
>>      > machines to widen out the ack ratio.
>>      >
>>      > If this works, you could open up a registry for the other 13
>> bits. So
>>      > instead of using up a precious bit, you generate 13 more ;)
>>
>>     Nice! :)
>>
>>     Actually, we had not considered such an idea.
>>
>>     In your proposal above, if a segment has the URG flag set to 1,
>>     would then
>>     the 3 ack_exp bits still be used as ack_exp bits, thus reducing the
>> size
>>     of the urgent pointer in practice to 13 bits?
>>
>>      > See:
>>      > https://tools.ietf.org/agenda/90/slides/slides-90-tcpm-10.pdf#page=5
>>      >
>>      > I think there were problems with some implementations and
>> middleboxes
>>      > (bound to be). I've checked the minutes of the IETF meeting where
>>     that
>>      > slide was presented
>>      > <https://www.ietf.org/proceedings/90/minutes/minutes-90-tcpm>,
>> but
>>      > there's no push-back in there. I suggest you delve back into the
>>      > archives of the tcpm mailing list around the time of that meeting
>> to
>>      > find out if there were any show-stoppers.
>>
>>     Thanks for the pointers!
>>
>>     Sure, we'll check the tcpm mailing list archive.
>>
>>     Cheers,
>>
>>     Carles
>>
>>
>>      > Cheers
>>      >
>>      >
>>      >
>>      > Bob
>>      >
>>      >
>>      > Paste from QUIC thread:
>>      >
>>     /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
>>      >
>>      >
>>      >
>>      >       Problem/goals:
>>      >
>>      >  1. Getting up to speed fast without inducing much queue is one
>>     of the
>>      >     main areas where latency reductions are needed (for short
>> and
>>     long
>>      >     flows). The sender needs frequent ACKing during this phase
>> in
>>     many
>>      >     approaches (hystart, etc). A Linux TCP receiver starts
>> with
>>      >     ack_ratio 1 and uses heuristics to identify the end of the
>>     sender's
>>      >     slow-start (or a re-start after idle). This has ossified a
>>     certain
>>      >     slow-start behaviour into all TCP senders (whether Linux
>> or
>>     not). We
>>      >     are trying to new sender behaviours (paced chirping being
>> an
>>      >     example), but we need a way for the sender to suppress
>>     delayed ACKs.
>>      >  2. At the other end of the scale, for long-running flows, many
>>     current
>>      >     CC approaches (e.g. BBR) use pacing not ACK-clocking so
>> they
>>     can use
>>      >     very few ACKs per RTT. And fewer means less cache
>> ejections
>>     for GRO.
>>      >     But the receiver doesn't know what CC the sender is using,
>> so it
>>      >     doesn't know how many or few is too many or too few.
>>      >  3. Some middleboxes and the majority of link technologies (e.g.
>>     DOCSIS,
>>      >     LTE, Satellite) thin TCP ACKs when they detect the
>> upstream is
>>      >     filled with TCP ACK stream(s), which are unresponsive.
>>     Otherwise the
>>      >     ACKs constrain downstream throughput (and any upstream
>> data
>>     either
>>      >     in the flow itself, or in others). We don't want these
>> links to
>>      >     attempt to guess which are the QUIC ACKs and try to thin
>>     them. QUIC
>>      >     can and should do this itself. QUIC has all the machinery
>> for the
>>      >     sender CC to detect ACK congestion, but not the protocol
>> to
>>     tell the
>>      >     receiver to do the thinning. This protocol addition would
>>     provide a
>>      >     sufficient hook for hosts to unilaterally add this to
>> their CC
>>      >     behaviour.
>>      >
>>      >
>>      >       Some scenarios where the sender's preferred ack_ratio
>> changes
>>      >       through the connection:
>>      >
>>      >  1.
>>      >
>>      >     A sender CC that wants the receiver to turn off DelAcks
>> during
>>      >     flow-start (e.g. it's using hybrid slow-start as in Cubic
>> and
>>     wants
>>      >     to get delay measurements more frequently) sets ack_exp=0
>> during
>>      >     flow-start (ack_ratio=1), then increases ack_exp during
>>     congestion
>>      >     avoidance. If it goes idle, then re-starts, it would set
>>     ack_exp=0
>>      >     again.
>>      >
>>      >     Note on heuristics: A Linux TCP receiver currently uses a
>>     heuristic
>>      >     to determine when the sender has exited slow-start.
>> However,
>>      >     heuristics -> ossification. A Linux receiver's heuristic
>> only
>>     works
>>      >     with the current pattern of slow-start. In TCP, when we
>> tried to
>>      >     improve the pattern on the sender (paced chirping), the
>>     heuristic on
>>      >     the receiver killed us.
>>      >
>>      >  2.
>>      >
>>      >     Imagine a paced sender has hardware generic receive
>> offload
>>     (GRO),
>>      >     so for a long-running flow it doesn't want a high rate of
>>     QUIC ACKs
>>      >     that are opaque to GRO. Let's say it would prefer at least
>> 8 ACKs
>>      >     per RTT. Again, it starts with ack_exp=0, but in
>> congestion
>>      >     avoidance it would use:
>>      >
>>      >     ack_ratio <= cwnd_in_packets/8
>>      >
>>      > Upshot: By remote controlling the receiver, the server offloads
>>     nearly
>>      > all the ACKs from large downloads, but still focuses its
>>     ACK-receiving
>>      > resources on getting each client up to speed.
>>      >
>>      > Note that calculation of ack_ratio lends itself to fast integer
>>      > arithmetic.
>>      >
>>      >
>>      >
>>      >
>>      >
>>      >
>>      >
>>      > Bob
>>      >
>>      >
>>      > --
>>      > ________________________________________________________________
>>      > Bob Briscoehttp://bobbriscoe.net/ <http://bobbriscoe.net/>
>>      >
>>      >
>>
>>
>>
>> _______________________________________________
>> tcpm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
>