Re: [re-ECN] RE-ECN without ECN.

[Matt Mathis <[e-mail]>]

2009/10/2 Bob Briscoe <[e-mail]>uk>:
> Matt,
>
> At 22:37 01/10/2009, João Taveira Araújo wrote:
>>
>> Inline
>>
>> Matt Mathis wrote:
>>>
>>> Here is a fun little tidbit for people to think about, while others
>>> resolve some of the non-technical issues surrounding launching the
>>> conex.
>>>
>>> For all modern reliable protocols over non-ECN networks (where loss is
>>> the primary congestion indicator), the amount of duplicate data is a
>>> direct measure of the down stream congestion.    (e.g. If I observe
>>> both a data packet and its retransmission, then the first transmission
>>> was lost someplace downstream).    This is because for all modern
>>> protocols, the sender almost never sends data that the receiver
>>> already holds.
>>>
>>
>> Except it's not transport oblivious, so you'd have to a) redesign the
>> solution for every new transport protocol or b) expect a single transport
>> protocol to be all-encompassing. Won't nitpick since this is a thought
>> experiment though...

Joao,
Which do you think is harder: adding duplicate data detection for a
new transport protocol to the ingress router or to get a (potentially
hostile) new transport to carry the ECN information on its ACKs?  (See
the ECI field for TCP Section 6.1.1 in -re-ecn-tcp- )   My bet that it
is a lot easier to add the duplicate detection to the ingress router,
since the owner is motivated to do so.

>>> If you add the assumption that the network itself does not reorder
>>> packets, then the number of out-of-order packets is a direct measure
>>> of the upstream congestion.  (e.g. the first transmission was lost,
>>> and I see the retransmission).
>
> I think you mean a measure of e2e congestion, not upstream.

Ok, It depends on how you define out-of-order when the packet is
duplicate.   Yea, perhaps your definition is better.

>>> This is sufficient information to implement many of the proposed uses
>>> for RE-ECN marking.
>
> It took me a while to unpick the subject line, but I understand now. You
> mean the network viewing e2e congestion with neither ECN nor re-ECN
> protocols.

Correct.

> A related reference (viewing e2e congestion with only ECN):
> [Purple] Pletka, R., Waldvogel, M., and S. Mannal, “PURPLE: Predictive
> Active Queue Management Utilizing Congestion Information,” Proc. Local
> Computer Networks (LCN 2003) , October 2003.

I will look it up.

>>> For example, the bottleneck for problematic flows can be moved
>>> upstream so that they are less disruptive to other users, by
>>> throttling the amount of duplicate data permitted through an ingress
>>> router.  In the extreme case, when a particular flow exceeds its quota
>>> for duplicate data, you drop the retransmissions which will
>>> (eventually) cause it to run out of receive window and take a timeout.
>
> We ought to try to get consensus on whether network-based policing of the
> amount of congestion per flow is desirable. It's not a goal I think we
> should aim for - it enforces a transport monoculture completely
> unnecessarily, as it's surely neither needed nor effective for any
> 'reasonable' form of fairness.
>
> Flow policing was indeed a use proposed for re-ECN (in our original
> SIGCOMM'05 paper). But we were conflicted over whether to include it. We
> ended up highliting it to pander to the SIGCOMM committee - I now wish we
> hadn't. Since then, we've written "The Case Against Bottleneck Policing"
> (S.3.1.2 of draft-briscoe-tsvwg-re-ecn-tcp-motivation-01) and I now find the
> whole idea of flow policing unneeded and actually harmful.
>
> But let's not throw out the baby with the bathwater. There may be something
> in what you're saying, even if we disagree how it might be used.

I misspoke - although the duplicate detection has to be done per flow
(since it requires per-flow state) the enforcement can be done on any
aggregate.   These two functions could even be done in different
places, if for example the duplicate data detection was used to apply
a pseudo-re-feedback mark (DSCP codepoint?) that was enforced further
downstream.   This also permits the duplicate detection to be
parallelized across multiple processors (so flow state becomes less of
a problem) and the aggregate congestion to be limited after the
streams from the different duplicate detectors are merged.

>>>  More polite would be to pass the retransmission, but then discard the
>>> next in-order packet on the same flow, normally causing it to slow
>>> down.  There are some details to be worked out...

 BTW Discarding excessive retransmissions clearly can enforce a hard
policy on an aggregate: e.g. no more than x% duplicate data.  However,
from the users point of view, this is a bad thing because losing the
same segment twice most often causes timeouts, and has the effect of
greatly increasing the variance of the application performance, as
seen by the user.    One would expect that discarding a future segment
to be better behaved from the users perspective, but it requires flow
state (or at least flow matching) at the point where the dropping is
happening.  My sense is that this also has a RED-like tuning problem
(the dropper has to guess what loss rate is required to attain a
particular data rate or congestion rate).  If the duplicate data
detection and enforcement are separated, then there is the additional
problem that you would like to exclude the enforcement drops and
subsequent retransmissions from the down stream statistics being
monitored by the enforcement.

I still don't understand the RE-ECN analogue of this latter problem.
If some aggregate is causing too much congestion, how can the network
throttle it, without causing it to exhibit even more congestion?

> I'm concerned that using a feature that a protocol exhibits when it is
> behaving (TCP retransmissions) in order to police misbehaviour of another
> aspect of the protocol (too much congestion) requires us to suspend too much
> disbelief. If I'm going to modify the protocol to misbehave in one way, I
> can modify the retransmission semantics too.

Actually it is extracting information from TCP's mechanism to
implement reliable data delivery.  It is a tautology that downstream
losses cause reliable protocols send duplicate data.   My proposed
mechanism would even work if the transport protocol was not responding
to losses as congestion signals, as long as it continued to implement
reliable delivery.

> Rather than per flow policing, a better framing might be bulk policing over
> time. Targeting misbehaviour of single flows presumes someone has modified
> the code. Whereas targeting excessive congestion caused by excessive use of
> unmodified TCP(s) is lower hanging fruit. Whether for excessive amounts of
> data over time, or excessive numbers of flows all at once.

Yea, I'd like to understand what "bulk policing over time" might mean.
  One of the big pieces missing from "per flow fairness" schemes is a
reasonable way to make them average over long time periods.

> Certainly apps or users can work round a re-transmission-based bulk policer
> by not using TCP, but they then have to hack an alternative that still does
> all the reliable delivery (or use IPsec).
>
>>> The point being that for non-encrypted standard transports, many
>>> RE-ECN like things can be implemented unilaterally, without global
>>> deployment of complicated, subtitle and hard to verify new protocol
>>> features.
>
> I know what you're trying to achieve in terms of unlilateral deployment.
> However, I think its characterisation of re-ECN is a little uncharitable.

Sorry, I was a little harsh.

> i) The point of the re-ECN design was that one network can deploy
> unilaterally without waiting for others, and one stack can deploy
> unilaterally without waiting for others. But you're right that not much can
> be achieved until a significant number of both have moved.
> ii) re-ECN is designed for very simple verification (balance of two
> counters).
> iii) I don't see complicated. I see simplicity relative to the complexity of
> any of the things it could replace, and the things that can't even be done
> at all any other way.

Consider pseudo-re-feedback as a possible deployment strategy for
RE-ECN.   One ISP can implement it unilaterally, and as real RE-ECN
gets deployed, the instrumentation and controls become more accurate.

> True, there might be simpler schemes that don't work, or can be bypassed.
> But network operators don't want the complexity of deploying many schemes
> one after the other as each one fails (they are driven to behave like this
> for lack of good solutions, but they don't want to).

Precisely the question I wanted to read for - what are the cases that
RE-ECN covers that pseudo-re-feedback can't address.

> Bob
>
>
>> You'd be missing one of the most important features however - that
>> information is verifiable across borders by both parties. An ISP can
>> throttle the duped packets, but why would it voluntarily hinder its
>> customers when the downstream transit domain is unable to verify whether
>> such enforcement is taking place?

I was actually thinking of the last ISP, peering with many "wholesale
ISPs" and having congested access networks feeding millions of  home
customers.   It would greatly help such an ISP to be able to discard
just enough traffic at it's ingresses from other ISP's to limit
congestion on its access links to its customers.   The whole problem
and my solution fall within a single ISP.

>> Congestion may come from within the ISP itself, but I'm not certain the
>> ISP can reliably infer the location of such congestion at the ingress.
>> Something like multipath TCP would also make it very hard to figure out
>> whether or not a packet is retransmitted ... which would make a great
>> incentive for deploying  mtcp.

I'm sort of missing the point here, but I think some of the problems
that you are concerned about are the same for RE-ECN and others are
solved by separating  duplicate detection from enforcement.

>> Joao.

Thanks,
--MM--