Re: [re-ECN] implementations

[Bob Briscoe <[e-mail]>]

Don,

One motivation I always highlight for using ECN rather than drop to 
indicate congestion is that it breaks the link between congestion and 
impairment.

Because ECN keeps queues short, it largely removes jitter & loss. It 
can then be used as an app-independent measure of *approaching* 
congestion. You then get a QoS meshanism for free, because you've got 
rid of delay & loss.

That's not to say there's not value in trying to manage traffic using 
contribution to loss rather than ECN. However, you have to be careful 
how you sell it - it makes sense to say you're capping contribution 
to loss, but if not carefully explained customers think it sounds 
perverse to pay for losses.

Queries/disagreements with stuff below (long); inline...

At 19:41 23/10/2009, Don Bowman wrote:
>From: Matt Mathis [mailto:[e-mail]]
>
> >Please explain "We've found packet loss to be unreliable means
> >of measuring congestion due to TCP rate control."  How do you define
> >and measure congestion?
>
>For the definition:
>
>Congestion is a function of the buffering in networking equipment. 
>When there is contention for an output link (i.e. instantaneously 
>more packets want to be transmitted than there is capacity for), the 
>buffer starts to fill. This buffering increases and adds variability 
>to latency, and thus can create quality problems for interactive 
>applications. As the buffers increase in depth eventually they 
>overflow and cause packet loss. Packet loss is a normal part of a 
>network since it is the mechanism by which TCP governs its 
>throughput. Thus for the loss part of congestion we would define it 
>as the situation in which an increase in data transmissions results 
>in a proportionally smaller or even a reduction in throughput.

[BB] Would it be precise to define loss-based congestion as the loss 
probability? That's what I would do.

>A non-congested network is one in which the latency (end-to-end 
>delay) is relatively constant, and has little packet loss.
>Congestion is also important to be framed by the user experience. A 
>situation of congestion which causes an instant message to be 
>delivered 500ms later is irrelevant to the average user. The same 
>delay on a gaming packet or VoIP packet is perceived as a full loss 
>by the consumer.
>
>Sandvine would therefore define congestion on a per-application 
>class basis as the variability in delay or packet loss beyond what 
>the application can withstand without the user noticing. The 
>Wikipedia article @ http://en.wikipedia.org/wiki/Congestion_collapse 
>provides a good description based on the assumption that all 
>applications perceive congestion the same.

[BB] We need to unpick two separate aspects: an app both i) suffers 
from the effects of congestion and ii) contributes to that congestion:

== i) Sensitivity to congestion ==
The different sensitivities of different apps to a common congestion 
metric are a property of the app, not the network - they are not 
reasons to re-define congestion itself. Once you have an objective 
congestion measure, the response to it can be down to each app (see 
below for the special case of an app-specific middlebox).

== ii) Contribution to congestion ==
Different apps cause different amounts of congestion per data sent, 
by responding more sensitively (LEDBAT) or less (unresponsive UDP), 
or somewhere in the middle (TCP). And apps on shorter RTTs cause less 
congestion for the same data, because they can get out the way more quickly.
All these differences are choices of the app, not anyone else. So 
it's right to define an objective measure of congestion independent 
of these things. If an app chooses to use a long RTT path thereby 
causing more congestion, making it accountable for its contribution 
to congestion is still the right thing to do.

=== A special case of i) with app-specific middleboxes ===
Admittedly, if a middlebox is trying to do the app's job, then the 
app-specific response will need to be in this middlebox. But within 
the box, the app-specific response should still be separated from the 
non-app-specific congestion metric. At least an objective congestion 
metric gives us an architecture where you don't /have/ to have 
anything app-specific in the network.

If you deploy ECN, even a traffic mgmt middlebox doesn't need an 
app-specific response to different impairments, because there are no 
impairments, only congestion signals. Nonetheless, there will need to 
be policy-induced impairment (policer drop) - if all a user's load is 
unresponsive to the congestion signals, or it's responsive but not 
responsive enough for the amount of it.

== Definitions of congestion ==
Congestion is shared by all packets arriving at a queue at a certain 
time, so it is simpler (& more correct IMHO) to define congestion as 
the instantaneous probability of impairment.

You're right that there are different sorts of impairment, so we need 
to define X-related congestion where
* X = loss for loss-related congestion,
* X = queuing delay for delay-related congestion and
* X = marking for ECN-related congestion (not actually an impairment).

In practice, if Reno is dominant, the prob of loss is closely related 
to the prob of extra delay (because delay precedes loss). But on 
faster links congestion delay is imperceptible before loss occurs.

However, modern end-system responses de-correlate the different 
measures. Delay-sensing congestion controls like LEDBAT or Vegas 
reduce the loss probability relative to the delay probability. And 
ECN reduces delay and loss probability to nearly nothing but of 
course doesn't reduce the ECN probability.

So, ECN is the target and its simple, reducing everything to just one 
dimension of congestion. But while we don't have ECN, it makes sense 
to just use loss as a metric and not worry about the differences 
between delay and loss (why perfect the interim solution?).

>So from this, congestion is the point @ which if an application 
>attempts to go faster, it gets no real return for the attempt, and 
>the quality delivered to the user suffers.

[BB] One could define some variant of congestion 
('Sandvine-congestion'?!) as relative to each app, but I don't think 
it's useful to do so.


>Now, for use of TCP packet loss.
>Since there is more than one TCP stack implementation in use, and 
>they vary in aggressiveness of ramp up/down, and since the network 
>is shared by so many sessions, and since congestion comes at many 
>hops along the way but TCP packet loss @ one location doesn't show 
>where it occurred, you have a blended mix. The loss may be in the 
>wifi in the home, in the home gateway, in the dsl modem, in the 
>dslam, in the bras, atm aggregation, aggregation routers, core 
>routers, peering, transit, etc. It may be due to the user's own 
>traffic, due to unreliable RF signals, due to other users traffic, 
>due to network switchover, etc.

[BB] Yup.

>So when we modelled TCP packet loss and correlated it against 
>congestion measured as % of link utilisation @ each choke point, and 
>eyeballed the quality of some key applications, we found the 
>correlation to be not that great.

[BB] I'm not exactly sure what the second correlation you mention is 
between. I'm assuming quality:loss is somehow compared to 
quality:utilisation.  This might means something if ':' means 
division, but I'm not sure what it means if ':' is the function 'to eyeball'.

Anyway, you could define 'Sandvine-congestion' like this, but it 
wouldn't be as useful as congestion = probability of impairment. Your 
approach assumes:
a) link utilisation is somehow a benchmark measure of 'congestion',
b) 'congestion' is meant to give a deterministic measure of app 
quality for all apps.

These both seem backwards:
b) I've already argued against an app-specific congestion metric at 
item (i) above.
a) Should congestion be defined at a human's timescale, or a 
computer's timescale? Link utilisation is measured over some 
averaging period (say 15mins or 1min or whatever). Congestion control 
algos can respond in msec. And scheduling algorithms can respond in 
nano or femto secs. If network operators are going to get their 
customers to co-operate towards beneficial ends for all, they need to 
treat their customers as machine-mediated customers.

It's most useful to define congestion at the fastest timescale 
possible (probability of impairment at queuing timescales). Then if 
there's a way for a computer to avoid congestion, the operator can 
give it an incentive to do so, by making it accountable for congestion.

For instance, there might be a machine on a short RTT path that can 
delay sending traffic at little detriment to itself (e.g. using 
LEDBAT). Or there might be a machine that can't use a short RTT, but 
it can use a lower Diffserv class instead. If no machine can avoid 
congestion, then that's fine. But if some machines have flexibility 
that can be harnessed, there's no point setting the bar so low that 
they don't use their flexibility.

Ideally, all the TCPs are trying to keep instantaneous link 
utilisation at 100% while there is some traffic to send and 0% 
otherwise. Humans understand utilisation as the average of this 
process - nominally the mark-space ratio between two states (but 
actually the sum of all the slow starts & AIMD sawteeth).

A source's contribution to loss (or marking) gives a much more useful 
instantaneous measure of congestion than some average of utilisation 
over time. And when accumulated over time the amount of loss (or 
marking) gives a useful average measure too.


>A related question relates to the use of policing. If i put a 10Mbps 
>policer on a network that is capable of 100Mbps, I do not see 90Mbps 
>of 'drops' from the policer, i typically see 0. Therefore i cannot 
>infer what the bandwidth need would be if i remove the policer (it 
>might still be 10, it might go to 100). Thus the packet loss cannot 
>be used to infer the bandwidth desired.

[BB] There's two separate issues I have with this:

1/ Whether loss is due to policy or to congestion, bandwidth desired 
isn't relevant to how much harm one user contributes to others. 
That's the bandwidth they actually make do with, weighted by the 
congestion when they use it (ie their contribution to congestion).

2/ If a policer is basing its decisions on contribution to 
congestion, it should have two separate internal processes:
- measuring the congestion a user's traffic causes
- dropping traffic when policy is exceeded

It shouldn't include policy drop in its measurements to decide how 
much policy drop. This is straightforward if it is not congested 
itself, but it's just measuring congestion everywhere else using 
(re-)ECN. But if it is itself the bottleneck, it needs to create a 
virtual bottleneck internally to do the measurement step, before 
introducing the policing drop.

That was a long set of responses, but I hope they are worth it.


Bob


>--don
>_______________________________________________
>re-ECN mailing list
>[e-mail]
>https://www.ietf.org/mailman/listinfo/re-ecn

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design