Re: [tsvwg] Extensive review of draft-ietf-tsvwg-circuit-breaker-05

[Bob Briscoe <[e-mail]>]

Gorry,

Understood.
Sorry for the shot out of the blue. But I missed the WG last call in Aug.


Bob

On 09/10/15 03:55, [e-mail] wrote:
> Thanks for the review.
>
> I can't now respond to this at this time, I wasn't expecting this review
> and will be away. Some parts I agree with, some parts I don't. I'll get to
> it in a couple of weeks, - I may not have answers to a few of the things
> we avoided saying.
>
> Gorry
>
>> Gorry,
>>
>> Despite being past the WG stage, here's my review anyway. Consider this
>> as early response to IETF last-call.
>>
>> In general I support the intent of this draft, but I am concerned at the
>> severity of the problems I have found with it given it is meant to be
>> about to go to the IESG. I am particularly concerned that I have found
>> numerous significant problems with the normative requirements section.
>>
>> Have you had a substantial review from anyone before this? The level of
>> review comments on the tsvwg list seemed quite light - picking on issues
>> of particular concern, but not seeming to review the draft as a whole.
>>
>> *1. Intro: **
>> *Congestion Collapse is a very specific case - CB is much more general.
>> it is clear from the draft that a CB is intended to mitigate
>> circumstances wider than solely the extreme case of congestion collapse.
>> For instance: a large unresponsive aggregate contributing to a high
>> level of congestion alongside congestion responsive traffic. This is
>> nowhere near congestion collapse, but it would be an applicable case for
>> a circuit-breaker. Congestion collapse is a specific well-defined
>> process that involves a cascade of congestion as a sequence of queues
>> fill in turn moving in the upstream direction. It is due to continual
>> retries or additional load arriving faster than existing flows are
>> departing. {Note 1}
>>
>> The introduction mentions that TCP-style cc is only an appropriate
>> remedy when long flows dominate. The implication that CB could be used
>> to deal with congestion induced by many short flows is a step too far,
>> IMO. This problem has not even been discussed in the IETF or IRTF to my
>> knowledge, let alone in the context of this draft. In 6.2 this draft
>> all-but says that a CB is a solution to this problem. I strongly object
>> to a BCP making that assertion. CB would be a very drastic and clumsy
>> solution to that problem.{Note 2}
>>
>> It says that the timescale at which a circuit-breaker operates must be
>> seconds or tens of seconds - much longer than the RTT timescale on which
>> TCP, SCTP and DCCP react. This disregards an important type of
>> application response to congestion; it must say that the timescale also
>> has to be longer than the timescale on which certain real-time
>> applications operate their own circuit-breakers i.e. adapt down their
>> codec rates, and eventually close the connection as a form of
>> self-admission control. Applications operate per-flow circuit-breakers
>> typically over the order of seconds or tens of seconds, so network CBs
>> MUST take longer than that - I would say "no less than a minute".
>>
>> We MUST not discourage voluntary self-regulation by overriding it
>> (end-to-end principle). I pick up this point later (comments on section
>> 51.), arguing that the fast-trip CB for RTP should be considered as an
>> application CB, and a network CB should always take longer to trigger
>> than these app CBs.
>>
>>
>> *1.1 Types of CB**
>> **
>> *I saw criticism on the list of the use of the term "protect" in this
>> section. Why hasn't it been changed? As the posting said, a CB does not
>> protect the aggregate that it monitors; rather it /regulates/ the
>> aggregate to protect the rest of the traffic that it is /not/ monitoring.
>>
>> *3.1 Functional Components.**
>> *
>> There is no mention of the problem of synchronising the ingress and
>> egress measurements to allow for transit time. Given you are trying to
>> measure loss, which is a relatively small difference between the traffic
>> entering and leaving, you can get very bad errors if you don't take path
>> delay into account. draft-ietf-tsvwg-tunnel-congestion-feedback
>> describes a nice (and commonly used) stateless way of doing that, by
>> sending the ingress measurement in-band to the egress, which triggers
>> the egress measurement so they are synchronized; allowing for transit
>> time. Then the egress can send them both back to the ingress to be
>> compared and acted on.
>>
>> *4. Reqs**
>> *
>>
>>         There MUST be a control path from the ingress meter and the egress
>>         meter to the point of measurement.  The Circuit Breaker MUST
>>         trigger if this control path fails.
>>
>> Either this is unclear terminology, or I strongly disagree. What do you
>> mean by a control path? We should only recommend that the CB triggers
>> due to lack of measurement signals if the measurement signals are
>> carried in-band with the data being monitored. That is only one way of
>> arranging the mechanism. The term control path, sounds like it is out of
>> band. If the measurement signals are out of band, the CB MUST NOT
>> trigger due to lack of measurement signals. I would recommend the
>> in-band method, but there are plenty of network designers who will want
>> to do this in centralised out of band ways, so we have to cater for that
>> way of thinking (even tho it's misguided).
>>
>>         The measurement period MUST be longer than the time that current
>>         Congestion Control algorithms need to reduce their rate following
>>         detection of congestion.
>>
>> This needs to be rewritten. Or just removed. It seems like ideas changed
>> after it was written, and the end was changed but not the normative
>> statement at the beginning. IMO, the measurement period can be
>> arbitrarily short, as long as multiple measurements are combined before
>> triggering the CB. It talks about unnecessarily penalizing long RTT
>> flows, but the measurement period is nothing to do with the period
>> before there is any penalization (defined later as the triggering
>> interval). There is no problem with short measurement periods as long as
>> any high congestion measured in these periods is averaged over all the
>> measurement periods in the triggering interval.
>>
>> In fact, there should be many measurement intervals per trigger
>> interval, so that there are many opportunities for measurement messages
>> to get through. Otherwise if there are only one or two measurement
>> periods per trigger interval, the possibility of a false trigger due to
>> lost control signals becomes too great.
>>
>>      o  A Circuit Breaker is REQUIRED to define a threshold to determine
>>         whether the measured congestion is considered excessive.
>>
>>      o  A Circuit Breaker is REQUIRED to define the triggering interval,
>>
>> A perfectly good CB could vary the trigger interval and threshold
>> depending on how rapidly congestion is rising, or how high its absolute
>> level is. Indeed one could say it is actually wrong to define a single
>> threshold or a single interval, so these normative statements are overly
>> restrictive and preclude designs that are smarter than just simple fixed
>> threshold.
>>
>> Also, see comment above about allowing time for application CBs, and
>> suggesting one minute minumum.
>>
>> o  A Circuit Breaker SHOULD be constructed so that it does not
>>         trigger under light or intermittent congestion, with a default
>>         response to a trigger that disables all traffic that contributed
>>         to congestion.
>>
>> The second half after the comma seems misplaced. If it does not trigger,
>> why does the sentence go on to talk about disabling all traffic that
>> contributed to congestion (which is what an /enabled/ trigger would do)?
>>
>> A reaction that results in a reduction SHOULD result in
>>         reducing the traffic by at least a factor of ten,
>>
>> What evidence have you got for this 10% number? It seems utterly
>> inappropriate to write a number here. The number depends on what
>> proportion of the traffic on the path between ingress and egress is
>> regulated by the CB. If the proportion is low, it needs to reduce by a
>> lot to make sufficient space for other traffic. If the proportion is
>> high relative to other traffic, it might be sufficient to reduce by 5%
>> to 95% of the previous load. If the tunnel traffic represented say 80%
>> of the load on the path, and it reduced by a factor of 10, that would
>> leave 92% of the path for other traffic, which might be unnecessarily
>> much greater than the normal proportion used by other traffic.
>>
>>         Manual operator
>>         intervention will usually be required to restore a flow.
>>
>> This sentence should be toned down to possibly, not usually. A human is
>> no more capable than a machine is of bringing together all the necessary
>> measurements to decide what other courses of action might be possible,
>> and when to release the brakes. I suggest the last para of 5.3.1 starting:
>>
>> "An operator-based response provides opportunity..."
>>
>> is more appropriate here, and doesn't really fit where it is.
>>
>> Section 4.1 contains no requirements text, only examples. It ought to be
>> moved from the normative requirements section to section 5 (Examples).
>>
>>
>> *5. Examples:**
>> *
>> *5.1.1 Fast-Trip CB for RTP**
>> *
>> The draft needs to make the distinction between an application doing its
>> own circuit breaking vs. functions on the path between the application
>> endpoints (even if in the hosts) doing CB. The extremely important
>> distinction is:
>> 1a) an app knows when congestion is too high for it to work properly
>> 1b) functions under the app can only infer congestion is possibly too
>> high for most apps to work properly
>> 2a) an app may be able to reduce the rate at which it sends data
>> 2b) a function under an app can only discard data, not remove it at
>> source.
>>
>> I believe that the requirements in section 4 do not apply to
>> application-controlled circuit-breakers. So, I would not include the
>> "Fast-Trip CB for RTP" as an example of a /network/ transport CB.
>>
>> As the requirements say, a network CB should never fast trip.
>> By misclassifying RTP CBs as network CBs, you've allowed the timescale
>> for network CBs to trigger after tens of seconds. When a network CB
>> should allow app CBs this long to trigger themselves (as I said earlier).
>>
>>
>> *Missing examples:**
>> *
>> * You might want to point to the flow termination function (as opposed
>> to admission control) in the PCN architecture [RFC5559], which is
>> precisely a network CB. It was precisely developed for cases where
>> failures caused traffic to reroute onto a previously well-provisioned
>> path (see 6.1).
>> * Andrew McGregor gave the examples of Google's BwE (bandwidth enforcer)
>> and B4, but you haven't referred to them. Given they are documented
>> existence proof of this beast, that seems remiss.
>>
>> *7. Security Consid's**
>> **
>> *
>>
>>      The circuit breaker MUST be designed to be robust to packet loss that
>>      can also be experienced during congestion/overload.
>>
>>
>> This implies reliable transmission - i.e. retransmit for ever until
>> acknowledged. This is NOT a good idea. In
>> ietf-tsvwg-tunnel-congestion-feedback we propose using SCTP partially
>> reliable transport. Then if congestion causes messages to be lost, they
>> don't have to be retransmitted if there are insufficient resources (thus
>> not risking contributing to congestion collapse - and here I use the
>> phrase correctly). Because they transmit counters, the missing counters
>> values do not matter. This is the tried-and-tested message delivery
>> approach used for IPFIX. The messages can still be given priority, but
>> should not be retransmitted.
>>
>>      Simple protection can be provided by using a
>>      randomized source port, or equivalent field in the packet header
>>      (such as the RTP SSRC value and the RTP sequence number) expected not
>>      to be known to an off-path attacker.
>>
>> I think the draft should recommend that for most scenarios, randomized
>> ports will be insufficient protection for CB control messages, which
>> should be properly crytographically authenticated. Otherwise, a
>> CB-controlled aggregate is too vulnerable to these off-path attacks.
>>
>> *Gap #1:**
>> ***The draft seems to think it is so obvious what a CB should measure
>> that it only says it vaguely as "the level of congestion", and only
>> suggests the difference between ingress and egress counters as an
>> example. Some readers might well think like this: Does congestion level
>> mean the percentage extra bit-rate relative to the aggregate's expected
>> or maximum bit-rate? That might actually be a correct measure of
>> congestion in some scenarios, but...
>>
>> The draft does not say that the congestion level is defined as dropped
>> bytes divided by ingress bytes. The draft should spell out that a CB
>> should measure the volume of bytes dropped and the volume of ECN-capable
>> bytes marked with CE, and express these as a fraction of resp. total
>> ingress non-ECT bytes and total ingress ECT bytes (assuming buffers
>> within the scope of the CB are ECN-enabled). Even this is problematic,
>> because the assumption in parentheses never holds, particularly during
>> excessive congestion. It could also discuss the relative merit of
>> measuring the percentage of packets dropped/marked instead of bytes.
>>
>> Also it should mention that care should be taken over how to combine the
>> measurements. For instance avoid the common mistake of averaging
>> fractions, because ave(c1/t1, c2/t2, c3/t3 ...) != (c1 + c2 + c3)/(t1 +
>> t2 + t3).
>>
>> *Gap #2:**
>> ***All the diags show multiple routers, but the text says congestion can
>> be measured by comparing ingress and egress traffic. Nowhere does it say
>> that only traffic with addressing that will have for-certain only passed
>> through both ends should be measured.
>>
>>
>> {Note 1}: A few years ago I dug deep into the history surrounding the
>> early congestion collapses on the Internet and found that those involved
>> were adamant that the term congestion collapse should not be waved
>> around for dramatic effect, because it has a very specific definition,
>> as paraphrased above.
>>
>> {Note 2}: The credit feature of ConEx was intended to address short-flow
>> overload if it becomes a problem. DOn't get me wrong; I'm not objecting
>> to the use of CBs for the short-flow problem because I want you to use
>> my solution. I'm just using this as an example of a fine-grained way to
>> solve the problem, rather than the sledge-hammer CB way.
>> Here's the intuition briefly: With ConEx, you have to attach 'congestion
>> credit' to the first packets of a flow to cover the risk of congestion
>> before you have feedback (and if you don't and there is congestion, your
>> packets are dropped by an audit function). Then congestion policers at
>> the network ingress can limit the amount of congestion credit consumed
>> without needing feedback, and thin out traffic if it consists of large
>> numbers of short flows. If short flows come to predominate, ConEx credit
>> was also designed to incentivize a new form of proxy that could regulate
>> short-flows with a push-back style of congestion control, without a full
>> feedback loop. That would be far preferable to such a drastic measure as
>> a circuit-breaker. This aspect of ConEx was not written into the IETF
>> docs, but it is mentioned in the re-ECN drafts that were the ancestors
>> of ConEx.
>>
>>
>>
>> *Nits**
>> *
>> 3.
>> s/last resort protection to the network paths that these are used./
>>    /last resort protection to the traffic sharing their network path./
>>
>> s/tunnels encapsulations/
>>    /tunnel encapsulations/
>>
>> 3. What makes a good CB?
>>
>>      Circuit Breakers are RECOMMENDED for IETF protocols and tunnels that
>>      carry non-congestion-controlled Internet flows and for traffic
>>      aggregates, e.g., traffic sent using a network tunnel.
>>
>> Delete "
>>
>> e.g., traffic sent using a network tunnel
>>
>> "
>> Reason: this implies all network tunnels are problematic, whereas the
>> rest of the sentence adequately says that only tunnels carrying
>> non-congestion controlled flows are of concern.
>>
>> 4.
>>
>> s/monitor the level congestion/
>>    /monitor the level of congestion/
>>
>> 4.1.1
>> (e.g. to implement a Section 5.1)
>> ?
>>
>> 4.1.2
>> s/pre-prosvisioned/
>>    /pre-provisioned/
>>
>> 6.1
>>
>>      One common question is whether a Circuit Breaker is needed when a
>>      tunnel is deployed in a private network with pre-provisioned
>>      capacity?
>>
>> Remove '?' from the end.
>>
>> 6.2
>>
>> s/in the event that persistent congestion occur./
>>    /in the event that persistent congestion occurs./
>>
>>
>>
>> Regards
>>
>>
>>
>> Bob
>>
>>
>> On 08/10/15 12:47, Gorry Fairhurst wrote:
>>>> [Gorry, I also have to deliver on my promise on a paragraph for
>>>> circuit-breaker. Do you have a deadline for that?]
>>>>
>>> The circuit-breaker ID is pending start of IETF last call, the
>>> deadline for doing an author rev passed, sorry.
>>>
>>> --
>>> ________________________________________________________________
>>> Bob Briscoe                               http://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/