Re: [tsvwg] draft-ietf-tsvwg-circuit-breaker-05 & draft-ietf-tsvwg-tunnel-congestion-feedback-00

[Bob Briscoe <[e-mail]>]

David,

On 04/11/15 22:48, Black, David wrote:
> Re: Response to the detailed review: draft-ietf-tsvwg-circuit-breaker-05
>
> Bob,
>
> As draft shepherd, let me start with the draft relationship concern:
>
> > *2. Scope: Measurement mechanism vs. response behaviour
> *
> > Secondly, there is a huge question-mark over the scope of the draft. 
> It ranges over the same territory as another draft already
>
> > adopted as tsvwg chartered work:
> >    draft-ietf-tsvwg-tunnel-congestion-feedback
> > The circuit-breaker draft does not even reference 
> tunnel-congestion-feedback. However, the two drafts should be totally 
> complementary.
>
> I believe the drafts are complementary, but I think the relationship
>
> between the drafts is:
>
> a) The tunnel congestion feedback draft is a measurement mechanism draft.
>
> Those measurements can be used for multiple purposes.
>
> b) One of those multiple purposes is to feed a circuit breaker of some
>
> sort. That is one of many possible uses of those measurements, and
>
> definitely *not* the only use.  I believe that prior discussion of
>
> the tunnel congestion feedback draft is consistent with this (i.e.,
>
> it is not solely for circuit breakers).
>
> c) The circuit breaker draft is about how to use measurements that may
>
> or may not come from the tunnel congestion feedback draft (in particular
>
> that’s unlikely to be used for a Managed Circuit Breaker) for a specific
>
> purpose. All of the measurements involved can be used  for other purposes.
>
I'm not sure why the managed aspect in particular makes 
tunnel-congestion-feedback inapplicable, but whatever, I agree that 
there will be other f/b mechanisms, e.g. in the PWE header.

So I agree with all the above so far.
>
> So, I think reducing the scope of the circuit breaker draft to exclude
>
> all mention of measurements is in appropriate - as a consumer of 
> measurements
>
> (item c), it seems appropriate for the circuit breaker draft to discuss
>
> the necessary and desirable characteristics of those measurements.
>
OK, that makes sense.

My point was more that the two drafts should be aligned in their 
requirements for the feedback.

tunnel-congestion-feedback contains a number of important requirements 
that either ought to be in CB, or CB ought to reference (or if there is 
disagreement, it should be resolved):
* need to ensure measurements of bytes in and bytes out are synched, 
otherwise incorrect calc of a small difference between two large numbers 
can false trigger CB or false fail to trigger when it should [CB says 
"could be synched", but I think it's much more important than that.]
* soft-state measurement (e.g. in case tunnel end-points move, e.g. 
tunnel from a train of all passenger sessions) [tunnel-congestion-fb 
gives a simple soft-state mechanism example]
* IPFIX style counter messages so that, if any messages are lost, the 
counters remain robust.
* SCTCP-style partial reliability for (in--band) measurement traffic, so 
that if congestion increases, the measurement traffic automatically 
thins, rather than contributing more to the problem.

I believe the last point (partial reliability) is controversial - it 
depends both on relative sizes and relative importance of measurements 
vs data, so both drafts ought to reflect whatever consensus arises on that.

> In my “copious spare time” this morning, I will put a/b/c above onto a
>
> slide for discussion in the tsvwg meeting today.
>
> > There is material in circuit-breaker about the measurement and 
> feedback mechanism that contradicts that in tunnel-congestion-feedback
>
> Please cite specific text that needs attention, preferably with respect
>
> to the above a/b/c relationship.  There is more work needed on the
>
> circuit breaker draft for other reasons, some of which were discussed
>
> in the meeting Monday, so there is an opportunity to revise text.
>
OK. I've given specifics are above.


Bob
>
> Thanks,
> --David
>
> *From:*Bob Briscoe [mailto:[e-mail]]
> *Sent:* Sunday, November 01, 2015 8:27 PM
> *To:* [e-mail]; [e-mail]; Black, David
> *Subject:* Re: Response to the detailed review: 
> draft-ietf-tsvwg-circuit-breaker-05
>
> Gorry,
>
> I've been taking a step back since I reviewed the detail of this 
> draft. I was carried along by the narrative of the draft, and didn't 
> think about the draft as a whole.
>
> I think there was also an ambiguity about the definition of 'flow' in 
> the draft, that I took one way (=microflow). Rereading, I think you 
> mostly meant the other way (=(aggregate OR microflow)). In the latter 
> case I would be very negative about the draft (whereas before I said I 
> was supportive). In the following, I use "flow-termination" with the 
> former meaning; I use "circuit-breaker" with the latter definition.
>
> *1. Circuit-breaking of real-time traffic without regard to microflows 
> considered (very) harmful
> *
> 1a. The main problem: the transport area sends out totally the wrong 
> message if it offers a network circuit-breaker as its apparent 
> recommended remedy for persistent congestion. The IETF transport area 
> has developed a wide range of remedies to this problem over the years, 
> and one of them will nearly always be better than a circuit breaker, 
> where the phrase "nearly always" leaves an extremely small exception 
> space.
>
> Why is the first thing that tsvwg has chosen to expedite, and give the 
> status of BCP, solely about this tiny exception space?
>
> 1b. Over the last fifteen years I have researched, designed, built, 
> evaluated and standardised numerous network-operator controlled 
> mechanisms for limiting congestion in networks. Below I outline a 
> number of ways of dealing with persistent congestion while maximizing 
> the satisfaction of everyone. These have been developed in the IETF 
> over the years. Then suddenly we get a draft saying, "it's OK to cut 
> off the whole tunnel if you have to, or at least a large fraction of 
> it," without saying that it will hardly ever be appropriate to take 
> this approach.
>
> I've realised that this is verging on vandalism: promotion of 
> vandalism of customer traffic, and vandalism of all the hard work of 
> dozens/hundreds of people over the years to address these problems in 
> a more considered way.
>
>   * That work started with pre-congestion notification (PCN) for 
> unresponsive flows, keeping per-flow operations to the edge of a 
> network. That included flow admission control and flow termination.
>
>   * We then developed the congestion policer to work at an aggregate 
> level, dealing with a mix of unresponsive and responsive traffic in 
> whole networks, or in tunnels or at single bottlenecks. It was 
> designed to cause the least disruption to each of the flows, when it 
> is impractical to police each flow separately. A congestion policer 
> deals effectively with persistent congestion, whether caused by 
> unresponsive flows, or an excess of responsive flow arrivals, or both. 
> As congestion rises, i) initially it gives responsive flows more drop 
> signals, while only removing a small fraction of unresponsive traffic, 
> so the latter should still be able to continue; ii) if congestion 
> persists, it removes larger and larger fractions of the aggregate, 
> irrespective of whether flows are responsive or unresponsive. iii) It 
> can be complemented by well-known techniques from the 1990s to 
> randomly identify heavy flows to give a compromise between per-flow 
> and aggregate processing.
>
>   * The draft cites the work of Yaakov Stein, David Black and myself 
> <draft-ietf-pals-congcons>, which the authors and WG have finally 
> reached consensus on (after years). Your draft is correct that it 
> proves that typical pseudowires for TDM traffic will become useless in 
> themselves before they consume more capacity than responsive traffic. 
> And the draft refers to this circuit-breaker as a possible solution. 
> But that depends how you define circuit-breaker:
>    - If it indiscriminately removes packets from the pseudowire 
> aggregate, it is /not/ the best solution at all.
>    - The best solution is briefly described in the pals draft 
> (briefly, 'cos remedies were out of scope): removal of inactive voice 
> channels, and admission control of individual flows from the 
> pseudowire. Failing that, the PW should remove random individual flows 
> until congestion is reduced sufficiently.
>    - Indiscriminate removal of packets is only appropriate, if the 
> network cannot see the microflows (e.g. due to encryption at the 
> aggregate level).
>
> If circuit-breaking is defined as per-aggregate not per-microflow, 
> then this draft should not recommend circuit-breaking. It should 
> recommend flow admission control and flow termination, not 
> circuit-breaking.
>
> Cutting off any indiscriminate proportion of the packets without 
> regard to microflows is equivalent to cutting off every microflow 
> within the  pseudowire, because real-time flows typically cannot 
> survive with loss levels above a few %. Cutting off the whole 
> pseudowire in this way is vandalism, and only appropriate if the 
> operator is a lazy good-for-nothing outfit that doesn't deserve any 
> customers.
>
> 1c. One remedy is to start this draft by saying :
>
> ***The idea of cutting down unresponsive traffic from a tunnel without 
> regard to microflows will rarely if ever be appropriate.***
>
> If a circuit-breaker is defined to include removing traffic without 
> regard to microflows, then the scope of applicability of this 
> circuit-breaker draft is tiny, and it should be rewritten to reflect 
> that. Ie. the next sentence might as well say "Therefore, don't read 
> this draft. There will nearly always be less harmful ways of 
> addressing persistent tunnel congestion, many of which are already 
> written up in RFCs or drafts."
>
> *2. Scope: Measurement mechanism vs. response behaviour
> *
> Secondly, there is a huge question-mark over the scope of the draft. 
> It ranges over the same territory as another draft already adopted as 
> tsvwg chartered work:
>     draft-ietf-tsvwg-tunnel-congestion-feedback
> The circuit-breaker draft does not even reference 
> tunnel-congestion-feedback. However, the two drafts should be totally 
> complementary. I.e.:
>    a) tsvwg-tunnel-congestion-feedback specifies a generic way to get 
> feedback from the egress of a tunnel to a decision point (which can be 
> the tunnel ingress or centralised entity), and it provides mechanism 
> for resilience of the feedback messages, ability to control timeliness 
> vs feedback message overhead, etc.
>    b) tsvwg-circuit-breaker should be limited in scope to solely 
> specifying one of the possible behaviours in response to congestion 
> measured over the tunnel: i.e.breaking the circuit
>    c) other drafts can specify other responses, e.g.:
>       * aggregate techniques:
>         -  provisioning additional capacity
>         - re-routing part of the load (traffic engineering)
>         - rate policing
>         - congestion policing {Notes 1, 2, 3}
>       * per-flow techniques
>         - flow admission control
>         - flow rate policing
>         - flow congestion policing
>         - flow termination
>
> There is material in circuit-breaker about the measurement and 
> feedback mechanism that contradicts that in 
> tunnel-congestion-feedback. Instead of writing two contradictory 
> standards, the WG should agree the division of scope between the two 
> drafts, and the superset of all the authors and reviewers should work 
> towards a single standard for each scope.
>
> The chairs of a WG are entitled to assign authors (including 
> themselves) to a chartered work item to ensure it is well-written, 
> technically sound and moves along in a timely manner.
>
> *Notes
> *
> {Note 1} when the conex WG completed, the responsible AD (Martin S) 
> suggested that drafts like draft-briscoe-conex-policing might be 
> picked up by tsvwg, which would be appropriate, because there is 
> nothing specific to conex in that draft - it assumes some mechanism is 
> getting information about congestion to the ingress of the network, 
> which might be conex or draft-ietf-tsvwg-tunnel-congestion-feedback.
>
> {Note 2} draft-briscoe-conex-data-centre specifies  the same mechanism 
> as draft-ietf-tsvwg-tunnel-congestion-feedback. Again, even tho it has 
> conex in the filename, it was written to give a general tunnel 
> congestion feedback mechanism where hosts do not support conex (and to 
> interwork with hosts that do). It focuses on a data centre deployment 
> scenario, but there is nothing in it that prevents it being applicable 
> in general networks.
>
> {Note 3} For those not familiar with a congestion policer, it is 
> essentially a token bucket, but rather than the tokens representing an 
> allowance to forward bytes, they represent an allowance to cause 
> congested bytes (ie loss or ECN marking).
> For the case of congestion feedback across a tunnel, when feedback 
> about loss (or ECN) comes back across the tunnel, it drains that 
> amount of bytes from the bucket (where intermittent feedback messages 
> are envisaged, token draining has to be averaged over the next 
> measurement interval). The closer the bucket is to empty the more it 
> blocks packets entering the ingress.
> This has the effect of thinning the ingress traffic so that the 
> congestion level is just below the defined threshold.
>
>
> *_Further detailed review comments
> _*
> *Abstract*
>
>       ... network tunnels, and other non-congestion controlled
>     applications
>
> I think you mean
>     non-congestion controlled network tunnels
>
> Reasoning: as it stands, this implies tunnels are always a type of 
> non-congestion controlled application, which is clearly not intended.
>
> *Introduction
> *
> FIrst para: the term 'flow' needs to be disambiguated here, not just 
> later. Does it mean microflow, or aggregate? I support this draft if 
> it means the former. but not if the latter.
>
>     It was countered by the requirement to use congestion
>     control (CC) by the Transmission Control Protocol (TCP) [Jacobsen88 
> <https://tools.ietf.org/html/draft-ietf-tsvwg-circuit-breaker-07#ref-Jacobsen88>]
>     [RFC1112 <https://tools.ietf.org/html/rfc1112>].
>
> There was no requirement in RFC1112 to use TCP or congestion control.
>
>     People have been implementing what this
>     draft characterizes as circuit breakers on an ad hoc basis
>
> This needs references.
>
>      ...either by disabling the flow or by significantly reducing
>     the level of traffic.
>
> Again, pls disambiguate "flow". I supported this drdaft, because I 
> thought it meant microflow. If it means aggregate, I don't support the 
> draft.
>
>    reflects a fair use of the available capacity
> This new text added in the latest draft you sent me is problematic. It 
> will need to say "according to the policy of the operator of the 
> circuit-breaker, not the IETF".
>
> Two more comments inline...
>
> On 17/10/15 12:32, [e-mail] <mailto:[e-mail]> 
> wrote:
>
> Sorry I've taken so long to reply.
>
>     Review by Bob Briscoe: 8th Oct 2015
>
>     Gorry,
>
>     Despite being past the WG stage, here's my review anyway. Consider this
>
>     as early response to IETF last-call.
>
>     In general I support the intent of this draft, but I am concerned at the
>
>     severity of the problems I have found with it given it is meant to be
>
>     about to go to the IESG. I am particularly concerned that I have found
>
>     numerous significant problems with the normative requirements section.
>
>     Have you had a substantial review from anyone before this? The level of
>
>     review comments on the tsvwg list seemed quite light - picking on issues
>
>     of particular concern, but not seeming to review the draft as a whole.
>
> I did see some reviews (and have had comments off-list - including from
> other WGs),
> but I also REALLY do appreciate this careful read of the whole.
> See my comments on comments below (a few notes from DB as document
> shepherd are also included).
> I'm guilty of reaching the deadline, and therefore have submitted these
> changes in a new draft (06), some of these points may benefit from further
> discussion if the new update did not address the issues.
> Gorry
> --------
>
>     *1. Intro: **
>
>     *Congestion Collapse is a very specific case - CB is much more general.
>
>     it is clear from the draft that a CB is intended to mitigate
>
>     circumstances wider than solely the extreme case of congestion collapse.
>
>     For instance: a large unresponsive aggregate contributing to a high
>
>     level of congestion alongside congestion responsive traffic. This is
>
>     nowhere near congestion collapse, but it would be an applicable case for
>
>     a circuit-breaker. Congestion collapse is a specific well-defined
>
>     process that involves a cascade of congestion as a sequence of queues
>
>     fill in turn moving in the upstream direction. It is due to continual
>
>     retries or additional load arriving faster than existing flows are
>
>     departing. {Note 1}
>
> GF: I see and I can do this, and will rephrase in the next version,
> to avoid using this term.
> ----
>
>     The introduction mentions that TCP-style cc is only an appropriate
>
>     remedy when long flows dominate. The implication that CB could be used
>
>     to deal with congestion induced by many short flows is a step too far,
>
>     IMO. This problem has not even been discussed in the IETF or IRTF to my
>
>     knowledge, let alone in the context of this draft. In 6.2 this draft
>
>     all-but says that a CB is a solution to this problem. I strongly object
>
>     to a BCP making that assertion. CB would be a very drastic and clumsy
>
>     solution to that problem.{Note 2}
>
>     It says that the timescale at which a circuit-breaker operates must be
>
>     seconds or tens of seconds - much longer than the RTT timescale on which
>
>     TCP, SCTP and DCCP react. This disregards an important type of
>
>     application response to congestion; it must say that the timescale also
>
>     has to be longer than the timescale on which certain real-time
>
>     applications operate their own circuit-breakers i.e. adapt down their
>
>     codec rates, and eventually close the connection as a form of
>
>     self-admission control. Applications operate per-flow circuit-breakers
>
>     typically over the order of seconds or tens of seconds, so network CBs
>
>     MUST take longer than that - I would say "no less than a minute".
>
>     We MUST not discourage voluntary self-regulation by overriding it
>
>     (end-to-end principle). I pick up this point later (comments on section
>
>     51.), arguing that the fast-trip CB for RTP should be considered as an
>
>     application CB, and a network CB should always take longer to trigger
>
>     than these app CBs.
>
> GF: I think want to encourage applications to do this, starting by
> ensuring that they have the opportunity to do so, added some text on this.
> I revised the upper bound to clarify this, please see if this helps.
>
>
> Timescales - the multiple tens of seconds you've added is better. 
> Here's a typed record of what I said to you f2f the other day:
>
> A network circuit-breaker needs to allow time for all possible 
> self-regulation techniques (transport and app layer) to complete:
> * TCP-like congestion control (RTT)
> * Smoothed TCP-like cc (tens of RTT)
> * Codec adaptation (tens of seconds)
> * flow termination (app circuit breaker) (~1 minute)
> * flow self-admission control (~minutes)
>
> These are /not/ the timescales written in IETF drafts like 
> rtp-circuit-breakers. These are the sort of timescales that commercial 
> real-time apps would be willing to use. There is a difference between 
> theory and practice.
>
> It would be very wrong for a circuit-breaker to fire before allowing 
> all these stages to take effect.
>
> The most appropriate thing the IETF could standardise (in a separate 
> draft) is:
> 1) the max time apps should take for each of these actions {ToDo: 
> check the RTP circuit-breaker draft}
> 2) that any timers in any of these mechanisms MUST be randomised
>
>
>
> GF: I added some more text on the coexistence of fast and other circuit
> breakers, which may help (within the fast circuit breakers section) - but
> did not create a new section dedicated to this. If this is desirable we
> could create a section (electrical circuit breakers have different
> "curves" for trigger in a similar way to what is described ... but that's
> just a side observation).
> GF-XXX: If these comments still apply on the new text, it is worth
> discussing further.
> ---
>
>     *1.1 Types of CB**
>
>     **
>
>     *I saw criticism on the list of the use of the term "protect" in this
>
>     section. Why hasn't it been changed? As the posting said, a CB does not
>
>     protect the aggregate that it monitors; rather it /regulates/ the
>
>     aggregate to protect the rest of the traffic that it is /not/ monitoring.
>
> GF: Totally agree, although guilty of not changing the wording as promised
> -   (people have the same problem describing Electrical circuit breakers)
> and sorry thatI inadvertently repeated this many times.  I have rephrased
> throughout.
> ---
>
> There are various forms of network transport circuit breaker.
>
> ...
> Fast-Trip Circuit Breakers:
>
> Repeating what I said before, Fast-Trip is /not/ a type of /network/ 
> circuit breaker. This really must be fixed.
>
>
>
>
>
> Bob
>
>     *3.1 Functional Components.**
>
>     *
>
>     There is no mention of the problem of synchronising the ingress and
>
>     egress measurements to allow for transit time. Given you are trying to
>
>     measure loss, which is a relatively small difference between the traffic
>
>     entering and leaving, you can get very bad errors if you don't take path
>
>     delay into account. draft-ietf-tsvwg-tunnel-congestion-feedback
>
>     describes a nice (and commonly used) stateless way of doing that, by
>
>     sending the ingress measurement in-band to the egress, which triggers
>
>     the egress measurement so they are synchronized; allowing for transit
>
>     time. Then the egress can send them both back to the ingress to be
>
>     compared and acted on.
>
> GF:I'd hope that measurements over longer periods would not result in
> inaccuracies high enough to lead to trigger, but added a note to the text.
> GF: I'm not sure the tunnel method is a CB, it seems more like a
> congestion control method, which is perhaps why it that case the
> synchronisation is required.
> ----
>
>     *4. Reqs**
>
>     *
>
>             There MUST be a control path from the ingress meter and the egress
>
>             meter to the point of measurement.  The Circuit Breaker MUST
>
>             trigger if this control path fails.
>
>     Either this is unclear terminology, or I strongly disagree. What do you
>
>     mean by a control path? We should only recommend that the CB triggers
>
>     due to lack of measurement signals if the measurement signals are
>
>     carried in-band with the data being monitored. That is only one way of
>
>     arranging the mechanism. The term control path, sounds like it is out of
>
>     band.
>
> GF: I fell into the trap of a multiply-defined term.  This needs resolved.
> I think we should say "communication path used for control messages", and
> edit accordingly to avoid this misunderstanding.
>
>     If the measurement signals are out of band, the CB MUST NOT
>
>     trigger due to lack of measurement signals. I would recommend the
>
>     in-band method, but there are plenty of network designers who will want
>
>     to do this in centralised out of band ways, so we have to cater for that
>
>     way of thinking (even tho it's misguided).
>
> DB: I think Bob's request for not triggering the CB if a separate control
> path fails is reasonable - that may need to be NOC/operator-mediated.
> GF-XXX: I'm not sure I agree with "MUST NOT", so this needs more thought -
> comments welcome on this point.
> ----
>
>             The measurement period MUST be longer than the time that current
>
>             Congestion Control algorithms need to reduce their rate following
>
>             detection of congestion.
>
>     This needs to be rewritten. Or just removed. It seems like ideas changed
>
>     after it was written, and the end was changed but not the normative
>
>     statement at the beginning. IMO, the measurement period can be
>
>     arbitrarily short, as long as multiple measurements are combined before
>
>     triggering the CB.
>
>     It talks about unnecessarily penalizing long RTT
>
>     flows, but the measurement period is nothing to do with the period
>
>     before there is any penalization (defined later as the triggering
>
>     interval). There is no problem with short measurement periods as long as
>
>     any high congestion measured in these periods is averaged over all the
>
>     measurement periods in the triggering interval.
>
> DB: There seem to be two meanings of "measurement period" here.  I've
> always viewed it as the period of time over which measurements are taken
> that result in triggering the CB, whereas Bob seems to view it as the
> period of time over which an individual measurement is taken.  I have no
> problem with the line of Bob's text quoted above, and think some
> clarification to make it clear that the means of measurement is
> unspecified (e.g., taking short measurements and combining them is fine),
> but the period of time that applies to the metric that is used to trigger
> the CB has to be sufficiently long.
> GF: My understanding was the "measurement period" are the samples that
> feed the trigger, so if the ingress or egress meter samples more
> frequently, these would be combined. I added text to clarify this.
>
>     In fact, there should be many measurement intervals per trigger
>
>     interval, so that there are many opportunities for measurement messages
>
>     to get through. Otherwise if there are only one or two measurement
>
>     periods per trigger interval, the possibility of a false trigger due to
>
>     lost control signals becomes too great.
>
> GF: This is the robustness issue.
>
>          o  A Circuit Breaker is REQUIRED to define a threshold to determine
>
>             whether the measured congestion is considered excessive.
>
>          o  A Circuit Breaker is REQUIRED to define the triggering interval,
>
>     A perfectly good CB could vary the trigger interval and threshold
>
>     depending on how rapidly congestion is rising, or how high its absolute
>
>     level is. Indeed one could say it is actually wrong to define a single
>
>     threshold or a single interval, so these normative statements are overly
>
>     restrictive and preclude designs that are smarter than just simple fixed
>
>     threshold.
>
> GF: There are many ways to do this, and individual specs can indeed react
> using more sophisticated algorithms. At some point these will become more
> like CC than an envelop CB.
> DB: A metric and threshold for that metric are needed.  A rate-of-increase
> metric or one based in part on that would be fine.
> ----
>
>     Also, see comment above about allowing time for application CBs, and
>
>     suggesting one minute minumum.
>
>     o  A Circuit Breaker SHOULD be constructed so that it does not
>
>             trigger under light or intermittent congestion, with a default
>
>             response to a trigger that disables all traffic that contributed
>
>             to congestion.
>
>     The second half after the comma seems misplaced. If it does not trigger,
>
>     why does the sentence go on to talk about disabling all traffic that
>
>     contributed to congestion (which is what an /enabled/ trigger would do)?
>
> GF: Split the second part as a separate clause.
>
>     A reaction that results in a reduction SHOULD result in
>
>             reducing the traffic by at least a factor of ten,
>
>     What evidence have you got for this 10% number? It seems utterly
>
>     inappropriate to write a number here. The number depends on what
>
>     proportion of the traffic on the path between ingress and egress is
>
>     regulated by the CB. If the proportion is low, it needs to reduce by a
>
>     lot to make sufficient space for other traffic. If the proportion is
>
>     high relative to other traffic, it might be sufficient to reduce by 5%
>
>     to 95% of the previous load. If the tunnel traffic represented say 80%
>
>     of the load on the path, and it reduced by a factor of 10, that would
>
>     leave 92% of the path for other traffic, which might be unnecessarily
>
>     much greater than the normal proportion used by other traffic.
>
> DB: Need to say something here - we can fall back to "order of magnitude"
> or something like that, but this'll need list discussion.
> GF: Text updated. This point was previously discussed, but I am happy to
> receive more feedback, rather than relying on what was said before. As I
> recall, many were happy with terminating flows once a CB triggered, but
> this was an attempt to be "kinder" and allow more flexibility - which I
> personally liked - but still the default reaction to successive triggers
> needs to be harsh. (It is a "SHOULD" though).
> -----
>
>             Manual operator
>
>             intervention will usually be required to restore a flow.
>
>     This sentence should be toned down to possibly, not usually. A human is
>
>     no more capable than a machine is of bringing together all the necessary
>
>     measurements to decide what other courses of action might be possible,
>
>     and when to release the brakes. I suggest the last para of 5.3.1 starting:
>
>     "An operator-based response provides opportunity..."
>
>     is more appropriate here, and doesn't really fit where it is.
>
> DB:  Operator intervention may be required to restore a flow.
> GF: Changed as above.
> ----
>
>     Section 4.1 contains no requirements text, only examples. It ought to be
>
>     moved from the normative requirements section to section 5 (Examples).
>
> GF: Resolved as a section on topologies.
> ----
>
>     *5. Examples:**
>
>     *
>
>     *5.1.1 Fast-Trip CB for RTP**
>
>     *
>
>     The draft needs to make the distinction between an application doing its
>
>     own circuit breaking vs. functions on the path between the application
>
>     endpoints (even if in the hosts) doing CB. The extremely important
>
>     distinction is:
>
>     1a) an app knows when congestion is too high for it to work properly
>
>     1b) functions under the app can only infer congestion is possibly too
>
>     high for most apps to work properly
>
>     2a) an app may be able to reduce the rate at which it sends data
>
>     2b) a function under an app can only discard data, not remove it at
>
>     source.
>
> GF: Although this is not the place to delve into details, I added a prefix:
> "Applications ought to use a full-featured transport (TCP, SCTP, DCCP),
> and if not, application (e.g. those using UDP and its UDP-Lite variant
> [RFC3828])they need to provide appropriate congestion avoidance. [RFC2309]
> discusses the dangers of congestion-unresponsive flows and states that
> "all UDP-based streaming applications should incorporate effective
> congestion avoidance mechanisms". Guidance for applications that do not
> use congestion-controlled transports is provided in [RFC5405.bis]. Such
> mechanisms can be designed to react on much shorter timescales than a
> circuit breaker, that only observes a traffic envelope. These methods can
> also interact with an application to more effectively control its sending
> rate."
> GF: Also updated some other parts of the section.
> ----
>
>     I believe that the requirements in section 4 do not apply to
>
>     application-controlled circuit-breakers. So, I would not include the
>
>     "Fast-Trip CB for RTP" as an example of a /network/ transport CB.
>
>     As the requirements say, a network CB should never fast trip.
>
> GF: But by design a RTP-CB should also not terminate flows.
>
>     By misclassifying RTP CBs as network CBs, you've allowed the timescale
>
>     for network CBs to trigger after tens of seconds. When a network CB
>
>     should allow app CBs this long to trigger themselves (as I said earlier).
>
> GF: I'm not sure, since understanding the difference between the two is
> indeed important. Apps that "trigger themselves" describes what the
> fast-trip CB does (in the absence of CC).
> -----
>
>     *Missing examples:**
>
>     *
>
>     * You might want to point to the flow termination function (as opposed
>
>     to admission control) in the PCN architecture [RFC5559], which is
>
>     precisely a network CB. It was precisely developed for cases where
>
>     failures caused traffic to reroute onto a previously well-provisioned
>
>     path (see 6.1).
>
> DB: I think that's a stretch.  This is not among the types listed in 1.1.
> It might be mentioned as a related concept that is outside the scope of
> this draft.
> GF: How different to rsvp and other admission-controlled schemes? - I
> think we're drifting away here... I did change in the new rev. Unless
> there is strong support from the WG, I'd rather not add these examples.
> ------
>
>     * Andrew McGregor gave the examples of Google's BwE (bandwidth enforcer)
>
>     and B4, but you haven't referred to them. Given they are documented
>
>     existence proof of this beast, that seems remiss.
>
> GF: I believe I discussed with Andrew and we didn't at the time have good
> text to add. This may well have changed, if there are good references
> please send them for consideration.
> ------
>
>     *7. Security Consid's**
>
>     **
>
>     *
>
>          The circuit breaker MUST be designed to be robust to packet loss that
>
>          can also be experienced during congestion/overload.
>
>     This implies reliable transmission - i.e. retransmit for ever until
>
>     acknowledged. This is NOT a good idea. In
>
>     ietf-tsvwg-tunnel-congestion-feedback we propose using SCTP partially
>
>     reliable transport. Then if congestion causes messages to be lost, they
>
>     don't have to be retransmitted if there are insufficient resources (thus
>
>     not risking contributing to congestion collapse - and here I use the
>
>     phrase correctly). Because they transmit counters, the missing counters
>
>     values do not matter. This is the tried-and-tested message delivery
>
>     approach used for IPFIX. The messages can still be given priority, but
>
>     should not be retransmitted.
>
> GF: I disagree this is a requirement for reliably transmitting packets.
> I'd be happy to add text to explain robustness does not imply reliability
> and that this is likely to be an evil thing to do.
> GF: Added text on duplicating messages.
> -------
>
>          Simple protection can be provided by using a
>
>          randomized source port, or equivalent field in the packet header
>
>          (such as the RTP SSRC value and the RTP sequence number) expected not
>
>          to be known to an off-path attacker.
>
>     I think the draft should recommend that for most scenarios, randomized
>
>     ports will be insufficient protection for CB control messages, which
>
>     should be properly crytographically authenticated. Otherwise, a
>
>     CB-controlled aggregate is too vulnerable to these off-path attacks.
>
> GF-XXX: I don't know why you state that a random port (or protocol field)
> is insufficient protection from off-path attack, please explain.  Are you
> saying a CB is more vulnerable to attack than other transport traffic. I
> don't see the problem yet, can you suggest what you would like to see
> added?
> --------
>
>     *Gap #1:**
>
>     ***The draft seems to think it is so obvious what a CB should measure
>
>     that it only says it vaguely as "the level of congestion", and only
>
>     suggests the difference between ingress and egress counters as an
>
>     example. Some readers might well think like this: Does congestion level
>
>     mean the percentage extra bit-rate relative to the aggregate's expected
>
>     or maximum bit-rate? That might actually be a correct measure of
>
>     congestion in some scenarios, but...
>
>     The draft does not say that the congestion level is defined as dropped
>
>     bytes divided by ingress bytes. The draft should spell out that a CB
>
>     should measure the volume of bytes dropped and the volume of ECN-capable
>
>     bytes marked with CE, and express these as a fraction of resp. total
>
>     ingress non-ECT bytes and total ingress ECT bytes (assuming buffers
>
>     within the scope of the CB are ECN-enabled). Even this is problematic,
>
>     because the assumption in parentheses never holds, particularly during
>
>     excessive congestion. It could also discuss the relative merit of
>
>     measuring the percentage of packets dropped/marked instead of bytes.
>
> GF:  A detailed discussion of how to measure congestion is out of scope
> here, IMHO.
> -----
>
>     Also it should mention that care should be taken over how to combine the
>
>     measurements. For instance avoid the common mistake of averaging
>
>     fractions, because ave(c1/t1, c2/t2, c3/t3 ...) != (c1 + c2 + c3)/(t1 +
>
>     t2 + t3).
>
> GF: Added some text - is this sufficient, given there is no intention to
> specify the algorithm here:
> "If necessary, MAY combine successive individual meter samples from the
> ingress and egresss to ensure observation of an average over a
> sufficiently long interval. (Note when meter samples need to be combined,
> the combination needs to reflect the sum of the  individual sample counts
> divided by the total time/volume over which the samples were measured.
> Individual samples over different intervals can not be directly combined
> to generate an average value.)"
> --------
>
>     *Gap #2:**
>
>     ***All the diags show multiple routers, but the text says congestion can
>
>     be measured by comparing ingress and egress traffic. Nowhere does it say
>
>     that only traffic with addressing that will have for-certain only passed
>
>     through both ends should be measured.
>
> GF: True, I think everyone who previously looked at this thought that
> section 3.1 described. I agree though that it needs to be stated, and will
> add text.
> --------
>
>     {Note 1}: A few years ago I dug deep into the history surrounding the
>
>     early congestion collapses on the Internet and found that those involved
>
>     were adamant that the term congestion collapse should not be waved
>
>     around for dramatic effect, because it has a very specific definition,
>
>     as paraphrased above.
>
> GF: I cited an RFC on congestion collapse and did not use the words
> thereafter.
> --------
>
>     {Note 2}: The credit feature of ConEx was intended to address short-flow
>
>     overload if it becomes a problem. DOn't get me wrong; I'm not objecting
>
>     to the use of CBs for the short-flow problem because I want you to use
>
>     my solution. I'm just using this as an example of a fine-grained way to
>
>     solve the problem, rather than the sledge-hammer CB way.
>
>     Here's the intuition briefly: With ConEx, you have to attach 'congestion
>
>     credit' to the first packets of a flow to cover the risk of congestion
>
>     before you have feedback (and if you don't and there is congestion, your
>
>     packets are dropped by an audit function). Then congestion policers at
>
>     the network ingress can limit the amount of congestion credit consumed
>
>     without needing feedback, and thin out traffic if it consists of large
>
>     numbers of short flows. If short flows come to predominate, ConEx credit
>
>     was also designed to incentivize a new form of proxy that could regulate
>
>     short-flows with a push-back style of congestion control, without a full
>
>     feedback loop. That would be far preferable to such a drastic measure as
>
>     a circuit-breaker. This aspect of ConEx was not written into the IETF
>
>     docs, but it is mentioned in the re-ECN drafts that were the ancestors
>
>     of ConEx.
>
> GF: I do agree ConEx can manage traffic. I'm confident that a
> ConEx-controlled flow would not need an additional circuit breaker
> mechanism - but I see this as a congestion control mechanism though - not
> a circuit-breaker.
> ---
>
>     *Nits**
>
>     *
>
>     3.
>
>     s/last resort protection to the network paths that these are used./
>
>        /last resort protection to the traffic sharing their network path./
>
> GF: Good you spotted this, changed to: " to provide last resort protection
> for traffic that shares the network path being used."
>
>     s/tunnels encapsulations/
>
>        /tunnel encapsulations/
>
> GF: Fixed in latest version
> ---
>
>     3. What makes a good CB?
>
>          Circuit Breakers are RECOMMENDED for IETF protocols and tunnels that
>
>          carry non-congestion-controlled Internet flows and for traffic
>
>          aggregates, e.g., traffic sent using a network tunnel.
>
>     Delete "
>
>     e.g., traffic sent using a network tunnel
>
>     "
>
>     Reason: this implies all network tunnels are problematic, whereas the
>
>     rest of the sentence adequately says that only tunnels carrying
>
>     non-congestion controlled flows are of concern.
>
> GF: Understood, suggest remove from this sentence, and instead place in a
> separate sentence that follows this saying,
> "This includes traffic sent using a network tunnel."
> -----
>
>     4.
>
>     s/monitor the level congestion/
>
>        /monitor the level of congestion
>
> GF: Fixed in latest version
>
>     4.1.1
>
>     (e.g. to implement a Section 5.1)
>
>     ?
>
> GF: Fixed tag in latest version
>
>     4.1.2
>
>     s/pre-prosvisioned/
>
>        /pre-provisioned/
>
> GF: Was fixed in -06
>
>     6.1
>
>          One common question is whether a Circuit Breaker is needed when a
>
>          tunnel is deployed in a private network with pre-provisioned
>
>          capacity?
>
>     Remove '?' from the end.
>
> GF: Fixed in latest version
>
>     6.2
>
>     s/in the event that persistent congestion occur./
>
>        /in the event that persistent congestion occurs./
>
> GF: Fixed in latest version
>
>     Regards
>
>     Bob
>
> Gorry
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/