Re: [mpls] AD review of draft-ietf-mpls-forwarding

["Adrian Farrel" <[e-mail]>]

Curtis,
Thanks for all this.
I'm good with the changes recorded here.
Time to post AFAIK.

A

> -----Original Message-----
> From: Curtis Villamizar [mailto:[e-mail]]
> Sent: 28 January 2014 04:09
> To: [e-mail]
> Cc: [e-mail]; [e-mail];
> [e-mail]
> Subject: Re: [mpls] AD review of draft-ietf-mpls-forwarding
> 
> 
> In message <022d01cf1bb4$1624fde0$426ef9a0$@olddog.co.uk>
> "Adrian Farrel" writes:
> >
> > Hello Curtis,
> >
> > These replies modulo the conversation with Carlos. I find I can't read that
> > conversation and back apply the results here :-(
> 
> I'll summarize the outcome of that conversation with Carlo at the
> bottom so we can have one thread.
> 
> > [snip]
> >
> > > > Your acronym list is commendably thorough, but a little enthusiastic.
> > >
> > > If I provide a section with a list of acronyms, do I still have to
> > > expand on first use.  If so, AC, NSP, OAM, and a few others appear
> > > before that section.
> >
> > Afraid so :-(
> 
> OK.  Expanded except RFC-Editor listed well known abbreviations.
> 
> > > Given that this takes up only 17 lines in a 50 page document I'd
> > > rather be thorough.
> >
> > Yup. OK. Go for it.
> >
> > [snip]
> >
> > > > I think Curtis may have heard this before :-)
> > > > The "preferred" (by the RFC editor) expansion of ECMP is
> > > > "Equal-Cost Multipath"
> > >
> > > The form without the hyphen is more common, even among recent
> > > documents.  I prefer to keep it without the hyphen.
> >
> > A discussion for a rainy day with the RFC Editor.
> > Leave as is.
> 
> Cheers.
> 
> > > > Section 1.3 bullet 5
> > > >
> > > >    5.  The implementer and system designer MUST support pseudowire
> > > >        control word (CW) if MPLS-TP is supported or if ACH [RFC5586] is
> > > >        being used on a pseudowire.
> > > >
> > > > The wording is a bit odd. "The implementation and system design..."?
> > > >
> > > > Ditto bullets 6 and 7
> > >
> > > Target audience is explained in Section 1.4.  If you like I can flip
> > > Section 1.3 and 1.4 so target audience is first, then use of the roles
> > > called for in the target audience section won't seem quite so odd.
> >
> > No issue with the section order.
> > Just puzzled by "the implementer MUST support" when I (pedantically) thought
> > that the implementer supporting something might not be the same as the
> > implementation supporting it.
> >
> > Not a big deal.
> 
> Good point.  Its changed.
> 
> > > > Section 1.3
> > > >
> > > > While there is not wrong with the statements made in the bullets, some
> > > > of the later ones refer to recent additions to the MPLS suite. Yet the
> > > > list is presented as "there were some misconceptions." Clearly the
> > > > early silicon did not have misconceptions about the inclusion of entropy
> > > > labels.
> > > >
> > > > Just tweak the words at the top of the list?
> > >
> > > I'd like to keep that as is and split into two lists.  The second list
> > > would have the last two items (fat-pw and EL).  The first list would
> > > end with "implement CW" (sic).
> > >
> > >  OLD
> > >
> > >    6.  The implementer and system designer SHOULD support adding a
> > >        pseudowire Flow Label [RFC6391].  Deployments MAY enable this
> > >        feature for appropriate pseudowire types.  See Section 2.4.3.
> > >
> > >    7.  The implementer and system designer SHOULD support adding an MPLS
> > >        entropy label [RFC6790].  Deployments MAY enable this feature.
> > >        See Section 2.4.4.
> > >
> > >  NEW
> > >
> > >    The following statements provide clarification regarding more
> > >    recent requirements that are often missed.
> > >
> > >    1.  The implementer and system designer SHOULD support adding a
> > >        pseudowire Flow Label [RFC6391].  Deployments MAY enable this
> > >        feature for appropriate pseudowire types.  See Section 2.4.3.
> > >
> > >    2.  The implementer and system designer SHOULD support adding an MPLS
> > >        entropy label [RFC6790].  Deployments MAY enable this feature.
> > >        See Section 2.4.4.
> > >
> > > I've made this change.  Let me know if this is not OK.
> >
> > That's fine.
> >
> > > > 2.1.1
> > > >
> > > > Maybe the first paragraph should clarify "special purpose labels at
> > > > the top of the label stack"
> > >
> > > That phrase doesn't appear anywhere in the document.  Exactly what am
> > > I clarifying?  What to do with unknown special purpose labels is in
> > > the last paragraph of this subsection.
> >
> > WTF?
> > You have to wonder where these senile ADs get their ideas.
> >
> > Complete brain fart. Sorry.
> >
> > [snip]
> >
> > > > While per platform label space is mentioned in 2.1.7 I wonder whether
> > > > more information on per platform and per interface label spaces is
> > > > needed. I recall early implementations that got very confused when
> > > > parallel interfaces used the same label for different purposes.
> > > >
> > > > I guess the point there is that you cannot assume that your neighbor
> > > > is or is not using the per platform label space.
> > > >
> > > > Upstream label allocation may also come into this.
> > >
> > > The only mention of label allocation is that MPLS FRR bypass method
> > > (more formally known as facilitles backup) uses platform label space.
> > >
> > > The only reason platform label space is of any significance in a
> > > document about forwarding is "The use of platform label space impacts
> > > the size of the LSR ILM for LSR with a very large number of
> > > interfaces."
> > >
> > > Label allocation, per platform or per interface and upstream or
> > > downstream, is not a forwarding issue.  It is a software issue
> > > and a matter of getting the protocol bits right.  Therefore I think
> > > expanding any further on label allocation should be out of scope.
> >
> > OK. I'm convinced.
> >
> > > > 2.1.8.1
> > > >
> > > >    3.  If the edge is not using pseudowire control word (CW) and the
> > > >        core is using multipath, reordering will be far more common.  If
> > > >        this is occurring, the best solution is to use CW on the edge,
> > > >        rather than try to fix the reordering using resequencing.
> > > >
> > > > Completely agree, but isn't the sequence number contained in a control
> > > > word meaning that the resequencing could, in any case, not be done
> > > > without using a control word?
> > >
> > > I suppose you can't fix reordering caused by not using CW without the
> > > sequence number in the CW.  That is going to require fixing the text.
> > >
> > >  OLD
> > >
> > >    3.  If the edge is not using pseudowire control word (CW) and the
> > >        core is using multipath, reordering will be far more common.
> > >        If this is occurring, the best solution is to use CW on the
> > >        edge, rather than try to fix the reordering using resequencing.
> > >
> > >  NEW
> > >
> > >    3.  If the edge is not using pseudowire control word (CW) and the
> > >        core is using multipath, reordering will be far more common.
> > >        If this is occurring, using CW on the edge will solve the
> > >        problem.  Without CW, resequencing is not possible since the
> > >        sequence number is contained in the CW.
> > >
> > > That was a big oops on our part.
> >
> > Well, on the scale of IETF oopsies, I don't think you score too high. Maybe
> > Narten could run a weekly script?
> >
> > [snip]
> >
> > > > Should 2.2 distinguish the order of magnitude of replication at branch
> > > > nodes? This impacts the replication method used (some devices make a
> > > > copy and cycle around, some devices can do multiple copies at once). On
> > > > the whole is no different from IP multicast processing except (as you
> > > > note) that each outgoing packet may be different by its label value.
> > >
> > > Is it possible to quantify the fanout?  YMMV?
> > >
> > > The only thing I could say is that an implementation may need to make
> > > lots of copies in some roles (access routers for example).
> > >
> > > Making a copy and cycling yields poor performance but for low
> > > multicast traffic volumes might be OK.  But you are right - some
> > > mostly low-end-ish chips to this.
> > >
> > > I'm not sure I can describe how multicast with high fanout is done
> > > without wading into implementation details of specific vendors.
> > >
> > > Perhaps the best I can do is add this:
> > >
> > >    Careful consideration should be given to the performance
> > >    characteristics of high fanout multicast for equipment that is
> > >    intended to be used in such a role.
> > >
> > > I'll add this before the last paragraph in the section.
> >
> > That works.
> >
> > > > 2.4
> > > >
> > > > So obvious you didn't say it?
> > > >
> > > >    In order to support an adequately balanced load distribution across
> > > >    multiple links, IP header information must be used.  Common practice
> > > >    today is to reinspect the IP headers at each LSR and use the label
> > > >    stack and IP header information in a hash performed at each LSR.
> > > >    Further details are provided in Section 2.4.5.
> > > >
> > > > Missing is the statement that a single "flow" must not be distributed
> > > > across multiple paths because of the implication for potentially
> > > > significant packet misordering. And feeding that is a common requirement
> > > > that such packet misordering must not occur because applications and
> > > > transport protocol implementations cannot survive such misordering.
> > >
> > > Yes.  That requirement was missed.  Add new second paragraph to this
> > > subsection.
> > >
> > >    The Differentiated Services requirements for good reasons dictate
> > >    that packets within a common microflow SHOULD NOT be reordered
> > >    [RFC2474].  Service providers generally impose stronger
> > >    requirements, commonly requiring that packets within a microflow
> > >    MUST NOT be reordered except in rare circumstances such as load
> > >    balancing across multiple links or path change for load balancing
> > >    or path change for other reason.
> > >
> > > Another SP requirement is stated here and I'm quite sure this
> > > requirement is well accepted.
> >
> > Looks good.
> >
> > > > 2.4.2 uses "composite link" and "component link". I suggest picking just
> > > > one term.
> > >
> > > They are two different things.  Two or more component links make up a
> > > composite link.  Knowing that, give it another read please.
> > >
> > > I'd rather not cite draft-ietf-rtgwg-cl-requirements as an
> > > informational reference just for this one term.  In favor of citing
> > > it, draft-ietf-rtgwg-cl-requirements is moving along.  Against citing
> > > it is there is far less than a ground swell of providers calling for
> > > the full set of things asked for in draft-ietf-rtgwg-cl-requirements.
> >
> > Yes. Sorry. It has been a looooooooong time since I had a pass on the CL
> > document. Atrophy.
> 
> Its also in Gwiz.8080 or something like that.
> 
> > > > 2.4.5.1 notes that special purpose and extended special purpose labels
> > > > need to be excluded from the hash. Good.
> > > > But it seems that some special purpose labels will indicate that the
> > > > next label stack entry contains a label with special meaning. (ELI is
> > > > an example that we specifically don't have to worry about.)
> > > > How do we handle that?
> > > > Should we be dividing up the extended special purpose label space to
> > > > have one set of code points meaning "just this label is special" and
> > > > another set meaning "this label is special and the next label stack
> > > > entry is magic"?
> > >
> > > I did list ELI (bullet 2) before the more general rule of not useing
> > > special purpose labels.  The ELI is not used, just the EL, so the text
> > > could be considered correct as-is.
> > >
> > > So far the only special purpose label that is not just ignored and
> > > skipped over is ELI.
> > >
> > > Regarding this being magic -- All of this is somewhat programable
> > > specialized silicon magic.  The silicon generally has some form of
> > > very fast, very light weight parsing engine at the front of the
> > > pipeline.  One thing it does is pick out fields for load balance.
> > >
> > > The better silicon hashes as it goes rather that pick out a set of
> > > fields and then hashes that set of fields when its done.  When it sees
> > > 13 it skips and hashes the next thing and stops hashing completely.
> > > If it sees 0-12,14 it skips and continues.  If it sees 15 it skips two
> > > labels and continues.  Its should be programable enough that if
> > > someone defines a new ELI like label it is likely to be able to deal
> > > with it.
> > >
> > > The not so good silicon has this all so hard wired that it won't be
> > > able to do ELI without at least a respin.
> > >
> > > At most I could add "If a new special purpose label or extended
> > > special purpose label is defined which requires special load balance
> > > processing then, as is the case for the ELI label, a spacial action
> > > may be needed rather than skipping the special purpose label or
> > > extended special purpose label."  I really don't think this is needed.
> >
> > You're right, and my worry is more about the special-purpose draft and the
> > consequences of possibly adding other special purpose labels that have child
> > labels associated. We certainly don't want to have to retrain the silicon at
> > transit LSRs to specially know what to do for each new special-purpose
label.
> > Currently we propose that you don't hash on a special purpose label, but you
> can
> > carry on hashing immediately after.
> >
> > If I introduce the foo-label, your silicon will recognise it as special
purpose,
> > but it I say the label after the foo-label is magic you won't know that.
> >
> > A way to fix this is to have (just punting here) the top bit of the extended
> > special purpose label range set mean "magic label follows".
> 
> I added this:
> 
>    4.  If a new special purpose label or extended special purpose
>        label is defined which requires special load balance
>        processing, then, as is the case for the ELI label, a spacial
>        action may be needed rather than skipping the special purpose
>        label or extended special purpose label.
> 
> This will be the new bullet 4, bumping down the old 4 and 5.
> 
> > > > An issue that arises from the multipath support (2.4.5.1) is that
> > > > hardware assumes that after a label stack entry with the S-bit set,
> > > > there are only three possible next bytes...
> > > > - a control word (indicated by b0000 or b0001)
> > > > - an IPv4 header
> > > > - an IPv6 header
> > > > This is the case regardless of how the LSP was set up, and the next
> > > > bytes cannot ever be further MPLS stack entries.
> > >
> > > Right.  Note that in (5) is says that some SP will require IP headers
> > > and some will require an ability to disable IP headers.
> > >
> > > The rule is really look for 4, 6, or anything else in the first
> > > nibble.  If 4 or 6 assume IP.  If anything else stop.
> > >
> > > And yes if the payload is MPLS after a S-bit you have a screwed up
> > > MPLS implementation to start with and you won't get load balance on
> > > any set of MPLS labels after the first S-bit.  This is a fact of life
> > > in the field and is as it should be.
> > >
> > > > While this comes up 2.4.5.1 it may merit further discussion in an
> > > > earlier section of the document.
> > >
> > > This text is part of 2.4. ("MPLS Multipath Techniques").  The third
> > > paragraph contains "Further details are provided in Section 2.4.5."
> > > Section 2.4.5. is "Fields Used for Multipath Load Balance".
> > >
> > > > I note that discussion of support of PWs without the CW drives you
> > > > to say that hashing beyond the S-bit should be a configurable option
> > > > which would (of course) support any payload including MPLS in MPLS
> > > > with repeated bottom of stack. However, you might want to specifically
> > > > preclude that.
> > >
> > > It says the same thing here in bullet 5 regarding being configurable.
> > > The wording "ability to disable" is same as "configurable option".
> > >
> > > At no point in this document do we imply that looking beyond the S-bit
> > > means looking at anything beyond the S-bit other than looking for IP.
> > > This is very clear in [RFC4385] and [RFC4928] which is cited in the
> > > text about PW CW.
> > >
> > > All it says in the places discusing PW is that without CW the traffic
> > > might get reordered.
> > >
> > > If you feel that we at any point imply that lack of PW CW allows
> > > looking at anything past the S-bit rather than just looking for an IP
> > > header please point to where and we will have to correct that.  I
> > > looked at all occurances of CW and did not find anything.
> > >
> > > Bullet 5 is very clear that a 4 or 6 has to be found in the first
> > > nibble of payload.
> >
> > OK. I misread bullet 5.
> 
> OK
> 
> > [snip]
> >
> > Thanks.
> > Adrian
> 
> Summary of conversation with Carlos:
> 
> Adrian wrote:
> >>   Tunneling encapsulations which may carry MPLS, such as MPLS in GRE,
> >>   L2TP, or LDP, are out of scope.
> >>
> >> I think s/LDP/UDP/
> 
> Carlos noted some wording issues in this sentence and suggested adding
> citations for each.  After some discussion:
> 
>  OLD (original)
> 
>    Tunneling encapsulations which may carry MPLS, such as MPLS in GRE,
>    L2TP, or LDP, are out of scope.
> 
>  NEW
> 
>    Tunneling encapsulations carrying MPLS, such as MPLS in IP
>    [RFC4023], MPLS in GRE [RFC4023], MPLS in L2TPv3 [RFC4817], or MPLS
>    in UDP [I-D.ietf-mpls-in-udp], are out of scope.
> 
> He also suggested the following.
> 
>  OLD
> 
>    Some IP encapsulations support tunneling, such as IP-in-IP, GRE,
>    L2TPv3, and IPSEC.  These provide a greater source of entropy which
>    some provider networks carrying large amounts of tunneled traffic
> -   may need.
>    The use of tunneling header information is out of scope for this
>    document.
> 
>  NEW
> 
>    Some IP encapsulations support tunneling, such as IP-in-IP, GRE,
>    L2TPv3, and IPSEC.  These provide a greater source of entropy which
>    some provider networks carrying large amounts of tunneled traffic
> +   may need, for example as used in [RFC5640] for GRE and L2TPv3.
>    The use of tunneling header information is out of scope for this
>    document.
> 
> As part of that conversation it was noted that the following also had
> no citations.
> 
>  OLD
> 
>    Support for other protocols that share a common Layer-4 header such
> -   as RTP, UDP-lite, SCTP and DCCP
>    SHOULD be provided, particularly for edge or access equipment where
>    additional entropy may be needed.
> 
>  NEW
> 
>    Support for other protocols that share a common Layer-4 header such
> +   as RTP [RFC3550], UDP-Lite [RFC3828], SCTP [RFC4960] and DCCP
> +   [RFC4340]
>    SHOULD be provided, particularly for edge or access equipment where
>    additional entropy may be needed.
> 
> btw- this is wrt fields used in load balancing and is saying in effect
> "use more than just port 6 and 17 (UDP and TCP)".
> 
> Other than that, I looked at the list of informational references and
> found the following do affect MPLS forwarding and therefore should be
> promoted to normative.
> 
>    [RFC4950]  Bonica, R., Gan, D., Tappan, D., and C. Pignataro, "ICMP
>               Extensions for Multiprotocol Label Switching", RFC 4950,
>               August 2007.
> 
>    [RFC5082]  Gill, V., Heasley, J., Meyer, D., Savola, P., and C.
>               Pignataro, "The Generalized TTL Security Mechanism
>               (GTSM)", RFC 5082, October 2007.
> 
>    [RFC5085]  Nadeau, T. and C. Pignataro, "Pseudowire Virtual Circuit
>               Connectivity Verification (VCCV): A Control Channel for
>               Pseudowires", RFC 5085, December 2007.
> 
>    [RFC5332]  Eckert, T., Rosen, E., Aggarwal, R., and Y. Rekhter, "MPLS
>               Multicast Encapsulations", RFC 5332, August 2008.
> 
>    [RFC5880]  Katz, D. and D. Ward, "Bidirectional Forwarding Detection
>               (BFD)", RFC 5880, June 2010.
> 
>    [RFC5884]  Aggarwal, R., Kompella, K., Nadeau, T., and G. Swallow,
>               "Bidirectional Forwarding Detection (BFD) for MPLS Label
>               Switched Paths (LSPs)", RFC 5884, June 2010.
> 
>    [RFC5885]  Nadeau, T. and C. Pignataro, "Bidirectional Forwarding
>               Detection (BFD) for the Pseudowire Virtual Circuit
>               Connectivity Verification (VCCV)", RFC 5885, June 2010.
> 
>    [RFC6374]  Frost, D. and S. Bryant, "Packet Loss and Delay
>               Measurement for MPLS Networks", RFC 6374, September 2011.
> 
>    [RFC6375]  Frost, D. and S. Bryant, "A Packet Loss and Delay
>               Measurement Profile for MPLS-Based Transport Networks",
>               RFC 6375, September 2011.
> 
>    [RFC6378]  Weingarten, Y., Bryant, S., Osborne, E., Sprecher, N., and
>               A. Fulignoli, "MPLS Transport Profile (MPLS-TP) Linear
>               Protection", RFC 6378, October 2011.
> 
>    [RFC6427]  Swallow, G., Fulignoli, A., Vigoureux, M., Boutros, S.,
>               and D. Ward, "MPLS Fault Management Operations,
>               Administration, and Maintenance (OAM)", RFC 6427, November
>               2011.
> 
>    [RFC6428]  Allan, D., Swallow Ed. , G., and J. Drake Ed. , "Proactive
>               Connectivity Verification, Continuity Check, and Remote
>               Defect Indication for the MPLS Transport Profile", RFC
>               6428, November 2011.
> 
>    [RFC6720]  Pignataro, C. and R. Asati, "The Generalized TTL Security
>               Mechanism (GTSM) for the Label Distribution Protocol
>               (LDP)", RFC 6720, August 2012.
> 
>    [I-D.ietf-mpls-psc-updates]
>               Osborne, E., "Updates to PSC", draft-ietf-mpls-psc-
>               updates-00 (work in progress), October 2013.
> 
> Upgrades were mostly due to BFD and MPLS-TP OAM.  For example, direct
> LM needs to be in hardware and most things related to protection.
> 
> Note that "Updates to PSC" affects MPLS-TP protection state machine
> which needs hardware assist if not direct support in hardware in order
> to be fast.  The race condition may be the most severe problem with
> RFC 6378 state machine.
> 
> If you think any of the above should be put back into informative,
> please let me know.
> 
> Curtis