Re: [Bier] WGLC - draft-ietf-bier-te-arch 1 WEEK

[Lou Berger <[e-mail]>]

Hi Toerless ,

Thanks for the super quick response

On 2/19/2020 10:56 PM, Toerless Eckert wrote:
> Thanks a lot, Lou
>
> [ Would have been nice if you could have commented a bit earlier.
>    But i can understand how a few years is not enough time ;-P
>    (actually no kidding, i really can.) ]

I raised this as an issue last march - at both the Chair and AD level.  
I thought I made the same point to you privately after one the 
presentations at IETF101, but if you don't remember it, I accept that I 
didn't.

> Originally i had planned to address the explanations you are missing
> in this doc in the BIER-TE traffic engineering framework document,
> for which i had written the -00 version and presented at TEAS WG, IETF101,
> but given how we first wanted to get BIER-TE out as RFC, i let that
> expire, and in hindsight it's certainly useful to have a short
> section summarizing this in the BIER-TE RFC itself:

Well that document seems like a fine place to describe how BIER-RP 
(routing policy) can be used to deliver BIER-TE.

> So, I just pushed -06 of the draft to address your concerns with
> additional explanations.
> Summary below, diff here:
>
> http://tools.ietf.org/tools/rfcdiff/rfcdiff.pyht?url1=https://tools.ietf.org/id/draft-ietf-bier-te-arch-05.txt&url2=https://tools.ietf.org/id/draft-ietf-bier-te-arch-06.txt
>
> - Title changed from "Traffic Engineering for..." to "Path Engineering
>    for...",

Do we really need a new term to describe here?  I find zero (0) 
instances of Path Engineering in any RFC.  Routing policy (and 
policy-based routing) on the other hand are fairly well established 
terms in the industry.  I think this just sows seeds for future 
confusion on this topic.

Why is "routing policy" not good enough for what is being defining 
here?  I again point out, this is the term being used in SPRING for 
basically the equivalent function/purpose. (Yes the forwarding mechanism 
is different, but the objective is not.)

> but kept name BIER-TE (see below).
>
> - Added section 1.1 explaining how BIER-TE relates to traffic
>    engineering, naming use-cases where its for example beneficial standalone and
>    and what it could be combined with for more comprehensive TE solutions,
>    but also stating that those integrations are outside the scope of this
>    document.

So this section seems to miss what has been going on in traffic 
engineering / TEAS for the last 5+ years (i.e., controller-based TE 
approaches)  and then goes on describe path steering  in a very brief 
way.  I read this as saying that BIER *could* do TE in the future and 
*can* support routing policy and path steering today.

> - changed "traffic engineering" term in the whole doc to "path engineering",
>    where appropriate.

While this is appreciated, I'm not sure it's helpful.  As stated above, 
I think the introduction of the new PE term is confusing as the 
continued use of BIER-TE.

> - Unrelated to you, there was one leftover fix from ietf106 review to rename BIER-TE
>    Controller Host to just BIER-TE Controller
>
> Wrt to naming:
>
> The mayority of customers i talked to only used RSVP-TE for path
> engineering, and not for anything more. Several didn't even know it
> can do bandwidth reservation. Nobody knew it could do latency
> guarantees, because nobody knows an implementation that supports that.
> [All reasons btw. why replacing RSVP-TE with SR happened in the industry.]

While I'm going to avoid getting into product differentiators and 
marketing, you're not mentioning that even those products and customers 
who didn't support/use per LSP queuing, did available resource 
bookkeeping and even admission control.

> In any case, the name BIER-TE was selected to reduce confusion
> with customers, not to maximize naming correctness in IETF.

umm, the IETF is a standards body not an industry marketing forum, so 
I'm unclear how this point helps your argument.  It seems that you're 
saying that if we call it BIER-TE we can market it in place of existing 
IETF TE solutions - even though it doesn't yet have the TE capability 
covered in draft-eckert-teas-bier-te-framework.

Assuming I'm reading it right, this just confirms to me that the current 
work needs to be renamed and that draft-eckert-teas-bier-te-framework 
will define BIER-TE.

> Something like "BIER-PE" (Path Engineering) would have
> probably confused more than it would have helped.  Think
> of the justification for the name BIER-TE not as
> "all you need to do TE", but "the variation of BIER to support TE".

I'm sorry to say, I'm left more confused by this response and doc update 
than I was before it.

> Wrt to SR:
>
> SR can actually NOT do the same as BIER-TE. It has no stateless multicast.

My point wasn't that BIER=SR, but rather both deliver support for path 
steering and policy based routing.

Thanks for being responsive!

Lou

> All the SR options do really require that you set up multicast trees with
> e.g.: replication-SIDs that together form the equivalent of a
> multicast tree, like you would have built with RSVP-TE. Except that
> the signaling how to build the tree is left for someone else, like
> PCECC. (AFAIK, i may not be on top of all details). In BIER-TE,
> there is no such per-tree state on transit nodes.
>
> Instead of a 256 bit bitstring in BIER-TE think of a header with up to 256 SIDs
> (e.g.: 128 bit per SID in SRv6). That would be the SR equivalent of BIER-TE.
> Just a bit less ( ;-) ) less efficient on the wire than BIER-TE and extremely
> harder to parse.
>
> Cheers
>      Toerless
>
>
> On Wed, Feb 19, 2020 at 06:38:38PM -0500, Lou Berger wrote:
>> Hi,
>>
>>      I have no issue or objection to the mechanisms being defined in this
>> document as much as they go, but I was quite disappointed that despite the
>> name of the document and use of 'bier-te'  to see that the document doesn't
>> define any traffic engineering support, at least as far as the term has been
>> used in IETF RFCs.  In particular it totally lacks any discussion of
>> resources usage and/or allocation.  What it currently describes certainly
>> provides good and useful path/traffic steering that can be used to support
>> policy-based routing.  Basically it does the same as what is defined by
>> draft-ietf-spring-segment-routing-policy.
>>
>> I personally (not speaking for the related WGs that I chair) would prefer to
>> see this document be revised  to include resource allocation that would
>> allow BIER-TE to support TE usage such as DetNet.  Barring such an addition,
>> I'm against publication of this document as is and I think the document
>> should be recast and renamed to be aligned with the SR example, i.e., BIER
>> routing policy (or path steering).
>>
>> Lou
>>
>> On 2/18/20 3:45 PM, Greg Shepherd wrote:
>>> Thanks Toerless and Jeffrey
>>>
>>> https://datatracker.ietf.org/doc/draft-ietf-bier-te-arch/
>>>
>>> One more week of WGLC. Please read the latest rev and respond to this
>>> thread w/wo support.
>>>
>>> Chairs
>>> (Shep)
>>>
>>>
>>> On Tue, Feb 18, 2020 at 12:07 PM Jeffrey (Zhaohui) Zhang
>>> <zzhang=[e-mail]
>>> <mailto:[e-mail]>> wrote:
>>>
>>>      Hi Toerless,
>>>
>>>      Thanks!
>>>      I support moving this to the next stage.
>>>
>>>      Jeffrey
>>>
>>>      On Fri, Nov 01, 2019 at 07:42:38PM +0100, Toerless Eckert wrote:
>>>      > Thanks Jeff
>>>      >
>>>      > I have now pushed out -05 with the answers and hopefully
>>>      resolution to
>>>      > your points in email below.  Biggest addition was a section about
>>>      > reuse of BPs (without DNR) which came out of the confusion i
>>>      think the
>>>      > reuse in the ECMP example raised. I was afraid so far to explan
>>>      that
>>>      > as it may not be easy to absorb and ultimately is stuff only
>>>      > controller developers need to understand, but hopefully useful.
>>>      > And then of course the summary of BP optimizatins you asked for
>>>      >
>>>      > Diff from last version i sent you:
>>>      >
>>>      >
>>>      https://urldefense.com/v3/__http://tools.ietf.org/*rfcdiff?url1=https:
>>>      > **Araw.githubusercontent.com
>>>      <http://Araw.githubusercontent.com>*toerless*bier-te-arch*master*draft-ietf-b
>>>      > ier-te-arch-05.1.txt&url2=http:**Atools.ietf.org
>>>      <http://Atools.ietf.org>*id*draft-ietf-bier-te
>>>      >
>>>      -arch-05.txt__;Ly8vLy8vLy8vLy8!!NEt6yMaO-gk!VuQCVHnqJy_aYI-FNt9A1a5EzH
>>>      > OCr0fZkLPbgg3CPNu0PyrWsrFx41_jWX2At8V-$
>>>      >
>>>      > full -04 -> 05 diff:
>>>      >
>>>      >
>>>      https://urldefense.com/v3/__http://tools.ietf.org/*rfcdiff?url1=http:*
>>>      > *Atools.ietf.org
>>>      <http://Atools.ietf.org>*id*draft-ietf-bier-te-arch-04.txt&url2=http:**Atools.
>>>      > ietf.org
>>>      <http://ietf.org>*id*draft-ietf-bier-te-arch-05.txt__;Ly8vLy8vLy8v!!NEt6yMaO-gk
>>>      > !VuQCVHnqJy_aYI-FNt9A1a5EzHOCr0fZkLPbgg3CPNu0PyrWsrFx41_jWancpziv$
>>>      >
>>>      > Comments inline below.
>>>      >
>>>      > Cheers
>>>      >     toerless
>>>      >
>>>      > On Mon, Oct 28, 2019 at 07:52:59PM +0000, Jeffrey (Zhaohui)
>>>      Zhang wrote:
>>>      > > I Thought u-turn is the most simple comparison leaf vs.
>>>      non-leaf BFR.
>>>      > >
>>>      > > Zzh> The text in the email is seriously misaligned. Looking at
>>>      the picture in the diff link, while you gave a U-turn example,
>>>      though even if BFER2 is not connected to BFR2  but only connected
>>>      to BFER1 (hence no U-turn), then BFER1 is still not a leaf BFER I
>>>      suppose. That's why I said the first sentence of the above
>>>      paragraph is enough to define Leaf BFER while the example itself
>>>      is actually not needed.
>>>      >
>>>      > Argh... ok, had to fix two words, BFIR->BFER and left-hand ->
>>>      right-hand:
>>>      >
>>>      > Consider how redundant disjoint traffic can reach BFER1/BFER2 in
>>>      above
>>>      > picture: When BFER1/BFER2 are Non-Leaf BFER as shown on the
>>>      right hand
>>>      > side, one traffic copy would be forwarded to BFER1 from BFR1,
>>>      but the
>>>      > other one could only reach BFER1 via BFER2, which makes BFER2 a
>>>      > non-Leaf BFER. Likewise BFER1 is a non-Leaf BFER when forwarding
>>>      > traffic to BFER2
>>>      >
>>>      > > Zzh> Additionally, in left part of the picture you added, if
>>>      some failure leads to BFR2 to be only reachable via BFER1, then
>>>      BFER1 is no longer a leaf BFER.
>>>      >
>>>      > Added sentence:
>>>      >
>>>      > <t>Note that the BFER in the left hand picture are only
>>>      guaranteed to
>>>      > be leaf-BFR by fitting routing configuration that prohibits transit
>>>      > traffic to pass through a PE, which is commonly applied in these
>>>      > topologies.</t>
>>>      >
>>>      > > I assume you don't reassign BPs when links go up and down.
>>>      >
>>>      > I didn't want to discuss that option in this document. Its
>>>      obviously
>>>      > perfectly feasible, but be yet a big amount of text (especially the
>>>      > considerations how to do this make-before-break. Future doc.
>>>      >
>>>      > > > but subsequent polarization example confuses me. It seems
>>>      that BP 0:6 is assigned to the routed adjacency BFR10 (which is
>>>      actually talked about in Section 4.8).
>>>      > >
>>>      > > Section 4.7 does not mention "routed" at all, so there are no
>>>      routed adjacencies at all used in 4.7. So i am not sure what you
>>>      are confused about.
>>>      > >
>>>      > > Zzh> "The BIFT of each BFR are only populated with BPs that
>>>      are adjacent to the BFR in the BIER-TE topology".
>>>      >
>>>      > Correct text from the introduction. Ok.
>>>      >
>>>      > > Zzh> Since the same 0:6 is in BIFTS of BFR1/BFR2/BFR3 (and I
>>>      suppose in BFR4~BFR9 as well even though not drawn), I assumed
>>>      it's for the "MP2P" routed adjacency to R10; though I then ruled
>>>      that out - but I don't know what 0:6 represent now on BFR1, BFR2,
>>>      and BFR3.
>>>      >
>>>      > Ah. Ok. I thought i could strip down the example to show only the
>>>      > adjacencies relevant to the following discusion, but seemingly this
>>>      > can introduce the confusion you have.
>>>      >
>>>      > So i completed the example with the BP assignment acoss all
>>>      nodes, but
>>>      > added text pointing to a new section further down to discuss the
>>>      > re-use of BP for which thi picture is also an example.
>>>      >
>>>      > (check out the diff, new reuse text to long to copy inline).
>>>      >
>>>      > > The whole purpose of the ECMP BPs is of course to save bits,
>>>      otherwise we'd give each link a separate BP, which would be 6 BP
>>>      to reach to BFR4...BFR7 from BFR1.
>>>      > >
>>>      > > Zzh> The trouble I am having is that the same 0:6 is assigned
>>>      to different things and it's present on all BFR1/BFR2/BFR3. It is
>>>      perhaps an intentional smart design but I have not wrapped my mind
>>>      around it. It's apparently different from the link bundle case, so
>>>      better separate it out and elaborate it (including the DNR flag
>>>      that might be needed here - If the packet arrives on BFR1 with
>>>      0:6, would the BP reset when it is sent to BFR2/3)?
>>>      >
>>>      > Yes, there was the bug of reusing BP 0:6 across sequential BFR
>>>      along
>>>      > the path, but now the example correctly reuses separate BP at
>>>      > different stages of the paths (BP 0:6 on BFR1, BP 0:7 on
>>>      BFR2/BFR3) and so on.
>>>      >
>>>      > Thanks!
>>>      >
>>>      > > > 4.8.  Routed adjacencies
>>>      > > >
>>>      > > > If I understand it correctly, there is a BP assigned to
>>>      L1/L2/L3
>>>      > > > respectively (p2p link), and then there are BPs assigned to
>>>      MP2P tunnels (routed adjacency from every BFR) to the L1/L2/L3
>>>      interface addresses and loopback addresses on BFR2/3.
>>>      > >
>>>      > > Ok that wasn't quite the read i expected. Let me clarify the
>>>      text/picture:
>>>      > >
>>>      > >                    ...............
>>>      > >          ...BFR1--...           ...--L1-- BFR2...
>>>      > >                   ... .Routers. ...--L2--/
>>>      > >          ...BFR4--...           ...------ BFR3...
>>>      > >                    ...............         |
>>>      > >                                           LO
>>>      > >                     Network Area 1
>>>      > >
>>>      > > Assume the requirement in the above picture is to explicitly
>>>      steer traffic flows that have arrived at BFR1 or BFR4 via a
>>>      shortest path in the routing underlay "network area 1" to one of
>>>      the following three next segments: (1) BFR2 via link L1, (2) BFR2
>>>      via link L2, (3) via BFR3.
>>>      > >
>>>      > > To achieve this, both BFR1 and BFR4 are set up with a
>>>      forward_routed adjacency BitPosition towards an address of BFR2 on
>>>      link L1, another forward_routed BitPosition towards an address of
>>>      BFR2 on link L2 and a third forward_routed Bitposition towards a
>>>      node address LO of BFR3.
>>>      > >
>>>      > > Does this clear ip the confusion ?
>>>      > >
>>>      > > Zzh> The picture is badly misaligned. I'll wait till 4.7
>>>      questions are cleared.
>>>      >
>>>      > Ok.
>>>      >
>>>      > > > If BFR2/3 are also BFERs, then they additionally will have
>>>      BFER BPs.
>>>      > > > On BFR1/4, the BIFT entries for the MP2P BPs for the
>>>      L1/L2/L3/loopback interface addresses of BFR2/3 will use
>>>      forward_routed(interface/loopback address). For a packet to be
>>>      decapsulated on a BFER, there is a need for both the BFER BP and
>>>      another BP (p2p/lan/hub-spoke/routed-adjacency) in the packet (the
>>>      former is for decapsulation and the latter is for getting it there).
>>>      > >
>>>      > > This is not discussed in this section, but you are right - unless
>>>      > > BFR2 or BFR3 is a leaf BFR. In that case, it would just
>>>      leverage the one shared "leaf-BFR" BP, so they do not need a
>>>      per-BFER BP for local_decap().
>>>      > >
>>>      > > Zzh> Right - shared leaf-BFR BP but still need that BP (the
>>>      key is that we need a BP to get packet to a BFER and then a BP for
>>>      decapsulation).
>>>      >
>>>      > You got it.
>>>      >
>>>      > > > If that???s the case, it???s worth point the above out.
>>>      > >
>>>      > > Hmm... The logic of BFER BPs is totally independent of the
>>>      logic of forward_routed adjacency, so i would worry that repeating
>>>      the explanation of BFER BPs would conflate the forward_routed
>>>      explanation.
>>>      > >
>>>      > > Zzh> It's just that this is a place where all kinds of BPs are
>>>      used so it's good to have a summary (could be a subsection 4.9).
>>>      >
>>>      > Yes, added such a summary. Pls. check.
>>>      >
>>>      > > > Actually, the reason that I thought this is MP2P is that 0:6
>>>      is present on R1, R2, and R3 (and more I assume) in Figure 12, but
>>>      now I think it can???t be MP2P (so it is not correct to have 0:6
>>>      present on those routers ??? only the p2p tunnel head/tail should
>>>      have the BP present in the BIFT). The reason is that if it were
>>>      MP2P, any router getting a copy will send it to the endpoint of
>>>      the routed adjacency, causing lots of duplicates.
>>>      > > >
>>>      > > > Am I getting this correct?
>>>      > >
>>>      > > I think you are still explaining from the misunderstsanding
>>>      that the ECMP explanations where about routed adjacencies.
>>>      > >
>>>      > > I have now expanded the somewhat terse text in the BIFT table
>>>      pictures, to make it clear that the ECMP is across multipe
>>>      forward_connected adjacencies in the examples. For example, first
>>>      BIFT picture:
>>>      > >
>>>      > >   BIFT entry in BFR1:
>>>      > >
>>>       ------------------------------------------------------------------
>>>      > >   | Index |  Adjacencies                  |
>>>      > >
>>>       ==================================================================
>>>      > >   | 0:6   |  ECMP({forward_connected(L1, BFR2),                 |
>>>      > >   |       |        forward_connected(L2, BFR2),                 |
>>>      > >   |       |        forward_connected(L3, BFR2)}, seed)
>>>           |
>>>      > >
>>>       ------------------------------------------------------------------
>>>      > >
>>>      > > Of course, an ECMP adjacency can be across any type of
>>>      adjacencies, but all the text/explanations used forward_connected,
>>>      and now the pictures show that explicitly.
>>>      > >
>>>      > > Zzh> I can understand the multi-link case, but the multi-hop
>>>      ECMP case (from BFR1 towards BFR10) is confusing me. It would help
>>>      to give an example how it can be used, WITHOUT worrying about
>>>      polarization.
>>>      >
>>>      > Please check -05 text that has the full set of BIFT listed now:
>>>      >
>>>      > There is  really nothing nothing unique in multi-hop ECMP for
>>>      BIER-TE
>>>      > that we do not also have in any other ECMP, except the
>>>      conclusion that
>>>      > we want to support fast HW hash mechanisms AND allow the
>>>      controller to
>>>      > set up non-polarized multi-hop ECMP AND be able to precalculate
>>>      paths.
>>>      > Hence the specification of ECMP adjacencies to have a controller
>>>      > configurable seed.
>>>      >
>>>      > Btw: The picture is maybe unnecessarily large because i've used
>>>      it for
>>>      > 20 years to explain the same polarization issue for unicast vs
>>>      > multicast, and for multicast only BFR10...BFR4 are relevant
>>>      (ECMP of
>>>      > the PIM/mLDP joins), whereas for unicast/BIER only
>>>      > BFR1...BFR7 are relevant. But being symmetric, the picture makes it
>>>      > clear its the same problem.
>>>      >
>>>      > > >    To inhibit looping in the face of such physical
>>>      misconfiguration,
>>>      > > >    only forward_connected adjacencies are permitted to have
>>>      DNR set, and
>>>      > > >    the link layer destination address of the adjacency
>>>      (e.g.  MAC
>>>      > > >    address) protects against closing the loop.  Link layers
>>>      without port
>>>      > > >    unique link layer addresses should not be used with the
>>>      DNR flag set.
>>>      > > >
>>>      > > > It???s not clear how link layer address helps?
>>>      > >
>>>      > > I have expanded this to
>>>      > > "link layer port unique unicast destination address"
>>>      > >
>>>      > > Aka: MPLS or ethernet have unique link layer destination
>>>      destination addresses (label or destination MAC). If you think
>>>      about incorrectly plugged HDLC links (such as old T1/T3/...
>>>      links), they only have 2 generic addresses, if i remember 1 or 3
>>>      in the HDLC frame. So when you misplug one of those p2p cables
>>>      wrong, the packets would be incrrectly received by the wrong
>>>      receiver node and then DNR could cause persistent loops only
>>>      solved by TTL.
>>>      > >
>>>      > > Zzh> "Consider in the ring picture that link L4 from BFR3 is
>>>      plugged into the L1 interface of BFRa" - still not sure how
>>>      label/mac helps here. I suppose the ring topology is
>>>      discovered/verified by the control plane and when the miscalling
>>>      happens then the ring will not include the BFR1/BFR2 part and BFR3
>>>      will not have the DNR set? If ring discovery/varication is not
>>>      done then perhaps we should point out that RPF based on link layer
>>>      address is needed - the key is RPF (which needs unique link layer
>>>      address)?
>>>      >
>>>      > Forget RPF. BIER(-TE) has no RPF (issues). Its just like
>>>      unicast. RPF
>>>      > is just a problem for receiver originated joins like in
>>>      PIM/mLDP, but
>>>      > not unicast/bier(-te)/RSVP-TE.
>>>      >
>>>      > Forward_connected is just like a unicast subnet adjacency to a
>>>      direct
>>>      > neighbor: Interface and L2 addresss of the destination.
>>>      >
>>>      > The controller (could be a human) "assumes" a particular physicial
>>>      > topology, from telemetry/knowledge/whatever. It then calculates the
>>>      > desired BIER-TE topology and pushes it down. This topology is
>>>      meant to
>>>      > be loop free of course wrt to the configured adjacencies.
>>>      > In this BIER-TE topology, BFR3 will have a BP with the
>>>      > forward_connected(L4, MAC-of-BFR2) adjacency.
>>>      >
>>>      > If the cable connecting to L4 is miswired, then BFR3 would still
>>>      send
>>>      > the packets to the MAC address of BFR2, but given how the cable
>>>      > connects to some other node, these packets will be discarded by
>>>      that
>>>      > node. because they're just L2 unicast packets.
>>>      >
>>>      > I think this is equally true when we have normal BIER/MPLS enacp.
>>>      > Those packets too are addressed to the unicast MAC address of the
>>>      > neighbor.
>>>      >
>>>      > Now, if/when he controller recognizes that the physical topology
>>>      has
>>>      > changed, thats a completely different story and not addressed here.
>>>      > Given how we assumed this was a cabling mistake, the controller
>>>      would
>>>      > probably only complain about the miswiring to operations but be
>>>      happy
>>>      > that the forwarding plane just makes packets fail instead of
>>>      loop. If
>>>      > this was a planned change process, then it will be similarily
>>>      > convoluted as it would today be with rewiring cables in an
>>>      > SR-MPLS/SRv6 topology and updating SIDs.
>>>      >
>>>      > > > Because the forwarding is different from BIER forwarding
>>>      (because of [1] above), we might as well introduce an optimization
>>>      here ??? for each BIFT, calculate the F-BM of the BIFT itself (the
>>>      logical ???or??? of all the BPs presented in this BIFT) and then
>>>      use (packet->bitstring & BIFT.F-BM) as the input to
>>>      GetFirst/NextBitPosition(). That should skip many bits.
>>>      > >
>>>      > > Right. But i explicitly removed those optimizations (i had
>>>      them in older draft versions) because the whole idea of this
>>>      picture is solely the comparison with figure 4 of RFC8279.
>>>      > >
>>>      > > Zzh> I think it's worth point that optimization out; you can
>>>      mark it optional if you want to emphasize the similarity to BIER
>>>      forwarding, but since BIER forwarding does do the maskoff step, it
>>>      is very efficient while BIER-TE forwarding does not it the maskoff
>>>      step so this optimization is important.
>>>      >
>>>      > Ok. I simplified the text comparison BIER/BIER-TE wrt. to the FBM
>>>      > rules [1] and [2] and added following paragraph:
>>>      >
>>>      > <t>In BIER, the order of BPs impacts the result of forwarding
>>>      because of [1].
>>>      > In BIER-TE, forwarding is not impacted by the order of BPs. It is
>>>      > therefore possible to further optimize forwarding than in BIER. For
>>>      > example parallelizing forwarding across multiple FPE cores or
>>>      > distributed linecards does only need to examine an arbitrary
>>>      subset of
>>>      > BP and not evaluate the dependency between BPs.</t>
>>>      >
>>>      > > >    The following pseudocode is comprehensive:
>>>      > > >
>>>      > > > The above sentence reads a bit strange (or lacks some segue).
>>>      > >
>>>      > > I hope not, but maybe best left to a native english speaker
>>>      (RFC-editor).
>>>      > >
>>>      > > The first (RFC8279) pseudocode was simplified. The second one
>>>      is comprehensive. If not comprehensive, whats a good opposite of
>>>      simplified ?
>>>      > >
>>>      > > Zzh> Perhaps "The above simplified pseudocode is elaborated
>>>      further as following"?
>>>      > > Zzh> Jeffrey
>>>      >
>>>      > Done.
>>>      >
>>>      > Thanks a lot.
>>>      >
>>>      >
>>>      > >
>>>      > > > ________________________________________
>>>      > > > From: BIER [[e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail]
>>>      <mailto:[e-mail]>>]
>>>      > > > on behalf of Toerless Eckert [[e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail] <mailto:[e-mail]>>]
>>>      > > > Sent: Tuesday, July 09, 2019 23:38
>>>      > > > To: Mike McBride
>>>      > > > Cc: Greg Shepherd; BIER WG; Pascal Thubert (pthubert)
>>>      > > > Subject: Re: [Bier] WGLC - draft-ietf-bier-te-arch
>>>      > > >
>>>      > > > Thanks, Mike
>>>      > > >
>>>      > > > The authors also reviewed the document and concluded that it
>>>      was
>>>      > > > really hard to get into the document context because of too
>>>      many
>>>      > > > forward dependencies. We tried to fix this by adding two
>>>      hopefully
>>>      > > > good & basic examples into the Introduction section and
>>>      using them
>>>      > > > to also add a better definition of the term "BIER-TE
>>>      Topology" in the Introduction.
>>>      > > > Hopefully this makes readin the rest of te document smoother.
>>>      > > >
>>>      > > > Also improved text of Abstract and refined text compariing
>>>      BIER-TE with SR.
>>>      > > >
>>>      > > >
>>>      https://urldefense.com/v3/__http://tools.ietf.org/*rfcdiff?url1=https:
>>>      > > > **Atools.ietf.org
>>>      <http://Atools.ietf.org>*id*draft-ietf-bier-te-arch-02.txt&url2=https:**A
>>>      > > > tool
>>>      > > > s.ietf.org
>>>      <http://s.ietf.org>*id*draft-ietf-bier-te-arch-03.txt__;Ly8vLy8vLy8v!8WoA6R
>>>      > > > jC81
>>>      > > >
>>>      c!XvH4AAxfrDjFoK_sercwZMsc0O5N42eENOs4l_qdsXF0KwZD82cJLDFFNV_eTUEh
>>>      > > > $
>>>      > > >
>>>      <https://urldefense.com/v3/__http:/tools.ietf.org/*rfcdiff?url1=https:
>>>      > > > **Atools.ietf.org
>>>      <http://Atools.ietf.org>*id*draft-ietf-bier-te-arch-02.txt&url2=https:**A
>>>      > > > tool
>>>      > > > s.ietf.org
>>>      <http://s.ietf.org>*id*draft-ietf-bier-te-arch-03.txt__;Ly8vLy8vLy8v!8WoA6R
>>>      > > > jC81
>>>      > > >
>>>      c!UBTGvWWpMHyeiSanxs6vIb_EnBVgyg6boAAW4nrqju8UCLOgiuXc8Y_6sNd1njcX
>>>      > > > $>
>>>      > > >
>>>      > > > Cheers
>>>      > > >     Toerless
>>>      > > >
>>>      > > > On Wed, Jun 26, 2019 at 10:39:36AM -0700, Mike McBride wrote:
>>>      > > > > How about three? I support.
>>>      > > > > mike
>>>      > > > >
>>>      > > > > On Tue, Jun 25, 2019 at 10:42 AM Greg Shepherd
>>>      <[e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail]
>>>      <mailto:[e-mail]>>> wrote:
>>>      > > > > >
>>>      > > > > > We cannot take two 'yes' votes and WG consensus.
>>>      > > > > > Please, read and respond. If you don't support, then
>>>      please vote as much publicly right here.
>>>      > > > > >
>>>      > > > > > Thanks,
>>>      > > > > > Greg
>>>      > > > > >
>>>      > > > > > On Mon, Jun 3, 2019 at 10:05 PM Pascal Thubert
>>>      (pthubert) <[e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail]
>>>      <mailto:[e-mail]>>> wrote:
>>>      > > > > >>
>>>      > > > > >> Support:
>>>      > > > > >>
>>>      > > > > >> I see great value in deterministic networks as well as
>>>      IOT (with RPL).
>>>      > > > > >>
>>>      > > > > >> All the best,
>>>      > > > > >>
>>>      > > > > >> Pascal
>>>      > > > > >>
>>>      > > > > >> > -----Original Message-----
>>>      > > > > >> > From: BIER
>>>      > > > > >> > <[e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail]
>>>      <mailto:[e-mail]>>> On
>>>      > > > > >> > Behalf Of Toerless Eckert
>>>      > > > > >> > Sent: mardi 4 juin 2019 02:03
>>>      > > > > >> > To: Greg Shepherd
>>>      > > > > >> > <[e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail]
>>>      <mailto:[e-mail]>>>
>>>      > > > > >> > Cc: BIER WG <[e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail] <mailto:[e-mail]>>>
>>>      > > > > >> > Subject: Re: [Bier] WGLC - draft-ietf-bier-te-arch
>>>      > > > > >> >
>>>      > > > > >> > +1
>>>      > > > > >> > Obviously support as co-author.
>>>      > > > > >> >
>>>      > > > > >> > On Wed, May 29, 2019 at 12:41:26PM -0700, Greg
>>>      Shepherd wrote:
>>>      > > > > >> > > Please read and respond to this thread w/ or w/o
>>>      support.
>>>      > > > > >> > >
>>>      > > > > >> > >
>>>      https://urldefense.com/v3/__https://datatracker..ietf.org
>>>      > > > > >> > > /doc
>>>      > > > > >> > >
>>>      /draft-ietf-bier-te-arch/__;!8WoA6RjC81c!XvH4AAxfrDjFoK_s
>>>      > > > > >> > > ercw ZMsc0O5N42eENOs4l_qdsXF0KwZD82cJLDFFNV9eClBj$
>>>      > > > > >> > >
>>>      <https://urldefense.com/v3/__https:/datatracker.ietf.org/
>>>      > > > > >> > > doc/
>>>      > > > > >> > >
>>>      draft-ietf-bier-te-arch/__;!8WoA6RjC81c!UBTGvWWpMHyeiSanx
>>>      > > > > >> > > s6vI b_EnBVgyg6boAAW4nrqju8UCLOgiuXc8Y_6sD40kmtH$>
>>>      > > > > >> > >
>>>      > > > > >> > > Vote ends 5 June 2019.
>>>      > > > > >> > >
>>>      > > > > >> > > Thanks,
>>>      > > > > >> > > Shep
>>>      > > > > >> > > (chairs)
>>>      > > > > >> >
>>>      > > > > >> > > _______________________________________________
>>>      > > > > >> > > BIER mailing list
>>>      > > > > >> > > [e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail] <mailto:[e-mail]>>
>>>      > > > > >> > >
>>>      https://urldefense.com/v3/__https://www.ietf.org/mailman/
>>>      > > > > >> > > list
>>>      > > > > >> > >
>>>      info/bier__;!8WoA6RjC81c!XvH4AAxfrDjFoK_sercwZMsc0O5N42eE
>>>      > > > > >> > > NOs4 l_qdsXF0KwZD82cJLDFFNT2WVXWX$
>>>      > > > > >> > >
>>>      <https://urldefense.com/v3/__https:/www.ietf.org/mailman/
>>>      > > > > >> > > list
>>>      > > > > >> > >
>>>      info/bier__;!8WoA6RjC81c!UBTGvWWpMHyeiSanxs6vIb_EnBVgyg6b
>>>      > > > > >> > > oAAW 4nrqju8UCLOgiuXc8Y_6sKn2KoAT$>
>>>      > > > > >> >
>>>      > > > > >> > _______________________________________________
>>>      > > > > >> > BIER mailing list
>>>      > > > > >> > [e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail] <mailto:[e-mail]>>
>>>      > > > > >> >
>>>      https://urldefense.com/v3/__https://www.ietf.org/mailman/li
>>>      > > > > >> > stin
>>>      > > > > >> >
>>>      fo/bier__;!8WoA6RjC81c!XvH4AAxfrDjFoK_sercwZMsc0O5N42eENOs4
>>>      > > > > >> > l_qd
>>>      > > > > >> > sXF0KwZD82cJLDFFNT2WVXWX$
>>>      > > > > >> >
>>>      <https://urldefense.com/v3/__https:/www.ietf.org/mailman/li
>>>      > > > > >> > stin
>>>      > > > > >> >
>>>      fo/bier__;!8WoA6RjC81c!UBTGvWWpMHyeiSanxs6vIb_EnBVgyg6boAAW
>>>      > > > > >> > 4nrq
>>>      > > > > >> > ju8UCLOgiuXc8Y_6sKn2KoAT$>
>>>      > > > > >
>>>      > > > > > _______________________________________________
>>>      > > > > > BIER mailing list
>>>      > > > > > [e-mail]
>>>      <mailto:[e-mail]><mailto:[e-mail] <mailto:[e-mail]>>
>>>      > > > > >
>>>      https://urldefense.com/v3/__https://www.ietf.org/mailman/listi
>>>      > > > > > nfo/
>>>      > > > > >
>>>      bier__;!8WoA6RjC81c!XvH4AAxfrDjFoK_sercwZMsc0O5N42eENOs4l_qdsX
>>>      > > > > > F0Kw
>>>      > > > > > ZD82cJLDFFNT2WVXWX$
>>>      > > > > >
>>>      <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listi
>>>      > > > > > nfo/
>>>      > > > > >
>>>      bier__;!8WoA6RjC81c!UBTGvWWpMHyeiSanxs6vIb_EnBVgyg6boAAW4nrqju
>>>      > > > > > 8UCL
>>>      > > > > > OgiuXc8Y_6sKn2KoAT$>
>>>      > > >
>>>      > > > --
>>>      > > > ---
>>>      > > > [e-mail] <mailto:[e-mail]><mailto:[e-mail]
>>>      <mailto:[e-mail]>>
>>>      > > >
>>>      > > > _______________________________________________
>>>      > > > BIER mailing list
>>>      > > > [e-mail] <mailto:[e-mail]><mailto:[e-mail]
>>>      <mailto:[e-mail]>>
>>>      > > >
>>>      https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/
>>>      > > > bier
>>>      > > >
>>>      __;!8WoA6RjC81c!XvH4AAxfrDjFoK_sercwZMsc0O5N42eENOs4l_qdsXF0KwZD82
>>>      > > > cJLD
>>>      > > > FFNT2WVXWX$
>>>      > > >
>>>      <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/
>>>      > > > bier
>>>      > > >
>>>      __;!8WoA6RjC81c!UBTGvWWpMHyeiSanxs6vIb_EnBVgyg6boAAW4nrqju8UCLOgiu
>>>      > > > Xc8Y
>>>      > > > _6sKn2KoAT$>
>>>      > >
>>>      > > --
>>>      > > ---
>>>      > > [e-mail] <mailto:[e-mail]>
>>>      >
>>>      > --
>>>      > ---
>>>      > [e-mail] <mailto:[e-mail]>
>>>      >
>>>      > _______________________________________________
>>>      > BIER mailing list
>>>      > [e-mail] <mailto:[e-mail]>
>>>      >
>>>      https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bier
>>>      >
>>>      __;!!NEt6yMaO-gk!VuQCVHnqJy_aYI-FNt9A1a5EzHOCr0fZkLPbgg3CPNu0PyrWsrFx4
>>>      > 1_jWV3YUA6D$
>>>
>>>      --
>>>      ---
>>>      [e-mail] <mailto:[e-mail]>
>>>
>>>      _______________________________________________
>>>      BIER mailing list
>>>      [e-mail] <mailto:[e-mail]>
>>>      https://www.ietf.org/mailman/listinfo/bier
>>>
>> _______________________________________________
>> BIER mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/bier
>