Re: Benjamin Kaduk's Discuss on draft-ietf-bfd-vxlan-09: (with DISCUSS and COMMENT)

[Benjamin Kaduk <[e-mail]>]

Hi Jeff,

I think I can only touch on a few points before telechat-time rolls around,
and will finish off afterwards.

On Wed, Dec 18, 2019 at 03:24:48PM -0500, Jeffrey Haas wrote:
> Benjamin,
> 
> On Mon, Dec 16, 2019 at 03:43:13PM -0800, Benjamin Kaduk via Datatracker wrote:
> > Benjamin Kaduk has entered the following ballot position for
> > ----------------------------------------------------------------------
> > DISCUSS:
> > ----------------------------------------------------------------------
> > 
> > I have a few points that I think merit IESG discussion.
> > 
> > (1) I see that several directorate reviewers expressed unease at the
> > destination (IP and) MAC address assignment procedure for the inner
> > VXLAN headers, and appreciate that there was extensive on-list
> > discussion (more than I could follow).  That said, I failed to find a
> > clear statement of why the current text is believed to be safe, and in
> > fact my reading of the current text is that the described procedure is
> > *not* safe.  Pointers to key parts of the WG discusison would be more
> > than welcome!
> 
> One high level point that likely didn't survive the rather verbose comment
> chain is there are two implementations of this draft.  Some of the
> considerations covered in the guidance here is "please don't break shipping
> code".
> 
> While this is IETF, and shipping code isn't always a blocking point to
> document changes, I'd suggest that as a consideration.

It did indeed not survive (at least my pass through) the comment chain, so
thank you for calling it out.  It is indeed a consideration, and I expect
some actual discussion on the call tomorrow.

> > To take something of a high-level view of my concerns, if we think of
> > the VXLAN as being a tunnel between VTEPs that carry encapsulated tenant
> > traffic, then what we're trying to do is roughly like BFD between VTEPs,
> > but we want to get fault-detection over as broad a coverage as we can
> > (the "outermost part of the tunnel"), so we want to have the option of
> > per-VNI BFD instead of just endpoint-to-endpoint (VTEP-to-VTEP).
> 
> You've summarized this clearly.  Joel Halpern, in particular, raised this
> point multiple times.  Effectively, "what are we testing?"  And the response
> not clearly converging on exactly one of the two possibilities.
> 
> As is noted in the various IESG discussion, each of the two test points
> raise slightly different considerations.
> 
> > However, we end up having to do this by trying to insert a thin filter
> > into the tenant's address space (i.e., the inner VXLAN header) and pick
> > out the specific stream of BFD traffic that we're introducing.  This is,
> > in some sense, a namespace grab in what is conceptually the tenant's
> > namespace, and we have to be careful that what we do is either
> > guaranteed to not impact the tenant or well-documented and
> > compartmentalized (akin to the "well-known URIs").
> 
> Possibly, and it's certainly a consideration.  However, I think I'm less
> convinced of it being quite the level of violation that seems to be
> reflected in the rest of the IESG comments in the various other threads.
> I'll respond to that detail a bit below.
> 
> > I've made comments at several places in the document that are more
> > directly tied to specific pieces of text, but in general, if we assume
> > that the tenant can add/remove new addresses at will within their VXLAN
> > abstration, then any attempt to preconfigure by mutual agreement the BFD
> > addresses to use at the VTEPs or to use the VTEP's normal (outer)
> > address as the sentinel value seems subject to the tenant coming in and
> > subsequently trying to use that address, leading to (some of) the
> > tenant's traffic getting silently filtered and interpreted by the VTEP.
> > If we were using domain names as identifiers, we could allocate
> > something under .arpa or similar, but I think our options are more
> > limited when numerical addresses are used.
> > 
> > The option suggested by the rtg-dir reviewer of always using the
> > management VNI does not suffer from this namespacing issue, though I
> > recognize that it does reduce the scope over which fault-detection is
> > available, for the cases when different VNIs' traffic are routed or
> > handled differently.
> 
> This is a clean summary of the considerations.  At least a portion of the WG
> seems to be comfortable with "test to the management VNI".  However, another
> (smaller, I believe) portion were wanting to test one layer further in.

It is reassuring that I at least managed to summarize the situation
tolerably.  Is it fair to say that testing "one layer further in" is a
superset of what "test to the managemenet VNI" can do?

> > (2) Section 6 says:
> > 
> >                                                          The selection
> >    of the VNI number of the Management VNI MUST be controlled through
> >    management plane.  An implementation MAY use VNI number 1 as the
> >    default value for the Management VNI.  All VXLAN packets received on
> >    the Management VNI MUST be processed locally and MUST NOT be
> >    forwarded to a tenant.
> > 
> > It seems like the management VNI concept is something that would apply
> > to the entire VXLAN deployment and not just to the BFD-using portions;
> > is this already defined somewhere (in which case we should reference
> > it), or is it new with this document?  In the latter case wouldn't it be
> > an update to the core VXLAN spec?  (I note that there are some
> > procedural hoops to jump through for an IETF-stream document to update
> > an ISE-stream document...)
> 
> The relevant portion of the archive will have the Subject: line text
> including:
> "Trapping BFD Control packet at VTEP"
> 
> A portion of the discussion relating to the magic number of the management
> VNI suggested '1', instead of '0'.
> 
> At least some implementations already use '0':
> https://mailarchive.ietf.org/arch/msg/rtg-bfd/6WfSATmfoPv4AD6RmD-Xb7zz4CE
> 
> The argument to not use '0' starts roughly here:
> https://mailarchive.ietf.org/arch/msg/rtg-bfd/z8E_a5k_r4pLLs5YfNsL_Xm9_Us
> 
> You're correct, IMO, that there's no standard practice and the above seems
> to support this.  I believe this leaves the document authors in the position
> of being requested to make a recommendation for the default value of this
> field and knowing that the default would be invalid on some platforms.
> 
> The alternative is requiring implementations to always configure this value.
> 
> I suggest the IESG determine whether it wants a default value here or not.
> If not, the text should be adjusted to require configuration.  If yes, the
> IESG should consider whether the nvo3 group should produce some document
> that covers current operational practices.

That does sound like something we should try to talk about on the telechat
as well; thanks for raising it so clearly.

> > ----------------------------------------------------------------------
> > COMMENT:
> > ----------------------------------------------------------------------
> >    0:0:0:0:0:FFFF:7F00:0/104 range for IPv6).  There could be a firewall
> >    configured on VTEP to block loopback addresses if set as the
> >    destination IP in the inner IP header.  It is RECOMMENDED to allow
> >    addresses from the loopback range through a firewall only if it is
> >    used as the destination IP address in the inner IP header, and the
> >    destination UDP port is set to 3784 [RFC5881].
> > 
> > I think we should reword this to make it clear that the default behavior
> > is still "block all incoming traffic with loopback destination" and that
> > the exception is tightly scoped to the encapsulated VXLAN traffic
> > discussed in this document and the specific destination port *and when
> > BFD has been configured for the VTEP*.  I note that well-known ports are
> > not reserved ports, and we have no guarangee that only a BFD
> > implementation would be listening on port 3784.
> 
> I don't think this consideration is necessarily critical.

I think I'm in agreement about its criticality, and will see if I can come
up with some actual text ... later.

-Ben

> BFD implementations residing in the related instance communicating to other
> instances across the vxlan environment would be using RFC 5881 or RFC 5883
> style BFD.  Since this isn't a tunneled BFD, the IP endpoints of the BFD
> control traffic will be unicast addresses rather than the reserved
> "loopback" ranges; i.e. 127/8 ::FFFF:127.0.0.0/104.  In order for those
> ranges to be problematic, it'd be necessary for the client to be able to
> manually encapsulate a vxlan packet - a security issue of its own.
> 
> A related point in this discussion is "we're hijacking an address managed by
> the local tenant".  While true, it's in the above ranges and thus somewhat
> under the auspice of the host OS to assert control.  I'm aware of some
> unusual applications that make use of configured addresses in those ranges
> for on-box communications, but they're also on the unusual end of things.
> 
> What sort of text would you want to cover the case that when BFD is run
> up-to-the-tenant mode in this circumstance that an address MUST be reserved
> for the BFD over vxlan application and that this address SHALL NOT be
> available to the tenant for its own use?
> 
> >    VXLAN packet.  The choice of Destination MAC and Destination IP
> >    addresses for the inner Ethernet frame MUST ensure that the BFD
> >    Control packet is not forwarded to a tenant but is processed locally
> >    at the remote VTEP.  [...]
> > 
> > This has to be 100% reliable, and I think we need to provide some
> > example mechanism that has that property even if we don't mandate that
> > it be the only allowed mechanism.
> 
> The consideration here, I believe, is that there's currently too much
> latitude by implementations as to what MAC addresses they use here.
> Restrict one case, you may break some implementation.
> 
> The missing element is how a pair of implementations of BFD for vxlan
> discover the necessary information?  As far as BFD is concerned, "tell me!"
> This seems like work that belongs in nvo3.
> 
> >          Destination MAC: This MUST NOT be of one of tenant's MAC
> >          addresses.  The destination MAC address MAY be the address
> > 
> > But the tenant can start using new MAC addresses at any time!  How is
> > BFD-over-VXLAN going to dynamically detect and avoid that?
> 
> See above.  Either it's coordinated with the ability to prevent the tenant
> from using it or the underlying vxlan environment needs to provide some
> mechanism to discover what's been provisioned.
> 
> >          associated with the destination VTEP.  The MAC address MAY be
> >          configured, or it MAY be learned via a control plane protocol.
> >          The details of how the MAC address is obtained are outside the
> >          scope of this document.
> > 
> > This all talks about the MAC address being relatively static
> > configuration, but per above, I don't think that's safe in the face of a
> > MUST-level requirement to avoid conflicting with tenant MAC addresses.
> 
> But is it BFD's responsibility to figure this out?  This is what the
> document is suggesting - a higher level with access to the implementation
> specifics should be supplying the BFD provisioning information.  Or manual
> provisioning in the absence thereof.
> 
> -- Jeff