Re: [nvo3] Review of draft-dt-nvo3-encap-01

[Sami Boutros <[e-mail]>]

Hi Tom,


Please find some comments inline beside DT:


Thanks,


Sami

FWIW here is some feedback on that draft.



Comments on sections I looked at indicated by '-'.



2. Design Team Goals



   As communicated by WG Chairs, the design team should take one of the

   proposed encapsulations and enhance it to address the technical

   concerns. Backwards compatibility with the chosen encapsulation and



- "Backwards compatibility" is at best a weak goal. As written:

"Internet-Drafts have no formal status, and are subject to change or

removal at any time; therefore they should not be cited or quoted in

any formal document.". Maintaining compatibility with an Internet

draft cannot be a requirement of standard protocol.



DT: Those points were also communicated by the WG chairs. We agree

the Backwards compatibility can be removed.





   the simple evolution of deployed networks as well as applicability to

   all locations in the NVO3 architecture are goals. The DT should

   specifically avoid a design that is burdensome on hardware

   implementations, but should allow future extensibility. The chosen



- I still don't understand all this focus on hardware. An nvo3

protocol is not a hardware nor a software protocol. Jumping through

hoops to make hardware implementation better at the expense of

software is not a reasonable trade-off.



DT: This has been a sentiment expressed on the list and multiple

IETF meetings. We are reflecting this as a requirement based on those

discussions as well to encourage HW implementation. Having said that

the DT has addressed the usage models while considering the requirements

and implementations in general that includes software and hardware.





   design should also operate well with ICMP and in ECMP environments.

   If further extensibility is required, then it should be done in such

   a manner that it does not require the consent of an entity outside of

   the IETF.



5.2 GUE



   - There were a significant number of objections related to the

   complexity of implementation in hardware, similar to those noted for

   Geneve above.



- The objections for complexity of hardware implementation raised for

GUE are not remotely similar to those raised for Geneve, the mechanism

of extensibility is completely different. This objection was

addressed.



DT: This feedback *also* came from the WG chairs. Out of curiosity, can

you summarize the objections? And how they were addressed?



   - In addition, there were concerns raised that GUE does not support a

   sufficient number of extensions due to its reliance on a limited

   flags field, which is already almost 45% allocated.



- As I mentioned in a rebuttal to this objection the flag-fields can

be extended with for flags. This has already been implemented, the

objection is not valid.



DT: We are documenting the concerns based on what is already out there –

a mailing list rebuttal or how it can be sorted out with “this has

already been implemented” doesn’t sound very helpful. The GUE draft

mention that "the set of available flags may be extended in the future

by defining a "flag extensions bit" that refers to a field containing

a new set of flags." Is that what you refer too as addressed

the objection?





6.5 Extension Ordering



   In order to support hardware nodes at the tunnel endpoint or at the

   transit that can process one or few extensions TLVs in TCAM. A

   control plane in such a deployment can signal a capability to ensure

   a specific TLV will always appear in a specific order for example the

   first one in the packet.



- I do not believe this is at all plausible. 1) This could only help

the endpoints and not intermediate devices. As point out two sentences

below transit nodes may need to process extensions. 2) This creates an

a dependency between data plane and control plane that is at odds with

the requirement for control plane independence (section 2.1 of Geneve

draft) 3) This would entail a serious design and implementation effort

that would likely only be ready long after the dataplane has been

deployed. 4) This creates new problems for interoperability, for

instance two devices could support same set of options but can't

interoperate because they need different orderings.



- Btw along these lines from the Geneve I noticed:



"Transit devices MUST maintain consistent forwarding behavior

   irrespective of the value of 'Opt Len', including ECMP link

   selection.  These devices SHOULD be able to forward packets

   containing options without resorting to a slow path."



- Making this requirement a SHOULD opens the door for devices to slow

path all packets with options and hence ossify the protocol in exactly

the same way that IP options were. It would not surprise me at all if

Geneve is already ossified so that options will never be deployed.

This really needs to be a MUST, but even so that probably won't

prevent vendors from throwing packets with options in the slow path.



DT: You can’t have it both ways – if you think “you can’t

understand all this focus on hardware” then why is this slow path etc.

a big issue? That is a function of the capability of the hardware – it

can become better in a few years and move the processing to the fast path.



   The order of the TLVs should be HW friendly for both the sender and

   the receiver and possibly the transit node too.



- This is exceedingly weak statement. If ordering is important then

just define a global ordering and dispense with all this hand waving

about a control plane solution and friendliness to everyone. Given

that the type space of Geneve TLVs is twenty-four, sparse assignment

of type values allows new options to be placed an appropriate order

relative to existing options.



DT: Control plane negotiated options and ordering is the most

flexible way to do this. We should consider writing a working group

draft to provide examples/guidance on this. Range of usage models and

deployments scenarios drive specific options and ordering that are

relevant for that specific deployment. This includes end points and

middle boxes using the options. So having the control plane negotiate

the constraints is most appropriate and flexible way to address these

requirement. Hardcoding the options and/or ordering would limit the

applications.





   A transit node may need to process some extensions like telemetry

   and/or OAM inband extensions.



- See comment above why this breaks TLV order negitation.



6.6 TLV vs Bit Fields



- Up front I will reiterate my previous point that I have made several

times now: _NO_ Geneve TLVs have been proposed. No TLVs have been

implemented and the AFAIK the required processing loop has not been

implemented. There are proposed bit-fields in GUE, at least one has

been implemented and deployed, and the core processing for bit-fields

has been implemented in software. All the discussion in these drafts

pertaining to TLVs and their benefits is completely academic!



DT: This is not true, there are implemented Geneve options by Vmware

today in production. There are as well new HW supporting Geneve TLV

parsing. In addition Inband Telemetry (INT) specification being

developed by P4.org illustrates the option of INT meta data carried

over Geneve.  OVN/OVS have also defined some option TLV(s) for Geneve.



   If there is a well-known initial set of options that are likely to be

   implemented in software and in hardware, it can be efficient to use



- There is such a set. There are described in draft-herbert-gue-extensions-01



   the bit-field approach as in GUE. However, as described in section

   6.3, if options are added over time and different subsets of options

   are likely to be implemented in different pieces of hardware, then it





   would be hard for the IETF to specify which options should get the

   early bit fields.



  TLVs are a lot more flexible, which avoids the need



- Yes, they are more flexible. In fact, as currently defined we can

define up to 16M TLVs each of which can be variable length. But _why_

do we need this? What are there requirements here? Most of the rest of

this section is trying to deal with the problems this "flexibility"

creates in the first place (limiting size, alignment, a new control

plane function to enforce order, etc.)





   to determine the relative importance different options. However,

   general TLV of arbitrary order, size, and repetition of the same

   order is difficult to implement in hardware. A middle ground is to

   use TLV with restrictions on the size and alignment, observing that

   individual TLVs can have a fixed length, and support in the control

   plane such that an NVE will only receive options that to needs and

   implements. The control plane approach can potentially be used to

   control the order of the TLVs sent to a particular NVE. Note that

   transit devices are not likely to participate in the control plane

   hence to the extent that they need to participate in option

   processing they need more effort,



- Which is a major problem with the whole control plane idea.



DT: We are not saying we want to limit the flexibility because of

the transit node, transit node will process a small subset of options

that will be consumed by tunnel endpoints.

The WG should consider developing a separate draft on guidance for

option processing and control plane participation.



   But transit devices would have

   issues with future GUE bits being defined for future options as well.



- That is not true. New options are added to the end of the flags so

this would not affect the the transit device processing of options it

knows about.





   A benefit of TLVs  from a HW perspective is that they are self

   describing i.e., all the information is in the TLV. In a Bit fields

   approach the hardware needs to look up the bit to determine the

   length of the data associated with the bit through some separate

   table, which would add hardware complexity.



- Yes, looking up the length of a bit field does require some

complexity, but this is a simple table lookup with a small number as

index. This pales in comparison to the lookup over a 24 bit type value

in the Geneve TLV.  And there is additional cost to verify that the

length in the TLV is appropriate for the type.



DT: The 24 bits consist of 16 bits class and 8 bits type,  so it is

not a 24 bits type. The ordering of TLV(s) can benefit from the split

of class/type.



   There are use cases where multiple modules of software are running on

   NVE. This can be modules such as a diagnostic module by one vendor

   that does packet sampling and another module from a different vendor

   that does a firewall. Using a TLV format, it is easier to have

   different software modules process different TLVs, which could be

   standard extensions or vendor specific extensions defined by the

   different vendors, without conflicting with each other. This can help

   with hardware modularity as well.



- This is weak, real implementation experience would be nice.



DT: There are some implementations with options that allows different

software like mac learning and security handle different options.



- Here are things that this section failed to address:



- The combinatorics of TLVs and sequential processing requirements are

hard to make efficient in both software and hardware implementations.

Bit-fields do not have this problem



DT:  This is not true according to the HW vendors presented in the

design team.



- Open ended TLVs, especially with the possibility of receiving ones

that can be ignored are a DOS vector.



DT: The ordering, what TLV(s) should be present and what can be

processed by which node transit/tunnel endpoint can address this.



- A survey of actual implementation of the protocols. Remember it's

"rough consensus and running code"-- Geneve is short on the running

code.



DT: Geneve is already running in shipping products.



- The rationale for a 24 bit type and cost of processing 24 bit type

fields. Deriving an expected rate of adding new extensions is not

difficult based on experience with other extensible dataplane

protocols. This will probably at most be one or two a year.

- Random access of options, for instance consider a device is trying

to find a specific option in a long list

- A comparison of Geneve TLVs to IP options, IPv6 options, or some

other protocol. Specifically, I would like to know why we should

believe Geneve would not suffer the same fate of protocol ossification

that those did.



DT: The argument that if IPv4 failed to process options in fast

path then Geneve will fail too, doesn’t sound right. IPv6 EHs for

SR are being defined and those will be processed by hardware platforms.

Also, there are hardware out there which can actually process options up

to a limited size.



1. We studied whether VNI should be in base header or in extensions

   and whether it should be 24-bit or 32-bit. The design team agreed

   that VNI is critical information for network virtualization and MUST

   be present in all packets. Design team also agreed that 24-bit VNI

   matches the existing widely used encapsulation format i.e. VxLAN and

   NVGRE and hence more suitable to use going forward.



- As I've stated before, there is simply no technical rationale behind

a 24 bit VNI. There is no reason to believe this is sufficient to

scale for large deployments over the lifetime of the protocol. Also,

as stated above, requiring compatibility of a standard protocol with a

draft is inappropriate. Just because this was 24 bits in VXLAN and

NVGRE and there may have been some deployment does not validate the

protocol element. The VNI should simply be extended to occupy 32 bits.

(btw if you don't do this then it is likely that the eight spare bits

will either be commandeered to extend the VNI or used for other some

purpose, in either case these unreserved eight bits will be abused and

create non-interoperability).



- The counter argument to this is that 32 bits is not enough, for

instance we might want to merge to large cloud providers and not force

them to renumber. That's why the VNI should itself _be_ an extension

so that the VNI is extensible. This has a huge advantage that it would

make at least one extension required for operation of the protocol

such that intermediate devices cannot ossify it. I suppose the counter

argument is that it's somehow too important of a value to and needs to

be accessed quickly so that we can't entrust it to the extension

mechanism, but then if we're not willing to commit the VNI to be an

extension why would we be willing to put anything in an extension?



DT: We are not against having the VNI in an extension (option) if

VNI value need more bits beyond the 24 bits in the header.



4. We compared the TLV vs Bit-fields style extension and it was

   deemed that parsing both TLV and bit-fields is expensive and while

   bit-fields may be simpler to parse, it is also more restrictive and

   requires guessing which extensions will be widely implemented so they

   can get early bit assignments for efficiency, as well Bit-fields are



- I don't understand this "early bit assignments" problem. Bit-field

allow easy random access to fields there is no need for sequential

processing. Please clarify this.



DT: Given that half the bits are already assigned in GUE, you may

need to set the bit for a widely deployed extension in a flag extension,

and this will require extra processing, to dig the flag from the flag

extension and then look for the extension itself.



- Also, I would advise the design team to be careful with use of the

word "efficiency" as applied to other protocols than one being

advocated. If you're going to claim someone else's protocol is then

you need to be prepared with the data to back this up.



   not flexible enough to address the requirement of variable length and



- What precisely is the requirement for variable length?



DT: Requirements came from OAM, Telemetry and even security extensions,

all require variable length option.



   different subtypes of the same option. While TLV are more flexible, a

   control plane can restrict the number of option TLVs as well the

   order and size of the TLVs to make it simpler for a dataplane

   implementation to handle.



- If this control plane idea doesn't go away I, for one, would really

likely to see the draft that describes _precisely_ how this will work.



DT: Agreed. And this why we need a control plane draft that

discuss this.