Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

[Tom Herbert <[e-mail]>]

On Fri, Apr 5, 2019 at 4:40 PM Jai Kumar <[e-mail]> wrote:
>
> Tom,
>
>
>
> You are right. There are two main cases where accessibility of transport header becomes an issue. And that is mainly because of current generation of fixed pipeline cell based architecture parse depth (not so much an issue for high end NPU where next packet header bytes can be DMA’ed in, though it impacts the performance. For eg. Cisco QFP NPU, EZ Chip, Juniper’s MX 3D NPU, or Broadcom’s Jerricho NPU ). This was mentioned earlier by Mikael as well.
>
>
>
> Case 1. SRv6
>
> Case 2. IPv6 with EH (mainly for 5G deployments where RH, DH, and IFA Header)
>
>
>
> Typically NPUs provide capability that it will handle 3 or 4 EH or SRH without performance penalty. With the presence of IFA header this means that for eg there can be only 2 or 3 SRH or EH. This is same as in MPLS label stack depth handling (PUSH or POP at PE router). Given the evolution SRV6 and MPLS-SR kind of headers, silicon vendors have taken a notice of it and are evolving the silicon arch to handle more chained headers.
>
>
>
> IFA supports fragmentation of metadata stack and also postcard mode so beyond the transport header it’s not an issue.
>
>
>
> Just a quick feedback on using IPv6 options, inserting hop by hop options for data path processing pushes the EH further down and impacts the node’s forwarding function (say if it is not able to reach Destination EH) vs if IFA header is the last EH then atmost transit node do not perform metadata collection which can be identified.

Maybe it does impact it, maybe it doesn't. It clearly depends on the
device capabilities. For a device with a parsing buffer of 256 bytes
that should be plenty of room for HBH options, SR header, transport
layer, etc.

>
>
>
> -Jai
>
>
>
> From: Tom Herbert <[e-mail]>
> Date: Friday, April 5, 2019 at 3:34 PM
> To: Jai Kumar <[e-mail]>
> Cc: "Pengshuping (Peng Shuping)" <[e-mail]>, 6man <[e-mail]>, ippm <[e-mail]>, "C. M. Heard" <[e-mail]>, Mikael Abrahamsson <[e-mail]>
> Subject: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)
>
>
>
> Jai,
>
>
>
> You might want to consider how IFA works in the presence of segment routing. The second someone puts an SRH in the packet the IFA headers get pushed down and the transport headers end up being deep in the packet anyway so that all the effort of IFA is nullified. If the requirement is to ensure that the transport layer information appear as early as possible in the packet, one could define a Hop-by-Hop option to carry some sort of pseudo header for the transport layer. This also would work in the case that the actual transport layer is obfuscated.
>
>
>
> Tom
>
>
>
> On Fri, Apr 5, 2019, 3:09 PM Jai Kumar <[e-mail]> wrote:
>
> “That is going to require justifying the protocol even in light of shortcomings in existing protocoIs.”
>
>
>
> Agreed !!! True for any proposal.
>
>
>
> -Jai
>
>
>
> From: Tom Herbert <[e-mail]>
> Date: Friday, April 5, 2019 at 2:34 PM
> To: Jai Kumar <[e-mail]>
> Cc: "Pengshuping (Peng Shuping)" <[e-mail]>, 6man <[e-mail]>, ippm <[e-mail]>, "C. M. Heard" <[e-mail]>, Mikael Abrahamsson <[e-mail]>
> Subject: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)
>
>
>
>
>
> On Fri, Apr 5, 2019, 2:07 PM Jai Kumar <[e-mail]> wrote:
>
> Tom,
>
> Its all explained in the requirements section of the draft. IFA header is essentially treated as an EH both in IPv4 and IPv6 (very similar to AH/ESP EH).
>
> Just like in AH/ESP the IP protocol is copied in the AH header and IP protocol field is replaced in the IP header with the AH proto.
>
> There are several advantages to this approach.
> - devise know how to parse AH header and can follow very similar approach
> - Protocol agnostics, silicon need not support N encaps (just like AH/ESP). Single implementation works for all protocol types.
> - No need to creation options and teach devices that options is NOT an exception path processing
> - Really worry about ordering in IPv6 options
> - Note that all the OAM data need to be deleted as well so position need to be optimal for removal. We can always do tail stamping of the data but this puts considerable constrains on the cell based architecture
> - L4 accessibility I have already mentioned
>
> I am happy to discuss with you in person and explain more but bottom line is that this proposal is what came out with MSDC discussions in light of shortcomings from other IETF proposals.
>
>
>
> As they say, you are free to run whatever you want in your proprietary network, but if you expect to create an interoperable, deployable protocoI and get a protocol number assignment, you'll need to work in ietf. That is going to require justifying the protocol even in light of shortcomings in existing protocoIs.
>
>
>
> Tom
>
>
>
>
> Regards,
> -Jai
>
>
> On 4/5/19, 12:28 PM, "Tom Herbert" <[e-mail]> wrote:
>
>     On Fri, Apr 5, 2019 at 12:03 PM Jai Kumar <[e-mail]> wrote:
>     >
>     > " Great, but this IETF"
>     > Does it mean that drafts have no relevance to deployment and actual adoption.
>     >
>     > For all your queries please take a look at IFA draft.
>     > https://datatracker.ietf.org/doc/draft-kumar-ippm-ifa/
>     >
>     I've looked at that draft. It's is creating a complete new IP
>     protocolas well as a new packet format for TCP and UDP that somehow
>     inserts meta data between the TCP header and payload. I don't
>     understand why this could possibly be better than just using flow
>     label for ECMP and teaching devices that need to extract transport
>     information how to walk over EH, or just using some UDP encpasulation
>     like in draft-herbert-ipv4-udpencap-eh-01.
>
>     Tom
>
>     > It handles IPv6, IPv4 TCP/UDP with altering the packet path (due to inserting options and/or encaping them in GRE header) as well as IPSec AH/ESP/WESP both tunnel and transport and guess what it is successfully deployed.
>     >
>     > -Jai
>     >
>     > On 4/5/19, 11:54 AM, "Tom Herbert" <[e-mail]> wrote:
>     >
>     >     On Fri, Apr 5, 2019 at 11:25 AM Jai Kumar <[e-mail]> wrote:
>     >     >
>     >     > Tom,
>     >     >
>     >     > I am speaking based on the actual deployment not based on if all the traffic is/will move to IPSec tunnel.
>     >     >
>     >     > I have 6 large customer deployments and one someone previously called as "Yahoo" who insist on visibility in L4 even for IOAM packets.
>     >     >
>     >     And I've worked in datcenters much larger, and my first rule is that I
>     >     don't want random router vendors dictating to me what protocols I'm
>     >     allowed to use in my datacenter.
>     >
>     >     > I can quote you on that and lose my business :-)
>     >     >
>     >     > ECMP argument is weak as well as it works for IPv6 flow label. What about IPv4 TCP/UDP traffic which typically is 80% of the traffic in MSDC.
>     >     >
>     >
>     >     draft-herbert-ipv4-udpencap-eh-01
>     >
>     >     > If someone tells me that proposal will work only for my 20% of traffic then again as I said before, they will lose my business :-)
>     >     >
>     >     Great, but this IETF. Where are the protocol requirements? If you are
>     >     saying that there are requirements for "visibility in L4" what are
>     >     they? If you are saying that ACLs are now required for packet
>     >     forwarding, where is that described? Is it a MUST that teh only
>     >     transport protocols we're allowed to use are  UDP or TCP? Are we
>     >     allowed to use extension headers at all? I suppose fragmentation is
>     >     right out... Are we violating the requirements if we encrypt the
>     >     transport headers? If you say that we shouldn't send long headers
>     >     because we'll overflow parsing buffers, then what is the limit? Please
>     >     be SPECIFC in setting requirements; we are trying to build protocols
>     >     that work and are interoperable and we need to get out of this morass
>     >     of reverse engineering to discover the least common denominator.
>     >
>     >     Tom
>     >
>     >     > Regards,
>     >     > -Jai
>     >     >
>     >     > On 4/5/19, 11:15 AM, "Tom Herbert" <[e-mail]> wrote:
>     >     >
>     >     >     On Fri, Apr 5, 2019 at 10:55 AM Jai Kumar <[e-mail]> wrote:
>     >     >     >
>     >     >     > Even if we use flow label for ECMP, lack of accessibility of L4 information (for various services, most simple is ACL) makes the proposal pretty much unusable.
>     >     >     >
>     >     >     That is convoluting router and firewall functionality and
>     >     >     requirements. This a discussion about packet forwarding which does
>     >     >     _not_ require ACLs. In a limited domain, for which IOAM is intended,
>     >     >     it is unlikely that they'll ACLs would be used in internal nodes, but
>     >     >     we do need good ECMP routing at all intermediate nodes need to be
>     >     >     consistent rather or not that the IOAM option is present. As side
>     >     >     note, the ability to extract L4 information out of packets is
>     >     >     dwindling anyway thanks to encryption and other techniques trying to
>     >     >     undo protocol ossification. For instance, good luck getting L4
>     >     >     information out of an IPsec tunnel :-).
>     >     >
>     >     >     Tom
>     >     >
>     >     >     > NIC cards are not used pervasively for ACLs and other services, I believe iptables in kernel are good enough for host but that’s not acceptable in networking elements.
>     >     >     >
>     >     >     > -Jai
>     >     >     >
>     >     >     > On 4/5/19, 10:31 AM, "ippm on behalf of Tom Herbert" <[e-mail] on behalf of [e-mail]> wrote:
>     >     >     >
>     >     >     >     On Fri, Apr 5, 2019 at 9:16 AM Pengshuping (Peng Shuping)
>     >     >     >     <[e-mail]> wrote:
>     >     >     >     >
>     >     >     >     > Hi Barak,
>     >     >     >     >
>     >     >     >     > We are certainly not designing for the lowest capable solutions but exploring optimal solutions which could help to keep the forwarding performance while inserting the iOAM data with variable length.
>     >     >     >
>     >     >     >     Shuping,
>     >     >     >
>     >     >     >     That's exactly the argument for using the flow label in ECMP.
>     >     >     >     Efficeint packet forwarding can be done solely by inspected the forty
>     >     >     >     byte fixed length header of the packet regardless of the contents of
>     >     >     >     the rest of the packet. That is an optimal solution. Anything else for
>     >     >     >     ECMP like doing DPI or hacking up IOAM to try do pull transport layers
>     >     >     >     into the parsing buffer (whatever size that may be) are nothing more
>     >     >     >     than hacks and unnecessarily perpetuate techniques used with IPv4 that
>     >     >     >     either don't work or are unnecessary in IPv6.
>     >     >     >
>     >     >     >     Tom
>     >     >     >
>     >     >     >
>     >     >     >     >
>     >     >     >     > In terms of the forwarding efficiency, we would all agree that a short and fixed header would be better than a variable header.
>     >     >     >     >
>     >     >     >     > Best regards
>     >     >     >     > Shuping
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > 发件人： Barak Gafni<[e-mail]>
>     >     >     >     > 收件人： Pengshuping (Peng Shuping)<[e-mail]>;Frank Brockners (fbrockne)<[e-mail]>;Mikael Abrahamsson<[e-mail]>
>     >     >     >     > 抄送： C. M. Heard<[e-mail]>;6man WG<[e-mail]>;Mark Smith<[e-mail]>;ippm<[e-mail]>
>     >     >     >     > 主题： RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)
>     >     >     >     > 时间： 2019-04-05 03:18:26
>     >     >     >     >
>     >     >     >     > Hi Shuping,
>     >     >     >     > I think that the question is whether we design for the lowest capable solutions. I assume someone can find solutions with even lower amount of parsing.. The low end solutions for these capabilities may need to either adopt with new designs or use solutions such as what is suggested in this thread by Tom Herbert.
>     >     >     >     > Personally, I don't think we should go this way, but to design appropriate solutions, with appropriate layers in the headers. For example, prevent dependencies between "remote" headers.
>     >     >     >     >
>     >     >     >     > Thanks,
>     >     >     >     > Barak
>     >     >     >     >
>     >     >     >     > -----Original Message-----
>     >     >     >     > From: ippm <[e-mail]> On Behalf Of Pengshuping (Peng Shuping)
>     >     >     >     > Sent: Wednesday, April 3, 2019 6:47 AM
>     >     >     >     > To: Frank Brockners (fbrockne) <[e-mail]>; Mikael Abrahamsson <[e-mail]>
>     >     >     >     > Cc: C. M. Heard <[e-mail]>; 6man WG <[e-mail]>; Mark Smith <[e-mail]>; [e-mail]
>     >     >     >     > Subject: [ippm] 答复: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)
>     >     >     >     >
>     >     >     >     > Hi,
>     >     >     >     >
>     >     >     >     > The parsing depth of the early chips is 96B/128B. For the new chips the parsing depth has been increased but still limited. So Mikael's concern makes sense especially in the tracing mode that the IOAM data fields are going to increase significantly along the path, which will even push the Routing Header out of the parsing depth of the chip. So it would make sense to split the IOAM data part from the IOAM header/instruction part, and place the IOAM data after the RH or even after the L4 header in order to be able to read the L4 information to enable ECMP, as stated in the draft https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-li-6man-ipv6-sfc-ifit-00&amp;data=02%7C01%7Cgbarak%40mellanox.com%7C612ba6dc117440e68c4608d6b83ae174%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636898960379485833&amp;sdata=Dob4zN%2F1jSGxLTlqGma29JJNvqJDBt4VzqHsN4OpxM4%3D&amp;reserved=0.
>     >     >     >     >
>     >     >     >     > BR,
>     >     >     >     > Shuping
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > -----邮件原件-----
>     >     >     >     > 发件人: ippm [mailto:[e-mail]] 代表 Frank Brockners (fbrockne)
>     >     >     >     > 发送时间: 2019年4月2日 17:40
>     >     >     >     > 收件人: Mikael Abrahamsson <[e-mail]>
>     >     >     >     > 抄送: C. M. Heard <[e-mail]>; 6man WG <[e-mail]>; Mark Smith <[e-mail]>; [e-mail]
>     >     >     >     > 主题: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)
>     >     >     >     >
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > -----Original Message-----
>     >     >     >     > From: Mikael Abrahamsson <[e-mail]>
>     >     >     >     > Sent: Dienstag, 2. April 2019 08:06
>     >     >     >     > To: Frank Brockners (fbrockne) <[e-mail]>
>     >     >     >     > Cc: Mark Smith <[e-mail]>; C. M. Heard <[e-mail]>; 6man WG <[e-mail]>; [e-mail]
>     >     >     >     > Subject: RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)
>     >     >     >     >
>     >     >     >     > On Mon, 1 Apr 2019, Frank Brockners (fbrockne) wrote:
>     >     >     >     >
>     >     >     >     > > ...FB: There is obviously no easy answer. Couple of thoughts:
>     >     >     >     > > * IOAM is considered a "domain specific" feature (see
>     >     >     >     > > draft-ietf-ippm-ioam-data-05, section 3). Routers in the IOAM domain
>     >     >     >     > > should be IOAM capable.  And IMHO,  we should add a statement to
>     >     >     >     > > draft-ioametal-ippm-6man-ioam-ipv6-deployment that an implementation
>     >     >     >     > > of IOAM MUST ensure "C1".
>     >     >     >     > >
>     >     >     >     > > * That said, there can be situations, where C1 cannot be achieved, e.g..
>     >     >     >     > > consider a network which does ECMP scheduling based on packet length.
>     >     >     >     > >
>     >     >     >     > > * What one could consider - and which is one suggested deployment
>     >     >     >     > > model
>     >     >     >     > > - is that by default IOAM data fields are added to _all_ customer
>     >     >     >     > > packets. The decapsulating node would decide whether the IOAM
>     >     >     >     > > information contained in a packet would be used (and exported) or not.
>     >     >     >     > > That way one would not need to deal with the situation that some
>     >     >     >     > > traffic carries IOAM while other does not - and might thus be treated
>     >     >     >     > > differently.
>     >     >     >     >
>     >     >     >     > Yes, I think this is the only way. Is there a risk that intermediate routers would not be IOAM capable? I think the C1 requirement is really really hard to fulfil and I'm also afraid that adding the IOAM header will actually make ECMP stop working on some platforms (because they would not have the capability to look deep enough into the packet to find L4 information, so it'll go back to 2 tuple ECMP instead of 5 tuple.
>     >     >     >     >
>     >     >     >     > But this question can only be answered by people with deep NPU knowledge...
>     >     >     >     >
>     >     >     >     > .....FB2: Given that IOAM is a domain specific feature - I expect that folks initially start to use it in domains (like e.g. a DC) where all boxes are IOAM capable and can trace the packet appropriately - or per Tom's note, can be configured so that one would do ECMP based on addresses and flow-label.  There are chips with pretty deep parsing capability (up to a few hundred bytes).
>     >     >     >     >
>     >     >     >     > > ...FB: The comparison to MPLS is interesting. How often do MPLS
>     >     >     >     > > packets leak and cause harm? For IOAM,
>     >     >     >     > > draft-ioametal-ippm-6man-ioam-ipv6-options-02 proposes a deployment
>     >     >     >     > > model similar to what we do for MPLS: Unless an interface is
>     >     >     >     > > explicitly configured for IOAM, packets with IOAM data fields MUST be dropped.
>     >     >     >     > > Hence leakage would only occur, if we have a clearly misbehaving
>     >     >     >     > > router which violates this rule. Similar to you, I'd also greatly
>     >     >     >     > > appreciate any pointers to research on how common MPLS leaked packets are.
>     >     >     >     >
>     >     >     >     > When it comes to "cause harm" I imagine there are (at least) two ways to cause harm, one is privacy/secrecy/security loss and the other one is actual operational loss.
>     >     >     >     >
>     >     >     >     > I know of bugs where labeled packets went the wrong way and caused them to be lost, I've also seen bugs where bugs caused traffic to "hop" between VRFs in MPLS VPN (or to "global" VRF). I don't have numbers on this though.
>     >     >     >     >
>     >     >     >     > Depending on the deployment scenario, it might make sense to make IOAM packets not valid for non-IOAM aware devices (basically encap entire packet/payload), but that might be a problem for intermediate non-IOAM nodes then. This would affect the ECMP requirement. I can see cases where one would operationally turn on IOAM for some customers traffic and then see the problem go away because now ECMP changed.
>     >     >     >     >
>     >     >     >     > > ...FB: One idea that Shwetha came up with to identify the source AS of
>     >     >     >     > > a leaked packet, would be to add a new new IOAM E2E option - as
>     >     >     >     > > proposed in section 5.1.1 bullet 2 of
>     >     >     >     > > draft-ioametal-ippm-6man-ioam-ipv6-deployment-01.
>     >     >     >     >
>     >     >     >     > Yes, that'd solve that problem.
>     >     >     >     >
>     >     >     >     > How much has the silicon people been involved so far in the design of the headers? What is the current thinking of amount of data going into the IOAM header? Considering things like trace options etc it seems to me the header can grow quite large? To satisfy the ECMP requirement what about putting the IOAM information as a trailer behind the regular payload? Or is there a problem for the hw to manipulate information that far into the packet (I also imagine this will considerably lower the forwarding performance of IOAM packets on IOAM aware platforms).
>     >     >     >     >
>     >     >     >     > .....FB2: There are quite a few folks with hardware background that co-author the IOAM drafts. Parsing capability differs between chips, as does the ability to deal with new/flexible headers and the ability to modify data in the extension headers. So the unfortunate answer to that question is "it depends" (what information do you gather, how large is your domain, what are the capabilities of your hardware/software forwarder, etc.), but I do expect that for specific deployment domains (e.g. DC, Enterprise) - but also for overlay networks / VPNs - we'll see an increasing amount of chips that are well suited for processing large extension header.
>     >     >     >     >
>     >     >     >     > Cheers, Frank
>     >     >     >     >
>     >     >     >     > --
>     >     >     >     > Mikael Abrahamsson    email: [e-mail]
>     >     >     >     >
>     >     >     >     > _______________________________________________
>     >     >     >     > ippm mailing list
>     >     >     >     > [e-mail]
>     >     >     >     > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fippm&amp;data=02%7C01%7Cgbarak%40mellanox.com%7C612ba6dc117440e68c4608d6b83ae174%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636898960379485833&amp;sdata=VjZ1PiwifYGCf159je0yef3BJFz6AQNNAtkgmAMDFKM%3D&amp;reserved=0
>     >     >     >     > _______________________________________________
>     >     >     >     > ippm mailing list
>     >     >     >     > [e-mail]
>     >     >     >     > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fippm&amp;data=02%7C01%7Cgbarak%40mellanox.com%7C612ba6dc117440e68c4608d6b83ae174%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636898960379485833&amp;sdata=VjZ1PiwifYGCf159je0yef3BJFz6AQNNAtkgmAMDFKM%3D&amp;reserved=0
>     >     >     >     > --------------------------------------------------------------------
>     >     >     >     > IETF IPv6 working group mailing list
>     >     >     >     > [e-mail]
>     >     >     >     > Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>     >     >     >     > --------------------------------------------------------------------
>     >     >     >
>     >     >     >     _______________________________________________
>     >     >     >     ippm mailing list
>     >     >     >     [e-mail]
>     >     >     >     https://www.ietf.org/mailman/listinfo/ippm
>     >     >     >
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>