Re: [rbridge] merging draft-tissa-trill-oam anddraft-ietf-trill-rbridge-oam

[Sam Aldrin <[e-mail]>]

Inline for my reply.

Sent from my iPhone

On Mar 15, 2012, at 2:48 PM, Santosh Rajagopalan <[e-mail]> wrote:

> I think that we're starting to talk past each other. May I suggest taking each of the ping/traceroute situations on a case-by-case basis to see how each proposal handles it? 
Draft is meant for that. If you have specific case where you feel it doesn't solve it, ask it away. Will be glad to answer them. Last thing we should do is to make it a design document. 
> 
> Basically, can one of the authors summarize the approach in draft-tissa-trill-oam for ping/traceroute to discover: 
> 
> a. multidestination unpruned trees. 
> 
> b. multidestination pruned trees (per multicast group). 
> 
> c. tracing the path taken by a real multicast flow. 
> 
> d. unicast path discovery for each multipath 
> 
> e. unicast path discovery for real flow. 
> 
> I understand this will be a restatement of information that's present in draft-tissa-trill-oam, and I apologize for that. I'm looking for a more concise presentation of the approach. I am having trouble arranging the information in draft-tissa-trill-oam into the categories I've suggested above. You can use as a template the email I sent out originally starting this thread. 
We plan to to present at ietf, again. Hopefully we can clarify them, there. As the status is still draft,we plan to add more content. We will definitely like to add data, if not present, to care for majority of the scenarios if not all.
> 
> As for the discussion below about whether properties should be queried via SNMP or ICMP OAM, I would simply suggest that we make a choice one way or the other. After all, if you add the ability to query mac addresses, DRBs etc to OAM (and these are already present in draft-ietf-trill-rbridge-mib), then why stop there? Why not incorporate everything in the SNMP MIB into the ICMP query framework?
Why just stop with mib?
> After all, if SNMP is truly "expensive and intensive", why stop with DRB queries? I would then suggest doing away with the trill mib and using icmp as the query mechanism for all properties.
You should ask mib authors that question. If you go by your logic, if snmp is present, we should not attempt any with other technology, including net conf.
> Really, I'm ok either way, but I think we should pick one so that people don't end up implementing the same thing twice. 
The goal is not to duplicate the effort, but to get something operationally useful to fulfill oam functions. We feel differently to what you think. It is not a magic wand trying to make others vanish, rather give additional tools to manage networks.


Sam
> 
> -- 
> Sunny Rajagopalan 
> 
> 
> 
> 
> From:        Sam Aldrin <[e-mail]> 
> To:        Santosh Rajagopalan/Santa Clara/IBM@IBMUS 
> Cc:        "[e-mail]" <[e-mail]> 
> Date:        03/13/2012 11:59 AM 
> Subject:        Re: [rbridge] merging draft-tissa-trill-oam        anddraft-ietf-trill-rbridge-oam 
> Sent by:        [e-mail] 
> 
> 
> 
> See inline for my response to your questions. 
> 
> -sam
> 
> Sent from my iPad 
> 
> On Mar 12, 2012, at 3:06 PM, Santosh Rajagopalan <[e-mail]> wrote:
> 
> 1) Let's say that you are trying to find all possible paths between two rbridges (or liveliness) for a)unicast, b) multicast (unpruned) and c) multicast (pruned). My proposal  below will let you discover all three, without needing IP.  Since we're using the rbridge channel ethertype, rbridges can trap these and prevent them from getting to hosts at the edge. Note that we rely on unicast forwarding of rbridge channel packets for unicast discovery and multicast forwarding of rbridge channel packets for multicast discovery. This happens naturally in any trill compliant switch. 
> %sam - Discovering all of the paths is one part and discovering a path a specific flow takes is another problem. What we are trying to say is channel draft does not allow the full flexibility, for Unicast and multicast combinations. As was answered in the original email by Tissa, that multicast pruning happens with IP VLAN combinations with channel drafts such things are not possible. IP invalid checksum method has danger of triggering invalid packet counters  and lead to administration nightmare in hosting providers. We encourage you to check this with providers. The same was attempted while doing OAM for PW's etc, and received a strong no-no. 
> 
> 2) Let's say that you already have the information from 1) above, but you are trying to figure out which of the many paths you know of will be used for a particular flow. Now, draft-tissa-trill-oam assumes that the inner ethernet header is never used for hashing for an IP flow. It assumes that as long as you keep the L3 and L4 fields the same as that of the original packet, hardware will hash to the same values as that from the original flow, even though the L2 header is different ("6.2. Identification of IP Flows" in  draft-tissa-trill-oam ). I do not think this is a valid assumption to make. The only way to make sure your probe packet takes the same path as a regular packet is to make it identical to the original packet, all the way including the L2 headers. The IP checksum is never used for hashing, so you can invalidate it to make sure such packets get dropped at the host. This is the approach I was advocating for doing path discovery for a particular flow. 
> %sam- You have not read section 6..3 of draft-tissa-trill-oam, (Tissa pointed this out in his earlier response as well). Section 6.3 is all about using unmodified header. Please read section 6.3. Also this method cannot be used for Multicast as answered above will trigger invalid packet count. 
> 
> 3) The only hash behavior we can be sure of is that if you give any asic a non-IP packet, it will do load balancing using the only available fields, which is ethernet smac + dmac. I'm exploiting this property to do path discovery (the rbridge ethertype makes asics treat these packets as non-IP). Therefore, if you vary the inner dmac, you can exercise all the available paths. Paths you discover using this approach are applicable to both IP and non-IP traffic, since trill routes are common for both. The only hash assumption I've made is that L2 unknown ethertype (rbridge channel)  packets get load balanced using the inner dmac/smac. I am making no assumption about the fields used by the hardware for hashing IP packets. 
> %sam- Are you suggesting ingress node to randomely change the DMAC, Point is how you going to randomly change a DMAC and ensure it is taking a specific path. ? 
> 
> 4) People adopting TRILL are more likely to come from an ethernet background than from an MPLS background. For our target audience, it's important for OAM to look like what they're already familiar with, which is ethernet OAM. TRILL competes with SPB for the same space, and note that they make a virtue of being ethernet OAM compatible. Do many people on this list feel MPLS OAM is the way to go for us? 
> %sam - that is your assumption, not necessary that others share the same view. OAM, irrespective of different layers, share common characteristics. Reusing what we learnt in other areas is very useful. Having said that, folks who use OAM do not see how it is implemented, rather it is for the implementor who cares for it. If you go by your logic, mpls OAM is not right because it uses IP model and mpls is not IP. 
> Trill as such has evolved by bringing ip and bridging technologies together, with best of both in each category. Just because one likes Ethernet OAM, doesn't mean, anything to do with Ethernet , should always do Ethernet OAM. The same argument was used for mpls TP OAM as well. We all know the result. 
> The proposal here is to use the proven methodology and is working. If it is solving problem, why reinvent the wheel all over again. Better make use of it. If the solution cannot work, then we can discuss about those points, rather than predefining the solution to fit based on what one feels. 
> As far as Qbp goes, the concept of flows and it's identification is different. You cannot just compare as is for TRILL. If you want, we can discuss in that list, to get complete perspective. 
> 
> 5) I'm assuming that when operators lay out a network, they have access to a picture that shows the interconnects (or at least have some management tool speaking snmp that does). So when they need to discover the DRB on a multi-access link, then can send an SNMP query to the rbridges connected on that link to get the DRB information. This also goes for finding the mac addresses on a given rbridge, finding the address bindings at a particular rbridge etc. All of these "property query" type messages are exactly what SNMP was made for. Some things like finding the vn-tag, vxlan ID, nvgre ID etc are within the domain of SNMP but out of scope for TRILL. What is trill-specific about these? 
> %sam - if everything is available through Snmp, even performing OAM functions is redundant. Basing a model on Snmp is expensive and intensive. If we go by the argument of yours, Ethernet OAM, why do you need to export performance stats in band, when you can query the same using snmp? 
> My answer is simple. If one feels the need to export data or query data, we provide a mechanism without relying solely on SNMP. While we find it useful, everyone may not, that doesn't mean, it is not useful. If a vendor chooses not to implement, it is fine. 
> 
> I haven't responded to any statements below which are based on a misunderstanding of my original email, please re-review it and I'll be happy to answer questions on it. 
> I do not see what was not understood. You can clarify if something was not addressed right. 
> 
> Sam 
> 
> best regards, 
> Sunny Rajagopalan 
> 
> 
> 
> 
> From:        "Tissa Senevirathne (tsenevir)" <[e-mail]> 
> To:        <[e-mail]> 
> Date:        03/08/2012 09:28 AM 
> Subject:        Re: [rbridge] merging draft-tissa-trill-oam        anddraft-ietf-trill-rbridge-oam 
> Sent by:        [e-mail] 
> 
> 
> 
> Firstly there are few critical high-level points that needed to be stated 
>  
> 1.      OAM cannot be just detecting Liveliness and Path discovery. It is about Operational aspects of networks which include but not limited to Detecting Liveliness, Path Discovery, Fault finding, Fault Isolation etc. OAM is never constrained to ping and trace route.  Look at where OAM is heading to in MPLS and MPLS-TP. 
> 2.      TRILL cannot be compared 1-1 with Ethernet or IP. TRILL encapsulate the user payload with the TRILL header and utilize that to forward using TRILL header and some parts of user payload (e,g, ECMP and mcast forwarding and pruning). TRILL utilize multiple paths and multi destination tress. In that regard it is similar to a tunneling protocol. Most logical comparison of TRILL OAM is RFC 4379 MPLS OAM If one takes a look at TRILL evolution, multipathing is the fundamental point of leveraging IP technology within bridging. If one cannot perform OAM functions to verify all the paths, it is a moot point to develop OAM functions. 
> 3.      What is fundamentally required in TRILL OAM is to have OAM framework that allows to keep OAM frames as close as possible to data packet. ICMP is not utilized as a Ping or Traceroute. Motivation here is to provide such framework in backward compatible manner without introducing additional restrictions. As an example, channel draft, which define the framework of OAM channel you are proposing require to use RBridge critical alert bit. RBridges that are not supporting that cannot claim it is Channel capable. Additionally, Channel header follows right after the inner MAC header. This limit including any L3 frames. It is a common practice to select ECMP based on IP flow information and multicast prune based on IP address. (Please see rfc6326 bis for Group IP Address sub-TLV.  draft-ietf-trill-rbridge-channel, which you are proposing to use do not provide that flexibility. In your response below you have proposed several alternatives but in our mind they do not achieve what is required in OAM. Please see specific responses embedded. 
> 4.      Current suggestion from WG members are to utilize RFC 4379 framework, as that allow to utilize lots of existing knowledge, implementation and user experience from MPLS OAM. 
> 5.      The embedded message channel is utilized so receiver can identify OAM frames from data frames and take appropriate actions.  Without limiting what payloads can be include in the OAM packet. 
> 6.      You have proposed to use incremental TTL value method for multicast with invalid IP checksum. Firstly given the tree depth is different in different branches, this will invariably leak packets out. And will lead to incrementing IP invalid packet count on end stations. In hosting application this may generate unwanted support calls, and something that should not be done. In draft-tissa-trill-oam-03 section 6.3 propose to use this exact method only for unciast. Additionally because we can identify OAM frames from real data with TTL expiry due to embedded message channel, we notify that as the last hop bridge and user would not generate N+1 TTL packet. 
> 7.      Below you have proposed to utilize multicast for path trace (trace route).  Whole purpose of OAM is data plane integrity verification. Trace route in particular, used for fault isolation. Multicast use a different forwarding logic than that of unicast. Use of Multicast to trace route unicat is not going to achieve what is needed and negate the use of OAM. Additionally there are several issues such multicast tree may not cover all links in the campus and so on. Please see specific response embedded in the relevant section 
> 8.      You are proposing to merge the two drafts and I do not see a commonality based on your responses below. 
>  
> Please see more specific answers below 
>  
> From: [e-mail] [mailto:[e-mail]] On Behalf Of Santosh Rajagopalan
> Sent: Tuesday, March 06, 2012 8:49 AM
> To: [e-mail]
> Cc: Santosh Rajagopalan
> Subject: [rbridge] merging draft-tissa-trill-oam anddraft-ietf-trill-rbridge-oam 
>  
>  
>  
> Hi,
> We currently have two competing approaches to TRILL OAM, draft-ietf-trill-rbridge-oam and draft-tissa-trill-oam. Both have their merits, which calls for a revised draft with the best ideas from both.
> 
> draft-tissa-trill-oam has a dependancy on IP/ICMP/ARP etc.  The major reason for this dependency seems to be for multipath path discovery, and I have some suggestions as to how to fix this in draft-ietf-trill-rbridge-oam without using IP. draft-tissa-trill-oam also has several features which should be rightly queried from a MIB.
> 
> I think the OAM draft should concentrate on facilitating actions that would be difficult or impossible to perform in its absence. So querying counters has no place here, because you could easily figure them out by doing an SNMP query or by logging into the switch. I also think it's worthwhile to look at the features traditionally supported in ethernet OAM and to see if they have a place here.
> 
> Using draft-ietf-trill-rbridge-oam as a basis:
> 
> 1) Ping/loopback: draft-ietf-trill-rbridge-oam handles this well enough (it does need a multicast addition). The purpose of ping is to verify connectivity between two rbridges, not to discover the various ways of connecting the two.  One suggestion I have here is to be able to include a cookie in the echo request message which gets reflected back by the receiver. This cookie could be used to include the sender's timestamp in the ping message. The response could then be used to measure round-trip path latency and jitter. If the sender and receiver's clocks are synchronized using some external mechanism then this can also be used to measure one-way delay and jitter. This is a feature borrowed from ethernet OAM. 
> [Answer] Seem the paragraph is contradicting itself. Firstly you say ping is to detect the liveliness. Then you say it can be detected to measure jitter, delay etc. If there are 10 ECMP paths and if you can only measure latency of one it does not give true picture of the information. 
> 
> 2) Traceroute: IP traceroute doesn't rely on icmp for the outgoing packet (you could use ICMP, but that's a different matter). In fact, UNIX implementations usually just send UDP packets with port numbers in the 33K range, with appropriate TTL values. Note that traceroute in the IP world gives you "a" path to a destination, not "all possible paths". However, being able to enumerate all possible paths, as well as to pinpoint the particular path a flow would take are good to have features. 
> [Answer] This is not a good to have feature, but a MUST, because it is fundamental to fault isolation. Take a look at MPLS OAM and its problem statement. It is no different here. If you model the way you say, running ICMP ping between RBridge suffice the need. Each Rbridge has IP stack anyway. Also, in Linux we do have –I options to trigger ICMP. 
> 
> Here're the modifications needed in draft-ietf-trill-rbridge-oam to achieve this: 
> [Answer] All of the below is based on using multicast trees. Are you suggesting to use multicast to do trace route. Whole purpose of OAM is data plane integrity verification. Trace route in particular, used for fault isolation. Multicast use a different forwarding logic than that of unicast. Use of Multicast to trace route unicat is not going to achieve what is needed. Secondly, multicast trees will not cover all links. Case in point is having a single tree. 
> a.        Multicast unpruned tree discovery: you can discover all existing unpruned multicast trees by setting the erbridge value of an rbridge channel message to the respective roots, M=1 and the dmac to "all-egress-rbridges". Just like regular ping, you can send this as an echo request message with incrementing ttl values, and the ttl expiry messages you get will tell you what the unpruned tree looks like. Changing the egress_rbridge will paint you a picture of the different trees. 
> [Answer] First reaction to this was am I reading this correctly. After reading this one more time it appeared I am reading it correctly. There are three main issues: 1. There can be 1000 of RBRidges, and getting a response from all of them going to overwhelm. Probably choke the CPU of the requester. 2. When you do incremental TTL value, it will leak out at the last RBridge. 3. Tree does not include all paths. Assume you have 1 tree, (Which is a perfectly a valid configuration), how are you going to cover links not in the tree. 
> b.        Multicast pruned tree discovery: just do a), but replace the "all-egress-rbridges" multicast address in the inner dmac with the mac address of the multicast group. As you vary the egress_rbridge field, this will enumerate all possible pruned trees for that group. Note that draft-ietf-trill-rbridge-oam uses draft-ietf-trill-rbridge-channel, which requires the inner dmac to be all-egress-rbridges. However, the ethertype field can be used by itself to classify an OAM packet, and I am advocating that the inner dmac requirement in draft-ietf-trill-rbridge-channel be relaxed to allow for alternative inner dmacs where needed. 
> [Answer] Fundamental problem:. Prune tree works based on VLAN and MAC address or VLAN and IP address. Are you suggesting to try all VLAN all address here ? 
> c.        Multicast pruned tree discovery for a particular flow: I recommend building a packet with an empty payload which looks like the actual flow (with the header fields from the flow being tested instead of the rbridge channel header). The IP address, mac address, L4 headers etc all match with that of the flow you're trying to debug. However, to ensure that any leaked packets get dropped, set the IP checksum to be invalid (the IP checksum is never used for the ecmp hash). Now send out successive packets with the ttl field in incrementing values. Note that you're sending a "pseudo-real" packet into the network and relying on received ttl expiry messages to figure out the path. 
> [Answer] Aha you are now acknowledging IP is used for ECMP hash. Now I am wondering how Channel header can facilitate that. The concept of utilizing invalid checksum is already covered in draft-tissa-trill-oam-03 Section 6.3. Secondly using TTL in multicast is a dangerous thing, as tree depth can be different and packets can leak out. You may not want to trigger invalid IP checksum counter on your hosting servers. As that may generate additional support calls. Checksum invalid SHOULD only be used to protect against unlikely case packets leaking out. Not the default. 
> Question: Are you suggesting to include IP Multicast packets in the native form here or encap within channel header ? 
> d.        Unicast path discovery: the erbridge ID is that of the switch you're trying to ping. Let's send this echo request message with a fake inner DMAC. Construct the dmac so that the U/L bit is 1 and the I/G bit is 0 (basically, its a local scope unicast mac). Set the last 32 bits to a random value. This is a fake mac address, but that's ok because the network routes only on the erbridge anyway. The M bit in the TRILL header is 0. Send this as an echo request message with incrementing ttl values. This gives you "a" path from the source to the destination rbridge. Increment the dmac used earlier, and repeat. All rbridges hash at least on the inner dmac, so this will allow you to enumerate all possible paths from source to destination. 
> [Answer] You are making big assumption here. Firstly hash is not defined in any standard and it is implementation depended. Some devices may use SMAC:DMAC VLAN combination, Some may use IP flow information. May be low ends use just DMAC. In anycase selecting Random values are not going to gurentee a covering all paths. Secondly one may not even know how many ECMP options available on an RBridge 3 hops away. This is the reason more elaborate methods are required RFC 4379 explain How this can be done in MPLS. Based on the feedback received during the last WG session in Taipei, we have also enhanced to included this. Please see section 9.9. This needed to be done in more systematic manner rather than randomly generating DMAC. DMAC means Destination MAC. SMAC means Source MAC 
> e.        Unicast path discovery for a particular flow: Similar to what we did for multicast, construct a real packet of length 0. The IP address, mac address, L4 headers etc all match with that of the flow you're trying to debug. However, to ensure that any leaked packets get dropped, set the IP checksum to be invalid. Now set the ttl field in incrementing values. Note that you're sending a "pseudo-real" packet into the network and relying on received ttl expiry messages to figure out the path. 
> [Answer] This same concept is in section 6.3 of draft-tissa-trill-oam.   
> Note that doing this eliminates the reliance on IP/ICMP, while preserving the objectives of draft-tissa-trill-oam. The fake packets generated are technically not OAM packets, just suggestions on how to achieve a certain objective. 
> 
> 3) Fault notification: although neither draft talks about this, this is a part of the ethernet OAM feature set. I think draft-ietf-trill-rbridge-oam needs to mention that the expectation is that BFD will work over the rbridge channel to achieve this.
> 
> 4) draft-tissa-trill-oam  has a big section on TTM. The equivalent in Ethernet OAM is performance management by checking counters and send/receive delays, etc. We can cover send/receive delay/jitter measurements using ping (as described earlier). I frankly don't see the benefit to including remote discovery of counters in this OAM proposal. The administrator can always just login to the rbridge of interest and get a read-out of the counters directly, or use snmp remotely. We shouldnt be replicating functionality which is better done elsewhere, and I think this entire section on TTM in draft-tissa-trill-oam should be taken out.
> 
> 
> 
> [Answer] Your understanding is incorrect. TTM  is utilized not for counters but to ensure that specific flows are reaching specifc devices. This is included as part of feedback received from the WG in Tapei. This is an adaptation of Data Driven CFM . 
> 
> 5) I don't see any benefits to DRB/AF discovery OAM messages. This is probably done most easily by logging into the rbridge in question and doing a "show l2 isis", or extending the SNMP MIB. 
> 
> 
> 
> [Answer] How do you know which device to login to use show commands/ 
> 
> 6) draft-tissa-trill-oam has a section on MAC discovery (basically, being able to query all rbridges to see what their rbridge<->mac binding is). This is again something that should be best queried using SNMP - this is a classic MIB walk case. 
> [Answer] How do you know which device is having the MAC so you can do the walk ?
> 
> 7) Address binding verification should be out of scope for a TRILL OAM draft, and besides, should just be queried at the rbridge with the CLI. 
> [Answer] Firstly how do you know which device to login to use CLI. Secondly, anything that improve usability and experience of users can be always included.
> 
> 8) End-Station Attachment Point Discovery: should be out of scope of this draft, and is too vendor specific (it uses VN-tags).
> 
> 
> 
> [Answer] Terminology can be changed to include what is defined in VXLAN, NVGRE etc. The case in point is when there are other overlay technologies how to identify different address association. Certain VM may use the same MAC address to associate with different VXLAN segments. 
> 
> best,
> Sunny Rajagopalan 
> _______________________________________________
> rbridge mailing list
> [e-mail]
> http://mailman.postel.org/mailman/listinfo/rbridge 
> _______________________________________________
> rbridge mailing list
> [e-mail]
> http://mailman.postel.org/mailman/listinfo/rbridge
> 
> _______________________________________________
> rbridge mailing list
> [e-mail]
> http://mailman.postel.org/mailman/listinfo/rbridge_______________________________________________
> rbridge mailing list
> [e-mail]
> http://mailman.postel.org/mailman/listinfo/rbridge
> 

_______________________________________________
rbridge mailing list
[e-mail]
http://mailman.postel.org/mailman/listinfo/rbridge