[i2rs] comments on draft-boucadair-network-automation-requirements

["George, Wes" <[e-mail]>]

"This is because existing standards have either failed to (1) be
   massively adopted so far ...
or (2) address service provider and network
   operator requirements about configuration framework and procedures."

[WEG] I think this is incomplete as currently presented. I don't think it's a failing of the existing standards, and I think the challenge is much larger.

First, a quote I've always found appropriate for framing this discussion: "The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency." - Bill Gates

There are two different but related issues with automation of network provisioning and management:
1) a lack of will to invest in a real, continually improved and updated automation environment - it's sometimes cheaper to "throw headcount at a problem" than to make a long-term investment in automation, especially since the industry is littered with attempts to do automation that have ultimately failed, making the return on the investment somewhat questionable. It also requires a permanent staff of people to maintain and improve the system, rather than it being a once and done sort of investment. Some companies are good about this, because they have management that understands the productivity multiplier that proper automation can provide and are willing to invest in it, others are not, sometimes because they try to force a tool to automate their current process as-is rather than trying to streamline their process around the capabilities of the tool.
2) healthy mistrust by operators and management of the automation's level of intelligence and ability to make the "right" decision given all possible scenarios it might encounter. This could be restated as an inability to either characterize or pre-build business rules around all possible scenarios. It's unclear to me whether this is because the tools available are simply too crude to adequately represent the series of scenarios that your average human operator is capable of handling, or because the average SP has too much variability in its scenarios such that "if A occurs, do B" isn't realistic for all possible values of A. We may need to differentiate between simple business logic and complex when it comes to automation requirements, since it's much harder to solve the latter, but we can maybe make good progress on the former.

When it comes to network automation, the options today are usually either "build your own system mostly from scratch" or "buy a COTS package and then work to adapt it for your needs and integrate it into your internal systems". Both of these options can use standards-based protocols, but those could be thought of as the modular building blocks for a "some assembly required" implementation that is almost always bespoke. Even if you have good north/southbound APIs to work with, someone has to work on the glue to hold everything together. A good implementation requires a lot of work to document functional spec and business logic (or adapt existing process to better fit the limitations and capabilities of the tool), a lot of testing to ensure that things are working properly, and the ability to make changes easy enough that it isn't simpler/cheaper to bypass the automation and have someone do it manually when something changes. So in a way, the necessary modularity for flexibility and adaptability and the necessary predefinition to make implementation easy "out of the box" run counter to one another. The good news is, it's extensible, so you can adapt it for your needs. The bad news? It's extensible, so you likely MUST adapt it for you needs in order for it to be useful.


3.4/3.5 - there are more performance and scaling impacts than this - most notably, a poorly-implemented discovery mechanism will do things like attempt synchronization too frequently, or do it inefficiently by attempting to pull down a complete configuration and then do diffs on its stored data, rather than requesting  diffs directly and storing those updates, or waiting for a notification that something it cares about has been changed before going out to synchronize with that particular network element ONLY. It may also try to discover network elements by walking through IP ranges, which doesn't really work for IPv6. It's also pretty common for there to be a lack of coordination between different tools that need access to the same data, and rather than pulling the data once and then replicating it to all of the different tools that need it, each tool polls separately, leading to situations where a box collapses under the load of too-frequent and overly-aggressive SNMP polling, etc.
I remember problems in production with a certain VPN provisioning software where its timeout for pulling the complete configuration was too short, and on routers with large and complex VPN configurations the config sync step would fail because it took the router too long to uncompress and "display" the config to the TTY, especially if the base CPU load was too high.
There are a good number of scaling/performance considerations for VPNs specifically discussed in draft-gs-vpn-scaling that may help with this discussion.

I generally agree with the requirements, but it's possible that the above discussion will generate additional requirements, so I didn't spend a lot of time pondering them yet.

Thanks
Wes George

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.