Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi Lucien,

after agreeing on the beginnings of a definition
(the one I dashed-off below):

  ... function that manages *and* controls SDN switches, <adding more qualifications>".

the scope police would say, "if you haven't got control capability,
the controller is out of scope for this memo".

ok?
Al

> -----Original Message-----
> From: Lucien Avramov (lavramov) [mailto:[e-mail]]
> Sent: Wednesday, October 14, 2015 5:31 PM
> To: MORTON, ALFRED C (AL); Castelli, Brian; Jacob Rapp; Boris Hassanov
> Cc: [e-mail]
> Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-
> benchmarking (terms and meth)
>
> some sdn controllers only 'manage' switches and don't control. Other are
> in the control plane. It's a very broad terminology prone to a lot of
> confusion, to be useful, the benchmark needs to clearly test one type of
> device or scope of performance (data center benchmarking?) and not go
> into a broad SDN controller term which i agree is not our charter to
> define.
>
>
>
> On 10/14/15 2:27 PM, MORTON, ALFRED C (AL) wrote:
> > ...and, as I  suggested in a separated thread (approximately), we say
> >
> > "For the purposes of this memo, the SDN Controller is a
> >
> > function that manages and controls SDN switches, <adding more
> > qualifications>".
> >
> > This is about scope of work, and the title needs to represent the
> scope.
> >
> > Al
> >
> > *From:*Castelli, Brian [mailto:[e-mail]]
> > *Sent:* Wednesday, October 14, 2015 5:21 PM
> > *To:* Jacob Rapp; Boris Hassanov
> > *Cc:* MORTON, ALFRED C (AL); [e-mail]
> > *Subject:* Re: [bmwg] WG ADOPTION:
> > draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)
> >
> > I think I'm beginning to understand Jacob's reservations about the
> > claim that this draft covers all SDN controllers. SDN is so broad that
> > it may include controllers of devices that are not switches, do not
> > have flows, etc. What if we choose a 3rd alternative: Call this a
> > benchmark for controllers of SDN switches. That is specific enough to
> > rule out inappropriate usage while avoiding coupling with one specific
> protocol.
> > Controlling switches implies many if not all of the tests included in
> > the draft. Would that satisfy the concerns?
> >
> > *From: *bmwg <[e-mail] <mailto:[e-mail]>> on
> > behalf of Jacob Rapp <[e-mail] <mailto:[e-mail]>>
> > *Date: *Wednesday, October 14, 2015 at 4:09 PM
> > *To: *Boris Hassanov <[e-mail]
> > <mailto:[e-mail]>>
> > *Cc: *"MORTON, ALFRED C (AL)" <[e-mail]
> > <mailto:[e-mail]>>, "[e-mail] <mailto:[e-mail]>"
> > <[e-mail] <mailto:[e-mail]>>
> > *Subject: *Re: [bmwg] WG ADOPTION:
> > draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)
> >
> > Also for the RFC 2544 is focused on the data plane not the control
> plane.
> >
> > --
> >
> > Jacob
> >
> >
> > On Oct 14, 2015, at 9:57 PM, Jacob Rapp <[e-mail]
> > <mailto:[e-mail]>> wrote:
> >
> >     I think it's worth a conversation live. I would argue that
> >     controllers are functionally so different right now, we are not at
> >     the point where we can define this. I would simply take the
> example
> >     of a Cisco ACI controller, VMWare Controller cluster & Open
> Daylight
> >     controllers and apply the standard to them and see the result. I'm
> >     happy to go along a help improve the draft to include them if we
> >     can, but functionally they are so different.
> >
> >     At some point we start defining what an SDN controller is and that
> >     is not our charter.
> >
> >     --
> >
> >     Jacob
> >
> >
> >     On Oct 14, 2015, at 9:34 PM, Boris Hassanov <[e-mail]
> >     <mailto:[e-mail]>> wrote:
> >
> >         Hi all!
> >
> >         My 0,05$:
> >
> >         I would agree with Bhuvan here, IMHO  common methodology for
> >         testing is definitely needed . I see here SDN-controller as
> >         general term, SBIs can be different today and probably there
> >         will be even more of them tomorrow. It will be very hard to
> >         support all of their message details in one draft. May be
> >         hierarchical approach( when details for particular
> SBI/protocol
> >         testing in another document will be correlated/referenced to
> the
> >         general methodology ), could be better in this case.
> >
> >         Jacob, may be this is not perfect example, but we have RFC2544
> >         which defines methodology that can be used for testing of
> >         devices with IPv4, labeled traffic etc.
> >
> >         Thank you.
> >
> >         SY,
> >
> >         Boris
> >
> >         On Tuesday, October 13, 2015 5:02 PM, Bhuvan (Veryx
> >         Technologies) <[e-mail]
> >         <mailto:[e-mail]>> wrote:
> >
> >         Hi Jacop,
> >
> >         Thanks for bringing this point for discussion in the mailing
> >         list. As you mentioned, the functionality of one protocol will
> >         be different from the another one.
> >
> >         But the intent of the draft is to define a generic methodology
> >         to benchmark the controller performance as a black-box basis
> >         independent of protocols it support. For instance, to
> benchmark
> >         the controller message processing rate (throughput), we need
> to
> >         measure the input/output  rate i.e., the input (a.k.a request
> >         such as OpenFlow Packet-ins or BGP route updates or Route
> >         install) to the controller could be from southbound/northbound
> >         interface and the output (response such as OpenFlow
> >         Packet-Out/Flow-Mod or XMPP Publish) could be sent to
> southbound
> >         interface. So this way we want to derive a methodology
> >         abstracting the protocols to benchmark different controller
> >         implementations.
> >
> >         If you feel some definitions or terminologies need to be
> >         clarified further/added, we would be very happy to take such
> >         inputs and work on it.
> >
> >         Thanks,
> >         Bhuvan
> >
> >         -----Original Message-----
> >         From: bmwg [mailto:[e-mail]
> >         <mailto:[e-mail]>] On Behalf Of Jacob Rapp
> >         Sent: 12 October 2015 20:54
> >         To: MORTON, ALFRED C (AL); [e-mail]
> <mailto:[e-mail]>
> >         Subject: Re: [bmwg] WG ADOPTION:
> >         draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)
> >
> >         I think the problem that I have is that I'm not sure if the
> >         southbound protocol can really be decoupled here. There are
> very
> >         different functionality if you use one protocol or another. It
> >         may not be a totally fair comparison, but it is like saying I
> >         can write a draft that tests a "router" that tests routing
> >         protocols, but I don't define if it is for OSFP of BGP.
> >         It would be one thing, if you were testing just controller HA,
> >         but the drafts test how the flows or paths or etc are
> >         programmed, which may be different and have special
> terminology
> >         depending on the southbound protocol.
> >
> >         -
> >         Jacob
> >
> >
> >
> >         On 10/12/15, 4:56 PM, "bmwg on behalf of MORTON, ALFRED C
> (AL)"
> >         <[e-mail] <mailto:[e-mail]> on
> behalf
> >         of [e-mail] <mailto:[e-mail]>> wrote:
> >
> >         >BMWG Folks, (this is the thread I was referring to in reply
> to Jacob)
> >         >
> >         >I support the adoption of these drafts as WG items.
> >         >
> >         >I'd also like to see some comments below addressed (e.g.,
> Brian's), if
> >         >they haven't already. We would have to look at controller-
> clusters to
> >         >do add of these metrics, I believe.
> >         >
> >         >regards,
> >         >Al
> >         >(as a participant)
> >         >
> >         >-----Original Message-----
> >         >From: Castelli, Brian [mailto:[e-mail]
> <mailto:[e-mail]>]
> >         >Sent: Monday, June 23, 2014 11:39 AM
> >         >To: Bhuvan (Veryx Technologies); MORTON, ALFRED C (AL);
> 'Banks, Sarah';
> >         >[e-mail] <mailto:[e-mail]>
> >         >Cc: 'Anton Basil'; 'Vishwas Manral'
> >         >Subject: Re: [bmwg] Feedback for Benchmarking Methodology for
> OpenFlow
> >         >SDN Controller Performance
> >         >
> >         >I really like Al¹s suggestion to generalize SDN controller
> benchmarking because 1) the ONF effort is OpenFlow specific and 2)
> OpenFlow does not define SDN. OpenFlow is one of several southbound
> protocols that can be used between a controller function and network
> node functions. Regardless of the maturity of current OpenFlow
> implementations, I believe that we can define a family of benchmarks
> that have general applicability to SDN environments because there are
> common functions/services that will be provided independent of protocol.
> At the end of the day, protocols are commodities. It¹s the service they
> provide that adds value.
> >         >
> >         >The current draft contains many of the elements required to
> sufficiently define a benchmarking methodology for a controller function
> in an SDN environment. We should follow Al¹s suggestion, define the
> taxonomy used in the draft, and consolidate wording to make the document
> easier to read and apply.
> >         >
> >         >What follows is an attempt to define a higher-level set of
> SDN controller test cases. It builds on the work done in the draft. I
> will provide some detail on the first test case in order to establish a
> cadence. Subsequent test cases will not have the detail, but the ability
> to iterate over variables and so on is implied.
> >         >
> >         >The SDN Controller test taxonomy would include:
> >         >
> >         >- Asynchronous Message Response Rate. A test case to measure
> AMRR would cover a variety of conditions that are analogous to the
> following OpenFlow-specific events:
> >         >
> >         >  o Packet_in received
> >         >  o Flow expiration received
> >         >  o Link down received
> >         >
> >         >In other words, AMRR would be used to measure the ability of
> an SDN controller to respond to a variety of asynchronous (i.e.
> unexpected) conditions. One methodology for measurement would be
> specified by the benchmark. The tester would then apply that methodology
> to packet_ins, flow expiration, link down, and whatever other AMR
> testing is appropriate.
> >         >
> >         >The AMRR methodology should include iteration variables and
> goal seeking.
> >         >The tester ought to be able to use the methodology to
> determine, for example, the maximum packet_in AMRR for a variety of
> conditions, such as variations in the number of connected nodes, the
> complexity of the packet_ins, and so on. Variables also include negative
> testing with invalid packets, for example.
> >         >
> >         >- Synchronous Message Response Rate. A test case for
> measuring SMRR
> >         >would cover a variety of conditions analogous to the
> following
> >         >OpenFlow-specific
> >         >events:
> >         >
> >         >  o Hello packet exchange
> >         >  o Echo request/response exchange
> >         >
> >         >- Node Acquisition Rate. A test case for measuring NAR would
> be a special case of SMRR, but it specifically measures the rate at
> which an SDN controller can connect to nodes in a network.
> >         >
> >         >- Node Acquisition Maximum. A test case for NAM would measure
> the maximum number of network nodes that an SDN controller can connect
> to without error.
> >         >
> >         >- FailOver Response Time. There are at least two test cases
> for measuring FORT. One would measure a control cluster¹s ability to
> survive a controller failure. How long does it take the controller
> function to be assumed by another controller element? Another would
> measure the controller¹s ability to respond to node failures to do
> things like re-route traffic. How long does it take for the controller
> to send out commands that handle the error? This is a bit like measuring
> convergence time, but care must be taken to keep the node variables out
> of the measurement.
> >         >
> >         >- FailBack Response Time. Test cases for FBRT would be used
> to measure the response time when the controller comes back up or the
> node comes back online. How long does it take the controller to send the
> commands that restore service?
> >         >
> >         >
> >         >
> >         >On 6/23/14, 10:53 AM, "Bhuvan (Veryx Technologies)"
> >         ><[e-mail]
> >         <mailto:[e-mail]>> wrote:
> >         >
> >         >>Hi Al,
> >         >>
> >         >>Thanks for your comment. I do agree with you that there
> needs to be a
> >         >>common benchmarking mechanism for controller designs
> performing the
> >         >>same tasks.
> >         >>But the technology is still relatively immature and the
> approach has
> >         >>various dimensions including centralized control vs
> distributed
> >         >>control (controller less).
> >         >>Again the centralized control approach uses different
> programming
> >         >>methods (OpenFlow vs Non OpenFlow).
> >         >>
> >         >>Having said that, for better usability/understanding of the
> draft and
> >         >>to avoid misinterpretation, I personally feel it would be
> good to have
> >         >>separate benchmarking methodology for each approach (though
> the
> >         >>metrics remain same).
> >         >>Also we would be happy to continue the effort to extend the
> same
> >         >>metrics for other approaches too.
> >         >>
> >         >>Please let me know if I've missed something.
> >         >>
> >         >>Sarah/Brain, please also share your thoughts on this.
> >         >>
> >         >>Thanks,
> >         >>Bhuvan
> >         >>
> >         >>-----Original Message-----
> >         >>From: MORTON, ALFRED C (AL) [mailto:[e-mail]
> <mailto:[e-mail]>]
> >         >>Sent: Saturday, June 21, 2014 6:00 PM
> >         >>To: Castelli, Brian; Banks, Sarah; Bhuvan (Veryx
> Technologies);
> >         >>[e-mail] <mailto:[e-mail]>
> >         >>Cc: 'Anton Basil'; 'Vishwas Manral'
> >         >>Subject: RE: [bmwg] Feedback for Benchmarking Methodology
> for OpenFlow
> >         >>SDN Controller Performance
> >         >>
> >         >>Hi Brian, Sarah, Bhuvan, and all,
> >         >>
> >         >>a couple of quick answers and a comment:
> >         >>
> >         >>> -----Original Message-----
> >         >>> From: bmwg [mailto:[e-mail] <mailto:bmwg-
> [e-mail]>] On Behalf
> >         Of Castelli,
> >         >>> Brian
> >         >>...
> >         >>> My two main concerns are getting the terminology right and
> making
> >         >>> sure that the proposed set of tests will achieve our
> goals. I
> >         >>> believe those goals ought to include a common taxonomy,
> enabling
> >         >>> apples-to-apples comparisons, and minimizing the time/work
> required
> >         >>> to execute the test cases.
> >         >>
> >         >>Great, that's exactly what we are chartered to do in BMWG.
> >         >>https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__datatracker.ietf.o
> >
> >>rg_wg_bmwg_charter_&d=BQIFAw&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt
> >         >>-
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_
> >         >>GjrhU&s=gJv3CuKB6C3osQoVuA3JGltD3bMZFTfuubj__2xVgMU&e=
> >         >>
> >         >>
> >         >>>
> >         >>> I am willing to help improve the benchmark. I understand
> that it is
> >         >>> an early draft. The time is right to make sure that we are
> >         >>> developing the best specification that we can.
> >         >>>
> >         >>> How do we continue this process? Via email to this list? I
> am
> >         >>> willing to help.
> >         >>
> >         >>It's principally e-mail (just as you've been doing) and
> face-to-face
> >         >>meetings three times a year.  Sarah mentions that our next
> meeting is
> >         >>in her home town:
> >         >>https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__www.ietf.org_meeti
> >         >>ng_90_index.html&d=BQIFAw&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-
> YihVMNtXt-uE
> >
> >>s&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_Gjr
> >         >>hU&s=PyQn5inWQ2S-mFAiKMB3UaRcIfzBr2syenp80SlaYxE&e=
> >         >>
> >         >>Make some text proposals for the draft, and we can discuss
> them on the
> >         >>list.
> >         >>
> >         >>Having said that, and recognizing that there appears to be a
> >         >>comparable activity in ONF that (I assume) is OpenFlow-
> specific,
> >         >>perhaps a way to add value to the industry is to approach
> the SDN
> >         >>controller problem more generically, such that different
> controller
> >         >>designs performing the same tasks could be benchmarked as
> black-boxes.
> >         >>I realize this approach has substantial implications for the
> draft,
> >         >>but the benefit of wider applicability.
> >         >>
> >         >>food for thought,
> >         >>Al
> >         >>(as a participant)
> >         >>
> >         >>
> >         >
> >         >Spirent Communications E-mail confidentiality.
> >         >-------------------------------------------------------------
> ----------
> >         >This e-mail contains confidential and / or privileged
> information belonging to Spirent Communications plc, its affiliates and
> / or subsidiaries. If you are not the intended recipient, you are hereby
> notified that any disclosure, copying, distribution and / or the taking
> of any action based upon reliance on the contents of this transmission
> is strictly forbidden. If you have received this message in error please
> notify the sender by return e-mail and delete it from your system.
> >         >
> >         >Spirent Communications plc
> >         >Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN,
> United Kingdom.
> >         >Tel No. [phone]
> >         >Fax No. [phone]
> >         >
> >         >Registered in England Number 470893
> >         >Registered at Northwood Park, Gatwick Road, Crawley, West
> Sussex, RH10 9XN, United Kingdom.
> >         >
> >         >Or if within the US,
> >         >
> >         >Spirent Communications,
> >         >26750 Agoura Road, Calabasas, CA, 91302, USA.
> >         >Tel No. 1-818-676- 2300
> >         >
> >         >_______________________________________________
> >         >bmwg mailing list
> >         >[e-mail] <mailto:[e-mail]>
> >         >https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailm
> >         >an_listinfo_bmwg&d=BQIFAw&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-
> YihVMNtXt-uEs
> >
> >&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_GjrhU
> >         >&s=K2L3UKeL6pXwVtMOMvWjpLaa6gT-IIk7WZ9-2384xZI&e=
> >         _______________________________________________
> >         bmwg mailing list
> >         [e-mail] <mailto:[e-mail]>
> >         https://www.ietf.org/mailman/listinfo/bmwg
> >
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mai
> > lman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_C4
> > AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
> >
> >         _______________________________________________
> >         bmwg mailing list
> >         [e-mail] <mailto:[e-mail]>
> >         https://www.ietf.org/mailman/listinfo/bmwg
> >
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mai
> > lman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_C4
> > AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
> >
> >     _______________________________________________
> >     bmwg mailing list
> >     [e-mail] <mailto:[e-mail]>
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail
> > man_listinfo_bmwg&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-u
> > Es&r=4eVZvh2C5Un5N5XXv1rh7g&m=mEwjLokjWWse4IGGeCVWyjlSoL-paCEYOSoERPwp
> > pfs&s=6XoMjPRvZDmcQHZUqjsHF-QFBiRwxG9Q_hAJbgUrluc&e=
> >
> >
> >
> > _______________________________________________
> > bmwg mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/bmwg
> >