[bmwg] Comments on SDN Benchmarking Drafts

["MORTON, ALFRED C (AL)" <[e-mail]>]

BMWG,

First, a public thank you to Sarah for preparing and
uploading our minutes from IETF-99, based on Marius'
extensive and "cross-checked with the audio" meeting notes.
If you missed the meeting or a trip to Prague, you can 
catch-up on our progress here:
https://datatracker.ietf.org/meeting/99/materials/minutes-99-bmwg

Thanks to you both!

Now, as a participant, I'm responding to one of my 
Action Items by following-up with comments on the 
SDN Drafts.

I have proposed text for the Terminology and Methodology
drafts to cover one of my comments at the meeting, about
more clearly including benchmarks for both:
   1. Loss-free Asynchronous Message Processing Rate
   2. Maximum Asynchronous Message Processing Rate

I haven't suggested how the 3x3 matrix should be edited yet;
these comments should be discussed first.  The proposed
text follows.

thanks and regards,
Al
(participant)


-=-=-=-=-=-=-=- Proposal Follows -=-=-=-=-=-=-=-

Terminology:
2.3.1.3. Asynchronous Message Processing Rate
OLD
Definition:
   The maximum number of asynchronous messages (session aliveness check
   message, new flow arrival notification message etc.) that the
   controller(s) can process, defined as the iteration starting with
   sending asynchronous messages to the controller (s) at the maximum
   possible rate and ending with an iteration that the controller(s)
   processes the received asynchronous messages without dropping.
NEW
Definition:
   The number responses to asynchronous messages (such as 
   new flow arrival notification message, etc.) for which the
   controller(s) performed processing and replied with a valid 
   and productive (non-trivial) response message. 


OLD
Discussion:
   As SDN assures flexible network and agile provisioning, it is
   important to measure how many network events that the controller can
   handle at a time. This benchmark is obtained by sending asynchronous
   messages from every connected Network Devices at the rate that the
   controller processes without dropping. This test assumes that the
   controller will respond to all the received asynchronous messages.
NEW
Discussion:
   As SDN assures flexible network and agile provisioning, it is
   important to measure how many network events that the controller can
   handle at a time. This benchmark is obtained by sending asynchronous
   messages from every connected Network Devices at the rate that the
   controller processes without dropping. This test assumes that the
   controller is respond to all the received asynchronous messages
   (and the messages can be designed to elicit individual responses).

   When sending asynchronous messages to the controller(s) at high
   rates, some messages or responses may be discarded or corrupted
   and require retransmission to to controller(s). Therefore, a useful
   qualification on Asynchronous Message Processing Rate is whether 
   the in-coming message count equals the response count in each trial.
   This is called the Loss-free Asynchronous Message Processing Rate.

   Note that several of the early controller benchmarking tools did
   not consider lost messages, and instead report the maximum 
   response rate. This is called the Maximum Asynchronous Message 
   Processing Rate.

   To characterize both the Loss-free and Maximum Rates, a test 
   could begin the first trial by sending asynchronous messages to 
   the controller (s) at the maximum possible rate and record 
   the message reply rate and the message loss rate.
   The message sending rate is then decreased by the step-size and
   and the message reply rate and the message loss rate are recorded.
   The test ends with a trial where the controller(s)
   processes the all asynchronous messages sent without loss.
   This is the Loss-free Asynchronous Message Processing Rate.

   The (intermediate) trial where the controller(s) produced the 
   maximum response rate is the Maximum Asynchronous Message 
   Processing Rate.

   Of course, the first trial could begin at a low sending rate
   with zero lost responses, and increase until the Loss-free and
   Maximum Rates are discovered.
   
Measurement Units:
   Messages processed per second.

See Also:
   None

-=-=-=-=-=-=-=-=-

Methodology 
(more substantial changes here, so OLD/NEW are not used)
5.1.3. Asynchronous Message Processing Rate

Objective:

   This test will measure two benchmarks on Asynchronous Message 
   Processing Rate using a single procedure. The two benchmarks
   are (see section 2.3.1.3 of [ref to Terminology]):

   1. Loss-free Asynchronous Message Processing Rate
   2. Maximum Asynchronous Message Processing Rate

   There two benchmarks are determined through a series of trials
   where the a number of messages are sent to the controller(s),
   and the responses from the controller(s) are counted over the
   trial duration. The message response rate and the message loss
   ratio are calculated for each trial. 

Reference Test Setup:

   The test SHOULD use one of the test setups described in section 3.1
   or section 3.2 of this document in combination with Appendix A.

Prerequisite:

   1. The controller(s) MUST have successfully completed the network
     topology discovery for the connected Network Devices.
   2. Choose and record the Trial Duration (Td), the sending rate step-size
     (STEP), the tolerance on equality for two consecutive trials (P%),
     and the maximum possible message sending rate (Ntx1/Td).

Procedure:

   1. Generate asynchronous messages continuously at the maximum
     possible rate on the established connections from all the
     emulated/simulated Network Devices for the given Trial Duration
     (Td).
   2. Record the total number of responses received from the controller
     (Nrx1) as well as the number of messages sent (Ntx1) to the
     controller(s) within the trial duration(Td).
   3. Calculate the Asynchronous Message Processing Rate (Tr1) and the 
     Message Loss Ratio (Lr1). Ensure that the controller(s) have returned
     to normal operation.
   4. Repeat the trial by reducing the asynchronous message sending
     rate used in last trial by the STEP size. 
   5. Continue repeating the trials and reducing the sending rate until
     both the maximum value of Nrxn and the Nrxn corresponding to zero loss
     ratio have been found.
   4. The trials corresponding to the benchmark levels the MUST be repeated 
     using the same asynchronous message rates until the responses received
     from the controller are equal (+/-P%) for two consecutive trials.
   5. Record the number of responses received from the controller (Nrxn)
     as well as the number of messages sent(Ntxn) to the controller in
     the last trial.

Calculations:

                                               Nrxn
   Asynchronous Message Processing Rate Trn = -----
                                               Td

   Maximum Asynchronous Message Processing Rate = MAX(Trn) for all n

                                              Nrxn
   Asynchronous Message Loss Ratio Lrn = 1 - -----
                                              Ntxn

   Loss-free Asynchronous Message Processing Rate = MAX(Trn) given Lrn=0


Reporting Format:

   The Asynchronous Message Processing Rate results MUST be reported in
   the format of a table with a row for each trial. 

   The table should report the following information in addition to
   the configuration parameters captured in section 5, with columns:

   - Offered rate (Ntxn/Td)

   - Asynchronous Message Processing Rate (Trn)

   - Loss Ratio (Lr)

   - Benchmark at this iteration (blank for none, Maximum, Loss-Free)

   The results MAY be presented in the form of a graph.  The X
   axis SHOULD be the Offered rate, and dual Y axes would represent
   Asynchronous Message Processing Rate and Loss Ratio, respectively.

   If this test is repeated with varying number of nodes over same
   topology, the results SHOULD be reported in the form of a graph. The
   X axis SHOULD be the Number of nodes (N), the Y axis
   SHOULD be the Asynchronous Message Processing Rate. Both the 
   Maximum and the Loss-Free Rates should be plotted for each N.

   If this test is repeated with same number of nodes over different
   topologies, the results SHOULD be reported in the form of a graph.
   The X axis SHOULD be the Topology Type, the Y axis
   SHOULD be the Asynchronous Message Processing Rate. Both the 
   Maximum and the Loss-Free Rates should be plotted for each topology.


@@@@ Comment: I think we may not need:

                                                 Tr1 + Tr2 + Tr3..Trn
   Average Asynchronous Message Processing Rate= --------------------
                                                 Total Test Iterations
   The last row of
   the table indicates the average Asynchronous Message Processing
   Rate.