Re: [bmwg] Comments on SDN Benchmarking Drafts

[[e-mail]]

Hi Al,
Thank you for sharing your comments on the terminology and methodology draft.
All your comments are valid and we will revise the draft based on your feedback.
Best Regards,
Bhuvan
On Mon, Aug 21, 2017 at 01:55 AM, "MORTON, ALFRED C (AL)"  wrote:
BMWG,

First, a public thank you to Sarah for preparing and
uploading our minutes from IETF-99, based on Marius'
extensive and "cross-checked with the audio" meeting notes.
If you missed the meeting or a trip to Prague, you can 
catch-up on our progress here:
https://datatracker.ietf.org/meeting/99/materials/minutes-99-bmwg (https://datatracker.ietf.org/meeting/99/materials/minutes-99-bmwg)

Thanks to you both!

Now, as a participant, I'm responding to one of my 
Action Items by following-up with comments on the 
SDN Drafts.

I have proposed text for the Terminology and Methodology
drafts to cover one of my comments at the meeting, about
more clearly including benchmarks for both:
1. Loss-free Asynchronous Message Processing Rate
2. Maximum Asynchronous Message Processing Rate

I haven't suggested how the 3x3 matrix should be edited yet;
these comments should be discussed first.  The proposed
text follows.

thanks and regards,
Al
(participant)

-=-=-=-=-=-=-=- Proposal Follows -=-=-=-=-=-=-=-

Terminology:
2.3.1.3. Asynchronous Message Processing Rate
OLD
Definition:
The maximum number of asynchronous messages (session aliveness check
message, new flow arrival notification message etc.) that the
controller(s) can process, defined as the iteration starting with
sending asynchronous messages to the controller (s) at the maximum
possible rate and ending with an iteration that the controller(s)
processes the received asynchronous messages without dropping.
NEW
Definition:
The number responses to asynchronous messages (such as 
new flow arrival notification message, etc.) for which the
controller(s) performed processing and replied with a valid 
and productive (non-trivial) response message. 

OLD
Discussion:
As SDN assures flexible network and agile provisioning, it is
important to measure how many network events that the controller can
handle at a time. This benchmark is obtained by sending asynchronous
messages from every connected Network Devices at the rate that the
controller processes without dropping. This test assumes that the
controller will respond to all the received asynchronous messages.
NEW
Discussion:
As SDN assures flexible network and agile provisioning, it is
important to measure how many network events that the controller can
handle at a time. This benchmark is obtained by sending asynchronous
messages from every connected Network Devices at the rate that the
controller processes without dropping. This test assumes that the
controller is respond to all the received asynchronous messages
(and the messages can be designed to elicit individual responses).

When sending asynchronous messages to the controller(s) at high
rates, some messages or responses may be discarded or corrupted
and require retransmission to to controller(s). Therefore, a useful
qualification on Asynchronous Message Processing Rate is whether 
the in-coming message count equals the response count in each trial.
This is called the Loss-free Asynchronous Message Processing Rate.

Note that several of the early controller benchmarking tools did
not consider lost messages, and instead report the maximum 
response rate. This is called the Maximum Asynchronous Message 
Processing Rate.

To characterize both the Loss-free and Maximum Rates, a test 
could begin the first trial by sending asynchronous messages to 
the controller (s) at the maximum possible rate and record 
the message reply rate and the message loss rate.
The message sending rate is then decreased by the step-size and
and the message reply rate and the message loss rate are recorded.
The test ends with a trial where the controller(s)
processes the all asynchronous messages sent without loss.
This is the Loss-free Asynchronous Message Processing Rate.

The (intermediate) trial where the controller(s) produced the 
maximum response rate is the Maximum Asynchronous Message 
Processing Rate.

Of course, the first trial could begin at a low sending rate
with zero lost responses, and increase until the Loss-free and
Maximum Rates are discovered.

Measurement Units:
Messages processed per second.

See Also:
None

-=-=-=-=-=-=-=-=-

Methodology 
(more substantial changes here, so OLD/NEW are not used)
5.1.3. Asynchronous Message Processing Rate

Objective:

This test will measure two benchmarks on Asynchronous Message 
Processing Rate using a single procedure. The two benchmarks
are (see section 2.3.1.3 of [ref to Terminology]):

1. Loss-free Asynchronous Message Processing Rate
2. Maximum Asynchronous Message Processing Rate

There two benchmarks are determined through a series of trials
where the a number of messages are sent to the controller(s),
and the responses from the controller(s) are counted over the
trial duration. The message response rate and the message loss
ratio are calculated for each trial. 

Reference Test Setup:

The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document in combination with Appendix A.

Prerequisite:

1. The controller(s) MUST have successfully completed the network
topology discovery for the connected Network Devices.
2. Choose and record the Trial Duration (Td), the sending rate step-size
(STEP), the tolerance on equality for two consecutive trials (P%),
and the maximum possible message sending rate (Ntx1/Td).

Procedure:

1. Generate asynchronous messages continuously at the maximum
possible rate on the established connections from all the
emulated/simulated Network Devices for the given Trial Duration
(Td).
2. Record the total number of responses received from the controller
(Nrx1) as well as the number of messages sent (Ntx1) to the
controller(s) within the trial duration(Td).
3. Calculate the Asynchronous Message Processing Rate (Tr1) and the 
Message Loss Ratio (Lr1). Ensure that the controller(s) have returned
to normal operation.
4. Repeat the trial by reducing the asynchronous message sending
rate used in last trial by the STEP size. 
5. Continue repeating the trials and reducing the sending rate until
both the maximum value of Nrxn and the Nrxn corresponding to zero loss
ratio have been found.
4. The trials corresponding to the benchmark levels the MUST be repeated 
using the same asynchronous message rates until the responses received
from the controller are equal (+/-P%) for two consecutive trials.
5. Record the number of responses received from the controller (Nrxn)
as well as the number of messages sent(Ntxn) to the controller in
the last trial.

Calculations:

Nrxn
Asynchronous Message Processing Rate Trn = -----
Td

Maximum Asynchronous Message Processing Rate = MAX(Trn) for all n

Nrxn
Asynchronous Message Loss Ratio Lrn = 1 - -----
Ntxn

Loss-free Asynchronous Message Processing Rate = MAX(Trn) given Lrn=0

Reporting Format:

The Asynchronous Message Processing Rate results MUST be reported in
the format of a table with a row for each trial. 

The table should report the following information in addition to
the configuration parameters captured in section 5, with columns:

- Offered rate (Ntxn/Td)

- Asynchronous Message Processing Rate (Trn)

- Loss Ratio (Lr)

- Benchmark at this iteration (blank for none, Maximum, Loss-Free)

The results MAY be presented in the form of a graph.  The X
axis SHOULD be the Offered rate, and dual Y axes would represent
Asynchronous Message Processing Rate and Loss Ratio, respectively.

If this test is repeated with varying number of nodes over same
topology, the results SHOULD be reported in the form of a graph. The
X axis SHOULD be the Number of nodes (N), the Y axis
SHOULD be the Asynchronous Message Processing Rate. Both the 
Maximum and the Loss-Free Rates should be plotted for each N.

If this test is repeated with same number of nodes over different
topologies, the results SHOULD be reported in the form of a graph.
The X axis SHOULD be the Topology Type, the Y axis
SHOULD be the Asynchronous Message Processing Rate. Both the 
Maximum and the Loss-Free Rates should be plotted for each topology.

@@@@ Comment: I think we may not need:

Tr1 + Tr2 + Tr3..Trn
Average Asynchronous Message Processing Rate= --------------------
Total Test Iterations
The last row of
the table indicates the average Asynchronous Message Processing
Rate.

_______________________________________________
bmwg mailing list
[e-mail] (mailto:[e-mail])
https://www.ietf.org/mailman/listinfo/bmwg (https://www.ietf.org/mailman/listinfo/bmwg)

DISCLAIMER: Privileged and/or Confidential information may be
contained in this message. If you are not the addressee of this message,
you may not copy, use or deliver this message to anyone. In such
event,you should destroy the message and kindly notify the sender by
reply e-mail.
It is understood that opinions or conclusions that do not relate to the
official business of the company are neither given nor endorsed by the
company.