Re: [bmwg] Software Update Doc

["MORTON JR., ALFRED (AL)" <[e-mail]>]

> On 2/25/13 2:16 PM, Sarah Banks wrote:
>David Newman wrote:
> ...
> > Having said this, I think that if or when downgrades are covered, the
> > methodology would be exactly the same. Would you agree?
> 
> Yes.
> 
> dn

I tend to agree, possibly without the download step in some DUTs
(though this seems to be included in section 6 on rollback).

I'm happy to see this draft in text form, after much discussion
and support for the topic in general (when on the agenda in the past).

Here are some other comments, below.

Al
(as a participant)

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Editorial:  this paragraph indent is a little distracting,
I assume this is a MSWord source?  Let's talk doc formatting...

1. Introduction

ISSU is a technique implemented by forwarding devices to upgrade or
   downgrade from one software version to another as applicable. The

-=-=-=-=-

Editorial:  it's a good idea to avoid wording that might lead to acceptance
criteria in bmwg specs. I'll try to flag them all, but here's an example:

3.1. Software Download
In this first phase, ...
... Such separation
allows an administrator to download the new code inside or outside
of a maintenance window; it is expected that downloading new code
and saving it to disk on the router will not impact operations.

s/expected/anticipated/

-=-=-=-=-

later in 3.1, Verification is purely functional testing, did it happen as planned
or not?

... Verification should include both positive verification (ensuring
that an ISSU action should be permitted) as well as negative tests
(creation of scenarios where the verification mechanisms would
report exceptions).

It would be good to separate procedural verifications as pre-requisites
for the benchmarking tests that follow. All good benchmarks assume/require
proper functions are performed. I see that similar verifications are
included in section 3.2, Software Staging, because at that stage they are an
inextricable part of the process.

-=-=-=-=-

3.3 Upgrade Run

2nd paragraph
This is the critical phase of the ISSU, where the control plane MUST
not be impacted and any interruptions to the forwarding plane should
be minimal to none.

This is a case where the ideal results have been expressed as a requirement
for the DUT to meet, rather than an anticipated result.

I suggest NEW:
There will be continuous monitoring of control-plane and forwarding-plane
functions, anticipating that any interruptions SHOULD be characterized.

-=-=-=-=-=-
Editorial:
4.1 Test Topology
The hardware configuration of the DUT (Device Under test) MUST be
identical to the one expected to be or currently deployed in
production.

suggest inserting the phrase below at end of first sentence:
... in order for the benchmark to have relevance in production.

-=-=-=-=-=-
4.2 Load Model
2nd para
For mixed protocol environments, frames SHOULD be distributed
between all the different protocols. The distribution SHOULD
approximate the network conditions of deployment. In all cases, the
details of the mixed protocol distribution MUST be included in the
reporting.

?? what protocols are being mixed?  IPv4 and IPv6? please clarify

-=-=-=-=-=-

later, at the end of 4.2:

All in all, the load model should attempt to simulate the production
network expectations to the greatest extent possible in order to
maximize the applicability of the results generated.

s/expectations/conditions/

-=-=-=-=-

5.2 Software Staging

General comment:  It is simpler in a procedure like this to ask the tester
to record the current time at different steps, then calculate the durations
later.

Start time:                          T0
Secondary RP stabilized to new code: T1
Start of Upgrade Run:                T2
Completion of Upgrade:               T3

Then when reporting (section 7), it looks like:

                                    Duration
Software Download/Secondary Update: T1 - T0
Upgrade Run:                        T3 - T2
...

worth a try?

-=-=-=-=-

5.3 Upgrade Run

4th para
... Examine the traffic
generators for any indication of traffic loss over this interval. If
the Test Set reported any traffic loss interval, note the duration
of the outage as "TP".

The above requires more detail, so that outage durations are 
reported the same way.  I think the recently approved MPLS
protection methods can provide some help here:
http://tools.ietf.org/html/draft-ietf-bmwg-protection-meth-14#page-12

-=-=-=-=-

5.4 Post ISSU
... last bullet

* Document the hitless or outage dark windows detected based upon
the (TP) counter value (provided by the Test Set)

update to reflect the metric chosen: although "outage dark windows"
sounds pretty cool, I don't know how to measure them.