[nmrg] RE: [IRSG] review of draft-irtf-nmrg-snmp-measure-04.txt

["Bert Wijnen - IETF" <[e-mail]>]

Karen,
Juergen has posted revision 5 with the fixes/updates as you
and he discussed. If you could quickly check that revision 5 is now OK
with you and let us all know if so, that would be appreciated.

The document is here:
  http://www.ietf.org/internet-drafts/draft-irtf-nmrg-snmp-measure-05.txt

NMRG members, pls check and speak up if you see any issues in the
changes that were made for revision 5 as a result of IRSG review.

Bert Wijnen
document shepherd

> -----Oorspronkelijk bericht-----
> Van: Karen R. Sollins [mailto:[e-mail]]
> Verzonden: maandag 19 mei 2008 19:25
> Aan: [e-mail]; Karen R. Sollins
> CC: [e-mail]; Bert Wijnen - IETF; Internet Research Steering
> Group
> Onderwerp: Re: [IRSG] review of draft-irtf-nmrg-snmp-measure-04.txt
>
>
> HI Juergen,
>
> It is all fine with me.  I think all you need now is concurrence from
> the research group.
>
> 			Cheers,
> 			Karen
>
> At 11:46 AM +0200 5/19/08, Juergen Schoenwaelder wrote:
> >On Sun, May 18, 2008 at 09:05:15PM -0700, Karen R. Sollins wrote:
> >
> >>  Thanks for your thoughtful responses.  I also did not think
> that what I was
> >>  suggesting was a lot of work.   At a high level, think about
> a reader who
> >>  is not part of your group, whom you are trying to convince
> that what you
> >>  are doing is valuable or whom you would like to convince to
> do such a data
> >>  collection exercise. I have interspersed my comments into
> your responses
> >>  below.
> >
> >I will repond to those things that are still "open" with suggestions
> >on how to "close" them.
> >
> >KS>  The second high level concern I have is that there is talk about
> >KS>  specific kinds of information to be collected and an interest in not
> >KS>  only the nature but longer term inferences with perhaps implications
> >KS>  for future redesign efforts in the SNMP context.    I have
> two levels
> >KS>  of concern here.  The most important one is that since network
> >KS>  management and in particular SNMP is NOT the primary
> objective of the
> >KS>  net (the primary objective being the transport of real payload), it
> >KS>  seems to me that the truly critical question with respect to network
> >KS>  management traffic is the impact that it has or does not
> have on that
> >KS>  real job.  To me this implies that the measurements MUST
> also include
> >KS>  contextual information.  As an example, it is probably more
> important
> >KS>  to understand whether  or not the network management traffic is
> >KS>  causing significant congestion for the payload traffic than the
> >KS>  particular mix or frequency pattern within the network management
> >KS>  traffic.  Without out the complementary contextual information, the
> >KS> whole measurement exercise seems to me to be of somewhat narrow
> >KS>  value.
> >
> >JS> The measurement may be of narrow value from your point of view but
> >JS> please keep in mind that this document is coming from the Network
> >JS> Management Research Group and not from a general Network Measurement
> >JS> Research Group. Our goal is to understand how network management
> >JS> protocols are being used because that has impact on their design and
> >JS> implementation strategies. Further note that in many networks, the
> >JS> management traffic is logically and sometimes even physically
> >JS> separated from the normal traffic and perhaps this is the reason why
> >JS> we did not even think about the question whether management traffic
> >JS> has an impact on normal traffic.
> >
> >KS> If you want to leave it as is, then I think it would be
> valuable to say as
> >KS> much.  Be specific about what you are not doing, because
> much of the rest
> >KS> of the world looks at network traffic from a broader perspective.
> >
> >I think the abstract and the introduction are pretty clearly spelling
> >out the scope of the measurement effort. So I am not sure changes are
> >needed, but see below.
> >
> >KS>  1. Section 1: It seems to me that there are TWO key questions with
> >KS>  respect to SNMP.  The first is how it is being used, which in turn
> >KS>  leads to the points made in this section, but the second is the
> >KS>  impact of that traffic.  I think that ought to appear in the
> >KS>  Introduction as well.
> >
> >JS> See my comments above. So far, the NMRG did not consider the
> impact of
> >JS> SNMP on other traffic a target of this activity. I don't want to add
> >JS> such text unless I see support from the NMRG and concrete proposals
> >JS> what should be added.
> >
> >KS> Again, as above, if you want to leave the scope as it is,
> then you should
> >KS> probably be up front about that, clearly leaving that work
> to another time
> >KS> and place.
> >
> >I propose to add the following paragraph just before the last
> >paragraph in section 1:
> >
> >    The measurement approach described in this document is by design
> >    limited to the study of SNMP traffic.  Studies of other management
> >    protocols or the impact of management protocols such as SNMP on
> >    other traffic sharing the same network resources is left to future
> >    efforts.
>
> This is what I was looking for.
>
> >KS>  2. Section 2.1.  The second paragraph begins with, "It is
> recommended
> >KS>  to capture at least a full week of data."  This is never
> justified or
> >KS>  explained.  Is one week really enough?  For what?  Why wouldn't
> >KS>  several weeks be critical, because one week might be anomalous?  Why
> >KS>  isn't a year critical, since we know that there are annual or
> >KS>  seasonal differences in traffic behaviors?  Typically, I find that
> >KS>  one-week data sets often leave me with lots of unanswered questions,
> >KS>  so justify this.
> >
> >JS> The text actually says:
> >JS>
> >JS>    It is recommended to capture at least a full week of
> data.  Operators
> >JS>    are encouraged to capture traces over even longer periods of time.
> >JS>
> >JS> The text tries to establish a lower bound of one week an encourages
> >JS> longer capture periods. I would love to get continuous traces but
> >JS> reality is such that this is not feasible. Our idea is
> simply to catch
> >JS> at least the weekly behaviour. Yes, there is of course also
> monthly or
> >JS> yearly behaviour but I believe it is not useful to set the
> bar so high
> >JS> that nobody gives us appropriate traces. I personally
> believe the text
> >JS> is fine as is.
> >
> >KS> So, what I was really getting at was the question of why one week
> >KS> was the minimum necessary.  So, something like that it is the
> >KS> minimum over which one can see the diurnal patterns in the weekly
> >KS> pattern and it is understood that both for computational and
> >KS> storage reasons the operators may not want to collect more.
> >
> >I have changed the text to the following:
> >
> >    It is recommended to capture at least a full week of data to capture
> >    diurnal patterns and one cycle of weekly behavior.  Operators are
> >    strongly encouraged to capture traces over even longer periods of
> >    time.
>
> OK by me.
>
> >KS>  4. Section 3.3, end of first paragraph:  The sentence reads, "Some
> >KS>  SNMP implementations approximate networking delays by measuring
> >KS>  request-response times and it would be useful to understand to what
> >KS>  extent this is a viable approach."  I agree, but traces
> will not tell
> >KS>  you anything about whether behaviors observed in packet traces are
> >KS>  for this reason or some other reason.  I do not believe you can get
> >KS>  at this question with the data you are collecting.
> >
> >JS> I think it is possible to analyze retransmission behaviour. Depending
> >JS> on the SNMP version used (and the other versions also depending on
> >JS> implementation choices), you can get information whether a
> response is
> >JS> just coming late for the original request or it is actually
> a response
> >JS> to a retranmitted request. We are not talking TCP here; we
> are talking
> >JS> about application layer retransmissions and SNMP has its own
> msgID and
> >JS> requestID fields.
> >
> >KS> The point I was trying to make here is that it is very
> difficult to intuit
> >KS> the reasons behind behaviors seen in the traffic, unless someone or
> >KS> something tells you.  So, you can see what the ends do, but not why.
> >
> >I agree, but this is a rather general observation and not necessarily
> >specific to 3.3 and it is not clear to me what I should do about it.
> >There is already a general remark in the last paragraph of section 2.5
> >that one has to be careful with drawing conclusions that go beyond
> >what you can really get out of traces.
>
> OK - I'll drop it.
>
> >
> >KS>  5. Section 3.4: Please explain why it is "interesting"
> (your word) to
> >KS>  identify whether concurrency or sequentiality is occurring?  What
> >KS>  will you "learn" if both are observed?  And, if one is
> occurring more
> >KS>  frequently or under specific identifiable conditions, what further
> >KS>  does that tell you?  Just knowing that one or the other occurs is
> >KS>  only the tip of the iceberg, and without acknowledging the fact that
> >KS>  these are important and unanswered questions, just learning first
> >KS>  ordered details suggests you are setting the bar too low.
> >
> >JS> The introduction of section 3 says:
> >JS>
> >JS>    The questions raised in the following subsections are meant to be
> >JS>    illustrative and no attempt has been made to provide a complete
> >JS>    list.
> >JS>
> >JS> I believe it is a good idea to first figure out whether there is an
> >JS> iceberg or not (keeping your analogy) and if there is one to ask
> >JS> questions how big the iceberg might be. For SNMP agent
> implementations
> >JS> that tend to do quite some caching, it is useful to know how well
> >JS> caching strategies are working in real-world networks. The
> concurrency
> >JS> level an agent experiences has clear impact on that. Furthermore, it
> >JS> will be useful to know how bursty the traffic tends to be or how well
> >JS> managers spread the traffic over polling intervals and this is again
> >JS> related to the concurrency we can extract from traces.
> >JS>
> >JS> I am not really sure what I should change, perhaps the word
> >JS> "interesting" is the source of the trouble and I should replace this
> >JS> with "valuable"?
> >
> >KS> What I was trying to ask was that you tell the reader a bit about
> >KS> what makes it interesting.  It certainly doesn't have to be
> >KS> complete, but just a> hint.  I don't think you should change the
> >KS> word "interesting", because if it isn't interesting, you probably
> >KS> shouldn't be doing it.
> >
> >I added the following sentence to section 3.4:
> >
> >    The concurrency level and the amount of redundant requests has
> >    implications on caching strategies employed by SNMP agents.
>
> Fine.
>
> >KS>  6. Section 3.5:  Please explain what you would do  with the
> >KS>  information about which approach to table retrieval is used.  Again,
> >KS>  what if the results tell you that both are used?  And, if not, of
> >KS>  what use is it to know which approach is prevalent?  Mighten it be
> >KS>  useful to know the conditions under which one or the other is used
> >KS>  more commonly?
> >
> >JS> This again has direct impact on agent implementation techniques and
> >JS> caching strategies.
> >
> >KS> And again, just explain a little.
> >
> >I changed the last sentence of section 3.5 to read as follows:
> >
> >    [...] It will be useful to know which of
> >    these approaches are used on production networks since this has a
> >    direct implication on agent implementation techniques and caching
> >    strategies.
>
> Fine.
>
> >/js
> >
> >--
> >Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> >Phone: [phone]         Campus Ring 1, 28759 Bremen, Germany
> >Fax:   [phone]         <http://www.jacobs-university.de/>
>
>
> --
>
> Karen R. Sollins, Ph. D.
> Principal Research Scientist
> M.I.T. CSAIL
> The Stata Center
> 32 Vassar St., 32-G818
> Cambridge, MA 02139
> V: 617/253-6006
> F: 617/253-2673
> E: [e-mail]
>