Re: [icnrg] [irsg] IRSG review for draft-irtf-icnrg-evaluation-methodology

[Dirk Kutscher <[e-mail]>]

Hi Aaron,

thanks a lot for the review! These are helpful comments.

Authors, can you discuss how you want to address this and let us know?

In case there is a need for discussion or further clarification, let’s do that on [e-mail]<mailto:[e-mail]>.

Cheers,
Dirk



From: Aaron Falk [mailto:[e-mail]]
Sent: Mittwoch, 30. Dezember 2015 00:50
To: Dirk Kutscher
Cc: Internet Research Steering Group ([e-mail]); [e-mail]
Subject: Re: [irsg] IRSG review for draft-irtf-icnrg-evaluation-methodology

Hi Dirk-

Here's my review.  It's an interesting doc.  I'm happy to correspond about any of my comments.

--aaron

High level comments:

  *   I'm not quite sure this document provides a methodology.  A better title might be "Evaluation Considerations"
  *   Section 2 isn't required by Section 3 and is a little off the focus of the document.  I recommend moving Section 2 to after Section 4, possibly to an appendix.

Other comments:

Section 1:

   This document incorporates input from ICNRG participants and their

   corresponding text contributions; it has been reviewed by several

   ICNRG active participants (see section 7<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#section-7>), and represents the

   consensus of the research group.

[AF] This is  first mention of the ICNRG and comes without expansion or definition.  Since you say earlier that the audience is the "ICN ... researchers and practitioners" I wouldn't expect them to know about the RG (indeed, RFCs live longer than RGs and the ICNRG may be a distant memory).  Thus, I suggest adding a short paragraph describing the RG's goals and members.

Section 2.2:
[AF] The section starts with a nice summary of requirements but the descriptions of the various tools should include some discussion of how well the requirements are met.


   The Icarus simulator [ICARUS<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-ICARUS>] implements ProbCache [PROBCACH<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-PROBCACH>],

   centrality-based in-network caching [CL4M<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-CL4M>]

[AF] "centrality-based" must be a ICN phrase.  It isn't meaningful to me.  Is there a less jargon-y phrase that can be used?

Section 2.3:
[AF] Given the discussion of requirements for tools, why no requirements for facilities?  E.g., low cost of use, reproducible configuration, easy-to-use tools, sharable, available background traffic...


   Often, central to the assessment of the features provided by

   a novel mechanism, lies the consideration of how it improves over

   already existing technologies, and by "how much."

[AF] Strike "Often,"

Section 3:

   This section considers techniques and options available for several

   key aspects of any evaluation method.

[AF] Given the draft title is "Information-centric Networking: Evaluation Methodology", I assume this section is the meat of the document and yet there is only a trivial introduction.  Throw the reader a bone, eh?  What am I going to hear about?  Am I to select one of the approaches described?  Or are these considerations for any evaluation technique?  Is there a relationship between the tools and facilities of the prior sections and methodologies?

Section 3.1.  Topology Selection
[AF] There's been a lot of prior work on topologies for simulation & performance evaluation.  Does ICN change any of that?  This section is a little of a hodgepodge of information.  It's hard to take away much insight on how topology selection applies to ICN evaluation.  It would benefit from a re-write that is clearer on what questions the section is supposed to answer (and then answers them).

Section 3.2.  Traffic Load

Table III. Content catalog



   Traffic | Catalog |  Mean Object Size  |  Popularity Distribution

   Load    | Size    |  [L4<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L4>][L5][L7<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L7>][L8]  |  [L3<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L3>][L5][L6<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L6>][L11][L12<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L12>]

           | [L1<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L1>][L2]|  [L9<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L9>][L10]         |

           | [L3<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L3>][L5]|                    |

   ==================================================================

   VoD     |  10^4   | Object: ~100 MB    | Zipf, 0.65 <= alpha <= 1

   ==================================================================

[AF] Video traffic has largely migrated to HTTP Adaptive Streaming where chunks are separate web objects and may be cached separately.  For example, the first 10 minutes of a movie may be more popular than the rest and cached in more places.  Treating VoD as a single 100MB object doesn't reflect this.

Section 3.3.  Choosing Relevant Metrics

   ICN is a networking concept that arose from the desire to align the

   operation model of a network with the model of its typical use.  For

   TCP/IP networks, this implies changing the mechanisms of data access

   and transport from a host-to-host model to a user-to-information

   model.  The premise is that the effort invested in changing models

   will be offset, or even surpassed, by the potential of a "better"

   network.  However, such a claim can be validated only if it is

   quantified.

[AF] This is a great introductory paragraph.  Suggest moving it to Section 1.


   ICN approaches have used system metrics, but unfortunately

   the situation is not as coherent as with the traffic metrics.

[AF] This statement isn't supported.  How is the use of system metrics not coherent?  What does this statement mean?



On Wed, Nov 25, 2015 at 8:07 AM, Dirk Kutscher <[e-mail]<mailto:[e-mail]>> wrote:
Hello IRSG,

ICNRG has reached consensus on moving forward with another document that now needs IRSG review:

https://datatracker.ietf.org/doc/draft-irtf-icnrg-evaluation-methodology/

This document surveys the evaluation tools currently available to
researchers in the information-centric networking (ICN) area and
provides suggestions regarding methodology and metrics.  Further,
this document sheds some light on the impact of ICN on network
security.

Looking at our review assignment table, next in line would be Aaron.

Aaron, do you have a chance to review this?

Thanks,
Dirk