[Icar] ICAR WG Draft Meeting Minutes

["Joel M. Halpern" <[e-mail]>]

Here is a draft of the minutes.  Please comment ASAP if there a need for 
correction / clarification, as I need to file these with the secreteriat soon.

Thank you,
Joel M. Halpern

ICAR (Improved Cross Area Review) WG
Minutes

Chairs:
Joel Halpern  [e-mail]
Mark Allman [e-mail]
with Alex Zinin guest starring for Mark who could not make it.

Minutes: April Marine (who apologizes for missing or messing up names)

Welcome:

Joel: welcome. Co-chair Mark could not make it to this one, so Alex Zinin
will help to ensure the chairs listen well.

What is icar?
  - provide a place for discussing cross area review
  - propose useful experiments in early and late stage review
  - select recommended BCPs on review techniques

We need to do recommendations of how to review early so we get on with 
trying stuff out.

WG email is [e-mail].  Please send pointers to drafts the group
should consider to that email list.  If you have a draft you think the
group should consider, please use the naming convention:
draft-<person>-icar-<name>-<nn>.txt You do not need approval to use that
name, and it is an easy way to get relevant docs.  Once the group
agrees on approaches to recommend, we will use the normal
draft-ietf-icar.... naming convention.  At this point, we want to be
aware of all docs, all ideas, half-baked or not.

The group will have an issues tracker.

Agenda:

First, existing experience. Reports from current efforts:
  - SIRS,  Crocker
  - MIB Doctors,  Wijnen
  - Security Area Directorate,  Bellovin
  - Routing Area Directorate, Zinin

As they speak, consider: what constitutes good review?
how can we help cause early review mechanisms to be used?


SIRS - Dave Crocker

SIRS is based on an idea of senior IETF folks doing external reviews (reviews
external to the WG).

Requests for review come from WGs or elsewhere. Contact:
  graybeards.net/sirs
  [e-mail]

SIRS did some reviews, very informal but with structure considering limited
resources.

Don't know if early reviews would have reduced the need for later reviews

No directions re what should be in a review and no evaluation of the
results of a review.  For example, icar is explicitly looking for
cross area reviews, but SIRS did not figure out reviews by type.

Started hearing rumors very early that SIRS had failed.
Getting reviews of the nature we need in the type we need is an 
evolutionary process.

The fact that SIRS got early signups was indicator of interest.

The criteria for who should/can review is a tough debate.

There was a low number of review requests, but that could be due to rumor 
of failure.

Need a sustained effort.

Need a central point for tracking review efforts, accessing reviews
and seeing collections of them. Such infratstructure takes time and
needs a real tracking system and someone to watch it.

Some types of reviews that need to happen:
  - cross area is one
  - review at design stage vs. later review

MIB Doctors - Bert Wijnen

psg.com//~bwijnen/iCarMib59.ppt

When Bert first became OPS AD, some MIBS could not even be run, so he
decided before approving any MIB document, it would be sanity checked.

So started organizing some "MIB doctors" to check syntax and etc.
www.ops.ietf.org/mib-doctors.html

They advise AD what to do re MIBs
and can review generic IETF docs from network management point of view.
Also, MIB doctors can advise WGs.

Guidelines are available in ID draft-ietf-ops-mib-review-guidelines-02.txt.

Now, this review takes place too much at the end of the process,
which is problematic.

They try to do consistent mib reviews so result doesn't vary depending
on which reviewer you get.

They send comments to authors and WG mailing list and relevant ADs so
that the doctors can follow up re comments or fixes.

Focus is on MIB syntax and structure/reuse, not on technology
specifically, so need to understand what the doctors do and don't do
and what they know/don't know to understand what is and isn't
reviewed.

Originally, this was just a volunteer effort, which didn't quite
work. It's better to ask a specific doctor. to review a specific MIB.
Ergo, that is a scaling issue because currently the AD is looking for
reviewers personally.


Security Directorate - Steve Bellovin

(no slides)

History of sec directorate:
  - first met for lunch to discuss issues, not so effective
  - then mailing list/web page that listed docs and asked for
    volunteers to read docs--big failure, no one volunteered
    so Jeff (Schiller, former security AD) gave up.

Steve re-started the Directorate and has been asking specific people
to review a specific doc. Also tracking who responds with reviews and
what the RTT is and using this to edit membership security
directorate.

It's not perfect - people don't always have the expertise and/or they
get overloaded.

Russ Housley (co-SEC AD) adds that pointing to a particular person is
the key vs. just asking for general help. Since we've been doing that,
we've got about 80% of the reviews we were asking for, vs. 5%.

Steve and Bert have to go to other WGs, questions for them now?
and none

Bert-- MIB Doctors haven't done much in early cross-area review. They once
started looking at --00 docs and that would be useful except that the doctors
use several tools to analyze a MIB, so if you want  MIB doctor review
early, then the doc version has to be mature enough for the tools to work.


Routing Directorate - Alex Zinin

The routing directorate has about 20 members today.  It was started by
the current ADs, Bill and Alex.  Members are selected by the ADs and
the group's official status is as an advisory body for the ADs.

[e-mail]  (closed list)

The Directorate is mainly used to review docs coming out of WGs to the IESG.
Sometimes docs go from IESG to  the directorate when ADs need advice.

General appeals for help rarely work.

Each doc is assigned 2 reviewers, depending on topic and expertise of
reviewers.  A deadline of usually 1 or 2 weeks is set.

Usually comments go to WG list or authors, but sometimes they are
discussed in rtg-dir list first or sometimes the AD proxys comments to
the WG.

ADs comments/feedback on things helps the directorate learn what IESG is 
looking for.

The log comments in tracker so is not lose and can be followed up on.

The directorate review usually happens during the AD review period.
Recently though, they have ask the chairs to cc the routing
directorate on final WG last call, so there is less of a delay.

Sometimes Alex forwards the document part of the IESG agenda to the rtg-dir 
so that folks
know what is coming up.

This is considered a "late" review, so they ask the directorate
reviewers to look for show stoppers, plus ID-nits.

They separate technical comments from editorial comments.


Presentations done. Next, discussion:

What constitues good reivew?
  first, for early reviews
    - is the number of reviews significant?
    - is the range of skill sets among reviewers important?

Harald: what you get in a review depends on what you are aiming
at. For example, directorates are helping AD make a decision. SIRS is
very informal, so what review you get depends and you don't really
know if you got what you wanted.

If you want a good review: figure out what that means. What is the
purpose of the review?

How many reviews are needed?
Are several different people needed?
Is breadth or depth of skill set of reviewers important?

Dave Thayler: Review should include both problems and also suggestions
for improvement.  This is especially important in early review process
as it is easier to make large changes at that stage.

Ole: early reviews are very important, both due to quality and
timing. They could speed up work on some things and save time on rat
holes.  Also broad reviews are important

Cathleen Moriarity:
Don't get thrown in the wrong direction --reviewer should fully
understand the problem in order to make useful suggestions.
Some suggestions could send person in wrong direction.

Joel: that raises an interesting question.  If we raise the bar too
high for reviewers, then we might discourage reviews. But it is
important for reviewer to understand what problem is being addressed
and the context of problem.

Dave T: Bert in jabber re the question of whether the number of
reivews is signifant says 2 is better than one. Dave agrees that 2 is
significantly better than 1, but after that the advantage drops.

Dimitri Papadimitriou: do reviews suggest areas that might come up in
future as problems in that space? e.g. technical content might generate
further issues?  Joel: sounds useful if reviewer can note future
implications.  They might be things that people working on topic don't
see because they don't have larger context that reviews might have.
Might be "obvious" that such review is needed to some, but not to the
folks working on the topic.

Loa Anderson Scope of review comment: if reviewer fully has to
understand the problem, you probably limit the number of reviewers to
folks who are in close cooperation with folks writing the docs. We
might want to have expertise coming in from different areas to
understand the scope of what you are doing.  e.g. if MPLS protocol
asks for security review, they need to understand that the feedback
comes back is from someone who knows X vs. Y.

Harald: I have a review team experiment that is 1 mo. old. They
read a doc and see if it is in good English.

We need docs that are technically sound, but we also have
responsibility to write docs that are comprehensible--can you figure
out the problem that they are trying to solve?  Useful feedbacd can
come from folks who don't totally understand technology for an area.

Anderson: good point

Dave Crocker via jabber: in near term focus on quality of issues, what topics
should be covered, what sorts of comments are helpful, etc. maybe a BCP?

Ole: supports Harald in re language point because that is part of
experience needing to be quite broad. e.g. getting comments from folks
who tried to understand what it is all about is useful too.

We'll have an interesting time writing a doc trying to capture depth vs. 
breadth in experience.

Harald: IESG discovered that when folks start reviewing, they don't
understand what to look for, but after awhile it's much more
obvious. Reviewers are made not born..

Tove, comment on review team "graybeard" and "sirs" as names seem to
exclude some folks :)

What group recommends will have a neutral name. It is important not
to exclude folks.

Greg Danly: Re issue of readability: the first RFC is just like a unix
man page! :) hard to write something that is technically correct but
also approachable by tech professionals in the field. a fairly wide
audience.  If we have docs that are easy to read, then that should be
part of review process.

Alex: one addition, important to think about the role of the review
process in the IETF consensus process.  We don't want WG to ignore
comments just because they already came to a consensus. We need balance
of WG ideas and input from other areas.

Any comment on late reviews?  Do comments also apply to late reviews?

Dmitri: early and late reviewers could be different people. Later
reviews could be more implementor focused.

Alex: early review needs as many eyes as possible. Later review is
touchier e.g. if directorate feels something was addressed but the
IESG doesn't agree, you have a problem.  It's important to have
consistency with IESG approval process.  AD delegation of review needs
to be done well.

Greg D: I think that there is a conceptual difference between early and 
late review.
Late review could show conceptual problems or changes if no early review was
done. If do an early review, then late review can clarify or show cross 
group consensus.

Dura..? nortel Early review should be done in WG for catch syntax,
language.  Late reviews should involve tech aspects as well as other
areas input, Early review w/in wg.

Ole: Don't completely agree. The real advantage is to have review outside 
of the WG.

Next agenda: discuss docs that currently talk about early review:
  SIRS experience
  allman--problem-wg-revcomm-00
  draft-ietf-ietf-ops-mib-review-guidelines
  draft-carpenter-solutions-sirs-02.txt

Joel: How can we identify docs for cross area review? Here "early"
means early cross area, not the within WG review.

Loa: agreed. If you say we should have folks not involved in WG do
early cross area reviews, you really need someone with a particular
expertise. He may be in the WG or not, don't exclude people who are in
WG.

James Polk: Is there a repository, so if i write a draft i can post it
and offer for review? and a tracking on progress?  Joel: not currently
such a repository or even a process.  This WG will figure out what
needs to happen and what tools are needed.  James: I think that a
posting process is a good idea, even though there is no way to ensure
someone will review a draft.

Harald: tools: given our limited resource environment, we'd like to have some
experimenting done before committing resources to the final choice.
We encourage folks to experiment because experience of 10 people doing 10
reviews using any mechanism to check them is valuable.  "genauth"
experiment visible on website, mailing list of reviews.  I know more
about tools needed after having tried a bit than I did before.

Jan? what is the relation between early review process and advisors for WG?

Joel: good question, and the answer is probably up to WG.
need to work with ADs and dirs if we want to tie to their advisors.

Russ: getting security advisors for WGs is really hard due to time
commitment greater than single review.

jabber: dave crocker: we should formulate a guide for reviews and add
a Sunday tutorial. BCP would be a separate output from the group.
also: sirs web/list suffered from lack of better tools, but had
limited resources.

Joel: many WG did not ask for SIRS reviews. Why? Did they need better
incentives?  We need to make sure WG pays attention to feedback.  Are
there incentives or motivations for encouraging WGs to get early
reviews?

Could this in some way speed up later iesg review? That could be an incentive.

Alex: the more eyes that see the doc, the faster it gets through the IESG.

Dave: I strongly suggest the review function be independent of IETF 
administrative
management team.  Strickly for technical commentary.  Also, getting review
requests and a  BCP will help motivate WGs.  IETF management culture
encouraging early reviews.  WG should have a doc that is like a technical
proposal, more than idea but less tech detail. "the boxes and arrows
draft."

Harald: if you want to create a body of power and responsibility
outside of the IETF management structure, be careful what you ask for. Creating
such a body with real power outside of IETF management structure would take
longer than other approaches.

Dmitri: what about cross body reviews?  e.g. ITU. How do we deal with
those reviews? can that process be integrated with this groups'
recommendations?

Alex: We had a comment like that during charter review to "improve
coordination with other standards bodies or external standards body review" 
etc.
But we should be careful about that as there are a lot of details that 
could get
in the way of process.  I think we should concentrate on our internal
review first because we understand those processes much better before
do interaction with other SDOs.  Joel: good point, that is not within
the charter as written, but an important question.

Ole: focus on internal issues, but don't forget cross relations. If
other SDOs have concerns or someone from there comes, address it.

Dave C's response to Harald: what power does a review group have? and
what benefit is there in making the review group subject to IETF power?
Harald: if review is to make any sense, must address quality, relevance,
and timeliness of IETF output. The management of the IETF exists to foster 
that quality,
relevance and timeliness.

We can already ignore reviewers? Harald: if you have reviewers that
make a difference, they have power. If you have reviews that don't
make a difference, they don't have power.  We have both kinds.

Some docs area already out there:

draft-zinin-early-review-00
draft-iesg-alvestrand-twolevel-00


Alex: Area Review Teams doc

Motivation for this doc was the same as led to creation of ICAR
working group.  We want to improve cross area review in order to
improve quality of documents and to get that review before docs reach
the IESG. Higher document quality should reduce IESG approval
time. make reviews consistent with the type of review the IESG does
later.

This doc assumes IESG still responsible for final approval.  ADs are
personally responsible to the community for the quality of approved
docs.

Each Area would have an Area Review Team with a well known mailing list
such as <area>[e-mail].

The team would include at least ADs and other experts selected by ADs.

ADs delegate review to team members.

Selection is arbitrary but may include an open call for nominations.

Two members assigned to each doc.

Other members are asked to review, but not required to.

The review is initiated by WG chairs or by AD early in process
or automatically started for IETF Last Call.  Team member review requires
AD approval.

The mechanism is an Email request and the AD approves assigns reviewers 
from ART.
Comments go to WG or authors. and WG looks at comments and addresses
them as usual.

Then as docs are updated, same reviewers review again.

During IESG review, the AD can bring up comments that haven't been
addressed or override a reviewer's comments.  This consistent with
personal responsibility ADs have for quality of doc.

Starting review during IETF last call means all areas ARTs can look at
it if necessary.

Documenting review results is important, vital for follow up.
WG chairs reponsible for issue tracking and summarizing resolution

For individual submissions, either AD or an ART person tracks issues.

The model of trust, responsibility and accountability does not change.
Delegation of review does not delegate responsibility of content.

Motivation and credit for team members important, but not sure how to do.

Questions:

Loa: if team picked personally by AD, does the change of an AD cause a
big change in the review team?  New AD will probably look at the
members and make changes or add some new people.  That has to due with
personal responsibility of AD
Loa: so review team is appointed by ADs?
Alex: yes

Dmitri: what if docs have several WG last calls? Alex: some WGs use
"last call" to really initiate early review. so WG chairs will use
judgement as to when to request cross-area review.

Ole: I'm worried about how reviewers are selected. If it is only to
off load AD work, then that may be good, but we also miss other views,
not just an extended AD.  It's important to see diversity of opinions on
review teams. Especially if their review is equal to the decision or just
input.  Alex: yes, there is a balance here.  Most Ads probably are
interested in diverse opinions. Not sure how to require that, but
seems we have the same goal.

One idea was to have the ART members selected by nomcom, but that has
a bunch of drawbacks. Alex is working from idea that current structure
stays the same.

Greg: there's a conception that the ART teams might be small personal
army for the ADs. However, when someone is asked to review a doc, they
are using their own technical judgement, which may or may not align
with the AD's. There is no guarantee of outcome. It is based on
individual integrity.  So even hand selecting the ART does not
guaranatee any particular result.  But getting folks who are qualified
and know the area is the most important thing.

Alex: yes, it's a  bi-directional process. The comments of reviewers influence
AD. Then there is also the process of educating each other.

Joel: this is one proposal we can consider for late state formal review

Harald: also has a draft.  The relevant section in draft very
short. Mainly it deals with noting how the IESG works when reviewing
docs and that it doesn't have to be those people doing that review. It 
suggests we
put together teams where the collective judgement of that team covers
all the aspects that need to be covered for late review.  For when a WG
thinks that they have covered everything.  In the IESG, people tend to
learn from each other and adjust to each other's reviews, knowing what the
other folks will usually catch.  Put together review teams that are
cross area from the beginning, vs. having one review team per area.
Vertical (Alex) vs. horizontal (Harald) approaches.  Harald suggested teams 
be able
to approve documents, but that is probably out of scope for this
WG. But it might be a worthwhile direction to experiment in.

Dave T: How is it determined which team approves a doc?  Harald:
Throw dice, except that if a group of docs refers to each other, the same
team should cover.
Dave:  but not all review teams will not be equal
in terms of quality or also re direction. if they have the power to
approve, one team could say yes and another no to same doc.  We need a
central management point of ultimate responsibility.  The WG can't
shop around, might be able to do so with Harald's group.  Harald:
yeah, valid concern. if we want to have judgement exercised by more
than one gourp of people, we'll have to expect and deal with
differences. We have to have a place in the system that can tell a
review team it is out of line, so need a central authority.

Greg Danly: we need to look at cross area section. Seems the ART thing
has 2 reviewers per area, but this is not as much cross area.

missed name: not sure how can come up with a team and know whether or
not they have authority to pass docs.  By the way, I don't think the team
should have authority to pass documents. Review teams should be helpers.

Harald: two possible outcomes for teams (3 if you count failure). 1)
get docs reviewed and IESG disagrees.  Review wasn't thorough enough, in
which case the review teams should be advisory. or 2) we run the
experiment and see the review team finds all significant problems with
the doc. The IESG sees review, the team  is happy, the WG deals with all 
comments,
IESG finds nothing more. Then if that is the normal case, why not
allow team to approve docs??  Need to run the experiment first.

Finally, AD has a question of the chairs: what are the next steps?

Joel:
  - put web site up,
  - get tracker set up,
  - get all relevant docs issued with the name convention so they can be 
reviewed,
  - then have discussion of those docs against the topics of early and
    late review as appropriate with the goal of trying to
  - clarify what experiments can be done soon and
  - recommend those experiments.
  - Then start BCP on what review contents and longer review procedures 
should be.

Harald hopes we have the energy to try it to the finish. 


_______________________________________________
Icar mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/icar