[Pesci-discuss] My Notes on draft-davies-pesci-initial-considerations-00.txt

Dear Pesci Design Team,

You guys have done a lot of work in not a lot of time. Most of it looks 
good. It wouldn't be the IETF, if no one had comments - pls. see the 
following notes.

FWIW, I'm trying to reflect my understanding of commonly-held and 
commonly-stated positions, not my own ideas about what should be happening 
(that would happen in other threads).

To cut to the chase, I have two fairly serious comments:

- In 1.2, the text says that the IETF Chair LED the design team. I don't 
believe that's actually what happened - it's not what Brian told me was 
happening - but if it was, Brian needs to recuse himself from the approval 
process. If the IETF chair REQUESTED that the design team produce this 
draft, that's fine (hint, wink).

- In 6.1, I'm concerned that I don't know who actually determines whether we 
have consensus for the action plan included in this draft. Is it the IETF 
chair? the IESG? or someone else?

Most else is editorial. My comments start with "SD:", and refer to the 
FOLLOWING text.

Spencer

p.s. I'm not going to look at Appendix A, because it was your raw inputs 
list, but I would like to see this carried forward in the document. It 
always bugged me that we threw away the raw inputs to the Problem Statement 
RFC after a couple of revisions - knowing what else was considered seems 
valuable.

1.  Introduction

SD: The following text makes "continual debate" sound a lot more positive 
than it really is. I'd suggest something like "The IETF almost continually 
debates its own process. While this is a positive indication of "openness", 
"closure" also matters, and IETF process debates are often inconclusive and 
often revisited."

   The IETF almost continually debates its own process and this is in
   many ways a healthy sign of its openness.  However, the debate is
   often inconclusive.

1.1.  Principles, Policy, Process and Procedure

SD: I have heard the phrase, "principles don't change, but procedures can 
change frequently" - I'm not sure if that's a helpful observation here or 
not, but if it is, it might give a smaller number of principles (do we 
really have 32? I thought us us as an unprincipled lot, most days).

   Before going on to discuss process change we should be clear about
   what we mean by "process", and some of the other "P" words which will
   be used in the document.

SD: This sentence was difficult to parse - if the phrase was "open 
development of documents and open resolution of organizational matters", and 
if the phrase was in quotes, I'd be able to parse it more easily.

               These
               include such things as requiring that development of
               documents and organizational matters are as far as
               possible open and "rough consensus and running code" as
               an operating principle.

1.2.  About This Document

SD: Please tell me PESCI was "simply the group of people who produced this 
document AT THE REQUEST of the IETF Chair" - if the IETF Chair LED a single 
design team, Brian should recuse himself from the approval process. Based on 
discussions with Brian, my understanding was "AT THE REQUEST", so I'm asking 
for a wording change, not a rewriting of history...

   PESCI
   has no special status in the IETF process; it is simply the group of
   people who produced this document under the leadership of the IETF
   Chair.

2.  Background reading
SD: Sadly, this probably should be "An earlier phase" (there were more than 
one).

   The early phase of the current round of process discussions led to a
   problem statement [RFC3774].

3.  Short problem analysis

SD: I would suggest "Insufficent cross-area review early in the process (an 
unsolved issue)"

   3.  Variable quality of output from WGs
       *  Unsolved issue of adequate cross-area review earlier in the
          process.

   5.  Lack of a "career path" for IESG members - "dropped in,
       overworked and chopped off"

SD: s/loose/lose/

       *  potential to loose both experience and capability of ex-ADs

SD: Expand "techspec" on first usage...

   Issues that are to be considered under the "techspec" banner are out
   of scope for PESCI.

4.1.2.  Change Management Guidelines

SD: I'm not AGAINST quantitative measures, but this comes up every time we 
discuss significant process change, just before EKR gets up and rails about 
how noisy the data is and how we change too many things at once to figure 
out what changed output can be assigned to each changed input, and we've got 
to be expecting two or three years of transition and retraining for 
significant process changes anyway. I'd drop this one for now, as much 
heartburn as it causes the engineers among us.

   G9.  Try to explicitly estimate the impact of changes before making
        them, and try to measure whether the expectations were met after
        making the change.

SD: It might be worth putting in an explicit pointer to RFC 3933 here, so 
that it's more clear that our existing BCPs would encourage this approach.

   G10. Acknowledge that some refinement of the initial proposals may be
        needed after trials.  To this end try to work expeditiously to
        provide a nearly right solution that delivers most of the gains
        rather than refining the solutions endlessly before any
        implementation (in line with the IETF's usual way of developing

4.2.1.  Development and Authorisation of the Changed Processes

SD: I got lost on "the development process" - is this actually "the process 
of developing process"? I know what I think a development process is, and 
this isn't it.

   P3. The development process must incorporate taking advice from
       *  the IESG, the IAB, the IAOC, and the Working Group chairs
       *  legal advisors

4.2.2.  The Content of the Changed Processes

SD: I would append "according to common sense, and relying on the appeals 
process to handle the breakdown of common sense in unusual cases".

   P7.  New process rules should be flexible enough to allow unusual
        cases to be handled according to common sense.

4.2.3.  Management and Leadership in the Changed Processes

SD: if we have to talk about miscarriages, could we at least call them 
"miscarriages of process"?

   The IETF prides itself on being a technically led organization (i.e.,
   our leaders are primarily technologists, rather than managers) with
   open processes which can, whenever possible, be challenged by
   participants if they feel there has been a miscarriage.

SD: I know what you're trying to say, but I think it's more like "processes 
should ALLOW that people in leadership positions remain knowledgeable", and 
even this seems snaky - I believe I remember Bradner explaining to me that 
ADs chair working groups just to make sure they don't turn into complete 
process weenies (to be fair, this was a number of years ago, and I could 
have dreamed the incident - but either way, I think we're only asking that 
the processes don't require a lobotomy, or a cessation of learning).

P13. Management and leadership of the standards process should be
        predominantly carried out by people who are technically
        competent in the area in which they are leading, and the
        processes should ensure that people in leadership positions
        remain knowledgeable about the state of the art (and beyond) in
        their areas of responsibility.

SD: "valuable" is an overstatement - "We should try to minimize the amount 
of "on the job" training required for leaders to become productive as 
quickly as possible after they are appointed."

   P14. The processes should ensure that it is possible for IETF
        participants to acquire or hone the particular skills needed for
        management and leadership positions in the IETF before taking on
        these positions, for example by forms of apprenticeship.  "On
        the job" training is valuable but we should try to give it in
        such a way that leaders are as productive as possible as soon as
        they are appointed.

SD: I would suggest "a period to contribute using their skill" - or are we 
really saying that long-term ADs can't maintain their technical skills? I 
thought not.

   P16. The processes should ensure that, so far as is possible, the
        pool of skills and experience developed by those in leadership
        and management positions remains available to the IETF and
        continues to be seen to be valued whilst giving these leaders a
        period to refresh their skill in the technical arena.

4.2.6.  Areas

SD: The majority of the IETF membership likely has little clue about "Kobe", 
except as a type of steak - it would be nice to provide any pointer as 
background for this point of first use (not everyone spends hours on IETF 
archeology, fascinating as it often is). Also - "the central focus of ... 
technical expertise in the IETF" seems slightly overblown - I'd go for "the 
central focus of ... cross-area technical expertise in the IETF", but at 
least one adjective seems indicated.

   Areas have become a fundamental structuring mechanism for IETF work
   since the Kobe reorganization.  Area Directors (ADs) are at the
   moment the central focus of management and technical expertise in the
   IETF.

SD: "ambit" is a perfectly lovely word, but I'm a 51-year-old native 
(American) English speaker and had never seen it used before. Perhaps, 
"sphere of influence" (from www.webster.com)?

   P25. Besides experts in particular areas, it is important that the
        IETF also takes an architectural view and encourages
        participants who are able to work at the overall architectural
        level both to direct the standards development work in the IESG
        ambit; and to set technical guidelines internally, monitor
        technical developments in the research and commercial world out
        side the IETF and interact with external groups.

4.2.7.  Working Groups

SD: I could be misspeaking here, but thought that we were headed toward a 
world where WG chairs are ALSO responsible for shepherding deliverables 
through the approval and publication process - perhaps including this as a 
principle is premature, but I needed to ask. Which also makes me wonder 
about using one continuous index across multiple sections for principle 
numbers - you'd like to add a P29 here, but then you have to renumber the 
rest of the way down.

   P27. WG Chairs are responsible for ensuring that WGs execute their
        charters, meet their milestones, and produce deliverables that
        are ready for publication.
   P28. WGs should be primarily responsible for the quality of their
        output, and for obtaining and responding to cross-area and other
        external review.

4.2.8.  Extended Review

SD: A number of us were SIRS volunteers, but I think the statement that "The 
SIRS experiment ... [has] shown the value of extended, cross-area review of 
documents" is grossly generous. SIRS showed that reviewers would volunteer 
(for some number of "reviewers"), that the IETF has the attention span of a 
hamster when it comes to evaluating process change, and that WGs had no 
appetite for additional review steps that didn't bypass review steps later 
on. IMO. I would also note that I've gotten pushback on technical objections 
based on how late in the approval cycle Gen-ART reviews happened - "haven't 
these guys suffered enough?", and "jeez, they deployed this stuff four years 
ago anyway, and it sort of works". Which leads me to suggest a P33 - 
"Technical show-stoppers are always relevant, at any point in the approval 
process".

   The SIRS experiment and the General Area review team have shown the
   value of extended, cross-area review of documents.  Currently most
   such review is carried out too late in the document life cycle for
   best effectiveness.

5.1.  Components Not Considered for Major Change in this Document

SD: "This process" - the PESCI process? or something else?

   ISOC      ISOC acts as IETF's legal parent and guarantor.  This
             process is not at liberty to change the relationship
             between ISOC and IETF.

SD: I cannot get from the following IAB writeup to the conclusion that there 
is no reason to change the IAB. The second paragraph seems justification 
enough to include IAB changes as in-scope, without reference to anything 
else..

   IAB       The main function of the IAB is architectural oversight of
             developments in the Internet.  This includes monitoring
             standards development work to ensure that it does not
             adversely impact the existing Internet, setting guidelines
             for architectural aspects of development, helping to
             determine what new working groups should be chartered,
             keeping a watching brief on technical developments outside
             the IETF, and providing statements on such developments as
             appropriate.

             The IAB currently acts as the second line of appeal for
             decisions of the IESG on standards development and other
             matters.  This function is not an ideal fit with the
             general architectural remit of the IAB: the competencies of
             the IAB members do not necessarily equip them to deal with
             the process appeals that come to them from the IESG.  Some
             of the members of the IAB may consider them a distraction
             from their architectural role.

5.2.  Components Considered for Major Change in the Document

SD: I would append "for final cross-area review, only for ensuring that 
final cross-area review has been performed".

   IESG      The IESG is central to the processes of standards
             development in the IETF.  The steering role of the IESG in
             acceptance of new work, formation, and chartering of WGs,
             monitoring of WGs, and final stages of document processing
             seems to be appropriate, as it is essential to coordinate
             these functions.  This does not imply that the IESG must be
             solely responsible for final cross-area review.

SD: It may be worth noting that IESG decisions can be appealed, but that 
this is not a fast or cheap process.

             Another area where process changes might lead to greater
             confidence in the standards development process is the
             degree of self-regulation which the IESG exercises: in many
             process related disputes the IESG is expected to act as
             judge and jury on actions in which it is itself involved.

  Working Groups

SD: It may be worth appending, "appears to be difficult to achieve, 
especially since so many working groups disband when their specifications 
are published as proposed standards:.

             The main areas for concern here are ensuring high quality
             output and maintaining momentum over the period of
             development, particularly in the later stages when the
             creative work is essentially finished and the work is
             mainly polishing.  Active management by supervisors
             together with more rigorous and earlier review of documents
             seems to be required.  Taking work through from initial
             standardisation to full standard appears to be difficult to
             achieve,

   NOMCOM    The NOMCOM provides the means of selecting leaders and the
             volunteer managers for the IETF.  The process by which this
             is performed is time consuming and could be considered
             onerous for participants, but it is vital that the process
             remains as open as possible and endeavours to keep the IETF
             as free of undue external influences as possible.

SD: If you're scoping out NOMCOM process change parameters, please include a 
note about the desire to get better community input on non-incumbent 
candidates - discussed on IETF Discussion List, I believe?

             Changes to these processes should ensure that overlaps of
             role (as regards those performing the selection and the
             leaders being selected) do not arise and that the workload
             of the NOMCOM does not get out of hand.

6.1.  Change Process Proposal

SD: It bothers me that this document handwaves approval of process change 
("the IESG (and ISOC Board of Trustees) will respect that consensus and 
approve it") - who actually DECLARES the consensus that the IESG and ISOC 
BoT then respect? Note that I haven't got a better answer, but the default 
now looks like "Brian declares consensus (or lack thereof) based on [fill in 
the blank here], and if no one successfully appeals to IAB/ISOC BoT, we know 
where we're going" - and if this isn't the default, what is?

   The teams should function with an open discussion list, in the same
   way that the PESCI committee has done.  The result of the committee
   should be tested against the IETF consensus in the normal fashion; we
   believe that if there is clear IETF consensus that the proposal makes
   sense, the IESG (and the ISOC Board of Trustees) will respect that
   consensus and approve of it. 

_______________________________________________
Pesci-discuss mailing list
Pesci-discuss@ietf.org
https://www1.ietf.org/mailman/listinfo/pesci-discuss