Re: [ietf-nomcom] Changing the candidate selection model

Sorry that I disappeared just before you sent this reply.

On balance I think that it was well-written and (even more exciting to me)
addresses MY issues that we've discussed, even the ones I think may very 
well not apply to you on corporate support.

If it seems to you that a positive statement would be helpful, please let me 
know, and I'll say so onlist. If it seems that I'd be more likely to wake up 
one or more flakes, please let me know, and I'll be quiet!

Thanks,

Spencer

> Sorry to have taken so long to get back to this.  Other work...
>
> Joel, at the risk of being heretical, the job of the Nomcom is
> not exclusively to "pick the best candidate".  It doesn't have
> enough information to do that, partially because (with one
> qualification that I'll discuss below) incumbents are a known
> quantity while non-incumbent nominees are not.  So the job, in
> practice, is to evaluate the strength of an incumbent's
> performance, the advantages and disadvantages of long tenure,
> and then to make a judgment about whether an alternate nominee
> would be worth the risk than any non-incumbent might constitute.
> Only if there is no incumbent does the Nomcom, in practice,
> actually get to make a "best candidate" decision with all of the
> Nominees evaluated on an equal basis.
>
> If you tell me that isn't what happens, I'll have to believe you
> somehow because I don't have the inside information that you do,
> but what I know about organizational behavior and selection/
> election processes causes me to doubt that any other formulation
> is plausible, even though "pick the best candidate" sounds good
> in theory.   Even if that were the only factor, if there is an
> incumbent who is willing to serve another term, a Nomcom
> realistically has to balance the qualities and record of that
> incumbent against the calculated risks posed by other nominees.
>
> Another job of the Nomcom --and the rules that enable it-- is to
> ensure that the maximum number of qualified people are willing
> to offer themselves as nominees for a given role.  If there are
> impediments that discourage people from being nominated who
> might be willing and able to serve, we should seek to eliminate
> or reduce those impediments.
>
> Consider, for example, that we keep pushing the date at which
> nominees are expected to tentatively commit themselves further
> and further back. The RFC5078 timeline implies about eight
> months between the first announcement of open positions and call
> for nominees and the date on which decisions are announced.  Now
> let's assume that we've got some potential nominees who have to
> go to their management in June and say
>
> "I'd like to sign up for a two-year position with the
> IETF.  It requires that you commit, right now, to
> dedicating two years of my time, starting next March, to
> this role which I may or may not get.   That commitment
> implies that I shouldn't be put on projects in the next
> eight months which would be in trouble if I had to drop
> them in March with, if we are lucky, a month's notice,
> so I'm on reduced functionality for eight months and
> then may largely disappear for two years.
>
> "Incidentally, while the rules would permit us to back
> out in February if circumstances change,
> draft-dawkins-nomcom-openlist will make that much more
> obvious, and possibly an embarrassment to the company,
> especially if I'm the only non-incumbent nominee.  And,
> since, at least statistically, the Nomcom routinely
> returns incumbents, especially those who have served
> only one previous term, the odds of my actually being
> selected are low."
>
> That scenario is _not_ a way to build large nominee pools.
> Worse, our current expectation is essentially that the same
> people will not only do it once but, after not being selected
> that year, will come back to their managers three or four months
> after the selections are announced and say "Remember what we
> went through last time?  I think we should do it again.".
> Perhaps large companies who are willing to donate a certain
> fraction of their staffs to the IETF don't care.  Perhaps people
> who work for such companies but who are in a position to
> allocate very significant fractions of their time however they
> like are unaffected.
>
> I think the IETF would be better off with much more diversity in
> the nominee pools -- while I'm grateful for the dedication of
> the companies involved, I don't believe that it is a good sign
> that, of the membership on the IESG and IAB, three companies
> support three people each and another one supports two.   I'm
> not concerned about conspiracy theories here but about the
> observation that, with nearly 25% of the Nomcom-appointed IESG
> and IAB positions in the hands of four organizations, how much
> trouble we would be in if one or more of them got into serious
> trouble and withdrew support.
>
> I do not believe that this proposal will solve the problem of
> concentration of slots in the hands of a few organizations (if
> indeed once considers it a problem).   But I think anything we
> can do that broadens the Nomcom's effective range of choices
> would be a good idea.
>
>>From my point of view, your reaction to the proposal is based on
> an assumption that the current model has no adverse
> consequences.  I think it does and am suggesting that those
> consequences are severe enough that it is time to make the
> tradeoffs somewhat differently.
>
> Some specific comments below.
>
> --On Wednesday, May 27, 2009 10:39 -0400 "Joel M. Halpern"
> <jmh@joelhalpern.com> wrote:
>
>> Having read the draft, it seems to me that this is a bad idea.
>>
>> Firstly, I am not sure what problem it solves.  There is
>> apparently some concern about whether the nomcom is returning
>> too many or too few incumbents.
>
> Actually, that is not a concern or problem we were trying to
> solve in this draft.  Instead, we started with the knowledge
> that, whether it was reasonable or not, about half were returned
> each time (as you note).  We examined the "ask the boss"
> scenario outlined above and some similar ones and added in the
> concerns about "running against" someone (especially an
> incumbent) from  the same or a related organization that have
> been expressed in conjunction with discussions of
> draft-dawkins-nomcom-openlist and concluded that, on balance, we
> would have much higher odds of having a good nominee pool when
> it was needed if we separated incumbent evaluation from
> consideration of new Nominees.
>
> We also believe (although we could be wrong, especially given
> the tendency for work to expand to fill the available time) that
> the separation would reduce the total workload, and time
> requirements, on the Nomcom.  If it did and the effect of that
> were to broaden the pool of Nomcom volunteers, that would also,
> in our opinion, be good for the IETF.
>
>>...
>> At the same time, this proposal seems to actually make it
>> harder for the nomcom to do a good job.  The goal is to
>> appoint the best available person to the job.
>> Choosing to re-appoint or not re-appoint an incumbent without
>> comparison with considering who else may be available and
>> qualified is not, it seems to me, able to meet that goal.
>
> Our disconnect here lies in your use of "best available".
> Certainly, that is correct if what you intended to say is "best
> available among those who volunteered to be nominees.  But the
> current model is almost certainly reducing the number of
> available nominees.  In some ways, that makes the Nomcom's job
> easier.  But encouraging people to nominate themselves against
> incumbents who, statistically, will be returned anyway does not
> give you a good pool of "available" nominees when the Nomcom
> needs them.
>
> If you don't consider it a breach of confidentiality (since you
> would not need to discuss any particular candidate, I think it
> does not), it would help in this discussion if you would reveal
> the number of plausible nominees you had available for each slot
> in the last year.
>
>> It seems to me that the general notion of how many incumbents
>> ought to be returned in any given cycle depends heavily on how
>> many good alternatives there are. So I would hate to require
>> the committee to make that decision without having that
>> information.
>
> It seems to me that, if a Nomcom starts making decisions about
> how many incumbents to return, as a count, we are in deep
> trouble.  Fred's rumored examples, in which "all of the
> incumbents were deemed good to return but the nomcom felt
> obliged to sacrifice one, and did so, with quite a bit of hurt
> feelings around the community. [...] And one could just as
> easily imagine a case in which the nomcom felt that none should
> be returned." both strike me as serious mistakes.
>
> Indeed, while nothing in either the current system or the
> proposed one could stop a Nomcom from going down some sort of
> "quota of incumbents to be returned" path if it decided it
> wanted to, part of the intent of the proposal is to focus
> attention away from that approach.   There are a few implicit
> assumptions in this model which I will try to clarify in the
> next draft.  One of them is that incumbents should be evaluated
> on performance --both theirs and that of the body to which they
> are contributing-- and returned there are signs of
> non-performance, bad behavior, or burnout.  Of course, that
> doesn't obligate those incumbents to be available for additional
> terms.  But the read-in costs and uncertainties of putting in a
> new person are sufficiently high that, if an incumbent is
> serving well and effectively in a body that is working well and
> wants to continue doing so, we should "let" him or her.
>
> If we don't believe that, then this is a bad proposal.  But,
> again, if one judges from the statistics, that is, with rare
> exceptions, exactly how Nomcoms are behaving.  The difference is
> that they are behaving that way after exerting more or less
> considerable efforts to collect other nominees (possibly making
> those people unavailable when they are really needed, as
> discussed above) and after spending lots of their time to see if
> they can do better.
>
>> More important than the abstract failure mode, there are
>> multiple specific failure modes that would do a disservice to
>> the community and the nomcom.  If the nomcom has many
>> qualified incumbents, if doing a stand-alone evaluation it may
>> choose to return rather more incumbents than it would if it
>> knew that some of those seats had good nominees available.
>
> You can't have it both ways.  One goal is well-qualified people
> with minimum risk that they cannot or will not do the job.  If
> that is it, then the discussion about incumbents above is
> relevant because "good nominees" are always going to be a risk,
> if only because we've had ample examples of the Peter Principle
> playing out in the IETF as well as in other organizational
> settings.  A different goal is to insure turnover by preferring
> "good nominees" to incumbents.  If that is your preference, the
> way to accomplish it is to be explicit about it and impose
> rotation rules (term limits, more or less) with an exception
> procedure.
>
>> Similarly, if the nomcom wishes to choose among the
>> incumbents, because it feels that the good ones would still
>> represent returning too many, the nomcom really needs to know
>> who else is available.  Conversely, it is quite possible that
>> the nomcom would, in a stand-alone evaluation, decide that
>> some incumbent was not quite as good as they would like, and
>> choose not to appoint him. Only to discover that said
>> incumbent is significantly better than any of the nominees
>> they evaluate.
>
> Here is where we have a significant difference of opinion that,
> I believe, would make a real difference.  Let's assume that we
> have an incumbent who is not very good but sort of getting by
> despite several liabilities, and potential other nominees who
> are unlikely to be any better.  At that point, it seems to me
> that you are suggesting that, under the current system, the
> Nomcom should select, not a "good" or "best" candidate (because
> there isn't one) but rather pick the least bad one.  The
> proposed system would certainly make that harder.
>
> By contrast, I believe that, if an incumbent is not performing
> well, returning him or her is almost always a mistake.  It
> weakens the IETF and the relevant body to have someone who is
> underperforming continue.  Perhaps worse, we've had a number of
> cases in which someone has been returned to a body in spite of a
> record of bad behavior, only to have that person behave as if
> reappointment constitutes endorsement of the behavior and
> increase it.  We've also found that comments from Nomcoms or
> others are of no help in preventing further inappropriate
> behavior.
>
>>From that perspective, if the Nomcom makes a decision that
> someone is not performing adequately, I believe that person
> should be retired or removed.   If the Nomcom cannot then find a
> nominee in whom it has high confidence, I believe that any of
> the three alternatives are better than going back to the
> incumbent: the Nomcom searches harder and gets more creative,
> the Nomcom decides to take the risk with a Nominee who has at
> least not already proven to be inappropriate for the role, or
> the Nomcom explicitly decides to not fill the position because
> there are no plausible candidates.  In the latter case, I'd
> expect the relevant body to be forced to either run short-handed
> or restructure and that either of those options would be better
> for the IETF that proceeding with weak leadership because some
> earlier structure required it.
>
> The proposed model makes the "retire the underperforming
> incumbent" scenario a little more efficient and likely to
> succeed, but the comments above apply even under the current
> model: if a Nomcom deliberately puts a "least unsatisfactory"
> nominee (whether incumbent or not) into a job, it is doing the
> community no favors.    And, as I have suggested above, the
> proposed model is reasonably likely to give the Nomcom a broader
> range of choices and thereby lower the odds of getting into that
> situation.
>
>> The document seems to suggest that the answer to some of these
>> issues is to assume that there are always enough good
>> nominees, and that if there are not, maybe the seat should go
>> unfilled.  This is sufficiently far from my understanding of
>> the realities that it seems a dangerous assumption.
>
> Again, my observation of the realities is that selection of the
> "best available" person, even if that person is known to be weak
> as a candidate, is not merely a danger, but an almost-certain
> problem.
>
>> There is another dimension in which I have serious concerns
>> with this proposal.  One of the things the nomcom evaluates,
>> for all bodies, is the balance of members.  That balance may,
>> in different years, be reflective of different aspects of
>> skills, knowledge, contacts, etc...   But That kind of
>> balancing has been seen as important in may cases.
>> Pre-selecting which incumbents will remain, and which will
>> not, means that such balancing can not be used as part of the
>> selection.
>
> Again, I don't think so.  Resignations and other rare and
> exceptional cases notwithstanding, a Nomcom does not get the
> option of replacing (or considering replacing) more than half of
> a given body in a given year.   Given that, any attempted
> "balancing" has to consider balance with at least half the
> membership of the relevant body as a given -- the balance must
> consider the incumbents who are not up for renewal, a group whom
> the Nomcom cannot affect.   So the question is whether choices
> of which incumbents to return can significantly aid or interfere
> with balance given that established base.  I believe that, if
> the body is sufficiently out of balance that special
> considerations of incumbents are needed, that lack of balance
> will usually show up as disfunction and make it clear which
> incumbents should be removed, regardless of the order in which
> things are done.  Conversely, if the perceived imbalance is
> dictated by principles too subtle to show up in that way, I'm
> willing to sacrifice giving the Nomcom an option that would seem
> to put "balance" ahead of "best candidate".  YMMD.
>
>> Also, it is my opinion that this would significantly increase
>> the time for the selection process.  The selection process is
>> not linear.  As such, while the number of seats to be filled
>> has some effect on the time, it is not a significant factor.
>> (Issues of collecting input, and whether particular choices
>> are easy or hard, make more difference.) While this work could
>> possibly be overlapped with some portion of the nomination
>> selection, it clearly would need to be completed, confirmed,
>> and announced before nominations were considered usable, since
>> nominations would clearly need these results if this process
>> were to take place.  As such, I believe it would add several
>> months to the process.
>
> Here, I am at a disadvantage because I don't have your in-Nomcom
> experience.   But my reading of the situation is that a review
> of, at most, a dozen people, without any need to compare them to
> others, is not going to take very long as compared to an
> in-depth review of several people per open slot.  And, by
> eliminating slots that need to be considered, and for which
> nominees need to be collected, one can further reduce workload.
> I don't see how it could be otherwise unless there is a desire
> to drag things out.
>
>> It is also the case that the differences in kinds of seats
>> makes some difference.  Even if it were somehow concluded that
>> this made sense for the IESG, I can not see why one would
>> apply it to the IAB.  In the case of the IAB, one is not
>> nominated against any one individual, but rather nominated for
>> a seat in a pool.  As such, none of the arguments presented
>> about the open nominee list seem to apply to the IAB.  And
>> even more than the IESG (where some balancing issues do apply)
>> the community consensus has seemed to me quite clear that
>> balance of various kinds are a set of very important factors
>> in selecting IAB members.
>
> When we started with this, we started with an "IESG only"
> assumption.  While it seemed useful to explore whether it could
> (and should) also be applied to the IAB and any other
> Nomcom-selected positions, I'd be happy to go back to "IESG
> only" if that proposal made sense.
>
>>...
>
> best,
>  john
>
> _______________________________________________
> ietf-nomcom mailing list
> ietf-nomcom@ietf.org
> https://www.ietf.org/mailman/listinfo/ietf-nomcom
>