Re: [ietf-nomcom] Changing the candidate selection model

[John C Klensin <[e-mail]>]

Sorry to have taken so long to get back to this.  Other work...

Joel, at the risk of being heretical, the job of the Nomcom is
not exclusively to "pick the best candidate".  It doesn't have
enough information to do that, partially because (with one
qualification that I'll discuss below) incumbents are a known
quantity while non-incumbent nominees are not.  So the job, in
practice, is to evaluate the strength of an incumbent's
performance, the advantages and disadvantages of long tenure,
and then to make a judgment about whether an alternate nominee
would be worth the risk than any non-incumbent might constitute.
Only if there is no incumbent does the Nomcom, in practice,
actually get to make a "best candidate" decision with all of the
Nominees evaluated on an equal basis.   

If you tell me that isn't what happens, I'll have to believe you
somehow because I don't have the inside information that you do,
but what I know about organizational behavior and selection/
election processes causes me to doubt that any other formulation
is plausible, even though "pick the best candidate" sounds good
in theory.   Even if that were the only factor, if there is an
incumbent who is willing to serve another term, a Nomcom
realistically has to balance the qualities and record of that
incumbent against the calculated risks posed by other nominees.  

Another job of the Nomcom --and the rules that enable it-- is to
ensure that the maximum number of qualified people are willing
to offer themselves as nominees for a given role.  If there are
impediments that discourage people from being nominated who
might be willing and able to serve, we should seek to eliminate
or reduce those impediments.  

Consider, for example, that we keep pushing the date at which
nominees are expected to tentatively commit themselves further
and further back. The RFC5078 timeline implies about eight
months between the first announcement of open positions and call
for nominees and the date on which decisions are announced.  Now
let's assume that we've got some potential nominees who have to
go to their management in June and say 

	"I'd like to sign up for a two-year position with the
	IETF.  It requires that you commit, right now, to
	dedicating two years of my time, starting next March, to
	this role which I may or may not get.   That commitment
	implies that I shouldn't be put on projects in the next
	eight months which would be in trouble if I had to drop
	them in March with, if we are lucky, a month's notice,
	so I'm on reduced functionality for eight months and
	then may largely disappear for two years.   
	
	"Incidentally, while the rules would permit us to back
	out in February if circumstances change,
	draft-dawkins-nomcom-openlist will make that much more
	obvious, and possibly an embarrassment to the company,
	especially if I'm the only non-incumbent nominee.  And,
	since, at least statistically, the Nomcom routinely
	returns incumbents, especially those who have served
	only one previous term, the odds of my actually being
	selected are low."

That scenario is _not_ a way to build large nominee pools.
Worse, our current expectation is essentially that the same
people will not only do it once but, after not being selected
that year, will come back to their managers three or four months
after the selections are announced and say "Remember what we
went through last time?  I think we should do it again.".
Perhaps large companies who are willing to donate a certain
fraction of their staffs to the IETF don't care.  Perhaps people
who work for such companies but who are in a position to
allocate very significant fractions of their time however they
like are unaffected.  

I think the IETF would be better off with much more diversity in
the nominee pools -- while I'm grateful for the dedication of
the companies involved, I don't believe that it is a good sign
that, of the membership on the IESG and IAB, three companies
support three people each and another one supports two.   I'm
not concerned about conspiracy theories here but about the
observation that, with nearly 25% of the Nomcom-appointed IESG
and IAB positions in the hands of four organizations, how much
trouble we would be in if one or more of them got into serious
trouble and withdrew support.

I do not believe that this proposal will solve the problem of
concentration of slots in the hands of a few organizations (if
indeed once considers it a problem).   But I think anything we
can do that broadens the Nomcom's effective range of choices
would be a good idea.

>From my point of view, your reaction to the proposal is based on
an assumption that the current model has no adverse
consequences.  I think it does and am suggesting that those
consequences are severe enough that it is time to make the
tradeoffs somewhat differently.

Some specific comments below.

--On Wednesday, May 27, 2009 10:39 -0400 "Joel M. Halpern"
<[e-mail]> wrote:

> Having read the draft, it seems to me that this is a bad idea.
> 
> Firstly, I am not sure what problem it solves.  There is
> apparently some concern about whether the nomcom is returning
> too many or too few incumbents.

Actually, that is not a concern or problem we were trying to
solve in this draft.  Instead, we started with the knowledge
that, whether it was reasonable or not, about half were returned
each time (as you note).  We examined the "ask the boss"
scenario outlined above and some similar ones and added in the
concerns about "running against" someone (especially an
incumbent) from  the same or a related organization that have
been expressed in conjunction with discussions of
draft-dawkins-nomcom-openlist and concluded that, on balance, we
would have much higher odds of having a good nominee pool when
it was needed if we separated incumbent evaluation from
consideration of new Nominees.  

We also believe (although we could be wrong, especially given
the tendency for work to expand to fill the available time) that
the separation would reduce the total workload, and time
requirements, on the Nomcom.  If it did and the effect of that
were to broaden the pool of Nomcom volunteers, that would also,
in our opinion, be good for the IETF.

>...
> At the same time, this proposal seems to actually make it
> harder for the nomcom to do a good job.  The goal is to
> appoint the best available person to the job.
> Choosing to re-appoint or not re-appoint an incumbent without
> comparison with considering who else may be available and
> qualified is not, it seems to me, able to meet that goal.

Our disconnect here lies in your use of "best available".
Certainly, that is correct if what you intended to say is "best
available among those who volunteered to be nominees.  But the
current model is almost certainly reducing the number of
available nominees.  In some ways, that makes the Nomcom's job
easier.  But encouraging people to nominate themselves against
incumbents who, statistically, will be returned anyway does not
give you a good pool of "available" nominees when the Nomcom
needs them.

If you don't consider it a breach of confidentiality (since you
would not need to discuss any particular candidate, I think it
does not), it would help in this discussion if you would reveal
the number of plausible nominees you had available for each slot
in the last year.

> It seems to me that the general notion of how many incumbents
> ought to be returned in any given cycle depends heavily on how
> many good alternatives there are. So I would hate to require
> the committee to make that decision without having that
> information.

It seems to me that, if a Nomcom starts making decisions about
how many incumbents to return, as a count, we are in deep
trouble.  Fred's rumored examples, in which "all of the
incumbents were deemed good to return but the nomcom felt
obliged to sacrifice one, and did so, with quite a bit of hurt
feelings around the community. [...] And one could just as
easily imagine a case in which the nomcom felt that none should
be returned." both strike me as serious mistakes.

Indeed, while nothing in either the current system or the
proposed one could stop a Nomcom from going down some sort of
"quota of incumbents to be returned" path if it decided it
wanted to, part of the intent of the proposal is to focus
attention away from that approach.   There are a few implicit
assumptions in this model which I will try to clarify in the
next draft.  One of them is that incumbents should be evaluated
on performance --both theirs and that of the body to which they
are contributing-- and returned there are signs of
non-performance, bad behavior, or burnout.  Of course, that
doesn't obligate those incumbents to be available for additional
terms.  But the read-in costs and uncertainties of putting in a
new person are sufficiently high that, if an incumbent is
serving well and effectively in a body that is working well and
wants to continue doing so, we should "let" him or her.  

If we don't believe that, then this is a bad proposal.  But,
again, if one judges from the statistics, that is, with rare
exceptions, exactly how Nomcoms are behaving.  The difference is
that they are behaving that way after exerting more or less
considerable efforts to collect other nominees (possibly making
those people unavailable when they are really needed, as
discussed above) and after spending lots of their time to see if
they can do better.

> More important than the abstract failure mode, there are
> multiple specific failure modes that would do a disservice to
> the community and the nomcom.  If the nomcom has many
> qualified incumbents, if doing a stand-alone evaluation it may
> choose to return rather more incumbents than it would if it
> knew that some of those seats had good nominees available.

You can't have it both ways.  One goal is well-qualified people
with minimum risk that they cannot or will not do the job.  If
that is it, then the discussion about incumbents above is
relevant because "good nominees" are always going to be a risk,
if only because we've had ample examples of the Peter Principle
playing out in the IETF as well as in other organizational
settings.  A different goal is to insure turnover by preferring
"good nominees" to incumbents.  If that is your preference, the
way to accomplish it is to be explicit about it and impose
rotation rules (term limits, more or less) with an exception
procedure.

> Similarly, if the nomcom wishes to choose among the
> incumbents, because it feels that the good ones would still
> represent returning too many, the nomcom really needs to know
> who else is available.  Conversely, it is quite possible that
> the nomcom would, in a stand-alone evaluation, decide that
> some incumbent was not quite as good as they would like, and
> choose not to appoint him. Only to discover that said
> incumbent is significantly better than any of the nominees
> they evaluate.

Here is where we have a significant difference of opinion that,
I believe, would make a real difference.  Let's assume that we
have an incumbent who is not very good but sort of getting by
despite several liabilities, and potential other nominees who
are unlikely to be any better.  At that point, it seems to me
that you are suggesting that, under the current system, the
Nomcom should select, not a "good" or "best" candidate (because
there isn't one) but rather pick the least bad one.  The
proposed system would certainly make that harder.  

By contrast, I believe that, if an incumbent is not performing
well, returning him or her is almost always a mistake.  It
weakens the IETF and the relevant body to have someone who is
underperforming continue.  Perhaps worse, we've had a number of
cases in which someone has been returned to a body in spite of a
record of bad behavior, only to have that person behave as if
reappointment constitutes endorsement of the behavior and
increase it.  We've also found that comments from Nomcoms or
others are of no help in preventing further inappropriate
behavior.

>From that perspective, if the Nomcom makes a decision that
someone is not performing adequately, I believe that person
should be retired or removed.   If the Nomcom cannot then find a
nominee in whom it has high confidence, I believe that any of
the three alternatives are better than going back to the
incumbent: the Nomcom searches harder and gets more creative,
the Nomcom decides to take the risk with a Nominee who has at
least not already proven to be inappropriate for the role, or
the Nomcom explicitly decides to not fill the position because
there are no plausible candidates.  In the latter case, I'd
expect the relevant body to be forced to either run short-handed
or restructure and that either of those options would be better
for the IETF that proceeding with weak leadership because some
earlier structure required it.

The proposed model makes the "retire the underperforming
incumbent" scenario a little more efficient and likely to
succeed, but the comments above apply even under the current
model: if a Nomcom deliberately puts a "least unsatisfactory"
nominee (whether incumbent or not) into a job, it is doing the
community no favors.    And, as I have suggested above, the
proposed model is reasonably likely to give the Nomcom a broader
range of choices and thereby lower the odds of getting into that
situation.

> The document seems to suggest that the answer to some of these
> issues is to assume that there are always enough good
> nominees, and that if there are not, maybe the seat should go
> unfilled.  This is sufficiently far from my understanding of
> the realities that it seems a dangerous assumption.

Again, my observation of the realities is that selection of the
"best available" person, even if that person is known to be weak
as a candidate, is not merely a danger, but an almost-certain
problem.

> There is another dimension in which I have serious concerns
> with this proposal.  One of the things the nomcom evaluates,
> for all bodies, is the balance of members.  That balance may,
> in different years, be reflective of different aspects of
> skills, knowledge, contacts, etc...   But That kind of
> balancing has been seen as important in may cases.
> Pre-selecting which incumbents will remain, and which will
> not, means that such balancing can not be used as part of the
> selection.

Again, I don't think so.  Resignations and other rare and
exceptional cases notwithstanding, a Nomcom does not get the
option of replacing (or considering replacing) more than half of
a given body in a given year.   Given that, any attempted
"balancing" has to consider balance with at least half the
membership of the relevant body as a given -- the balance must
consider the incumbents who are not up for renewal, a group whom
the Nomcom cannot affect.   So the question is whether choices
of which incumbents to return can significantly aid or interfere
with balance given that established base.  I believe that, if
the body is sufficiently out of balance that special
considerations of incumbents are needed, that lack of balance
will usually show up as disfunction and make it clear which
incumbents should be removed, regardless of the order in which
things are done.  Conversely, if the perceived imbalance is
dictated by principles too subtle to show up in that way, I'm
willing to sacrifice giving the Nomcom an option that would seem
to put "balance" ahead of "best candidate".  YMMD.

> Also, it is my opinion that this would significantly increase
> the time for the selection process.  The selection process is
> not linear.  As such, while the number of seats to be filled
> has some effect on the time, it is not a significant factor.
> (Issues of collecting input, and whether particular choices
> are easy or hard, make more difference.) While this work could
> possibly be overlapped with some portion of the nomination
> selection, it clearly would need to be completed, confirmed,
> and announced before nominations were considered usable, since
> nominations would clearly need these results if this process
> were to take place.  As such, I believe it would add several
> months to the process.

Here, I am at a disadvantage because I don't have your in-Nomcom
experience.   But my reading of the situation is that a review
of, at most, a dozen people, without any need to compare them to
others, is not going to take very long as compared to an
in-depth review of several people per open slot.  And, by
eliminating slots that need to be considered, and for which
nominees need to be collected, one can further reduce workload.
I don't see how it could be otherwise unless there is a desire
to drag things out.

> It is also the case that the differences in kinds of seats
> makes some difference.  Even if it were somehow concluded that
> this made sense for the IESG, I can not see why one would
> apply it to the IAB.  In the case of the IAB, one is not
> nominated against any one individual, but rather nominated for
> a seat in a pool.  As such, none of the arguments presented
> about the open nominee list seem to apply to the IAB.  And
> even more than the IESG (where some balancing issues do apply)
> the community consensus has seemed to me quite clear that
> balance of various kinds are a set of very important factors
> in selecting IAB members.

When we started with this, we started with an "IESG only"
assumption.  While it seemed useful to explore whether it could
(and should) also be applied to the IAB and any other
Nomcom-selected positions, I'd be happy to go back to "IESG
only" if that proposal made sense.

>...

best,
  john