Re: [Eligibility-discuss] some numbers

[Ted Lemon <[e-mail]>]

I think you’re missing my point, or I’m misunderstanding your response.

The thing I’m concerned about is that this algorithm that’s been proposed is going to disenfranchise a lot of people who would have been nomcom eligible if there had been an IETF 107.  But we don’t really have a way to test that, because we don’t know who would have shown up for IETF 107.   So we can’t do an A/B test on IETF 107.  But we do know who showed up for IETF 106, and for IETF 105, and for IETF 104, and so on.

So what I’m proposing is that you do an A/B test comparison using your algorithm as if it were now just past IETF 106, which didn’t happen, using the list of RFCs published at that time, and make a list of all the people who would not be nomcom eligible using this scenario, but are noncom eligible because IETF 106 did happen.  And since each IETF has slightly different attendance implications, run the same scenario for a few IETFs before that.

If nobody is disenfranchised, then the algorithm is perfect.  If some people are disenfranchised, that would be useful to know as we evaluate the effectiveness of the proposal.

Given how late it is where you are, I would be happy to take my answer in the morning… :)

> On Apr 21, 2020, at 9:08 PM, Stephen Farrell <[e-mail]> wrote:
> 
> 
> Hiya,
> 
> On 22/04/2020 01:59, Ted Lemon wrote:
>> Stephen, what results do you get if you run this on historical data,
>> say for the past two years?  The problem with this run is that it
>> doesn’t really tell us who this would disenfranchise under normal
>> circumstances, given that circumstances are anything but normal.
>> 
>> IOW, pretend that it’s just after IETF 106, and now run the old
>> algorithm, run the new algorithm (in the second case pretending IETF
>> 106 didn’t happen) and see who loses eligibility.  Do this again for
>> IETF 105, 104 and 103, so that we have more than one test case.
>> 
>> Does that make sense?  Is it hard to do?
> 
> I'm not 100% sure I'm answering the right question but I
> think that'd be the 357 people who are only eligible based
> on f2f meetings and not otherwise. (See the Venn diagram
> near the bottom of [2].)
> 
> Put another way: if we have no f2f meetings before the 2021
> nomcom are selected, and we adopted draft-02, then those
> are the people who'd "lose" (assuming nomcom-eligibility is
> a "win";-).
> 
> It is absolutely a non-trivial, interesting and subtle
> question as to whether we'd be ok with dropping that set
> of 357 people and replacing them with the set of 452 who
> would be eligible without having been at a f2f meeting
> recently.
> 
> Optimistically, if we do have another f2f meeting before
> nomcom2021 are picked, I'd guess many of those 357 people
> might re-gain eligibility, but I'd have to change code
> to put a number on that.
> 
> I guess I could also check how many of the 290 who are
> in the intersection have previously volunteered to be
> on a nomcom. (Another one for tomorrow or the next day:-)
> 
> Cheers,
> S.
> 
>> 
>>> On Apr 21, 2020, at 8:50 PM, Stephen Farrell
>>> <[e-mail]> wrote:
>>> 
>>> 
>>> Hiya,
>>> 
>>> With many thanks to Robert Sparks for writing up some initial code
>>> to query the datatracker DB exploring the results of drafts such as
>>> the one Brian and I wrote, we have some initial numbers to report.
>>> 
>>> If one applied the rules from our -02 draft [1] then if seems we'd
>>> end up with a similarly sized set of nomcom eligible folks without
>>> requiring f2f meeting attendance.
>>> 
>>> Details are at [2].
>>> 
>>> The status quo with only f2f meetings gives 647 people, the
>>> draft-02 scheme gives us 742 eligible people without requiring f2f
>>> attendance. 290 of those are eligible both ways. From eyeballing
>>> the lists, the eligible people who'd not usually be eligible seem
>>> no more crazy than any random subset of IETFers;-)
>>> 
>>> I didn't put the actual lists of names up as I'm allergic to
>>> publishing lists of people's names, but I'd be fine with publishing
>>> hashes of them if that helped, or with answering any on or off list
>>> queries. [2] does have instructions that allow you to replicate
>>> creating the lists yourself if you are willing to do a bit of work.
>>> (Before we could consider ourselves "done" here, I do think we'd
>>> want someone to have independently replicated results like this.)
>>> 
>>> For me, this means that draft-02 seems credible enough to pursue
>>> some more, based on the data and not only based on a hunch, which
>>> is good.
>>> 
>>> Happy to try answer queries or check out alternatives if people
>>> have any to suggest...
>>> 
>>> Cheers, S.
>>> 
>>> PS: I'm sure there'll be some errors in this somewhere. Apologies
>>> in advance for those and they are of course my fault and not
>>> Robert's nor Brian's ;-)
>>> 
>>> [1]
>>> https://tools.ietf.org/html/draft-carpenter-eligibility-expand-02 
>>> [2] https://github.com/sftcd/elig/blob/master/queries.md 
>>> <0x5AB2FAF17B172BEA.asc>-- Eligibility-discuss mailing list 
>>> [e-mail] 
>>> https://www.ietf.org/mailman/listinfo/eligibility-discuss
>> 
> <0x5AB2FAF17B172BEA.asc>