Re: [Last-Call] OT: change BCP 83 [Re: Last Call: BCP 83 PR-Action Against Dan Harkins]

[Colin Perkins <[e-mail]>]

Hi Adam,

On 2 Oct 2022, at 20:42, Adam Roach wrote:
> On 10/2/22 04:26, Stephen Farrell wrote:
>> I wonder if there's any less subjective metric that could be
>> applied to mailing list archives?
>
>
> If you're offering to put in the footwork, the general outline of what 
> I know how to do would take one of two paths. Both would start with 
> getting as complete a copy of the email archives as possible. This 
> used to be easily found online, and may yet be; but if it isn't, I'm 
> sure the tools team could give you assistance.

The IETF sill maintains IMAP access to the mail archive for lists hosted 
on ietf.org, so access to the emails is straightforward.

> Then you either:
>
>  * Take a suitably large random sample of messages over the past 37
>    years (work out the size of the corpus and determine what you want
>    your confidence interval to be), and assign a team to score which
>    ones they believe meet some relevant criteria (e.g., violate 
> today's
>    code of conduct). You'll want at least two people -- and preferably
>    more -- of differing backgrounds to look at each message to
>    countervail certain kinds of biases. Or

This would be time-consuming and expensive, but would likely give an 
interesting result.

>  * Use one of the several available forum management tools to
>    automatically score each message. Details vary, but most such tools
>    will generate both "toxicity" and "sentiment" scores that you can
>    plot over time. The ones I'm familiar with are run as a service, so
>    you'd need to perform some light API integration (which might be as
>    easy as piping formail into a curl command); although it's entirely
>    possible that offline tools are also available.
>
> Again, I know how to do this, but can't invest the resources. Let me 
> know if you're earnest, and I'll happily consult with you on getting 
> it to work.

I’m part of [a project](https://sodestream.github.io) that’s doing 
mailing list analysis of IETF data. and the recent [IAB AID 
workshop](https://www.iab.org/activities/workshops/aid/) also explored 
this topic.

We haven’t spent too much time looking at sentiment analysis, but my 
colleagues took a quick look at messages on the [e-mail] list.

The plots below show the average extent, expressed in the range 0…1, 
to which text in emails sent to that list in each year rate as positive, 
negative, or neutral sentiment, according to the [VADER Sentiment 
Analysis](https://github.com/cjhutto/vaderSentiment) library:


![](cid:[e-mail] 
"Unknown.png")

Redrawing the plot with a different range, to focus on the positive and 
negative sentiment categories, it’s clear that messages labelled as 
positive sentiment outweigh those labelled as negative, but there’s a 
significant fraction of negativity. Proportions don’t look to be 
changing significantly over time.

![](cid:[e-mail] 
"Unknown.png")

Sentiment analysis, of course, is a crude measure that doesn’t 
necessarily correlate with toxicity. It’d be interesting to analyse 
further, and look at the other mailing lists too.

If anyone’s interested in exploring the data further, [our 
project](https://sodestream.github.io)  will be at the hackathon at the 
London IETF in a few weeks - come talk to us.

Colin
(with thanks to Mladen Karan and Ravi Shekhar, cc’d, for wrangling the 
data)