megan's blog

Pastebin use on FLOSS social media, and the downsides

A pastebin is a web site where developers can paste in some code, get back a URL, and then share that with others. The usage of pastebins is handy for IRC chat or in email, when a lot of source code will look ugly or be unformatted. However, the pastebin URLs disappear over time and this presents a problem for those of us who collect old data, or want to study the software evolution.

My former student Amber Smith (Elon, '14) and I wrote a paper investigating the rise of the pastebin as it is used in FLOSS mailing lists specifically. We wanted to know how they developed and whether their adoption curve followed classic Diffusion of Innovation theory. We also wondered how different types of pastebins evolved differently, and whether there was pushback about using them.

The paper was presented at OSS2015 in Florence May 16-17. Get the paper over at the FLOSShub.org/biblio!

FLOSS as a source for insults

FLOSSmole is hosting the data from a new paper: FLOSS as a source for profanity and insults: Collecting the data

Get the data

Get the slides

Hearsay Culture radio show link

I appeared (?) on David Levine's radio show Hearsay Culture out of KZSU-FM (Stanford U.) today. 10am PST for the stream, or listen later on a podcast at HearsayCulture.com.

A few times in the show we referred to various things; here are the relevant links:
--some slides we were looking at
--a forthcoming paper about insults and profanity in FLOSS will be presented at HICSS in January 2015
--if you want to check out the data, we have flat files and database access available. There are a few datasources not yet cleaned enough for public consumption (mostly irc and email data) but you can get a sense of the types of data we have

Slides for All Things Open presentation

I'm presenting today at All Things Open in Raleigh on Why and How Researchers are Studying Open Source. My goal is to show a wide variety of papers (not necessarily the "best" or most oft-cited papers, but a variety of techniques and motivations) in order to give developers a high-level idea of what kind of research is happening with the open source artifacts that they create.

Slides

UPDATE: based on a cursory count of Twitter attention, it seems like the most popular part of the presentation was the findings from Guzman et al from MSR 2014 showing that Java code comments were the most negative and the code comments were more negative on a Monday. :)

New schema for IRC data

In my continuing quest to be organized, I've created a new schema to hold just the IRC log data. On the database server (access instructions here), there is a new schema called 'irc' and it includes (for now) Ubuntu logs, Django logs, 7 Apache projects, and the topic lines from Freenode for all channels with 3+ users.

Coming soon: email updates, including Linux Kernel Mailing List (LKML) and more IRC (Wordpress, etc).

Enjoy!

Freecode is no longer updating

Freecode (formerly Freshmeat), the directory of Free and Open Source Software Projects, is no longer accepting new submissions. As of June 18, their site has this message on top:

The last full scrape of the Freecode RDF files took place in March. The data for the March 2014 Freecode collection is available for download from the FLOSSmole FC data site (or in the MySQL database).

RIP Freecode, nee Freshmeat.

Last Rubyforge Collection

The last Rubyforge collection happened yesterday. The datasource_id = 12987. All the data is located on our file downloads site, or in the database (ossmole_merged schema, tables prefixed 'rf', use datasource_id=12987 in your SQL queries).

RIP Rubyforge! We have been collecting from there for 10 years. Charts and graphs coming soon.

rubyforge shuts down

Django IRC data loaded into database

Django is a Python web framework. And of course it is an open source project. I have downloaded the entire collection of IRC logs for this project starting with the first logs from 2011. The logs are split into lines, parsed into fields (message, sender, time, date, etc) are now loaded into ossmole_merged database on our live MySQL server in a table called django_irc.

Each datasource_id represents one day's log file. Right now we have datasource_id 8442-9435.

We will update the collection periodically.

Usage example:

SELECT about_user, count(*)
FROM django_irc
GROUP BY about_user
ORDER BY 2 desc;

Like the Apache IRC logs, the Django IRC data will not be released as flat files since it's already available at the original django-irc-logs site.

New March 2014 data released

Some new forge data has been released collected 04-Mar-2014.

Datasource_id's are as follows:

8079 - freecode
8080 - rubyforge
8081 - objectweb
8082 - savannah
8083 - tigris
8084 - alioth

IRC data:
8085 - 8134: Apache ServiceMix
8135 - 8185: Apache Camel
8186 - 8236: Apache ActiveMQ
8237 - 8287: Apache CXF
8288 - 8338: Apache-Aries
8339 - 8389: Apache Kalumet
8390 - 8440: Apache Karaf

Data is available either in the flat files or by direct database access. Happy digging!

New Apache project IRC data

Hello moles! Happy January. Here are some fresh new data sources for your mining pleasure:

1. Freenode channel list and topics (all public channels with 3 or more users). The table is called "fn_irc_channels".
2. Apache Activemq IRC logs (one datasource_id per day, one row per message).
3. Apache Aries IRC logs
4. Apache Camel IRC logs
5. Apache CXF IRC logs
6. Apache Karaf IRC logs
7. Apache Kalumet IRC logs
8. Apache Servicemix IRC logs

here is a sample of what the structure looks like for 2-8:

CREATE TABLE `apache_servicemix_irc` (
`datasource_id` int(11) NOT NULL,
`line_num` int(11) NOT NULL,
`full_line_text` varchar(500) NOT NULL,
`line_message` varchar(500) NOT NULL,
`date_of_entry` date NOT NULL,
`time_of_entry` varchar(5) NOT NULL,
`type` enum('action','system','message') NOT NULL,
`about_user` varchar(50) NOT NULL,
`last_updated` datetime NOT NULL,
PRIMARY KEY (`datasource_id`,`line_num`)
)

These are available on the live MySQL connection.

Syndicate content