Database reorganization

Today my new research assistant Gavan & I are performing some maintenance tasks on the database, including a reorganization of the places where the data tables live. Hopefully this will mean that the data is much better organized.

Here is the github summary of what we are doing, and a brief summary below.

We will leave old copies of the most popular tables for a few days, in order to give everyone time to rework scripts, etc.

Data Resources: 

Software Archaeology: GNUe IRC data & summaries

Back in the 2000's, the GNU Enterprise (GNUe) project chat logs (and human-created chat log summaries!) were used by several papers in the area of text summarization, especially dialogue summarization.

The reason the GNUe chat logs and summaries were used is that the logs were accompanied by summaries that were compiled periodically (manually) by a human. The summarized chat logs can thus be considered a kind of "gold standard" for what kind of summary a machine summarizer should produce.

Data Resources: 

Pastebin use on FLOSS social media, and the downsides

A pastebin is a web site where developers can paste in some code, get back a URL, and then share that with others. The usage of pastebins is handy for IRC chat or in email, when a lot of source code will look ugly or be unformatted. However, the pastebin URLs disappear over time and this presents a problem for those of us who collect old data, or want to study the software evolution.

FLOSS as a source for insults

FLOSSmole is hosting the data from a new paper: FLOSS as a source for profanity and insults: Collecting the data

Get the data

Get the slides

Data Resources: 

Hearsay Culture radio show link

I appeared (?) on David Levine's radio show Hearsay Culture out of KZSU-FM (Stanford U.) today. 10am PST for the stream, or listen later on a podcast at

Slides for All Things Open presentation

I'm presenting today at All Things Open in Raleigh on Why and How Researchers are Studying Open Source. My goal is to show a wide variety of papers (not necessarily the "best" or most oft-cited papers, but a variety of techniques and motivations) in order to give developers a high-level idea of what kind of research is happening with the open source artifacts that they create.


New schema for IRC data

In my continuing quest to be organized, I've created a new schema to hold just the IRC log data. On the database server (access instructions here), there is a new schema called 'irc' and it includes (for now) Ubuntu logs, Django logs, 7 Apache projects, and the topic lines from Freenode for all channels with 3+ users.

Coming soon: email updates, including Linux Kernel Mailing List (LKML) and more IRC (Wordpress, etc).


Data Resources: 

Freecode is no longer updating

Freecode (formerly Freshmeat), the directory of Free and Open Source Software Projects, is no longer accepting new submissions. As of June 18, their site has this message on top:

The last full scrape of the Freecode RDF files took place in March. The data for the March 2014 Freecode collection is available for download from the FLOSSmole FC data site (or in the MySQL database).

RIP Freecode, nee Freshmeat.

Data Resources: 

Last Rubyforge Collection

The last Rubyforge collection happened yesterday. The datasource_id = 12987. All the data is located on our file downloads site, or in the database (ossmole_merged schema, tables prefixed 'rf', use datasource_id=12987 in your SQL queries).

RIP Rubyforge! We have been collecting from there for 10 years. Charts and graphs coming soon.

rubyforge shuts down

Data Resources: 

Django IRC data loaded into database

Django is a Python web framework. And of course it is an open source project. I have downloaded the entire collection of IRC logs for this project starting with the first logs from 2011. The logs are split into lines, parsed into fields (message, sender, time, date, etc) are now loaded into ossmole_merged database on our live MySQL server in a table called django_irc.

Each datasource_id represents one day's log file. Right now we have datasource_id 8442-9435.

Data Resources: