Microsoft CodePlex data

Codeplex was Microsoft's open source code forge. It began in 2006 and shut down in 2017. We collected the data at the time of shutdown, and provided it here at FLOSSmole for anyone to use.

Data is available in raw format or in the FLOSSmole database.

Sample graphics

SELECT create_date, count(*)
FROM cp_projects
GROUP BY 1
ORDER BY 1 ASC;

Data Model

cp_projects table:
This table holds 400k records of projects that were - at some point - created on Codeplex. This includes "spam" projects and removed projects. However, not every project will have many details filled in about it. This is because the details come from their "index" page (see next table) and we were not able to get an index page for about 300k of the spam/canceled projects.

proj_name: the short name of the project
datasource_id: the collection number (70910 in this case)
proj_long_name: the long name of the project
proj_url: the url on codeplex
description: the long description of the project (some of these are quite long)
current_dl_version: the latest version number of the software that you can download
current_dl_date: the date that the current downloadable version was posted
download_count: the count of downloads of the latest version
proj_status: the last state the project was in (alpha, beta, etc)
proj_license: what license does this project use?
create_date: the date the project was first created (month/year only - comes from the history page)
last_updated: the last datetime that this record was updated in our database

cp_projects_indexes table (database only):
This table holds 100k home page, history page, and developers page records for every non-canceled project on Codeplex. NOTE: even though some spam projects were removed from the 400k list above, there are still many projects in this list that I would consider "spam" since they are gibberish and hold no real code or data, and are being used for advertising.

proj_name: the short name of the project
datasource_id: the collection number (70910)
home_html: the Codeplex home page for the project
history_html: the Codeplex history page for the project (this is where we get the cp_projects.create_date from)
people_html: the Codeplex developer page where the developers for each project are listed

cp_project_history:
Lists the 864k unique history events (such as page changes) for the projects, during each collection (70910 in this case). Each project can have multiple of the same event during a given period.

history_id: a unique identifier for each history event (primary key)
proj_name: the short name for a project
datasource_id: the collection number
month, year: when the history event took place
page_name: what page was updated
page_url: what is the url of the page that was updated
author, author_url: who updated the page?
last_updated: the timestamp for when this record was last updated

cp_project_people
Lists the 49k individual people that are working on ANY Codeplex project.

username: the Codeplex system username of the person
datasource_id: the collection number
personal_statement: typed by the person, tells something about themselves
member_since: the date the person joined
last_visit: the last date they used the site
user_html: the home page for the user on Codeplex
last_updated: the timestamp for when this record was last updated

cp_project_people_roles
Lists the person, project, and what role they had on that project at the time of collection. A person can only have a single role on a given project at one time.

proj_name: the short name of the project
datasource_id: the collection number
username: the person's Codeplex username
role: what their job was on this project
project_member_since: when they joined the project
last_updated: the timestamp for when this record was last updated

Google Code Project Create Dates

Project creation dates for every Google Code project from February 4, 2011 (when they first started tracking project creation dates) and when Google Code was shut down March 12, 2015.

Click to enlarge

Data is available in a gzipped CSV file or in the FLOSSmole database.

LKML (email) study: data/paper available

We presented this paper at the 2016 OpenSym this week.

Schneider, D., Spurlock, S., and M. Squire. (2016). Differentiating Communication Patterns of Leaders on the Linux Kernel Mailing List. In Proceedings of the 12th International Symposium on Open Collaboration (OpenSym 2016).

Biblio
PDF
Slides (Google Docs)
Data Available (Torvalds & Kroah-Hartman emails, source code removed): data dump, request MySQL database access

How do various corporations populate the Apache projects?

With the FLOSSmole Apache Project/Contributor/Roles data we updated earlier today, we thought an interesting initial analysis would be to figure out how various corporations populate the Apache projects (at least according to the official lists of contributors posted on each Apache project page).

Here is a list of the Apache projects with the highest density of participation by a single corporation:


We only show the first page of results here.

How did we get this result?

1. We used the Apache Contributor Data Set described in this previous FLOSSmole blog posting. Each project in Apache family of projects lists their members, and sometimes they list what company that person works for. Here is an example from the Geronimo Project.

Not every project lists its members, and not every project lists its members' affiliations.

2. We limited our analysis to datasource_id 65935, or May 18, 2016

3. We created a view in SQL so that we could more easily calculate the percentage of the total number of developers for each project:

SELECT project_name, count(*) as 'devcount'
FROM apache_people_projects
WHERE real_name IN (
SELECT distinct(real_name) FROM `apache_people_projects` WHERE datasource_id=65935
ORDER BY `apache_people_projects`.`real_name`)
AND datasource_id=65935
GROUP BY 1 ORDER BY 1 ASC;

4. Then we ran this SQL query to generate the data shown in the table above. The rows are sorted by the highest percent.

SELECT app.project_name, app.organization, count(*) as 'org dev count', app2.devcount as 'all devs', cast((count(*)/devcount)*100 as decimal(4,2)) as 'pct of team' FROM apache_people_projects app
INNER JOIN apache_project_dev_count_65935 app2
ON app.project_name = app2.project_name
WHERE app.real_name
IN (SELECT distinct(real_name) FROM `apache_people_projects` where datasource_id=65935)
AND app.organization IS NOT NULL
AND app.organization !=""
AND app.datasource_id=65935
GROUP BY 1, 2
ORDER BY 5 DESC, 1 asc;

Interested in getting this data? Apache Contributor Data Set

Want to see more examples of how to use FLOSSmole data? Examples

New "Apache Projects & Contributors" data dump

I spent a few days in May updating the list of all the Apache project contributors (full name & Apache system name when available) and their organizations when available. This data set was first released in 2013 in the MSR paper entitled "Project Roles in the Apache Foundation: A Data Set".

Fields:

  • svn_id
  • real_name
  • web_site
  • datasource_id
  • project_name
  • role_on_project
  • details
  • email
  • organization
  • timezone
  • last_updated

here is a sample of what the data looks like:


click to enlarge

Most of the fields are nullable since many times the data is incomplete.

Download the flat file, or use the live database ('apache') on the FLOSSmole MySQL server.

Django IRC Contributions Graph

Django IRC D3 CONTRIBUTIONS GRAPH

This graph represents The number of posts in the Django IRC logs. The lighter green squares represent days with less posts than the darker green squares. Months go from left to right and are separated by the darker lines. Days go by columns from left to right.

SQL Code

SELECT YEAR( date_of_entry ) , MONTH( date_of_entry ) , DAY( date_of_entry ) , COUNT( `type` )
FROM `django_irc`
WHERE TYPE = 'message'
GROUP BY 1 , 2, 3
ORDER BY YEAR( date_of_entry ) ASC, MONTH( date_of_entry ) ASC,DAY( date_of_entry ) ASC

Source code

This visualization was created using D3 and JavaScript. To view similar source code on GitHub head over to the FLOSSmole Github repositories

Want to see more examples of how to use FLOSSmole data? Examples

Ubuntu IRC Contributions Graph

Ubuntu IRC D3 CONTRIBUTIONS GRAPH

This graph represents The number of posts in the UbuntuIRC logs. The lighter green squares represent days with less posts than the darker green squares. Months go from left to right and are separated by the darker lines. Days go by columns from left to right.


SQL Code

SELECT YEAR( date_of_entry ) , MONTH( date_of_entry ) , DAY( date_of_entry ) , COUNT( `type` )
FROM `ubuntu_irc`
WHERE TYPE = 'message'
GROUP BY 1 , 2, 3

Source code

This visualization was created using D3 and JavaScript. To view similar source code on GitHub click here

Perl6 Data Visualizations

The following are a few examples of some quick queries and visualizations we made to show how to use the Perl6 IRC data.

(1) Posts by hour of day over the years

SQL code:

SELECT YEAR(`date_of_entry`),HOUR(`time_of_entry`),COUNT(HOUR(`time_of_entry`))
FROM `perl6_irc`
GROUP BY 1,2

(2) Perl6 IRC Posts by hour

SQL Code:

SELECT HOUR( `time_of_entry` ) , COUNT( HOUR( `time_of_entry` ) )
FROM `perl6_irc`
GROUP BY 1

(3) Perl6 Developer IRC Posts Over Time

SQL Code:

SELECT `send_user` , YEAR( `date_of_entry` ) , MONTH( `date_of_entry` ) , `date_of_entry` , COUNT(
TYPE )
FROM `perl6_irc`
WHERE `send_user` = 'pmichaud'
OR `send_user` = 'audreyt'
OR `send_user` = 'TimToady'
OR `send_user` = 'jnthn'
GROUP BY 1 , 2, 3

Bitcoin IRC Graphs

Here are some graphs and SQL to show how to use the Bitcoin IRC data. These charts track the activity of three main developers of Bitcoin: Mike Hearn, Gregory Maxwell, and Gavin Andresen. These three developers were featured in a New York Times article about Bitcoin developers.

(1) Bitcoin Developers IRC Posts Over Time




Click to enlarge image

SQL Code

SELECT `send_user` , YEAR( date_of_entry ) , MONTH( date_of_entry ) , COUNT( `type` )
FROM `bitcoindev_irc`
WHERE `send_user` = 'gavinandresen'
GROUP BY 1, 2 , 3;

(2)Bitcoin Developers IRC Posts by Total Percentage




Click to enlarge image

SQL Code

SELECT YEAR( date_of_entry ) , MONTH( date_of_entry ) , COUNT( `type` )
FROM `bitcoindev_irc`
WHERE TYPE = 'message'
GROUP BY 1 , 2;

(3) Bitcoin IRC Log by Day of the Week




Click to enlarge image

SQL Code

SELECT DAYOFWEEK( date_of_entry ) , COUNT( `type` )
FROM `bitcoindev_irc`
WHERE TYPE = 'message'
GROUP BY 1 ;

Bitcoin IRC D3 Contributions Graph

BITCOIN IRC D3 CONTRIBUTIONS GRAPH

This graph represents The number of posts in the Bitcoin IRC logs. The lighter green squares represent days with less posts than the darker green squares. Months go from left to right and are separated by the darker lines. Days go by columns from left to right.

SQL Code

SELECT YEAR( date_of_entry ) , MONTH( date_of_entry ) , DAY( date_of_entry ) , COUNT( `type` )
FROM `bitcoindev_irc`
WHERE TYPE = 'message'
GROUP BY 1 , 2, 3

Source code

This visualization was created using D3 and JavaScript. To view the source code on GitHub click here

Syndicate content