Digg, Digg Statistical Data, Python

Tracking the top Diggers

For the past 15 months, I’ve been maintaining a list of the top 100 (and top 1000) Digg users. As Digg has become less and less relevant to my interests (and the interests of the greater tech community), I’ve decided to stop updating these lists. However, I have chosen to provide the tools I have used in case someone else has an interest in tracking Digg statistics.

Caveat: This whole thing was thrown together without any regard for coding or design conventions. Also, I can’t guarantee that you won’t be banned from Digg for using any of this information – I was, a few times. Proceed with caution.

First, the data structures. The Top Diggers lists are supported by two database tables: `digg` and `digg_users`. `digg` holds the set of front-page stories and their submitters, while `digg_users` stores aggregated information about each user in the system.

CREATE TABLE `digg` (
  `user` varchar(128) NOT NULL default '',
  `submission` varchar(255) NOT NULL default '',
           `date` datetime NOT NULL default '0000-00-00 00:00:00',
  PRIMARY KEY  (`submission`),
  KEY `user` (`user`)
);

CREATE TABLE `digg_users` (
  `user` varchar(128) NOT NULL default '',
  `frontpage` int(11) NOT NULL default '0',
  `dugg` int(11) NOT NULL default '0',
  `submitted` int(11) NOT NULL default '0',
  `profileViews` int(11) NOT NULL default '0',
  `frontpagestatic` int(11) NOT NULL default '0',
  `frontpagetotal` int(11) NOT NULL default '0',
  `submittedstatic` int(11) NOT NULL default '0',
  `image` varchar(128) NOT NULL default '',
  UNIQUE KEY `username` (`user`)
);

Next, some data to get you started. Here is a SQL dump of the two tables described above, including information on about 13000 Digg users (for `digg_users`), as well as the last 4 stories on the Digg homepage at the time of the last update (for `digg`). Import this data into your database in preparation for the next step.

The next step: data retrieval. The main work of updating the top 100 list is done by a Python script:

import digg

# Grab the last X pages of popular stories
digg.update_news(100)

# Update the top X profiles
digg.update_profiles(110)

Of course, this code makes no sense without the digg.* methods, downloadable here. This script also requires the excellent BeautifulSoup Python HTML parser. You will have to modify digg.py to change the database connection parameters.

For those who don’t care to read through the code, it achieves two main objectives: Find out who submitted any frontpage stories since the last update (it stops when it hits a story already in the `digg` table), and using that information, determine the new top 100 users and update their profile information.

The last step is data presentation. The information in the database tables needs to be transformed into a readable HTML file. I’ll leave this step as an exercise for the reader, but to get you started, this SQL query will get you the data you want in an easy-to-read format:

SELECT 
   user `Username`, 
   frontpagetotal `Frontpage Stories`, 
   submitted `Stories Submitted`, 
   dugg `Stories Dugg`, 
   profileViews `Profile Views`
FROM digg_users 
WHERE 
   frontpagetotal <= submitted 
ORDER BY 
   `Frontpage Stories` DESC, 
   `Stories Submitted` ASC, 
   `Stories Dugg` DESC 
LIMIT 100

So to sum up, if you want to manage your own “Top 100 Diggers” list, take the following steps:

1. Import the dump of digg data linked above.
2. Set up and run your scripts
3. Create a readable version of the data.

Have fun, and beware the Digg ban-hammer.

Standard
Digg

Giving up on Digg

I won’t be reading Digg any more. I’ve visited it almost daily for the past 2 years or so, but it’s been a long time since I’ve seen something on Digg that I haven’t already seen at another site (except for stories about Digg itself). The front-page news is always at least a day old, and there are far too few real news stories and way too many “LOL FUNNY PIC” and “OBAMA CAN HAZ ELEKSHUN” stories.

Oh, and where is that OpenID support that was promised to be added “sometime this year” in early 2007? Here is what I wrote in an e-mail to Tom Drapeau after hearing that:

I’m not normally a betting man, but I’d put a week’s pay against Digg actually rolling that out before December 31, 2007. Their ratio of features promised to features delivered is, how you say, crappy.

I’d say that I won that bet quite handily.

Standard
Digg, Digg Statistical Data, Social Media

Kevin Rose removed from Top Diggers Lists

Some of you may have noticed that Kevin Rose (Digg user kevinrose) no longer appears in both the Digg’s Top 100 and Top 1,000 Users lists. It was recently brought to my attention that Kevin’s profile page shows that he has submitted 295 stories to Digg, but somehow, 298 of his submissions have reached the front page.

Kevin Rose is more powerful than math

Gaming the system like this will not be tolerated, and any Digg users caught breaking the laws of mathematics will immediately be banned from both Top Users lists.

 

Standard
Digg, Digg Statistical Data, Social Media

Top 1000 Diggers as of 2007/09/26

This CSV files contains the username, number of frontpage stories, number of submitted stories, number of stories dugg, and number of profile views for the top 1,000 users on Digg.

Top 1000 Diggers as of 2007/09/26

If you’d like to be notified whenever I release a new dataset, you can subscribe to the Digg Statisical Data RSS feed, which will include only the dataset posts, or my main RSS feed, which is updated with all of my blog posts.

Standard
Digg, Digg Statistical Data, Social Media

Top 1000 Diggers as of 2007/09/12

This CSV files contains the username, number of frontpage stories, number of submitted stories, number of stories dugg, and number of profile views for the top 1,000 users on Digg.

Top 1000 Diggers as of 2007/09/12

If you’d like to be notified whenever I release a new dataset, you can subscribe to the Digg Statisical Data RSS feed, which will include only the dataset posts, or my main RSS feed, which is updated with all of my blog posts.

Standard
Digg, Digg Statistical Data, Social Media

Top 1000 Diggers as of 2007/09/05

This CSV files contains the username, number of frontpage stories, number of submitted stories, number of stories dugg, and number of profile views for the top 1,000 users on Digg.

Top 1000 Diggers as of 2007/09/05

If you’d like to be notified whenever I release a new dataset, you can subscribe to the Digg Statisical Data RSS feed, which will include only the dataset posts, or my main RSS feed, which is updated with all of my blog posts.

Standard