Less Talk, More Do » Digg Statistical Data

Tracking the top Diggers

Christopher Finke — Sat, 24 May 2008 04:02:54 +0000

For the past 15 months, I've been maintaining a list of the top 100 (and top 1000) Digg users. As Digg has become less and less relevant to my interests (and the interests of the greater tech community), I've decided to stop updating these lists. However, I have chosen to provide the tools I have used in case someone else has an interest in tracking Digg statistics.

Caveat: This whole thing was thrown together without any regard for coding or design conventions. Also, I can't guarantee that you won't be banned from Digg for using any of this information - I was, a few times. Proceed with caution.

First, the data structures. The Top Diggers lists are supported by two database tables: `digg` and `digg_users`. `digg` holds the set of front-page stories and their submitters, while `digg_users` stores aggregated information about each user in the system.

CREATE TABLE `digg` (
  `user` varchar(128) NOT NULL default '',
  `submission` varchar(255) NOT NULL default '',
           `date` datetime NOT NULL default '0000-00-00 00:00:00',
  PRIMARY KEY  (`submission`),
  KEY `user` (`user`)
);

CREATE TABLE `digg_users` (
  `user` varchar(128) NOT NULL default '',
  `frontpage` int(11) NOT NULL default '0',
  `dugg` int(11) NOT NULL default '0',
  `submitted` int(11) NOT NULL default '0',
  `profileViews` int(11) NOT NULL default '0',
  `frontpagestatic` int(11) NOT NULL default '0',
  `frontpagetotal` int(11) NOT NULL default '0',
  `submittedstatic` int(11) NOT NULL default '0',
  `image` varchar(128) NOT NULL default '',
  UNIQUE KEY `username` (`user`)
);

Next, some data to get you started. Here is a SQL dump of the two tables described above, including information on about 13000 Digg users (for `digg_users`), as well as the last 4 stories on the Digg homepage at the time of the last update (for `digg`). Import this data into your database in preparation for the next step.

The next step: data retrieval. The main work of updating the top 100 list is done by a Python script:

import digg

# Grab the last X pages of popular stories
digg.update_news(100)

# Update the top X profiles
digg.update_profiles(110)

Of course, this code makes no sense without the digg.* methods, downloadable here. This script also requires the excellent BeautifulSoup Python HTML parser. You will have to modify digg.py to change the database connection parameters.

For those who don't care to read through the code, it achieves two main objectives: Find out who submitted any frontpage stories since the last update (it stops when it hits a story already in the `digg` table), and using that information, determine the new top 100 users and update their profile information.

The last step is data presentation. The information in the database tables needs to be transformed into a readable HTML file. I'll leave this step as an exercise for the reader, but to get you started, this SQL query will get you the data you want in an easy-to-read format:

SELECT
   user `Username`,
   frontpagetotal `Frontpage Stories`,
   submitted `Stories Submitted`,
   dugg `Stories Dugg`,
   profileViews `Profile Views`
FROM digg_users
WHERE
   frontpagetotal <= submitted
ORDER BY
   `Frontpage Stories` DESC,
   `Stories Submitted` ASC,
   `Stories Dugg` DESC
LIMIT 100

So to sum up, if you want to manage your own "Top 100 Diggers" list, take the following steps:

1. Import the dump of digg data linked above.
2. Set up and run your scripts
3. Create a readable version of the data.

Have fun, and beware the Digg ban-hammer.

Kevin Rose removed from Top Diggers Lists

Christopher Finke — Wed, 26 Sep 2007 21:27:34 +0000

Some of you may have noticed that Kevin Rose (Digg user kevinrose) no longer appears in both the Digg's Top 100 and Top 1,000 Users lists. It was recently brought to my attention that Kevin's profile page shows that he has submitted 295 stories to Digg, but somehow, 298 of his submissions have reached the front page.

Gaming the system like this will not be tolerated, and any Digg users caught breaking the laws of mathematics will immediately be banned from both Top Users lists.

Top 1000 Diggers as of 2007/09/26

Christopher Finke — Wed, 26 Sep 2007 21:09:56 +0000