The ten questions currently displayed at 10questions.com under the "Answers" tab are believed to be consistent with voting from individual visitors to the site, and have been so certified. The following is a summary of an internal audit conducted to establish this. It includes an examination of the error associated with each video's vote count and a brief discussion of patterns found in voting data.

In addition to limiting votes by IP address, we collected an assortment of contextual data to help aid in determining whether votes were consistent with the intended usage of the site (i.e., one vote per computer.) Though the use of IP address limiting does not guarantee this intended usage, the examination of additional data and trends helps to establish error bars. These speak to any deviations introduced by issues such as IP pooling and the repeated voting of users with non-static IP addresses.

Taking these into account, it was possible to estimate the percentage of votes for each video which may have been subject to such issues, including the deliberate reseting of IP addresses to facilitate multiple votes. For all questions in the top ten, this uncertainty fell between three and six percent. Of course this uncertainty makes the difference between the tenth and eleventh video harder to discern. However, since the eleventh video has a similar uncertainty and the range of net votes for the tenth video has higher numbers, the tenth video remains in place. Given that the twelfth question has a net vote count less than 96% of the tenth, we can conclude that the top ten questions really do represent those chosen by individual voters.

The Story Behind the Votes

After examining this uncertainty, we sought to determine if the patterns of voting seen were consistent with votes from individual people. Our driving question was simple, "Can we locate reasonable explanations behind peaks in voting that can be attributed to individuals?" If we could not, this would raise a red flag. In the end, we found the votes received by each of the top ten videos did indeed appear consistent with individual voting.

As you can see in this graph displaying all votes by day and type (positive or negative), there were three main peaks in voting.

The first occurs on the weekend of Friday the 26th of September. This weekend followed the first full week of round one. Additionally, we announced on Friday that Monday's top video would be asked of Barack Obama during his MySpace/MTV. If we look to possible traffic drivers, we find the peak correlates with a number of mobilization campaigns aimed at getting people to vote on specific questions. The most successful of these appears to have been an email sent on Sunday by MoveOn.org asking its members to vote on the the video "Question for Pres Candidates about Net Neutrality." Looking at the video's history we can clearly see this. We also know of organized mobilization from the Marijuana Policy Project and the Electronic Frontier Foundation presumably benefiting the questions "Medical Marijuana " and " Warrantless Wiretapping" respectively.

This peak can also be seen prominently in the voting histories of "Why won't you pledge to abolish corporate personhood?", "Is our voting system broken?", and "Two Party System Bad." To a lesser extent, it can be seen in the histories of "Is America unofficially a theocracy?" and " Transparency." ("Fair Elections Campaign Reform" and "Size of Government" were not yet on the site.) Additionally, webmail sites make up one of the top traffic sources for this weekend, adding credence to the inference that many people came to the site thanks to direct mailing efforts. However, it's worth noting that votes for the MoveOn video only accounted for about a third of this weekend's votes. Further discussion of the weekend can be found here.

The second peak occurs on November 4th and corresponds to increased traffic on the site due primarily to a Digg posting late the previous night. It was a call to action, asking people to vote on "Is America unofficially a theocracy?" Looking at the video's history this is very easy to see. However, voters on this question for the most part remained on site to vote on other videos as well. A more detailed discussion can be found here.

The third peak occurred at the closing of round one, and appears consistent with that expected from increased traffic and many different groups advocating for videos. Notably, this is the only peak found in the history of "10questions: Size of Government" as that video did not appear until the last day of voting. It is worth looking at closely as the video's rise was thanks to careful mobilization and some have questioned its legitimacy. Suffice it to say, no rules were broken. A more detailed discussion of this can be found here.

To quote Micah's summary blog posting the day after:

A look at our site traffic from yesterday shows 13,367 unique visits from more than 700 sources. The top referrers, other than direct visitors (who numbered nearly 5,500) were Hugh Hewitt, Patrick Ruffini, Crooks and Liars , RonPaulForums, Townhall, organic Google search, the Marijuana Policy Project, Digg, the Politico, ConservativeGrapevine, the New York Times, Why Tuesday, and Fair Elections Now. The sites Ruffini mobilized generated about 2600 unique visits out of that total. Throw in another 1000 visitors from random sources like email or Facebook that he presumably generated. That's not even a third of all the visitors we had during the day.

Most of the peaks visible in the videos' voting histories can be attributed to the increased traffic accompanying one of these three peaks. However, it's worth noting some additional activity. Question 4, "Warrantless Wiretapping" experienced a peak in the first week before other videos were on the site. This was sparked, at least in part, by a posting from the Electronic Frontiers Foundation on the 18th which found ripples in the blogosphere. Additionally, this video was referenced in many of the early blog posting and news reports surrounding 10questions.

Smaller scale peaks can be found for each of the other videos and appear consistent with similar moralization efforts and the sharing of questions among friends.

Did Anyone Try to Cheat?

The question on everyone's mind, however, is "Did anyone try to game the system?" The answer is "yes." We know of a few attempts by people to stuff the ballot box. None of them, however, were able to rise much above background noise. It appears that some videos were the target of attempts to vote them down while voting similarly ranked videos up to quicken their subject's decent. Ironically, such an attack was launched against one of the current top ten before it experienced any positive peaks--something it more than recovered from. There's also circumstantial evidence that some people may have attempted to use an ISP's IP pooling to vote more than once. However, this was taken into account when calculating the uncertainty of net votes (see discussion above).

The most interesting attempt to "cheat" had to be a scheme to pay people for votes over at Amazon's Mechanical Turk, a site which seeks to harness large pools of workers to do simple tasks. The specific request has since expired, but someone posted a job paying four cents per up vote. The person making the request was pointing people to vote on a video posted by someone with the same name. He was asking for 1000 votes, but in the end he only seemed to get a few dozen takers.

Overall we have strong confidence in the fact that the top ten questions really do hold their place thanks to the votes of individuals, not the stuffing of the ballot box. It is worth noting that while many benefited from organized efforts to get votes, in the end individuals had to decide to cast their votes as they saw fit, and many of the vistors driven to the site from elsewhere didn't necessarily vote on the video they were pointed to and/or stayed to vote on other videos.