Lucky vs. Good

The Washington Post had an article last week entitled The Top Pickers vs. The Pack describing web sites that attempt to discover "experts" through investment or sports picking...

A small number of Web sites seeking to turn the wisdom of the Internet on its head by sifting through its vast number of users to identify a handful of experts...the wisdom of the crowd could be outsmarted by what Michael Arrington, editor of the TechCrunch blog, recently dubbed the "wisdom of the few." Sites like PicksPal rely on input from the masses chiefly as a venue for auditioning prospective experts, on the theory that these virtuosos could provide even more accurate information and predictions than the crowd.

The author Alan Sipress appears excited by the potential of this approach, though he does allow for the possibility that this "phenomena" may be an illusion ("Some business professors remain skeptical, warning that luck can often be mistaken for expertise.") Meanwhile, Tom Jessiman, the founder of PicksPal, breathlessly remarks: "A very small set of people keep winning," Jessiman said. "It kind of blew me away."

Before we crown a gaggle of new geniuses, we thought it would be interested to simulate the results of a site like PicksPal if only luck was at play. PicksPal has a user base of 100,000 and identifies the 30 most successful players over the previous five weeks. ("Jessiman said he tries to rule out flukes at PicksPal by requiring that his experts play actively in three of the most recent five weeks."). What those top 30 experts would look like if they were entirely based on luck? To find out, we ran a simple Monte Carlo simulation for 13 games picked a week for three weeks (39 total games) with a 50% chance of winning each game.

    The results: The top 30 winners predicted almost 80% of the games correctly, boasting records of 30-9 or better. You can see why it would be tempting to wrap these leading pickers in plastic and take them to Vegas. But if you did, you’d find their streak would end in a hurry.