Disinterested Evaluators

I’ve just read a fascinating article about a proposed open source meritocratic search engine by Bennett Haselton on Slashdot . Haselton proposes that if you had a random selection of voters vote on the quality of a given site within a given search term, you could create a meritocratic search engine that strongly resists gaming. This is basically the Holy Grail for all content producers and really for a large swath of the economy if Haselton’s analysis stands. In theory, I totally agree.

The issue is this, how do you gather up and motivate disinterested evaluators? This is a problem that exists across so many world systems: police forces, juries, democracy. If we can’t get more than 50% turnout for the U.S. presidential election, how can we get folks to vote on search results? Haselton’s research needs to be coupled with a new plan for truly global ROI analysis to work. If you work in the digital marketplace, and the tasks you complete add value to that marketplace, barring some privacy-based lack of access to information, you should be able to calculate the return on your investment for actions within the digital marketplace, exactly.

In so doing you should be able to dramatically tighten the scope of almost all content creation online. Only items from an entirely new genre or format or concept would really have unknown, or at least unpredictable performance. As a business owner, predictability is highly valuable. Sure, every segment’s going to have a standard deviation and some areas will have wider variability than others. However, in a market currently ruled almost exclusively by the value judgments of individual, experienced (read expensive) evaluators, study of this sort could really level the playing field. Creative Science? Will this serve to homogenize and dull the creative vibrancy found online? Perhaps; the more something is studied, known, the more boring it gets really. Sadly, there are lots of really boring companies that happen to be quite profitable.

This profitability, gained from predictability, should also benefit the evaluators. Paying voters is the only thing that makes sense to me, but orchestrating micro-payments on the kind of scale required sounds somewhat nightmarish. Money seems to be the most reasonable way to motivate disinterested evaluators, but perhaps there are other models.

In grade school, my teachers marked “evaluation” as the highest form of human thought. Sounds like pretty heinous drudgery to me. As hard as we try to get away from the factory, no matter how white our collars get, the steam whistle never seems that far off, does it?