Prediction Market Trials
byA recent Mercury Research blog listed three key points to consider when testing out internal prediction markets:
- solving a real problem
- starting with a small, diverse group
- keeping it low-profile
All three are very important to a successful prediction market trial, but I would add one more: a way to measure how well they performed.
When selecting your trial business problems, be sure that you choose ones that
- Have a quantifiable “correct” answer
- Have an existing predictor “control group”
Without these strong measurement aspects, it is easy to dismiss the results of the prediction market as “we probably could have done as well using process X instead”. It is only by leveraging your control group and your quantifiable answer that you can demonstrate their value. Remember: prediction markets are still a new and strange idea to many people, and having a demonstrated value will allow you to move them out of a trial phase and increase their adoption company-wide more successfully.
Found in
• Permalink
http://www.sundog.net/index.php/sunblog/entry/prediction-market-trials/
Comments
Hello, Nathan. Thanks for commenting on my blog post.
I wanted to clarify a couple of points here.
Regarding having a quantifiable “correct” answer: That is a fundamental need in order to run a prediction market in the first place. In order to close out a futures market (a prediction market is essentially a futures market on an event) the outcome must be a certifiable, discrete answer. There is no answer to a question like “Which brand will be better in 2009: Google or Yahoo” since there is no certifiable answer to “better.” If you’re running a prediction market, you will have a correct answer.
When you mention “Have an existing predictor ‘control group’” I think it’s more complicated. I’ve never heard of an existing forecasting method being dropped completely for a prediction market without any sort of comparison. It’s fairly obvious that you’d want to do that, though for some of my clients the difference is so clear that the original forecasting method is dropped almost immediately.
Sometimes, however, a prediction market develops a prediction for something that has never been forecasted before, for whatever reason. In this case, the calculation is more difficult since there is no easy computation about the value. However, what value is that information to decision-makers? Each instance is going to be different, though I doubt a prediction market will ever be run on a variable that’s not important to a business.
Thanks again for your reply to my post!
Hi Jed, I’m glad you found my post. You make a good point about a quantifiable answer being a requirement of a prediction market. The goal of my post was to build on your post with some additional pointers for people who are new to the idea of prediction markets, and who are looking to trying to introduce them into their organization. I highlighted the quantifiable answer mainly to keep them from trying the “Which brand will be better?” question and blaming the failure on the market.
As far as the control group requirement, the point I was trying to get at was the value they serve when first evaluating and presenting a prediction market. For many organizations it is a strange concept that will run into opposition. Having a direct comparison between the predictive power of the old system and the market is the best defense against this opposition.
A good example is listed in my Predicting Potter and Consoles post. The game console market would make a great trail because you can say “the market out-predicted the expert 3 out of 5 times”.
If the expert’s prediction (existing predictor control group) wasn’t available, it would be easy for the decision makers to say “The market’s prediction of Wii sales was off by over 13%, I’m sure we can can find a more reliable predictor than THAT! Don’t bother with these prediction markets anymore.”
Next entry
"Unbelievable savings"
Previous entry
Instant Messaging Maintains its Workplace Presence
<< Back