Comments on: Probability and Prediction Markets – What matters and what doesn’t http://blog.mercury-rac.com/2007/08/14/probability-and-prediction-markets-what-matters-and-what-doesnt/ A blog on prediction markets and innovation Mon, 22 Feb 2010 12:22:07 +0000 http://wordpress.org/?v=2.9 hourly 1 By: oyvind http://blog.mercury-rac.com/2007/08/14/probability-and-prediction-markets-what-matters-and-what-doesnt/comment-page-1/#comment-197 oyvind Fri, 17 Aug 2007 12:04:42 +0000 http://blog.mercury-rac.com/2007/08/14/probability-and-prediction-markets-what-matters-and-what-doesnt/#comment-197 Say that you are going to evaluate 100 forecasting tools, and you only have access to one event. (for example probability of nuclear attack or of finding space aliens). The 100 different scores will have a correlation with the average scores when predictions are repeated. A tool with larger score on predicting a single event, will also have a higher probability of having a large score in the long run. So in one way, it makes sense to evaluate single events. Evaluating a tool based on this is flawed, but it is not impossible. Say that you are going to evaluate 100 forecasting tools, and you only have access to one event. (for example probability of nuclear attack or of finding space aliens). The 100 different scores will have a correlation with the average scores when predictions are repeated. A tool with larger score on predicting a single event, will also have a higher probability of having a large score in the long run.

So in one way, it makes sense to evaluate single events. Evaluating a tool based on this is flawed, but it is not impossible.

]]> By: Jed Christiansen http://blog.mercury-rac.com/2007/08/14/probability-and-prediction-markets-what-matters-and-what-doesnt/comment-page-1/#comment-196 Jed Christiansen Fri, 17 Aug 2007 10:09:53 +0000 http://blog.mercury-rac.com/2007/08/14/probability-and-prediction-markets-what-matters-and-what-doesnt/#comment-196 Hello, Oyvind, and thanks for commenting. Scoring rules still are really only appropriate for repeated decisions, and using them on a one-off forecast is still problematic to me. I believe that there is simply no way of assessing the quality of a single (binary) forecast. Hello, Oyvind, and thanks for commenting.

Scoring rules still are really only appropriate for repeated decisions, and using them on a one-off forecast is still problematic to me.

I believe that there is simply no way of assessing the quality of a single (binary) forecast.

]]> By: oyvind http://blog.mercury-rac.com/2007/08/14/probability-and-prediction-markets-what-matters-and-what-doesnt/comment-page-1/#comment-195 oyvind Fri, 17 Aug 2007 09:41:19 +0000 http://blog.mercury-rac.com/2007/08/14/probability-and-prediction-markets-what-matters-and-what-doesnt/#comment-195 Yeah, but you can use a scoring rule, like Good's scoring rule. ln(r / 0.5) if the WTA turns out to happen, ln(1 - r / 0.5) if not. A prediction of, say 0.20 is bad according to the scoring rule. Of course, maybe the PM will get a good score in the long run. But if we are speaking single events, then Chris Masse has a point. If another forecasting tool turns out to get a better score, and we only have access to one prediction, then the other forecasting tool is the best. Yeah, but you can use a scoring rule, like Good’s scoring rule. ln(r / 0.5) if the WTA turns out to happen, ln(1 – r / 0.5) if not. A prediction of, say 0.20 is bad according to the scoring rule.

Of course, maybe the PM will get a good score in the long run. But if we are speaking single events, then Chris Masse has a point. If another forecasting tool turns out to get a better score, and we only have access to one prediction, then the other forecasting tool is the best.

]]> By: Midas Oracle .ORG » Blog Archive » Jed Christiansen strongly believes that Chris Masse has a bad understanding of probabilities. http://blog.mercury-rac.com/2007/08/14/probability-and-prediction-markets-what-matters-and-what-doesnt/comment-page-1/#comment-193 Midas Oracle .ORG » Blog Archive » Jed Christiansen strongly believes that Chris Masse has a bad understanding of probabilities. Tue, 14 Aug 2007 11:28:04 +0000 http://blog.mercury-rac.com/2007/08/14/probability-and-prediction-markets-what-matters-and-what-doesnt/#comment-193 [...] Christiansen strongly believes that Chris Masse has a bad understanding of probabilities. And he could be right. The only way to evaluate accuracy of predictions is with a sufficient group or series of [...] [...] Christiansen strongly believes that Chris Masse has a bad understanding of probabilities. And he could be right. The only way to evaluate accuracy of predictions is with a sufficient group or series of [...]

]]>