Solving the attractiveness of incredible results with prediction markets
The idea that science is in crisis is one that has been building now for at least two decades. Evidence for this crisis revolves around studies that provide evidence of publication bias and especially the lack of repeatability of high profile studies. The findings appear bemusing because all studies are subjected to peer-review before being published and therefore go through some kind of quality check. Certainly this would suggest that it would not be easy to pick which studies are replicable and which are not. Yet this does not appear to be the case.
A study that gave subjects the possibility of predicting which studies were replicable and which were not found that it was possible to predict an advance of any replication studies being made. Moreover they found that once replication studies were made they followed the prediction market (Dreber et al 2015). Thus, this suggests that individuals in peer review are not particularly good at determining whether or not the study is replicable, but that a prediction market is.
Incredible results
Humans have a bias toward wanting to believe significant results (Tivers 2011), even when the potential for these to be the result of aType I error is quite high. Perhaps the positive feedback gained from incredible results by the media (traditional and social) gives the impetus to drive selection. But a new study suggests that, all else being equal (including gender bias, author seniority, etc.), these incredible results also generate more citations (Serra-Garcia & Gneezy 2021).
As we are aware, citations are a form of currency in current day science. Increased citations to journals (within 2 years of publication) give them higher Impact Factors, which in turn allow them to leverage better manuscripts and higher APCs. Increased citations to authors allow them to compete in a competitive job market, opening the door to tenure, grants and awards. We should be aware of the increasing number ofretractions associated with fraud, which has placed the perpetrators in advantageous jobs.
The research by Serra-Garcia & Gneezy (2021) is of particular note as they only selected publications from two journals,NatureandScience, meaning that this journal playing field was very similar. They used the dataset from Dreber et al (2015) allowing them to see which of the studies was actually repeatable, and the ones that weren’t being literally incredible. They found that incredible studies received more citations, even after the replication studies (see Dreber et al 2015) showed that they lacked credibility. Moreover, after the failure to replicate, only 12% of these additional citations reported their incredible nature and hence the increased citations are not generated from those that report on failure to replicate.
The recognition that incredible results are attractive to high-ranking journals and those who cite research in their own fields helps to lift the veil on the way in which today’s science has a positive feedback forchancers and crooks. We are susceptible to scientific fraud because we appear to be drawn by the incredible, presumably because the credible simply doesn’t seem exciting enough. Given that as individuals we perform poorly, can we use prediction markets to give us the edge on our inbuilt biases?
Finding a use for prediction markets
Prediction markets are simply crowdsourcing to determine the outcome of a particular event, in this case whether or not a study was replicable. However, there is a gambling twist that borrows from the stock market. For example, you might think that there is an 85% chance that the study is replicable, and this is how you enter the market. Once all participants place their predictions in, a consensus prediction is reached, and now the trading begins. If the consensus prediction is at 0.62, and you really believe that the chance is 0.85 you should buy stocks valued at 0.62 because if you are right you will make money. However, if the consensus stock is 0.95, then you would be better off selling your share of 0.85 if you really believe that there’s a 0.15 chance the replication will fail. Like the real market, there is no reason for this market to be static. For example, one of the authors of the original study could give a talk, and during the talk participants start buying or selling their stock as extra confidence or skepticism is gained. Likewise, during the questions an astute member of the audience may rattle the author, resulting in a fall of the ‘price’ or consensus outcome.
Potential uses of prediction markets
If individual reviewers aren’t good at spotting incredible results, perhaps this should be passed to a crowdsourcing platform to determine whether or not high profile studies should be published in high profile journals. As all of the editorial board team members have expertise in the journal’s content, perhaps they could make up the panel of experts that judge on the replicability of each issue’s content. Over time, each editorial board team members’ quality would be recorded, and over time their ‘usefulness’ might be quantitatively valued.
Prediction markets are likely to work well when complex decisions have to be taken by changing a small committee of people with limited information to a much larger group with better collective experience.
Choosing student projects to fund
Each year a limited number of people apply for bursaries to conduct projects in invasion biology at the CIB. The number of bursaries available is between a tenth and a quarter of the number of applicants. The committee (of 5 people) examines each application on a number of criteria from the application including qualities of the student, the project, the focus and past performance of the advisor. Knowledge of the panel is imperfect as they don’t know all of the information behind each application. Occasionally, phone calls during decision meetings are made to fill in blanks, but decisions are made on scoring each project with projects that have the top scores getting funded.
Enter the prediction market.Now a larger number of people can get involved - this could be the entire Core Team of the CIB, or the core team and all existing students. Some will have much better information than the original panel, and will have some impetus to trade with greater confidence. Those with less information are less likely to participate or buy less stock. Advisors who have several student applications are similarly forced to either split their stakes evenly, or potentially back a preferred application over one they consider less likely to succeed. Once the projects are funded (based on the outcome of the prediction market), their success or otherwise will be gauged by whether the degree is gained by the student within the time allotted. For students that fail to produce in time (or meet any set of milestones) will be regarded as having failed, and the payout will be for those that had stock predicting this.
Other potential uses
A similar position is faced by anyone looking at applications from stakeholders that have imperfect knowledge, but where a community of experts exists that have better collective knowledge. Outcomes of proposed projects need to have clear milestones, and on a timescale whereby the participants are still likely to be around to see the return of their knowledge investment. There are a lot of potential uses within the academic environment including:grant applications; hiring committees; etc.
Recognition of weakness
Once we have recognised where we are likely to perform badly at decision making, we should be willing to look to other solutions to improve our performance. There is clearly some distaste in the idea of a money market on making decisions, so could this instead form part of the reputation of those that take part? Would you be willing to wager your scientific reputation on the outcome of a hire, a student bursary or a grant application?
References:
Dreber, A, T Pfeiffer, J Almenberg, S Isaksson, B Wilson, Y Chen, BA Nosek, and M Johannesson. Using Prediction Markets to Estimate the Reproducibility of Scientific Research. Proceedings of the National Academy of Sciences112, no. 50 (December 15, 2015): 15343–47.https://doi.org/10.1073/pnas.1516179112.
Measey, John.How to Write a PhD in Biological Sciences: A Guide for the Uninitiated, 2021.http://www.howtowriteaphd.org/.
Serra-Garcia, M, and U Gneezy. Nonreplicable Publications Are Cited More than Replicable Ones. Science Advances7, no. 21 (May 1, 2021): eabd1705.https://doi.org/10.1126/sciadv.abd1705.
Trivers, Robert.The Folly of Fools: The Logic of Deceit and Self-Deception in Human Life. 1st edition. New York, NY: Basic Books, 2011.
Want to read more?
This blog is the basis of two books currently in press, but you can read them free online now:
