Proposal: A System for Disclosing and Answering Concerns About Potential Flaws in Published Research
January 8, 2016
Transparency measures for empirical social science research (e.g., replication data and pre-analysis plans) make mistakes and misconduct easier to discover, but do not announce them on their own; this requires third-party researchers to investigate and articulate concerns about mistakes or misconduct in others' research. Currently, there are significant disincentives for researchers to do so. This is especially the case when concerns about others' research have a `personal' valence, such as concerns that another researcher has misrepresented results, analyzed data incorrectly, deviated from a pre-analysis plan, p-hacked, or fabricated data. Articulating concerns like these often involves great effort and can introduce negative personal consequences for critics and potentially innocent researchers. These costs dissuade many potential critics from articulating concerns about published research. That potential critics can be expected to remain silent reduces the disincentives for dishonest conduct and the accuracy of the scientific record. It also does not give innocent authors the opportunity to respond to concerns about their research.
This note proposes a web platform for disclosing concerns about published research and allowing authors to respond. The key features of the proposed system are revokable anonymity, revokable confidentiality, and an information escrow for communications between author and critic (Ayers and Unkovic 2012). Critic anonymity allows for critics to express their concerns without fear of reprisal. Confidentiality allows innocent accused authors to answer concerns without having their reputation unjustly tarnished before they have prepared a response. Both protections are revokable together by the critic at any time, while only confidentiality and not critic anonymity is revokable by the author at any time. The existence of a potentially public verified record of the confidential exchange is expected to facilitate appropriate behavior during the confidential period. Minimal additional structure is proposed, deferring to the judgment of critics and authors about what steps will be viewed as appropriate within their fields and what information must be provided during a communication on the platform. The role of the technology is providing 1) a third-party information escrow and 2) communication platform, 3) with the public disclosure features previously described.
David Broockman is an Assistant Professor of Political Economy at the Stanford GSB. David received his PhD in political science at UC Berkeley in 2015. His research focuses on public opinion and how politicians represent it, and has appeared in the American Journal of Political Science, Political Analysis, and other journals.