Five ways to fix statistics
Nature recently commissioned a comment piece in which they asked five influential statisticians to recommend one change to improve science. The contributors agree that the problem does not lie with maths but with ourselves. More accurately, it is in the interaction between statistics and human researchers/audiences that we discover the roots of the reproducibility crisis in science. Here are the main points from each of the contributors:
- Jeff Leek: Adjust for human cognition. We should spend more time studying how people collect, manipulate, analyze, communicate, and consume data - and then use those findings to prevent cognitive mistakes.
- Blakeley B. McShane & Andrew Gelman: Abandon statistical significance. P-values should be considered as just one piece of evidence among many, along with prior knowledge, plausibility of mechanism, study design, data quality, real-world costs and benefits, and so on.
- David Colquhoun: State false-positive risk too. Another way to put p-values in their place is do a better job of reporting risks of finding false positives.
- Michèle B. Nuijten: Share analysis plans and results. “Better than rules about how to analyse data are conventions that keep researchers accountable for their analyses.”
- Steven N. Goodman: Change norms from within. There is no straightforward way to address the reproducibility crisis other than instigating a broad process of norm change, bringing leaders, funders, and journals on board.
What I take away from this piece is that even the best statisticians are now openly addressing the reflexivity of human subjects and advocating for mixed methods and open science. Replication and reproducibility aren’t discussed much in the social sciences, probably because experiments feature much less centrally in the methodological toolbox for most of us. But openness and transparency when it comes to sharing data, analytical steps, and results ought to be the goal for any type of study. The Open Science Framework might be the way forward here.