Biomedical science is suffering a replication crisis, suggesting that a significant proportion of research resources are wasted and that many results are unreliable. However, this can be countered effectively with simple actions, such as properly accounting for the known risks of bias in in vivo research. Although few studies report the measures taken to reduce these risks, doing so is a stated requirement for many research stakeholders (such as journals, funding bodies and universities) who wish to increase the standards of their research.
A text-mining tool has been developed which can identify the reporting of measures such as randomisation, sample size calculations, and blinded assessment of outcomes, in research publications. Aggregating this information for a large number of publications allows research output to be monitored, and comparisons – such as changes in practice over time, or between different journals – to be made. The technology is a new method to drive improvement in the standards of biomedical research by providing a rigorous metric for the quality of research publications.
The text mining method has been validated by several complementary studies (Macleod et al., 2015), some with samples of more than 2500 publications. Comparisons of the reported risks of bias are made between different time periods, journals of different impact factor, and universities. More recent work has shown the method to have sensitivities of 100%, and specificities >80%, for each of three different reported measures.
Allows research stakeholders to track whether their publications meet required standards.
Can guide future policies for further improvements in quality.
Proof of concept
M. R. Macleod et al. (2015) PLoS Biol. 13(10) e1002273