What is RobotReviewer?

RobotReviewer is a machine learning system aiming which aims to support evidence synthesis. Our demonstration website allows users to upload RCT articles and see automatically determined information concerning the trial conduct (the 'PICO', study design, and whether there is a risk of bias).

Please note: the machine learning works well, but is not a substitute for human systematic reviewers. We recommend the use of our demo as an assistant to human reviewers, who can validate the machine learning suggestions, and correct them as needed. Machine learning used this way is often described as semi-automation.

Overview

Evidence Based Medicine (EBM) aims to systematically use the best available evidence to inform medical decision making. This paradigm has revolutionized clinical practice over the past 30 years. The most important tool for EBM is the systematic review, which provides a rigorous, comprehensive and transparent synthesis of all current evidence concerning a specific clinical question. These syntheses enable decision makers to consider the entirety of the relevant published evidence.

Systematic reviews now inform everything from national health policy to bedside care. But producing these reviews requires researchers to identify the entirety of the relevant literature and then extract from this the information to be synthesized; a hugely laborious and expensive exercise. Moreover, the unprecedented growth of the biomedical literature has increased the burden on those trying to make sense of the published evidence base. Concurrently, more systematic reviews are being conducted every year to synthesize the expanding evidence base; tens of millions of dollars are spent annually conducting these reviews.

RobotReviewer aims to mitigate this issue by (semi-) automating evidence synthesis using machine learning and natural language processing.

People

RobotReviewer is led by Byron Wallace, Iain Marshall, Joël Kuiper and Frank Soboczenski. We also collaborate with Ani Nenkova and Zachary Ives at UPenn, James Thomas at the EPPI Centre, UCL, and Anna Noel-Storr at the University of Oxford and Cochrane Dementia group. We have many other collaborators and graduate students who contribute to the effort.

Acknowledgements

We are enormously grateful for the support of very many people and groups who have generously allowed us use of their data to train and evaluate our systems. In particular, we would like to express our gratitude to the Cochrane Collaboration, and especially to David Tovey and Chris Mavergames among many others who facilitated getting access to data, and made many useful introductions. We are hugely appreciative to the volunteers of the Cochrane Crowd, and to Anna Noel-Storr and Gordon Dooley, whose efforts and data we depend on to build our machine learning systems for identifying RCTs.

Support

This work is supported by: National Institutes of Health (NIH) under the National Library of Medicine, grant R01-LM012086-01A1, "Semi-Automating Data Extraction for Systematic Reviews", and by NIH grant 5UH2CA203711-02, "Crowdsourcing Mark-up of the Medical Literature to Support Evidence-Based Medicine and Develop Automated Annotation Capabilities", and the UK Medical Research Council (MRC), through its Skills Development Fellowship program, grant MR/N015185/1