Automatic bias assessment for articles in the Trip Database
We are excited to announce that the Trip Database now uses RobotReviewer to automatically identify trials that are likely to have the lowest biases in conduct and reporting. Bias, when present, leads to over- or under-estimates of true intervention effects, in turn complicating treatment choice. Bias can be introduced in several ways in the context of clinical trials (e.g., improper blinding or poor randomization); these domains have been codified in the Cochrane risk of bias tool. RobotReviewer predicts the risk of bias for each of these.
RobotReviewer's risk of bias estimates are based on the words and short phrases used in the titles and abstracts of papers. RobotReviewer has ingested thousands of such articles and identified statistical correlations between word use and manually assessed risks of bias. This general approach is known as “machine learning”.
The full RobotReviewer system (shown above) performs a risk of bias assessment using the Cochrane Risk of Bias tools. In general, RobotReviewer accepts as input the full text of clinical trial reports (as PDFs). For the version deployed within Trip, RobotReviewer performs a more limited assessment based on the title and abstract only. Moreover, in this case bias predictions are limited to the Random sequence generation, Allocation concealment, and Blinding domains.
How does RobotReviewer work?
RobotReviewer has ‘learned’ how to assess bias by examining the titles and abstracts of tens of thousands of articles describing clinical trials that are also included in the Cochrane Database of Systematic Reviews. These trials have all been manually assessed for bias by systematic review authors, using the Cochrane Risk of Bias tool. As mentioned above, this tool is used to assess various common biases in Randomized Controlled Trials, including whether a trial used robust randomization, and whether participants were adequately blinded to which intervention they received. Specifically, the version of RobotReviewer used in Trip focuses on biases in random sequence generation, allocation concealment, and blinding.
While the predictions are reasonably accurate, we caution that they are not perfect, especially because they rely only on the titles and abstracts of articles (rather than full-texts). Nonetheless, in the context of Trip searches, the predictions can help quickly identify what is likely to be the highest quality research for a given search topic. If you require very high accuracy bias assessments (for example if you are conducting a systematic review) you will of course need to consult the full text of the paper. You may wish to use the full RobotReviewer tool, which provides detailed information on all the domains of the Cochrane Risk of Bias tool from full text papers and can aid in your assessments.
For more information on the algorithm used in RobotReviewer, please see the references below.
[1] Iain J. Marshall, Joël Kuiper, and Byron C. Wallace. "RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials." Journal of the American Medical Informatics Association (2015): ocv044.
[2] Iain J. Marshall, Joël Kuiper, and Byron C. Wallace. "Automating risk of bias assessment for clinical trials." IEEE journal of biomedical and health informatics 19.4 (2015): 1406-1412.