- Open Access
Epistemonikos: a comprehensive database of systematic reviews for health decision-making
BMC Medical Research Methodology volume 20, Article number: 286 (2020)
Systematic reviews allow health decisions to be informed by the best available research evidence. However, their number is proliferating quickly, and many skills are required to identify all the relevant reviews for a specific question.
Methods and findings
We screen 10 bibliographic databases on a daily or weekly basis, to identify systematic reviews relevant for health decision-making. Using a machine-based approach developed for this project we select reviews, which are then validated by a network of more than 1000 collaborators. After screening over 1,400,000 records we have identified more than 300,000 systematic reviews, which are now stored in a single place and accessible through an easy-to-use search engine. This makes Epistemonikos the largest database of its kind.
Using a systematic approach, recruiting a broad network of collaborators and implementing automated methods, we developed a one-stop shop for systematic reviews relevant for health decision making.
A landmark study in 2010 estimated that 11 systematic reviews were published each day. Other researchers have reported an exponential growth of epidemic proportions afterwards. Our estimate is that 104 systematic reviews relevant for health decision-making are currently published each day.
The growth of systematic reviews and the skills needed to retrieve them from across multiple databases make it almost impossible for the scientific community, health care providers and policymakers to keep up.
Using a systematic approach, which includes a broad network of collaborators and the use of automated methods, we developed Epistemonikos, an easy-to-use, one-stop shop for systematic reviews relevant for health decision-making.
A wide consensus has been reached about the need for making health decisions informed by the best available research evidence. Such decisions help to assure quality and efficiency and maximise the benefits while minimising harms and costs .
For most health questions there is already a substantial number of studies that can inform decisions. In fact, in several areas the information exceeds what clinicians or policymakers trying to keep up with the evidence can handle . Systematic reviews and meta-analyses (henceforth referred to as systematic reviews) were invented to deal with the problem of having several studies answering a similar question. A well-conducted systematic review is considered the best available evidence according to Evidence-Based Health Care methodology .
Because systematic reviews are highly valued by the scientific community and health care providers, their number has increased. Other incentives, such as systematic reviews being relatively easy to perform and publish in high impact journals  and becoming a marketing tool  have also fueled their proliferation. A landmark study estimated their production at 11 per day in 2010 , and others have reported an exponential growth afterwards .
An easy-to-use, comprehensive database for systematic reviews would allow the scientific community, health care providers and policymakers to find and use the best available research evidence efficiently in the limited time they have to make a decision. The Database of Abstracts of Reviews of Effects (DARE), maintained by the Centre for Reviews and Dissemination, University of York, UK, intended to play this role, but it was discontinued in March 2015 .
Epistemonikos is a collaborative project started in 2009 with the objective of gathering, organising and comparing all of the relevant research evidence for health decision-making in a single database . This article describes the methods designed and the results obtained during the first stage of the project which consists in identifying all of the relevant systematic reviews.
Construction and content
Criteria for considering systematic reviews for Epistemonikos Database
In accordance with the Cochrane Collaboration and the PRISMA Statement [11, 12], we have adopted the following definition: ‘A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria to answer a specific research question. It uses explicit systematic methods that are selected with a view to minimising bias, thus providing reliable findings from which conclusions can be drawn and decisions made’ .
The operational criteria to consider a systematic review for inclusion in Epistemonikos Database are:
Its main purpose is to synthesise primary studies.
It states at least one explicit eligibility criterion.
It reports searching in at least one electronic database.
Additionally, we include any synthesis of primary studies that do not fulfil the above definition but is judged to add valuable information, such as individual patient or unpublished data meta-analysis where studies have not been identified through a systematic search process.
Evidence-syntheses that fulfil criteria 3 but not all of the above are not excluded from Epistemonikos Database but are classified under a different category (i.e. broad synthesis plus a specific subtype, such as guideline, overview of systematic reviews), which is not the subject of this article.
We exclude reviews that:
Do not address a human health problem.
Synthesise studies that do not evaluate individuals or groups of individuals (e.g. preclinical or animal studies).
Explore a methodological issue (i.e. research about research).
Are only presented as conference abstracts.
Search methods for identification of systematic reviews
Epistemonikos was developed and is maintained by systematically searching 10 databases in a daily or weekly basis: Cochrane Database of Systematic Reviews, Pubmed/MEDLINE, EMBASE, CINAHL, PsycINFO, LILACS, DARE, Campbell library, JBI Database of Systematic Reviews and Implementation Reports and EPPI-Centre Evidence Library.
We do not restrict our search by language, publication status or publication date (i.e. databases have been searched from inception). In the case of databases of structured summaries (i.e. DARE database), we retrieve the article being summarised and assess it using the same inclusion criteria.
The search strategies were pragmatically adapted from previously reported strategies to retrieve systematic reviews  and improved by a team of search experts who analysed the search terms obtained from the text mining of relevant and irrelevant records.
The detailed search strategies currently used in Epistemonikos Database are described in additional file 1.
In order to identify systematic reviews potentially missed by our search in electronic databases we:
Include systematic reviews identified in overviews of reviews, guidelines, scoping reviews or other types of broad syntheses (which are also included in Epistemonikos Database but are classified under a different category).
Check references of selected included reviews.
Run cross-citation searches in Google Scholar and Microsoft Academic.
Evaluate potentially eligible reviews sent by users through the contact page or other means (e.g. email, twitter).
Data collection and analysis
Selection of reviews
The selection is conducted in two steps. First, all potentially eligible articles are classified as they enter the database using automated methods specifically created for this project (a machine learning classifier for the records with an abstract and a heuristic classifier for the records without an abstract). Secondly, a collaborative network of Epistemonikos users validates this classification. Records with a high probability of being false positives or false negatives are regularly checked by a dedicated team of method experts.
Development of the classifier for records with an abstract
The dataset used to develop the classifier includes all the records with an abstract that had been manually screened by at least one reviewer by January 2019. This dataset was formed by 102,011 systematic reviews and 42,321 records not corresponding to systematic reviews, most of them classified before 2016 when Epistemonikos Database selection process was conducted only by human screeners (earlier versions of the classifier had been in use during 2017 and 2018).
The dataset was arbitrarily divided into two splits as training and validation (80 and 20% respectively). The training split was used to build a classifier using a supervised learning random forest and the validation split was used to test its predictive power . The terms composing the classifier were iteratively analysed and improved by a team of software engineers with expertise in information retrieval, methodologists and information specialists until reaching a stable version. Finally, the results were manually validated with a set of 500 unseen records to make sure we had not overfit the model during the tuning of the random forest model.
Development of the classifier for records without an abstract
Acknowledging the limitations of any classifier using a language-based technique to manage records without an abstract, we approached these as a separate problem. We reviewed the sample iteratively to identify characteristics associated with a high probability of being or not being a systematic review and custom-built a heuristic classifier using specific terms and other characteristics of the records.
Automated classification (classifiers)
All the records retrieved by the search strategy are immediately processed and automatically classified into included/excluded systematic reviews in the database. Later, the classifications are manually validated by at least one human screener.
All the titles and abstracts included by the classifier are uploaded to Collaboratron™ , a screening software specifically developed by Epistemonikos Foundation for this purpose. The documents are screened by at least one human using this tool, starting from the most recent records. The records without an abstract are regularly reviewed by a dedicated team. The full text of the article is retrieved if it is not possible to make a decision based on the title or abstract.
Discrepancies between the classifier and a human screener (i.e. included by the classifier and excluded by the human screener) or between different human screeners are resolved by a senior researcher.
Measures of performance
In order to estimate the performance of the classifiers, we used the validation set as a gold standard for the machine learning classifier and a convenience sample of 500 unseen records for the heuristic classifier. We calculated the following measures (and their 95% confidence interval): sensitivity or recall (true positives/(true positives + false negatives)), precision or positive predictive value (true positives/(true positives + false positives)), specificity (true negatives/(false positives + true negatives)) and accuracy ((true positives + true negatives)/total). For estimation of misclassified reviews in Epistemonikos Database, we applied these numbers to the total amount of records without human validation.
Utility and discussion
Results of the search
On January 24, 2020, the literature search had retrieved 1,431,972 records, which after removing duplicates correspond to 704,150 potentially eligible systematic reviews (626,121 with and 78,029 without an abstract). The total number of included systematic reviews in Epistemonikos is 307,119 (104,050 already validated by at least one human). We have identified 13,369 reviews from sources different than the search strategy in electronic databases.
A flow diagram summarizing the screening and selection process is presented in Fig. 1. A living report (updated daily) can be consulted at https://www.epistemonikos.org/about_us/updated_report
The number of systematic reviews has increased at a rate of 18.2% per year since 1990. In the last decade (January 1st, 2010 to December 31st, 2019), it went from 11,233 per year to 37,944, that is a 3.5-fold increase. On average, 104 systematic reviews are published each day.
The number of systematic reviews per year is shown in Fig. 2.
Performance of the classifiers
The sensitivity of the machine learning classifier for the detection of systematic reviews in records with an abstract was 96.8% (95% CI, 96.58 to 97.06%), the specificity was 80.4% (95% CI, 79.55 to 81.23%), and the accuracy was 92.0% (95% CI, 91.65 to 92.28%). The sensitivity of the heuristic classifier for the detection of systematic reviews in records without an abstract was 94.1% (95% CI, 88.74 to 96.97%); the specificity was 55.6 (95% CI, 50.49 to 60.63%); and the accuracy was 66.0% (95% CI, 61.74 to 70.02%).
Applying these figures to the records that have only been evaluated by the classifiers, the number of actual systematic reviews missing in Epistemonikos Database (false negatives) can be estimated at 9122 (3.0%), and the number of records classified erroneously as systematic reviews (false positives) can be estimated at 44,242 (14.4%).
The complete performance of the classifiers is presented in Table 1.
Estimation of the number of systematic reviews
Our project confirms the dramatic increase in the number of systematic reviews that has been reported by other researchers [2, 4, 5]. A well-known study  using data from 2007 estimated the average number of published systematic reviews at 11 per day. Our estimation is slightly superior during the same period (17.6 reviews per day), which might be partially explained by the rate of false positives in Epistemonikos or by an underestimation of the numbers in the former. Moreover, the difference might reflect the larger number of search sources and improved strategies developed by Epistemonikos. More recent studies have reached to lower  or higher  estimations, probably because of the variation in the methods employed. For instance, some authors have screened a sample of potentially eligible reviews in PubMed (i.e. records published during a one-month period)  or simply counted PubMed records tagged as systematic review or meta-analysis . The use of different definitions of what is considered a systematic review might also explain the differences.
Comparison against other databases/repositories of systematic reviews
In comparison with other searchable and freely available repositories of systematic reviews, Epistemonikos Database has multiple advantages. In terms of comprehensiveness, it contains substantially more systematic reviews than any other repository. For instance, DARE  included about 45,000 reviews when it was discontinued in 2015, TripDatabase contains about 65,000 reviews , and there were over 40,000 systematic reviews in PubMed Health when it was discontinued in 2018 . In 2019 PubMed announced the addition of a new publication type MeSH term for systematic reviews , which retrieves about 110,000 citations as of January 2020.
Another advantage of our approach is the transparency in the reporting of methods and results. A systematic approach and a clear report make it possible to be certain on the contents of the database and to be aware of its limitations. In terms of reporting, we have adapted standards for systematic reviews , and we have developed the technology to apply those standards to a massive amount of records. Our PRISMA flowchart, for example, is updated daily, even though it is one of the largest, if not the largest project to be summarised in this format.
Probably the more important question, for both researchers and decision-makers, is if they can rely on Epistemonikos Database and not conduct their own searches elsewhere. One study comparing searches for systematic reviews in 7 key databases, including Epistemonikos, concluded that no single database was able to retrieve all of the reviews used as a reference set . Epistemonikos was selected as part of a combination of 3 databases which, combined, retrieved the highest number of unique records. However, it is important to note that this comparison was made when Epistemonikos was at an early stage, before the current methods and technologies were fully deployed, and included only 60,000 reviews (currently, there are 198,000 records in the database that would have been available at the time of that comparison). So, if a similar study were conducted nowadays, it is reasonable to expect a substantially better recall.
Limitations of this article
The main limitation of this article is that the performance of Epistemonikos has not been estimated using a proper gold standard, such as the one used in previous studies [8, 13]. Recent studies by our research group are addressing this limitation and have preliminary shown Epistemonikos includes most of the reviews that would be retrieved when using an exhaustive approach as gold standard [20, 21]. We think it is reasonable to affirm Epistemonikos constitutes a one-stop shop for systematic reviews from the perspective of most users but more studies are needed in order to establish if it can be used as a unique source for reviews in more rigorous contexts, such as the conduction of guidelines or overviews of systematic reviews.
Limitations of Epistemonikos Database
The proliferation of systematic reviews has been described as one of epidemic proportions  and as a mega-silliness , and portrayed as a massive production of unnecessary, misleading, and conflicted evidence [5, 22]. Nevertheless, systematic reviews are recognised as an invaluable tool to make health decisions , as a key source of information in policymaking , as the staple of practice guidelines , as the main input for textbooks and point-of-care tools supporting health professionals and students , and as the most effective way to prioritise research needs and to reduce waste in evidence production .
In this context, a one-stop shop of systematic reviews is a conditio sine qua non for the adoption of an evidence-based decision-making model in real-life practice and policy making. However, most of the existing systematic reviews have major limitations in terms of currency and quality [5, 23, 27, 28], so still, a substantial effort by users to separate the wheat from the chaff is required. This problem is not yet addressed by Epistemonikos Database.
In the current scenario, users will often need to critically appraise dozens of overlapping reviews to answer a single question , most of them of poor quality  or out of date [27, 28]. This represents an insurmountable barrier to the adoption of evidence-based decision-making.
Furthermore, even with this massive production of systematic reviews there is a large proportion of primary studies that are not yet covered by systematic reviews, which are not routinely collected by Epistemonikos . The lack of coordination in the review production has led researchers to cover much the same territory.
The present state of a wide acceptance of systematic reviews amidst important challenges to their development has been portrayed as a ‘midlife crisis’ . The transition to a mature field requires that we expand the existing efforts to improve the quality and reporting of reviews , establish initiatives to discourage the conduct of redundant reviews , and promote the adoption of better and more sustainable review models .
Availability of data and materials
The datasets used and analysed during the current study or datasets needed to reproduce the results of this study (e.g. a list of included/excluded records) are available from the corresponding author on reasonable request.
Sackett D, Straus S, Richardson W, Rosenberg W, Haynes R. Evidence-based medicine: how to teach and practice EBM. Edinburgh: Churchill Livingstone; 2000..
Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326..
Oxford Centre for Evidence-based Medicine – Levels of Evidence (March 2009). [cited 2 April 2018] Available from: http://www.cebm.net/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/.
Tebala GD. What is the future of biomedical research? Med Hypotheses. 2015;85(4):488–90. https://doi.org/10.1016/j.mehy.2015.07.003.
Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514. https://doi.org/10.1111/1468-0009.12210.
Ely J, Osheroff J, Ebell M, Chambliss M, Vinson D, Stevermer J, et al. Obstacles to answering doctors’ questions about patient care with evidence: qualitative study. BMJ. 2002;324:710.
Shariff SZ, Bejaimal SA, Sontrop JM, Iansavichus AV, Haynes RB, Weir MAGA. Retrieving clinical evidence: a comparison of PubMed and Google scholar for quick clinical searches. J Med Internet Res. 2013;15:e164.
Rathbone J, Carter M, Hoffmann T, Glasziou P. A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. Syst Rev. 2016;5:27. https://0-doi-org.brum.beds.ac.uk/10.1186/s13643-016-0197-5.
Petticrew M, Song F, Wilson P, Wright K. Quality-assessed reviews of health care interventions and the database of abstracts of reviews of effectiveness (DARE). Int J Technol Assess Health Care. 1999;15(4):671–8.
Rada G, Perez D, Capurro D. Epistemonikos: a free, relational, collaborative, multilingual database of health evidence. Stud Health Technol Inform. 2013;192:486–90.
Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. https://0-doi-org.brum.beds.ac.uk/10.1371/journal.pmed.1000097.
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from: www.training.cochrane.org/handbook.
Montori VM, Wilczynski NL, Morgan D, Haynes RB, Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005;330(7482):68.
Breiman L. Random forests. Mach Learn. 2001;45(1):5–32.
Epistemonikos Foundation. Collaboratron™[Software]. https://collaboratron.epistelab.com.
Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13(5):e1002028. https://0-doi-org.brum.beds.ac.uk/10.1371/journal.pmed.1002028.
TRIP database. [accessed 27 Jan, 2020] Available from: https://www.tripdatabase.com/.
PubMed Health to be discontinued October 31, 2018; content will continue to be available at NLM. [accessed 27 January, 2020] Available from: https://ncbiinsights.ncbi.nlm.nih.gov/2018/07/31/pubmed-health-discontinued-october-2018-nlm/.
National Library of Medicine. Support for systematic reviews. NLM Tech Bull. 2019;(427):b6.
Bravo-Soto GA, Schulze CE, Morel-Marambio M, Lobos-Urbina D, Vergara C, Verdugo-Paiva F, et al. How comprehensive is Epistemonikos Database to identify systematic reviews in health: a methodological study. Santiago: Cochrane 26th Cochrane Colloquium (virtual); 2019.
Araya V, Kraemer P, Burdiles P, Herrera P, Castillo C, Sepulveda D, et al. The Living Overview of Evidence database (LOVE) may be more efficient than a traditional search of systematic reviews and randomized trials. Santiago: Cochrane 26th Cochrane Colloquium (virtual); 2019.
Page MJ, Moher D. Mass production of systematic reviews and meta-analyses: an exercise in mega-silliness? Milbank Q. 2016;94(3):515–9. https://0-doi-org.brum.beds.ac.uk/10.1111/1468-0009.12211.
Lavis JN, Oxman AD, Grimshaw J, Johansen M, Boyko JA, Lewin S, et al. SUPPORT Tools for evidence-informed health Policymaking (STP) 7: Finding systematic reviews. Health Res Policy Syst. 2009;7(Suppl 1):S7. https://0-doi-org.brum.beds.ac.uk/10.1186/1478-4505-7-S1-S7.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jclinepi.2010.04.026.
Jeffery R, Navarro T, Lokker C, Haynes RB, Wilczynski NL, Farjou G. How current are leading evidence-based medical textbooks? An analytic survey of four online textbooks. J Med Internet Res. 2012;14(6):e175. https://0-doi-org.brum.beds.ac.uk/10.2196/jmir.2105.
Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65. https://doi.org/10.1016/S0140-6736(13)62229-1.
Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147(4):224–33.
Beller EM, Chen JK, Wang UL, Glasziou PP. Are systematic reviews up-to-date at the time of publication? Syst Rev. 2013;2:36.
Siontis KC, Hernandez-Boussard T, Ioannidis JP. Overlapping meta-analyses on the same topic: survey of published studies. BMJ. 2013;347:f4501. https://0-doi-org.brum.beds.ac.uk/10.1136/bmj.f4501.
Iqbal SA, Wallach JD, Khoury MJ, Schully SD, Ioannidis JP. Reproducible research practices and transparency across the biomedical literature. PLoS Biol. 2016;14(1):e1002333. https://doi.org/10.1371/journal.pbio.1002333.
Gurevitch J, Koricheva J, Nakagawa S, Stewart G. Meta-analysis and the science of research synthesis. Nature. 2018;555(7695):175–82. https://0-doi-org.brum.beds.ac.uk/10.1038/nature25753.
Tugwell P, Welch VA, Karunananthan S, Maxwell LJ, Akl EA, Avey MT et al. When to replicate systematic reviews of interventions: consensus checklist. BMJ. 2020;370:m2864.
Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JP, Mavergames C, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2):e1001603. https://0-doi-org.brum.beds.ac.uk/10.1371/journal.pmed.1001603.
We thank Jaime Cerda and Jorge Dagnino who suggested the name Epistemonikos after long and stimulating conversations. We also thank Simon Lewin, Vivian Welch, Andy Oxman, Jordi Pardo, Sarah Rosenbaum, Gordon Guyatt, Iain Chalmers, Ludovic Reveiz and Marit Johansen, who provided valuable feedback, encouragement and support, especially in the first years of the project. This project would not have been possible without the support of all the organisations that have provided funding, a network of volunteers who have contributed with invaluable work, and the countless people in Chile and Ibero-America that have embraced the project as their own and have acted as uninterested ambassadors.
The authors received no financial support for the research, authorship, and/or publication of this article.
Ethics approval and consent to participate
This article does not report on or involve the use of any animal or human data or tissue, so this section is not applicable. Details about the ‘Terms and conditions’ of Epistemonikos Database can be accessed at Epistemonikos Website (https://www.epistemonikos.org/en/terms_and_conditions).
Consent for publication
Epistemonikos Database was developed and is maintained by Epistemonikos Foundation, a non-profit organisation with a strict policy to avoid financial conflict of interests. Details about this policy and financial support can be found on the website of the Foundation (http://www.epistemonikos.cl). The salary of CA, FV2, VC2, MP, JV, CV is totally or partially paid by Epistemonikos Foundation.
All authors, as founders, board members, developers or contributors, have some degree of academic conflict of interests with this article.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Rada, G., Pérez, D., Araya-Quintanilla, F. et al. Epistemonikos: a comprehensive database of systematic reviews for health decision-making. BMC Med Res Methodol 20, 286 (2020). https://0-doi-org.brum.beds.ac.uk/10.1186/s12874-020-01157-x
- Bibliographic database
- Systematic reviews
- Evidence-based practice