Table 1 Characteristics of the 4 evaluations
Edwards 2002 | Doust 2005 | Pham 2016 | Shemilt 2016 | |
|---|---|---|---|---|
Aim of evaluation | Estimate the accuracy and reliability of reviewers when screening records for relevant trials for a systematic review | Assess the reliability and accuracy of reviewers’ screening | Assess the implications of applying methodological shortcuts: one of the shortcuts is a single screening approach | Compare the costs and effects of a single screening approach. |
Other aims/comparisons included in the evaluation | – | The other aim was to assess the sensitivity and precision of five published search strategies | The other shortcuts were: - one bibliographic database plus ancillary sources - limiting the search to bibliographic databases - only papers available electronically | Four variant screening approaches were analysed. The other approaches were: - safety first screening - double screening - single screening with text mining |
Number of - reviews examined | - 1 | - 2 | - 3 | - 1 |
- reviewers involved | - 4 | - 2 | - 2 | - 1 |
- sets of screenings examined | - 6* | - 2 | - 3 | - 1 |
- individual screenings analysed | - 12 | - 4 | - 6 | - 1 |
Screening step assessed | Title/abstract screening only | Title/abstract screening only | Title/abstract screening only | Title/abstract screening only |
Piloting screening | Prior meeting to discuss inclusion criteria | No information provided | Pre-test screening of 50 potentially relevant records | No information provided |
Gold standard | Studies identified as relevant by a double screening approach.Disgreements were resolved by consensus | Studies identified as relevant by a double screening approach. Disagreements were resolved by consensus. | Original reviews’ study pool. No further information provided. | Studies identified as relevant by a double screening approach. Disagreements were resolved by consensus. |
Number of hits needed to be screened | 22,571 hits, (each reviewer was to screen approximately 11,286 records) | Tympanometry: 638 hits Natriuretic peptides: 373 hits | Wilhelm 2011: 1890 hits Greig 2012: 3091 hits Bucher 2015: 690 hits | 12,477 hits |
Study type(s) included in the review | RCTs | Diagnostic test accuracy studies | Wilhelm 2011: n/a; no limitations Greig 2012: limited to experimental (control and challenge trials), quasi-experimental (before-and-after trials), cohort designs Bucher 2015: no limitations on study designs | Studies of any design, using quantitative and/or qualitative methods |
Reviewer experience as reported in the evaluation | “Each reviewer had substantial experience of screening records for systematic reviews, apart from reviewer 2 who was relatively inexperienced.” | “The first reviewer had more content knowledge and more experience in completing systematic reviews” [...] | “Reviewer A was a veterinarian, had a master’s degree in epidemiology and had over 5 years of experience in relevance screening for reviews in agri-food public health. Reviewer B had a master’s degree in public health and over 2 years of experience in relevance screening for reviews in agri-food public health”. | […] “conducted by an experienced team of systematic reviewers with substantial experience in primary care and medical education” |
Reviewer experience: classification | Reviewer 1: experienced Reviewer 2: less experienced Reviewer 3: experienced Reviewer 4: experienced | Reviewer 1: experienced Reviewer 2: less experienced | Reviewer 1: experienced Reviewer 2: less experienced | Reviewer 1: experienced |
Re-analysis meta-analysis | No | No | Yes | Yes |
Number of missed records presented | Yes | Yes | Yes | Yes |