Table 2 Judgments Involved in Systematic Reviews

Judgments pertaining to study quality assessment

• What method will be used to assess study quality?

• If a quality scale is to be used, what scoring method will apply?

• What is the threshold for excluding a study from analysis due to poor quality?

• How will the individual study quality ratings be summarized to yield a single overall rating of quality to the evidence base (High, Moderate, or Low)?

Judgment pertaining to sufficient evidence for quantitative estimate

• What is the minimum number of studies required to permit a quantitative estimate?

• What is the minimum percentage of studies reporting accurate information (i.e., calculable effect sizes) required to permit a quantitative estimate?

• What imputation methods will be used for studies that did not report sufficient information for a calculable effect size?

Judgments pertaining to initial meta-analysis

• What effect size measure will be used?

• How will heterogeneity be measured (Q or I²)?

• What is the threshold for considering an evidence base heterogeneous?

• Will the summary effect size estimate be derived from a fixed-effects or random-effects meta-analysis?

Judgments pertaining to quantitative robustness testing

• What robustness tests will be used?

• If a cumulative meta-analysis is one of the robustness tests, will it be a fixed-effects or random-effects model?

• If a cumulative meta-analysis is one of the robustness tests, in what order will studies be entered into the cumulation?

• If a cumulative meta-analysis is one of the robustness tests, how many steps (i.e., study removals) will be examined to determine robustness?

• If a cumulative meta-analysis is one of the robustness tests, what threshold for a change in the summary effect size will be used to determine robustness?

• If publication bias testing is one of the robustness tests, which method of testing for publication bias will be used?

• If confidence interval width is one of the robustness tests, how narrow must the interval be for the summary estimate to be considered robust?

• Will overall robustness be judged based on passing all of the robustness tests, or simply a majority, or what percentage?

Judgments pertaining to meta-regression

• What is the minimum number of studies required to perform meta-regression?

• Which covariates will be included in multiple regression models?

• How many covariates are permitted in any given regression model?

• Does "explaining heterogeneity" require a statistically significant covariate, or the lack of resultant heterogeneity, or both of these?

• What robustness tests will be performed for the meta-regression?

Judgments pertaining to qualitative robustness testing

• What robustness tests will be performed?

• If a cumulative meta-analysis is one of the robustness tests, in what order will studies be entered into the cumulation?

• If a cumulative meta-analysis is one of the robustness tests, how many steps (i.e., study removals) will be examined to determine robustness?

Other judgments

• What is the size of a lowest possible effect that is still clinically important?

• What is the p value for statistical significance?

• What is the definition of a large magnitude of effect?

• How will qualitative consistency be defined (based on point estimates, confidence intervals, some percentage of studies, etc)?