Skip to main content
Download PDF

Multivariate meta-analysis of critical care meta-analyses: a meta-epidemiological study

BMC Medical Research Methodology volume 21, Article number: 148 (2021) Cite this article

Abstract

Background

Meta-analyses typically consider multiple outcomes and report univariate effect sizes considered as independent. Multivariate meta-analysis (MVMA) incorporates outcome correlation and synthesises direct evidence and related outcome estimates within a single analysis. In a series of meta-analyses from the critically ill literature, the current study contrasts multiple univariate effect estimates and their precision with those derived from MVMA.

Methods

A previous meta-epidemiological study was used to identify meta-analyses with either one or two secondary outcomes providing sufficient detail to structure bivariate or tri-variate MVMA, with mortality as primary outcome. Analysis was performed using a random effects model for both odds ratio (OR) and risk ratio (RR); borrowing of strength (BoS) between multivariate outcome estimates was reported. Estimate comparisons, β coefficients, standard errors (SE) and confidence interval (CI) width, univariate versus multivariate, were performed using Lin’s concordance correlation coefficient (CCC).

Results

In bivariate meta-analyses, for OR (n = 49) and RR (n = 48), there was substantial concordance (≥ 0.69) between estimates; but this was less so for tri-variate meta-analyses for both OR (n = 25; ≥ 0.38) and RR (≥ -0.10; n = 22). A variable change in the multivariate precision of primary mortality outcome estimates compared with univariate was present for both bivariate and tri-variate meta-analyses and for metrics. For second outcomes, precision tended to decrease and CI width increase for bivariate meta-analyses, but was variable in the tri-variate. For third outcomes, precision increased and CI width decreased. In bivariate meta-analyses, OR coefficient significance reversal, univariate versus MVMA, occurred once for mortality and 6 cases for second outcomes. RR coefficient significance reversal occurred in 4 cases; 2 were discordant with OR. For tri-variate OR meta-analyses reversal of coefficient estimate significance occurred in two cases for mortality, nine cases for second and 7 cases for third outcomes. In RR meta-analyses significance reversals occurred for mortality in 2 cases, 6 cases for second and 3 cases for third; there were 7 discordances with OR. BoS was greater in trivariate MVMAs compared with bivariate and for OR versus RR.

Conclusions

MVMA would appear to be the preferred solution to multiple univariate analyses; parameter significance changes may occur. Analytic metric appears to be a determinant.

Peer Review reports

Background

Meta-analyses typically consider more than one outcome, and the conventional approach is to report multiple univariate effect size estimates of these separate outcomes. Such an approach has two attendant consequences; it ignores the effect of outcome correlation upon individual estimates, assuming that they are independent [1], and engenders multiplicity of the Type I error rate [2]. Confounding such effects is the selective reporting of outcomes, or outcome reporting bias (ORB), whereby secondary outcomes are selectively reported based upon outcome results [3, 4]. Multivariate meta-analysis (MVMA), whereby direct evidence and results from related outcomes are synthesised to yield a summary outcome result [5,6,7], is an elegant solution to the above problems.

In meta-analyses of interventions in the critically ill, where mortality is a common primary outcome, it would be expected that secondary outcomes such as intensive care unit (ICU) and hospital length of stay, infections and the requirement for mechanical ventilation would demonstrate substantial correlation [6], and with the primary mortality event. MVMA in such meta-analyses would allow joint inference upon multiple outcomes and be of relevance from a methodological and clinical viewpoint. Price et al. suggested that where multiple outcomes routinely occur, MVMA would be “…more likely to have an impact” [8]. From a previous study which reported mortality outcome of a series of meta-analyses in the critically ill [9] utilising only randomised controlled trials, a meta-analytic cohort was identified where secondary outcomes were reported in such detail as to yield bivariate or tri-variate data structures. Tri-variate data structures have been rarely subjected to MVMA; in the Price et al. analysis [8], only one such MVMA was reported. Univariate and multivariate analyses were undertaken and compared with respect to differences between estimated outcome variable coefficients, their standard errors (SE) and 95% confidence interval (CI) width and statistical significance, with no selection of meta-analyses based upon the number of RCTs per meta-analysis. As a by-product of MVMA coefficient estimation, variable correlations, direct information and borrowing of strength (BoS) were determined. Whereas direct information describes the contribution of data from the same outcome, BoS represents the contribution of data from all other outcomes [10, 11]. One problematic requirement of MVMA is the provision of with-study correlations which are rarely reported, although methods based upon individual patient [12] or aggregated data [13] and within the Bayesian framework [14] have been undertaken. Any recommendation for the practical application of MVMA must be accompanied by appropriate software. As such, the “alternative” MVMA model of Riley [15] was employed, whereby an overall correlation, the total marginal correlation between outcomes, was modelled, enabling seamless application to all meta-analyses considered. As results based upon indices of risk, odds ratio (OR) and risk ratio (RR), are not generally inter-translatable [16], both OR and RR estimates were compared.

Methods

Ethics

The data for this study was abstracted from published studies and an Ethics clearance was not appropriate.

Data management

A previous study [9] was used to identify meta-analyses with either one or two secondary outcomes that provided sufficient detail to generate a bivariate or tri-variate MVMA data structure, with mortality as the primary outcome; all meta-analyses were of randomised controlled trials (RCT). Usable second and third outcomes were identified as presented in the original meta-analysis.

Statistical analysis

  1. 1.

    To facilitate rapid data processing over a large number of models, initial univariate meta-analytic point estimates and standard errors (SE) were computed within Stata™ V17 [17] using the “meta” suite of commands [18]; default estimation used restricted maximum likelihood (REML [19]).

  2. 2.

    Subsequently, both univariate and multivariate outcomes were estimated using the user written Stata command “mvmeta” ([20], Version 3.2.0 6apr2018) in a random effects (RE) formulation. Estimation employed REML with an unstructured covariance and the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm for likelihood maximisation or the Davidon-Fletcher-Powell (DFP) algorithm if there were convergence difficulties. Maximisation employed the “difficult” option (use a different stepping algorithm in nonconcave regions) provided by Stata™.

    1. (i)

      The two sets of univariate estimates were subsequently compared.

    2. (ii)

      Under persistent convergence difficulties of “mvmeta”, the model was refit assuming the overall correlation matrix was fixed and known, with values set equal to the estimates from either the BGFS or DFP algorithm using the “bscovariance” option of “mvmeta” ([8], Appendix 4).

    3. (iii)

      In the MVMA note was taken of very small β coefficient standard errors (SE) with consequent large z values for coefficient significance and very small p-values and CI width, such that the estimates were implausible.

    4. (iv)

      To avoid the requirement for specific within study correlations [1], the “alternative” model of Riley was used [15], whereby an overall correlation, the total marginal correlation between outcomes [21], was modelled; that is an amalgam of the within and between-study correlations [6, 8]. The reported correlation(s) in this paper were these overall correlation(s) [22].

    5. (v)

      Direct information and BoS between estimates were also reported [8, 10, 11], using the default (“sd”) method of “mvmeta”. BoS may be conceptualised as a comparison of variances of the estimated rth component of β under the uni- and multivariate models \(BoS_{r}^{{RV}} = 1 - \frac{{{\text{var}}\left( {\hat{\beta }_{{mv,r}} } \right)}}{{{\text{var}}\left( {\hat{\beta }_{{uv,r}} } \right)}}\), where RV refers to relative variance [11]. This ratio has also been described as the efficiency, “E” [10, 23]. An equivalent but alternative method, decomposition of the score function for β, has advantage in that it defines appropriate study weights within an MVMA [11]. In a univariate RE meta-analysis study weights are inversely proportional to the sum of the within- and between- study variances. In an MVMA analysis, as undertaken by “mvmeta”, weights were derived using the score decompensation method, where the score function \(S\left( \theta \right)\) is the first derivative of the log-likelihood function \(l\left( \theta \right);\)\(S\left( \theta \right) = \frac{{dl\left( \theta \right)}}{{d\theta }}\) and \(l(\theta )\) is the likelihood. The weights were broken down into direct information, the contribution of data from the same outcome, and BoS, the contribution of data for all other outcomes. For a univariate analysis, the weights sum to 1, or when expressed as a percentage, 100, as in the “mvmeta” output. In a MVMA, a simple tabulation of direct information and BoS will sum to 100 for each outcome. In particular, the methodology takes the variance components as fixed and the precisions of the point estimates from a MVMA have an expectation of being greater than or equal to those from separate univariate meta-analyses [24], albeit the latter study employed the methods of Van Houwelingen et al. [25] using “Proc Mixed’’ with SAS statistical software, not the “Riley” method [15].

      1. a.

        The use of the “Riley” method [15] excluded the computation of multivariate I2 for each outcome.

  3. 3.

    The reported confounding effect of small study effects upon changes of statistical significance between univariate and MVMA [5, 8] was explored by inspection of contour-enhanced funnel plots [26] and formal regression based tests, in particular, the Harbord (for binary outcomes) and Egger (for continuous outcomes) tests [27]. Small study effects were reported for all meta-analyses, as a matter of complete reporting, but suffer from the problem of multiple testing. The power and interpretation of the tests are problematic for small RCT number (< 10) meta-analyses and in the presence of moderate (see 4., below) heterogeneity [28, 29]. More importantly, univariate tests may be underpowered compared with the recently described multivariate small study effect test (MSSET), a multivariate extension of Egger’s regression test [30].

  4. 4.

    Meta-analytic heterogeneity was reported as the I2 index and adjudged as medium and high if I2 ≥ 50 and 75% respectively. The I2 index was preferred, compared with τ2, as it is comparable across different metrics and number of RCTs [31].

  5. 5.

    The analyses, using frequentist methods, were performed for bivariate and tri-variate models with both OR and RR metrics.

  6. 6.

    Agreement or otherwise between univariate and multivariate estimation results was undertaken using Lin’s concordance correlation coefficient (CCC) via the user written Stata command “concord” [32]. The CCC combines measures of both precision and accuracy to determine how far the observed data deviate from the line of perfect concordance (that is, the line at 45 degrees on a square scatterplot). Other measure to characterise the comparison were:

    1. (i)

      Estimate differences average and standard deviation (SD); univariate versus MVMA.

    2. (ii)

      95% (Bland and Altman) limits of agreement (LOA)

    3. (iii)

      An F test (Bradley-Blackwood) of equality of means and variances; non-significance implies concordance.

  7. 7.

    Boxplots [33] were used to visualise the density distribution of BOS and total marginal correlations of both OR and RR for bivariate and tri-variate models.

Statistical significance was ascribed at p < 0.05.

Results

The cohort was composed of forty-nine meta-analyses, 18% nutritional therapeutic, 18% non-pharmaceutical therapeutic and 64% pharmaceutical therapeutic, published between 2002 and 2018. The primary outcome in all was mortality; forty-nine were bivariate in outcome data composition and 30 were tri-variate. Details of the mortality, second and third outcome meta-analyses are shown in Tables 1, 2 and 3, respectively. Heterogeneity, as the I2 index, of ≥ 50 and ≥ 75% was found in mortality, second and third outcomes in 12, 31 and 50%, and 0, 16 and 23%, respectively. Of the 49 mortality meta-analyses, five [38, 43, 53, 60, 66] demonstrated evidence of small study effects on formal testing (p < 0.05); for the second outcome, five [37, 53, 59,60,61]; and for the third, five [37, 49, 67, 68, 78]. The disparity between the formal test of small study effects (p < 0.05) and the increased frequency of “query” for contour-enhanced funnel plot assessment in second and third outcomes versus mortality outcome (37, 40 and 4%, respectively) was noted and may be a function of the power of the test (see Methods, 3.). There was uniform agreement (to the second or third decimal point) between univariate estimates of “mvmeta” and “meta” in Stata™.

Table 1 Details of primary (mortality) outcome for meta-analyses
Full size table
Table 2 Details of second outcomes for meta-analyses
Full size table
Table 3 Details of third outcomes for meta-analyses
Full size table

Bivariate model: OR

For the 49 meta-analyses, median (minimum, p25, p75, maximum) number of RCT per meta-analysis for the primary mortality outcome was 13(4, 10, 17, 31); for the second outcomes, 8(4, 6, 10, 31); see Tables 1 and 2. In only 11 meta-analyses was there equality between the reported primary and secondary outcome study numbers. In the MVMA the “bscovariance” option was used once only and there were no instances of “large” Z values. Second outcomes were binary in 39 and continuous in 10 (Tables 1 and 2). Estimate analysis is given in Table 4. Across all outcomes and estimates, the concordance, univariate versus multivariate, was substantial, with a general relative increment, albeit uneven, in the magnitude of multivariate estimates. Means and variances demonstrated little concordance. Reversal of coefficient estimate significance, univariate versus MVMA, occurred no cases for mortality and 6 cases for second outcomes (significant to non-significant in five [36, 43, 70, 71, 79], one meta-analysis exhibiting small study effects [43]; non-significant to significant in in one [59]).

Table 4 Concordance analysis for bivariate model (OR): univariate versus multivariate
Full size table

Bivariate model: RR

In the MVMA, 49 meta-analyses were considered and there were no instances of “large” Z values. Concordance estimate analysis is given in Table 5. Substantial concordance was seen between uni- and multivariate estimates, with a variable relative increment of multivariate estimates (SE and CI width) across outcomes. Multivariate β estimates were variable with respect to univariate and means and variances lacked concordance. Reversal of coefficient estimate significance, univariate versus MVMA, occurred in one case for mortality outcome (significant to non-significant, [52]) and 3 cases for second outcomes (significant to non-significant [36, 61, 70]; one instance [61] was discordant with the OR metric and one instance exhibiting small study effects [61]).

Table 5 Concordance analysis for bivariate model (RR): univariate versus multivariate
Full size table

The bivariate distributions of BoS are displayed in Fig. 1, where an increment of BoS for RR compared with OR, for both mortality and the second outcome is evident.

Fig. 1
figure 1

Bivariate distribution of BoS for OR (left) and RR (right)

Full size image

The bivariate total marginal correlations, mortality vs second outcome, are shown in Fig. 2; both metrics displayed similar distribution.

Fig. 2
figure 2

Bivariate correlations (mortality: second outcome) for OR (left) and RR (right)

Full size image

Tri-variate model: OR

For the 30 meta-analyses, the median (minimum, p25, p75, maximum) number of studies per meta-analysis for the primary mortality outcome was 13(4, 9, 16, 24); for the second outcome 8(4, 6, 10, 20); and the third 7(3, 5, 10, 15). In only 2 meta-analyses [37, 49] was there equality between the reported primary, second and third outcome study numbers. In the MVMA the “bscovariance” option was used on 13 occasions [37, 38, 46, 54, 57, 62, 66, 72, 73, 77, 78] and there were 5 instances of “large” Z values [37, 54, 72, 77, 78] which were sufficient to render estimates implausible and they were not further considered (median number of RCT per meta-analysis for primary, second and third outcomes 12, 6 and 5 respectively). The outcome data set was thus 25 meta-analyses. Second outcomes were binary in 18 and continuous in 7.; third outcomes were binary in 6 and continuous in 19; the “bscovariance” option being used in eight cases.

Concordance estimate analysis is given in Table 6. Variable concordance between uni- and multivariate estimates was observed. Multivariate estimate precision (SE) increased, and confidence interval width tended to decrease compared with univariate, across and within outcomes. A tendency for concordance between means and variances was apparent. Reversal of coefficient estimate significance, univariate versus MVMA, occurred in two cases for mortality ([38, 73] non-significant to significant, one meta-analysis exhibiting small study effects [38]); nine cases for second outcomes (significant to non-significant in 3 [67, 69, 79], one meta-analysis exhibiting small study effects [67]; non-significant to significant in 6 [37, 57, 59, 62, 66, 73]) and 7 cases for third outcomes (significant to non-significant in 3 [40, 46, 67], one meta-analysis exhibiting small study effects [67]; non-significant to significant in 4 [37, 38, 69, 73] with one demonstrating small study effects [37]).

Table 6 Concordance analysis for trivariate model (OR): univariate versus multivariate
Full size table

Tri-variate model: RR

Of the 30 tri-variate meta-analyses, there was one instance of complete convergence failure [46] and seven instances of “large” Z values [37, 38, 56, 66, 72, 73, 78] which were sufficient to render estimates implausible (median number of RCT per meta-analysis for primary, second and third outcomes 10, 6 and 5 respectively); the outcome data set was thus 22 meta-analyses [37, 40, 47, 49, 50, 53,54,55, 57, 59, 60, 62, 67,68,69, 71, 76, 77, 79, 80]. The median (minimum, maximum) number of studies per meta-analysis for the primary mortality outcome was 13(4, 24); for the second outcome, 8(4, 20); and the third 7(4, 15). Second outcomes were binary in 15 and continuous in 7; third outcomes were binary in 7 and continuous in 15. In the MVMA the “bscovariance” option was used on 3 occasions [57, 62, 67]. Concordance estimate analysis is given in Table 7. Concordance between uni- and multivariate estimates was uneven, with no consistent relative change in multivariate estimates, compared with univariate, across or within outcomes. A tendency for concordance between means and variances in second and third outcomes was apparent. Reversal of coefficient estimate significance, univariate versus MVMA, occurred in two case for mortality ([37, 62] with no small study effects, non-significant to significant, not concordant with the OR cases); five cases for second outcomes with no small study effects (significant to non-significant in 2 [69, 79], concordant with OR cases; non-significant to significant in 3 [54, 57, 77]; concordant with one OR cases only [57]); and 3 cases ([54, 60, 67] non-significant to significant, one exhibiting small study effects [67]) for third outcomes with no concordance with OR cases.

Table 7 Concordance analysis for trivariate model (RR): univariate versus multivariate
Full size table

The tri-variate distributions of BoS are displayed using boxplots in Fig. 3. The increment of BoS for OR compared with RR for mortality and the third outcome is evident. In the panel (right top) showing BoS mortality RR there were points of large BoS for two MVMA meta-analyses, 99.3 and 93.6 [57, 62]. Both these MVMA utilized the “bscovariance” option of “mvmeta” as there was initial unresolved convergence. The estimated between study mortality variance was minimal for both (5.24e-06 and 0.005, respectively) and the status of these estimates may be circumspect.

Fig. 3
figure 3

Tri-variate distribution of BoS (mortality, second and third outcomes) for OR (left) and RR (right)

Full size image

The tri-variate total marginal correlations for both OR (left) and RR (right)are shown via boxplots in Fig. 4; with progressive movement to positive correlations from mortality-second outcome through second-third outcome. Positive correlations appeared more frequent with the RR metric.

Fig. 4
figure 4

Tri-variate total marginal correlations (mortality-second outcome, mortality-third outcome, second-third outcome) for both OR (left) and RR (right)

Full size image

Discussion

It is easy to forget that the MVMA approach has a long history dating back to at least 1993 [81] and has subsequently been formally implemented in popular statistical software packages [82,83,84,85]. This being said, MVMA still appears rarely used by practitioners, a decade after a 2009 review by Riley [1]. From within the social science paradigm Becker, in 2000, pointed out that ignoring outcome dependence in meta-analysis will affect Type I error rates and precision and bias of estimates: “No reviewer should ever ignore dependence among study outcomes” [82]. In the current study the total marginal correlations for both bi- and tri-variate analyses was sizeable overall and, depending upon the composition of the non-primary outcomes, more positive than negative and more so for the tri-variate case.

One of the principal attractions of MVMA is estimation of the BoS between parameters, well demonstrated in Fig. 3. Most of the BoS would appear to derive from studies which are more “atypical” in design. In particular, the BoS of secondary outcomes of the ith study is a function of the within-study variance matrix \(\left( {V_{i} } \right)\) and the harmonic average \(\bar{V}\) of all the \(V_{i} {\text{s}}\). BoS can only arise if there are differences between the \(V_{i} {\text{s}}\); which would entail studies of”..substantive difference[s] in background and research methods…”, not simply different sample sizes [10]. The magnitude of outcome BoS would appear to be bounded by percentage of missing data for that outcome [6, 24], which in the current study was substantial (see Results). A percentage missingness of 30–35% of studies informing an outcome was found to result in a “more pronounced” BoS in one empirical study [14]. Any nexus between BoS and missingness requires a missing at random (MAR) assumption, as opposed to missing completely at random (MCAR) for univariate meta-analysis [21]. The notions of MAR and MCAR are well recognised in the bio-medical literature [86], albeit inconsistency of usage has been documented [87]; in particular, the conflation of (non)”ignorable” and MAR [22]. Perhaps not surprisingly, within the domain of outcome reporting bias (ORB) [4], MVMA has been a method of choice to investigate the impact of ORB upon meta-analytic conclusions [22, 88].

Computationally, MVMA requires both within- and between-study correlations and the former are typically not known and are likely not to be available, especially in higher order (trivariate) models [24]. Riley provided four alternate methods to overcome these problems [1]; the most straightforward, yet laborious, being a sensitivity analysis by correlation imputation over the entire range (-1 to + 1). Riley’s alternate model [15] has been found to have good asymptotic statistical properties compared with a fully hierarchical REML model, with known within-study correlations, and with separate univariate meta-analyses. The performance may be problematic when the overall correlation \(\left( {\hat{\rho }} \right)\) is very close to 1 or -1. In the current study, only two instances were found; in the bivariate RR MVMA, \(\hat{\rho }\) = 0.999 [75], and the trivariate OR MVMA, \(\hat{\rho }\) = -0.986 ([57], second versus third outcome); both MVMA utilised the “bscov” option. As the Riley model is a “working” model when the true data generating mechanism is a RE model, the standard variance estimates may not provide confidence interval coverage at the nominal level [21]. Complete failure of convergence in the current study was rare, occurring in one instance [46], but problematic SE estimation was exhibited in the trivariate series, 5 instances in OR metric and 7 in RR. This may relate to the small number of RCT in second and third outcomes (see Table 8), but these numbers were not substantially different compared with meta-analyses not demonstrating this feature, as shown in Table 8.

Table 8 Number of studies per meta-analysis (minimum, median and maximum), a propos large z values
Full size table

Frequentist and Bayesian empirical comparisons between univariate meta-analyses and MVMA have appeared in the literature [8, 14, 89,90,91,92,93] with results demonstrating similar (pooled) parameter estimates between the two analytic forms. However, papers by Riley and co-workers [1, 15, 24, 94], which included formal simulation studies, found advantage; a smaller standard error and mean-square error of pooled estimates, predicated upon the presence of missing data; again, assuming missing at random. That is, in the presence of complete data a bivariate analysis would not be expected to produce a gain in statistical efficiency. The extension to trivariate and higher order outcome data and the inability to provide within study correlations was thus identified as a “pressing research issue”; to wit, the “alternative” model of Riley [15]. Price et al. suggested that estimates of clinical and /or statistical conclusions from MVMA may occasionally differ from those from univariate analyses and observed, somewhat wryly, that any claimed discrepancy “…says more about the dangers of using concepts of statistical significance than it does the use of MVMA” [8]. The results from the current analysis were somewhat at odds with these sentiments and with the general results of bivariate studies, both empirical and simulation (see below), albeit the caution about the variance estimates of the Riley model, above, are noted. A variable change in the multivariate precision of primary mortality outcome estimates compared with univariate analysis was present for both bivariate and tri-variate meta-analyses and for metric. For second outcomes, precision tended to decrease, and CI width increase for bivariate meta-analyses; for third outcomes, precision increased, and CI width decreased. The latter finding appears not to have been previously reported although analytic reports of the tri-variate structure are rare; one case only reported by Price et al. study [8] and two by Trikalinos et al. [14]. With respect to the observed relative changes (univariate versus multivariate) across four concordance analyses, the magnitude of the difference was rather small and accompanied by a more substantial SD, suggesting a heterogeneity of the MVMA effect, grounded in the individual meta-analyses and dependent upon the nature of the outcome, binary or continuous. As MVMA allows for correlation between outcomes, CIs may be wider on the basis of increased between-study variance [8], but this was observed only in the bivariate case in the current analysis. The experience of Price et al. that “MVMA methods can be applied only in a minority of reviews of interventions in pregnancy and childbirth” [8] was not consistent with the current study.

A reviewer pointed to the wide LOA of the β estimates for the second continuous outcome (days) in Table 6 (trivariate OR MVMA), this being -10.894, 8.122. Of the seven meta-analyses considered, two had stand-out differences between univariate and MVMA estimates; the study of Wan et al. ([57], intra-meta-analytic study number = 4), -11.31, -18.17 and Chen et al. ([69],intra-meta-analytic study number = 10), -11.27 and -4.26. The former study used the “bscov” option, recording a BOS for the second outcome of 54% and correlation between second and third outcomes of 0.986; the latter had normal convergence but record a BOS for the second outcome of 92.4% with a correlation between second and third outcomes of 0.995. This may be indicative of problematic estimation, which has been mentioned above and further addressed in “Limitations”, below. Trikalinos et al. ([14], point 4.1), using Bayesian methods, observed that “Generally, point estimates are comparable”; Price et al. ([8], Table 2) using “mvmeta” recorded differences in β between univariate and MVMA, but did not focus attention on such; and in the bivariate simulation study of Riley et al. ([94], Table 4), bias of the mean for \(\beta _{1} \,{\text{and}}\,\beta _{2}\) was comparable with coverage for both between 93–98%; similar results were also observed when considering the “alternate” model of Riley ([15], Table 1).

The differences between the results of the current study and those referenced above [8, 14, 89,90,91,92,93] needs some further explication with regard to data structure. The bi- and tri-variate meta-analyses under consideration were relatively conventional; a primary mortality outcome and second and third recorded outcomes which were not direct extensions of the primary outcome. For second and third outcomes, both categorical (binary) and continuous outcomes were considered, unlike Trikalinos et al. [14] where outcomes were categorical. No repeated measures of a primary outcome, such as different mortality time-points or different types of mortality (all cause or disease specific) were considered; the latter structure featured in the studies of Trikalinos et al. [14, 91], Arends [93] and also in an empirical example Riley et al. [15]. Within the critical care domain the use of MVMA analysis with different mortality time-points has been recently presented [95].The current study did not focus on the impact of different meta-analytic estimators as in Berkey et al. [92], generalized least squares and multivariate maximum likelihood, nor adopt the Bayesian framework of Trikalinos [14]. That bivariate models have been used in systematic reviews of diagnostic test studies for some years, was noted in 2009 by Riley and both Simel et al. [90] and Dahabreh et al. [89] found little advantage for bivariate approaches when considering estimates of sensitivity and specificity. With respect to the change of estimate significance reported here, univariate versus MVMA, the use of the MVMA “bscov” option may have been consequential. For the OR metric, where 24 significance changes occurred, there were seven instances [37, 46, 57, 62, 66, 69, 73], all in trivariate MVMA. For the RR metric, again with the trivariate data structure, there were three [57, 62, 67].

These changes of statistical significance are shown in forest plots as couplets, univariate versus MVMA, for binary (null line unity, Fig. 5) and continuous (null line zero, Fig. 6) outcomes. A majority of the CI width changes that achieved a change of significance about the null appear substantial; the clinical import of such changes would require case by case determination [96].

Fig. 5
figure 5

Binary outcome variables (OR left panel, RR right panel): univariate versus MVMA as couplets. For OR: Bangalore [36], Singh [43], Manzanares [59], Qureshi [70] and Elke [71], second outcome bivariate meta-analysis; Chan [38] and Davies [73], mortality tri-variate meta-analysis; Manzanares [59], Nunez-Patino [62]and Davies [73], second outcome trivariate meta-analysis; Marik [37], Osandik [67] and Chen [69], third outcome tri-variate meta-analysis. For RR meta-analysis: Masip [52], mortality bivariate meta-analysis; Bangalore [36], Rhodes [61] and Qureshi [70], bivariate meta-analysis; Marik [37] and Numez-Patino [62], mortality tri-variate meta-analysis; for Wang [77] and Szakmany [54], trivariate meta-analysis, second outcome

Full size image
Fig. 6
figure 6

Continuous outcome variables (OR metric left panel, RR metric right panel): the scale is integer days (one case (OR [46]) reporting blood loss in ml was omitted due to scaling incompatibilities). For the (OR) left panel: Tang [79] bivariate meta-analysis; Marik [37], Wan [57], Osandik [67], Chen [69] and Tang [79] second outcome tri-variate meta-analysis; Chan [33], Ho [39] and Davies [73] third outcome tri-variate meta-analysis. For the (RR) right panel: Tang [79], Wan [57] and Chen [69] second outcome tri-variate meta-analysis; Szakmany [54], Tian [60] and Osandik [67] third outcome tri-variate meta-analysis

Full size image

Disparities between the OR and RR occurred over a range of indices and may be a function of the current cohort. However, OR and RR are not merely interchangeable metrics and there is no monotone relationship between them [16]. Recent papers have drawn attention to potential estimation problems with the RR. First, the RR effect magnitude is dependent upon the underlying baseline prevalence, shifting toward 1 as prevalence increases, and is a ratio of two conditional probabilities, whereas the OR is a likelihood ratio whose magnitude reflects the fold increase in odds, baseline to intervention, independent of prevalence [97]. Second, under both the DerSimonian-Laird [98] and REML formulations, the requirements of log(RR) estimation to be compatible with study level event rates in the [0,1] interval \((\pi _{j} {\text{treat}}\,{\text{ < }}\,{\text{1}}\,{\text{and 0}}\, < \,\pi _{j} {\text{control}}\,{\text{ < }}\,{\text{1)}}\) demand restriction on the parameter space with ensuant bias in estimates of both \(\tau ^{2}\) and log(RR). Thus risk relativism may be an “illusion “ [97] and the OR “appears to be a safer option” [99]. This being said, Xiao and colleagues argued that interpretability issues of the OR, lack of collapsibility and a dependence on the baseline risk, negates any in-principle recommendation for the OR [100].

Limitations

The current study utilised a single meta-analytic cohort from the critical care domain and had a modest number of bivariate meta-analyses, but less so in the trivariate series. The preference for the alternate model of Riley was a potential limitation, but when reviewing a number of bivariate and tri-variate studies in two metrics the use of sensitivity analysis by specifying within study correlations (via the “wscor” option of “mvmeta”) would be unwieldy and potentially uninterpretable. This being said, the recommendation of Riley et al. in the landmark 2008 paper [15], was that in the presence of overall correlations > 0.9 in absolute value, practitioners “..should assess the robustness of pooled results to small changes in \(\hat{\rho }\) as a sensitivity analysis”. In the MVMA where large z values were found and subsequently not considered, for the OR studies [37, 54, 72, 77, 78] and for the RR studies [37, 38, 56, 66, 72, 73, 78], all meta-analyses had \(\hat{\rho }\) > 0.9 in at least one of the correlations. Whether such a modus operandi would yield credible z values and pooled estimates has not been explored.

The current study has adopted a workable and practical solution to the particular requirements of MVMA. Future studies should replicate or otherwise the findings in this paper using the “alternate” meta-analytic model of Riley and consider meta-analyses from specific disciplines, moving beyond the bivariate data structure to encompass “…three or more end points…” [1], albeit such estimation may be challenging.

Conclusions

MVMA elucidates the structure and correlation between multiple reported outcomes in univariate meta-analyses and resolves outcome reporting bias. Change in estimate precision and CI width with MVMA appeared context dependent. The BoS entailed in this technique may be quantified and change of parameter significance may be a consequence. MVMA is a feasible solution to the meta-analytic estimation of multiple univariate effects.

Availability of data and materials

The collection of the original data set (current reference [9]) from published papers was the joint work of the author (John L Moran) and Dr Petra Graham (see below) and ownership of the data resides with both persons and is thus not available in the public domain, nor in repository.

References

  1. Riley RD. Multivariate meta-analysis: the effect of ignoring within-study correlation. J R Stat Soc Ser A Stat Soc. 2009;172(4):789–811.

    Article  Google Scholar 

  2. Lopez-Lopez JA, Page MJ, Lipsey MW, Higgins JPT. Dealing with effect size multiplicity in systematic reviews and meta-analyses. Res Synth Methods. 2018;9(3):336–51.

    Article  Google Scholar 

  3. Dwan K, Gamble C, Williamson PR, Kirkham JJ, Reporting Bias G. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One. 2013;8(7):e66844.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340(feb15_1):c365.

    Article  PubMed  Google Scholar 

  5. Jackson D, Riley R, White IR. Multivariate meta-analysis: Potential and promise. Stat Med. 2011;30(20):2481–98.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Riley RD, Jackson D, Salanti G, Burke DL, Price M, Kirkham J, White IR. Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples. BMJ. 2017;358:j3932.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Jackson D, White IR, Riley RD. Multivariate meta-analysis. In: Schmid CH, Stijnen T, White IR, editors. Handbook of meta-analysis. Boca Raton: CRC Press; 2021. p. 163–86.

    Google Scholar 

  8. Price MJ, Blake HA, Kenyon S, White IR, Jackson D, Kirkham JJ, Neilson JP, Deeks JJ, Riley RD. Empirical comparison of univariate and multivariate meta-analyses in Cochrane Pregnancy and Childbirth reviews with multiple binary outcomes. Res Synth Methods. 2019;10(3):440–51.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Moran JL, Graham PL. Risk related therapy in meta-analyses of critical care interventions: Bayesian meta-regression analysis. J Crit Care. 2019;53:114–9.

    Article  PubMed  Google Scholar 

  10. Copas JB, Jackson D, White IR, Riley RD. The role of secondary outcomes in multivariate meta‐analysis. J R Stat Soc Ser C Appl Stat. 2018;67(5):1177–205.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Jackson D, White IR, Price M, Copas J, Riley RD. Borrowing of strength and study weights in multivariate and network meta-analysis. Stat Methods Med Res. 2017;26(6):2853–68.

    Article  PubMed  Google Scholar 

  12. Riley RD, Price MJ, Jackson D, Wardle M, Gueyffier F, Wang J, Staessen JA, White IR. Multivariate meta-analysis using individual participant data. Res Synth Methods. 2015;6(2):157–74.

    Article  CAS  PubMed  Google Scholar 

  13. Wei Y, Higgins JP. Estimating within-study covariances in multivariate meta-analysis with multiple outcomes. Stat Med. 2013;32(7):1191–205.

    Article  PubMed  Google Scholar 

  14. Trikalinos TA, Hoaglin DC, Schmid CH. An empirical comparison of univariate and multivariate meta-analyses for categorical outcomes. Stat Med. 2014;33(9):1441–59.

    Article  PubMed  Google Scholar 

  15. Riley RD, Thompson JR, Abrams KR. An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown. Biostatistics. 2008;9(1):172–86.

    Article  PubMed  Google Scholar 

  16. Feng C, Wang B, Wang H. The relations among three popular indices of risks. Stat Med. 2019;38(23):4772–87.

    Article  PubMed  Google Scholar 

  17. StataCorp. Stata release 17. 2021. https://www.stata.com/products/.

    Google Scholar 

  18. StataCorp. Stata meta-analysis reference manual: release 17. 2021. https://www.stata-press.com/manuals/meta-analysis-reference-manual/.

    Google Scholar 

  19. Partlett C, Riley RD. Random effects meta-analysis: coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med. 2017;36(2):301–17.

    Article  PubMed  Google Scholar 

  20. White IR. Multivariate random-effects meta-regression: updates to mvmeta. Stata J. 2011;11(2):255–70.

    Article  Google Scholar 

  21. Hong C, Riley RD, Chen Y. An improved method for bivariate meta-analysis when within-study correlations are unknown. Res Synth Methods. 2018;9(1):73–88.

    Article  PubMed  Google Scholar 

  22. Kirkham JJ, Riley RD, Williamson PR. A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Stat Med. 2012;31(20):2179–95.

    Article  PubMed  Google Scholar 

  23. Lin L, Xing A, Kofler MJ, Murad MH. Borrowing of strength from indirect evidence in 40 network meta-analyses. J Clin Epidemiol. 2018;106:41–9.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Riley RD, Abrams KR, Lambert PC, Thompson JR. An evaluation of bivariate random-effects meta-analysis for the joint synthesis of two correlated outcomes. Stat Med. 2007;26(1):78–97.

    Article  CAS  PubMed  Google Scholar 

  25. Van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med. 2002;21(4):589–624.

    Article  PubMed  Google Scholar 

  26. Palmer TM, Peters JL, Sutton AJ, Moreno SG. Contour-enhanced funnel plots for meta-analysis. Stata J. 2008;8(2):242–54.

    Article  Google Scholar 

  27. StataCorp: meta bias. Stata V17 documentation. 2021. https://www.stata.com/manuals/metametabias.pdf. Accessed 4 May 2021.

  28. Ioannidis JP, Trikalinos TA. The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. Can Med Assoc J. 2007;176(8):1091–6.

    Article  Google Scholar 

  29. Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol. 2000;53(11):1119–29.

    Article  CAS  PubMed  Google Scholar 

  30. Hong C, Salanti G, Morton SC, Riley RD, Chu H, Kimmel SE, Chen Y. Testing small study effects in multivariate meta-analysis. Biometrics. 2020;76(4):1240–50.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Huedo-Medina TB, Sánchez-Meca J, Marin-Martinez F, Botella J. Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychol Methods. 2006;11(2):193–206.

    Article  PubMed  Google Scholar 

  32. Steichen TJ, Cox NJ. A note on the concordance correlation coefficient. Stata J. 2002;2(2):183–9.

    Article  Google Scholar 

  33. Cleveland WS. Graphical methods. In: The elements of graphing data. edn. Summit: Hobart Press; 1994. p. 139–143.

  34. Griesdale DEG, de Souza RJ, van Dam RM, Heyland DK, Cook DJ, Malhotra A, Dhaliwal R, Henderson WR, Chittock DR, Finfer S, et al. Intensive insulin therapy and mortality among critically ill patients: a meta-analysis including NICE-SUGAR study data. Can Med Assoc J. 2009;180(8):821–7.

    Article  Google Scholar 

  35. Annane D, Bellissant E, Bollaert PE, Briegel J, Confalonieri M, De Gaudio R, Keh D, Kupfer Y, Oppert M, Meduri GU. Corticosteroids in the treatment of severe sepsis and septic shock in adults: a systematic review. JAMA. 2009;301(22):2362–75.

    Article  CAS  PubMed  Google Scholar 

  36. Bangalore S, Wetterslev J, Pranesh S, Sawhney S, Gluud C, Messerli FH. Perioperative β blockers in patients having non-cardiac surgery: a meta-analysis. Lancet. 2008;372(9654):1962–76.

    Article  CAS  PubMed  Google Scholar 

  37. Marik P, Zaloga G. Immunonutrition in critically ill patients: a systematic review and analysis of the literature. Intensive Care Med. 2008;34(11):1980–90.

    Article  PubMed  Google Scholar 

  38. Chan EY, Ruest A, Meade MO, Cook DJ. Oral decontamination for prevention of pneumonia in mechanically ventilated adults: systematic review and meta-analysis. Br Med J. 2007;334(7599):889.

    Article  Google Scholar 

  39. Gonzalez R, Zamora J, Gomez-Camarero J, Molinero LM, Banares R, Albillos A. Meta-analysis: combination endoscopic and drug therapy to prevent variceal rebleeding in cirrhosis. Ann Intern Med. 2008;149(2):109–22.

    Article  PubMed  Google Scholar 

  40. Ho KM, Dobb GJ, Webb SAR. A comparison of early gastric and post-pyloric feeding in critically ill patients: a meta-analysis. Intensive Care Med. 2006;32(5):639–49.

    Article  PubMed  Google Scholar 

  41. Ho KM, Ng JY. The use of propofol for medium and long-term sedation in critically ill adult patients: a meta-analysis. Intensive Care Med. 2008;34(11):1969–79.

    Article  CAS  PubMed  Google Scholar 

  42. Siempos II, Ntaidou TK, Falagas ME. Impact of the administration of probiotics on the incidence of ventilator-associated pneumonia: a meta-analysis of randomized controlled trials. Crit Care Med. 2010;38(3):954–62.

    Article  PubMed  Google Scholar 

  43. Singh S, Amin AV, Loke YK. Long-term use of inhaled corticosteroids and the risk of pneumonia in chronic obstructive pulmonary disease a meta-analysis. Arch Intern Med. 2010;169(3):219–29.

    Article  Google Scholar 

  44. Peterson K, Carson S, Carney N. Hypothermia treatment for traumatic brain injury: a systematic review and meta-analysis. J Neurotrauma. 2008;25(1):62–71.

    Article  PubMed  Google Scholar 

  45. Silvestri L, van Saene HKF, Milanese M, Gregori D, Gullo A. Selective decontamination of the digestive tract reduces bacterial bloodstream infection and mortality in critically ill patients. Systematic review of randomized, controlled trials. J Hosp Infect. 2007;65(3):187–203.

    Article  CAS  PubMed  Google Scholar 

  46. Whitlock RP, Chan S, Devereaux PJ, Sun J, Rubens FD, Thorlund K, Teoh KHT. Clinical benefit of steroid use in patients undergoing cardiopulmonary bypass: a meta-analysis of randomized trials. Eur Heart J. 2008;29(21):2592–600.

    Article  PubMed  Google Scholar 

  47. Piccini JP, Berger JS, O’Connor CM. Amiodarone for the prevention of sudden cardiac death: a meta-analysis of randomized controlled trials. Eur Heart J. 2009;30(10):1245–53.

    Article  CAS  PubMed  Google Scholar 

  48. Landoni G, Mizzi A, Biondi-Zoccai G, Bignami E, Prati P, Ajello V, Marino G, Guarracino F, Zangrillo A. Levosimendan reduces mortality in critically ill patients. A meta-analysis of randomized controlled studies. Minerva Anestesiol. 2010;76(4):276–86.

    CAS  PubMed  Google Scholar 

  49. Brar SS, Leon MB, Stone GW, Mehran R, Moses JW, Brar SK, Dangas G. Use of drug-eluting stents in acute myocardial infarction a systematic review and meta-analysis. J Am Coll Cardiol. 2009;53(18):1677–89.

    Article  PubMed  Google Scholar 

  50. Landoni G, Biondi-Zoccai GGL, Tumlin JA, Bove T, De Luca M, Calabro MG, Ranucci M, Zangrillo A. Beneficial impact of Fenoldopam in critically ill patients with or at risk for acute renal failure: a meta-analysis of randomized clinical trials. Am J Kidney Dis. 2007;49(1):56–68.

    Article  CAS  PubMed  Google Scholar 

  51. Mazaki T, Ebisawa K. Enteral versus parenteral nutrition after gastrointestinal surgery: a systematic review and meta-analysis of randomized controlled trials in the English literature. J Gastrointest Surg. 2008;12(4):739–55.

    Article  PubMed  Google Scholar 

  52. Masip J, Roque M, Sanchez B, Fernandez R, Subirana M, Exposito JA. Noninvasive ventilation in acute cardiogenic pulmonary edema - systematic review and meta-analysis. JAMA. 2005;294(24):3124–30.

    Article  CAS  PubMed  Google Scholar 

  53. Oldani M, Sandini M, Nespoli L, Coppola S, Bernasconi DP, Gianotti L. Glutamine supplementation in intensive care patients a meta-analysis of randomized clinical trials. Medicine. 2015;94(31):e1319.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Szakmany T, Russell P, Wilkes AR, Hall JE. Effect of early tracheostomy on resource utilization and clinical outcomes in critically ill patients: meta-analysis of randomized controlled trials. Br J Anaesth. 2015;114(3):396–405.

    Article  CAS  PubMed  Google Scholar 

  55. Alkhawaja S, Martin C, Butler RJ, Gwadry-Sridhar F. Post-pyloric versus gastric tube feeding for preventing pneumonia and improving nutritional outcomes in critically ill adults. Cochrane Database Syst Rev. 2015;2015(8):CD008875.

    PubMed Central  Google Scholar 

  56. van Zanten ARH, Dhaliwal R, Garrel D, Heyland DK. Enteral glutamine supplementation in critically ill patients: a systematic review and meta-analysis. Crit Care. 2015;19(1):294.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Wan X, Gao XJ, Bi JC, Tian F, Wang XY. Use of n-3 PUFAs can decrease the mortality in patients with systemic inflammatory response syndrome: a systematic review and meta-analysis. Lipids Health Dis. 2015;14(1):1–9.

    Article  CAS  Google Scholar 

  58. Teo J, Liew Y, Lee W, Kwa AL-H. Prolonged infusion versus intermittent boluses of beta-lactam antibiotics for treatment of acute infections: a meta-analysis. Int J Antimicrob Agents. 2014;43(5):403–11.

    Article  CAS  PubMed  Google Scholar 

  59. Manzanares W, Dhaliwal R, Jiang XR, Murch L, Heyland DK. Antioxidant micronutrients in the critically ill: a systematic review and meta-analysis. Crit Care. 2012;16(2):1–3.

    Article  Google Scholar 

  60. Tian FMD, Heighes PTMPS, Allingstrup MJP, Doig GSP. Early enteral nutrition provided within 24 hours of ICU admission: a meta-analysis of randomized controlled trials*. Crit Care Med. 2018;46(7):1049–56.

    Article  PubMed  Google Scholar 

  61. Rhodes NJ, Liu J, O’Donnell JN, Dulhunty JM, Abdul-Aziz MH, Berko PY, Nadler B, Lipman J, Roberts JA. Prolonged infusion piperacillin-tazobactam decreases mortality and improves outcomes in severely ill patients: results of a systematic review and meta-analysis. Crit Care Med. 2018;46(2):236–43.

    Article  CAS  PubMed  Google Scholar 

  62. Nunez-Patino RA, Zorrilla-Vaca A, Rivera-Lara L. Comparison of intensive versus conventional insulin therapy in traumatic brain injury: a meta-analysis of randomized controlled trials. Brain Inj. 2018;32(6):693–703.

    Article  PubMed  Google Scholar 

  63. Kawano-Dourado L, Zampieri FG, Azevedo LCP, Correa TD, Figueiro M, Semler MW, Kellum JA, Cavalcanti AB. Low-versus high-chloride content intravenous solutions for critically ill and perioperative adult patients: a systematic review and meta-analysis. Anesth Analg. 2018;126(2):513–21.

    Article  CAS  PubMed  Google Scholar 

  64. Dallimore J, Ebmeier S, Thayabaran D, Bellomo R, Bernard G, Schortgen F, Saxena M, Beasley R, Weatherall M, Young P. Effect of active temperature management on mortality in intensive care unit patients. Crit Care Resusc. 2018;20(2):150–63.

    PubMed  Google Scholar 

  65. Chong MA, Krishnan R, Cheng D, Martin J. Should transfusion trigger thresholds differ for critical care versus perioperative patients? A meta-analysis of randomized trials. Crit Care Med. 2018;46(2):252–63.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Yang X-M, Tu G-W, Zheng J-L, Shen B, Ma G-G, Hao G-W, Gao J, Luo Z. A comparison of early versus late initiation of renal replacement therapy for acute kidney injury in critically ill patients: an updated systematic review and meta-analysis of randomized controlled trials. BMC Nephrol. 2017;18(1):264.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Osadnik CR, Tee VS, Carson-Chahhoud KV, Picot J, Wedzicha JA, Smith BJ. Non-invasive ventilation for the management of acute hypercapnic respiratory failure due to exacerbation of chronic obstructive pulmonary disease. Cochrane Database Syst Rev. 2017;7(7):CD004104.

    PubMed  Google Scholar 

  68. Lu C, Sharma S, McIntyre L, Rhodes A, Evans L, Almenawer S, Leduc L, Angus DC, Alhazzani W. Omega-3 supplementation in patients with sepsis: a systematic review and meta-analysis of randomized trials. Ann Intensive Care. 2017;7(1):58.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  69. Chen QH, Zheng RQ, Lin H, Shao J, Yu JQ, Wang HL. Effect of levosimendan on prognosis in adult patients undergoing cardiac surgery: a meta-analysis of randomized controlled trials. Crit Care. 2017;21(1):253.

    Article  PubMed  PubMed Central  Google Scholar 

  70. Qureshi SH, Rizvi SI, Patel NN, Murphy GJ. Meta-analysis of colloids versus crystalloids in critically ill, trauma and surgical patients. Br J Surg. 2016;103(1):14–26.

    Article  CAS  PubMed  Google Scholar 

  71. Elke G, van Zanten ARH, Lemieux M, McCall M, Jeejeebhoy KN, Kott M, Jiang X, Day AG, Heyland DK. Enteral versus parenteral nutrition in critically ill patients: an updated systematic review and meta-analysis of randomized controlled trials. Crit Care. 2016;20(1):117.

    Article  PubMed  PubMed Central  Google Scholar 

  72. Parikh HG, Miller A, Chapman M, Moran JL, Peake SL. Calorie delivery and clinical outcomes in the critically ill: a systematic review and meta-analysis. Crit Care Resusc. 2016;18(1):17–24.

    PubMed  Google Scholar 

  73. Davies ML, Chapple L-AS, Chapman MJ, Moran JL, Peake SL. Protein delivery and clinical outcomes in the critically ill: a systematic review and meta-analysis. Crit Care Resusc. 2017;19(2):117–27.

    PubMed  Google Scholar 

  74. Abroug F, Ouanes I, Abroug S, Dachraoui F, Ben Abdallah S, Hammouda Z, Ouanes-Besbes L. Systemic corticosteroids in acute exacerbation of COPD: a meta-analysis of controlled studies with emphasis on ICU patients. Ann Intensive Care. 2014;4:32.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  75. Manzanares W, Lemieux M, Langlois PL, Wischmeyer PE. Probiotic and synbiotic therapy in critical illness: a systematic review and meta-analysis. Crit Care. 2016:262.

  76. Peter JV, Moran JL, Phillips-Hughes J, Warne D. Non-invasive ventilation (NIV) in acute respiratory failure: a meta-analysis update. Crit Care Med. 2002;30(3):555–62.

    Article  PubMed  Google Scholar 

  77. Wang CH, Ma MH-M, Chou HC, Yen ZS, Yang CW, Fang CC, Chen SC. High-dose vs non high-dose proton pump inhibitors after endoscopic treatment in patients with bleeding peptic ulcer a systematic review and meta-analysis of randomized controlled trials. Arch Intern Med. 2010;170(9):751–8.

    Article  CAS  PubMed  Google Scholar 

  78. Tao W, Li P-S, Shen Z, Shu Y-S, Liu S. Effects of omega-3 fatty acid nutrition on mortality in septic patients: a meta-analysis of randomized controlled trials. BMC Anesthesiol. 2016;16:39.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  79. Tang H, Huang T, Jing J, Shen H, Cui W. Effect of procalcitonin-guided treatment in patients with infections: a systematic review and meta-analysis. Infection. 2009;37(6):497–507.

    Article  CAS  PubMed  Google Scholar 

  80. Muscedere J, Rewa O, McKechnie K, Jiang XR, Laporta D, Heyland DK. Subglottic secretion drainage for the prevention of ventilator-associated pneumonia: a systematic review and meta-analysis. Crit Care Med. 2011;39(8):1985–91.

    Article  PubMed  Google Scholar 

  81. Vanhouwelingen HC, Zwinderman KH, Stijnen T. A bivariate approach to meta-analysis. Stat Med. 1993;12(24):2273–84.

    Article  CAS  Google Scholar 

  82. Becker BJ. Multivariate meta-analysis. In: Tinsley HEA, Brown SD, editors. Handbook of applied multivariate statistics and mathematical modeling. San Diego: Academic; 2000. p. 499–525.

    Chapter  Google Scholar 

  83. Gasparrini A, Armstrong B, Kenward MG. Multivariate meta-analysis for non-linear and other multi-parameter associations. Stat Med. 2012;31(29):3821–39.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  84. Luo S, Chen Y, Su X, Chu H. mmeta: an R package for multivariate meta-analysis. J Stat Softw. 2014;56(11):26.

    Article  Google Scholar 

  85. White IR. Multivariate random-effects meta-analysis. Stata J. 2009;9(1):40–56.

    Article  Google Scholar 

  86. Vesin A, Azoulay E, Ruckly S, Vignoud L, Rusinova K, Benoit D, Soares M, Azeivedo-Maia P, Abroug F, Benbenishty J, et al. Reporting and handling missing values in clinical studies in intensive care units. Intensive Care Med. 2013;39(8):1396–404.

    Article  PubMed  Google Scholar 

  87. Seaman S, Galati J, Jackson D, Carlin J. What is meant by “missing at random”? Stat Sci. 2013;28(2):257–68.

    Article  Google Scholar 

  88. Frosi G, Riley RD, Williamson PR, Kirkham JJ. Multivariate meta-analysis helps examine the impact of outcome reporting bias in Cochrane rheumatoid arthritis reviews. J Clin Epidemiol. 2015;68(5):542–50.

    Article  PubMed  Google Scholar 

  89. Dahabreh IJ, Trikalinos TA, Lau J, Schmid CH. Univariate and bivariate likelihood-based meta-analysis methods performed comparably when marginal sensitivity and specificity were the targets of inference. J Clin Epidemiol. 2017;83:8–17.

    Article  PubMed  Google Scholar 

  90. Simel DL, Bossuyt PMM. Differences between univariate and bivariate models for summarizing diagnostic accuracy may not be large. J Clin Epidemiol. 2009;62(12):1292–300.

    Article  PubMed  Google Scholar 

  91. Trikalinos TA, Olkin I. Meta-analysis of effect sizes reported at multiple time points: a multivariate approach. Clin Trials. 2012;9(5):610–20.

    Article  PubMed  Google Scholar 

  92. Berkey CS, Hoaglin DC, Antczak-Bouckoms A, Mosteller F, Colditz GA. Meta-analysis of multiple outcomes by regression with random effects. Stat Med. 1998;17(22):2537–50.

    Article  CAS  PubMed  Google Scholar 

  93. Arends L. Multivariate meta-analysis: modelling the heterogeneity. PhD Disseration. The Netherlands: Erasmus University; 2006. https://core.acu.k/download/pdf/19187601.pdf, downloaded 4th June 2016.

  94. Riley R, Abrams K, Sutton A, Lambert P, Thompson J. Bivariate random-effects meta-analysis and the estimation of between-study correlation. BMC Med Res Methodol. 2007;7(1):3.

    Article  PubMed  PubMed Central  Google Scholar 

  95. Moran JL, Graham PL. Multivariate meta-analysis of the mortality effect of prone positioning in the acute respiratory distress syndrome. J Intensive Care Med. 2021:08850666211014479. https://0-doi-org.brum.beds.ac.uk/10.1177/08850666211014479.

  96. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, p values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–50.

    Article  PubMed  PubMed Central  Google Scholar 

  97. Doi SA, Furuya-Kanamori L, Xu C, Lin L, Chivese T, Thalib L. Questionable utility of the relative risk in clinical research: a call for change to practice. J Clin Epidemiol. 2020. https://www.jclinepi.com/article/S0895-4356(20)31171-9/fulltext, Downloaded 19th November 2020.

  98. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–88.

    Article  CAS  PubMed  Google Scholar 

  99. Bakbergenuly I, Hoaglin DC, Kulinskaya E. Pitfalls of using the risk ratio in meta-analysis. Res Synth Methods. 2019;10(3):398–419.

    Article  PubMed  PubMed Central  Google Scholar 

  100. Xiao M, Chen Y, Cole S, MacLehose R, Richardson D, Chu H. Is OR “portable” in meta-analysis? Time to consider bivariate generalized linear mixed model. medRxiv : the preprint server for health sciences 2020. https://0-doi-org.brum.beds.ac.uk/10.1101/2020.11.05.20226811.

Download references

Acknowledgements

Dr Petra Graham (Macquarie University, NSW Australia) for diligent help in assembling the original series of meta-analyses as reported in Reference [8].

Professor Ian White (Medical Research Council Clinical Trials Unit at University College London, UK) for advice on command syntax of his user-written Stata module “mvmeta”.

Funding

Local Intensive Care Unit funds only.

Author information

Authors and Affiliations

  1. Department of Intensive Care Medicine, The Queen Elizabeth Hospital, Woodville, SA, 5011, Australia

    John L. Moran

Authors
  1. John L. Moran
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JLM is the sole author of this paper. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to John L. Moran.

Ethics declarations

Ethics approval and consent to participate

The data for this study was abstracted from published studies and an Ethics clearance was not appropriate.

Competing interests

The author declares no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moran, J.L. Multivariate meta-analysis of critical care meta-analyses: a meta-epidemiological study. BMC Med Res Methodol 21, 148 (2021). https://0-doi-org.brum.beds.ac.uk/10.1186/s12874-021-01336-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12874-021-01336-4

Keywords

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us