Skip to main content
Download PDF

Low methodological quality of systematic reviews on acupuncture: a cross-sectional study

BMC Medical Research Methodology volume 21, Article number: 237 (2021) Cite this article

Abstract

Background

While well-conducted systematic reviews (SRs) can provide the best evidence on the potential effectiveness of acupuncture, limitations on the methodological rigour of SRs may impact the trustworthiness of their conclusions. This cross-sectional study aimed to evaluate the methodological quality of a representative sample of SRs on acupuncture effectiveness.

Methods

Cochrane Database of Systematic Reviews, MEDLINE, and EMBASE were searched for SRs focusing on the treatment effect of manual acupuncture or electro-acupuncture published during January 2018 and March 2020. Eligible SRs must contain at least one meta-analysis and be published in English language. Two independent reviewers extracted the bibliographical characteristics of the included SRs with a pre-designed questionnaire and appraised the methodological quality of the studies with the validated AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2). The associations between bibliographical characteristics and methodological quality ratings were explored using Kruskal-Wallis rank tests and Spearman’s rank correlation coefficients.

Results

A total of 106 SRs were appraised. Only one (0.9%) SR was of high overall methodological quality, zero (0%) was of moderate-quality, six (5.7%) and 99 (93.4%) were of low-quality and critically low-quality respectively. Among appraised SRs, only ten (9.4%) provided an a priori protocol, four (3.8%) conducted a comprehensive literature search, five (4.7%) provided a list of excluded studies, and six (5.7%) performed meta-analysis appropriately. Cochrane SRs, updated SRs, and SRs that did not search non-English databases had relatively higher overall quality.

Conclusions

Methodological quality of SRs on acupuncture is unsatisfactory. Future reviewers should improve critical methodological aspects of publishing protocols, performing comprehensive search, providing a list of excluded studies with justifications for exclusion, and conducting appropriate meta-analyses. These recommendations can be implemented via enhancing the technical competency of reviewers in SR methodology through established education approaches as well as quality gatekeeping by journal editors and reviewers. Finally, for evidence users, skills in SR critical appraisal remain to be essential as relevant evidence may not be available in pre-appraised formats.

Peer Review reports

Introduction

The delivery of traditional, complementary, and integrative medicine (TCIM) services in an evidence-based manner is advocated by the World Health Organization (WHO) in its Traditional Medicine Strategy 2014–2023 [1]. As a popular form of TCIM, the use of acupuncture is increasing globally. In China, traditional Chinese medicine (TCM) constitutes a formal part of the health system, of which 20% of all outpatient services were delivered by the TCM sector, including acupuncture [2]. In Taiwan, acupuncture services are covered by the National Health Insurance [3], and the prevalence of acupuncture use was 11% in 2011 [4]. Meanwhile, in Australia [5], Germany [6], and Norway [7], acupuncture is not only regulated by the government or relevant authorities but is also partially or fully covered by statutory health insurance.

In response to the WHO’s initiative, there is a need to synthesise up-to-date evidence on the effectiveness of acupuncture, so as to facilitate the implementation of evidence-based acupuncture services. With the increasing numbers of acupuncture trials being published, keeping up with new trial results continually has become almost impossible for clinicians, managers, and policymakers [8]. Accordingly, decision-makers rely on systematic reviews (SRs) as one of the key tools for making informed decisions on the use of acupuncture interventions [9]. High-quality SRs provide a tool to assist decision-making based on a trustworthy, clear, and comprehensive synthesis of the best available evidence on a particular clinical question [9].

Although the number of SRs on acupuncture effectiveness has been increasing recently [10], there are still concerns over their methodological quality [11, 12]. For instance, inappropriate literature search, absence of critical appraisal of included primary studies, and meta-analysis of highly heterogeneous studies may give rise to biased conclusions [13, 14]. These methodological limitations may mislead decision-making in clinical practice. It is necessary to evaluate the rigour of existing SRs and consider their trustworthiness for informing decision-making.

This cross-sectional study aimed to (i) describe the bibliographical characteristics of SRs on acupuncture trials; (ii) appraise the methodological quality of SRs on acupuncture trials using AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2) [12].

Methods

Eligibility criteria

To be eligible, SRs must report at least one of the following six defining features: (i) research question; (ii) information sources searched; (iii) inclusion and exclusion criteria; (iv) screening and selection methods; (v) risk of bias assessment of the primary studies; or (vi) data synthesis and analysis methods [15]. SRs published in English with at least one meta-analysis on the treatment effect of acupuncture, including traditional manual acupuncture and electro-acupuncture, were eligible. Acupuncture refers to the use of stainless-steel filiform needles to puncture specific acupoints on the body to trigger specific therapeutic effects [16]. Meta-analysis refers to the quantitative combination of results from two or more separate trials [11]. SRs on acupuncture with moxibustion, a TCM therapy involving the burning of herbs over the skin [17], were also included. SRs on transcutaneous electrical nerve stimulation and laser acupuncture were excluded. Animal studies, narrative reviews, protocol, and network meta-analyses were also ineligible. For duplicates of SRs, the most updated versions were included for appraisal.

Literature search

A comprehensive literature search was conducted in three international electronic databases, including the Cochrane Database of Systematic Reviews, MEDLINE, and EMBASE, for a representative sample of SRs published from January 2018 to March 2020. It is recommended that SR should be updated every two years, and hence we have chosen a sampling time frame that allowed us to focus on current SRs [11]. Details on the search strategies are shown in eTable 1, Additional file 1. The search strategies were adopted from previous SRs on acupuncture [18, 19]. Validated search filters for SRs were applied to maximise the specificity of search on MEDLINE and EMBASE [20, 21]. In this cross-sectional study of SRs, the three databases of Cochrane Database of Systematic Reviews, MEDLINE, and EMBASE were considered as the sampling frame where individual SRs were sampled. To ensure representativeness, we sampled all eligible SRs in this cross-sectional study as long as they were identified in the search using the validated search filters. This census-like sampling procedure facilitated the generation of a representative sample of SRs which are most commonly utilised by clinicians and policymakers. The use of these databases for identifying SRs is recommended in the Comprehensive Framework of Methods for Conducting, Interpreting and Reporting Overviews [22].

Literature screening and data extraction

All retrieved citations were imported into Endnote X9. After deduplication, titles and abstracts of retrieved citations were screened against the eligibility criteria. Full texts of potentially eligible citations were subsequently retrieved for further assessment. For included SRs, bibliographic characteristics were extracted using a pre-designed questionnaire (eTable 2, Additional file 1) [23,24,25,26]. In academia, journal impact factor (JIF) is a widely accepted metric for measuring journal quality, evaluating the performance of researchers and institutions, and more importantly, influencing academic promotion and funding allocation [27]. Despite criticisms of its over-simplistic algorithm [28], journals with higher JIF are still regarded as more prominent in their fields [29]. Indeed, publications in journals with higher JIF are assumed to demonstrate higher methodological and reporting quality [30]. Given the existing wide acceptance of JIF, we included this as a bibliographic characteristic of SR. We also investigated the potential relationship between JIF and SR methodological quality.

Literature selection and data extraction were conducted by two independent reviewers (FYTK and AKLC). Disagreements and discrepancies were resolved via consensus between reviewers, or by arbitration of a third reviewer (CHLW).

Methodological quality assessment

Methodological quality of included SRs was appraised by the validated AMSTAR 2 [12], across all 16 domains. AMSTAR 2 has moderate inter-rater reliability as supported by a median kappa value of 0.51 [31]. Its validity has been demonstrated by a strong positive correlation with scoring from ROBIS (A Risk of Bias Assessment Tool for Systematic Reviews) (r = 0.84) [31].

Seven domains were considered as critical:

  1. i.

    Protocol registered before commencement of the review (item 2)

  2. ii.

    Adequacy of the literature search (item 4)

  3. iii.

    Justification for excluding individual studies (item 7)

  4. iv.

    Risk of bias from individual studies being included in the review (item 9)

  5. v.

    Appropriateness of meta-analytical methods (item 11)

  6. vi.

    Consideration of risk of bias when interpreting the results of the review (item 13)

  7. vii.

    Assessment of presence and likely impact of publication bias (item 15)

Based on their performance on each domain, each SRs were rated as being “high”, “moderate”, “low”, and “critically low” in terms of overall methodological quality [12], in accordance with published operational guidelines. Methodological quality assessment was conducted by two authors (FYTK and LH) independently. Disagreements and discrepancies were resolved via consensus between authors, or by arbitration of a senior researcher (VCHC).

Data analysis

Data on bibliographical characteristics and AMSTAR 2 methodological quality assessment results were summarised using descriptive statistics. Differences in the overall methodological quality of SRs across different bibliographical characteristics were investigated using Kruskal-Wallis rank tests and Spearman’s rank correlation coefficients. A p-value < 0.05 was considered statistically significant. All statistical analyses were performed using SPSS 26.

Results

Literature selection

The literature search yielded a total of 1065 citations. After deduplication, titles and abstracts of 764 citations were screened. Then, 185 publications proceeded to full-text eligibility assessment. Finally, 106 SRs fulfilled the eligibility criteria and were included (eTable 3, Additional file 1). Details on literature selection are illustrated in Fig. 1. A full list of excluded records is presented in eTable 4, Additional file 1.

Fig. 1
figure 1

Process of literature selection

Full size image

Bibliographical characteristics of the included systematic reviews

The 106 included SRs contained 1864 randomised controlled trials with 204,784 participants. Only five SRs (4.7%) were Cochrane reviews. Nineteen SRs (17.9%) were an update of previous SRs. JIF ranged from 0 to 6.8 with a median of 2.0. Number of review authors ranged from two to 13 with a median of six. The corresponding authors of 93 (87.7%) SRs were from Asia, seven (6.6%) from America, four (3.8%) from Europe, and two (1.9%) from Oceania. Over a half (66; 62.3%) of the SRs had their funding sources located in Asia, while 24 (22.6%) SRs did not receive any funding support.

One-hundred-and-five (99.1%) SRs involved English database searching, while 88 (83.0%) involved non-English database searching. Most SRs reported both starting and ending years of search (81; 76.4%) and search terms for one or more electronics databases (101; 95.2%). Seventy-six (71.7%) SRs reported intervention harms. Nevertheless, 59 (55.7%) SRs did not report the language of the included primary studies.

Ninety-nine (93.4%) applied the Cochrane risk-of-bias tool for assessing risk of bias, two used Jadad scale (1.9%) or Pedro scale (1.9%) respectively, and two did not perform risk of bias assessment (1.9%). One-hundred-and-two (96.2%) SRs included a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analysis) -like flow diagram to illustrate the process of literature selection. Details on bibliographical characteristics are shown in Table 1.

Table 1 Bibliographical characteristics of the 106 included systematic reviews on acupuncture
Full size table

Methodological quality

Performance among the included SRs was inadequate across four of the seven AMSTAR 2 critical domains, with that less than 20% satisfying the following: ten (9.4%) SRs established an a priori protocol and justified deviations from the protocol (item 2); four (3.8%) implemented a comprehensive literature search strategy (item 4); five (4.7%) listed excluded studies and justified the exclusions (item 7); and six (5.7%) conducted appropriate meta-analysis (item 11).

Included SRs performed relatively better across the remaining three critical domains: 97 (91.5%) had the risk of bias of individual studies assessed by appropriate instruments (item 9); 78 (73.6%) accounted for risk of bias among individual studies when interpreting results (item 13); and 23 (21.7%) investigated publication bias, and discussed its potential impact on the results (item 15).

Performance was unsatisfactory among four of the nine non-critical domains, with less than 20% fulfilling relevant criteria: four (3.8%) explained the selection of study designs for inclusion (item 3); 12 (11.3%) described included studies in adequate details (item 8); four (3.8%) reported sources of funding among individual studies included (item 10); and 19 (17.9%) assessed potential impact of risk of bias among individual studies on the results of meta-analysis (item 12).

More than 75% of SRs performed well across the remaining five non-critical domains: all SRs reported the PICO (Problem/Patient/Population, Intervention/Indicator, Comparison, and Outcome) components in their research questions and inclusion criteria (item 1); 94 (88.7%) and 97 (91.5%) SRs performed study selection (item 5) and data extraction (item 6) in duplicate, respectively; 84 (79.2%) provided a satisfactory explanation for heterogeneity in the results (item 14); and nearly all (104; 98.1%) reported the potential sources of conflict of interest (item 16). Details on the overall and individual assessment results of the included SRs are illustrated in Table 2 and Additional file 2, respectively.

Table 2 Results of the AMSTAR 2 items for the 106 systematic reviews on acupuncture
Full size table

Relationship between bibliographical characteristics and overall methodological quality

Among the 106 appraised SRs, only one (0.9%) of them was of high overall methodological quality, while six (5.7%) were of low-quality. The remaining 99 (93.4%) SRs were of critically low-quality.

Results of Kruskal-Wallis tests indicated that there were statistically significant between-group differences across three bibliographical characteristics (Table 3). Cochrane reviews (P < 0.001), an update of a previous non-Cochrane review (P = 0.007), and SRs that did not search non-English databases (P = 0.048) had higher overall methodological quality than their counterparts. The Spearman’s rank correlation coefficient also showed that SRs published in higher JIF journals (rs = 0.36; P < 0.001) were associated with higher overall methodological quality. No significant associations were identified between overall methodological quality and reporting of harms, funding location, year of coverage, search terms reporting, publication language restriction, risk-of-bias assessment tools used, and the inclusion of PRISMA-like flow diagram.

Table 3 Overall methodological quality of the 106 systematic reviews on acupuncture by bibliographical characteristics
Full size table

Discussion

Summary of results

This cross-sectional study investigated the methodological quality of a representative sample of 106 SRs on acupuncture effectiveness published between 2018 to 2020. Our results revealed that the methodological rigour of recent acupuncture SRs is weak, with more than 93% being critically low-quality. The observation that the majority of the SRs are of critically low-quality resembles findings from a similar study [32], and the floor effect caused by high standards set by the AMSTAR 2 might be an explanation. Nevertheless, such poor ratings also reflect a real need in improving SR methodology in this field [33], as only high-quality SRs should be used for guiding decision-making.

Being Cochrane review, an update of a previous non-Cochrane review, SRs that did not search non-English databases, and being published in journals with higher JIF were associated with better quality, but they only constituted a small number of SRs. In this cross-sectional study, we revealed that Cochrane reviews are likely to have a higher methodological quality than non-Cochrane reviews. It might be due to the Cochrane Collaboration’s stringent editorial requirements consisting of peer-reviewing of SR protocols [34]. This requirement acts as a gatekeeper to ensure the rigour of Cochrane reviews. Being an update of a previous SR was found to be associated with higher methodological quality as well. This observation might be attributable to improved methodological competency and experience among authors over time. The positive association between JIF and rigour echoes previous findings [30], showing the link between methodological quality and higher JIF in the context of SRs. Finally, we observed that SRs that did not search non-English databases have better rigour than their counterparts. This observation could be incidental, as it seems counterintuitive. It is known that the conduct of literature search on non-English databases is determined by the availability of funding and resources [35], and indeed these are usually more abundant in more experienced teams with more methodological expertise [11].

Comparisons with other cross-sectional studies on systematic review rigour

The proportion of acupuncture focused SRs with high or moderate overall methodological quality (0.9%) is substantially lower than recent SRs on asthmatic treatments (15.4%) [36], osteoarthritic interventions (9.0%) [37], and osteoporosis treatments (4.0%) [26]. However, among AMSTAR 2 critical domains, SRs on acupuncture performed better than SRs of interventions for asthma, osteoporosis, and osteoarthritis [26, 36, 37]: (i) using satisfactory techniques for assessing the risk of bias in primary studies; (ii) conducting comprehensive literature search; and (iii) accounting for risk of bias among primary studies when interpreting synthesised results.

Recommendation for future systematic reviews

Publishing an a priori review protocol

As SR authors tend to include primary studies with positive results [38], publication of an a priori SR protocol would reduce selective outcome reporting and enable comparison of SR protocol and its publications [11, 39]. This also minimises influence of reviewers’ biases caused by foreknowledge on preliminary results, allows peer-reviewing of planned methods, and reduces research waste due to duplication [11]. Our study showed that only 9.4% of SRs satisfied this criterion. Future authors should publish SR protocols in open-access journals, or register them on international databases [11, 12], like the International Prospective Register of Systematic Reviews (PROSPERO) [40].

Conducting comprehensive literature search

Only 3.8% fulfilled the criteria for completing a comprehensive literature search, as many did not conduct searches on trial registries, conference abstracts, theses, and grey literature, examining reference lists of included studies, and consulting the experts in the field of acupuncture. Such incomprehensiveness may give rise to publication bias, leading to over-estimation of effectiveness [41, 42]. On average, the exclusion of grey literature may result in an overestimation of intervention effect by approximately 12% [43]. It is also noteworthy that 55.7% did not report the language of included primary studies, which cast doubts on whether non-English publications were included. If this is the case, language bias may occur [11], leading to an overestimation or underestimation of intervention effect [44, 45]. For future acupuncture SR, ensuring a search for grey as well as non-English literature is a clear area for future improvement.

Providing a list of excluded studies and detailed description of included primary studies

A list of excluded studies with justifications for exclusion promotes transparency and reproducibility of SRs [46, 47]. Such a list may reduce potential publication bias and exclusion errors caused by unavoidable subjectivity during the study selection process [11, 12]. On the other hand, for included primary studies extensive details on PICO elements should be reported. These details can assist evidence users in evaluating the external validity and applicability of the findings [11], as well as in facilitating the exploration of clinical heterogeneity across primary studies [11, 12, 48]. Unfortunately, only 4.7 and 11.3% of SRs provided a list of excluded studies with rationales, or described the included primary studies in detail, respectively. Future reviewers should avoid these limitations.

Conducting appropriate meta-analysis

Our findings indicated that 94.3% applied inappropriate meta-analysis methods, mainly due to improper choice for a fixed-effect model. This model assumes that there is only one true effect size among all included studies, and the pooled effect estimate is common to all studies. On the contrary, a random-effect model assumes that the true effect size varies among studies, and pooled effect estimate is the mean of a distribution of true effects [49]. In the context of a meta-analysis of acupuncture trials, a random effect model is the correct choice in most of the cases. This is because some heterogeneity across trials is expected, and the assumption that there is only one true effect size is unlikely to hold [11, 49].

Strengths and limitations

This study applied AMSTAR 2 to evaluate an up-to-date representative sample of SRs on acupuncture effectiveness. While our results indicated an urgent need to improve the methodological quality of SRs in the field, there are also several limitations. This cross-sectional study only appraised SRs published in English, and we did not search for SRs indexed in grey literature databases or prospective registers like the PROSPERO. These might have limited the representativeness of our sample. That said, it is expected that English SRs indexed in major international databases remained to be key sources of evidence impacting clinical decisions on acupuncture internationally.

In the context of SR, methodological quality and reporting quality are two distinct concepts. The former refers to the capability of an SR in providing an accurate and comprehensive summary of the results of available primary studies [12]. The latter entails the extent to which an SR described its rationale, methods, findings, and other relevant information transparently, completely, and accurately [50]. However, in this cross-sectional study, the reliability of our appraisal depended on how comprehensive the SRs reported their methodology. Poor reporting quality and journals word limit might have negatively influenced the accuracy of assessment in our sample [10].

Implications

Most SRs on acupuncture effectiveness are of critically low methodological quality, of which these may give rise to an underestimation or overestimation of treatment effectiveness. Healthcare providers, guideline developers, and other evidence users should critically appraise the methodological quality of SRs before applying relevant evidence in policy and clinical decision-making. Journal editors and peer-reviewers are also recommended to use AMSTAR 2 and Cochrane Handbook for Systematic Reviews of Interventions as guidelines for verifying the quality of SRs submitted [11, 12]. Critical appraisal of SRs is time-consuming and requires specialised training which may not be available to decision-makers. To facilitate access to quality evidence, pre-appraised evidence resources have been established to inform policy or clinical decision-making. Platforms that gather appraised evidence-based information about specific clinical topics with regular updates, such as DynaMed and UpToDate, are particularly useful [51]. However, if the clinical question is yet answered by these platforms, decision-makers need to depend on SRs, and a prudent evaluation of SRs’ trustworthiness is still required.

Internationally, recommendations on the use of acupuncture have been increasing in clinical practice guidelines [52, 53]. In recent years, the Chinese Government has been upscaling the resources allocated to acupuncture research and development, as well as advocating evidence-based acupuncture practice [54]. Unfortunately, findings of this study revealed that the methodological rigour of SRs on acupuncture may not be robust enough to support these guidelines and policy initiatives. Prior to extensive implementation of evidence-based acupuncture services, clinical epidemiology education, particularly on performing high-quality clinical research and synthesis, should be emphasised. To implement the recommendations above, enhancing technical competency in SR methodology via established education approaches is warranted [11]. SR courses focusing on protocol preparation, search strategy formulation, meta-analytic methods, and result reporting have already been developed, and trainers may make use of those curriculums [55,56,57]. Student-led discussions, in-class activities, and student-teacher interactions are essential for increasing students’ engagement and ensuring pedagogical quality [55,56,57]. An evaluation of an SR methodology course showed that students were confident in selecting appropriate databases, and understanding the importance of reproducible and systematic search strategies after training, with a mean confidence score reaching 4.88 (out of 5.0) [55]. These educational outcomes were achieved with a contact time of two to three hours per week for eight to thirteen weeks [55].

Conclusions

Methodological quality of SRs on acupuncture published in recent years is unsatisfactory, with only 0.9% of them being of high overall quality. For future SR authors, improvement efforts should focus on publishing a priori research protocols, conducting comprehensive literature search, providing lists of excluded studies with justifications for exclusion, and employing appropriate methods for meta-analysis. Technical competency of reviewers in SR methodology may be enhanced via established education approaches and quality gatekeeping by journal editors and reviewers. For evidence users, skills in SR critical appraisal remain to be essential as pre-appraised evidence may not be available.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

AMSTAR 2:

A MeaSurement Tool to Assess systematic Reviews 2

JIF:

Journal impact factor

PICO:

Problem/Patient/Population, Intervention/Indicator, Comparison, and Outcome

PRISMA:

Preferred Reporting Items for Systematic Reviews and Meta-analysis

PROSPERO:

International Prospective Register of Systematic Reviews

SR:

Systematic review

ROBIS:

A Risk of Bias Assessment Tool for Systematic Reviews

TCM:

Traditional Chinese medicine

TCIM:

Traditional, complementary, and integrative medicine

WHO:

World Health Organization

References

  1. World Health Organization. WHO traditional medicine strategy: 2014–2023. Geneva: World Health Organization; 2013.

    Google Scholar 

  2. Chung VCH, Ma PHX, Wang HHX, Wang JJ, Hong LC, Wei X, et al. Integrating traditional Chinese medicine Services in Community Health Centers: insights into utilization patterns in the Pearl River region of China. Evid Based Complement Alternat Med. 2013;2013:426360.

    PubMed  PubMed Central  Google Scholar 

  3. Chen FP, Kung YY, Chen TJ, Hwang SJ. Demographics and patterns of acupuncture use in the Chinese population: the Taiwan experience. J Altern Complement Med. 2006;12(4):379–87.

    Article  PubMed  Google Scholar 

  4. Wu MY, Lee YC, Lin CL, Huang MC, Sun MF, Yen HR. Trends in use of acupuncture among adults in Taiwan from 2002 to 2011: a nationwide population-based study. PLoS One. 2018;13(4):e0195490.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Zheng Z. Acupuncture in Australia: regulation, education, practice, and research. Integr Med Res. 2014;3(3):103–10.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Trinczek K. Reimbursement for acupuncture treatments in the German statutory health insurance system. Forsch Komplementmed. 2015;22(2):118–23.

    PubMed  Google Scholar 

  7. Jacobsen R, Fønnebø VM, Foss N, Kristoffersen AE. Use of complementary and alternative medicine within Norwegian hospitals. BMC Complement Altern Med. 2015;15(1):275.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Mulrow CD. Rationale for systematic reviews. Bmj. 1994;309(6954):597–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Liu Y, Zhang R, Huang J, Zhao X, Liu D, Sun W, et al. Reporting quality of systematic reviews/Meta-analyses of acupuncture. PLoS One. 2014;9(11):e113172.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  11. The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions. London, United Kingdom: The Cochrane Collaboration. Accessed Jan 4 2021. Available from: https://training.cochrane.org/handbook.

  12. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358(3):252–60.

    Article  CAS  PubMed  Google Scholar 

  15. Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Zhang J, Shang H, Gao X, Ernst E. Acupuncture-related adverse events: a systematic review of the Chinese literature. Bull World Health Organ. 2010;88(12):915–21c.

    Article  PubMed  PubMed Central  Google Scholar 

  17. World Health Organization. Acupuncture: review and analysis of reports on controlled clinical trials. Geneva: World Health Organization; 2003.

    Google Scholar 

  18. Chan MWC, Wu XY, Wu JCY, Wong SYS, Chung VCH. Safety of acupuncture: overview of systematic reviews. Sci Rep. 2017;7(1):3369.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  19. Lau CH, Wu X, Chung VC, Liu X, Hui EP, Cramer H, et al. Acupuncture and related therapies for symptom Management in Palliative Cancer Care: systematic review and Meta-analysis. Medicine (Baltimore). 2016;95(9):e2901.

    Article  Google Scholar 

  20. McMaster Health Information Research Unit. Search Filters for MEDLINE in Ovid Syntax and the PubMed translation. Hamilton, Ontario: McMaster Health Information Research Unit. Accessed Jan 4 2021. Available from: https://hiru.mcmaster.ca/hiru/HIRU_Hedges_MEDLINE_Strategies.aspx.

  21. McMaster Health Information Research Unit. Search Strategies for EMBASE in Ovid Syntax. Hamilton, Ontario: McMaster Health Information Research Unit. Accessed Jan 4 2021. Available from: https://hiru.mcmaster.ca/hiru/HIRU_Hedges_EMBASE_Strategies.aspx.

  22. Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev. 2017;6(1):231.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Wu XY, Lam VCK, Yu YF, Ho RST, Feng Y, Wong CHL, et al. Epidemiological characteristics and methodological quality of meta-analyses on diabetes mellitus treatment: a systematic review. Eur J Endocrinol. 2016;175(5):353.

    Article  CAS  PubMed  Google Scholar 

  24. Wu XY, Du XJ, Ho RS, Lee CC, Yip BH, Wong MC, et al. Characteristics and methodological quality of Meta-analyses on hypertension treatments-a cross-sectional study. J Clin Hypertens (Greenwich). 2017;19(2):137–42.

    Article  Google Scholar 

  25. Ho RST, Wu X, Yuan J, Liu S, Lai X, Wong SYS, et al. Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (assessing the methodological quality of systematic reviews) tool. NPJ Prim Care Respir Med. 2015;25(1):14102.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Tsoi AKN, Ho LTF, Wu IXY, Wong CHL, Ho RST, Lim JYY, et al. Methodological quality of systematic reviews on treatments for osteoporosis: a cross-sectional study. Bone. 2020;139:115541.

    Article  PubMed  Google Scholar 

  27. Saha S, Saint S, Christakis DA. Impact factor: a valid measure of journal quality? J Med Libr Assoc. 2003;91(1):42–6.

    PubMed  PubMed Central  Google Scholar 

  28. Lei L, Sun Y. Should highly cited items be excluded in impact factor calculation? The effect of review articles on journal impact factor. Scientometrics. 2020;122(3):1697–706.

    Article  Google Scholar 

  29. Wang J, Shapira P. Is there a relationship between research sponsorship and publication impact? An analysis of funding acknowledgments in nanotechnology papers. PLoS One. 2015;10(2):e0117727.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  30. Mackinnon S, Drozdowska BA, Hamilton M, Noel-Storr AH, McShane R, Quinn T. Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies. BMJ Open. 2018;8(3):e020331.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Lorenz RC, Matthias K, Pieper D, Wegewitz U, Morche J, Nocon M, et al. A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. J Clin Epidemiol. 2019;114:133–40.

    Article  PubMed  Google Scholar 

  32. Leclercq V, Beaudart C, Tirelli E, Bruyère O. Psychometric measurements of AMSTAR 2 in a sample of meta-analyses indexed in PsycINFO. J Clin Epidemiol. 2020;119:144–5.

    Article  PubMed  Google Scholar 

  33. Lorenz RC, Matthias K, Pieper D, Wegewitz U, Morche J, Nocon M, et al. AMSTAR 2 overall confidence rating: lacking discriminating capacity or requirement of high methodological quality? J Clin Epidemiol. 2020;119:142–4.

    Article  PubMed  Google Scholar 

  34. The Cochrane Collaboration. Proposing and registering new Cochrane Reviews. London, United Kingdom: The Cochrane Collaboration. Accessed Oct 1 2021. Available from: https://community.cochrane.org/review-production/production-resources/proposing-and-registering-new-cochrane-reviews.

  35. Neimann Rasmussen L, Montgomery P. The prevalence of and factors associated with inclusion of non-English language studies in Campbell systematic reviews: a survey and meta-epidemiological study. Syst Rev. 2018;7(1):129.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Wu IXY, Deng Y, Wang H, Chen Y, Wong CHL, Chung VCH. Methodological quality of systematic reviews and Meta-analysis on asthma treatments. A cross-sectional study. Ann Am Thorac Soc. 2020;17(8):949–57.

    Article  PubMed  Google Scholar 

  37. Wu IX, Wang H, Zhu L, Chen Y, Wong CH, Mao C, et al. Methodological quality of systematic reviews on interventions for osteoarthritis: a cross-sectional study. Ther Adv Musculoskelet Dis. 2020;12:1759720x20959967.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA. 2002;287(21):2831–4.

    Article  PubMed  Google Scholar 

  39. The PLoS Medicine Editors. Best practice in systematic reviews: the importance of protocols and registration. PLoS Med. 2011;8(2):e1001009.

    Article  PubMed Central  Google Scholar 

  40. Centre for Reviews and Dissemination - University of York. About PROSPERO York, United Kingdom: Centre for Reviews and Dissemination - University of York. Accessed Jan 4 2021. Available from: https://www.crd.york.ac.uk/prospero/#aboutpage.

  41. Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012;344:d7202.

    Article  PubMed  Google Scholar 

  42. Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2014;2014(4):Cd008965.

    PubMed Central  Google Scholar 

  43. McAuley L, Pham B, Tugwell P, Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000;356(9237):1228–31.

    Article  CAS  PubMed  Google Scholar 

  44. Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, et al. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int J Technol Assess Health Care. 2012;28(2):138–44.

    Article  PubMed  Google Scholar 

  45. Wu XY, Tang JL, Mao C, Yuan JQ, Qin Y, Chung VC. Systematic Reviews and Meta-Analyses of Traditional Chinese Medicine Must Search Chinese Databases to Reduce Language Bias. Evid Based Complement Alternat Med. 2013;2013:812179.

    PubMed  PubMed Central  Google Scholar 

  46. Faggion CM. Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. BMC Med Res Methodol. 2015;15(1):63.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Wegewitz U, Weikert B, Fishta A, Jacobs A, Pieper D. Resuming the discussion of AMSTAR: what can (should) be made better? BMC Med Res Methodol. 2016;16(1):111.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Thompson SG. Why sources of heterogeneity in meta-analysis should be investigated. BMJ. 1994;309(6965):1351–5.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods. 2010;1(2):97–111.

    Article  PubMed  Google Scholar 

  50. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Dicenso A, Bayley L, Haynes RB. Accessing pre-appraised evidence: fine-tuning the 5S model into a 6S model. Evid Based Nurs. 2009;12(4):99–101.

    Article  PubMed  Google Scholar 

  52. Birch S, Lee MS, Alraek T, Kim TH. Overview of treatment guidelines and clinical practical guidelines that recommend the use of acupuncture: a bibliometric analysis. J Altern Complement Med. 2018;24(8):752–69.

    Article  PubMed  Google Scholar 

  53. Guo Y, Zhao H, Wang F, Li S-n, Sun Y-x, Han M-j, et al. Recommendations for acupuncture in clinical practice guidelines of the national guideline clearinghouse. Chin J Integr Med. 2017;23(11):864–70.

    Article  PubMed  Google Scholar 

  54. The State Council of the People's Republic of China. Opinions of the CPC Central Committee and the State Council on Promoting the Preservation, Innovation, and Development of Traditional Chinese Medicine (Chinese). China: The State Council of the People's Republic of China. Accessed Jan 4 2021. Available from: http://www.gov.cn/zhengce/2019-10/26/content_5445336.htm.

  55. McGowan BS, Reed JB, Yatcilla JK. Graduate student confidence following a for-credit systematic review course pilot. J Med Libr Assoc. 2021;109(2):323–9.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Tsujimoto H, Kataoka Y, Sato Y, Banno M, Tsujino-Tsujimoto E, Sumi Y, et al. A model six-month workshop for developing systematic review protocols at teaching hospitals: action research and scholarly productivity. BMC Med Educ. 2021;21(1):98.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Li T, Saldanha IJ, Vedula SS, Yu T, Rosman L, Twose C, et al. Learning by doing-teaching systematic review methods in 8 weeks. Res Synth Methods. 2014;5(3):254–63.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

This work was supported by the National Natural Science Foundation of China (81973709) and the Hunan Nature Science Foundation (2019JJ40348).

Author information

Author notes

  1. Leonard Ho and Fiona Y. T. Ke contributed equally to this work.

Authors and Affiliations

  1. School of Chinese Medicine, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong

    Leonard Ho & Vincent C. H. Chung

  2. The Jockey Club School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong

    Fiona Y. T. Ke, Charlene H. L. Wong, Andy K. L. Cheung & Vincent C. H. Chung

  3. Xiangya School of Public Health, Central South University, 5/F, 238 Shang-Ma-Yuan-Ling Alley, Kai-Fu District, Changsha, Hunan, China

    Irene X. Y. Wu

  4. Department of Epidemiology, School of Public Health, Southern Medical University, Guangzhou, China

    Chen Mao

Authors
  1. Leonard Ho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fiona Y. T. Ke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charlene H. L. Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Irene X. Y. Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andy K. L. Cheung
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chen Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Vincent C. H. Chung
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LH: Collection and assembly of data; Data analysis and interpretation; Manuscript writing. FYTK: Collection and assembly of data; Data analysis and interpretation. CHLW: Data analysis and interpretation; Manuscript writing. IXYW: Conception and design; Resources; Funding acquisition. AKLC: Data analysis and interpretation. MC: Conception and design. VCHC: Conception and design; Manuscript writing; Supervision. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Irene X. Y. Wu.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: eTable 1.

Syntaxes for literature search. eTable 2. 17-item bibliographical characteristics questionnaire. eTable 3. List of included systematic reviews. eTable 4. List of excluded systematic reviews with justifications.

Additional file 2.

Additional file 3:

STROBE Statement—Checklist of items that should be included in reports of cross-sectional studies.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ho, L., Ke, F.Y.T., Wong, C.H.L. et al. Low methodological quality of systematic reviews on acupuncture: a cross-sectional study. BMC Med Res Methodol 21, 237 (2021). https://0-doi-org.brum.beds.ac.uk/10.1186/s12874-021-01437-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12874-021-01437-0

Keywords

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us