Evaluation of the reporting quality of observational studies in master of public health dissertations in China

Abstract

Backgrounds

Master of public health (MPH) plays an important role in Chinese medical education, and the dissertations is an important part of MPH education. In MPH dissertations, most are observational studies. Compared with randomized controlled trial (RCT), observational studies are more prone to information bias. So, the reporting of the observational studies should be transparent and standard. But, no research on evaluating the reporting quality of the MPH dissertation has been found.

Methods

A systematic literature search was performed in the Wanfang database from January 1, 2014 to May 31, 2019. The Strengthening the Reporting of Observation Studies in Epidemiology (STROBE) statement was adopted to evaluate the reporting quality of the selected studies. Articles that met the following criteria were selected: (1) observational studies, including cross-sectional studies, case-control studies, and cohort studies; (2) original articles; (3) studies on humans, including both adults and children.

Results

The Median of compliance to individual STROBE items was 74.79%. The mean (standard deviation) of STROBE score was 14.29 (1.84). Five items/sub-items were 100% reported (“reported” and “partly reported” were combined): background, objectives, study design, report numbers of individuals at each stage, and key result. Fifteen items/sub-items were reported by 75% or more. Reporting of methods and results was often omitted: missing data (6.67%), sensitivity analyses (3.63%), flow diagram (15.15%), and absolute risk (0%). Logistic regression analysis indicated that cohort studies (OR = 3.41, 95% CI = 1.27–9.16), funding support (OR = 4.37, 95% CI = 1.27–9.16) and more published papers during postgraduate period (OR = 3.46, 95% CI = 1.40–8.60) were related to high reporting quality.

Conclusion

In short, the reporting quality of observational studies in MPH’s dissertations in China is suboptimal. However, it’s necessary to improve the reporting of method and results sections. We recommend that authors should be stricter to adhere STROBE statement when conducting observational studies.

Background

Public health in the twenty-first century faces problems that are very different from those in previous centuries [1]. With the economic development in China, an increasing number of public health problems are appearing, which poses a serious challenge to public health practitioners [2, 3]. As of 2014, the total number of professional staff in Chinese public health institutions was only 87.5 million, which was well below the target of reaching 95 million in 2015. In addition, only 4.2% of public health professionals had a postgraduate degree [4]. The low quantity and insufficient quality of health professionals have hindered the development of public health services in China. Therefore, it is urgent to optimize training programmes to train more highly educated, application-oriented public health personnel. To fulfill this requirement, the Ministry of Education of China launched the full-time master of public health (MPH) postgraduate programme in 2009 [5]. Although the dissertation is an important part of MPH education, studies on the quality of these dissertations are still limited [6].

The randomized controlled trial (RCT) has been advocated as the gold standard for evaluating causal effects in medical studies. However, it is difficult to verify many studies by RCTs due to various ethical problems and side effects of intervention in practice [7,8,9]. Well-designed observational studies can not only provide abundant clues for investigating the causal relationship between exposure and diseases but also be more suitable for investigating the long-term and rare side effects of treatment modalities. Moreover, one study showed that approximately 90% of the papers published in medical journals are observational studies [10].

Compared with RCTs, observational studies cannot randomly assign study factors to the participants. They can rely only on comprehensive, objective descriptions or well-designed programmes to analyse, compare and summarize population phenomena and further explore the causal relationships between disease and exposure factors. Hence, the reporting of observational studies should be transparent and complete. Standardizing the reporting of observational studies can not only help editors and reviewers of medical journals to better understand the study designs but also provide important information for readers in related fields so that they can clearly understand the content and results of the research and improve their professional skills. Several incipient studies on reporting quality recognized deficiencies in medical studies, but all of these were limited by incomplete reporting quality evaluation standards [11,12,13,14].

In 2008, the Strengthening the Reporting of Observation Studies in Epidemiology (STROBE) statement was published to “improve transparency in reporting of observational studies” [15,16,17,18]. The STROBE statement contains a total of 22 items, including evaluations of research design, data collection, analytical techniques, and potential deviations. Since the STROBE statement was published, several studies have evaluated the reporting quality of clinical medicine articles and found that the reporting of observational studies needs to be improved [19,20,21,22,23].

At present, no research has been performed to evaluate the reporting quality of the dissertation for the Chinese MPH degree. Therefore, we used the STROBE statement to evaluate the reporting quality of observational studies in MPH dissertations in China, identify factors associated with high-quality reporting, and provide direction for writing the MPH dissertation.

Methods

Search strategy

We searched the relevant studies in the Wanfang database. The Wanfang database contains mainly Chinese dissertation and journal papers, including all the dissertations of higher education institutions or scientific study institutions that are approved for granting the MPH degree. The language was limited to Chinese, and the search strategy was (theme:(“cohort studies” OR “cohort analyses” OR “case-control studies” OR “case-control analyses” OR “cross-sectional studies” OR “prevalence studies” OR “current situation studies”) *profession:(Master of Public Health) * degree:(master)) * Date:2014–2019.

Study selection

Articles that met the following criteria were selected: (1) observational studies, including cross-sectional studies, case-control studies, and cohort studies; (2) original articles; and (3) studies on humans, including both adults and children. The exclusion criteria were as follows: (1) review articles; (2) case reports; (3) quasi-randomized trials, randomized controlled trials and other interventional studies; and (4) articles for which the database provided only abstracts and not full texts.

The articles retrieved were preliminarily reviewed on the basis of titles and abstracts by two investigators independently. Any disagreement was resolved by consulting a senior author. After the initial screening, the full texts of the relevant research were searched, and the two investigators determined the final content of the literature review based on the inclusion and exclusion criteria.

Data extraction

The extraction of data from the included articles was performed independently by the two investigators. The general information extracted included publication time, type of study, number of papers published during master studies, funding support, and number of statistical methods used. This information was extracted from each article. For dissertations, the publication time and funding support were indicated on the cover page. In China, all graduate students must list the titles of their own published paper at the end of the dissertation, from which we extracted the number of published papers. Published papers were not limited in terms of country, language, and database, but the papers published after graduation are not included in this information. The number of number of statistical methods used was collected by a “common statistical methods in medical studies” checklist (supplement.1).

Quality assessment

On the basis of the detailed item descriptions of the STROBE statement, the reporting appraisal was performed by two investigators. Differences between the two investigators were resolved by discussion with the senior author until all differences were resolved. The STROBE statement contains 22 items: title and abstract (item 1), introduction (items 2 ~ 3), method (items 4 ~ 12), results (item 13 ~ 17), discussion (items 18 ~ 21), and other information (item 22). A score of 1 was assigned to items for which all the detailed information was reported, a score of 0.5 was assigned to items for which the detailed information was partly reported, and a score of 0 was assigned to items for which none of the information was reported. For items with sub-parts, fractional points were assigned depending on the number of sub-items met. Sub-item 6b is applicable only to match studies, and 14c is applicable only to cohort studies. Sub-items were removed from the denominator if they were not applicable [24]. Therefore, every study had an overall STROBE score rated from a maximum score of 22. To simplify the statistical analysis, we combined the “reported” and “partly reported” categories to calculate the “reporting rate” when describing the reporting rate of items.

Data analysis

The continuous data subjected to normal distribution were presented as the mean and standard deviation (SD). Categorical variables were expressed as numbers and percentages. Comparisons of STROBE scores between dichotomous groups were conducted using the independent Student’s t-test. Comparisons of STROBE scores between multiple groups were conducted using one-way analysis of variance (ANOVA) with the LSD-t test. The included articles were further divided into high and low reporting quality groups according to the cut-off value (the 75th percentile of the STROBE score). Univariate logistic regression models were used to analyse the associations between high reporting quality and study type, publication time, papers published during master studies, funding support, and types of statistical methods used. Candidate variables for which P ≤ 0.05 in the univariate logistic regression analyses were included in the multivariate logistic regression model. The odds ratio (OR) and 95% confidence interval (CI) were calculated from the logistic regression analyses. Statistical analyses were performed using SPSS version 18.0. All reported probabilities (P values) were two-sided, and P ≤ 0.05 was considered significant.

Results

Search results

After the search of the database, we confirmed 425 articles without duplication. After the titles and abstracts were screened, 201 articles were excluded. A total of 224 full articles were further reviewed, and 59 additional articles were excluded because 32 articles were review studies and 27 were intervention studies. Finally, 165 relevant articles that met the inclusion criteria were included. Of the 165 articles, 61 articles were cross-sectional studies, 66 articles were case-control studies, and 38 articles were cohort studies (Fig. 1).

Fig. 1

Flow diagram of the literature

The compliance of the included studies with the STROBE statement

The adherence to the 22 items of the included studies is shown in Table 1. The median of compliance with individual STROBE items was 74.79% (range: 0–100%). The overall reporting quality was relatively good. Five items/sub-items (14.71%) were reported by 100% of the studies: background/rationale, objectives, study design, reported numbers of individuals at each stage of the study, and key results; 17 items/sub-items (50.00%) were reported by 75% or more of the studies. Specifically, the reporting on the title and abstract section and the introduction section was satisfactory: reporting rates for all four items exceeded 95%. Reporting on methods and results was often omitted: examine sub-groups and interactions (n = 11, 6.67%), missing data (n = 11, 6.67%), sensitivity analyses (n = 6, 3. 63%), reasons for non-participation (n = 14, 8.48%), flow diagram (n = 25, 15.15%), and absolute risk (n = 0, 0%). The reporting rates of the items were similar in the three study types, but the reporting of the cohort studies was slightly better (Fig. 2).

Table 1 Adherence to the STROBE Reporting Criteria

Fig. 2

The compliance of three study types in each item

STROBE score

The mean of the STROBE score was 14.29 (range: 10.03–18.93) with a standard deviation of 1.84. We found that the STROBE score of the cohort studies was significantly higher than that of the other two types of studies. Dissertations with funding support were more likely to receive high STROBE scores. The STROBE scores of dissertations that listed more papers published during the postgraduate period and more statistical methods was higher than that of others. The mean of the STROBE score of dissertations published in 2018 was higher than that of those published in other years. The characteristics of the 165 included articles are shown in Table 2.

Table 2 The main characteristics of included studies

Univariate and multivariate logistic regression analyses

According to the cut-off value of the STROBE score (15.40), the included articles were divided into low (n = 124) and high reporting quality groups (n = 41). The univariate logistic regression analyses showed that the following factors were related to superior reporting quality: cohort study (OR = 4.08, 95% CI = 1.67–10.00), funding support (OR = 2.71, 95% CI = 1.05–7.02), more papers published during the postgraduate period (OR = 2.52, 95% CI = 1.11–5.73), and number of statistical methods used (OR = 3.33, 95% CI = 1.46–7.56).

The multivariate logistic regression analysis demonstrated that cohort studies (OR = 3.41, 95% CI = 1.27–9.16), funding support (OR = 4.37, 95% CI = 3.52–7.48), and more papers published during the postgraduate period (OR = 3.46, 95% CI = 1.40–8.60) were related to superior reporting quality (Table 3).

Table 3 Univariate and Multivariate Logistic Regression Analyses of Predictive Factors Associated With Superior Reporting Quality

Discussion

Summary of findings

Our study evaluated 165 MPH dissertations. Although the overall reporting quality was relatively good, some essential aspects of methods and results were seldom reported, which makes it difficult for readers to assess the validity and reliability of an observational study [16]. Moreover, dissertations of superior reporting quality usually contained the following predictive factors: cohort study, funding support and more papers published during the postgraduate period.

Reporting on the title and abstract section and the introduction section was satisfactory. The reason may be that each MPH candidate needs to undergo strict opening and midpoint screening stages in the early stage of the dissertation writing. The deficiency of the reporting of MPH dissertations occurred mainly in methods and results. In particular, there was a need for dissertations to improve their reporting of variable definitions, statistical methods, and flow diagrams.

In actual studies, the outcome, exposure, predictors, potential confounders, and effect modifiers of the study should be clearly defined, but less than half of the dissertations fully reported these contents. Inadequate reporting of statistical methods may indicate that the research results are not fully exploited, resulting in a waste of valuable information and varying degrees of bias. However, only a few articles described any methods used to examine sub-groups and interactions, explained how missing data were addressed, and described any sensitivity analysis. Only 25 dissertations (15.15%) used flow diagrams, while others did not take advantage of the simple and direct features of the flow diagram. In addition, all of the articles summarized the key results with reference to the study objectives, but only approximately one-quarter of the articles discussed the generalizability of the study results.

The results of the multivariate logistic regression analysis indicated that funding support was associated with high reporting quality. To receive funding, projects require rigorous research designs and need to be screened and approved. Therefore, masters candidates are strictly required and trained to learn more knowledge to ensure that their thesis quality will be higher. Moreover, a positive association between more papers published during the postgraduate period and high reporting quality was observed. Masters candidates who published more papers during the postgraduate period have stronger academic ability, are more familiar with writing articles, and know what should be reported in detail. In addition, the results of the univariate logistic regression analyses showed that a higher number of statistical methods was associated with high reporting quality. Masters candidates who use more statistical methods have a deeper understanding of methodology, are more proficient in using statistical methods, and tend to be more complete in reporting their methods in their dissertations.

Compared with other studies

A few articles have evaluated the reporting quality of observational studies in other medical disciplines. Several studies have found that the reporting quality of articles that used the STROBE statement for standardization was better than that of others [19, 20, 25, 26]. Jacqueline Ramke et al. used the STROBE statement to evaluate reporting in blindness prevalence surveys and found that the mean of the STROBE score of studies published in journals requiring the STROBE statement was higher than that for others [19]. Swords C’s study indicated that the STROBE statement had increased the reporting quality of observational otology and audiology studies [20]. Hence, we strongly recommend that masters candidates should be familiar with the STROBE statement.

On the other hand, many studies have found defects in reporting the methods and results of observational studies [27,28,29]. Adams AD et al. discovered poor reporting in obstetrics observational studies for study size, missing data, and absolute studies [27]. Karaçam Z evaluated the reporting quality of observational studies in Turkish nursing journals and found that methods sections were mostly omitted [29]. Our research yielded similar results.

Educational implications

Our study has highlighted the important deficiencies in the reporting of observational studies in MPH dissertations. Based on these findings, we believe that if universities adopt the STROBE criteria to guide MPH candidates, it will help improve the reporting quality of MPH dissertations. In the course of master training, it is necessary to strengthen students’ understanding and flexible application of statistical methods, and graduate tutors should pay more attention to masters students who published fewer papers during the postgraduate period.

Strengths and limitations of this study

As a systematic review of MPH dissertations, our study has some advantages. First, it is a comprehensive assessment and used logistic regression analyses to identify factors associated with high-quality reporting. For the evaluation of dissertations, we included not only adherence to STROBE items but also the STROBE score. Second, some of the evaluation items, such as 6b and 14c, are not applicable to all dissertations, and some items are not adequately reported. To minimize biases against systematic review, we identified items as fully reported, partly reported, not reported, and not applicable and assigned the scores accordingly. Thus, different articles have a more consistent score criterion. Third, since the STROBE statement was published in 2007, no studies have used this guideline to evaluate the reporting quality of master dissertations. Therefore, our work is innovative and will provide a reference for subsequent similar research. Fourth, the study includes the independent assessment of all articles by two authors. All details of our search are transparent and clearly stated, and it can therefore easily be reproduced.

There are also some limitations of this study. First, the scoring of items remains a subjective task and easily leads to subjective bias. However, the two investigators independently used the STROBE statement to evaluate the included studies, and differences were resolved by discussion. In this way, we minimized subjective bias. Second, given that our research was restricted to MPH dissertations published by Chinese masters candidates in the past 5 years, the results reflect only the integrity and standardization of the reporting of Chinese MPH dissertations to a certain extent. Finally, since there is no literature to be found on using the STROBE statement to evaluate the reporting quality of medical masters dissertation, it is impossible to compare the reporting quality of these dissertations with that of dissertations in other professions.

Conclusion

In summary, the reporting quality of observational studies in MPH dissertations is sub-optimal. There is a need to improve the reporting of methods and results sections, especially statistical methods reporting. The STROBE statement was intended to help researchers to improve transparency in reporting observational studies. Therefore, we think it is highly plausible that using the STROBE statement will improve the quality of reporting. We recommend that masters candidates who conduct observational studies use the STROBE statement and recommend that research supervisors use this statement to guide MPH candidates.