Incentive delivery timing and follow-up survey completion in a prospective cohort study of injured children: a randomized experiment comparing prepaid and postpaid incentives

Abstract

Background

Retaining participants over time is a frequent challenge in research studies evaluating long-term health outcomes. This study’s objective was to compare the impact of prepaid and postpaid incentives on response to a six-month follow-up survey.

Methods

We conducted an experiment to compare response between participants randomized to receive either prepaid or postpaid cash card incentives within a multisite study of children under 15 years in age who were hospitalized for a serious, severe, or critical injury. Participants were parents or guardians of enrolled children. The primary outcome was survey response. We also examined whether demographic characteristics were associated with response and if incentive timing influenced the relationship between demographic characteristics and response. We evaluated whether incentive timing was associated with the number of calls needed for contact.

Results

The study enrolled 427 children, and parents of 420 children were included in this analysis. Follow-up survey response did not differ according to the assigned treatment arm, with the percentage of parents responding to the survey being 68.1% for the prepaid incentive and 66.7% with the postpaid incentive. Likelihood of response varied by demographics. Spanish-speaking parents and parents with lower income and lower educational attainment were less likely to respond. Parents of Hispanic/Latino children and children with Medicaid insurance were also less likely to respond. We found no relationship between the assigned incentive treatment and the demographics of respondents compared to non-respondents.

Conclusions

Prepaid and postpaid incentives can obtain similar participation in longitudinal pediatric critical care outcomes research. Incentives alone do not ensure retention of all demographic subgroups. Strategies for improving representation of hard-to-reach populations are needed to address health disparities and ensure the generalizability of studies using these results.

Introduction

Prospective studies evaluating health outcomes over time depend on successful completion of follow-up assessments by enrolled participants. Longitudinal studies that rely on recontacting participants to collect additional data require more from participants than one-time cross-sectional studies. Repeated outcome measurement places an additional burden on participants and may contribute to higher levels of non-response [1, 2]. Minimizing non-response at follow-up is especially critical for ensuring continued representation of enrolled participants from hard-to-reach populations. Minimizing participant attrition in longitudinal studies ensures statistical power, demographic representation, and preserves study validity and integrity [3,4,5,6,7,8].

Incentives are frequently used to encourage participants to complete follow-up assessments. Previous studies have evaluated how strategies such as incentives and contact methods affect retention in health and epidemiological research, including cohort studies and randomized trials [9,10,11,12]. Findings from this research suggest that monetary incentives are effective, but nonmonetary incentives are not [9]. Additionally, increasing the dollar amount of incentives has improved retention [10]. Systematic reviews have shown that monetary incentives improve retention in randomized trials [9, 13] and prospective cohort studies [11].

Although types of incentive and incentive amounts have been studied, minimal attention has been given to incentive timing, i.e., whether incentives are postpaid or prepaid, in health outcomes research. Prepaid incentives are commonly used by social scientists and public opinion researchers. Prepaid incentives resulted in higher response rates than the promise of a postpaid incentives in cross-sectional [14,15,16,17] and longitudinal [18,19,20] survey research. It is less clear whether prepaid incentives are similarly effective in health outcomes studies. In prospective clinical research, postpaid incentives are more commonly used. One study that compared the effect of prepaid and postpaid incentives on retention in a randomized trial found inconsistent effects [21]. Few investigators have examined whether prepaid incentives are more effective at retaining cohort participants for follow-up data collection [12]. The existing literature offers no conclusive guidance on whether prepaid incentives improve retention in health outcomes research.

Survey researchers have also examined whether incentive timing influences the demographic composition of respondents. In some cases, prepaid incentives improved demographic representation in cross sectional surveys [22, 23]. In other instances, prepaid incentives skewed the representativeness of participants [24, 25]. Demographic characteristics such as lower socioeconomic status have been associated with higher attrition in prior cohort studies [26]. These studies support the need to evaluate the impact of incentive timing on the demographics of retained participants in prospective health research. Prepaid incentives may also encourage faster response than postpaid incentives, which can shorten the data collection period or reduce the number of contact attempts [27, 28]. Outcomes such as response speed have received less attention than response rates. Factors affecting response speed in longitudinal health outcomes research have not been examined.

The purpose of this study was to determine whether prepaid incentives result in higher retention than postpaid incentives in a longitudinal study of injured children. We conducted a randomized experiment to evaluate the timing of incentive delivery on parents’ response to a six-month follow-up survey. We hypothesized that a prepaid incentive would result in a higher survey response rate compared to the promise of a postpaid incentive. A secondary aim examined whether a prepaid incentive would reduce the required number of contact attempts needed to reach participants. We also assessed whether parent and child demographic characteristics were associated with retention and evaluated whether prepaid incentives aid in the retention of a demographically representative sample.

Materials and methods

Study design and participants

We conducted a parallel, 1:1 randomized trial (“experiment”) nested within a prospective cohort study (“cohort study”) to assess the effect of incentive timing on participant response to a follow-up survey. This experiment was embedded within the “Assessment of Functional Outcomes and Health-Related Quality of Life after Pediatric Trauma Study.” The aim of this study was to identify factors associated with injured children’s functional status at hospital discharge, and the relationship between functional status at discharge with six-month functional status and health-related quality of life. Functional status—the ability to perform activities of daily living—measure six domains: mental status, sensory, communication, motor, feeding, and respiratory function [29]. The cohort study was conducted at seven sites in the United States from March 2018 to February 2020. The Institutional Review Board at the University of Utah approved this study through a central mechanism (Approval #00105435). Additional details about the children’s injuries and the survey measures used in this study have been previously reported [30].

Children under 15 years in age who were treated for a serious, severe, or critical injury to one or more major body regions (head, thorax, abdomen, spine, or extremity) were eligible for enrollment in the cohort study. Patients with major burn injuries were excluded, as were children whose parents or guardians (hereafter referred to as parents) did not speak English or Spanish. Eligible children were enrolled at seven hospital sites, all level 1 pediatric trauma centers within the National Institutes of Health-funded Collaborative Pediatric Critical Care Research Network (CPCCRN). The participant of interest for this experimental study was the parent who provided consent for the child’s participation. Parents received all follow-up communications, were asked to complete the survey, and were assigned to an experimental arm.

Recruitment, follow-up protocol, and randomization

At each hospital (study site), research coordinators reviewed the daily census to identify eligible children. The cohort study sampling strategy was designed to promote equal enrollment of patients with isolated injuries in each body region. We set a goal to enroll 50 patients per study site per year, with 70% comprised of children with one injured body region and 30% children with multiple injured body regions. Every three months, we adjusted enrollment across sites to ensure enrollment in each category. The original goal was to enroll up to 840 patients into the cohort study. Statistical power was calculated to detect a difference in the proportion of parents who completed the survey between experimental arms. We initially planned to perform interim monitoring of experimental results after 210, 420, and 630 participants had completed follow-up. If interim analyses indicated one incentive type to be superior to the other, we planned to stop randomization and proceed with the superior method for subsequently enrolled participants. The cohort study ended early due to funding after enrolling 427 children.

Research coordinators obtained written informed consent and collected baseline data using medical records and standardized questionnaires administered to parents at discharge. Six months after hospital discharge, the parent who signed the consent form was asked to complete a telephone survey about the child’s current functional status and health-related quality of life. The Pediatrics Clinical Trials Office at the University of Utah made all contacts associated with the follow-up survey. These contacts consisted of a reminder letter at three months, a second reminder letter one week before the telephone survey, and a text message reminder one day before the telephone survey. To collect the survey data, at least three telephone call attempts were made on different days and times of day according to preferences parents specified at enrollment. If these attempts were unsuccessful, the study team attempted to reach a designated alternate contact to confirm the parent’s availability and contact information. If these contacts were also unsuccessful, the parent was emailed a link to an abbreviated web version of the survey two weeks after the last call attempt. If the parent did not respond to the web survey within another two weeks, medical records were reviewed to determine if six-month outcomes could be assessed from this source.

The contact protocol and materials for both experimental arms were the same except for the six-month reminder letter, which contained the intervention. This letter was either accompanied by a cash card (prepaid incentive arm) or informed the parent that a cash card would be sent after survey completion (postpaid incentive arm). After enrollment, parents were randomly assigned to one of the two incentive arms. We stratified incentive assignments within each study site and by injury type using a pre-generated randomization sequence that was created by the study biostatistician using statistical software. The sequence was concealed from all study staff except for the central research coordinator overseeing incentive delivery. The randomization sequence was reset daily by IT staff and parents were automatically assigned to a study arm as they enrolled in the study using the REDCap (Research Electronic Data Capture) platform [31]. The study site that enrolled the participant did not know which incentive arm was assigned. Because they participated in the contact protocols and delivery of the incentives, the interviewers who administered the surveys were aware of parents’ incentive assignments.

Intervention, outcomes, and additional variables

The experimental intervention was incentive timing, categorized as a prepaid or postpaid incentive. Parents either (1) received a US$50 cash card in advance of the follow-up survey or (2) were informed that they would receive the cash card after completing the survey. Similar to a debit card, the cash card could be used at any retailer.

The primary outcome of this experiment was six-month survey completion, categorized as completed or not completed. A telephone survey response was classified as complete if sufficient information was obtained for scoring at least four of the five instruments included in the survey. The web survey was considered complete if containing enough information for scoring three out of the four instruments. A survey was classified as not completed if the parent did not respond to any of the survey requests or did not complete enough items to reach the completion thresholds. The secondary outcome was the number of call attempts needed to reach participants, as recorded by the interviewers making the telephone calls.

Another secondary aim was to assess whether demographic characteristics were associated with the primary outcome of survey completion. Available demographic variables included parental educational attainment and the parent’s preferred language (English or Spanish), household income and household size, and the race, ethnicity, age, sex, and insurance status of the child. These variables were obtained from a baseline survey.

Statistical analysis

We calculated descriptive statistics for patient and parent demographics by each experimental arm and within the entire sample. Categorical measures were summarized with counts and percentages. Continuous measures were summarized using medians and the 25^th and 75^th percentile values to account for non-normal distributions. We calculated the proportion of parents who completed the survey by experimental arm. To test for differences in survey completion by experimental arm, we performed logistic regression models predicting survey completion, controlling for incentive assignment and study site. To test for demographic differences in survey completion, we performed logistic regression models predicting survey completion, controlling for the demographic variable and study site. In a supplemental analysis, we assessed the effect of the interaction of each demographic variable with incentive type on likelihood of survey response. Significance was defined at p<0.05. Analyses were performed using SAS version 9.4 (Cary, NC).

Results

Across all study sites, 835 children were assessed for eligibility in the cohort study, 654 met inclusion criteria, 493 were approached for consent, and 428 were consented to participate (See Flow Diagram, Additional file 1). One patient was excluded due to the absence of a qualifying injury, resulting in a final sample of 427 children with their consenting parents. To evaluate the effect of incentive timing, we limited the analyses to the 420 parents who were randomly assigned to an experimental arm. The 420 randomized parents were analyzed according to their originally assigned treatment arm (prepaid incentive n=204, postpaid incentive n=216).

Most parents reported a household size of three or four individuals (54.5%; Table 1). The largest share of parents had a high school education or less (35.7%) and 15.7% reported an annual household income of less than $15,000. Only 3.6% of parents were surveyed in Spanish. The median age of the injured children was 7.2, 36.9% were female, 11.2% were Hispanic, and 64.8% were white. A summary of the children’s injury characteristics is included in Additional file 2.

Table 1 Demographic characteristics of children in the study sample, overall and by experimental arm

We obtained survey responses from 67.4% (283/420) of the parents. Survey completion did not differ based on incentive timing. The response rate for the prepaid incentive was 68.1%, and 66.7% for the postpaid incentive (p=0.61, Table 2). A median of two telephone calls was associated with successful contact with the parent. The number of telephone calls needed to reach parents also did not differ based on when the incentive was provided (p=0.22). Regardless of the incentive offered, the highest percentage of parents were reached on the first call (39.1% for the prepaid incentive, 49.6% for the postpaid incentive).

Table 2 The effect of incentive timing on six-month follow-up survey completion outcomes in a prospective cohort study

Survey response varied by demographic characteristics (Table 3). Hispanic or Latino children comprised 6.4% of respondents and 21.2% of non-respondents (p<0.001). Spanish-speaking parents were less likely than English speakers to complete the survey (0.4% of respondents compared to 10.2% of non-respondents; p<0.001). Children with Medicaid were less likely to be represented among the responses (37.8% of respondents compared to 63.5% of non-respondents; p<0.001). Children whose parents had a high school diploma or less education were under-represented among respondents (27.6%) compared to non-respondents (52.6%; p<0.001). Fewer respondents than non-respondents were from families earning less than $15,000 per year (11.7% vs. 24.1%; p<0.001). Parents reporting household sizes of five or more individuals were underrepresented among respondents (34.6% of respondents compared to 48.2% of non-respondents; p<0.013). The interaction of each demographic variable with incentive assignment did not predict on survey completion (Additional file 3). Completion patterns by demographics were similar regardless of incentive timing.

Table 3 Follow-up survey completion by child and parent demographic characteristics

Discussion

Collecting long-term health outcomes data after discharge is time-consuming and costly. The resources and optimal approaches for conducting longitudinal follow-up are not well-defined. In this study, we evaluated whether prepaid incentives retained more parents than postpaid incentives in a study of injured children. Contrary to our hypothesis, incentive timing did not influence the likelihood of survey completion. This result departs from research showing that prepaid incentives were more effective than postpaid incentives for improving cross-sectional survey completion [15,16,17] and longitudinal survey retention [18,19,20].

Several explanations may account for our results. Evidence in support of prepaid incentives comes primarily from cross-sectional survey research [15, 17]. Incentives are used in longitudinal surveys, but less is known about the optimal use of incentives to reduce attrition in these studies [19]. Even fewer studies have addressed how incentive timing affects retention in prospective cohort studies. Incentive timing may not predict retention in longitudinal health outcomes research. Systematic reviews show that retention in health studies improves as more response-inducing strategies are incorporated [9, 11]. Strategies other than incentives include multiple reminders, varied modes of contact, and sending a second copy of a paper questionnaire to non-respondents.

The effect of incentive timing on retention may also vary based on the subject matter of a survey or across different target populations. A survey’s subject matter predicts cross-sectional survey participation [15]. Survey response is higher when questions are interesting to the participants. In the current study, we asked parents about their children’s functional status and quality of life after injury. This subject matter is relevant to parents because of the family burden associated with a child’s long-term impairment [32]. The relevance of the content may have affected survey completion more than incentive timing. In a similar study, parents of injured children expressed gratitude for follow-up calls evaluating their child’s status [33], supporting interest in this subject.

The cash card we used as an incentive could also account for our results. The effect of prepaid incentives can depend on the currency offered. A prepaid cash incentive retained more participants than a prepaid gift card in a longitudinal survey of recent high school graduates [34]. A prepaid cash card produced a lower response rate than a prepaid check in a survey of physicians [35]. The incentive dollar amount is also relevant to the timing of delivery. Small, prepaid cash incentives were associated with more responses than larger, postpaid incentives [36]. A smaller prepaid incentive could have produced different results in our study. The magnitude of an incentive’s effect on survey response also depends on the survey mode. Many studies that obtained higher response with prepaid incentives used mailed paper surveys [15, 16]. The effect of prepaid incentives on response in telephone surveys has been smaller [17].

We also assessed whether incentive timing led to differences in the demographic composition of respondents and non-respondents. Prepaid incentives improved demographic representation in some cross-sectional studies [22, 23], but decreased representation in others [24, 25]. We found no demographic differences between the respondents and non-respondents based on incentive timing. The association of incentive timing with responses and demographic representation in survey research is an evolving area of investigation [17].

We observed that some demographic characteristics were associated with follow-up completion, including child ethnicity and insurance status, household income, and parental education. Others have noted the challenge researchers have in recruiting participants from all racial and ethnic groups and diverse socioeconomic backgrounds [37]. Methods to improve participation among underrepresented populations must be tailored to address the multiple barriers faced when participating in research [38]. Prior research suggests barriers to participation include mistrust of medical research, language barriers, and demands and inconveniences of participation [38, 39]. Community-based, tailored, and personalized recruitment efforts may facilitate continued engagement with underrepresented populations [39,40,41]. More research is needed to identify specific strategies that ensure demographic representation in health outcomes research and ensure the generalizability of the findings of this research [42].

We anticipated that a prepaid incentive would encourage parents to answer the study’s telephone calls and reduce the need for multiple call attempts. Prepaid incentives can reduce the level of effort required to obtain follow-up responses [27, 28]. In this study, the number of calls research coordinators placed did not vary by incentive timing. We contacted most parents on the first or second attempt regardless of incentive type. Contact at the initial attempts may be related to parents’ interest in the study or the reminder letter’s impact.

Unlike prior incentive timing experiments, the respondents in our study were proxies providing information about the enrolled children. Parent-proxy reporting of children’s quality of life or other health outcomes is frequently used in pediatrics [43]. Compared to other medical specialties, pediatrics is more family-oriented, and parents play a larger role in healthcare decision-making [44]. These unique circumstances may require different retention methods than studies that use direct reports from the participant.

It is difficult to retain participants in prospective cohort studies [45]. This experiment provides guidance for designing future longitudinal studies of critically ill and injured children. Our results show that parents in longitudinal pediatric critical care studies can be retained with either a prepaid or postpaid incentive. Postpaid incentives are more commonly used in health outcomes research. In our experience, postpaid incentives are less likely to be restricted by organizational accounting policies. Postpaid incentives may be more suitable when these restrictions exist. Prepaid incentives must be carefully considered because of cost. Prepaid cash incentives are less expensive when offered in smaller amounts than postpaid incentives. Recipients of prepaid incentives delivered as a check are unlikely to cash them if not participating in the study, making them more cost-effective [46]. Prepaid incentives can also establish goodwill and trust [47]. Prepaid incentives do not require additional follow-up contact for delivery. Incentives enhance retention regardless of timing. Our study shows researchers have options for how to incorporate incentives.

This study has several limitations. First, although we obtained several measures of demographic characteristics, additional parental characteristics may account for group differences. The demographic characteristics in our study predicting follow-up completion were similar to those in other cohort studies and longitudinal surveys [48,49,50] suggesting relevant measures were included. Second, this study was conducted with a sample of children treated at level 1 pediatric trauma centers for a serious or greater injury. These results may not generalize to studies of children with less severe injuries or those treated in other care settings. The results may also not apply to adults or patients with other conditions. Our findings should be confirmed in other populations to evaluate generalizability. Third, our experiment did not include a no-incentive condition. Without this baseline for comparison, we could not assess how incentives encouraged survey completion regardless of timing. Because the study was closed to enrollment earlier than anticipated, we were unable to reach our originally targeted sample size.

Conclusions

This study assessed whether a prepaid incentive could improve retention in a longitudinal health outcomes study. Our approach provides a framework to apply and evaluate other response-inducing techniques from survey research in prospective studies. Because incentive timing did not affect retention in this study of injured children, researchers can choose either option for similar studies. Additional investigation is needed to identify methods that improve participation among underrepresented socioeconomic and ethnic subgroups. Without adequate representation, the conclusions drawn from health outcomes research may miss insights that are critical for addressing health disparities.