Skip to main content

Table 1 Outline of the Target Trial Framework for Natural Experiment Studies

From: Conceptualising natural and quasi experiments in public health

Protocol Component [29]

Theorising the causal contrast*

Strengthening causal claims*

Eligibility Criteria

• Does the study include a precise and detailed description of the population who have/will feasibly be exposed to the intervention, with special focus on the boundaries of the intervention which may be fuzzy and/or may not overlap with boundaries of (routine) data collection or risk of the outcome?

• Is a definition and description of the eligibility of potential control populations to ensure independence and exclude spill-over effects included? [30]

• Are potential issues of collider bias [31] or other forms of selection bias considered?

• Consider broadening out the eligibility criteria for multiple control groups that differ in some consequential way [14]; to include, for example, comparable groups or areas from other geographical locations for sensitivity analyses.

Treatment strategies

• Are the intervention, the dose and treatment regimes, and what it aims to affect, including when and where it is introduced defined?

• Has the baseline timepoint been defined?

• Has the control condition (including the potential for reactions even if intervention was not received) in the post-intervention period been defined, and/or has the counterfactual been defined?

• Does the study describe the plausibility of the Stable Unit Treatment Value Assumption (SUTVA)? [32]

• Consider the possibility of pre-implementation changes resulting from anticipating the intervention (for example changes in behaviour or reactions from industry [33]).

• Consider additional other, likely earlier, baseline timepoints to exclude anticipation behaviour in sensitivity analyses.

Assignment procedures

• Given that the assignment procedure of the intervention is not controlled by the researcher, has the assignment rationale and procedures been reported in detail?

Note that the intervention group can also be the whole population (e.g. if exposed to the intervention at a well-defined timepoint). Further note that, in the absence of a suitable control population defined by a temporal or spatial boundary, that the control group can be a synthetic counterfactual

• Has the plausibility of as-if randomization of the assignment been discussed?

• Has conditional exchangeability been formally evaluated for observed factors? Note that this cannot be done for unobserved factors and requires knowledge about exposure allocation procedures.

• Has the parallel trends assumption been assessed prior to the intervention implementation (when analysis based on timeseries data)?

• Has the plausibility of intervention and control groups remaining in their allocation group throughout the study been discussed?

• Consider whether partial control of assignment of intervention is possible.

• Consider the selection of controls that are geographically locally to the intervention units

• Consider selection of intact control groups that are matched to intervention units based on pre-intervention measures of the outcome

• Consider control groups for whom measurement of the exposure, outcome, and covariates is performed similarly to that for the intervention group [6].

• Consider inclusion of (additional) control groups or use of synthetic counterfactuals to improve assessment of conditional exchangeability for observed and unobserved factors [14].

• Consider the inclusion of additional controls hypothesized to not be affected by the intervention (negative controls)

Follow-up period

• Has the follow-up period, which starts prior to assignment of intervention to groups, includes assignment, and ends after a priori defined period post-intervention, been described?

• Consider different follow-up periods to assess evidence of pulse impacts (short-term temporal effect followed by regression to the mean)

Outcome(s)

• Does the study describe the outcome (or outcomes) of interest in detail, and does the description include a priori hypothesized individual-level or population-level parameters at a priori defined period post-intervention or cumulative/average outcomes from start of intervention until a priori defined period post-intervention?

Consider evaluation of additional outcomes:

• also hypothesised to be affected by intervention (positive control)

• hypothesised to be unaffected by intervention (negative control)

Causal contrasts of interest

• Has the causal contrast, or contrasts, to be evaluated been precisely defined?

• Has the causal contrast of interest been specified as an ‘average-treatment-effect’ (ATE) for the population, or as ‘average-treatment-effect-treated’ (ATT) for self-selected interventions? [34]

• Consider, and report, whether Natural Experiment Study enables the estimation of intention-to-treat effects and/or per-protocol effects (although in natural experiments the latter may be rarely available)

• Consider additional causal contrasts, for example in subgroups

Analysis plan

• Is there a pre-specified analytic plan?

• Is the measure of the result specified as a relative and/or absolute measure?

• Is the measure of the result specified as the difference between post-intervention minus pre-intervention outcome of interest in intervention group and post-intervention minus pre-intervention outcome of interest in control group?

• Has the statistical methodology used to calculate the impact or effect of the event or intervention been described in sufficient detail to allow replication?

• Consider the inclusion of temporal falsification analyses by choosing different, randomly assigned, implementation times for the intervention

• Consider the inclusion of spatial falsification analyses using different combinations of units, irrespective of true assignments

• Consider improving causal claims by methodological triangulation using different statistical methods [35, 36].

  1. *: Sources [1, 4, 37, 38] (unless otherwise indicated)