What to write
Approach chosen for assessing the impact of the intervention(s)
Approach used to establish whether the observed outcomes were due to the intervention(s)
Explanation
Broadly, the study of the intervention is the reflection upon the work that was done, its effects on the systems and people involved, and an assessment of the internal and external validity of the intervention. Addressing this item will be greatly facilitated by the presence of a strong rationale, because when authors are clear about why they thought an intervention should work, the path to assessing the what, when, why and how of success or failure becomes easier.
The study of the intervention may at least partly (but not only) be accomplished through the study design used. For example, a stepped wedge design or comparison control group can be used to study the effects of the intervention. Other examples of ways to study the intervention include, but are not limited to, stakeholder satisfaction surveys around the intervention, focus groups or interviews with involved personnel, evaluations of the fidelity of implementation of an intervention, or estimation of unintended effects through specific analyses. . The aims and methods for this portion of the work should be clearly specified. The authors should indicate whether these evaluative techniques were performed by the authors themselves, or an outside team, and what the relationship was between the authors and the evaluators. The timing of the ‘study of the intervention’ activities relative to the intervention should be indicated.
In the first example,1 the cross-functional team study, the goal was to improve utilisation of operating room time by having a multidisciplinary, interprofessional group proactively manage the operating room schedule. This project used a prespecified study design to study an intervention, including an intervention and a control group. They assessed whether the observed outcomes were due to the intervention or some other cause (internal validity) by comparing operating room utilisation over time at the intervention site to utilisation at the control site. They understood the possible confounding effects of system-wide changes to operating room policies, and planned their analysis to account for this by using a quasi-experimental time series design. The authors used statistical results to determine the validity of their findings, suggesting that the decrease in variation in use was indicative of organisational learning.
In a subsequent section of this report, the authors also outlined an evaluation they performed to make sure that improved efficiency of operating room was not associated with adverse changes in operative mortality or complication rates. This is an example of how an assessment of unintended impact of the intervention—an important component of studying the intervention—might be completed. An additional way to assess impact in this particular study might have been to obtain information from staff on their impressions of the programme, or to assess how cross-functional teams were implemented at this particular site.
In the second example,2 a programme to improve the transition from paediatric to adult cystic fibrosis care was implemented and evaluated. The authors used a robust theoretical framework to help develop their work in this area, and its presence supported their evaluative design by showing whose feedback would be needed in order to determine success: healthcare providers, patients and their families. In this paper, the development of the intervention incorporated the principle of studying it through PDSA cycles, which were briefly reported to give the reader a sense of the validity of the intervention. Outcomes of the intervention were assessed by testing how patients’ physical parameters changed over time before and after the intervention. To test whether these changes were likely to be related to the implementation of the new transition programme, patients and families were asked to complete a survey, which demonstrated the overall utility of the intervention to the target audience of families and patients. The survey also helped support the assertion that the intervention was the reason patient outcomes improved by testing whether people actually used the intervention materials as intended.
Examples
Example 1
The nonparametric Wilcoxon-Mann-Whitney test was used to determine differences in OR use among Radboud UMC (University Medical Centre) and the six control UMCs together as a group. To measure the influence of the implementation of new regulations about cross functional teams in May 2012 in Radboud UMC, a (quasi-experimental) time-series design was applied and multiple time periods before and after this intervention were evaluated.1
Example 2
To measure the perceptions of the intervention on patients and families and its effect on transition outcomes, a survey was administered in the paediatric cystic fibrosis clinic at the start of the quality improvement intervention and 18 months after the rollout process. The survey included closed questions on demographics and the transition materials (usefulness of guide and notebook, actual use of notebook and guide, which specific notebook components were used in clinic and at home). We also elicited open-ended feedback…..
A retrospective chart review assessed the ways patients transferred from the paediatric to adult clinic before and after the transition programme started. In addition, we evaluated differences in BMI (body mass index) and hospitalizations 1 year after transfer to the adult centre.2
Training
The UK EQUATOR Centre runs training on how to write using reporting guidelines.
Discuss this item
Visit this items’ discussion page to ask questions and give feedback.
