10. Measures

What to write

1. Measures chosen for studying processes and outcomes of the intervention(s), including rationale for choosing them, their operational definitions, and their validity and reliability

2. Description of the approach to the ongoing assessment of contextual elements that contributed to the success, failure, efficiency, and cost of the improvement

3. Methods employed for assessing completeness and accuracy of data

Explanation

Studies of healthcare improvement should document both planned and actual changes to the structure and/or process of care, and the resulting intended and/or unintended (desired or undesired) changes in the outcome(s) of interest.¹ While measurement is inherently reductionistic, those evaluating the work can provide a rich view by combining multiple perspectives through measures of clinical, functional, experiential, and cost outcome dimensions.²

Measures may be routinely used to assess healthcare processes or designed specifically to characterise the application of the intervention in the clinical process. Either way, evaluators also need to consider the influence of contextual factors on the improvement effort and its outcomes.^{34 5} This can be accomplished through a mixed method design which combines data from quantitative measurement, qualitative interviews and ethnographical observation.⁶ In the study described above, triangulation of complementary data sources offers a rich picture of the phenomena under study, and strengthens confidence in the inferences drawn.

The choice of measures and type of data used will depend on the particular nature of the initiative under study, on data availability, feasibility considerations and resource constraints. The trustworthiness of the study will benefit from insightful reporting of the choice of measures and the rationale for choosing them. For example, in assessing ‘staff morale, attitudes, and aspects of ’culture’ that might be affected’ by the SPI1, the evaluators selected the 11 most relevant of the 28 survey questions in the NHS Staff Survey questionnaire and provided references to detailed documentation for that instrument. To assess patient safety, the authors’ approach to reviewing case notes ‘was both explicit (criterion based) and implicit (holistic) because each method identifies a different spectrum of errors’.⁷

Ideally, measures would be perfectly valid, reliable, and employed in research with complete and accurate data. In practice, such perfection is impossible.⁸ Readers will benefit from reports of the methods employed for assessing the completeness and accuracy of data, so they can critically appraise the data and the inferences made from it.

Example

Improvement in culture of safety and ‘transformative’ effects—Before and after surveys of staff attitudes in control and SPI1(the Safer Patients Initiative, phase 1) hospitals were conducted by means of a validated questionnaire to assess staff morale, attitudes, and aspects of culture (the NHS National Staff Survey)…

Impact on processes of clinical care—To identify any improvements, we measured error rates in control and SPI1 hospitals by means of explicit (criterion based) and separate holistic reviews of case notes. The study group comprised patients aged 65 or over who had been admitted with acute respiratory disease: this is a high risk group to whom many evidence based guidelines apply and hence where significant effects were plausible.

Improving outcomes of care—We reviewed case notes to identify adverse events and mortality and assessed any improvement in patients’ experiences by using a validated measure of patients’ satisfaction (the NHS patient survey)…

To control for any learning or fatigue effects, or both, in reviewers, case notes were scrambled to ensure that they were not reviewed entirely in series. Agreement on prescribing error between observers was evaluated by assigning one in 10 sets of case notes to both reviewers, who assessed cases in batches, blinded to each other’s assessments, but compared and discussed results after each batch.⁷

Training

The UK EQUATOR Centre runs training on how to write using reporting guidelines.

Discuss this item

Visit this items’ discussion page to ask questions and give feedback.

References

1.

DONABEDIAN A. Evaluating the quality of medical care. The Milbank Quarterly. 2005;83(4):691-729. doi:10.1111/j.1468-0009.2005.00397.x

2.

Provost l , murray, s . The health care data guide: Learning from data for improvement. 1st edn. San francisco, CA: Jossey-bass, 2011.

3.

Harvey G, Jas P, Walshe K. Analysing organisational context: Case studies on the contribution of absorptive capacity theory to understanding inter-organisational variation in performance improvement. BMJ Quality & Safety. 2014;24(1):48-55. doi:10.1136/bmjqs-2014-002928

4.

Pettigrew a WRMcfcs. Managing change for competitive success. Cambridge, MA: blackwell.

5.

Pawson RaNT . Realistic evaluation. London: Sage, 1997.

6.

Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs—principles and practices. Health Services Research. 2013;48(6pt2):2134-2156. doi:10.1111/1475-6773.12117

7.

Benning A, Ghaleb M, Suokas A, et al. Large scale organisational intervention to improve patient safety in four UK hospitals: Mixed method evaluation. BMJ. 2011;342(feb03 1):d195-d195. doi:10.1136/bmj.d195

8.

Robson c 2011. Real world research: A resource for users of social research methods in applied settings. 3rd edn. Chichester: Wiley, 2011.