What to write
Trial limitations, addressing sources of potential bias, imprecision, generalisability, and, if relevant, multiplicity of analyses
Examples
“The preponderance of male patients (85%) is a limitation of our study . . . We used bare-metal stents, since drug-eluting stents were not available until late during accrual. Although the latter factor may be perceived as a limitation, published data indicate no benefit (either short-term or long-term) with respect to death and myocardial infarction in patients with stable coronary artery disease who receive drug-eluting stents, as compared with those who receive bare-metal stents.”1
“Our study had several limitations. The early changes to the protocol to accommodate patients with a shorter injury history (but still not acute) to improve recruitment altered the characteristics of the study population. Overall, patients had less long-standing injury than was originally planned. Moreover, the study addressed a deliberately specific population of patients who continued to have ACL injury-related symptoms of instability and had not undergone any previous formal treatment. Another potential limitation is the proportion of patients who did not undergo surgical reconstruction, despite allocation to that group. The true benefit of surgical reconstruction could be somewhat greater than the ITT analysis suggests. The 18-month follow-up period ideally could have been longer but was constrained by various factors including funding. Notwithstanding, most patients had established their level of instability at this timepoint since being included in the trial. The trial design and analysis accounted for delayed surgery in both groups.”2
Explanation
An essential part of a good discussion section is summarising the limitations of the trial. Limitations are frequently omitted from research reports3; identification and discussion of the weaknesses of a study have particular importance.
Some journals have attempted to remedy this problem by encouraging more structure to authors’ discussion of their results.4,5 For example, The BMJ recommends that authors structure the discussion section by presenting (1) a statement of the principal findings; (2) the strengths and weaknesses of the study; (3) the strengths and weaknesses in relation to other studies, discussing important differences in results; (4) the meaning of the study, its possible explanations and implications for clinicians and policymakers; and (5) any unanswered questions and future research.6 We recommend that authors follow these sensible suggestions, perhaps also using suitable subheadings in the discussion section.
Authors should also discuss any imprecision of the results. Imprecision may arise in connection with several aspects of a study, including measurement of a primary outcome (item 14) or diagnosis (item 12a). Perhaps the scale used was validated on an adult population but used in a paediatric one, or the assessor was not trained in how to administer the instrument.
The difference between statistical significance and clinical importance should always be borne in mind. Authors should particularly avoid the common error of interpreting a non-significant result as indicating equivalence of interventions. The CI (item 26) provides valuable insight into whether the trial result is compatible with a clinically important effect, regardless of the P value.7
Authors should exercise special care when evaluating the results of trials with multiple comparisons. Such multiplicity arises from several interventions, outcome measures, time points, subgroup analyses, and other factors. In such circumstances, some statistically significant findings are likely to result from chance alone.
Authors should also consider the extent to which the results of a study can be generalised to other circumstances; also known as external validity. For example, can the results be generalised to an individual participant or groups that differ from those enrolled in the trial with regard to age, sex, severity of disease, and comorbid conditions? Are the results applicable to other drugs within a class of similar drugs, to a different dose, timing, and route of administration? Can similar results be expected in different healthcare settings? What about the effect on related outcomes that were not assessed in the trial, and the importance of length of follow-up and duration of treatment, especially with respect to harms?8 Internal validity, the extent to which the design and conduct of the trial eliminates the possibility of bias, is a prerequisite for external validity: the results of a flawed trial are invalid and the question of its external validity becomes irrelevant. External validity is a matter of judgement and depends on the characteristics of the participants included in the trial, the trial setting, the treatment regimens tested, and the outcomes assessed.
Training
The UK EQUATOR Centre runs training on how to write using reporting guidelines.
Discuss this item
Visit this items’ discussion page to ask questions and give feedback.
