Case vignettes are so popular in the assessment of CME thanks to a series of three articles published between 2000 and 2004 [1-3]. These articles described three separate comparative analyses between medical charts and case vignettes. In each article, the authors assessed how well medical charts and case vignettes measured physician performance as compared to standardized patients (their “gold standard”). And in each article, case vignettes out-performed medical chart abstraction. Here’s the conclusion from the first article in this series (published in JAMA in 2000):
CONCLUSIONS: Our data indicate that quality of health care can be measured in an outpatient setting by using clinical vignettes. Vignettes appear to be a valid and comprehensive method that directly focuses on the process of care provided in actual clinical practice. Vignettes show promise as an inexpensive case-mix adjusted method for measuring the quality of care provided by a group of physicians.
Armed with this JAMA citation, CME providers have been regularly assessing physician performance changes associated with CME participation via case vignettes. In this setting, the typical case vignette format is a 3-5 sentence patient presentation followed by multiple-choice questions regarding diagnosis, treatment, and maybe even perceived practice barriers. And if you’re using that format, here’s the problem...it doesn’t reflect the format utilized in the research upon which it’s based.
The case vignettes utilized in the literature cited above [1-3] were developed only in clinical areas with strong clinical guideline support (i.e., there was little argument in regard to proper diagnostic or treatment approaches). They were also designed in regard to the sequence of a typical patient visit, requiring physicians to provide open-ended responses to a given case vignette about their approach to patient history, physical exam, tests, diagnosis, and management. Finally, these open-ended responses were assessed by a trained abstractor based on a scoring sheet developed by a panel of physicians.
Unless you’re using this approach, there is no evidence to support that your case vignettes are actually capturing physician performance. That doesn’t mean what we’re doing is wrong, it’s just not an evidence-based assessment method. As to what we’re actually measuring (i.e., knowledge, competence), there may be lots of opinion, but the burden is on us to provide some evidence.
References:
- Peabody JW, Luck J, Glassman P, Dresselhaus TR, Lee M. Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA. 2000;283:1715-22. (article)
- Peabody JW, Luck J, Glassman P, et al. Measuring the quality of physician practice by using clinical vignettes: a prospective validation study. Ann Intern Med. 2004;1414:771-80. (article)
- Dresselhaus TR, Peabody JW, Luck J, Bertenthal D. An evaluation of vignettes for predicting variation in the quality of preventive care. J Gen Intern Med. 2004;19:1013-18. (article)
