When I saw the JCEHP editorial title lead with “How Significant Is Statistical Significance…” I knew I’d be blogging about it. As I remember the progression through graduate school statistic courses, it began with learning how to select the appropriate significance test, progressed to application, and then concluded with all the reasons why the results didn’t really mean much. So I was ready to build a “cut-and-paste” blog post out of old class papers detailing an unhealthy dependence on the results of statistical tests (which I expected to be the opinion of this editorial). And that would have worked fine, but then I found a rabbit hole: script concordance test (SCT).
Casually introduced by the authors via an educational scenario illustrating the limitations of statistical significance, SCT is a case-based assessment method designed to measure the clinical decision-making process (as opposed to simply identifying whether someone knows a correct diagnosis or treatment). As educators, this could be quite helpful in clarifying educational gaps. For evaluators, this approach has some encouraging validity data. I’ve got a way to go before I can even claim familiarity with SCTs but will be diving into the literature immediately (and assuming expert status by hopefully next week). If anyone else is interested, here are some suggestions to learn more:
- Fournier JP, Demeester A, Charlin B. Script concordance tests: guidelines for construction. BMC Med Inform Decis Mak. 2008;8:18. (click here for full article)
- Charlin B, Roy L, Brailovsky C, Goulet F, van der Vleuten C. The script concordance test: a tool to assess the reflective clinician. Teach Learn Med. 2000;12:189-95. (click here for abstract)
- Dory V, Gagnon R, Dominique V, Charlin B. How to construct and implement script concordance tests: insights from a systematic review. Med Educ. 2012;46:552-63. (click here for full article)
- Lubarsky S, Charlin B, Cook DA, Chalk C, van der Vleuten C. Script concordance testing: a review of published validity evidence. Med Educ. 2011;45:329-38. (click here for full article)
FYI – it turns out that SCTs were introduced in the late 1990s. So I’m less than 20 years behind the curve and perfectly in tune with the traditional adoption curve of evidence to clinical practice (which hovers around 17 years).
