Over my previous three posts, I introduced effect size, discussed its calculation and interpretation, and even provided an example of how you can use effect size to demonstrate the effectiveness of your overall CME program. My intention was to present a method for CME assessment that is both practical and powerful.
For those who are a bit more statistically savvy, you likely noticed that my previous effect size example focused on paired, ordinal data. That is, I used a pre- vs. post-activity survey (i.e., paired) composed of rating-scale (i.e., ordinal) questions. I chose this path because it’s fairly common in CME outcome assessments, and it’s the most straightforward calculation of Cohen’s d (which was the effect size measure of interest).
Here are some other scenarios:
- If you’re using pre- vs. post-activity case-based surveys, you’re now working with paired, nominal (or categorical) data that have most likely been dichotomized (e.g., transformed into correct/evidence-based preferred answer = 1, all other responses = 0). In this case, the road to effect size is a bit more complex (i.e., use McNemar’s to test for statistical significance, calculate an odds ratio [OR], and convert the OR to Cohen’s d). Of note, an OR is itself an effect size measure, and converting this to Cohen’s d is optional. The formula for this conversion is d = ln(OR)/1.81. (Chinn S. A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med. 2000;19:3127-31.)
- If you’re using post-activity case-based surveys administered to CME participants and a representative control group, you’re now working with unpaired, nominal data (that are typically dichotomized into correct answer vs. incorrect answer). In this case, you’ll use a chi-square test (if the sample is large) or Fisher’s exact test (if the sample is small) and also calculate a Cramer’s V. Unfortunately, converting Cramer’s V to Cohen’s d is convoluted and will not find much support in statistician circles. Good news – Cramer’s V is an effect size, just of a different type. You can learn more about types of effect size – and how to interpret each here.
If you’ve been doing this or any other analysis incorrectly (as I have in the past, often do in the present, and bet on doing in the future), don’t fret. Statisticians are constantly pointing out examples of the faulty use of statistics in the peer-reviewed literature (even in such prestigious journals as JAMA and NEJM). Keep making mistakes, it means you’re moving forward.
