The script concordance test as a measure of clinical reasoning: a

The script concordance test as a measure of clinical reasoning: a national validation study Nouh et al The American Journal of Surgery 2012; 203: 530-534

SCT-Clinical Reasoning national validation study • SCT- prior single center study, differentiated between Jr and Sr SR • To determine validity and reliability, administered on a national level • Test given to 202 Residents - 9 Gen Surg programs across Canada • SCT distinguishes intra-operative decision making skills of Jr. vs Sr. gen surgical residents

SCT • Based on script theory from cognitive psychology • Uses authentic clinical scenarios to compare a trainee’s judgment skills with those of experts • SCT scores- intended to reflect respondents’ competence, interpreting clinical data under conditions of uncertainty

Script Theory Elements • Clinical encounter: – perceiving features of situation – assessing relevant hypotheses – checking for sx & signs that confirm or refute hypotheses – Using related knowledge to guide investigations + Rx

Charlin et al Med Educ 2007: 41,1178-1184

Script Theory Elements • Provides explanations for how events are processed • Fields of computer science and psychology • Organization of experiences in script-like formations – Referred to in future – Understand same or similar new situations

SCT A Practical + Reliable Instrument • • • • • • • • •

2004 Radiology 2006 Surgery 2009 Radiation oncology Neurology Pediatric emerg med 2011 Dermatology ENT Preclinical med students 2012 Geriatrics

SCT Example If you are considering the following investigation ...

and then you find . . .

you would consider the proposed investigation to be . . .

A V/Q scan to R/O pulmonary embolism

CXR Shows areas of collapse

∘ much less useful ∘ slightly less useful ∘ neither less nor more useful ∘ slightly more useful ∘ much more useful

SCT- Natl Study Methods • • • •

Res level of training: post grad yrs 1-5 Jr = R1-2 Sr= R3,4,5 Test development: 9 prgrm dir; scenario,26(?)s • Ans key: 5 pt scale, ranging from (-2), comp contra-ind to (+2), comp indic • 153 (?)s

SCT- Natl Study Methods Scoring • Every choice sel by expert→ Credit • 22 Gen Sur experts took test • Credit for each answer was transformed proportionally to get a max score of 1 for the modal experts’ choice on each item • Stat analysis

SCT- Natl Study Methods Results • Scores  progressively from R1to R4 with a dip in R5 • Test differentiated R1+2 from R3+4+5 • Site of training had sig effect on scores

Discussion • Limitations – – – – – –

No observed drop in variability from R1 to R5 Choice of scale unsettled Scores depend on cohort of experts Influence of generalist vs subspecialist Item production Setting of “passing” scores

• Advantages – Ease of administration; online – Reliable; valid (?)