Establishing Content Validity of a Novel Written Examination to

Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein MEd PhD, R Nowygrod MD, A Fingeret MD

Agenda • • • • • •

Needs assessment Exam Development Exam Design Methodology Results Discussion

Needs Assessment • Historical assessment of students – Clinical evaluations – Unstructured oral exam – NBME surgery subject exam

• Goals of new evaluation tool – Objective and reliable – Congruent with educational objectives – Based on ACGME core competencies

Exam Design • Novel Exam: Clinical Skills Exam (CSE) – 20+ clinical scenarios • 5 randomly selected per exam

– Short answer or essay response – Administered over 3 hours – ExamSoft computer software

• Grading – Rubric – Senior residents

Example 1 • Identify operation • Obtain surgical consent • Determine order • Describe steps shown

Example 2 • Generate differential based on history • Describe risk factors • Interpret radiographic study • Identify next step in management

Methodology • July, 2011 - February, 2013 – n = 262

• Analysis – Descriptive statistics – Multivariate analysis

• Comparison data for validity assessment – – – –

USMLE Step 1 and 2CK NBME Subject Exams in Medicine and Surgery Clerkship clinical evaluations Clerkship order

Results: Descriptive Statistics – CSE Content and Score Distribution 45 40

Communication Anatomy 12% 8%

35 30

History 13%

Physical 9% Differential 8% Radiology 10%

Frequnecy

Management 40%

25 20 15 10 5 0

48 54 60 66 72 78 84 90 96 Score (%)

Results: Trend Analysis – CSE and NBME Surgery Subject Exam Score (%)

Clinical Skills Examination 100 90 80 70 60 50

p = 0.45

p = 0.32 Median Mean +SD Mean -SD

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

Score

NBME Surgery Subject Examination 100 90 80 70 60 50

p = 0.04

p = 0.02 Median Mean +SD Mean -SD

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

Results: Multivariate Statistics Validity Analysis CSE Linear Correlation Regression r p value NBME Surgery 0.36* 0.82 NBME Medicine 0.01 0.54 Step 1 0.34* 0.82 Step 2 0.44* 0.007 Clerk - Clinical 0.48* 0.001 Reasoning Clerk - Fund of 0.56* 0.004 Knowledge * = p < 0.001

NBME Correlation r

Linear Regression p value

0.08 0.55* 0.56*

0.20 0.02 0.004

0.43*

0.09

0.63*

< 0.001

Results: Multivariate Statistics – Validity Analysis CSE

NBME

Bivariate Analysis p value

Bivariate Analysis p value

Logistic Regression p value

ANOVA

0.30

< 0.001

0.01

OB/Gyn prior

X2

0.34

< 0.001

0.02

Medicine prior

X2

0.69

< 0.001

0.03

Clerkship order

Results: Student Feedback on CSE 9 8 7 6 5 4

3 2 1

Objectives

Relevance

Clarity

Feedback

Overall Clerkship

Results: Student Feedback - Word Cloud

Limitations –Study • Generalizability • Statistical analysis

Future Directions • Future directions – New content – Open access – Computer algorithm based grading – Use as formative assessment mid-clerkship

Conclusions • CSE is a useful adjunct to NBME surgery subject exam – Independent of clerkship order – Course goals and objectives – ACGME core competencies

• Questions / Comments