30 Years of Evidence on the Comparability of Exam Standards
Short Description
Download 30 Years of Evidence on the Comparability of Exam Standards...
Description
30 Years of Evidence on the Comparability of Exam Standards: Myths, Fiascos and Unrealistic Expectations Paul E. Newton Centre for Evaluation & Monitoring, University of Durham, 30th Anniversary Conference: 30 Years of Evidence in Education. 23 September 2014. London.
Statistics vs. Judgement: What Does 30 Years of Research Tell Us About the Best and Worst Way to Maintain Exam Standards?
What does it mean to ‘maintain’ an exam standard? Grade Awarding The process of identifying: which marks on this year’s exam correspond to levels of attainment (i.e. levels of knowledge, skill and understanding) that were associated with grade boundary marks on last year’s exam.
Why do exam boards need to move grade boundaries? Because even exams that are designed to measure: exactly the same kind of attainment in exactly the same way may end up being slightly different in terms of the overall difficulty of their questions
Have we always maintained exam standards like this? 30 years ago – in 1984? 60 years ago – in 1954?
Have we always maintained exam standards like this? 30 years ago – in 1984? 60 years ago – in 1954? … yes, pretty much!
Attainment-referencing
From one examination to the next, corresponding grade boundaries should be located at marks associated with equivalent levels of attainment.
The myth
HYPOTHETICAL A level pass-rates for UCLES (Summer examinations, Home candidates only) 100 90 80 70 Latin
60
French
50
Physics
40
Biology
30 20 10
1986
1985
1984
1983
1982
1981
1980
1979
1978
1977
1976
1975
1974
1973
1972
1971
1970
1969
1968
1967
1966
1965
1964
1963
1962
1961
1960
0
The myth… debunked
A level pass-rates for the 'Cambridge' board UCLES (1960 to 1984) (Summer examinations, Home candidates only)
100 95 90 85 80
Latin
75
Physics
70 65 60 55 1984
1983
1982
1981
1980
1979
1978
1977
1976
1975
1974
1973
1972
1971
1970
1969
1968
1967
1966
1965
1964
1963
1962
1961
1960
50
How do you operationalise attainment-referencing?
Cumulative percentage of A level Sociology students awarded grade E (blue) against total number of results awarded (red) (for All Boards, Summer Awards, All Modes, by Syllabus Group)
100.0 34000 95.0 32000 90.0 30000 85.0 28000
80.0
Cum.% E No. Results
26000
75.0 70.0
24000
65.0
22000
60.0
20000 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Scrutiny of scripts (undertaken by examiners)
Comparing levels of attainment ‘directly’ by inspecting performances in examination scripts a.k.a. ‘Judgement’
Scrutiny of data (undertaken by the Board)
Cumulative percentage of A level Sociology students awarded grade E (blue) against total number of results awarded (red) (for All Boards, Summer Awards, All Modes, by Syllabus Group)
100.0 34000 95.0 32000 90.0 30000 85.0 28000
80.0
Cum.% E No. Results
26000
75.0 70.0
24000
65.0
22000
60.0
Comparing levels of attainment indirectly by ‘modelling’ the causal determinants of attainment
20000 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
a.k.a. ‘Statistics’
Which is better – statistics or judgement?
Cumulative percentage of A level Sociology students awarded grade E (blue) against total number of results awarded (red) (for All Boards, Summer Awards, All Modes, by Syllabus Group)
100.0 34000 95.0 32000 90.0 30000 85.0 28000
80.0
Cum.% E No. Results
26000
75.0 70.0
24000
65.0
22000
60.0
20000 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Which is better – statistics or judgement?
Cumulative percentage of A level Sociology students awarded grade E (blue) against total number of results awarded (red) (for All Boards, Summer Awards, All Modes, by Syllabus Group)
100.0 34000 95.0 32000 90.0 30000 85.0 28000
80.0
Cum.% E No. Results
26000
75.0 70.0
24000
65.0
22000
60.0
20000 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
The battle of grade awarding Examiners
The Board
We are just so impressed by the quality of performances that we see in our French exams.
But do you really have enough evidence to justify raising the pass-rate yet again? After all: pass-rates haven’t been rising in German or Spanish the French cohort is expanding massively
What Does 30 Years of Research Tell Us About the Best and Worst Way to Maintain Exam Standards?
Evidence from Exam Boards
Evidence from Academia
Evidence from Regulators
What have we learned since 1984?
We shouldn’t put too much confidence in statistics
Cumulative % candidates with grade E (or higher) Averaged across 13 UCLES A level subjects, 1960-1984 (Summer examinations, Home candidates only, Main syllabuses only)
100 90 80 70 60 50 40 30 20 10 0
4 NEAB maths A levels P&A, P&M, P&S, SMP
MLM to control for prior achievement, gender, etc. even after control, SMP still appeared too lenient
However the SMP syllabus more motivating excellent support materials more time-consuming
We shouldn’t put too much confidence in judgement
Grade boundaries set by examiner judgement alone for two exam papers same subject different tiers
sat by same candidates Many more students ended up with higher grades on the lower tier exam (than on the higher tier).
Judgemental innovations
We have learned how to harness examiner judgement more effectively
Statistical innovations
We have learned how to compute statistical analyses more effectively
It is extremely hard to predict and control comparability threats.
The ‘fiascos’ Summer 2002 Curriculum 2000 anomaly
Summer 2012 GCSE English anomaly
January awarding, 2012 Clear tendency to ensure students marked ‘comfortably’ above historical boundaries
June awarding, 2012 Same tendency, but many students no longer ‘comfortably’ above the raised boundaries
So, which is better – statistics or judgement?
Cumulative percentage of A level Sociology students awarded grade E (blue) against total number of results awarded (red) (for All Boards, Summer Awards, All Modes, by Syllabus Group)
100.0 34000 95.0 32000 90.0 30000 85.0 28000
80.0
Cum.% E No. Results
26000
75.0 70.0
24000
65.0
22000
60.0
20000 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Unrealistic expectations Three ‘stages’ in understanding comparability 1. statistical auditing problems are routine solutions require ‘back of the envelope’ sums
2. scientific research problems are difficult solutions require rigorous and objective investigations
3. art criticism problems are perhaps insurmountable solutions require value judgements (Bardell, Forrest and Shoesmith, 1978)
Realistic expectations + Persuasive justifications Four ‘stages’ in understanding comparability 1. statistical auditing 2. scientific research 3. art criticism
4. engineering pragmatism
many comparability problems are technically insurmountable… but some are less insurmountable than others and should be prioritised all comparability solutions are inevitably imperfect… but some are less imperfect than others and should be prioritised technically insurmountable problems and inevitably imperfect solutions highlight the fundamental importance of strong arguments in defence of policy and practice
View more...
Comments