Peter Hendry: CEM Consultant
Using CEM Data in Practice: Feedback from CEM Secondary Assessments Glasgow Conference 13th Feb 2013
[email protected] [email protected]
Assessment for Excellence: 2 systems S1/S2 Baseline Assessment (MidYIS): A Test of Developed Ability at start of S1 or S2 •
4 sections: Vocabulary, Maths, Non-Verbal and Skills (Proof Reading, and Perceptual speed and accuracy
•
Computer adaptive* or paper-based
•
Any time Term 1 + catch ups
•
Value Added measures to SOSCA and to Scottish Qualifications
S2 Curriculum-based Assessments (SOSCA): A curriculum assessment of knowledge, skills and understanding at end of S2
•
Assesses what has been taught in the classroom
•
Covers, Maths, Science, Reading (English)
•
Computer adaptive*
•
Four week assessment window mid April to mid May + catch ups
•
Value Added measures from PIPs, InCAS and MidYIS and to Scottish Qualifications
AfE: key elements • School and students’ test performances are compared to a nationally representative sample • Assessment data from the tests provides school and pupil data to inform teaching and learning • ‘Predictive’ data indicates future potential performance to SQ, and will to National 4 and 5, to inform target setting and subsequent monitoring
• ‘Value-added data’ provides measures of relative performance at SQ (and will for National 4 and 5) and SOSCA from earlier baselines to inform self-evaluation
AfE Feedback: MidYIS and SOSCA Assessment Feedback: • Standardised score system: 50-150, mean 100, SD 15
• School data including component averages and band profiles (historical and current) • Pupil data in the form of spreadsheets and Individual Pupil Records (IPRs)
AfE Feedback: MidYIS and SOSCA ‘Predictions’ Feedback: • Currently to S2 SOSCA from S1/S2 baseline (MidYIS) and to SQ from MidYIS and SOSCA
‘Value-Added’ Feedback: • To SQ from curriculum baseline (SOSCA), and from S1/S2 baseline (MidYIS) • To S2 curriculum assessment (SOSCA) from P7 baseline (PIPS or InCAS) and/or S1/S2 baseline (MidYIS)
www.cemcentre.org
Detailed Information about the assessment. Well worth a look. Data for your school.
Bands, percentiles, standardised scores…
25% 25%
25%
C B
25%
A
D 0
50
60
70
80
90
100
110
120
130
90
95 98
140
Standardised scores Percentiles:
1
5
10
20
30 40 50 60 70
80
Mean 100, Standard Deviation of 15
99
150
SCHOOL A: S1 baseline data Year 2010/2011 YIS Year 8/S1 Test Academic
MidYIS Year 8/S1 Test Academic Year 2010/2011
es Standardised On A Nationally Representative Sample of Schools Scores Standardised On A Nationally Representative Sample of Schools
e Surname Forename Forename CEM Id JAMES JAMES COREY COREY JOHN JOHN SAMUEL SAMUEL DANIELLE DANIELLEFRAZER FRAZER
Standardised Scores
Sort Data By...
Standardised Scores
Mean Score: 100
Sort Data By...
Sort Data By...
Sort Data By...
Mean Score: 100
Standard Deviation: 15
Standard Deviation: 15
CEMForm Id
AverageAverage93.3
93.3
St. error St. error 1.2
1.2
1.2
81 D 76 D 93 C 88 D 117 98A
D 69 D 61 C 87 D 91 A C98
69 D 61 D 87 D 91 C 98 89 C
C
89
D
Form Sex
M M M M F M
Sex Cust Cust M 81 M 76 M 93 M 88 F 117 M 98
89.9 89.9
90.9 90.9
97.397.3
90.7 90.7
1.2
1.2 1.2
1.2 1.2
1.21.2
D 75 D 64 D 101 C 82 C D 107
98
75 64 101 82 107 98
D D B D B C
D 68 68 D 57 57 B 92 92 D 91 91 B 105 105 C 117
117
D D C C B A
D D C C B A
7272 65 65 89 89 88 88 109 10993 93
Band
Band
St. Score
St. Score
MidYIS Score MidYIS Score Band
Band
St. Score
St. Score
Band
Band
St. Score
St. Score
Non Verbal Non Verbal Skills Skills Band
Band
St. Score
St. Score
Band
Band
St. Score
St. Score
Vocabulary MathsMaths Vocabulary
DD D D D D D D B BC C
SCHOOL A: S1 baseline data Intake profiles: overall and test component bands
A school with a ‘completely average’ intake would have 25% of pupils within each band.
SCHOOL A: S1 Baseline MidYIS Year 8/S1 Historical Band Profiles Scores Standardised on a Nationally Representative Sample of Schools Year
Band A
Band B
Band C
Band D
2013 2012 2011 2010 2009
5 5 6 11 8
12 18 18 20 18
17 28 30 29 38
65 49 46 41 36
Historical Band Profiles: Overall Score Percentage of pupils falling into each MidYIS Band over time. (Nationally Representative Standardisation) (Band A = Top 25% of the sample, Band D = Bottom 25% of sample.) 100% 90%
80%
Percentage
70%
60%
Band D
50%
Band C
40%
Band B
30%
Band A
20% 10% 0%
2009
2010
2011
2012
2013
MidYIS Baseline Testing Year Test Section
Overall Show Band Percentage Values
Start Year
2009
End Year
2013
based Standardised Feedback 2012 Curriculum 2AfE S2 Curriculum based Standardised Feedback 2012
tandardised On AfE Curriculum based Sample 2012
Scores Standardised On AfE Curriculum based Sample 2012
SCHOOL A
Scores Standardised Standardised Scores Sort Data SortBy... Data By...
100 Mean Score: Mean Score: 100 15 Standard Deviation: Standard Deviation: 15
Surname
Forename AfE Id James Form Id Forename AfE Corey James John Corey Samuel John Danielle Samuel Frazer
Danielle Frazer
St. error Form
Sex M M M M F M
A D
119 106
D D B C A B
77 77 87 101 90 94 11987 10678
110 90
87 D 90 D 84 D 87 D 75 D 78 D D110 86 B D 90 81 D
B D
117 92
92.5
92.5
1.3
1.3
1.3
84 75 D 86 D 81 D117 D92
D D 88 D 66 D 50A 50C
88 D 66 D 78 D 50 D 78 D 50 D 50 D 105 B D 96 C
A C
105 96
B C
100 91
1.3
1.3
1.3
D D 89 D
91
89 63 D 53 D 63 66 53 B 90D D 66 C 100
D D 84 D 66 D C50 B
B C
90 100
C B
98 95
D D 100
1.3
96.9
Energy & Forces
97.6
96.9
96.1
97.6
96.9
1.3
1.3
1.3
1.3
1.3
1.3
137 107
A B
1.3
D 104 B 91 D 57 D 74 91 104 B D 111 74 D 57 D 73 D 90 A 139 A 111 121 B 7396 D C 9095
139 96
A C
121 95
Band Band
St. St. Score Score
Band
Overall OverallScience Science
Energy & Forces
Band
St. Score
St. Score
Earth & Space Band
Earth & Space St. Score
Band
Life Band
96.1
96.9
84 D 84 66 D 50 D 84 D 50 D 78 D 50 D 84 98D C 78 137 D 95 C 84 107 D
C C
St. Score
St. Score
Band
Mathematics
Living Things & theLiving Processes Thingsof & the Processes of Life
Band
Band
Overall Mathematics Overall 92.1
92.1
1.3 78 78 D 50
St. Score
Band
St. Score
Band
92.6
92.6
St. Score
Shape, Position & Movement
Science Science
Shape, Position & Movement
92.6
92.6
St. Score
Band St. Score
Information Handling
St. Score Band
Information Handling
St. Score Band
Number, Money & Measurement
Band St. Score
Number, Money & Measurement Band
Overall Reading
1.3
1.3 D D B C A B
St. Score
St. Score
Overall Reading
Band
Band
Passage Comprehension St. Score
St. Score
Comprehension
Band
Band
St. Score
Text Passage Comprehension
1.3
1.3 C D D D A D
95.9
95.9
1.3
1.3
1.3
Sex Cust 94 CustM M 71 77 C 94 M 76 77 D 71 M 81 101 D 115 F 76 D 81 M 8594
Mathematics Mathematics
97.2
97.2
1.3
St. error
St. Score
Band
98.1
98.1
94.2
115 85
Band
St. Score
94.2
Average
Average
St. Score
Band
St. Score
Speed Reading
Speed Reading Text Comprehension
ReadingReading
1.3
C D C A D D AA DC
A C
92 50
C D
92 50
C D
138 99
A C
79 138 79 99
D A D C
SCHOOL A: S2 curriculum (SOSCA) data
School no.1
School no.2
STANINES Stanine 1
2
4%
50
60
3
4
5
6
7
8
9
7% 12% 17% 20% 17% 12% 7%
70
80
90 100 110 120 Standardised Test Score
4%
130
140
150
Individual Pupil Record Date Of Birth: Sex:
Vocabulary Maths Non-Verbal Skills MidYIS Score
Student Number: Form:
22/02/99 F
Band
Stanine
Percentile
Standard Score
A
7
87
117
C
5
45
98
B
6
67
107
B
6
63
105
B
6
73
109
Nationally Standardised Scores with 95% Confidence Band 150
Nationally Standardised Scores
S1 baseline IPR
DANIELLE
140 130 120
110 100
90 80 70
60 50
Vocabulary
Maths
Non-Verbal
Skills
MidYIS Score
S2 curriculum (SOSCA) IPR Individual Pupil Record Date Of Birth: 22/2/1999 Sex: F
Danielle Student Number: Form:
AfE Band
Stanine
Percentile
AfE Standardised Score
Speed Reading Text Comprehension Passage Comprehension Overall Reading Number, Money & Measurement Information Handling Shape, Position & Movement Overall Mathematics Living Things & the Processes of Life Earth & Space
A A B A
7 8 6 7
84 90 74 87
115 119 110 117
B
6
62
105
B
5
51
100
C
4
26
90
C
5
46
98
A
9
99.3
137
A
9
99.5
139
Energy & Forces
A
8
92
121
Overall Science
A
9
99.4
138
S2 curriculum (SOSCA) IPR Individual Pupil Record Date Of Birth: 22/2/1999 Sex: F
Danielle Student Number: Form:
Predictions Feedback • Currently available to Standard Grade from AfE baselines • In the future to National 4 & 5 Qualifications from AfE baselines • Also to AfE S2 curriculum (SOSCA) from S1 and/or S2 baseline (MidYIS) • Includes Chances Graphs
How is a ‘prediction’ generated?
******************************* 2
******************************************* ‘PREDICTION’ (expected grade) GRADE
************************************* ************************************** ******************************** ***************************** 50% on or below the trend line ***************** BASELINE SCORE
3 4
GRADE
Trend line (regression line)
50% on or above the trend line *********************** 1
Grade
Data for 2012 examinations
AfE S2 Curriculum based Assessments 2012 Predictions & Chances Graphs to Standard Grade subjects based on 2012 AfE scores Predictions & Chances Graphs to Standard Grade subjects based on 2012 AfEInformation scores Please refer to the Sheet for details of
the method used to make these predictions Please refer to the Information Sheet for details of the method used toPoints make these predictions To Grades
Adjust
Standard Grade Point Predictions
M
Kieran Roland M Megan
F
Danielle
F
Frazer
M
84 75 68 100 117 92
84 66 60 89 98 95
92 85 3.8 50 54 5.9 72 63 5.3 93 93 3.3 138 120 1.5 99 95 3.2
3.1 4.2 3.9 2.8 1.8 2.7
4.3 6.6 5.9 3.6 1.5 3.5
3.3 5.4 4.8 2.7 1.0 2.6
3.8 5.6 5.1 3.3 1.7 3.2
3.8 5.6 5.1 3.3 1.7 3.2
3.3 4.7 4.3 2.9 1.7 2.8
3.0 4.3 3.9 2.6 1.4 2.5
3.4 4.8 4.4 3.0 1.8 2.9
4.0 5.9 5.4 3.5 1.9 3.4
4.2 6.4 5.8 3.6 1.7 3.5
4.2 6.0 5.5 3.7 2.1 3.6
3.7 5.2 4.8 3.3 2.0 3.2
84 75 68 100 117 92
84 66 60 89 98 95
92 85 4 3 50 54 6 4 72 63 5/6 4 93 93 3/4 3 138 120 1/2 2 99 95 3 2/3
4/5 6/7 6 3/4 1/2 3/4
3/4 4 4 3/4 3 3/4 4 4 4 3/4 5/6 5/6 5/6 4/5 4/5 5 6 6/7 6 5 5 5 5 4/5 4 4/5 5/6 6 5/6 5 2/3 3/4 3/4 3 2/3 3 3/4 3/4 3/4 3/4 1 1/2 1/2 1/2 1/2 2 2 1/2 2 2 2/3 3 3 3 2/3 3 3/4 3/4 3/4 3
Overall AfE Score
Graphic Communication
M
Corey
German
James
Geography
M
French
Frazer
English
F
Drama
Danielle
Craft & Design
F
Computing Studies
Megan
Chemistry
Kieran Roland M
Business Management
M
Biology
Corey
Cust
Art & Design
M
AfE Id
Administration
James
Form
AfE Science Score
Sex
AfE Maths Score
Forename
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
AfE Reading Score
Teacher's Adjustment (Levels)
S
Predictions: Chances Graphs
Point Prediction =1.8 Grade Prediction = 2
Chances Graphs based on Pupil’s SOSCA Test Score
50% chance of a grade 2 – the most likely single grade. 50% chance of a different grade
Not a label for life ...just another piece of information • The Chances graphs show that, from almost any baseline score, students come up with almost any grade - - -there are just different probabilities for each grade depending on the baseline score. • In working with students these graphs are more useful than a single predicted or target grade • Chances graphs show what can be achieved: – By students of similar ability – By students with lower baseline scores
Latest information as at 13 Feb 2102 • Replacing SG with Nat 4 & 5 predictions: CEM statisticians have put together a proposal. • This has been looked at by colleagues in Scottish Authorities. • The CEM Technical team have now published the new predictive data. • This data is now available on the CEM secondary+ website alongside the existing SG predictive data
The equivalence we have assumed between CfE outcomes and Standard Grade outcomes is outlined below: National Level National Level 5 A,B National Level 5 C,D National Level 4 National Level 3 National Level 2
Standard Grade Credit Level 1 Credit Level 2 General Level (3 and 4) Foundation Level (5 and 6) Standard grade level 7
For each student in each CfE subject, a predicted grade is given. The range of predicted grades is as follows Grade National Level 5 A National Level 5 B National Level 5 C National Level 5 D National Level 4 National Level 3 National Level 2
CEM Predicted Grade A B C D N4 N3 N2
Where a subject is only available at National Level 4 (e.g. Business), only grades N4, N3, N2 will be reported. For subjects only available at National Level 5 (e.g. Accounting), the full range of grades will be reported. Whilst we recognise that grades N4, N3, N2 are not available we have included them in the chances graphs to aid the teacher in supporting the student in making realistic subject choices.
AfE S2 Curriculum based Assessments 2012
New slide!!
Predictions & Chances Graphs for CfE National Qualifications based on 2012 AfE scores The grade 'predictions' and chances data presented here are simulated based on the relationship between AfE S2 Curriculum based, Standard Grade and Curriculum for Excellence levels, in the absence of any actual CfE examination data being available. They should therefore be treated with caution. The relationships will be updated once the first cohort are awarded CfE qualifications in 2014.
C
N4
B
C
C
C
C
D
C
C
B
N4
A
B
B
B
B
C
B
103 116
108
C
D
D
115
112 123
119
B
C
125
135 117
129
A
B
Art & Design
102
N4/D N4/D
Drama
D
N4
Design & Manufacture
D
D
Dance (N5)
D
99
Computing Science
D
99
Chemistry
D
104
Care
Business Management (N5)
D
95
Biology
D
Administration & It
C
Accounting (N5)
N4
Overall AfE Score
D
AfE Science Score
D
AfE Maths Score
N4
AfE Reading Score
Business (N4)
CfE Grade Predictions
N4/D N4/D N4/D N4/D N4/D N4/D
. Measuring Value-Added – Terminology Subject X
Exam grade Outcome
120 100 80
-ve VA
+ve VA
60 40 20 04
25
46
BASELINE SCORE
67
8 Raw Residual
Trend Line/Regression Line
SCHOOL A: S2 curriculum (SOSCA) data Year 8 MidYIS Score 2010/2011 Against AfE Overall Mathematics 2012 170
160 150 140
AfE Score
130 120 110
100 90 80 70 60
50 50
60
70
80
90
100
110
120
130
140
MidYIS Test Score (Standardised on a Nationally Representative Sample)
150
SCHOOL C: AfE data
SCHOOL C: AfE data
Standardised Residual Bar Chart
99.7% confidence limit 95% confidence limit
ANY VALUE IN THE INNER SHADED AREA SHOWS THE EXPECTED RESULTS i.e. AVERAGE VALUE ADDED
SCHOOL C: SQ data
