Peter Hendry: CEM Consultant
Interpreting Feedback from Baseline Tests – Predictive Data Course: CEM Information Systems for Beginners and New Users
Day 1 Session 3 Wednesday 17th October 2012
[email protected]
The word ‘PREDICTION’: Quite probably the most contentious term that is used!! Concerns include: • A prediction for GCSE at start of year 7? • What has the baseline test got to do with my subject? • I know my pupils! • The predictions are too low: not valid!!! • And what about my professional judgement?
How is a ‘prediction’ generated? *
** *** ******************* A*
* ** *** ******************************** A * ** *** ********************************* *** ** * B C
GRADE
GRADE
* ** *** ************************************ *** ** * * ** *** ******************************** **** ** * * ** ***************************** *** ** * * ** *** ******************* ** * *
* ** *********** ** * 50
100 BASELINE SCORE
150
3 key points are: • The higher the baseline score the higher the final grade • Any one grade is achievable from a range of baseline scores • From any baseline score, a range of grades are possible
How is a ‘prediction’ generated?
Subject National trend line * ** (regression line)
50% on or above the trend line * ** *** ******************* A* *** ******************************** A
* ** *** ********************************* *** ** * B ‘PREDICTION’ (expected grade)
* ** *** ******************************** **** ** * * ** ***************************** *** ** * * ** *** ******************* ** * * 50% on or below the * ** *********** ** * trend line
BASELINE SCORE
C
GRADE
GRADE
* ** *** ************************************ *** ** *
2008 GCSE Maths (representative sample) 9 8 7
Result
6 5 4 3 2 1 0 0
20
40
60
baseline score
80
100
‘Predictions’…...are based on Average performance by similar pupils in past examinations The problem with the word ‘prediction’ is…? An alternative is ‘expected’ grade
Predictions
6.6 points = A/B
4 points = D
Trend line
The graph below shows the middle 2/3 of some subject trend lines MidYIS Year 7 to GCSE 2011 Value Added Analysis
Art & Design
8
Biology 7
English
GCSE Points Score
6
5
French
4
History 3
Mathematics
2
1 50
60
70
80
90
100
110
120
MidYIS Test Score
Comments?
130
140
150
Science
Some Subjects are More Equal than Others….
A
Grade
B >1 grade
C
Photography Sociology English Lit Psychology Maths Physics Latin
D
E
C
B
A Average GCSE
A*
FACTORS THAT WILL INFLUENCE RELIABILITY OF PREDICTIONS: • • • • • • • • • •
Knowledge of student Parental support/home life Peer influences/social life Student attitude, interest, language Expectations of staff Department/institution ethos Resources Quality of teaching and learning: pace of lessons Understanding how children learn……… And the reliability of the predictions......
Result
Correlation = 1
12
Correlation = 0
Correlation = 0.7
CORRELATIONS SUBJECT ART and DESIGN BIOLOGY ECONOMICS ENG LIT FRENCH HISTORY MATHS MUSIC PHYS EDUC PSYCHOLOGY
GCSE to A2 0.54 0.69 0.65 0.71 0.6 0.69 0.6 0.65 0.65 0.62
MIDYIS( Yr 7) subject to GCSE ART 0.49 BIOL 0.56 ENG LANG 0.7 FRENCH 0.64 HISTORY 0.63 MATHS 0.74 MUSIC 0.57
A
6.8 7.2
2
F
7E 120
A
3
F
7C 110
4
F
7J
5
M
7D
1
7.2
7.5
7.0
6.3
6.7 6.4 6.1
6.4
6.5
6.3
B
5.8
6.2 5.7 5.4
5.5
5.6
5.5
101
B
5.4 5.8
5.1 4.8
4.9
4.9
4.9
82
D
4.5 4.9
3.8 3.5
3.4
3.2
3.5
M
7E 131
A
2
F
7E 120
A
3
F
7C 110
B
B
B
4
F
7J
101
B
B/C
B
C
5
M
7D
82
D
C/D
C
D
A
A
English
7.1 6.8
Biology
Science
7E 131
Mathematics
Art & Design
M
History
MidYIS Band
1
French
Form
• Look at Art and Design, Biology and French. Comments?
Sex
What pattern do you notice?
student no.
• Compare the predictions for English and Mathematics.
MidYIS Score
Point and grade ‘predictions’ to GCSE
A
A/B A/B A/B
A B
A A*/A
A
A/B A/B A/B
B/C B/C B/C B/C B/C C
C
D/E D/E
C
C
E
D/E
The graph below shows the middle 2/3 of some subject trend lines MidYIS Year 7 to GCSE 2011 Value Added Analysis
Art & Design
8
Biology 7
English
GCSE Points Score
6
5
French
4
History 3
Mathematics
2
1 50
60
70
80
90
100
110
120
MidYIS Test Score
Comments?
130
140
150
Science
Individual Chances Graph for Student 4- GCSE English MidYIS Score 101 MidYIS Band B Teacher's Adjustment : 0 grades / levels / points
45
39
40
35
Percent
30
Prediction/expected grade: 5.1 grade C
27
25
20
19
Most likely grade
15
9
10
5
5
0
0
U
G
1
1
0 F
E
D
Grade
C
B
A
A*
MidYIS Year 9 Test Academic Year 2011/2012 Predictions based on 2011 Regression Analysis Predictions & Chances Graphs to GCSE subjects
to place school at 75th percentile of VAD
based on Year 9 nationally standardised scores Please refer to the Information Sheet for details of the method used to make these predictions
Points To Grades
Adjust
GCSE Point Predictions
Teacher's Adjustment (Grades)
Display Legend
4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.6
3
F
91400042
108
B
4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.4
4
F
91400043
101
B
4.2 5.0 5.6 5.7
4.8 5.7
Geography
A
French
111
English Literature
9140005K
5.1 5.9
English
F
Cust
Drama
2
CEM Id
Design & Technology
4.4 5.3 5.8 5.9
Form
Chemistry
B
Sex
Business Studies
105
Forename
Biology
9140005J
Surname
Art & Design
Additional Applied Science
F
Generate File for MIS Import
Additional Science
MidYIS Band
1
Prior Value added
Student
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
MidYIS Score
Upper Quartile of Schools
5.3 5.2 5.4 5.5 5.0 5.1
5.0 5.0 5.1 5.3 4.7 4.7
MidYIS Year 9 Test Academic Year 2011/2012 Predictions based on 2011 Regression Analysis Predictions & Chances Graphs to GCSE subjects based on Year 9 nationally standardised scores Please refer to the Information Sheet for details of the method used to make these predictions
Points To Grades
Adjust
GCSE Point Predictions
Teacher's Adjustment (Grades)
Display Legend
4.9 6.0 6.3 6.5 5.8 6.5 5.9 5.8 6.0 6.1 5.6 5.8
3
F
91400042
108
B
4.8 5.8 6.2 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.6
4
F
91400043
101
B
4.4 5.2 5.8 5.9
5.0 5.9
Geography
A
French
111
English Literature
9140005K
5.3 6.1
English
F
Cust
Drama
2
CEM Id
Design & Technology
4.6 5.5 6.0 6.1
Form
Chemistry
B
Sex
Business Studies
105
Forename
Biology
9140005J
Surname
Art & Design
Additional Applied Science
F
Generate File for MIS Import
Additional Science
MidYIS Band
1
Prior Value added
Student
0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2
MidYIS Score
Reset Adjustments
5.5 5.4 5.6 5.7 5.2 5.3
5.2 5.2 5.3 5.5 4.9 4.9
MidYIS Year 9 Test Academic Year 2011/2012 Predictions based on 2011 Regression Analysis Predictions & Chances Graphs to GCSE subjects based on Year 9 nationally standardised scores Please refer to the Information Sheet for details of the method used to make these predictions
Points To Grades
Adjust
GCSE Point Predictions
Teacher's Adjustment (Grades)
Display Legend
4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.6
3
F
91400042
108
B
4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.4
4
F
91400043
101
B
4.2 5.0 5.6 5.7
4.8 5.7
Geography
A
French
111
English Literature
9140005K
5.1 5.9
English
F
Cust
Drama
2
CEM Id
Design & Technology
4.4 5.3 5.8 5.9
Form
Chemistry
B
Sex
Business Studies
105
Forename
Biology
9140005J
Surname
Art & Design
Additional Applied Science
F
Generate File for MIS Import
Additional Science
MidYIS Band
Or insert own values and click adjust
1
Prior Value added
Student
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
MidYIS Score
Upper Quartile of Schools
5.3 5.2 5.4 5.5 5.0 5.1
5.0 5.0 5.1 5.3 4.7 4.7
MidYIS Year 9 Test Academic Year 2011/2012 Predictions based on 2011 Regression Analysis Predictions & Chances Graphs to GCSE subjects based on Year 9 nationally standardised scores Please refer to the Information Sheet for details of the method used to make these predictions
Points To Grades
Adjust
GCSE Point Predictions
Teacher's Adjustment (Grades)
Display Legend
B
4.6 5.9 7.4 6.8 5.4 6.8 5.5 6.3 6.9 6.5 5.9 6.8
4
F
91400043
101
B
4.2 5.3 7.0 6.4
4.8 6.4
Geography
108
French
91400042
English Literature
F
5.1 6.6
English
3
Cust
Drama
4.7 6.1 7.5 7.0 5.6 7.0 5.7 6.5 7.1 6.7 6.1 7.0
CEM Id
Design & Technology
4.4 5.6 7.2 6.6
A
Form
Chemistry
B
111
Sex
Business Studies
105
9140005K
Forename
Biology
9140005J
F
Surname
Art & Design
Additional Applied Science
F
2
Generate File for MIS Import
Additional Science
MidYIS Band
1
Reset Adjustments
Student
0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4
MidYIS Score
Upper Quartile of Schools
5.3 6.1 6.7 6.3 5.7 6.5
5.0 5.9 6.4 6.1 5.4 6.1
MidYIS Year 9 Test Academic Year 2011/2012 Predictions based on 2011 Regression Analysis Predictions & Chances Graphs to GCSE subjects based on Year 9 nationally standardised scores Please refer to the Information Sheet for details of the method used to make these predictions
Adjust
A
A/B
B
A
B A*/A
A
B/C
A
B/C A/B
A
A/B
B
A
A/B
C
A/B
A/B
B
B/C
B
2
F
9140005K
111
A
C/D
3
F
91400042
108
B
C/D
4
F
91400043
101
B
D
Cust
B/C
A
A/B B/C
C
B
B
Geography
B/C A/B
C/D B/C
CEM Id
French
A
B
Form
English Literature
B/C
105
Sex
English
Business Studies
A
9140005J
Forename
Drama
Biology
B A*/A
F
Surname
Design & Technology
Art & Design
C
MidYIS Band
A/B
1
Generate File for MIS Import Student
A
MidYIS Score
Reset Adjustments
0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4
Additional Science
Upper Quartile of Schools
Additional Applied Science
Teacher's Adjustment (Grades)
GCSE Grade Predictions
Chemistry
Grades To Points
Display Legend
A/B A/B B/C A/B
Prediction/expected grade: 6.4 grade A/B
Most likely grade
MidYIS Year 9 Test Academic Year 2011/2012 Predictions based on 2011 Independent Sector Regression Analysis Predictions & Chances Graphs to GCSE subjects based on Year 9 Independent Sector standardised scores Please refer to the Information Sheet for details of the method used to make these predictions
Points To Grades
Design & Technology
C
6.1 6.8 6.7 6.0 6.7 6.1 6.4 6.4 6.5 6.4 6.1 6.5 6.2
3
F
91400042
90
C
6.0 6.7 6.6
5.9 6.6
4
F
91400043
81
D
5.5 6.5 6.1
5.4 6.1 5.6 5.8 5.9 5.9 5.9 5.4 5.8 5.6
German
94
Geography
9140005K
French
F
5.7 6.4
English Literature
2
Cust
English
5.8 6.6 6.4
CEM Id
Drama
D
Form
Chemistry
87
Sex
Business Studies
9140005J
Forename
Biology
Additional Science
F
Surname
Art & Design
MidYIS Band
1
Generate File for MIS Import Student
Classical Civilisation
GCSE Point Predictions
MidYIS Score
Display Legend
5.8 6.1 6.1 6.1 6.2 5.7 6.1 5.8 6.0 6.3 6.3 6.3 6.3 5.9 6.3 6.0
Independent Sector Prediction/expected grade: 5.9 grade C
Most likely grade
Not a label for life ...just another piece of information • The Chances graphs show that, from almost any baseline score, students come up with almost any grade - - -there are just different probabilities for each grade depending on the baseline score. • In working with students these graphs are more useful than a single predicted or target grade • Chances graphs show what can be achieved: – By students of similar ability – By students with lower baseline scores
Yellis predictive data: baseline score 103 (55%) Yellis Average Subject Predicted Business Studies 5.4 (B/C) English 5.7 (B/C) French 5.4 (B/C) Geography 5.6 (B/C) Mathematics 5.7 (B/C) Physical Education 5.7 (B/C) Science: GCSE 5.6 (B/C) Science: GCSE Additional 5.6 (B/C) SC Religious Studies 5.3 (B/C)
Yellis Average Subject Predicted Business Studies 5.6 (B/C)* English 5.9 (B)* French 5.6 (B/C)* Geography 5.8 (B)* Mathematics 5.9 (B)* Physical Education 6.0 (B)* Science: GCSE 5.8 (B)* Science: GCSE Additional 5.8 (B)* SC Religious Studies 5.6 (B/C)*
Weighted Average
Weighted Average
5.6 (B/C)
Yellis Average Subject Predicted Business Studies 5.4 (B/C) English 5.7 (B/C)* French 5.4 (B/C) Geography 5.8 (B)* Mathematics 5.9 (B)* Physical Education 6.7 (A/B)* Science: GCSE 6.3 (A/B)* Science: GCSE Additional 5.6 (B/C) SC Religious Studies 5.7 (B/C)* Weighted Average
5.8 (B)
5.8 (B)
Polly Bolton
B
SECURE
5
ORG
MC
E
B
UNLIKELY
3
Science
CPa
E
C
UNLIKELY
Science Additional
CPa
D
C
French
CK
C
History
KM
RS
CG
CB
Maths
B
SECURE
5
C
B
LIKELY
5
HW
D
B
POSSIBLE
4
4
ORG
D
C
POSSIBLE
4
ORG
LIKELY
4
ORG
B
C
LIKELY
4
ORG
B
LIKELY
4
C
B
LIKELY
5
A
A
SECURE
5
B
A
SECURE
5
D
B
POSSIBLE
4
C
B
POSSIBLE
4
KEY - target is:
SECURE
LIKELY
POSSIBLE
UNLIKELY
effort: 5 excellent - 4 good - 3 satisfactory - 2 poor - 1 very poor concern: WW working well - ATT attitude - BEH behaviour TEN attendance - PUN punctuality - HW homework CON confidence - ORG organisation - EAL language
concern
B
English Literature
B
is:
effort
ORG
target grade
5
year 10 exam
LIKELY
concern
concern
B
CB
is:
effort
effort
C
English
target grade
is:
Teach
Subject
SUMMER
current grade
target grade
SPRING
current grade
AUTUMN
Student 1
Student 2
Why is the ‘predicted’ grade not always equal to the highest bar ? Predicted (‘expected’) grade
Most likely grade
AT WHICH POINT WILL THE SEESAW BE BALANCED? Predicted (‘expected’) grade i.e. the lower grades ‘pull’ the prediction to the left
Student 3
Student 4
Student 4 - IPR
Vocab Item Responses
Correct
30
Response Time (s)
25
20
15
10
5
0 1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 Question Wrong
Basing Targets on Prior VA – One Methodology from an Alis School •
Discuss previous value added data with each HoD
•
Start with an agreed REALISTIC representative figure based, if available on previous (3 years ideally) of value added data
•
add to each pupil prediction, and convert to grade (i.e. in-built value added)
•
Discuss with students, using professional judgment and the chances graphs, adjust target grade
•
calculate the department’s target grades from the addition of individual pupil’s targets
theet to tadl e tasrg d ello s a th e to ta l ta rg g ra ae ret g a ra s fo wre s : a s fo llo w s :A F ro m a n d in c lu d in g 2 0 0 2 , a ra w re s id u a l o f 2 0 .0 is e q u iva le n t to o n e gBra d e
1A 01 2B 32 C 6C 56 S U G G E S T E D T A R G E T S F O R 2 0 0 7 , b a s e d o n A L IS p re d a n d d e p D t's v a lu e a d1Dd e d h is to ry 11 T h e ta rg e t g ra d e h a s a n in -b u ilt va lu e a d d e d o f 1 5 p o in ts (o n e g ra d e E is 2 0 p o in ts 0 )E 00 ta rg e t dept adj D EPAR T M EN T : A g ratadrg e et taTrg T a rg e t S e ttin g etaaerg ct heet r tn g dee a sA d eE T a dgj ra ta0d rgee t S u rn aFth m e to FAoeve re Are ve C fo S ElloPwresTd: D reRdAG ic E tioTn T g Ara R1G S u rn a m e oere n ata ml eta rg Gara CmS Es a TG D icA tio n P TA no. of S td . 3 yr. A v. B 2 3 D 48 .7 2 .3 8 .3 6D 4 .3 D 4 .7 2 49 6449.3 p u p ils R e s id S td re s id ye a r a v. 5G.8 C S E a v.3 T D A ra w re s id . C 6 5 50 .8 0 .2 8C 8 .2 C 7 3 .2 8783.2 C 11 .4 1 2002 2 6 .8 44698.9 .0 29 4 .5 16D .2 8 101A.7 .9 6 .4 19 1.4 .4 BA 0.8 0 .2 6 .8 1 .8 9 5.8 .2 0 98 .8 CB 2003 7 76 .1 46 19 18 3 .3 050E .6 0B
0 3 5 1 0
Teacher a dEj StaUrg et R R LT D D D C C C B B B C C C .1 3 .8 9 .8 7 2.8.8 C B B .1 7 75 .8 B B 2004 6 65 .6 535 91 15 8 .2 027.7 0C ta8C rg et T e aC 5 .53 .8 4 1 2 .8626 3±.3 04 .0 5 0 .686 6.3 1 .3 Cc h e r D C D 2005 12 6 .1 75±.50 .2 2 4 4 .5 3 00 ± 01± 0.3.2 9 0 .6 5 g ra d e a d j ta rg e t S u rn a m e F o re n a m e A ve5G.4C S E T5D4 A P re d ic tio n T A R G E T R E S U L T .4 5 .4 4 .4 7C 8 .4 C C B 63 7683.4 C B D .7 8 is e q u iva 9.9 .3 D.9 D 52.0 .2 3 3.3 5g44 9ra .9d e 7C 4 C C C 3 549 .9 C C F ro m a n d in c lu d in g 2 0 0 2 , a ra w re54s.2 id u a l o f 2 03 le n t to o n e76 .8 0 .2 C.1 C C 65.1 5 779 98 B B 633 .1 53.1 3.2 748 9.1 .1 9B 4 B B B 6 .9 4 8 9 6 .4 1 1 1 .4 A B B RTASG F EOARV2E0R 50.7 .8A L IS p re 6491d.6 49.6 A7G 51n .7 .8a n d d e p86t's .6v a lu e a 8dC 4d.6 S U G G E S T E D T AARVGEE , bEa s e d4o e d h is to ryC 0 .8 5 .82 0 p o in ts )B C C T h e ta rg e t g ra d e h a s a n in -b u ilt va6lu.2e a d d e d o6f 11 5 p o in ts8(o n e g ra d e9is 5 .1 39 5 7 .8 7 2 .8 B ta rg eC t dept B adj 5 .5 30 6 6 .3 8 1 .3 D g ra d eC ta rg eC t .4 d e s a re a 5s 4 fo llo w s6: 3 .4 7A 8 .4 C C B th e to ta l ta rg e t g5ra 1 0 5 .2 33 5 9 .9 7B 4 .9 C C C 2 3 6 .1 53 7 9 .1 9C 4 .1 B 6B 5B AVERAGE 5 .7 4 1 .8 6 9 .6 8D 4 .6 C 1 1 E 0 0
S u rn a m e
F o re n a m e
A ve G C S E 4 .7 5 .8
TDA 28 30
P re d ic tio n 4 9 .3 7 3 .2
TARGET 6 4 .3 8 8 .2
ta rg e t g ra d e D C
Teacher a d j ta rg e t R E S U L T D D C C
Key Questions for Target Setting • What type of valid and reliable predictive data should be used to set the targets? • Should students be involved as part of the process (ownership, empowerment etc.)? • Should parents be informed of the process and outcome?
Key points to consider might include: • • • • • • •
Where has the data come from? What (reliable and relevant) data should we use? Enabling colleagues to trust the data: Training (staff) Communication with parents and students Challenging, NOT Demoralising, students……. Storage and retrieval of data Consistency of understanding what the data means and does not mean
