100% - CEM

Performance Monitoring with feedback: a holistic approach

PERFORMANCE MONITORING USING VALUE ADDED DATA (Post – 16) Keith Murdoch

Performance Monitoring with feedback: a holistic approach

Overview of Presentation • Context: The College, the locality and some political drivers • An overview of Woodhouse College’s Performance Monitoring approach • Performance Monitoring – College – Departments – Students

• Concluding musings

Performance Monitoring with feedback: a holistic approach

Woodhouse College: 1,145 16-19 full time students • 99% A – level provision

• 58% Female • 54% BME

• Enrolments from 140+ schools

Performance Monitoring with feedback: a holistic approach

Value Added & the Common Inspection Framework Outcomes for Learners •

Learners’ attainment and progress

Quality of Provision •

Effectiveness of teaching, training and assessment in supporting learning and development

•

Effectiveness of the care, guidance and support learners receive

Leadership and Management •

Raising expectations and promoting ambition

•

Actively promoting equality and diversity to narrow the achievement gap

•

Effectiveness of self assessment

Performance Monitoring with feedback: a holistic approach

PERFORMANCE MONITORING Key Elements: Using value added data • Monitoring Student Progress / Student Reviews • Lesson Observation • Internal Inspections • Every Child Matters • Diversity and Equality • Self Assessment: Department / College

Performance Monitoring with feedback: a holistic approach

SELF ASSESSMENT • How well do you know your Department / College?

• How well do you know your data? • Can you accurately identify your weaknesses? • Do you have the capacity to make improvements?

• Can you provide evidence to demonstrate improvements?

Performance Monitoring with feedback: a holistic approach

Performance Monitoring Using ALIS to monitor achievement & attitudes with feedback to 3 levels How are we doing and how do we know? with

ALIS helps us measure:

Students

Progress on Course

Departments

Achievement

Whole College

Achievement & Attitudes

What are we doing about the poor bits? e.g. Target Setting, Action Plans, Operational Plans, Strategic Plans

Quality Monitoring with feedback: a holistic approach

Aug & Sept

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May

Jun

Strategic & Operational Planning Dept. SAR Achievement + Action Plans

Whole College SAR

Dept SAR Phase 1 + Action Plans

Curriculum Quality Monitoring & Internal Inspections Data Capture GCSE A/AS

Achievement Feedback

MAG

Yr 12 Reviews

Yr 13Data Capture: Attitudes

Input Feedback

Yr 12 & 13 Reviews

On-course student assessment and monitoring, subject by subject

Yr 12 Reviews

Jul

Performance Monitoring with feedback: a holistic approach

Monitoring the overall performance of the COLLEGE

Summative monitoring by Senior Leaders and Governors

Performance Monitoring with feedback: a holistic approach

Performance Monitoring with feedback: a holistic approach

Performance Monitoring with feedback: a holistic approach

Performance Monitoring with feedback: a holistic approach

Performance Monitoring with feedback: a holistic approach

A Level results 2010 No of students on a given MAG achieving A*-U grades A* A B C A/B 56 112 40 7 B 67 330 302 139 B/C 3 20 28 26 C 8 47 87 96 C/D 2 4 D 1 4 6 D/E 1 1 E 2 5 1 Grand Total 134 512 469 280

D 2 36 10 60 3 10 1 2 124

% of students on a given MAG achieving A*-U grades A* A B C A/B 26% 51% 18% 3% B 8% 37% 34% 16% B/C 3% 22% 31% 29% C 2% 14% 26% 28% C/D 17% 33% D 3% 14% 21% D/E 33% 33% E 17% 42% 8%

D 1% 4% 11% 18% 25% 34% 33% 17%

E

U

15 2 22 2 6

2 1 19 1 2

2 49

25

E

U

2% 2% 6% 17% 21%

1% 6% 8% 7%

17%

Grand Total 218 891 90 339 12 29 3 12 1594

Grand Total 100% 100% 100% 100% 100% 100% 100% 100%

Performance Monitoring with feedback: a holistic approach

Report to Governors 2010 Subject

No of Retention % Student s 40 100 96

97

41

93

117

98

61

94

124

97

60

100

106

100

36

100

2008-2010

Raw Resid.

Std. Resid.

08-10

A-B %

A-E %

A-B %

A-E %

63 45 64 42 73 36 56 46 69 45 57 51 45 48 65 47 89 42

100 97 100 95 100 98 100 98 100 98 96 95 95 96 100 97 100 99

62

100

2.2

0.1

-0.1

65

100

10.0

0.4

0.5

65

100

4.5

0.2

0.2

64

100

1.7

0.1

0.1

70

100

6.1

0.3

0.3

62

98

1.1

0.0

0.2

50

96

-3.6

-0.1

0.0

76

99

5

0.2

0.4

78

100

11.1

0.6

0.5

Value-Added

Performance Monitoring with feedback: a holistic approach

Monitoring the Performance of DEPARTMENTS 



SELF ASSESSMENT:  summative monitoring by departments of their own performance using fair comparisons of achievement levels. Analysing student achievement by subject

Performance Monitoring with feedback: a holistic approach

Key Performance Indicators Success rates % Grade A* - B at A2 and A - B at AS Pass rate Value added (A2, AS): exam year / 3 year Performance compared to MAG Retention rate Learner satisfaction Attendance Enrolment number Gender and Ethnicity monitoring Lesson Observation grade profile Internal Inspection grade Average class size

Performance Monitoring with feedback: a holistic approach

Year Institns in cohort

GCSE

IT DA

Av. A

Residual

2004

6.4

50.2

7.4

0.36

10%

2005

6.2

48

5.7

-0.4

10%

2006

6.4

49

93.1

0.04

2007

6.5

48

95

-0.04

2008

6.5

48

101

0.31

10%

2009

6.3

94

0.1

25%

2010

6.5

102

0.29

10%

2011

Year

Error

Attitude to Subject

Attitude to College

Advising College

(+/ -)0.2 0.1 (+/ -)0.18 0.09 (+/ -)0.22 0.11 (+/ -)0.2 0.1 (+/ -)0.2 0.1 (+/ -)0.18 0.09 (+/ -)0.2 0.1 (+/ -)0.

3.6

3.7

70%

3.6

3.7

82%

3.4

3.6

3.3

3.9

3.5

3.7

3.3

3.7

3.5

3.7

Analysis of Residuals by GCSE scores 8 - 6.2

8 - 6.9

6.8 - 6.2

6.1 - 5.5

2002 2003 2004 3yr. 2005 3 yr. 2006

0.2 0.3 0.55 0.35 -0.9 -0.02 1.4

56 63 48

0.29

21

0.76

27

54

-0.2

19

-1.3

35

59

1.4

27

1.4

32

2007 2yr. 2008 3yr. 2009 3yr. 2010 3yr. 2011 3yr.

-1 0.20 5.6 2.00 1.5 2.03 4.2 3.77

76

-1.3 0.05 4.6 1.57 2.2 1.83 1.8 2.87

36

-0.7 0.35 6.2 2.30 1.0 2.17 6 4.40

40

80 65 78

32 27 34

48 38 44

5.4 - 4.7

0.3 -0.1 0.9 0.37 -0.7 0.03 -2.4

26 24 32

2.7 0.15 -2 -0.57 2.4 1.03 7.1 2.50

22

38 20

11 42 23

ExtraMural

LSE

Enrol "A"

Complete Retention "A" "A"

1 1

95 102 110

90 94 95

95% 92% 86%

1

115

106

92%

92

90

98%

1

109

104

95%

1

105

104

99%

1

120

117

98%

1

108

105

97%

4.6 - 4

0.5 1.6 0.85 0.98 -0.6 0.62 2.6

8 6 14

-7.8 -2.60 12.6 2.47 -4.4 0.13 7.9 5.37

5

13 11

12 7 3

0 -0.9 -0.6 -0.50 -0.6 -0.70 0 13.7 6.85 8 7.23 31.1 17.60 23.7 20.93

Performance Monitoring with feedback: a holistic approach

Ye ar

GCSE

2010

6.5

T DA

Av. A

Re sidual

102

0.29

E rror 10% (+/-)0.2 0.1

Attitude Attitude to to Subje ct Colle ge 3.5

3.7

Advising Colle ge

Performance Monitoring with feedback: a holistic approach

SELF-ASSSESSMENT: DEPARTMENTS

ALIS DATA ANALYSIS 1.

Summary of Raw Results •

2.

Analysis by GCSE Score • •

3.

•

An explanation of data points which lie outside the control lines A comment on the moving average

Analysis by Teaching Group •

7.

Can we identify common features within the high and low achieving groups? Do the extremes distort the overall picture of the subject performance?

Analysis of Variance • •

6.

In terms of actual scores and standardised residuals, how have students from different ethnic minority/gender groups performed? Are the differences significant?

Analysis of Extreme Cases • •

5.

In terms of residuals how have students in different bands been performing? Does a preponderance of students in any one band help explain the overall residual?

Analysis by Ethnic Minority and Gender •

4.

Comparison of raw subject A-level results with the national percentage for the subject

Is there any apparent correlation between set and residual or, over the 3 year period, between member(s) of staff teaching a set and the residuals achieved, after sets have been analysed for the range of ability?

Students’ Attitudes and learning and Teaching Processes • •

Analysis in trends in students’ attitudes to subject Issues raised by analysis of perceived learning activities.

Performance Monitoring with feedback: a holistic approach

1. Analysis by GCSE Score – – –

In terms of residuals, how have students in different bands been performing? Does a preponderance of students in any one band explain the overall residual? How does your analysis impact on strategies for teaching and learning? Analysis of Residuals by GCSE scores 8 - 6.2

8 - 6.9

2010 4.2 78 3yr. 3.77

1.8 2.87

6.8 - 6.2 34

6.0 4.40

6.1 - 5.5 44

7.1 2.50

5.4 - 4.7 23

7.9 5.37

4.6 - 4 3

24 20.93

1

Performance Monitoring with feedback: a holistic approach

2. Analysis by Gender & Ethnic Minority – – –

In terms of standardised residuals, how have students from different e-m/gender groups performed? Are the differences significant? What impact will these differences have on your teaching and learning strategies?

Ye ar

Ge nde r

No

AvGCSE

AvS

AvR

StdR

Diffe re nce

2010

F M

54 45

6.8 6.7

95.6 100.4

4.8 12.8

0.2 0.6

0.40

08-10

F M

178 117

0.35 0.49

0.14

Performance Monitoring with feedback: a holistic approach

Performance Monitoring with feedback: a holistic approach

3. Analysis of Extreme Cases – An extreme is a student with a raw residual of + or – 30 (15 for an AS) – Can we identify common features within the low and high achieving groups? – Do the extremes distort the overall picture of the subject performance?

Performance Monitoring with feedback: a holistic approach

Student Level Residuals Average GCSE Score

Actual Grade

Predicted Grade

Raw Residual

Jaymin

5.7

60

50

10

Davy-Martin

Lee

6.7

120

85

35

Lai

King-Ho

6.1

0

62

-62

Mansigani

Ilaha

6.0

120

60

60

Taylor

Hannah

6.3

100

75

25

Surname

Forename

Chandaria

Performance Monitoring with feedback: a holistic approach

4. Analysis of Variance – An explanation of data points which lie between control lines – A detailed explanation of data points which lie outside 3SD control line – A comment on the moving average

Performance Monitoring with feedback: a holistic approach

Performance Monitoring with feedback: a holistic approach

KEY ELEMENT of

SELF ASSESSMENT ANALYSIS  Identifying strengths and weaknesses  understanding your weaknesses and identifying actions for improvement  impact on strategies for teaching and learning and supporting students

Performance Monitoring with feedback: a holistic approach

Using ALIS data to identify and ‘unpack’ weaknesses e.g. -0.7 Female -0.1 Male

-0.28 Standardised Residual

AS Physics

-0.64 without Maths -0.19 with Maths

•Work with Maths Department on creating resources to support student not taking AS Maths • Introduce problem-solving consolidation sessions • Further investigation of girls underachievement – Institute of Physics, Standards Unit, focus groups etc

Performance Monitoring with feedback: a holistic approach

e.g. … contd. -0.23 Standardised Residual

3 distinct groups 1. Additionality: 0.1 AS Critical Thinking

2. Full programme : -0.8 3. Applied AS: -0.5

• Restructure external assessment of course – January module • Increase hours for Applied AS students, formalise requirements, ‘integrate’ into main programme

• Review appropriateness

Performance Monitoring with feedback: a holistic approach

Monitoring the Performance of STUDENTS to raise aspirations and achievement

Performance Monitoring with feedback: a holistic approach

You know you’ve gone to Woodhouse when..... MAG/CAG. They do mean something.

Performance Monitoring with feedback: a holistic approach

PROBLEM Motivating, and monitoring student progress is a FORMATIVE process ALIS is  RETROSPECTIVE  SUMMATIVE  STATISTICAL

HOW CAN WE ‘SQUARE THE CIRCLE?’ Possible because correlations are high and the variation in the association between AVGCSE and the statistically ‘PREDICTED’ A-Level grade, from one year to the next, in any given subject, tends to be very SMALL

Performance Monitoring with feedback: a holistic approach

KEY PROCESSES 

Departmental Monitoring and Assessment Practices which feed into



Student Reviews 



Scheduled monitoring of a student’s progress across their programme Mutually supporting academic and pastoral functions

using COMMON DATA MINIMUM ACCEPTABLE GRADES

Performance Monitoring with feedback: a holistic approach

BENCHMARKING and TARGET SETTING: Defining Grades 1. MAG: Minimum Acceptable Grade The statistically predicted grade for each subject (Scale A/B – E [E] based on the ALIS trend line) which will not change during the year. Provides an initial benchmark (with associated health warnings) against which a student’s progress can be judged.

2. CAG: Current Achievement Grade The grade a student is currently working at.

Performance Monitoring with feedback: a holistic approach

Defining Grades (contd.) 3. TAG: Target Achievement Grade The grade (above the MAG) a student is considered capable of working at and should aim to achieve. Student Review discussions (where appropriate) would be focused on negotiating strategies to enable students to attain this grade.

4. PG: Predicted Grade The grade that is written on a student’s UCAS form and subsequently changed or confirmed as part of the return to Examination Boards

Performance Monitoring with feedback: a holistic approach

CALCULATING THE MINIMUM ACCEPTABLE GRADE The student MAG for each subject is based on the previous year’s ALIS ‘trend-line’ for that subject Principle = ‘good enough for purpose not statistically flawless’

Example: Lisa Fry has GCSE grades 2A*, 3A, 4B and 1C Total Points = 66

(i.e. 2x8 + 3x7 + 4x6 + 1x5)

AVGCSE = 66 / 10

= 6.6

MAP for subject (15.78 x 6.6) – 62.65 = 41.5 MAG = C (at AS)

Performance Monitoring with feedback: a holistic approach

Technical problem!!! calculation creates number and decimal points which need to be translated into UCAS grades College Conversion Table

A-Level

AS Level

MAG

‘Predicted’ Point Score

‘Predicted’ Point Score

A/B

109 – 140

53 – 60

B

>91 - 44 - 71 - 34 - 51 - 24 -