PPT Presentation Template: This PPT includes all slides to present a district or building level overview of PVAAS. This was used with a districtwide K-12 faculty group in a small school district, where one grade level represented one building versus multiple buildings on a district level report.
PVAAS: Evaluating Growth, Projecting Performance
Today’s Session
Concept of PVAAS Overview of Reporting • • • •
Math and Reading, Grades 4-8 Math and Reading, Grade 11 Science, Grades 4, 8 and 11 Writing, Grades 5, 8 and 11
School District Data Questions and Answers
What Is PVAAS?
PVAAS IS: • Analysis of existing assessment data – Longitudinal PSSA scores in available content areas • Math, Reading, Science, Writing
– Provides reliable estimates
• A measure of a school’s effectiveness • Critical Relationship:
Growth Data vs. Achievement Data
PVAAS: •
Is not another test
•
Is not a comparison of single test scores
•
Critical concept: Does not provide a measure of growth for an individual student
Current Data Collection Approaches High Achievement Average Achievement
Low Achievement Achievement is based on each student’s ending point.
PVAAS Adds Another Variable To Achievement…Progress Below Growth Standard
At or Above Growth Standard
Progress is based on each student’s starting point.
Viewing Achievement and Progress Together Scatter Plots: Grades 4-8 Reading and Math
The Scatter Plot Coordinate System Vertical Axis PSSA Percent Proficient or Advanced
Achievement and Growth - 4th Math
% Proficient or Advanced
AYP Percent Proficient Target
Growth Standard Boundary
One Year Growth 56
0 -20
0
PVAAS Growth Measure
20
Horizontal Axis PVAAS Growth Measure
Scatter Plot Quadrants and Their Meanings Quad 2 Adequate Achievement Negative Growth
Achievement and Growth - 4th Math
% Proficient or Advanced
Quad 3 Below Adequate Achievement Negative Growth
Quad 1 Adequate Achievement Positive Growth
56
Quad 4 Below Adequate Achievement Positive Growth
0 -20
0
PVAAS Growth Measure
20
SD Scatter Plots Grade 4 Reading
Math
Insert District Scatter Plots Here
SD Scatter Plots Grade 5 Reading
Math
Insert District Scatter Plots Here
SD Scatter Plots Grade 6 Reading
Math
Insert District Scatter Plots Here
SD Scatter Plots Grade 7 Reading
Math
Insert District Scatter Plots Here
SD Scatter Plots Grade 8 Reading
Math
Insert District Scatter Plots Here
Viewing Achievement and Progress Together Scatter Plots: Grade 11 Reading and Math
Grade 11 Scatter Plots The PSSA-PVAAS Scatter Plots visually display the following data from the statewide distribution of PSSA results for the most current school year.
Achievement • The mean predicted performance of the cohort. • The actual mean performance of all students in the state is displayed for reference
Growth • School Effect ≈ Observed Mean – Predicted Mean • Standardized School Effect = School Effect relative to its error – displays degree of confidence
The following two slides explain the meaning of each of the regions of an 11th grade scatter plot.
The Scatter Plot Coordinate System for Grade 11 Vertical Axis Mean Predicted Scaled Score
2010 Mean Scaled Score
Growth Descriptor Boundaries (±2SEs)
Horizontal Axis Growth Measure in Standard Errors
Scatter Plot Regions and Their Meanings for Grade 11
SD Scatter Plots Grade 11 Reading
Math
Insert District Scatter Plots Here
The TWO Types of PVAAS Information Looking Forward/Planning… Looking Back/Evaluation…
PVAAS Projection Reports For Individual Students and Cohorts of Students
Value-added Growth Reports For Cohorts of Students
Today NO individual measure of student growth.
PVAAS Reporting
Student Projections Wouldn’t it be great to know the likelihood that a student will be proficient on a future PSSA?
Student Projection Methodology
A separate longitudinal modeling process focused on the performance scores of individual students. The model uses all of the available data (both math and reading) to project future PSSA performances in Math, Reading, Science and Writing.
Projections can be calculated for an individual student or groups of students.
How do I know that the projection is accurate?
The projection is precise, and is created using a model that has been reviewed and approved by four different peer review panels and the GAO (US Government Accountability Office).
The most recent 2008 growth model proposal to USDOE includes information regarding the statistical model and projection reliability study. (http://www.ed.gov/admins/lead/account/growthmodel/pa/index.html)
Recent studies have confirmed that the PVAAS projections (even as far as 3 years into the future) are more reliable at looking at the future performance of a student than the most recent PSSA score.
PVAAS Student Projection Report
PVAAS Projections for Groups of Students Student Search
Identified Students Meeting Search Requirements
PVAAS Reporting Projections to Basic, Proficient, and Advanced Math and Reading: 3 4 or 5 4 5 or 6 5 6 or 7 6 7 or 8 78 8 11
Grade Last Tested
Writing: 35 45 58 68 78 8 11
Grade Last Tested
Science: 34 4 NA 58 68 78 8 11
Grade Last Tested
Projection Summary Reports
What is it? • Report that summarizes the numbers and percents of students in likelihood ranges of performing at the proficient level on a future PSSA exam. How might a school use this report? • Intervention Planning • Resource Allocation • Strategic Planning • School Improvement Planning Cautions • This report provides another indicator about likelihood of future performance.
School (Single Grade) Projection Summary Report
Value-Added Wouldn’t it be great to know if your students made one year’s growth for the year of schooling they experienced last year? In Math and Reading (4-8 and 11)? In Science (4, 8 and 11)? In Writing (5, 8 and 11)?
PVAAS Growth Methods Testing
Subjects – Grades
Methodology
PSSA in consecutive years
Math & Reading – 4-8
Growth Standard
PSSA not in consecutive years
Writing – 5, 8, 11 Science – 4, 8, 11 Math & Reading – 11
Predictive
Value-added Measure of Growth Math and Reading (Grades 4-8)
Comparison to a State Growth Standard The Growth Standard specifies the minimal acceptable academic gain from grade to grade for a cohort of students. The use of a Growth Standard creates the possibility that ALL schools can demonstrate appropriate growth.
PVAAS Value-Added Growth Descriptors – Grade 4 – 8 Math and Reading The PA Growth Standard specifies the minimal acceptable academic gain from grade to grade for a cohort of students. The Growth Standard is met when the students in a school or district on average maintain their relative position when compared to the 2006 statewide distribution of student scores (2006 is the base year). Estimated mean NCE gain equal or greater than zero. There is evidence that the school has met or exceeded the Growth Standard. All schools can achieve this rating. Estimated mean NCE gain less than zero by one standard error or less. There is slight evidence that the school has not met the Growth Standard. Estimated mean NCE gain less than zero by between one and two standard errors. There is moderate evidence that the school has not met the Growth Standard. Estimated mean NCE gain more than two standard errors less than zero. There is significant evidence that the school has not met the Growth Standard.
PVAAS School Report Grades 4 through 8 - Math and Reading
What Value-added Gain Means! Cohort’s Achievement 7th grade 66.6
Cohort’s Achievement 6th grade 60.7 0
10
20
30
40
50
60
70
80
90
100
NCE Scale State Mean = 50
PVAAS Gain ≈ Change in position from 6th to 7th grade ≈ 66.6 – 60.7 ≈ 5.9 NCE points
PVAAS Growth Methods Testing
Subjects – Grades
Methodology
PSSA in consecutive years
Math & Reading – 4-8
Growth Standard
PSSA not in consecutive years
Writing – 5, 8, 11 Science – 4, 8, 11 Math & Reading – 11
Predictive
Value-added Measure of Cohort Growth Math and Reading (Grade 11) Science (Grades 4, 8, and 11) Writing (Grades 5, 8, and 11) Based on a predictive methodology…
A predicted score is calculated for each student; Mean of predicted scores is calculated for students in a cohort; Mean of observed scores of students in the cohort; School Effect is the difference between the Observed and the Predicted Scores.
PVAAS Predictive Analysis 1. We collect all of the individual student data that is available for a minimum of two years. 2. All of these data are used to develop individual student profiles. 3. PVAAS then calculates a Predicted Performance Value based on the student profile for each student. Year 1 PSSA
Year 2 PSSA
Year 3 PSSA (11th)
7th
8th
Predicted
Actual
Reading
1280
1300
1290
??
Math
1330
1305
1320
??
PVAAS Ratings of School Effect Remember: School Effect Mean of Student Scores – Mean of Predicted Scores So, if the students perform as expected, School Effect 0. The ratings are in terms of the expected value of 0.
Not Detectably Below
Above
Different
-2 SEs
0
+2 SEs
Standard Error (SE) is a measure of variation in the prediction.
Writing, Science & Grade 11 Reading/Math Growth Descriptors Progress is defined by the district or school effect, which is a measure of the difference between observed performance and predicted performance. Progress is significantly above the progress of the average district in the state. Progress is not detectably different from the progress of the average district in the state. Progress is significantly below the progress of the average district in the state.
PVAAS School Report Grades 11- Math and Reading Grades 4, 8 & 11 – Science Grades 5, 8 & 11 – Writing
Summary of PVAAS Growth Methods Testing
Subjects – Grades
Methodology
PSSA in consecutive years
Math & Reading – 4-8
Growth Standard
PSSA not in consecutive years
Writing – 5, 8, 11 Science – 4, 8, 11 Math & Reading – 11
Predictive
Summary: One Year’s Growth Growth Standard Method Math and Reading – Grades 4 through 8
Cohort maintains its achievement position. Predictive Method Math & Reading – Grade 11 Writing – Grades 5, 8, 11 Science – Grades 4, 8, 11 Cohort’s actual performance is as expected.
PVAAS Performance Diagnostic Report The Key is the RED Whisker!
Growth Standard Methodology What the whiskers tell us… Exceeded Growth Standard; More than One Year’s Growth Green
Met the Growth Standard; Made One Year’s Growth Yellow
Growth Standard (One Year’s Growth)
Below Growth Standard; Less than One Year’s Growth Rose
Predictive Methodology What the whiskers tell us… ABOVE Expectations -Observed Score Was Greater Than Predicted Score NDD – No Detectable Difference Between Observed Score and Predicted Score
Observed – Predicted (One Year’s Growth)
BELOW Expectations -Observed Score Was Less Than Predicted Score
Performance Diagnostic Report
Greater Detail – based on % of students
Click on Hot Keys for lists of students.
Student Lists from Performance Diagnostic Reports
Your District and School Data
PA’s Intent for PVAAS
Another Tool for Continuous Improvement
Measure of Growth to Add to Achievement/Status Measures Not to Be Used in Isolation
Questions and Feedback
Questions: PVAAS Materials or Statewide Implementation
[email protected] 717-606-1911
PVAAS Report Web Site https://pvaas.sas.com
www.pde.state.pa.us
333 Market Street Harrisburg, PA 17126
