Pennsylvania Value-Added Assessment System (PVAAS):
Communicating PVAAS to Your School Board PVAAS Statewide Core Team Winter, 2011
Session Description
This 60 minute session will provide strategies and approaches for discussing the concept of PVAAS with your local school board, as well as sharing actual PVAAS results. Approaches that have been used by actual districts in Pennsylvania will be discussed. Materials and resources to use as tools for communication about PVAAS to your local school board will be reviewed. Opportunities for questions and discussion will be provided. There are trainer notes included for this presentation.
Agenda
Documents & Suggested Handouts for School Board Members Considerations for Presenting PVAAS to School Boards Examining the School Board Presentation Template Power Point Questions
Suggested Approach for Initial Session
Keep it simple…keep it conceptual Discuss what PVAAS is and is not Start: Focus on Scatter Plots: Achievement vs. Growth Summarize important reports! • Listed in suggested priority order • Suggest starting with Reading and Math only • District Value-Added Report • Value-Added Summary Report • District (Single Grade) Projection Summary Report • Performance Diagnostic Summary Report
Go For The BIG IDEAS!
Supports and Resources to Prepare & Present at PVAAS Sessions with Your Board Members
Show: DVD for School Boards PVAAS for School Board Members by School Board Members Tom Gentzel, PSBA 2 Superintendents School Board Members 12 minutes
Suggested Resources & Documents for Board Members
One-page handout titled “What is PVAAS? Evaluating Growth, Projecting Performance” One-page handout titled “Effective Use of PVAAS Student Projection Reports”
Suggested Resources & Documents for Board Members
Copies of Key District Reports • District Value-Added Summary Reports • District Value-Added Reports • District Projection Summary Reports
Copies of District Scatter Plots • District vs IU region vs State • See next slide for example template
Copy of Presentation PPT
District Scatter Plot – Template Statewide Grade 4 Reading
IU Region Grade 4 Reading
District Grade 4 Reading
PDE PVAAS Webpage •Help Menus •PowerPoint Presentations with Trainer Notes • Resource Materials
• Video Clips • District Case Studies
PDE PVAAS Website, New & Archived Webinars, Intermediate Unit Supports, Other Resources
Key Concepts for EVERY PVAAS Session
Key Concepts: ANY PVAAS Session Achievement AND Growth (PVAAS Scatter Plots)
Key Concept: ANY PVAAS Session PVAAS reporting reflects the
effectiveness of your district’s or school’s Standards-Aligned System PVAAS reporting reflects the district’s or school’s system regarding curriculum, assessment, and instruction
Key Concepts: ANY PVAAS Session There is NO relationship between demographic variables such as socioeconomic status and growth. • There are high achieving schools making high growth; • There are high achieving schools making low growth; • There are low achieving schools making high growth; • There are low achieving schools making low growth. Growth reporting reflects what WE do with students in terms of academic growth in schools/districts. -- VAAS can remove the effects of factors not under the control of the school. (McCaffrey, Lockwood, Koretz & Hamilton, 2003; Ross, Wang, Sanders, Wright & Stringfield, 1999a; Wright, Horn & Sanders, 1997).
Key Concepts: ANY PVAAS Session One of the inherent benefits of the PVAAS analyses is that all scores for all students are included in the analyses. Simplistic approaches are less precise and sometimes biased. While PVAAS does use individual student data to yield value-added reporting, there are NO individual student measures of growth yielded from PVAAS!
Key Concepts: ANY PVAAS Session
No measures of individual student growth! Why not?
Key Concepts: ANY PVAAS Session
The PSSA observed scores are simply ‘snapshots in time’ making comparisons of the observed scores as a measure of growth unreliable.
The error in an estimate for a data set with only one record (one student) is too large to make the estimate meaningful…and
Error depends on variation in the data AND the sample size (the number of student records in a data set).
An estimate of progress, or growth, based on only one student would have a much larger error; therefore, it would be considerably less precise and useful than when considering a group of students.
Key Concept: ANY PVAAS Session PVAAS District and School Value-Added Reports do not provide a measure of teacher effectiveness! Often
people ask about the meaning of these reports, especially in situations where there is only one classroom teacher in a grade level. This is NOT a report on the effectiveness of the classroom teacher, even in that situation! Teacher effectiveness reporting would be done using a different statistical methodology than the one used for reporting on district or school effectiveness. Additional data would be collected as well.
DECISION POINTS Who? - What? -When? - How?
Decision Point: Audience What are the backgrounds of school board members? Do the board members have experience with growth data/PVAAS? Do the board members have any background knowledge/foundation in Statistics?
Decision Point: Intended Goal for Session (s)
PVAAS Overview?
Review of Actual District/School Data?
Part of Planning Process? • Strategic Planning • School Improvement Planning
Decision Point: Focus of Data
District? School? Grade? Subgroups? Which subjects? • Math, Reading, Science, Writing
Remember, less is more!
Decision Point: Reports Needed District-wide Focus • Scatter Plots
State to IU to District
• District Value-Added
Think ahead if there is only one building per grade
• Value-Added Summary (shows individual schools)
• District (Single Grade) Projection Summary Report • (Option) Performance Diagnostic Summary
Examples of Reports
Talking About Your Results Think about your message before sharing your results!
What do the results mean? Do you have a plan of action regarding the areas needing improvement (Red)? What is your plan of action for attending to the Green areas?
Talking About Your Results Think about your message before sharing your results!
What do the results mean? Do you have a plan of action regarding the areas needing improvement (Red)? What is your plan of action for attending to the Green areas?
Decision Point: Session Facilitators(s)
Principal Team of Principals District Administrator(s) District Administrator(s) + Principal(s) Support from Intermediate Unit PVAAS Contacts Support from PVAAS Statewide Team for PDE – call-in for questions
Decision Point: Next Steps
Session Focus • Awareness- Follow with a second session to review actual district/school reports • Analysis – share plan of action to use data with faculty, making sure to pair PVAAS data with other data available on your students
Result: Plan of ACTION!
OTHER CONSIDERATIONS
PVAAS Projections are an Approved Way for Schools to Make AYP (Growth Model)
Preparing for Parent Requests
There is NO student level growth data in PVAAS. This is a common request from parents. What is available at the student level? Projections! • % likelihood of a student reaching Basic, Proficient and/or Advanced levels on future PSSA Examples of districts that share PVAAS student projections with parents: Corry Area SD, Derry Township SD Plan in advance for how to respond such requests
Questions? Comments?
Presenting PVAAS to an Individual School Board Power Point Template with Trainer Notes
Pennsylvania Value-Added Assessment System (PVAAS): A Compass for Improvement Presented at [District] School Board Meeting [Date]
DVD for School Boards PVAAS for School Board Members by School Board Members Tom Gentzel, PSBA 2 Superintendents School Board Members 12 minutes
Two Types of PVAAS Information Looking Forward/Planning… PVAAS Projection Reports For Individual Students and Cohorts of Students
Looking Back/Evaluation… Value-added Growth Reports For Cohorts of Students
Today
NO individual student growth measure!
PVAAS IS: • Analysis of existing assessment data – Longitudinal PSSA scores in available content areas • Math, Reading, Science, Writing
– Provides reliable estimates
• A measure of a school’s effectiveness • Critical Relationship:
Growth Data vs. Achievement Data
PVAAS: •
Is not another test
•
Is not a comparison of single test scores
•
Critical concept: Does not provide a measure of growth for an individual student
Current Data Collection Approaches High Achievement Average Achievement
Low Achievement Achievement is based on each student’s ending point.
PVAAS Adds Another Variable To Achievement…Progress Below Growth Standard
At or Above Growth Standard
Progress is based on each student’s starting point.
Viewing Achievement and Progress Together Scatter Plots: Grades 4-8 Reading and Math
The Scatter Plot Coordinate System Vertical Axis PSSA Percent Proficient or Advanced
Achievement and Growth - 4th Math
% Proficient or Advanced
AYP Percent Proficient Target
Growth Standard Boundary
One Year Growth 56
0 -20
0
PVAAS Growth Measure
20
Horizontal Axis PVAAS Growth Measure
Scatter Plot Quadrants and Their Meanings Quad 2 Adequate Achievement Negative Growth
Achievement and Growth - 4th Math
% Proficient or Advanced
Quad 3 Below Adequate Achievement Negative Growth
Quad 1 Adequate Achievement Positive Growth
56
Quad 4 Below Adequate Achievement Positive Growth
0 -20
0
PVAAS Growth Measure
20
District Scatter Plots District vs. IU Region vs. State Math and Reading Grades 4 – 8
District Scatter Plot – Grade 4
District Scatter Plot – Grade 5
District Scatter Plot – Grade 6
District Scatter Plot – Grade 7
District Scatter Plot – Grade 8
Viewing Achievement and Progress Together Scatter Plots: Grade 11 Reading and Math
Grade 11 Scatter Plots The PSSA-PVAAS Scatter Plots visually display the following data from the statewide distribution of PSSA results for the most current school year.
Achievement • The mean predicted performance of the cohort. • The actual mean performance of all students in the state is displayed for reference
Growth • School Effect ≈ Observed Mean – Predicted Mean • Standardized School Effect = School Effect relative to its error – displays degree of confidence
The following two slides explain the meaning of each of the regions of an 11th grade scatter plot.
The Scatter Plot Coordinate System for Grade 11 Vertical Axis Mean Predicted Scaled Score
2010 Mean Scaled Score
Growth Descriptor Boundaries (±2SEs)
Horizontal Axis Growth Measure in Standard Errors
Scatter Plot Regions and Their Meanings for Grade 11
District Scatter Plot – Grade 11
Overview of PVAAS Reporting
PVAAS Reporting
District Level Reporting School Level Reporting Grade Level Reporting • • • •
Reading (Grades 4-8, 11) Math (Grades 4-8, 11) Science (Grades 4, 8, & 11) Writing (Grades 5, 8, & 11)
Subgroup Reporting (ex. IEP, ED, ELL) Student Level Reporting • Projections to Future PSSA Performance • Reading, Math, Science, Writing • Grades vary based on subject area
Have all of our groups of students experienced one year of academic growth during the last school year? Value-Added Growth Reports Grades 4-8, Reading and Math
PVAAS Value-Added Growth Descriptors – Grade 4 – 8 Math and Reading The PA Growth Standard specifies the minimal acceptable academic gain from grade to grade for a cohort of students. The Growth Standard is met when the students in a school or district on average maintain their relative position when compared to the 2006 statewide distribution of student scores (2006 is the base year). Estimated mean NCE gain equal or greater than zero. There is evidence that the school has met or exceeded the Growth Standard. All schools can achieve this rating. Estimated mean NCE gain less than zero by one standard error or less. There is slight evidence that the school has not met the Growth Standard. Estimated mean NCE gain less than zero by between one and two standard errors. There is moderate evidence that the school has not met the Growth Standard. Estimated mean NCE gain more than two standard errors less than zero. There is significant evidence that the school has not met the Growth Standard.
PVAAS Value-added Growth Descriptors Grade 4 – 8, Math and Reading Favorable Indicator Caution Indicator Stronger Caution Strongest Warning
District/School Value-Added Report Example
Value-Added Summary Report Example
Have all of our groups of students experienced one year of academic growth during the last school year?
Value-Added Reports Science, Writing Grade 11, Reading and Math
Writing, Science & Grade 11 Reading/Math Growth Descriptors Progress is defined by the district or school effect, which is a measure of the difference between observed performance and predicted performance. Progress is significantly above the progress of the average district in the state. Progress is not detectably different from the progress of the average district in the state. Progress is significantly below the progress of the average district in the state.
Writing, Science & Grade 11 Reading/Math Growth Descriptors Below
Below Expectations
NDD
Not Detectably Different from Expectations
Above
Above Expectations
District/School Value-Added Report Example
How well are we supporting our highest achieving students, with our average performing students, and with our lowest achieving students?
Performance Diagnostic Summary Report
Performance Diagnostic Growth Descriptors, Grade 4-8, Math/Reading Below Growth Standard
Met Growth Standard Exceeded Growth Standard
Performance Diagnostic Growth Descriptors Science, Writing & Grade 11 Math/Reading Below Expectations Not Detectably Different from Expectations Above Expectations
Performance Diagnostic Summary Report Example 1
Performance Diagnostic Summary Report Example 2
What is the likelihood that a student will be reach proficiency on a future PSSA? What is their likelihood of reaching the Basic or Advanced levels?
PVAAS Student Projections
Student Projection Methodology • A separate longitudinal modeling process focused on the performance scores of individual students. • The model uses all of the available data (both math and reading) to project future PSSA performances in Math, Reading, Science and Writing. • Projections can be calculated for an individual student or groups of students.
PVAAS Student Projections Projections to Basic, Proficient, Advanced Math and Reading: 3 4 or 5 4 5 or 6 5 6 or 7 6 7 or 8 7 8 8 11
Writing: 3 5 4 5 5 8 6 8 7 8 8 11
Science: 3 4 4 NA 5 8 6 8 7 8 8 11
Grade PSSA
Grade PSSA
Grade PSSA
Last Taken
Last Taken
Last Taken
PVAAS Student Projection Report
PVAAS Projections for Groups of Students Student Search
Identified Students Meeting Search Requirements
District (Single Grade) Projection Summary Report
PVAAS Help Menus
Questions: PVAAS Materials or Statewide Implementation
[email protected] 717-606-1911
PVAAS Report Web Site https://pvaas.sas.com
www.pde.state.pa.us
333 Market Street Harrisburg, PA 17126