Pro Bono Publico: Avoiding Child Welfare Data Abuse

Pro Bono Publico: Avoiding Child Welfare Data Abuse Panel on Meaningful Measurement in the Context of Litigation and Consent Decrees Daniel Webster, MSW, PhD

Center for Social Services Research, University of California at Berkeley

Isabel Blanco

Deputy State Child Welfare Director, South Carolina

Andy Barclay

Fostering Court Improvement, Atlanta, GA

Susan Smith, PhD

Casey Family Programs Data Advocacy Group Presentation Originally Created by: Barbara Needell, MSW, PhD Melissa Correia, MSW Emily Putnam-Hornstein, MSW, PhD Bryn King, MSW

AAPWA Preconference Symposium Demonstrating Sustained Well Being of Children and Youth in a Compliance Environment CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

San Diego, CA September 9, 2012

The Current Placement System* (highly simplified) the foster care system

CHILD IN

a bunch of stuff happens

CHILD OUT

*adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Tracking Child Welfare Outcomes rate of allegations/ substantiated allegations

home-based services vs. out of home care

reentry to care permanency through reunification, adoption, or guardianship

counterbalanced indicators of system performance

length of stay stability of care CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

use of least restrictive form of care

positive attachments to family, friends, and neighbors

Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill: Jordan Institute for Families

There are three kinds of lies: Lies, Damned Lies and Statistics

^

Abused Statistics

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Public Data: Putting it All Out There • PROS: – greater performance accountability – community awareness and involvement, encourages public-private partnerships – ability to track improvement over time, identify areas where programmatic adjustments are needed – Region/region and region/county collaboration – transparency

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Public Data: Putting it All Out There • CONS: – Potential for misuse, misinterpretation, and misrepresentation – Available to those with agendas or looking to create a sensational headline – Misunderstood data can lead to the wrong policy decisions – “Torture numbers, and they’ll confess to anything” (Gregg Easterbrook)

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Six Ways to Abuse Data (without actually lying!): 1) Compare Apples and Oranges 2) Use ‘snapshots’ of Small Samples 3) Rely on Unrepresentative Findings 4) Logically ‘flip’ Statistics 5) Falsely Assume an Association to be Causal 6) Rely on Summary Statistics

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

1) Compare Apples and Oranges Two doctors in San Diego, CA… Doctor #1

Doctor #2 Lowest mortality rate?

2/1000

20/1000

What if Doctor #1 is a podiatrist and Doctor #2 is a cardiologist? CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

2) Data Snapshots… Crime in San Diego, CA… Number of Crimes Period 1: 76 Period 2: 51 No change. Average = 73.5 Crime jumped by 49%!! Period 3: 91 Crime dropped by 16% Period 4: 76

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

2) Data Snapshots… A politician recently claimed that 92.3% of all the jobs lost since Obama took office were lost by women.*

*http://www.forbes.com/sites/susanadams/2012/04/12/romney-claims-on-womens-job-loss-paint-a-misleading-picture/

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

3) Unrepresentative Findings… Survey of people in San Diego, CA… 90% of respondents stated that they support using tax dollars to build a new football stadium. The implication of the above finding is that there is overwhelming support for the stadium…

But what if you were then told that respondents had been sampled from a list of season football ticket holders?

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

4) Logical “Flipping”… Headline in U-T San Diego: 60% of violent crimes are committed by men who did not graduate from high school.

“Flip” 60% of male high school drop-outs commit violent crimes?

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

5) False Causality… A study of San Diego residents makes the following claim: Adults with short hair are, on average, more than 3 inches taller than those with long hair. Hair Length

X

Height

Gender

Finding an association between two factors does not mean that one causes the other… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

6) Reliance on Summary Statistics The average human has one breast and one testicle.*

Disaggregation is required for this analysis to be really useful.

* ~Des McHale www.quotegarden.com/statistics.htm

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Disaggregation • One of the most powerful ways to work with data… • Disaggregation involves dismantling or separating out groups within a population to better understand the dynamics • Useful for identifying critical issues that were previously undetected Aggregate Permanency Outcomes Race/Ethnicity County

Age Placement Type

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

In San Diego County, entry rates are highest for infants. Entries are higher for African American children for almost all ages. Rate of First Entry into Out of Home Care in 2011 by Age, Gender, & Race 45 40 35 30

Rate per 1,000 Children

Total

25

Black Female

20

Black Male 15

White Female

10

White Male

5 0