Diapositiva 1
Short Description
Download Diapositiva 1...
Description
The counterfactual logic for public policy evaluation
Alberto Martini 1
Everybody likes “impacts” (politicians, funders, managing authorities, eurocrates) Impact is the most used and misused term in evaluation
Impacts differ in a fundamental way from outputs and results
Outputs and results are observable quantities
Can we observe an impact?
No, we can’t This is a major point of departure between this and other paradigms
As output indicators measure outputs, as result indicators measure results, so supposedly impact indicators measure impacts
Sorry, they don’t
Almost everything about programmes can be observed (at least in principle): outputs (beneficiaries served, activities done, training courses offered, KM of roads built, sewages cleaned) outcomes/results (income levels, inequality, well-being of the population, pollution, congestion, inflation, unemployment, birth rate)
Unlike outputs and results, to measure impacts
one needs to deal with
unobservables
To measure impacts, it is not enough to “count” something, or compare results with targets, or to check progress from baseline
It is necessary to deal with
causality
“Causality is in the mind” J.J. Heckman Nobel Prize Economics 2000
How would you define impact/effect? “the difference between a situation observed after an intervention has been implemented
and the situation that …………………………………………………………………..
?
would have occurred without the intervention
10
There is just one tiny problem with this definition
the situation that would have occurred without the intervention cannot be observed 11
The social science scientific community has developed the notion of potential outcomes “given a treatment, the potential outcomes is what we would observe for the same individual for different values of the treatment” 12
Hollywood’s version of potential outcomes
13
A priori there are only potential outcomes of the intervention, but later one becomes an observed outcome, while the other becomes the counterfactual outcome
A very intuitive example of the role of counterfactual analysis in producing credible evidence for policy decisions
Does learning and playing chess have a positive impact on achievement in mathematics?
Policy-relevant question: Should we make chess part of the regular curriculum in elementary schools, to improve mathematics achievement? Or would it be a waste of time? Which kind of evidence do we need to make this decision in an informed way? 17
Let us assume we have a crystal ball and we know “truth”: for all pupils we know both potential outcomes—the math score they would obtain if they practiced chess or the score they would obtain if they did not practice chess
General rule: what we observe can be very different than what is true
Types of pupils
What happens to them
High ability 1/3
Practice chess at home and do not gain much if taught in school
Mid ability 1/3
Practice chess only if taught in school, otherwise they do not learn chess
Low ability 1/3
Unable to play chess effectively, even if taught in school 20
Potential outcomes
If they do play chess at school
If they do NOT play at school
difference
math test scores High ability Mid ability
Low ability
70 50
20
70
0
40
10
20
0
Try to memorize these numbers: 70 50 40 20 10 21
SO WE KNOW THAT
1.
there is a true impact but it is small
2. the only ones to benefit are mid ability students, for them the impact is 10 points 22
The naive evidence: observe the differences between chess players and non players and infer something about the impact of chess The difference between players and non players measures the effect of playing chess.
DO YOU AGREE? 23
The usefulness of the potential outcome way of reasoning is to make clear what we observe and we do not observe, and what we can learn and cannot learn from the data, and how mistakes are made 24
What we observe High ability
70
Mid ability average=30
Low ability
20
DO YOU SEE THE POINT? 25
Results of the direct comparison Pupils who play chess
Average score = 70 points
Pupils who do not play chess
Average score = 30 points
Difference = 40 points is this the impact of playing chess? 26
Can we attribute the difference of 40 points to playing chess alone?
OBVIOUSLY NOT There are many more factors at play that influence math scores
27
Play chess
Does it haveCSan impact on?
SELECTION PROCESS
Math ability
Math test scores
DIRDIRE DIRECT INFLUENCE
Ignoring math ability could severly mislead us, if we intend to interpret the difference in test scores as a causal effect of chess 28
First (obvious) lesson we learn
Most observed differences tell us nothing about causality We should be careful in general to make causal claims based on the data we observe 29
However, comparing math test scores for kids who have learned chess by themselves and kids who have not
is pretty silly, isn’t it? 30
Almost as silly as: Comparing participants of training courses with non participants and calling the difference in subsequent earnings “the impact of training” Comparing enterprises applying for subsidies with those not applying and call the difference in subsequent investment “the impact of the subsidy” 31
The raw difference between selfselected participants and nonparticipants is a silly way to apply the counterfactual approach the problem is selection bias (pre-existing differences) 32
Now we decide to teach pupils how to play chess in school Schools can participate or not 33
We compare pupils in schools that participated in the program and pupils in schools which did not in order to get an estimate of the impact of teaching chess in school
34
We get the following results Pupils in the Pupils in the non treated schools treated schools
Average score = 53 points
Average score = 29 points
Difference = 24 points is this the TRUE impact? 35
High ability Mid ability Low ability
Schools that participated
Schools that did NOT participate
30%
10%
60% 10%
20% 70%
There is an evident difference in composition between the two types of schools 36
High ability Mid ability Low ability
Schools that participated
Schools that did NOT
30%
10 %
60% 10%
WEIGHTED Average of 70, 50 and 20 = 53
20 % 70 % WEIGHTED Average of 70, 40 and 20 = 29
Average impact = 53 – 29 = 24 37
The difference of 24 points is a combination of the true impact and of the difference in composition If we did not know the truth, we might take 24 as the true impact on math score, and being a large impact, make the wrong decision 38
We have two alternatives: statistically adjusting the data or conducting an experiment The mostt
39
Any form of adjustment assumes we have a model in mind, we know that ability influences math scores and we know how to measure ability
40
But even if we do not have all this information we can conduct a randomized experiment The schools who participate get free instructors to teach chess , provided they agree to exclude
one classroom at random 41
Results of the randomized experiment Pupils in the treated classes in the volunteer schools
Average score = 53 points
Pupils in the excluded classes in the volunteer schools
Average score = 47 points
Difference = 6 points this is very close the TRUE impact 42
Schools that volunteered
High ability
30%
Mid ability
60%
Low ability
10%
Schools that did NOT volunteer
random assignment: flip a coin EXPERIMENTALS CONTROLS average of 70, 50 & 20 = 53 mean of 70, 40 & 20 = 47
Impact = 53 – 47 = 6
43
Experiments are not they only way to identify impacts However, it is very unlikely that an experiment will generate grossly mistaken estimates
If anything, they tend to be biased toward zero
On the other hand, some wrong comparisons can produce wildly mistaken estimates 44
View more...
Comments