DEFINE what is to be evaluated Simon Hearn, ODI, London
Part 2 of 8 AEA Coffee Break Webinars 2013
The Rainbow Framework
Why do we need to start with a clear definition?
Source: Hobbies on a Budget / Flickr
Three evaluation tasks in DEFINING Identify potential unintended results
Develop program theory or logic model
Develop initial description
1. Develop initial description
Peak Experience
Thumbnail description
Aims & Objectives
Vision & Mission
2. Develop program theory or logic model
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
INPUTS • Apples • People at risk of poor health
IMMEDIATE RESULTS
SHORT-TERM RESULTS
LONGER-TERM RESULTS
• Apples eaten
• Improved nutritional status
• Improved health
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
RESULTS AREA
INDICATOR
MEANS OF VERIFICATION
ASSUMPTIONS/ RISKS
GOAL:
Decreased sick days
School and work records
Good status largely due to nutrition
Adequate levels of Vitamin C
Sample blood tests
Apples retain Vitamin C
Apples replaced by apple cores
Visual inspection
Apples eaten on location
Required numbers delivered
Delivery receipts
Required numbers bought
Purchase receipts Visual inspection
Improved health
PURPOSE: Improved nutritional status
OUTPUT: Apples eaten
ACTIVITIES: Deliver apples
ACTIVITIES: Buy apples
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
Outcomes chain showing possible alternative causal paths Improved health
Improved vitamin C levels
Improved quercetin levels
Decreased Body Mass Index
Apples eaten (whole or juice)
Red apple skin eaten
Apples eaten instead of junk food snacks
Apples delivered to schools
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
Context
Mechanism
Outcome
Healthy people at high risk of gastric cancer
Quercetin from red apple skin (or red onion) protect cells from free radicals
Reduced risk of cancer
Smokers or people with regular exposure to radiation
Chemicals in cigarette smoke or radiation damages DNA in healthy cells
Continued increased risk of cancer
Individuals with vitamin deficiency through malnutrition
Sufficient vitamin C absorbed when juice from apples (or oranges) consumed
Stronger immune system
Individuals with vitamin deficiency through excessive alcohol
Alcohol destroys vitamin C
No change in vitamin C level
Individuals with obesity due to excessive snacking on high fat, high calorie foods
Decreased calorie intake due to substitution of apples (or carrot sticks)
Reduced obesity and related conditions
Decrease sugar intake through substitution of apples
No change in obesity but lower risk of diabetes
Individuals with obesity due to genetic condition
Context
Mechanism
Outcome
Healthy people at high risk of gastric cancer
Quercetin from red apple skin (or red onion) protect cells from free radicals
Reduced risk of cancer
Smokers or people with regular exposure to radiation
Chemicals in cigarette smoke or radiation damages DNA in healthy cells
Continued increased risk of cancer
Individuals with vitamin deficiency through malnutrition
Sufficient vitamin C absorbed when juice from apples (or oranges) consumed
Stronger immune system
Individuals with vitamin deficiency through excessive alcohol
Alcohol destroys vitamin C
No change in vitamin C level
Individuals with obesity due to excessive snacking on high fat, high calorie foods
Decreased calorie intake due to substitution of apples (or carrot sticks)
Reduced obesity and related conditions
Decrease sugar intake through substitution of apples
No change in obesity but lower risk of diabetes
Individuals with obesity due to genetic condition
http://betterevaluation.org/resource/example/rubrics-oakden
3. Identify potential unintended or negative results
Source: www.smarttoolkit.net
Negative program theory
Key informant interviews
Six Hats Thinking
Risk assessment
Defining – Framing iteration Initial description Agree scope of evaluation
DEFINE
Discuss evaluation purpose
FRAME
FRAME
Develop logic model
Refine description
DEFINE
DEFINE
Define evaluation questions FRAME
http://betterevaluation.org/plan/define facebook.com/betterevaluation @bettereval
Simon Hearn Research Fellow, ODI
[email protected]
