March 13—Evaluation research and meta

Significance and effect sizes  What is the problem with just using p-levels to determine

whether one variable has an effect on another?  Don’t EVER just give p-range!  Sample results:  For boys, r (87) = .31, p = .03  For girls, r (98) = .24, p = .14

 Significance test = effect size x study size  Why are effect sizes important?  What is the difference between statistical, practical, and

clinical significance?

What should you report?  2 group comparison—treatment vs. control on anxiety    

symptoms 3 group comparison—positive prime vs. negative prime vs. no prime on number of problems solved 2 continuous variables—relationship between neuroticism and goal directedness 3 continuous variables—anxiety as a function of selfesteem and authoritarian parenting 2 categorical variables—relationship between answers to 2 multiple choice questions

Narrative vs. quantitative reviews  When was the first meta-analysis?  When was the term first used?  What are the advantages of quant reviews?

 What are particular critiques of them?  What are the three basic principles to guide meta-

analysis?

Steps to meta-analysis

1. define your variables/question  1 df contrasts  What is a contrast?

2. Decide on inclusion criteria  What factors do you want to consider here?

3. Collect studies systematically  Where do you find studies?  File drawer problem  Rosenthal’s fail-safe N  # studies needed at p < .05= (K/2.706) (K(mean Z squared) = 2.706)  

Z = Z for that level of p K = number of studies in meta-analysis

 Funnel plot  Rank correlation test for pub bias  What can you do if publication bias is a problem?  Trim and fill  Sensitivity analysis  Weight studies

Fig. 3. Funnel plots of 11 (subsets of) meta-analyses from 2011 and Greenwald, Poehlman, Uhlman, and Banaij (2009).

Marjan Bakker et al. Perspectives on Psychological Science 2012;7:543-554 Copyright © by Association for Psychological Science

3. Calculate effect sizes  If there is more than 1 effect per study, what do you do?  What does the sign mean on an effect size?  What are small, medium, and large effects?

 How can you convert from one to another?  r or d?  http://www.soph.uab.edu/Statgenetics/People/MBeasl

ey/Courses/EffectSizeConversion.pdf

Families of effect sizes  2 group comparisons (difference between the means)  Cohen’s d  Hedge’s g  Glass’s d or delta  Continuous or multi-group (proportion of variability)  Eta squared η2  Partial eta-squared ηp2  Generalized eta-squared η G2  r, fisher’s z, R2, adjusted R2  ω2 and its parts  difference between η2 and R2 family

 Nonparametric effect sizes  Nonnormal data: convert z to r or d  Categorical data:   

Rho Cramer’s V Goodman-Kruskal’s Lambda

 How can you increase your effect sizes?  How can you calculate confidence intervals around your

effect sizes?  http://www.latrobe.edu.au/psy/research/cognitive-and-

developmental-psychology/esci  http://www.cem.org/effect-size-calculator

Interpretation of effect sizes  Recommended for at least most important findings  PS U

 Binomial effect size display (p. 76)  Relative risk  Odds ratio  Risk difference

4. Look at heterogeneity of effect sizes  Chi-square test  I2 (measure based on Chi-square)  Cochran’s Q

 Standard deviations of effect sizes  Stem and leaf plot (p. 671)  Box plot

 Forest plot  What are common moderators you might test? How

would you do that?

Forest plot

5. Combine effect sizes  When should you do fixed vs. random effects?  Should you weight effect sizes, and if so, on what?  How can you deal with dependent effect sizes?

 Hunter and Schmidt method vs. Hedges et al. method  Credibility intervals vs. confidence intervals

6. Calculate confidence intervals/ 7. Look for moderators  What are common moderators you might test?  How do you compare moderators?

“Meta-analysis”  Comparing and combining effect sizes on a smaller 

  

level—when might you want to do this? How would you do it? Average within-cell r’s with fisher z transforms To compare independent r’s: Z = z1-z2/sqrt ((1/n-3) + (1/n-3)) To combine independent r’s: z = z1+z2/2

Write-up  Inclusion criteria, search, what effect size  Which m-a tech and why  Stem and leaf plots of effect sizes (and maybe mods)

 Forest plots  Stats on variability of effect sizes, estimate of pop

effect size and confidence intervals  Publication bias analyses

Side note  Analysis of power (Appendix)

Terms  Evolutionary epistemology  Evidence-based practice  Systems thinking  Dynamical systems approaches  Evaluation research

Issues with evaluation research  What questions are asked?  What methods are used?  What unique issues emerge?

Types of evaluation  Formative  Needs assessment  Evaluability assessment  Structured conceptualization  Implementation evaluation  Process evaluation  Summative  Outcome evaluation  Impact evaluation  Cost-benefit analysis  Secondary analysis  Meta-analysis

Methods used for different ?s  What is the scope of the problem?  How big is the problem?  How should we deliver the program?

 How well did we deliver it?  What type of evaluation can we do?  Was the program effective?

 What parts of the program work?  Should we continue the program?

Evidence based medicine (Sackett et al.)  Convert problem into question  Find evidence  Evaluate validity, impact, applicability

 Integrate patient experience and clinical judgment  Review evaluation

What does the book author  Mean by an “evaluation culture”?  Is it a good thing?

Post spring break  Readings on analyses (some to be emailed out)  Quant article critique is separate from thought paper

(look for questions at end of syllabus)  One more week then rough drafts due