2-Problem_Formulation_Oslo_2011_1

C2 Training: May 9 – 10, 2011 Introduction to Systematic Reviews

The Campbell Collaboration

www.campbellcollaboration.org

Systematic Review Methods • Systematic reviews are a form of research – secondary observations – in which studies are the unit of analysis • Follow basic steps in the research process • Aim to minimize bias and error – But SRs are not immune to bias and error (not a

panacea) C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Stages of a research synthesis (Cooper, 1982) The seminal article outlining five stages of a research review is by Harris Cooper: • Cooper, H.M. (1982). Scientific guidelines for conducting integrative research reviews. Review of Educational Research, 52, 291-302. • Cooper, H. M. (2009). Research synthesis and metaanalysis: A step-by-step approach. Thousand Oaks, CA: Sage. 3 C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Stages of a research synthesis (Cooper, 1982) • Problem formulation – Clarifying your questions and writing a protocol – Set explicit inclusion/exclusion criteria

• Data collection – Literature search – Information-gathering from studies

• Data evaluation – Criteria for including and excluding studies – Assessing study quality 4 C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Stages of a research synthesis (Cooper, 1982) • Data analysis and interpretation – Integrating the effects from collected studies – Interpreting analysis results

• Report preparation – Narrative, statistics, graphs, tables

5 C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

What’s required? • A team with – Substantive expertise – Methodological expertise – Statistical expertise

– Information retrieval expertise • Time and money – SRs are labor intensive – $50-$150K depending on scope, complexity, and number

of studies C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Protocols for SRs • A detailed protocol (plan) for the SR should be developed

and made available to readers (Higgins & Green, 2008; Moher et al., 2009) – Protocols increase transparency, limit ad hoc decisions • The review process is iterative and plans may change during the process – The final report should document and explain changes made (deviations from the protocol) C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Stages of a research synthesis • Most of the work involved in conducting a review is not spent

in statistical analysis. • The scientific contribution of the final product is dependent

on all stages of the review and not just the statistical analysis stage.

8 C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

An example of the stages of a review We will illustrate each of the stages by considering the C2 systematic review on Multisystemic Therapy (MST): Littell, J.H., Popa, M., & Forsythe, B. (2005). Multisystemic Therapy for social, emotional, and behavioral problems in youth aged 10 – 17. (Campbell version)

9 C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Stage 1 Problem Formulation

10 C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Problem formulations • Conceptual issues arise in attempts to combine results of

studies that vary (to some degree) in their methods, treatments, samples, outcome measures – Apples, oranges, and other fruits – Parallel problems in studies of individuals (no 2 people are identical and there is no “average” person) • Given variation in primary research – What should be included in a synthesis? – How should we synthesize results?

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

SRs vary in scope • Specific, narrow questions – Useful for testing effects of specific treatments • Broad, global questions – Useful for generating new knowledge

• Identify common elements of effective programs

(Lipsey, 2008) • build better intervention theories to guide program development and evaluation design (Lipsey, 1997)

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Scope of Systematic Reviews • Not limited to questions about effects – Can address trends, epidemiology, accuracy of diagnostic

and prognostic tests • Not limited to randomized controlled trials or quantitative data – Qualitative synthesis (e.g., meta-ethnography, narrative analysis of qualitative research reports) – Mixed/multiple methods synthesis (e.g., Thomas, Harden, et al. on progams to combat childhood obesity) C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

What kinds of research questions can be asked in a systematic review? • Questions about intervention effects: – What are the effects of x intervention on y

outcomes for z populations/problem? – Variations on this theme (e.g., differences in effects of interventions x1 vs x2)

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Kinds of research questions (continued) • Questions about associations – How does x1 relate to x2 for population z? (direction and

strength of correlation) – Variations on this theme (e.g., differences in relation of x1 and x2 between populations z1 and z2 ) • Diagnostic/Prognostic questions – Which test (A vs. B) is a better predictor of y? – Which test (A vs. B) is a better predictor of y for z1 vs. z2 populations? C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Problem formulation • To assess the impacts of MST on out-of-home living

arrangements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families.

16 C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Problem formulation • What research evidence will be relevant to the problem or

hypothesis of interest in the synthesis? – Studies where youth (age 10-17) with social, emotional, and/or

behavioral problems were randomised to licensed MST programs or other conditions (usual services or alternative treatments).

17 C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Problem formulation • Define the (a) variables and (b) relationships of

interest so that relevant and irrelevant studies can be distinguished.

Multisystemic Therapy (MST) is a multi-faceted, short-term, home- and community-based intervention for families of youth with severe psychosocial and behavioral problems. Based on social ecological and family systems theories, and on research on the causes and correlates ofserious antisocial behavior in youth (Henggeler 1998, Henggeler 2002a), MST is designed to address complex psychosocial problems and provide alternatives to out-of-home placement of children and youth.

18 C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Problem formulation (continued) • Measures of behavioral, psychosocial, and family outcomes

were examined. – Behavioral outcomes included antisocial behavior (as measured by arrest or

conviction of a criminal offense), drug use (self-reports and drug tests), and school attendance. – Psychosocial outcomes included measures of psychiatric symptoms (on standardized scales),school performance (teacher reports), peer relations (self-reports and parent or teacher reports),and self esteem. – To assess the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families.

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Step two: Establish study inclusion and exclusion criteria • The PICOS framework: – Population/Participants (problems/conditions) – Interventions (if applicable)

– Comparison group (e.g., absolute vs. relative effects,

counterfactual conditions) – Outcomes (primary and secondary outcomes, acceptable outcome measures) – Study Design (should be fit for purpose) Geographic area, time, language, other criteria C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Participants: Who is included in the sample of subjects? • Why 10 - 17 only? • Exclude special populations?

• Exclude studies with only males or females? • Include families of these children? • What is the rationale for the choice?

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Interventions: How do we define MST? Comparisons: What is the comparison?

• What kinds of programs are considered MST? • What types of treatments are the “right” comparisons to

MST?

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

What are the Outcomes? • Research question states: – Placement in out-of-home living arrangements

– Crime and delinquency – Behavioral outcomes – Psychosocial outcomes

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Study Designs: What types of study designs are relevant for our review? • If the research question is the assessment of the impacts of

MST on out-of-home living arrangements, crime and delinquency, and other behavioral and social outcomes for youth and families, what designs are most appropriate?

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

What studies are relevant? • Will we attempt to generalize to all studies done? • Will we look only at “high quality” studies? • Will we look at both published and unpublished? • Will we set a time period for the studies?

• Will we look at only randomized controlled trials or will we

include quasi-experimental studies? Why?

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Study eligibility criteria • Usually focus on inclusion criteria, with few

exclusion criteria • Studies should not be excluded based on: – Sample size (or statistical power considerations) – Publication status

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

What kinds of studies should be included? Garbage in, garbage out • Synthesis of invalid studies produces invalid

conclusions • What constitutes credible evidence? – Depends on the question • Need to set clear standards – Based on methodological principles – Relevant for topic and context C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org

Small group exercise: Problem formulation • Form a working group of 2 – 5 people and develop a

preliminary research question for a comprehensive review • Elect a recorder for the group who will report back to the whole • Each group will share the research question, and any challenges that they will face in studying this question • See the first exercise on problem formulation

C2 Training Materials – Oslo – May 2011

www.campbellcollaboration.org