Systematic reviews: Social science perspective

Systematic reviews in the social sciences Sarah Whitehead, Cardiff Institute of Society and Health

Outline • Why do systematic reviews in the social sciences? • Examples of types of reviews:  Correlates of physical activity  Effectiveness of tobacco harm reduction approaches  Barriers and facilitators to tobacco harm reduction

• Overview of different types of evidence generated • Lessons learnt

Why do systematic reviews in the social sciences? Chaos in the brickyard • • • • • •

Goal of scientific research to discover explanations Research findings to be assembled like bricks to create edifices Initially bricks carefully made and only those required for the edifice produced Over time, brick making became an end in itself until the land was flooded with bricks, many of them random and useless Now we need to search through so many bricks it’s difficult to find proper ones for the task “Saddest of all, sometimes no effort was made even to maintain the distinction between a pile of bricks and a true edifice”1 1. Forscher 1963

Why do systematic reviews in the social sciences? • Systematic reviews allow us to make sense of the chaos in the brickyard • They allow us to:  develop a comprehensive understanding of research to date  identify areas of evidence that can be translated into policy and practice  determine which areas are in need of further research

Examples of types of reviews

Example 1: correlates of physical activity • Research question:  What factors are associated with physical activity participation among adolescent girls?

• Suitable design:  Correlates review of quantitative studies reporting a measure of physical activity and at least one potential correlate

Example 1: methods • 4 databases searched, manual searches of journals • Inclusion criteria:      

Quantitative research design Published 1999-2003 Participants in age range 10-18 years Results available for females separately Articles published in English language Any type of physical activity

Example 1: methods (cont’d) • Potential correlates classed as demographic/biological, psychological, behavioural, social/cultural, or physical environmental • Variables classed as related/ not related to physical activity • Strength of association noted1 • Variables only included if studied three or more times2 • Quality of physical activity measure coded2:  Self-report of poor or unknown reliability/validity  Self-report with acceptable reliability/validity  Acceptable objective measure 1. Cohen 1988 2. Sallis et al. 2000

Example 1: findings • 50 published papers yielding 51 independent samples • Majority of studies (63%) carried out in US • Mean sample size (females only) = 1,280 (range 22-9,039) • 41 (80.4%) cross-sectional in design • Majority (61%) used self-report with acceptable reliability/validity

Example 1: findings • Demographic variables:  Non-white ethnicity (-)  Age (-)  SES (+)

• Psychological variables:    

Enjoyment (+) Perceived competence (+) Self-efficacy (+) Physical self-perceptions (+)

• Behavioural variables:  Smoking (-)  Organised sport involvement (+)

• Social/cultural variables:  Family/parental support (+)  Father’s physical activity (+)

• No consistent trends for environmental variables

Example 1: conclusions • Several variables related with PA across 3+ studies • Many potential correlates understudied • Several weak designs and measures of poor or unknown validity • Most promising modifiable correlates:  Family support  Positive psychology  Organised sport

Example 2: effectiveness of tobacco harm reduction approaches • Research questions:  How effective are tobacco harm reduction approaches with the intention of quitting (i.e. ‘cutting down to quit’ or ‘reduction to stop smoking’), with and without assistance?  How effective are long-term harm reduction approaches without the prior intention of quitting (i.e. reducing consumption without the aim of quitting), with and without assistance?

• Suitable design:  Two effectiveness reviews of experimental studies of tobacco harm reduction approaches, e.g. randomised controlled trials, controlled trials

Example 2: methods • 21 databases, 28 websites, range of 'snowballing’ techniques • Included populations:  People of all ages wanting to quit smoking but unable to do so abruptly / not willing to quit but wanting to reduce/ stop temporarily  Participants in cut down to quit/ reduction/ temporary abstinence interventions

• Included interventions:    

Pharmacotherapies licensed for cutting down/ temporary abstinence Other non-tobacco nicotine containing products, e.g. e-cigarettes Behavioural support or counselling Self-help

Example 2: methods • Data extraction to capture:    

Study design, setting and participants Intervention details Methods of analyses Outcomes

• Quality appraisal to assess:  Methodological quality  External validity

 Narrative synthesis and meta-analyses to summarise findings

Example 2: included studies and types Cut down to quit

Cut down without quitting

•

•

•

•

11 studies  10 RCTs or quasi-RCTs  1 uncontrolled before & after Quality  Generally moderate Applicability  Only one UK study

• • •

45 studies  29 RCTs or quasi-RCTs  3 non RCTs  2 controlled before & after  10 uncontrolled before & after  1 secondary analysis 1 systematic review Quality  Generally moderate Applicability  5 UK studies  6 in UK-comparable countries

Example 2: types of evidence examined • Effectiveness of pharmacotherapies in helping people cut down to quit or cut down indefinitely without the aim of quitting • The effectiveness of behavioural support, counselling or self-help • Whether different approaches had differential impacts on different groups

Example 3: barriers and facilitators to tobacco harm reduction approaches • Research question:  What are the barriers and facilitators to implementing tobacco harm reduction approaches, including users’ and providers’ perspectives?

• Suitable design:  Mixed methods review of qualitative and quantitative views and opinions studies

Example 3: methods • 21 databases, 28 websites, range of 'snowballing’ techniques • Included populations:  Smokers interested in THR or who take part in THR interventions, or those who provide opinions or experiences regarding THR approaches  Service providers, healthcare personnel and policy makers who may deliver/ commission/ refer smokers to THR interventions

• Included study types:  Views studies relating to cut down to quit/ long term reduction/ temporary abstinence intervention studies previously identified  Views studies relating to tobacco harm reduction approaches in general

Example 3: methods • Data extraction to capture:  Study design, setting and participants  Methods and analytical approach  Outcomes / key themes

• Quality appraisal to assess:  Methodological quality  External validity

• Thematic synthesis of views, experiences and perspectives • Mapping of whether effectiveness studies addressed the identified barriers and facilitators to THR approaches

Example 3: included studies and types • 41 studies     

22 cross-sectional surveys 13 qualitative studies 3 mixed methods studies 2 process evaluations 1 longitudinal study

• Quality  Generally moderate

• Applicability  9 UK studies and 5 UK applicable

Example 3: types of evidence examined • Factors that might act as barriers or facilitators to tobacco harm reduction approaches • Smokers’/ their families’ and healthcare professionals' views and experiences on whether specific tobacco harmreduction approaches are perceived to have a differential impact on particular groups • Smokers’/ their families’ and healthcare professionals' views and experiences on the potential or actual unintended consequences from adopting a harmreduction approach

Summary of different types of evidence that systematic reviews can generate • Correlates of behaviours or outcomes • Effectiveness of interventions / intervention type studies • Views and opinions • All can be useful in informing policy and practice and identifying evidence gaps where further research is needed

Lessons learnt • Develop a solid search strategy • Identify appropriate tools for the type of review being undertaken, for example quality assessment measures and approaches to synthesising the data; • Most importantly – work with a team of experienced information specialists

Acknowledgement Sarah Whitehead’s role at Cardiff Institute of Society and Health is supported by the Welsh Assembly Government’s National Institute for Social Care and Health Research