Stats-Day-WKSP-non-sampling-errors

Evaluate Statistically Based Reports ( AS 3.12)

Workshop 1 Dru Rose (Westlake Girls High School, Ministry of Education Study Award) Rachel Passmore (Endeavour Teacher Fellow) Angela Hawkins (Rosehill College)

What does AS 3.12 cover? i. Polls and Surveys  

Non-sampling errors and survey concerns Sampling error (margin of error, 95% confidence intervals, testing claims made - Workshop A)

ii.

Experimental and Observational Studies

Why teach AS3.12 ? 



   

Statistical Literacy is an essential life-skill to function effectively in the information age (Wallman, 1993; Gal, 2002) Broadens students’ horizons, taking statistical understanding beyond the classroom into the real world (a motivational aspect for students in the trial) Accessible to less academic students External standard Only pre-requisite is AS2.9 (possibly just 1.10) Links to other standards students may be taking (formal inference AS3.10, experiments AS3.11, bi-variate data AS 3.9)

Suggested timeline for 2013 About 4 to 5 weeks  Polls and surveys : • Introduction to non-sampling errors and survey concerns ≈ 6 periods • margin of error, 95% confidence intervals, testing claims ≈ 7 periods  Experimental and observational studies ≈ 7 periods

Evaluating reports involving polls and surveys How should we teach this ? (some approaches that seem to work) Teacher modelling + group work + individual practice  Literacy support: 1. Pre-reading strategies-look at the structure of media reports, skim

2. 3.

reading to “get the gist”, identify and discuss problem vocabulary While-reading strategies-read text aloud asking “worry” questions, record answers on a writing frame Post-reading strategies- write a 3 sentence summary, note 2 positive aspects, note 2 concerns, organise into a short report

How should we teach this ?  Begin with familiar contexts and gradually move to

unfamiliar (examples in resource pack- ask students to find some too)  Use “stories” to illustrate the various non-sampling errors and survey concerns (power-point in resource pack- ask students to add examples of their own)

 Rephrase student comments from informal to formal

statistical language (Encourage students to use the correct terminology in their reports - a step up from AS2.11) 

Allow Time (e.g. one report per lesson – one or two non-sampling errors at time until all covered – can keep non-sampling errors and survey concerns going even when moving on to look at sampling error)

What are polls and surveys?

Do you have an internet capable mobile device with you today? % of New Zealanders carry an internet capable mobile device, survey finds” “

Any concerns about this statement?

A possible student poll : What type of driving licence do you hold ? (full, restricted, learner, none)  Use poll to explore and extend students’ existing statistical knowledge

base regarding: target population, sample, random selection, making an inference

 extend students’ view of sampling from classroom activities to sampling of

the wider NZ population Students need an idea of how key characteristics such as : gender, age, ethnicity, regions of residence, income are distributed across NZ population (A common student misconception : for a NZ survey on health the sample should have equal numbers of smokers and non-smokers)  Look at methodologies used by research companies eg DIGIPOLL,

COLMAR BRUNTON, HORIZON etc.(in power-point on political polls in resource pack)

Sampling

Sampling Error (random process)

Non-sampling Errors

Selection bias

Non-response bias

Self selection

Question effects

Behavioural considerations Interviewer effects

Survey-format effects

Transfer findings C1, L2, S9

Interviewer Effects in Racial Questions In 1968, one year after a major racial disturbance in Detroit, a sample of black residents were asked: “Do you personally feel that you trust most white people, some white people or none at all?” • White interviewer: 35% answered “most” • Black interviewer: 7% answered “most” C1, L2, S10

The resource pack for this workshop has 4 media reports:  Driving infringements –Digipoll (good methodology), question concerns-illustrates structure of media reports

 Road tolls – Horizon on-line panel, potential selection bias, non-response bias, question concerns, funding concerns-”Worry Questions” sheet and writing frame

 Potential for DVT in office workers – a more difficult context but relevant and interesting to students -funding concerns, question concerns, selection concerns- a chance to discuss definitions

 NZ-US military –Digipoll (good methodology), a move to a more difficult context- demonstrate “reading aloud”-how to make meaning from context-need to discuss background re ANZUS, NZ non-nuclear stance

Experiments & Observational Studies

What is an Experiment? • Participants randomly allocated to treatments • Variable of interest is measured and results compared between treatment groups • Causal claim can be made Dru Rose, Angela Hawkins and Rachel Passmore

KEY TERMINOLOGY • Response Variables – the variable measured as the outcome of an experiment. Measure blood pressure to assess effect of new drug.

• Explanatory Variables – the variable that attempts to explain differences in the response variable • Treatment- one or a combination of explanatory variables assigned by the researcher • Confounding Variables – variables connected to the explanatory variable that may be the actual cause of the differences in the response variable.

Dru Rose, Angela Hawkins and Rachel Passmore

How to design a good experiment • Random Allocation – participants randomly allocated to treatments and random order of treatment • Use of a Control Group – use to establish base line effect • Use of a Placebo – dummy, existing or no treatment • Use of Blinding – single and double blinding

Dru Rose, Angela Hawkins and Rachel Passmore

What is an Observational Study? A study which observes differences in the explanatory variable and then assesses whether these differences are related to differences in the response variable A CAUSAL CLAIM CANNOT BE MADE Famous NZ Observational Study is the Dunedin Multidisciplinary Health and Development Study (DMHDS) .A study of 1037 children who were born in Dunedin, in 197273. Where they now live shown below.

Dru Rose, Angela Hawkins and Rachel Passmore

Potential Problems with Observational Studies Confounding Variables

“Happy People Live Longer” • Possible Confounding variables – Health, Life-style, Economic status, Family……… • Why is this an Observational Study? Cannot force people to be happy ! • Problem – a causal claim has been made. Dru Rose, Angela Hawkins and Rachel Passmore

Potential Problems with Observational Studies Extending Results inappropriately

“WOMEN HIT HARDER BY HANGOVERS”

PROBLEM The study observed only undergraduate students at one University so results cannot be extended to cover ALL WOMEN

Dru Rose, Angela Hawkins and Rachel Passmore

Potential Problems with Observational Studies Using the past as a source of data “LEFT HANDED PEOPLE DIE YOUNGER” In early part of twentieth century children forced to write with right hands, hence greater numbers of righthanders. PROBLEM Confounding variables can change over time. Recalled information often unreliable. Dru Rose, Angela Hawkins and Rachel Passmore

COMPARISON OF EXPERIMENTS AND OBSERVATIONAL STUDIES Experiments

Observational Studies

Explanatory Variable specified

Explanatory variable specified

Response variable specified

Response variable specified

Participants randomly allocated to treatments

Participants NOT randomly allocated to treatments, sometimes because it is unethical to do so

Treatment is the only factor that is varied.

Changes in response variable recorded

Changes in response variable recorded

CAUSAL CLAIM CAN BE MADE

CAUSAL CLAIM CANNOT BE MADE

Good experiments include random allocation to treatments, control groups, placebos and use blinding

Good observational studies acknowledge & account for all potential problems

Dru Rose, Angela Hawkins and Rachel Passmore

Writing frame for critically evaluating a report Pre-Reading: “Getting the gist” Read the media report and summarise what it is about in 3 sentences or bullet points.

MEDIA REPORT EVALUATION Media Headline

While reading: “Worry Questions” Read the media report again, asking appropriate “worry questions” as you go. Record your answers in the boxes below:

Source

Observational Unit Explanatory variable Response variable Treatment(s)

Study Type

Original Source Design of study - describe

methodology of observational study

Method Target Group Who was sampled? Selection method

OBSERVATIONAL STUDY - potential problems

Sample size Margin of error Questions asked Evaluate critical components

Key Findings

Claims Made

What is missing?

Critical Evaluation: Discuss 2 good aspects of this report

CONCLUSION -

good or bad study? Any areas of improvement?

Discuss 2 concern

Dru Rose, Angela Hawkins and Rachel Passmore

EXAMPLE OF EXAM STYLE QUESTION QUESTION TWO Extract from Newspaper article “GRADUATING IS GOOD FOR YOUR HEALTH” College graduates feel better emotionally and physically than their high-school drop-out counterparts, a study of over 400,000 US adults reveals. Participants were asked a series of questions including level of education and on how many of the past 30 days they felt physically and emotionally healthy. •Define the explanatory and response variables •Explain whether any of the potential problems that observational studies can suffer from might apply to this study. •Evaluation steps 5, 6 & 7 are concerned with measurements taken and who was studied. Discuss these steps in relation to this study

Dru Rose, Angela Hawkins and Rachel Passmore

POSSIBLE RESPONSES TO QUESTION 1. Explanatory Variable is level of education, Response variable is Emotional & Physical Health 2. Recall of information over last 30 days may be unreliable, generalising to ‘ good for your health’ but study only in US, confounding variables – many other factors contribute to Emotional & Physical Health – social network, economic status, family, illness etc 3. No information provided about who participants were or how they were selected other than that they were from USA. Unclear exactly what questions were asked to determine how healthy students were so not sure if they provided a good indicator of health or not. Responses to these questions might vary depending on when the question was asked & how the student was feeling at that time. They might also vary depending on who was asking the questions or how responses were collected. Dru Rose, Angela Hawkins and Rachel Passmore