Survey Research: Structured Interviewing and Questionnaires

Chapter 4

10/28/2013 12:15

Geog 3250

1

Thoughts on researcheradministered surveys?  Keep a list of things not to do in a researcher-

administered survey from this tongue-in-cheek video:

10/28/2013 12:15

Geog 3250

2

Structured Interviews Method

Question Order

Context

Structured interview schedule interview

strict

usually face-toface

Survey

questionnaire

strict

various

In-depth Interview

Interview guide, and the interviewer

flexible to keep it conversational

usually face-toface

10/28/2013 12:15

Instrument

Geog 3250

3

Structured Interviews: errors and interviewer effects Method

“Errors”

Opportunities for Clarification

Structured minimize interview

minimize

high: but clarify in exact same way in all situations

Survey

minimize

minimize

low: due to social distance

In-depth Interview

use to sustain rapport and keep the conversation meaningful

No “errors”, just opportunities for deeper understanding

high: clarification may be the source of meaningful conversation

10/28/2013 12:15

Attitude towards interviewer effects

Geog 3250

4

Structured Interviews and “errors”  Variation due to error can come from:  Intra-interviewer variability: an interviewer is not consistent in asking questions or recording answers (with the same respondent or a different one).  Inter-interviewer variability: lack of consistency in asking questions or recording answers between different interviewers.

10/28/2013 12:15

Geog 3250

5

Structured Interviews: responses Method

Response mode

Data recording

Structured interview

mostly closed-ended

on the interview schedule

Survey

mostly closed-ended

on the questionnaire

In-depth Interview

open-ended and conversational

Field notes or audio recording

Example of close-ended question; data recorded on the document 10/28/2013 12:15

Geog 3250

6

Interview Contexts  Work with your neighbour to fill out this table - as

many as you can think of per cell Method of Administration

Strengths

Limitations

Researcher-Administered

Telephone e.g. with CATI Face-to-Face Self-Administered Drop-off, mail-back Hand-out, hand-back Email, 0nline 10/28/2013 12:15

Geog 3250

7

Conducting Structured Interviews: Basic Points  Schedule – Interviewers should be familiar with the

interview schedule.  Introduction – Respondents should be provided with

a credible rationale for the research.  Rapport – Interviewers should try to develop rapport

with interviewees but refrain from being too chatty (unlike semi-structured or unstructured) 10/28/2013 12:15

Geog 3250

8

Conducting Interviews: Basic Points, cont’d. Asking Questions  The question should be asked

exactly as stated.

 Small changes to wording can

make a big difference

 Similarly, the answers should be

recorded as exactly as possible.

 Questions should be asked in

the order they are given on the interview schedule.

10/28/2013 12:15

Geog 3250

9

Conducting Interviews: Basic Points, cont’d.  show cards - to provide detailed

information, e.g., Likert scale options  appropriate if  

long list of possible answers; same set of answer categories applies to several questions.

10/28/2013 12:15

Geog 3250

10

Conducting structured interviews: Basic Points Leaving the interview  The interviewer must remain

professional and have a set response to disengage respondents so that they may do more interviews (i.e., if researcher is motivated to have large sample size) 10/28/2013 12:15

Geog 3250

Sometimes they want to be done and move on, sometimes it is the other way around…

11

Conducting (survey) interviews: Basic Points Training  Good interviewing is a skill

set to be developed. It usually requires considerable training and supervision to become good at it…but you have to start somewhere!

10/28/2013 12:15

Geog 3250

12

Questionnaires (self-administered)  Questionnaires are essentially

structured interviews without an interviewer.

 They involve filling out a form which

is then returned to the researcher, often by mail or in person.

 Because the respondent has to read

the questionnaire without the aid of an interviewer, it has to be very clear and easy to follow – layout matters!!

10/28/2013 12:15

Geog 3250

Example of Statistics Canada selfadministered survey question

13

Questionnaires (self-administered)  Compared to interviews, questionnaires tend to  have fewer ‘open’ questions, because ‘closed’ questions are easier to answer;  have simple designs, so the respondent can complete all sections without difficulty;  be short, to avoid respondent becoming discouraged.

10/28/2013 12:15

Geog 3250

14

Questionnaires (self-administered)  Advantages over structured interviews:  Cheaper, quicker, more convenient to administer  No interviewer effects, including errors in administering the instrument  Social desirability bias seems to be reduced. 

Respondents are more likely to give genuine answers to ‘sensitive’ questions, e.g., those regarding sexual practices, drug use, criminal activity, etc.

10/28/2013 12:15

Geog 3250

15

Questionnaires (self-administered)  Disadvantages  Researcher cannot explain the question – re: validity.  Greater risk of missing data, lack supervision.  Researcher cannot probe.  Difficult to ask a lot of questions.  Difficult to ask different kinds of questions, especially ‘open’ ones and questions requiring a filter.  Order effects hard to control – may answer out of order  Literacy issues  Designated respondent may not have completed questionnaire. 10/28/2013 12:15

Geog 3250

16

Questionnaires (self-administered)  Online survey  Growing in popularity  Finding participant lists…random?  Email surveys 

Tend to be aimed at smaller, more homogenous groups

10/28/2013 12:15

Geog 3250

This one is free and can be restricted to UWO usernames

17

10/28/2013 12:15

Geog 3250

18

Questionnaires (self-administered)  Researcher-driven diaries  

A form of questionnaire Participants record their feelings, perceptions, actions, etc. on a form shortly after they occur.  Participants should be given explicit instructions on how to complete the diary, the time periods for recording responses, and the types of experiences to be recorded.  Used for quantitative or qualitative research.

 ‘structured’ 

like a regular questionnaire with closed questions, or

 ‘free text’ 

experiences written down without fixed responses, like answers to ‘open’ questions

10/28/2013 12:15

Geog 3250

19

Diaries  Advantages of diaries over interviews and questionnaires:  Fairly accurate data, re: frequency or time spent on behaviours of interest  Fairly accurate data about sequencing of behaviours  Good for getting data on sensitive matters, e.g., sexual practices  Disadvantages  More expensive than personal interviews  Attrition: people may tire of using them  Details may not be recorded quickly enough. 

Errors, omissions

10/28/2013 12:15

Geog 3250

20

Respondent Problems - Structured Interviews and Questionnaires  Response sets  the respondent is not motivated to provide a genuine response – answer all the same without reading

I am not paying attention to each question as I respond

10/28/2013 12:15

Geog 3250

21

Respondent Problems - Structured Interviews and Questionnaires Acquiescence: trying to please the researcher  e.g., respondent agrees just to be ‘cooperative’  partial antidote: mix items so that they have logically

opposite positions

Do you like this image? 10/28/2013 12:15

Geog 3250

22

Respondent Problems - Structured Interviews and Questionnaires Social desirability: avoiding appearance of being bad/unlikeable

I hope these are the answers she is looking for.

 Answers are not sincere, e.g.

questions on racism or the environment

10/28/2013 12:15

Geog 3250

23

Respondent Problems - Structured Interviews and Questionnaires Laziness or boredom: answers just to get the end of the process  E.g., respondent checks all the same answer to get it done

Assume all respondents might be this lazy?

10/28/2013 12:15

Geog 3250

24

Feminist (and other) Critiques  Structured interviews and

questionnaires - exploitative.  asymmetrical power relationship between the researcher and the respondent.

 Link to Ethics: Increased attention

given to the rights of research respondents in recent years, e.g., privacy rights, the right to end the interview at any time, etc.

Power imbalances are not always this overt

 Is this enough? 10/28/2013 12:15

Geog 3250

25

Questionnaire Design  Identify at least one problem with each question in the following

questionnaire snippet:

10/28/2013 12:15

Geog 3250

26

Open or Closed Questions?  Closed Questions  Present the respondent with a set of answers from which to choose  E.g., see previous slide  Open Questions  Response decisions are left completely to the respondent  Answers must be coded into to numerical data 

Tend to be used in quantitative research

 E.g., “What do you think about using alternatives to

chemical pesticides on your lawn”?

10/28/2013 12:15

Geog 3250

27

Open Questions  Advantages  Respondents can answer in their own terms  Allow for unusual, unanticipated responses  No suggested answers, so responses may expose knowledge and be more genuine  Good for exploring new or changing areas of research  Answers may lead to fixed-choice responses

10/28/2013 12:15

Geog 3250

28

Open Questions  Disadvantages  Time-consuming to record answers  Answers have to be coded  In self-administered questionnaires, respondents may balk at the request to write long answers  Recording inaccuracies for verbal answers 10/28/2013 12:15

Geog 3250

Why did your degree take longer than the funding period? •Not really sure. I worked away on it diligently but the analysis just took longer than expected, I guess. • There are worse places to be than grad school. • Closer or more effective supervision was the primary reason. • Too much procrastination and family issues. What code/categories? Can one response go in multiple categories? 29

Coding Open Questions  Post-coding  Data are gathered, then themes or categories of behaviour are discerned.  e.g., an open question on the legalization of marijuana might have codes ‘very hostile opposition’, ‘indifferent’, etc.  Each code would then be assigned a number, and attributed to respondent

10/28/2013 12:15

Geog 3250

30

Coding Open Questions, cont’d  Pre-coding  Themes or categories of behaviour are decided upon before the data are gathered.  May be done with fixed-response items (i.e. closedended) 

e.g., ‘What is your view on the legalization of marijuana? Are you strongly opposed, opposed, neither opposed nor in favour, in favour, or strongly in favour?’

 Data may be gathered freely through open responses

and sorted into pre-set categories afterward. (What if there are responses that seem to require a new code?) 10/28/2013 12:15

Geog 3250

31

Coding Open Questions, cont’d  Basic principles:  The categories must not overlap.  The list of categories must be exhaustive.  

Must cover all possibilities. ‘Other’ category is usually required.

 Coders should be provided with a ‘coding frame’ or

‘coding manual’ that establishes clear rules about how codes should be applied.

10/28/2013 12:15

Geog 3250

32

Closed Questions  Advantages  Easy to process answers  Standardization allows comparison of answers  Fixed responses may help clarify what the question means  Easier and quicker for the respondent to complete  Reduced bias in recording answers 

No interpretation required by researcher

10/28/2013 12:15

Geog 3250

33

Closed Questions  Disadvantages  Loss of spontaneity and

authenticity - relevant answers may be excluded from choices provided.  Antidote: ‘Other’ category with open area to elaborate (but you may want to code it afterwards!)  Defining categories that do not overlap.  Antidote: Pre-test as open-ended to establish appropriate and distinct categories.  Difficult to make forced-choice answers exhaustive

10/28/2013 12:15

Geog 3250

24. Why do you oppose the construction of more wind turbines in your community? Bird/bat collisions Economic costs outweigh benefits Health threats Impacts on farm and domestic animals Unfair siting process Other _____________________ 34

Closed Questions, cont’d  Disadvantages (cont’d)  Respondents may differ in their interpretation of the wording of fixed responses. 

e.g., the meaning of ‘strongly’ in ‘strongly agree’

 Respondents may not find a fixed response that they feel

applies to them.  Large numbers of closed questions can reduce rapport in interviews 

Antidote: include open-ended responses even if you may not code them?

10/28/2013 12:15

Geog 3250

35

Types of Questions  Personal factual questions  e.g., age, occupation, number of cars owned

 Factual questions about others,

events etc?  Avoid these: makes respondent feel

they are being tested and look stupid  Antidote: Questions about perceptions are better – puts them in position of being “the expert”. 10/28/2013 12:15

Geog 3250

This “survey” tests, but it is for education, not research purposes

36

Types of Questions, cont’d.  Beliefs  e.g., “Do you believe climate change has happened globally over the last 50 years?”  Attitudes  Very common, subjective, “taste”, or perception  use of Likert scales  E.g., “Do you feel we should do more to mitigate the effects of global climate change.”

“Gabe Kotter” tests attitudes where participant is the “expert” regardless of how they choose

Likert “agree” scale, often used to create an index from multiple items 10/28/2013 12:15

Geog 3250

37

General Rules for Designing Questions  Keep the research questions/hypotheses in mind.  Focus on exactly what you want to know.  Be specific.  e.g., “How many children 18 and under are living at home?” vs. “How many children do you have?”

 Put yourself in the position of the respondent  How would you answer the question? 10/28/2013 12:15

Geog 3250

38

Specific Rules for Designing Questions AVOID:  ambiguous terms  Even commonly used terms may be vague, e.g., does ‘income’

include tips? Investments income? Does it mean after tax income?

 long questions: respondents may lose track of what the

question is asking  multiple phenomena in the same question  E.g., “Do you agree your network of closest friends and family is helpful and supportive?”

10/28/2013 12:15

Geog 3250

39

Specific Rules for Designing Questions AVOID:  several very open-ended questions (do unstructured interviews instead?)  e.g., ‘What do you think of the prime minister?’

 leading questions  e.g., ‘Do you feel the monarchy, which costs taxpayers over $5 million each year, be abolished?’

10/28/2013 12:15

Geog 3250

40

Specific Rules for Designing Questions AVOID:  questions that include negatives, especially double negatives.  e.g., ‘Do you oppose the rule stating that laptops are not

allowed in classrooms?’

 jargon/technical terms. And be sure that respondents

have the knowledge needed to answer the question.  E.g., ‘How would you rate the level of social capital in your

neighbourhood?’ 10/28/2013 12:15

Geog 3250

41

Specific Rules for Designing Questions  Ensure symmetry between a closed question and its

answers.  e.g., avoid:

 Responses should be ‘very satisfied, somewhat satisfied,

…

 Ensure the answers are balanced.  e.g., avoid: Don’t let instructors design their own course evaluations?

10/28/2013 12:15

Geog 3250

42

Specific Rules for Designing Questions  Memory  Beware of expecting respondents to recall what they’ve done or observed in detail.  e.g., number of hours spent online per week  ‘Don’t know’ option  Consider avoiding this(e.g., use “neither agree/disagree” instead if you do not want to force them to choose)  Thought to prevent forcing the respondent to choose, but it actually serves to offer an easy way out.

10/28/2013 12:15

Geog 3250

43

Exercise  Design a survey question that violates one rule of

questionnaire design identified above Text 335633 and your question to 37607 Or Respond at this link

NB You need to be in slideshow mode (full screen) for any of the links to work

10/8/2013 10:15

Baxter Geog 2153

YES peek back up the notes…

poll results here 44

Specific Rules for Designing Questions  Question order  All respondents should receive questions in the same order (unless testing for order effects).  Asking a particular question may affect the responses given to subsequent questions.  Early questions should be directly related to the announced research topic.  Questions likely to be of interest to respondents should be asked early in the interview.  ‘Sensitive’ questions (e.g., ‘Have you ever been a victim of sexual assault?’) should be asked well into the schedule, but not at the very end.

10/28/2013 12:15

Geog 3250

45

Specific Rules for Designing Questions  Question order, cont’d  Questions should be grouped logically and into related sections.  Within a group of questions, general questions should precede specific ones.  Opinion and attitude questions should precede behaviour questions.  Even if a respondent provides an answer to a question before it is asked, the question should be repeated at the appropriate time. 

i.e., there could be question order issues.

10/28/2013 12:15

Geog 3250

46

Designing the Questionnaire  Provide a clear layout and

presentation  Choose vertical or horizontal answers, as appropriate  Vertical tends to be more

pleasing to the eye

Vertical layout

Horizontal layout

10/28/2013 12:15

Geog 3250

47

Designing the Questionnaire  Provide clear instructions on how to respond.  Is more than one response allowed?  Should the choice of answer be circled? Underlined?  Keep a question and its answer together on the same

page.

10/28/2013 12:15

Geog 3250

48

Vignette Questions  Presenting people with one or more

scenarios and asking them how they would respond  Anchor the choices in a realistic situation.  Creates distance between question and respondent.  Hopefully more candid response.  Weakness: how people say they would act

in a particular situation may be very different from how they would really act.

 Can provide useful information, or at least

Michelle and Philip bought their home 5 years ago and 2 years ago wind turbines were installed nearby….Please provide your thoughts about their situation in the following 5 questions.

a starting point for further research. 10/28/2013 12:15

Geog 3250

49

Pilot Studies  Used to test whether individual items or the

instrument as a whole operate well.  Used with open questions to generate closed questions for subsequent studies.  Provide interviewers with experience in administering the instrument.  Can be used to ensure that there is variability in the answers given.

10/28/2013 12:15

Geog 3250

50

Pilot Studies  Can identify questions that are embarrassing,

uninteresting, etc.  Can identify questions that are difficult to understand.  Can be used to determine the adequacy of the instructions.  Can be used to determine whether the instrument has satisfactory flow.

10/28/2013 12:15

Geog 3250

51

Using Existing Questions  If certain questions have worked well for other

researchers, they may be appropriate for your study.  The original source must be cited.  Allows results from different studies to be compared.  Questions may be modified to suit one’s purposes better.

10/28/2013 12:15

Geog 3250

52