Open-Ended Survey Questions - SNAAP

Open-Ended Survey Questions: Non-Response Nightmare or Qualitative Data Dream? American Educational Research Association

April 2013

Angie L. Miller, Ph.D. Amber D. Lambert, Ph.D. Center for Postsecondary Research, Indiana University

Literature Review  There is an increasing trend for requiring colleges

and universities to show measures of their effectiveness (Kuh & Ewell, 2010)  Combination of the struggling economy, funding cuts to

higher education, and the evolution of the traditional higher education model (i.e. distance education, MOOCs, etc.)

 Alumni surveys are an important tool for

assessment, but often have lower responses rates (Smith & Bers, 1987)  Due to bad contact information, suspicion of money

solicitation, and decreased loyalty after graduation

Literature Review  Despite lower response rates, qualitative data

from open-ended questions can still provide rich information from relatively few respondents (Geer, 1991; Krosnick, 1999)  Disadvantages of open-ended questions:  Heavy burden on respondents (Dillman, 2007)  Some personal characteristics, such as language fluency

and positive affect, can impact likelihood of responding to open-ended questions (Wallis, 2012)

Research Questions  Do open-ended responses represent the opinions of the

entire sample? Are some types of respondents more likely to complete these questions? Does question placement on the survey impact responses?

The purpose of this study is to explore whether those with certain demographic and personal characteristics, including gender, age, cohort, number of children, marital status, citizenship, race, current employment status, income, and institutional satisfaction level, are more or less likely to respond to open-ended questions placed at the beginning, middle, and end of an online alumni survey.

Method: Participants  Data from the 2011 administration of the Strategic

National Arts Alumni Project (SNAAP)  Participants were 33,801 alumni from 57 different

arts high schools, arts colleges, or arts programs within larger universities  Sample consisted of 8% high school level, 70%

undergraduate level, and 22% graduate level alumni  38% male, 62% female, .2% transgender  Majority (87%) reported ethnicity as Caucasian  Average institutional response rate: 21%  Only used those who completed the entire survey (i.e. did not drop out before the end) N = 27,212

What is SNAAP?  On-line annual survey designed to assess and improve

various aspects of arts-school education  Investigates the educational experiences and career

paths of arts graduates nationally  Questionnaire topics include:       

Formal education and degrees Institutional experience and satisfaction Postgraduate resources for artists Career Arts engagement Income and debt Demographics

Method: Open-Ended Measures  From beginning of survey (Question 17 of 82)  “Is there anything that [your institution] could have done

better to prepare you for further education or for your career? Please describe.”  From middle (Question 44 of 82)  “Please describe how your arts training is or is not relevant

to your current work.”  From end (Question 80 of 82)  “If there are additional things you would like to tell us

about your education, life, and/or career that were not adequately covered on the survey, please do so here.”

Method: Demographic Measures  Demographic information collected for:  Gender (Categorical- 3)  Age group (Ordinal ranges)  Graduation cohort (Ordinal ranges)  Number of children (Ordinal ranges)  Marital status (Categorical – 4)  Citizenship (Binary)  Race/ethnicity “check all” (Binary – 7 total)  Current employment status (Categorical – 7)  Income (Ordinal midpoints of ranges)  Institutional satisfaction level (Ordinal - 4-point

Likert)

Analyses  Series of 14 chi-squared analyses was done for each

of the 3 open-ended question binary variables  For gender, age group, graduation cohort, number of

children, marital status, citizenship, each race/ethnicity option, and current employment status  3 independent samples t-tests completed for

institutional satisfaction level and each of the 3 open-ended variables  3 non-parametric Mann-Whitney U tests were

completed for income comparisons  Skewed variance violated t-test assumptions

Results: Descriptive Statistics Much higher percentages of responses for beginning and middle questions than for the end question: % Responding 100% 80%

79% 68%

60% 40%

24% 20%

0% Beginning

Middle

End

Results: Chi-Squared Analyses

Results: Chi-Squared Analyses  Females more likely to respond

 Over 50 years of age more likely to respond  Graduating in or before 1990 more likely to respond  Singles less likely to respond  Those with no dependent children more likely to

respond  Unemployed, retired, or with “other” employment status more likely to respond  U.S. citizens more likely to respond  Race: Asians less likely to respond, but those with “other” race/ethnicity more likely to respond

Results: Means and Other Ordinal Comparisons Those answering beginning and end questions were significantly less satisfied with overall institutional experience Didn’t Answer Item: Mean

Answered Item: Mean

t value

df

Effect Size (d)

Near-Beginning Item

3.57

3.40

20.33***

19914.16

.26

Middle Item

3.44

3.45

-.907

27082

.01

Near-end Item

3.46

3.42

4.20***

9767.14

.06

*p