Module Four: Analyzing and Reporting Assessment Results
Short Description
Download Module Four: Analyzing and Reporting Assessment Results...
Description
Module 4: Analyzing & Reporting Assessment Results
© 2013 Christie Cruise-Harper, PhD All Rights Reserved
The purpose of this module is to assist you with analyzing the data you gathered for the program/service you assessed in the 2013 – 2014 academic year.
Department Office of Multicultural Programs Personal Counseling
Health and Wellness
Program Multicultural Scholars Program/Dean’s Award Program Mandated Substance Abuse Assessment Program HEROs Program
Campus Ministry and Community Service Student Involvement
KLILV Sophomore Colloquium
Athletics
SAAC and Champs
Residence Life
Resident Assistant Program
CAB/MSG/CSI
Student Life/Associate Dean of Students Habitat for Humanity
In module 1 you developed learning outcomes for your program/service using Bloom’s Taxonomy as a guide. Module 2 allowed you to strengthen those learning outcomes and guided you through the process of choosing learning activities. Module 3 assisted you with choosing appropriate assessment tools/methods for the learning outcomes and learning activities you established. This module will help you with analyzing and reporting the assessment data.
Learning Outcome
Learning Activity
Assessment
Analysis
What should your students be able to do?
What activity will help your students achieve the learning outcome?
How will you know whether students have achieved the learning outcome?
What will you do with all the information you collected from your assessment plan?
Module 1
Module 2
Module 3
Module 4
Assessment results must be analyzed to determine if student learning outcomes were met. Data is analyzed for context, understanding and to draw conclusions. Analysis of data gives the information meaning.
Taken from Academic Program Assessment: Tools & Techniques for Program Improvement
Determining how to organize, synthesize, interrelate, compare and present the assessment results are all part of analyzing the data. Assessment data can be compared to findings from previous assessments, baseline data and existing criteria.
Taken from Academic Program Assessment: Tools & Techniques for Program Improvement
Quantitative ◦ Also known as “empirical research” ◦ Refers to any research based on something that can be accurately and precisely measured. ◦ Collects numerical data in order to explain or predict a particular phenomena.
Taken from University of Wisconsin – Madison’s Ebling Library & R. Ouyang’s Basic Inquiry of Quantitative Research
For more information visit: http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225
Qualitative ◦ Refers to any research based on something that cannot be accurately and precisely measured. ◦ Collects narrative data to gain insights into a particular phenomena.
Taken from University of Wisconsin – Madison’s Ebling Library & R. Ouyang’s Basic Inquiry of Quantitative Research
For more information visit: http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225
It’s all about the numbers!
There are four types of quantitative research methods:
◦ Descriptive: collecting data for hypotheses testing* ◦ Correlational: determining whether and to what degree a relationship exists ◦ Cause-Comparative: establishing the cause-effect relationship ◦ Experimental: establishes the cause-effect relationship, but manipulates the cause
See notes section for more detailed information.
Taken from R. Ouyang’s Basic Inquiry of Quantitative Research
The following are common methods of data collection in quantitative research: ◦ ◦ ◦ ◦ ◦
Surveys and Questionnaires Structured Interviews Observation or Interaction Analysis Secondary Data or Content Analysis Experiments
See notes section for more detailed information.
In quantitative research there are two ways in which data are analyzed: Descriptive Statistics ◦ Procedures used to describe a given collection of data. ◦ The purpose is to describe the sample at hand-the collection of cases that we have examined.
Inferential Statistics ◦ Procedures that let us generalize our findings beyond the particular sample at hand to the larger population represented by that sample. Taken from Diekhoff, G.M.(1996). Basic Statistics for the Social and Behavioral Sciences.
Most Student Life assessment projects do not seek to generalize its findings to the entire Maryville University student body. Because our goal is to learn about the sample at hand, descriptive statistics will be the focus of the quantitative data analysis for module 4. There are three types of descriptive statistics to provide you with an overview of your data: ◦ Central Tendency Measures* ◦ Variability Measures ◦ Frequency and Percentages*
More on descriptive statistics later. Before beginning any data analysis, you must first identify the level of measurement associated with your quantitative data. There are four levels of measurement: ◦ ◦ ◦ ◦
Nominal Ordinal Interval Ratio (Scale)
Nominal: basic classification data; do not have meaningful numbers attached to them, but are broader categories Ordinal: have numbers attached to them and the numbers are in a certain order, but there are not equal intervals between the numbers Interval: have equal intervals between the numbers; the distance between attributes have meaning Ratio: have equal intervals between the numbers; there is an absolute zero that is meaningful
Taken from: http://www.uni.edu/commstudies/researchmethods/chapterfour1.html
Once you have decided on your data collection method, decided on the level of measurement for your variables, and collected the data, you are ready to begin analyzing the data. There are two software programs I recommend and they are available on campus: Qualtrics SPSS (Statistical Package for the Social Sciences)
The following quantitative data analysis procedures are used to describe the data and can be done in Qualtrics and SPSS: ◦ ◦ ◦ ◦
Data Tabulation (Frequency Distributions, Percentiles) Descriptive Data (Central Tendency) Data Disaggregation Moderate and Advanced Analytical Methods
For a detailed description of these analyses visit: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/ OR Chapters 1-3 in Diekhoff’ s Basic Statistics for the Social and Behavioral Sciences
Surveys and questionnaires can be developed in Qualtrics and the data can be analyzed within the software. There are tutorials to assist you with analyzing your data. Crosstabs:
http://qualtrics.com/university/researchsuite/reporting/crosstabs/about-cross-tabulations/
Understanding Statistics:
http://qualtrics.com/university/researchsuite/reporting/crosstabs/understanding-statistics/
SPSS ◦ Below are several links to resources to assist you with using SPSS. http://www.slideshare.net/sspink/seminar-on-spss http://www.youtube.com/watch?v=eTHvlEzS7qQ http://www.youtube.com/watch?v=HT0Skh2UP1U&feat ure=related http://calcnet.mth.cmich.edu/org/spss/toc.htm
In addition to a narrative about your data analysis, quantitative results are presented in the following ways: ◦ Charts ◦ Graphs ◦ Tables
The following websites provide examples, in APA, of how quantitative results are presented: https://owl.english.purdue.edu/owl/resource/560/19/ https://owl.english.purdue.edu/owl/resource/560/20/
Tables and graphs can be created in Qualtrics. For more information visit: ◦ http://qualtrics.com/university/researchsuite/reporting/re porting-beta/tables/#AboutTables ◦ http://qualtrics.com/university/researchsuite/reporting/re porting-beta/graphs/
Graphs can also be created in SPSS. For directions visit:
◦ http://julius.csscr.washington.edu/pdf/spss.pdf ◦ http://academic.udayton.edu/gregelvers/psy216/spss/gra phs.htm ◦ http://www.ats.ucla.edu/stat/spss/seminars/SPSSGraphics /spssgraph.htm
I want to know how you feel!
It is important to achieve empathic understanding to comprehend the participant’s experience with a minimum of distortion or bias; The researcher must attempt to recognize their own personal prejudices, stereotypes, myths, assumptions and other thoughts or feelings that may cloud the perception of other people’s experiences; Knowledge of other’s experience cannot be assumed regardless of familiarity with their subcultural landscape; and Do not expect participants to hold the same values as you. M. Ely, M. Anzul, T. Friedman, D. Garner, A.M. Steinmetz in Doing Qualitative Research: Circles within Circles
Qualitative Research Methodologies ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦
Basic Interpretive Qualitative Study* Grounded Theory Phenomenology Case Study Ethnography Postmodern Research Critical Qualitative Research Narrative Analysis
For more information visit: http://www.fctl.ucf.edu/researchandscholarship/sotl/creatingsotlprojects/implementingmanaging/qu alitativeresearchtypes.php
Participant Observation Interviewing (formal & informal) Focus Groups Document Analysis Logs (notes/reflections from observations and interviews) Audio and videotaping On-going data analysis
M. Ely, M. Anzul, T. Friedman, D. Garner, A.M. Steinmetz in Doing Qualitative Research: Circles within Circles
Coding is the process of organizing data into chunks or segments before making meaning of the information. Coding involves taking text data or pictures gathered during data collection, segmenting it into categories, and labeling those categories with a term, often a term used by the actual participant.
Taken from: Creswell, J.W.(2009). Qualitative, Quantitative and Mixed Methods Approach, chapter 9, Qualitative Procedures.
Begin the coding process by first reviewing your learning outcomes as a reminder of what you are assessing. Your coding scheme will be based on your learning outcomes. ◦ For example: Undergraduate students who
participate in the Multicultural Scholars Program will be able to describe their talents, strengths and social group memberships.
Coding involves assigning a word, phrase, number or symbol to each coding category. ◦ For example: describe talents and strengths
Codes can be pre-set or emergent. You should have both.
◦ Pre-set: A list of codes created in advance by the researcher based on the research question, learning outcomes, or conceptual framework. ◦ Emergent: Ideas, concepts, actions and meanings that come up from reading and analyzing the data that are not in the pre-set codes.
Coding will serve as a system to help you to organize your data.
For more information and an example of coding visit: http://programeval.ucdavis.edu/documents/Tips_Tools_18_2012.pdf
Once you have gone through all documents and coded them, they can now be gathered into families of codes or categories. Materials are sorted by these categories, identifying similar phrases, patterns and relationships. As you code and categorize the data, look for the interrelationships among categories. Sorted materials are examined to isolate meaningful patterns. Identified patterns are used to create themes.
It is best to start any report of qualitative results with an overview of how data were processed and coded. The results are presented as “findings”. The findings are organized by themes with substantial evidence that links to the themes included in the findings (i.e. quotes from participants).
So now what?
Closing the loop - using the assessment results for program improvement. Assessment Plan
Implement Changes
Data Collection
Identify Changes
Data Analysis
Assessment Report
The findings from your quantitative and/or qualitative research will yield rich information that will be included in your assessment report. These findings will provide you with insights into what’s working with your program/service and areas of improvement. From these findings you will make recommendations for improvement of your program/service.
You may find it helpful to create a program improvement plan to transform the recommendations made into actions for improvement. Recommendation: Action step(s)
What action steps must be completed to implement the recommendation?
Estimated implementation date
When does the program expect to begin to implement the action steps?
Estimated completion date
When does the program expect the recommendation to be fully implemented and/or achieved?
Person(s) responsible
Who will take responsibility for seeing that the actions steps are implemented?
Expected outcome
What is the expected impact/outcome the recommendation will have on the program if it is implemented?
Estimated cost(s)
What is the estimated cost of implementing the recommendation?
Status update
What progress has been made towards achieving the recommendation?
Accreditation Planning and budgeting Maryville University requirements Student Life requirements Program promotion/marketing Recruitment/retention initiatives Publications Conference presentations Student development opportunities Professional development opportunities Grant applications Taken from: Academic Program Assessment: Tools & Techniques for Program Improvement
Academic Program Assessment: Tools & Techniques for Program Improvement.(2013). SUNY. Analyzing Assessment Data.(2013). http://www.sunyorange.edu/assessmentapa/docs/Analy zingandUtilizingAssessmentData.pdf. State University of New York (SUNY) Orange County Community College. Basic Inquiry of Quantitative Research.(n.d.). http://ksumail.kennesaw.edu/~rouyang/EDresearch/details.htm. Kennesaw State University. Center for Evaluation and Research.(2012). http://programeval.ucdavis.edu/documents/Tips_Tools_ 18_2012.pdf. University of California at Davis.
Creswell, J.W.(2009). Qualitative, Quantitative and Mixed Methods Approach, chapter 9, Qualitative Procedures. Diekhoff, G.M.(1996). Basic Statistics for the Social and Behavioral Sciences. Differences Between Quantitative and Qualitative Research. (2013). http://researchguides.ebling.library.wisc.edu/conte nt.php?pid=325126&sid=2940225. University of Wisconsin-Madison-Health Sciences Ebling Library Ely, M., Anzul, M., Friedman, T., Garner, D., Steinmetz, A.M.(1991). Doing qualitative research: Circles within circles. RoutledgeFalmer: New York.
Faculty Center for Teaching and Learning.(2013). Types of qualitative research: Explained within a SOTL framework. http://www.fctl.ucf.edu/researchandscholarship/sot l/creatingsotlprojects/implementingmanaging/quali tativeresearchtypes.php. University of Central Florida. Gathering Data and Assessing Results.(2013). http://nnlm.gov/evaluation/guide/stage 5.pdf. National Network of Libraries of Medicine. The Pell Institute and Pathways to College Network.(2013). Evaluation Toolkit. http://toolkit.pellinstitute.org/evaluationguide/analyze/analyze-quantitative-data/ University of Northern Iowa.(2013). Communication Studies – Research Methods Chapter Four. http://www.uni.edu/commstudies/researchmethods/chapte rfour1.html
View more...
Comments
