+1 - ETS

Combining Cognitive and NonCognitive Measures: Expanding the Domain of College Performance and its Prediction Neal Schmitt Michigan State University Presented at College Board, ETS, AERA Conference December 10, 2010

1

Acknowledgements Jessica Fandre  Alyssa Friede  Michael Gillespie  Anna Imus  Brian Kim  Stephanie Merritt  Fred Oswald  Matt Reeder  College Board (Wayne Mattern) 

Tim Pleskac Abigail Quinn Lauren Ramsay Smriti Shivpuri Ruchi Sinha Tae-Young Yoo Mark Zorzie Juliya Golubovich Camara and Krista 2

Outline       

 

History and Background “Job analysis” or conceptualization Instrumentation (description of noncognitive measures used and outcomes examined) Validity data Subgroup Differences and Implications Faking issues Acceptability Research (profiles, fit, goal orientation, dif) Limitations and future research 3

Developing Alternative Measures of Student Potential Motivation for our work  Broaden the scope of student outcomes and capabilities considered in college admissions. 

Reduce adverse impact.



Test the feasibility of developing “noncognitive” measures that are   

valid practical in terms of time and effort required to assess less susceptible to faking 4

Developing Alternative Measures of Student Potential (Oswald et al, 2004, JAP) 



Identify a broader domain of college student performance: 

Review university mission statements and department objectives



Interview with university staff responsible for student life at Michigan State University



Review of the education literature on student outcomes

Our systematic search resulted in 12 dimensions of student performance… 5

Dimensions of College Student Performance intellectual

intrapersonal

1.

Knowledge and mastery of general principles

7.

Social responsibility and citizenship

2.

Continuous learning, intellectual interest and curiosity

8.

Physical and psychological health

3.

Artistic and cultural appreciation

9.

Career orientation

10.

Adaptability and life skills

11.

Perseverance

12.

Ethics and integrity

interpersonal 4.

Appreciation for diversity

5.

Leadership

6.

Interpersonal skills 6

Two “Noncognitive” Measures 

1. Situational judgment inventory  



A situation is presented along with several alternative courses of action. The respondent is asked to indicate what she/he would be most likely and least likely to do.

2. Biodata 

Short, multiple choice reports of past experience/background and/or interests/preferences. 7

Situational Judgment Inventory (SJI) Developed situational judgment items for each of the 12 performance dimensions 

Student generated critical incidents (CIs) for each dimension



Translated CIs to stems for each item



Other students generated solutions to these questions



Researchers edited the options



Re-sorting back into 12 dimensions



3 Answer Keys: (see Motowidlo, Dunnette, & Carter, 1990)   

Expert student (junior and senior students) scoring Resident Advisor scoring African American key

8

Sample SJI Item for Leadership You are assigned to a group to work on a particular project. When you sit down together as a group, no one says anything. a) -1 Look at them until someone eventually says something b) Start the conversation yourself by introducing yourself c) +1 Get to know everyone first and see what they are

thinking about the project to make sure the project’s goals are clear to everyone

d) Try to start working on the project by asking everyone’s

opinion about the nature of the project

e) You would take the leadership role by assigning people to

do things or ask questions to get things rolling

9

Sample SJI Item for Interpersonal Skills You and some other students in your dorm area feel that a small group of students are highly disruptive during times when you would like to study or sleep. What would you do? a) b) c) d)

e) f)

Talk to the resident assistant about it, as that is one of the responsibilities of their job. -1 That’s part of life in the dorms. Let it go. Bring it up at the next floor meeting. +1 Politely talk to the disruptive students and ask them to be more considerate. If the problem persists, talk to the resident assistant. Wear earplugs or headphones when necessary. Develop and implement appropriate rules to address the problem. 10

Sample Item for Knowledge 

  

 



You decided early in the term to do a paper on a topic very interesting to you. However, you have found it difficult to find information on your topic, your job has taken more time than you wanted, and you have had more work in your other courses than you anticipated. Now it seems like you may have to engage in several "all-nighters" to complete your paper on time. What would you do? a. Seek help from other students who may have had a similar experience. b. (-1)Pick a topic that can be completed quicker. An “A” is an “A”. c. (1)Set up a schedule on which you can complete all of the other work you need to do, spend as much time on the paper as possible, and meet with the instructor to discuss what you have so far and get suggestions. d. Do whatever it takes to complete the paper, including “all-nighters”. e. Talk to the instructor about the situation and ask for advice. f. Make the paper a priority, but take into account how much the paper is worth in the class.

11

Biodata Measure Developed biodata items for each dimension   



Reviewed biodata item pools and adapted items related to each major performance dimension. Resorted items into dimensions. Asked a pilot sample to respond to open-ended versions of quantitative response options to determine appropriate scale anchors. We used a rational scoring approach to these items, but also developed empirical keys against 3 criteria 12

Sample Biodata Items for Leadership 1. The number of high school clubs and organized a) b) c) d) e)

activities (such as band, sports, newspapers, etc.) in which I took a leadership role was: 4 or more 3 2 1 I did not take a leadership role

2. How often do you talk your friends into doing

a) b) c) d) e)

what you want to do during the evening? most of the time sometimes (about half the time) occasionally (about as often as others in my group seldom or infrequently never

13

Sample Biodata Items for Multicultural Appreciation 1. How often have you participated in social service or a) b) c) d) e)

charity organizations? Four or more times Three times Two times Once Never

2. If given a choice at a restaurant, would you order any a) b) c) d)

food with which you are unfamiliar? Never, I would always order foods that I know and enjoy Sometimes I might try a new food if someone else ordered it Occasionally I will order something new provided I can also order familiar food at the same time If given a chance, I will always order a new food and try it 14

Sample Biodata Items for Social Responsibility 1.

a) b) c) d) e) 2. a) b) c) d) e)

In the past year how many times have you considered the environment when purchasing a product (for example hairspray, or a car?) Never Once Twice Three or four times Five times or more How often do you work with not for profit groups? Never Not very often Sometimes Often Always

15

Outcomes Examined Self Ratings on behaviorally anchored rating scales built around the 12 dimensions  Self rated class attendance  University archives (grades)  Organizational citizenship behavior  Deviance  Continuation in school and graduation 

16

Interpersonal skills—Self Rated Performance Outcome Definition: Communicating and dealing well with others, whether in informal social situations or more formal school-related situations. Being aware of the social dynamics of a situation and responding appropriately. Fulfills Expectations

Unsatisfactory

1 |

2 |

3 |

4 |

Exceptional

5 |

6 |

7 |

Before you make your rating, please read these two examples: Example 1 Your roommate, usually a tidy person, has recently experienced some personal difficulties. As a result, the roommate has become quite distracted and has left much of the household responsibilities to you. You have talked to the roommate about your concerns, and empathetically requested that the roommate resume his/her share of the responsibilities as soon as possible. A month passes and you are still doing too much of the roommate’s work. What do you expect you would do? Unsatisfactory

Fulfills Expectations

Exceptional

You ask to change rooms.

You do his/her share of the work, and put anything of the roommate’s that affects you in the roommate’s area of the room.

You talk with the roommate again and explain that you are suffering as a result of the roommate’s behavior. You attempt to come up with a mutually acceptable plan of action.

Example 2 You have been standing in line for the restroom for some time after a campus event, and someone cuts into the line ahead of you. What do you expect you would do? Unsatisfactory

Fulfills Expectations

Exceptional

You comment loudly to someone nearby how rude it is that people cut in line.

You tell the person that there is a line.

You calmly and politely inform the person that there is a line and ask that they move to the back.

17

Validity Data: College GPA               

Knowledge Learning Art Appr. Mltcult.Appr. Lead Int. Skl. Soc. Resp. Health Cr. Ornt. Adapt Perser Ethics SJI HSGPA ACT/SAT

2001-02 (N=614)

2003-04 (N=568)

.22 .05 .01 .07 .14 .04 .08 .23 -.02 .21 .15 .14 .16

.22 -.02 -.03 -.04 -.01

.33

.07 .14 -.06 .13 .07 .22 .11 .39 .34

2004-8 (N>1900)

2009-10 (N>550)

.26 .13 .18 .11 .09

.22 .10 .16 .12 .07

.13 .11 -.14 .05 .07 .17 .22 .53 .53

.08 .04 -.11 .01 .10 .13 .09 .29 .44 18

Validity Data: Class Absences 2001-02 (N>630)

     

        

Knowledge Learning Art Appr. Mltcult.Appr. Lead Int. Skl. Soc. Resp. Health Cr. Ornt. Adapt Perser Ethics SJI HSGPA ACT/SAT

-.18 .00 .07 .02 -.03 -.09 -.09 -.23 -.06 -.15 -.20 -.31 -.27 .11

2005-06 (N>900)

2008 (N >556)

2009-10 (N>600)

-.15 -.07 -.03 -.04 -.06

-.14 -.06 -.08 -.10 -.02

-.03 .04 -.10 -.06 -.04

-.08 -.17 -.05 -.10 -.18 -.17 -.16 -.04 .17

-.08 -.12 -.08 -.10 -.16 -.24 -.17 -.01 .14

-.11 -.05 -.05 .07 -.01 -.11 -.14 -.02 .07 19

Validity Data: Self Report-BARS      

        

2001-02 (N=614) Knowledge .33 Learning .27 Art Appr. .51 Mltcult.Appr. .38 Lead .43 Int. Skl. .15 Soc. Resp. .35 Health .28 Cr. Ornt. .22 Adapt .24 Perser .34 Ethics .11 SJI .53 HSGPA ACT/SAT -.01

2003-04 (N=568) .46 .37 .33 .34 .34

2008 (N>547) .21 .24 .20 .29 .29

.32 .16 .36 .26 .46 .39 .54 .08 -.01

.26 .26 .22 .28 .36 .23 .23 .08 .01

2009-10 (N>600) .32 .39 .33 .30 .38

.30 .29 .31 .30 .37 .22 .25 .02 .08

20

Incremental Validity: College GPA Step 1: Act/SAT, HSGPA ∆R2 Step 2: Biodata, SJI ∆R2 Adjusted R N Noncog. Vars. Significant

01-02 .103 .089 .438 610 Know Lrng. Health Adapt

03-04 .179 .070 .499 331

08 .398 .029 .419 1155

09-10 .200 .069 .480 296

Know Lrng. Cr. Ornt. Lrng. Health Know Ethics Cr.Ornt. SJI SJI

21

Incremental Validity: Absenteeism and BARS 05-06 Absenteeism Step 1: Step 2:

Act/SAT, HSGPA ∆R2 Biodata, SJI ∆R2

Adjusted R N Noncog. Vars.

BARS

08 follow-up Absenteeism BARS

.028

.019

.033

.008

.060

.215

.116

.240

.297 800 Health Ethics SJI

.484 801 Health Ethics SJI Adapt Cr.Ornt. Persev.

.386 556 Lead Health Ethics SJT

.497 547 Multic Health Persv Ethics 22

Subgroup Differences in Standardized Units (04sample) SJT Know Cont.Lrn. Artistic Multi.Appr. Leader Respons. Health Car.Ornt. Adapt. Persev. Ethics HSGPA SAT/ACT

Male-Female -.15 .00 .10 -.17 -.16 -.15 -.25 .52 -.17 .00 -.23 -.19 -.04 .43

Cauc-Afr.Am. .13 .36 .05 .31 -.08 .14 .16 .44 -.55 .11 -.19 .22 -.61 -1.46

Cauc-Hisp.Am. .10 .23 .00 .01 -.46 .10 .01 .30 -.08 .16 .00 .29 .42 1.01

23

Percent of subgroups admitted under various strategies Top 15% Top 50% Top 85%

All

Hispanic Cog Cog+ 4.3 6.4 4.0 4.6 4.5 4.7 3.7

Asian African Caucasian Cog Cog+ Cog Cog+ Cog Cog+ 17.8 14.9 .9 4.1 77.0 74.6 10.5 10.1 8.3 10.0 77.1 75.3 7.6 7.7 18.4 18.7 69.5 69.0 9.0 19.4 67.8

Cog=equally weighted composite of HSGPA and SAT/ACT Cog+=equally weighted composite of HSGPA, SAT/ACT, and Non-cognitive measures.

24

Average Cumulative GPA for Subgroup Members who graduated at various levels of selectivity Hispanic Asian African Caucasian N Cog N Cog+ N Cog N Cog+ N Cog N Cog+ N Cog N Cog+ Top15 4 3.67 6 3.57 23 3.88 18 3.84 0 1 3.66 208 3.88 197 3.87 Top50 17 3.51 17 3 .42

55 3.65 54

3.63 15 3.46 15

3.38 651 3.66 638 3.66

Top85 23 3.44 22

64 3.62 64

3.62 49 3.05 47

3.08 929 3.52

3.47

905 3.54

25

Proportion of Subgroups Graduating in Four Years under Different Levels of Selectivity Hispanic Cog Top15 93 Top50 79 Top85 77

Cog+ 90 78 76

Asian Cog Cog+ 89 91 72 75 68 68

African Cog Cog+ 100 92 87 87 79 78

Caucasian Cog 72 59 51

Cog+ 70 60 52

26

Admits versus Applicants: Standardized mean differences (d)            

Knowledge Continuous Learning Artistic Appreciation Multicultural Appreciation Leadership Social Responsibility Health Career Orientation Adaptability Perseverance Ethics Situational Judgment

2004 .55 .43 .35 .48 .43 .46 .26 .25 .22 .39 .62 .50

2006 .54 .49 .30 .40 .35 .46 .30 .21 .25 .32 .53 .52 27

Conclusions on Faking Research (represents summary of a series of studies   

 



Applicants score higher than current students (d=.2 to .6) Coaching has a significant impact on the degree of score inflation Elaboration can minimize score inflation, but…feasibility is an issue and its effects do not appear to generalize to nonelaborated items Elaboration has no impact on validity Warnings do not appear to have much effect, but…generalizability to an applicant situation has not been evaluated. Are they any less fakable than essays or less inflated than letters of recommendation??? 28

Sample Bogus Items: Carelessness 

How often in the past year have you programmed in AJMR?



never once twice three or four times five times or more

    

How often, in the past three years, have you operated a rhetaguard?



never once twice three or four times five times or more



  

29

Profiling Subgroups of Students based on HSGPA, SAT/ACT and Noncognitive Variables

30

31

32

33

34

35

36

Outcomes and Implications  







There were predicted outcome differences (GPA, absenteeism, satisfaction) across profile groups Highly motivated, career-oriented group is most likely to respond to remedial efforts particularly if they relate to their career objectives Marginal group without career objectives is a high risk group. Career counseling and intensive remediation may be necessary Efforts to broaden the scope of interests of the high ability, culturally limited may be desirable in some universities Students in the high ability well rounded group might be identified as potential student leaders and peer mentors/tutors. 37

Overall Conclusions 

  



We can develop valid noncognitive measures that relate to GPA and other important student outcomes Faking of the biodata and SJI remains a problem. Reactions to their use are not significantly different than reactions to the ACT/SAT Subgroup differences are minimal and certainly much less than those we find for cognitive ability measures There may be useful other ways to employ these instruments; that is, to identify subgroups for whom interventions designed to retain them will be needed. 38

Thank you for your attention!

39