to view the PowerPoint presentation

Measurement Problems within Assessment: Can Rasch Analysis help us? Mike Horton Bipin Bhakta Alan Tennant

Mental Arithmetic - Test 1 

  



4+5 = 2x3 = 18 ÷ 2 = Arithmetic is one of the ‘3 R’s’. True or False? = 17 x 13 =

Mental Arithmetic - Test 1 - Answers 

  



4+5 =9 2x3 =6 18 ÷ 2 = 9 Arithmetic is one of the ‘3 R’s’. True or False? = True 17 x 13 = 221

Assumptions underpinning test score addition 

All questions must be mapping onto the same underlying construct • Unidimensionality



All questions must be unbiased between groups • Item Bias  Differential Item Functioning (DIF)



Raw score is a sufficient statistic

Mental Arithmetic - Test 1 – Potential Problems 

  



4+5 =9 2x3 =6 18 ÷ 2 = 9 Arithmetic is one of the ‘3 R’s’. True or False? = True 17 x 13 = 221

Mental Arithmetic - Test 1 – Potential Problems 

  



4+5 =9 2x3 =6 18 ÷ 2 = 9 Arithmetic is one of the ‘3 R’s’. True or False? = True 17 x 13 = 221 Plus: Item Bias -Gender DIF has been shown to be a particular problem in mathematics exams. (e.g. Scheuneman & Grima, 1997., Lane et al. 1996)

Mental Arithmetic - Test 2 

  



17 x 13 = 47 x 64 = 768 ÷ 16 = 532 = 73 =

Mental Arithmetic - Test 2 - Answers 

  



17 x 13 = 221 47 x 64 = 3008 768 ÷ 16 = 48 532 = 2809 73 = 343

Assumptions of Test Equating (Holland & Dorans, 2006) 

Tests measure the same characteristic



Tests measure at the same level of difficulty



Tests measure with the same level of accuracy

Requirements of Test Equating (Dorans & Holland, 2000) 

The tests should measure the same construct



The measures from the tests should have the same reliability



The function used to equate measures from one test to another should be inversely symmetrical



Examinees should be indifferent about which of the equated test forms will be administered



The function for equating tests should be invariant across subpopulations of examinees

Are these elements currently assessed? 



Unidimensionality – is assumed on face validity • Cronbach’s alpha Exam Difficulty Equivalence • Subjective procedures • Classical Test Theory = sample dependent

What is Rasch Analysis?

Mesa Press, Chicago 1980

Rasch Analysis 

The Rasch model is a probabilistic unidimensional model • the easier the question the more likely the correct response • the more able the student, the more likely the question will be passed compared to a less able student.



The model assumes that the probability that a student will correctly answer a question is a logistic function of the difference between the student's ability and the difficulty of the question

Rasch G. Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press, 1980

Assumptions of the Rasch Model 

Stochastic Ordering of Items



Unidimensionality



Local Independence of Items

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Easy Least

Most Able

Hard

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

The Guttman Pattern

1 0 1 1 1 1 1 1

2 0 0 1 1 1 1 1

3 0 0 0 1 1 1 1

4 0 0 0 0 1 1 1

5 0 0 0 0 0 1 1

6 0 0 0 0 0 0 1

Total Score 0 1 2 3 4 5 6

The Rasch Guttman Pattern

1 0 1 1 1 1 1 1

2 0 0 1 1 1 1 1

3 0 0 0 1 0 1 1

4 0 0 1 0 1 1 1

5 0 0 0 0 0 1 1

6 0 0 0 0 0 0 1

Total Score 0 1 3 3 3 5 6

The Probabilistic Rasch Model Probability of a student’s success on an item student

27% 12% 5%

95% 88% 73%

-3

-2

-1

0

1

2

3

Difference (in logits) between the ability of the student and the difficulty of the item

Rasch Analysis 





When data fit the model, generalisability of Item difficulties beyond the specific conditions under which they were observed occurs (specific objectivity) In other words… Item Difficulties are not sample dependent as they are in Classical Test Theory

What Else Does Rasch Offer us? 

When data fit the Rasch Model, the assumptions of summation are met • All questions must be mapping onto the same underlying construct • All questions must be unbiased between groups (DIF) • Raw score is a sufficient statistic



We can then test for other things • Quality of Distractors



It gives us the mathematical basis to compare test scores via equating

Limitations of Rasch Analysis 

The model tests the internal psychometric properties



The model assumes unidimensionality



The model cannot set standards

Summary 

The Rasch model offers a unified framework under which all of the assumptions can be tested together



It gives us a lot of information about individual items which can be utilised to ensure that item and test construction is of a high quality



It provides a rigorous mathematical basis for test equating

References 









Rasch G. Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press, 1980 Dorans NJ & Holland PW. Population invariance and the equatability of tests: Basic theory and the linear case. Journal of Educational Measurement, 2000: 37; 281-306 Holland PW & Dorans NJ. Linking and Equating. In RL Brennan (Ed.), Educational Measurement (4th ed., p187-220). Westport, CT: American Council on Education and Praeger Publishers, 2006. Scheuneman JD & Grima A. Characteristics of quantitative word items associated with differential item functioning for female and black examinees. Applied Measurement in Education, 1997; 299-320. Lane S, Wang N, Magone, M. Gender-Related Differential Item Functioning on a Middle- School Mathematics Performance Assessment. Educational Measurement, 1996: 15(4); 21-27