We are a sharing community. So please help us by uploading **1** new document or like us to download:

OR LIKE TO DOWNLOAD IMMEDIATELY

Measurement Problems within Assessment: Can Rasch Analysis help us? Mike Horton Bipin Bhakta Alan Tennant

Mental Arithmetic - Test 1

4+5 = 2x3 = 18 ÷ 2 = Arithmetic is one of the ‘3 R’s’. True or False? = 17 x 13 =

Mental Arithmetic - Test 1 - Answers

4+5 =9 2x3 =6 18 ÷ 2 = 9 Arithmetic is one of the ‘3 R’s’. True or False? = True 17 x 13 = 221

Assumptions underpinning test score addition

All questions must be mapping onto the same underlying construct • Unidimensionality

All questions must be unbiased between groups • Item Bias Differential Item Functioning (DIF)

Raw score is a sufficient statistic

Mental Arithmetic - Test 1 – Potential Problems

4+5 =9 2x3 =6 18 ÷ 2 = 9 Arithmetic is one of the ‘3 R’s’. True or False? = True 17 x 13 = 221

Mental Arithmetic - Test 1 – Potential Problems

4+5 =9 2x3 =6 18 ÷ 2 = 9 Arithmetic is one of the ‘3 R’s’. True or False? = True 17 x 13 = 221 Plus: Item Bias -Gender DIF has been shown to be a particular problem in mathematics exams. (e.g. Scheuneman & Grima, 1997., Lane et al. 1996)

Mental Arithmetic - Test 2

17 x 13 = 47 x 64 = 768 ÷ 16 = 532 = 73 =

Mental Arithmetic - Test 2 - Answers

17 x 13 = 221 47 x 64 = 3008 768 ÷ 16 = 48 532 = 2809 73 = 343

Assumptions of Test Equating (Holland & Dorans, 2006)

Tests measure the same characteristic

Tests measure at the same level of difficulty

Tests measure with the same level of accuracy

Requirements of Test Equating (Dorans & Holland, 2000)

The tests should measure the same construct

The measures from the tests should have the same reliability

The function used to equate measures from one test to another should be inversely symmetrical

Examinees should be indifferent about which of the equated test forms will be administered

The function for equating tests should be invariant across subpopulations of examinees

Are these elements currently assessed?

Unidimensionality – is assumed on face validity • Cronbach’s alpha Exam Difficulty Equivalence • Subjective procedures • Classical Test Theory = sample dependent

What is Rasch Analysis?

Mesa Press, Chicago 1980

Rasch Analysis

The Rasch model is a probabilistic unidimensional model • the easier the question the more likely the correct response • the more able the student, the more likely the question will be passed compared to a less able student.

The model assumes that the probability that a student will correctly answer a question is a logistic function of the difference between the student's ability and the difficulty of the question

Rasch G. Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press, 1980

Assumptions of the Rasch Model

Stochastic Ordering of Items

Unidimensionality

Local Independence of Items

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Easy Least

Most Able

Hard

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

The Guttman Pattern

1 0 1 1 1 1 1 1

2 0 0 1 1 1 1 1

3 0 0 0 1 1 1 1

4 0 0 0 0 1 1 1

5 0 0 0 0 0 1 1

6 0 0 0 0 0 0 1

Total Score 0 1 2 3 4 5 6

The Rasch Guttman Pattern

1 0 1 1 1 1 1 1

2 0 0 1 1 1 1 1

3 0 0 0 1 0 1 1

4 0 0 1 0 1 1 1

5 0 0 0 0 0 1 1

6 0 0 0 0 0 0 1

Total Score 0 1 3 3 3 5 6

The Probabilistic Rasch Model Probability of a student’s success on an item student

27% 12% 5%

95% 88% 73%

-3

-2

-1

0

1

2

3

Difference (in logits) between the ability of the student and the difficulty of the item

Rasch Analysis

When data fit the model, generalisability of Item difficulties beyond the specific conditions under which they were observed occurs (specific objectivity) In other words… Item Difficulties are not sample dependent as they are in Classical Test Theory

What Else Does Rasch Offer us?

When data fit the Rasch Model, the assumptions of summation are met • All questions must be mapping onto the same underlying construct • All questions must be unbiased between groups (DIF) • Raw score is a sufficient statistic

We can then test for other things • Quality of Distractors

It gives us the mathematical basis to compare test scores via equating

Limitations of Rasch Analysis

The model tests the internal psychometric properties

The model assumes unidimensionality

The model cannot set standards

Summary

The Rasch model offers a unified framework under which all of the assumptions can be tested together

It gives us a lot of information about individual items which can be utilised to ensure that item and test construction is of a high quality

It provides a rigorous mathematical basis for test equating

References

Rasch G. Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press, 1980 Dorans NJ & Holland PW. Population invariance and the equatability of tests: Basic theory and the linear case. Journal of Educational Measurement, 2000: 37; 281-306 Holland PW & Dorans NJ. Linking and Equating. In RL Brennan (Ed.), Educational Measurement (4th ed., p187-220). Westport, CT: American Council on Education and Praeger Publishers, 2006. Scheuneman JD & Grima A. Characteristics of quantitative word items associated with differential item functioning for female and black examinees. Applied Measurement in Education, 1997; 299-320. Lane S, Wang N, Magone, M. Gender-Related Differential Item Functioning on a Middle- School Mathematics Performance Assessment. Educational Measurement, 1996: 15(4); 21-27

View more...
Mental Arithmetic - Test 1

4+5 = 2x3 = 18 ÷ 2 = Arithmetic is one of the ‘3 R’s’. True or False? = 17 x 13 =

Mental Arithmetic - Test 1 - Answers

4+5 =9 2x3 =6 18 ÷ 2 = 9 Arithmetic is one of the ‘3 R’s’. True or False? = True 17 x 13 = 221

Assumptions underpinning test score addition

All questions must be mapping onto the same underlying construct • Unidimensionality

All questions must be unbiased between groups • Item Bias Differential Item Functioning (DIF)

Raw score is a sufficient statistic

Mental Arithmetic - Test 1 – Potential Problems

4+5 =9 2x3 =6 18 ÷ 2 = 9 Arithmetic is one of the ‘3 R’s’. True or False? = True 17 x 13 = 221

Mental Arithmetic - Test 1 – Potential Problems

4+5 =9 2x3 =6 18 ÷ 2 = 9 Arithmetic is one of the ‘3 R’s’. True or False? = True 17 x 13 = 221 Plus: Item Bias -Gender DIF has been shown to be a particular problem in mathematics exams. (e.g. Scheuneman & Grima, 1997., Lane et al. 1996)

Mental Arithmetic - Test 2

17 x 13 = 47 x 64 = 768 ÷ 16 = 532 = 73 =

Mental Arithmetic - Test 2 - Answers

17 x 13 = 221 47 x 64 = 3008 768 ÷ 16 = 48 532 = 2809 73 = 343

Assumptions of Test Equating (Holland & Dorans, 2006)

Tests measure the same characteristic

Tests measure at the same level of difficulty

Tests measure with the same level of accuracy

Requirements of Test Equating (Dorans & Holland, 2000)

The tests should measure the same construct

The measures from the tests should have the same reliability

The function used to equate measures from one test to another should be inversely symmetrical

Examinees should be indifferent about which of the equated test forms will be administered

The function for equating tests should be invariant across subpopulations of examinees

Are these elements currently assessed?

Unidimensionality – is assumed on face validity • Cronbach’s alpha Exam Difficulty Equivalence • Subjective procedures • Classical Test Theory = sample dependent

What is Rasch Analysis?

Mesa Press, Chicago 1980

Rasch Analysis

The Rasch model is a probabilistic unidimensional model • the easier the question the more likely the correct response • the more able the student, the more likely the question will be passed compared to a less able student.

The model assumes that the probability that a student will correctly answer a question is a logistic function of the difference between the student's ability and the difficulty of the question

Rasch G. Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press, 1980

Assumptions of the Rasch Model

Stochastic Ordering of Items

Unidimensionality

Local Independence of Items

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Easy Least

Most Able

Hard

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

What Would We Expect When These People Meet These Items? Hard

Easy Least

Most Able

Item 1

Item 2

Item 3

Person 1

Correct

Incorrect

Incorrect

Person 2

Correct

Correct

Incorrect

Person 3

Correct

Incorrect

Correct

Person 4

Incorrect

Correct

Correct

Person 5

Correct

Correct

Correct

The Guttman Pattern

1 0 1 1 1 1 1 1

2 0 0 1 1 1 1 1

3 0 0 0 1 1 1 1

4 0 0 0 0 1 1 1

5 0 0 0 0 0 1 1

6 0 0 0 0 0 0 1

Total Score 0 1 2 3 4 5 6

The Rasch Guttman Pattern

1 0 1 1 1 1 1 1

2 0 0 1 1 1 1 1

3 0 0 0 1 0 1 1

4 0 0 1 0 1 1 1

5 0 0 0 0 0 1 1

6 0 0 0 0 0 0 1

Total Score 0 1 3 3 3 5 6

The Probabilistic Rasch Model Probability of a student’s success on an item student

27% 12% 5%

95% 88% 73%

-3

-2

-1

0

1

2

3

Difference (in logits) between the ability of the student and the difficulty of the item

Rasch Analysis

When data fit the model, generalisability of Item difficulties beyond the specific conditions under which they were observed occurs (specific objectivity) In other words… Item Difficulties are not sample dependent as they are in Classical Test Theory

What Else Does Rasch Offer us?

When data fit the Rasch Model, the assumptions of summation are met • All questions must be mapping onto the same underlying construct • All questions must be unbiased between groups (DIF) • Raw score is a sufficient statistic

We can then test for other things • Quality of Distractors

It gives us the mathematical basis to compare test scores via equating

Limitations of Rasch Analysis

The model tests the internal psychometric properties

The model assumes unidimensionality

The model cannot set standards

Summary

The Rasch model offers a unified framework under which all of the assumptions can be tested together

It gives us a lot of information about individual items which can be utilised to ensure that item and test construction is of a high quality

It provides a rigorous mathematical basis for test equating

References

Rasch G. Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press, 1980 Dorans NJ & Holland PW. Population invariance and the equatability of tests: Basic theory and the linear case. Journal of Educational Measurement, 2000: 37; 281-306 Holland PW & Dorans NJ. Linking and Equating. In RL Brennan (Ed.), Educational Measurement (4th ed., p187-220). Westport, CT: American Council on Education and Praeger Publishers, 2006. Scheuneman JD & Grima A. Characteristics of quantitative word items associated with differential item functioning for female and black examinees. Applied Measurement in Education, 1997; 299-320. Lane S, Wang N, Magone, M. Gender-Related Differential Item Functioning on a Middle- School Mathematics Performance Assessment. Educational Measurement, 1996: 15(4); 21-27

We are a sharing community. So please help us by uploading **1** new document or like us to download:

OR LIKE TO DOWNLOAD IMMEDIATELY

Thank you for interesting in our services. We are a non-profit group that run this website to share documents. We need your help to maintenance this website.

To keep our site running, we need your help to cover our server cost (about $400/m), a small donation will help us a lot.