File

Test Construction and Reliability WISC-IV By Jill Hutzel, A.M, K.W & L.K

What Does This Test Measure (3.2)? • The Wechsler Intelligence Scale for Children-Fourth Edition (WISC-IV) was designed to measure the intellectual functioning in specific cognitive areas such as: • Verbal Comprehension • Perceptual Reasoning • Working Memory • Processing Speed

This test also provides a composite score (ex. Full Scale IQ) that represents a child’s general intellectual ability.

These Four Index Scores measures a child’s overall: Crystallized Ability (Gc)- acquired skills and knowledge that are developmentally dependent on exposure to the culture

Visual Processing (Gv)- a facility for visualizing and manipulating figures and responding appropriately to spatial forms

Fluid Reasoning (Gf)- a broad pattern of reasoning, seriation, sorting, and classifying

Processing Speed (Gs)- an ability to scan and react to simple tasks rapidly

(Sattler, 2008)

What Are the Test Specifics (3.3)? Age of Examinees- 6:0- 16:11

Number of Subtests- 10 subtests and 4 indexes including: (VCI) Similarities, Vocabulary, Comprehension (PRI) Block Design, Picture Concepts, Matrix Reasoning (WMI) Digit Span, Letter-Number Sequencing (PSI) Coding & Symbol Search Number of Supplemental Subtests- 5 including: (VCI) Information & Word Reasoning (PRI) Picture Completion (WMI) Arithmetic (PSI) Cancellation Administration time is approximately 65 to 80 minutes Qualification of Examiners- Graduate or Professional level of training in psychological assessment

Procedures to Norm and Standardize Test Scores (3.4) • • •

•

• • • • •

This test developed in 5 general stages: Conceptual Development, Pilot, National Tryout, Standardization, Final Assembly and Evaluation Sample size 2,200 children ages 6:0 to 16:11 (the arithmetic subtest was normed a subsample of 1,100 children-100 children per age group) In order to prove evidence of the scales validity, additional children were administered the WISC-IV and other cognitive measures including the WISC III, WAIS III, WPPSI-III, WASI, WIAT-II,CMS,GRS, BarOn EQ and ABAS-II Description of the Sample- To ensure the standardization samples included representative proportions of children according to selected demographic variables including sex, age, race/ethnicity, parent education level, geographic region. Researchers used the March 2000 Census from the U.S. Bureau AGE- 2,200 Children divided into 11 age groups- 200 participants in each age group (ex. 6:0-6:11, 7:0-7:11… 16:11) SEX- Equal number of males and females in each age group (100 in each group) RACE- The proportions of racial groups were based on the racial proportions of children within that age group of the US population according to the Census Parent Education Level- The sample was divided according to 5 parent education levels based on years of education completed Geographic Region-Divided into the 4 major geographic regions specified by the census reports

Procedures Used to Develop Test Items 3.5 (Conduct and document review by relevant, independent experts, including review process and experts’ qualifications, relevant experiences, and demographics)

• Specific procedures were utilized in the WISC IV research program to optimize the quality of obtained data and to assist in the formulation of final scoring criteria. • One of the first steps was to recruit examiners with extensive experience testing children and adolescents. Potential examiners completed a questionnaire by supplying information about their educational and professional experience, administration experience with various intellectual measures, certification, and licensing status. The majority was certified or licensed professional working in private or public facilities.

• Potential standardization examiners were provided training material, which consisted of training video, a summary of common administration and scoring errors, and a two part training quiz. The content of the training quiz included questions on administration and scoring rules as well as a task that required the examiner to identify administration and scoring errors in a fictitious test protocol.

3.5 continued… • Selected examiners scored at least 90% correction on both parts of the training quiz. Any errors or omissions on the training quiz were reviewed with the examiner. As an oversight measure, examiners were required to submit a review case prior to testing additional children. Every attempt was made to discuss administration and scoring errors on the review case with the examiner within 48 hours of its submission. Subsequent cases were reviewed within 72 hours of receipt if possible, and any errors resulting in loss or inaccuracy of data were discussed with the examiner. A periodic newsletter was sent to all examiners, alerting them to potentially problematic areas. • All scorers had a minimum of a bachelor’s degree and attended a 5-day training program led by members of the research team. Scorers were required to score at least 90% correct on a quiz that required them to identify scoring errors in a fictitious protocol. Each protocol collected during the national tryout and standardization stages of development was rescored and entered into a database by two qualified scorers working independently. Any discrepancies between the two scorers were resolved daily by a third scorer (resolver).The resolvers were chosen based on their demonstration of exceptional scoring accuracy and previous scoring experience.

3.6 (Empirical analyses and/or expert judgment as to the appropriateness of test items, content, and response formats for different groups of test takers)

• To ensure the validity of the WISC IV, 16 special group studies were conducted during the WISC IV Standardization. The results from the special group studies support for the validity and clinical utility of the WISC IV. The majority of results are consistent with expectations based on previous research and theoretical foundations of the scales development. It is expected that future investigations utilizing the WISC IV in different clinical settings and populations will provide additional evidence of the scales utility for clinical diagnosis and intervention purposes.

3.7 (Procedures used to develop, review, and tryout items from item pool)

• Early in the development process, 45 assessments professional from eight major cities met in a focus group with members of a marketing research firm to refine revision goals and assist in the formulation of the scales working blueprint. • Also, a telephone survey (N=308) was conducted with users of the WISC-III as well as professionals in child and adolescent assessment. The research team, advisory panel that was composed of national recognized experts in school psychology and clinical neuropsychology, and clinical measurement consultants from the Harcourt assessment reviewed the feedback from the focus groups and telephone surveys. Based on findings, the working blueprint was established and the first research version of the scale was developed for the use in the initial pilot study.

3.7 continued…

• The primary goal of the pilot stage was to produce a version of the scale for use in the subsequent national tryout stage. A number of research questions were addressed through a series of five pilot studies (N= 255, 151, 110, 389, and 197) and three mini pilot studies (N=31,16, and 34). Each of these studies utilized a research version of the scale that included various groupings of subtests retained from the WISC III and new, experimental subtests that were being considered for inclusion at the national tryout stage. • The primary research questions at this stage of development focused on such issues as content and reliance of items, adequacy of subtests floors and ceilings, clarity of instructions to the examiner and child, identification of response processes, administration procedures, scoring criteria, item bias and other relevant psychometric properties.

3.8 (selection procedures and demographics of item tryout and/or standardization sample) •

•

•

The national tryout stage utilized a version of the scale with 16 subtests; Data were obtained from a stratified sample of 1,270children, who reflected key demographic variables in the national population. An analysis gathered by the U.S Bureau of the Census (1998) provided basis for stratification along the following variables: age, sex, race, parent education level, and geographic region. Using this larger, more representative sample of children, research questions from the pilot phased were reexamined, and additional issues were addressed. Refinements to the item order were made based on more precise estimates of their relative difficulty level, and exploratory and confirmatory factor analyses were conducted to determine the underlying factor structure of the scale. In addition, data were collected at this stage from a number of special groups (children identified as intellectually gifted, children with intellectual disability or learning disorders, and children with ADHD) to provide additional evidence regarding the adequacy of the subtest floors and ceilings, as well as clinical utility of the scale. An oversample of252 African American children and 186 Hispanic children was collected to allow for a statistical examination of item bias using IRT methods of analysis.

3.8 Continued… • After reviewing the accumulated evidence from the pilot and national tryout studies a standardization edition of the WISC IV was created.

• The standardization sample consisted of 2,200children who were divided into 11 age groups where each age group consisted of200 participants. The U.S. Bureau of the Census collected an analysis of data in March 2000 along the variables of: age, sex, race/ethnicity, parent education level, and geographic region. For each age group, the proportions of Whites, African Americans, Hispanics, Asians, and other racial groups were based in the racial proportions of children within the corresponding age group of the U.S> population according to March 2000 census data. The sample was stratified according to five-parent education levels based on the number of years of school completed. If the child resided with only one parent or guardian, the educational level of that parent of guardian was assigned. If the child resided with two parents, a parent and a guardian, or two guardians, the average of both individuals’ educational levels was used, with partial levels rounded up to the next highest level.

Evidence for Internal Consistency 2.7 According to the Technical Manual…

• The evidence for internal consistency was obtained using the normative sample and the split half method. “The split-half method is done by sorting the items on a test into two parallel subtests of equal size. Then you compute a composite score for each subtest and correlate the two composite scores. By doing so, you have created two parallel tests from the items within one test. It is possible to use these subtest scores to compute and estimate of total test reliability” (Furr and Bacharach, 2008). • As stated by the WISC IV technical manual, the split half method was used on all subtests excluding Coding, Symbol Search and Cancellation due to these being Processing Speed subtests. Therefore, test-retest stability coefficients were used as the reliability estimates for these particular subtests.

2.7 continued… • The reliability coefficients for the WISC IV composite scales range from .88 (Processing Speed) to .97 (Full Scale). These coefficients are generally higher than those of the individual subtests that comprise these composite scales. The average reliability coefficient for the Processing Speed composite scale is slightly lower (.88) because it is based on the test-retest reliabilities which tend to be lower then the split half reliabilities. The reliability coefficients for the WISC IV composite scales are identical to or slightly better than corresponding scales in the WISC III. • The evidence of Internal Consistency Reliability was obtained from the split half method from a group of children ranging from the ages of 6 to 16. The overall averages for these special clinical groups are as follows: Verbal Comprehension (VCI) .94, Perceptual Reasoning (PRI) .92, Working Memory (WMI) .92, Processing Speed (PSI) .88, and Full Scale (FSIQ) .97.

Test-Retest Approaches (Are alternate-form or testretest approaches used, and if so, what were the results? Were separately timed administrations used to investigate a practice effect, and if so, what were the results? Additional information includes procedures used to estimate this type of reliability)

2.9

• Test-Retest Approaches Yes. A test-retest approach was used. According to Wechsler (2004), the sample consisted of: 243 children 18 to 27 participants in each of the 11 age groups Each participant was given two separate WISCIV administrations: Ranging from 13 to 63 days between test and retest (Mean interval of 32 days)

2.9 Continued…

• The sample consisted of: •52.3% Female vs. 47.7% Male •74.1% White •7.8% African American •11.1% Hispanic •7.0% Other • The Parent Education Level: •0 – 8 Years (Y): 4.9% •9 – 11 Y: 9.1% •12 Y: 25.9% •13 – 15 Y: 36.2& •> 16 Y: 23.9%

2.9 Continued… • Used Pearson’s product-moment correlation to estimate TEST-RETEST RELIABILITY for 5 different age groups (Wechsler, 2004) Age groups: 6-7, 8-9, 10-11, 12-13, 14-16 r = (SP)/(SqRt (SSxSSy)) Table 4.4 in the WISC-IV Integrated Technical and Interpretive Manual displays: • Mean subtest scaled scores and composite scores with SD • Standard Differences (effect sizes) between the first and second testing's • Correlation coefficients corrected for the variability of the standardization sample Continuation and Chart Follows

2.9 Continued…

• (Williams, Weiss, & Rolfhus, 2003) • Used Fisher’s Z Transformation to calculate TESTRETEST COEFFICIENTS for the Overall Sample (Wechsler, 2004) • Standard Difference calculated using: (The mean score difference between the first and second testing session) divided by (the pooled standard deviation) Effect Size – A measure intended to provide a measurement of the absolute magnitude of a treatment effect, independent of the size of the sample(s) being used (Gravetter & Wallnau, 2009) Cohen’s d = mean difference/standard deviation Comprehension had the smallest effect size (.08), Picture Completion had the largest (.60), FSIQ had an effect size of (.46)

2.9 Continued… • RESULTS: The WISC-IV scores have adequate stability across time for all five age groups (Wechsler, 2004) • Corrected Stability Coefficient •Excellent (.92) -Vocabulary •Good (.80) -Block Design -Similarities -Digit Span -Coding -Letter-Number Sequencing -Matrix Reasoning -Comprehension -Symbol Search -Picture Completion -Information -Word Reasoning •Adequate (.70) -Other subtests

Composite Scores have better stability than individual subtest scores Good (.80) or better

2.9 Continued… • Retest score means for the subtests of the WISC-IV are higher than the scores from the first testing session, possibly due to practice effects due to a short time interval between test and retest– Practice Effects •Re-test gains were smaller for the VCI and WMI subtests compared to the PRI and PSI subtests •Score Differences between test-retest primarily due to practice effects: VCI +2.1, PRI +5.2, WMI +2.6, PSI +7.1 , FSIQ +5.6

– Stability of the WISC-IV in a Sample of Elementary and Middle School Children Ryan, Glass, and Bartels (2010) investigated test-retest stability of the WISC-IV in 43 elementary/middle school students in a rural location, tested on two separate occasions, roughly 11 months apart Believed that the stability found from the WISC-IV standardization sample does not generalize to clinically realistic test-rest intervals and does not generalize to other populations

2.9 Continued… •

•

Participants – 76 students from a small private school in a Midwestern community – 43 were rested 25 female 18 male Stability Coefficients ranged from: .26 (Picture Concepts) .84 (Vocabulary) .88 (FSIQ)

Table follows-

2.9 Continued…

• Results

2.9 Continued… • Stability Coefficients from the standardization sample were slightly larger than from the sample in this study (Ryan et al., 2010) -FSIQ .91 in the Standardization Sample Vs. .88 in the abovementioned sample -Similar to the standardization sample, stability coefficients for the composite scores were slightly more stable than individual subtest scores, with the FSIQ being the most stable • Ryan et al. (2010) believe that the test-retest interval of 11 months, compared to the 32 day test-retest interval, accounted for an overall smaller stability coefficient and an overall smaller practice effect • This study supports Wechsler’s statistical evidence that (Ryan et al., 2010): -The FSIQ is the most stable score provided by the WISC-IV over time -During long test-retest intervals, only the FSIQ has sufficient stability for interpretation -Individual subtest scores should NOT be used for any diagnostic and/or decision-making purposes

Evidence Provided for Both Interrater Consistency & Consistency Over Repeated Measurements (2.10) According to the WISC IV Technical Manual…

•

The test-retest sample for the WISC-IV was composed of 243 children. There were 18-27 participants from each of the 11 age groups.

•

The WISC-IV was given one time to all of the children. The test was then administered a second time anywhere from 13-63 days later. The mean interval was 32 days. There was %52.3 females and %47.7 males in the sample.

•

“The test-retest reliability was calculated for five age groups (6:0-7:11, 8:0-9:11, 10:0-11:11, 12:0-13:11, and 14:0-16:11) using Pearson’s product-moment correlation.” The coefficients of the test-retest for the general sample were calculated using Fisher’s z transformation.

•

“The standard difference was calculated using the mean score difference between two testing divided by the pooled standard deviation.”

•

The mean scores for the retest for all of the seven scaled process scores are higher than that from the first testing, in which the effect sizes ranged from .14 to .41. “In general, test-retest gains are less pronounced for the process scores in the Working Memory domain than for the process scores in the Perceptual and Processing Speed domains” (pg. 136 of technical manual)

2.10 Continued… • In a Study done by Ryan, Glass and Bartels they had 76 students in a Midwestern community take the WISC-IV, 43 of the students agreed to take a second WISC-IV examination, those 43 students were the participants of the investigation. • According to Ryan, Glass and Bartels (2010), in all of the dependent samples, except for one, the t-tests failed to discover significant differences in scores from the first time the WISC-IV was administered to the second time. • “Stability coefficients in the present sample were consistently smaller than those reported in the WISC-IV Technical and Interpretive Manual (Wechsler, 2003b) for children 8 to 9 years of age.” • This study did have some limitations though, the study was done in a rural community that is composed of mainly white students attending a private school and it is not a good representation of an ethnically diverse population (Ryan, Glass, Bartel, 2010).

Reference •

Furr, R. M. & Bacharach, V. R. (2008). Psychometrics: An Introduction. Thousand Oaks, CA: Sage Publications.ISBN: 978-1-412-927604

•

Gravetter, F. & Wallnau, L. (2009). Statistics for the Behavioral Sciences-Eighth Edition. Wadsworth, CA: Cengage Learning.

•

Ryan, J., Glass, L., Bartels, J. (2010). Stability of the WISC-IV in a Sample of Elementary and Middle School Children. Applied Neuropsychology, 17: 68-72.

•

Sattler, J.M. (2008a). Assessment of children: Cognitive foundations (5th ed.). San Diego: Author

•

Wechsler, D. (2004). WISC-IV Technical and Interpretive Manual. San Antonio, TX: Psychological Corporation.

•

Williams, P., Weiss, L., Rolfhus, E. (2003). WISC-IV Technical Report # 1 Psychometric Properties. WISC-IV Technical Manual # 1. San Antonio, TX: Psychological Corporation.

•

Williams, P., Weiss, L., Rolfhus, E. (2003). WISC-IV Technical Report # 2 Psychometric Properties. WISC-IV Technical Manual # 2. San Antonio, TX: Psychological Corporation.