Essex County College Center for Academic Foundations

A Mixed-Method Evaluation of a Series of Strategies Designed to Improve Student Success in Mathematics and English Developmental Education Courses at Essex County College

Dr. Leigh Bello de Castro Essex County College Dr. Charyl Yarbrough The Heldrich Center for Workforce Development at Rutgers

The Center for Academic Foundations ECC Incoming Freshman • Accuplacer and Companion Tests • Results • 85% - Math 086 • 10% - Math 092 • 5% - Math 100 or higher

ECC Center for Academic Foundations Objectives • Established in 2007 • Target Group: students scoring between 20 -29 Accuplacer • 25% pass rate in Math 086 (Target Group) • Develop Learning Communities for students in need of Math and English developmental/transitional courses. • Provide an array of support services. • • • •

Counseling Early Intervention Career Counseling Intensive Tutoring

• Implement new teaching strategies into the curriculum.

ECC Developmental Sequence Mth 100

Mth 092

AFM 083

Mth 086

Eng 101

Eng 092

AFE 083

*** AFM and AFE Divisional Midterms and Finals

Eng 085

ECC Center for Academic Foundations Staff • • • • • • •

Full time Coordinator Full time Teacher Advisor Math Full time Teacher Advisor English Adjunct Math and English Faculty Supplemental Instruction Leaders Resource Specialists Math and English Faculty Advisors

ECC Center for Academic Foundations •

Implement new teaching strategies into the curriculum. • • • • •

•

Block Scheduling • • •

•

Customized Book ALEKS My Skills Lab Early Intervention Supplemental Instructional Leaders Math, English and Computer Course (CSS 101) Tutoring with SI Leader after class 2 class periods of lecture, 1 class period in Lab with ALEKS.

Stand Alone Math Sections

ESSEX COUNTY COLLEGE INSTITUTIONAL RESEARCH BASIC SKILL TRACKING - CAF and Regular Spring 2009, Spring 2010 and Fall 2010 Cohorts Criteria Cohort

Original Cohort

Return Following Sem.

Success* Rate in

Percent W

Return Following Sem.

Success* Rate in

Size/ N Who

took MTH 092

MTH 092

MTH 092

took ENG 096

ENG 096

Percent W ENG 096

Passed Course FALL 2010

N original/ passed

%

N

%

N

original/ passed

%

N

%

%

original/ passed

AFM 083

609/ 335

270

44.3%/ 80.6%

104

38.5%

25.9%

MTH 086

2202/ 1146

896

40.6%/ 78.2%

417

46.5%

19.0%

AFE 083

444/ 278

233

52.5%/ 83.8%

156

67.0%

11.6%

ENG 085

957/ 708

578

60.4%/ 81.6%

395

68.3%

11.1%

SPRING 2010 AFM 083

470/ 220

99

21.1%/ 45.0%

12

12.1%

41.4%

MTH 086

1528/ 670

348

22.8%/ 51.9%

120

34.5%

24.1%

AFE 083

305/ 159

73

23.9%/ 45.9%

34

46.6%

23.3%

ENG 085

426/ 237

92

25.6%/ 38.8%

54

58.7%

9.8%

SPRING 2009 AFM 083

302/ 91

42

13.9%/ 42.6%

9

21.4%

19.0%

MTH 086

1400/ 577

278

19.9%/ 48.1%

104

37.4%

15.5%

AFE 083

183/ 112

42

23%/ 37.5%

19

45.2%

23.8%

ENG 085

321/ 200

73

22.7%/ 36.5%

45

61.6%

11.0%

*Success rate = A,B+,B,C+,C/All Grades The numbers in RED refer to the number or percent of students who successfully completed their first remedial course and would be allowed to take the next level course. For Example: For Fall 2010 AFM 083 - originally there were 609 students registered for this course. Of these, 335 passed the course and would qualify to take The next level mathematics course. In fact, 270, or 44.3 % of the original cohort took MTH 096. This increases to 80.6% of those qualified to take MTH 092. Of the 270 who took MTH 092, 104 or 38.5% successfully completed MTH 092 and 25.9% withdrew.

Center for Academic Foundations Math 092 • Students were not succeeding in next course • Spring 2011 • PBI Funds for CAF 09 level • • • • •

Block scheduling Intrusive Support Resource Specialist Mandatory Tutoring Supplemental Instructional Leaders

• MTH 092 and ENG 096 • Curriculum • Books • Instructional Software

• Implemented the original CAF model • Additional Staffing • Evaluation

Purpose of Evaluation • Describe ECC’s PBI Formula Grant Activities • Generate actionable information on the implementation of the PBI Formula Grant initiatives • Assess the overall effectiveness of ECC’s PBI Formula Grant initiatives in achieving positive outcomes

Key Research Questions 1. Do the ECC project services have the intended positive effects on student outcomes (e.g., higher percentage of students passing mathematics and English courses)?

2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? 3. In what ways can the services be modified to maximize their effectiveness?

Evaluation Design • Consisted of formative and summative components

• Utilized qualitative and quantitative methodologies • Evaluators from multiple academic disciplines participated in the evaluation. • Submit interim reports • Focused on understanding the culture and politics of the school as part of the evaluation process • Considered the views of all stakeholders

Qualitative Data Collection (Spring 2011) Method

Time Frame

Number of Participants/Sessions

Course Observations

2-3 times per course per semester

sixteen observations

Instructor Interviews

Pre-Post Semester

Five interviews

Student Focus Groups

Post Semester

Four groups consisting of 5-8 students

Self-Efficacy and Subject Matter Confidence Surveys

Pre-Post Semester

N = 42-67

Competing Priorities and Barriers Scales

Pre Semester

Quantitative Data Collection Administrative Data • The Heldrich Center collects class level data from the program administrator.

Program Description Profile of Students • 121 students in five developmental math sections • 109 students in five developmental English sections • Most students were comfortable with use of computers at baseline • 67% of students worked 20 hours per week or less • Moderate levels of confidence at the beginning of the semester

RQ 1. Do the ECC project services have the intended positive effects on student outcomes? 90.0% 80.0%

Spring 2011

70.0% 60.0% 50.0%

Average College Wide Success Rate

40.0%

CAF Program Success Rate Goal

30.0%

CAF Overall Success rate (withdrawals included)

20.0% 10.0% 0.0% Math092

ENG096

Success rate is the number of students who passed the course divided by the number of students enrolled.

RQ 1. Do the ECC project services have the intended positive effects on student outcomes? Table 3. MTH-090 Grade Distribution and Success and Withdrawal Rates, by Section Total Total Success Rate Enrolle Completing Withdrawals Withdrawal Section d Withdrawals Course Included Rate AF1 30 0 96.7% 0.0% 30 AF2 26 4 53.8% 15.4% 22 AF4 21 3 76.2% 14.3% 18 AF5 29 4 75.9% 13.8% 25 AF6 15 5 13.3% 33.3% 10 Table 4. ENG-090 Grade Distribution and Success and Withdrawal Rates, by Section

Total Total Success Rate Enrolle Completing Withdrawals Withdrawal Section d Withdrawals Course Included Rate AF1 24 1 87.5% 4.2% 23 AF2 25 8 48.0% 32.0% 17 AF4 23 2 73.9% 8.7% 21 AF5 21 0 81.0% 0.0% 21 AF6 16 2 62.5% 12.5% 14

Spring 2011 The charts below representthe grade distributionfor CAF and the Division of Mathematicsand Physics. Data reflectsa 68% successrate for those studentstakingthe Mathcourse in comparison to their counterpartstakingthe traditionalMTH 092 whose successrate is 40%. In the ENG 096, the CAF courseshad 69% successrate as comparedto 60% in the traditionalENG 096 courses.

MTH 092 Sect. AF 1 AF 2 AF 4 AF 5 AF 6 TOTAL

Total 30 26 22 29 15 122

A's 4 4 2 6 0 16

B+ 7 1 4 5 0 17

B 2 3 4 5 1 15

C+ 5 4 0 1 0 10

C 11 2 6 5 1 25

D 0 0 0 0 0 0

F 1 8 2 3 8 22

I 0 0 0 0 0 0

W 0 4 4 4 5 17

A-C 29 14 16 22 2 83

A-F 30 22 18 25 10 105

All 30 26 22 29 15 122

Pass* 97% 64% 89% 88% 20% 79%

Success 97% 54% 73% 76% 13% 68%

25 26 23 21 16 111

3 0 1 2 0 6

2 2 3 4 7 18

5 2 3 4 2 16

4 4 0 1 0 9

7 4 10 6 1 28

0 0 0 0 0 0

2 5 4 4 4 19

0 0 0 0 0 0

2 9 2 0 2 15

21 12 17 17 10 77

23 17 21 21 14 96

25 26 23 21 16 111

91% 71% 81% 81% 71% 80%

84% 46% 74% 81% 63% 69%

1944 122 1822

170 16 154

129 17 112

140 15 125

136 10 126

232 25 207

80 0 80

634 22 612

16 0 16

407 17 390

807 83 724

1521 105 1416

1944 122 1822

53% 79% 51%

42% 68% 40%

1504 111 1393

168 6 162

152 18 134

212 16 196

141 9 132

244 28 216

15 0 15

363 19 344

18 0 18

191 15 176

917 77 840

1295 96 1199

1504 111 1393

71% 80% 70%

61% 69% 60%

ENG 096 AF 1 AF 2 AF 4 AF 5 AF 6 TOTAL ALL MTH 092 CAF less CAF ALL ENG 096 CAF less CAF

* Pass rate = A,B+,B,C+,C/A,B+, B, C+, C, D, and F * Success Rate = A, B+, B, C+, C/ all grades including W and I.

RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? • Scheduling The CAF program successfully grouped the components of the courses to support block scheduling. • Each CAF course required the same group of students to meet in three different session types (i.e., classroom, tutorial, and lab).

• Students in CAF Math were not necessarily in CAF English with the same group of students

RQ2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? Supplemental Instructors • Typically advanced ECC students or recent graduates attending four year college • Varied in experience and skill • Established positive rapport with students • Administrators matched inexperienced supplemental instructors with lead teachers

RQ2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? Inclusion of technology: • Students and instructors reported that the math lectures, computer programs, and tutorials aligned rather seamlessly in the MTH-100 course. • Students interpreted the grammatical issues presented by the webbased tutorial program and the supplemental instructors (in tutorial sessions) as distinct and discrete topics, unrelated to writing and reading comprehension topics presented by the course instructor. • Students were more motivated to complete computer-aided assignments in math than in English

RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? • Program Culture • The CAF program built a trusting environment in which students felt comfortable seeking help from instructors and supplemental instructors.

Table 8. CAF ENG-096 Student Survey Responses Related to General Perceptions of School Support English - How much does this college emphasize each of the following? Please select the most appropriate answer. Very Little Some Quite a Bit Very Much Encouraging you to spend significant 3.2% 12.7% 46.0% 38.1% amounts of time studying Providing the support you need to help you succeed at this college

3.2%

11.1%

38.1%

47.6%

Encouraging contact among students from different economic, social, and racial or ethnic backgrounds

3.2%

25.8%

25.8%

45.2%

Helping you cope with your nonacademic responsibilities (work, family, etc.)

6.5%

21.0%

33.9%

38.7%

Providing the support you need to thrive socially

3.2%

11.3%

37.1%

48.4%

Providing the financial support you need to afford your education

4.8%

11.1%

33.3%

50.8%

Using computers in academic work

4.8%

11.3%

24.2%

59.7%

Table 9. CAF MTH-092 Student Survey Responses Related to General Perceptions of School Support Math - How much does this college emphasize each of the following? Please select the most appropriate answer. Very Little Some Quite a Bit Very Much Encouraging you to spend significant 4.1% 16.3% 34.7% 44.9% amounts of time studying Providing the support you need to 2.1% 16.7% 37.5% 43.8% help you succeed at this college Encouraging contact among students from different economic, social, and racial or ethnic backgrounds

6.1%

14.3%

38.8%

40.8%

Helping you cope with your nonacademic responsibilities (work, family, etc.)

8.2%

16.3%

36.7%

38.8%

Providing the support you need to thrive socially Providing the financial support you need to afford your education Using computers in academic work

4.2%

18.8%

35.4%

41.7%

0.0%

18.4%

30.6%

51.0%

2.0%

16.3%

32.7%

49.0%

RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? Program Culture • The students did not develop the sense of community that learning communities are designed to foster and that have been shown to have positive effects on student achievement. • CAF instructors did not provide substantive guidance to students related to collaborative learning.

Table 10. CAF Math and English Pre-Post Confidence and Self-Efficacy Mean Pre-test sample size

Pre-test mean

Post-test sample size

Post-test mean

Math Confidence English Confidence

67 71

3.18 3.51

46 59

3.41 3.55

Math Self-Efficacy English Self-Efficacy

75 63

7.58 7.76

55 42

7.74 7.7

RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? General Teaching Approach • Across math and English courses, the instructors implemented procedural teaching techniques. • The teaching methods did not encourage students to engage proactively in the “process” of learning.

RQ3. In what ways can the services be modified to maximize their effectiveness? • Administrators should provide professional development training in “scaffolding” • Instructors should emphasize analysis and critical thinking • Instructors should ensure that all components of a course are clearly connected • Instructors should lead activities that foster interaction and collaboration between students • Ensure that instructors are punctual and present at each class session

Fall 2011 – Success Rate Update 90.0% 80.0%

Average College Wide Success Rate

70.0% 60.0%

CAF Program Success Rate Goal

50.0% 40.0%

CAF Overall Success rate Spring 2011 (withdrawals included)

30.0% 20.0%

CAF Overall Success rate Fall 2011 (withdrawals included)

10.0% 0.0%

Math092

ENG096

Success rate is the number of students who passed the course divided by the number of students enrolled.

Fall 2011 Comparison Data

course AFE083 ENG085 AFM083 MTH086 ENG096AF ENG096MTH092AF MTH092-

NO_ NO_A NO_B+ NO_B NO_C+ NO_C NO_D NO_F NO_I W A-C A-F A-W Previous Enrolled Success Rate 45 43 85 46 96 5 109 2 74 315 429 505 431 505 62.4% 114 103 202 108 132 1 230 2 58 659 890 950 950 957 69.4% 105 54 82 39 95 13 148 33 118 375 536 687 603 694 54.6% 250 136 213 122 217 57 701 19 342 938 1696 2057 2057 2164 45.6% 20 27 17 17 19 0 21 0 17 100 121 138 138 138 72.5% 155 123 194 134 189 9 323 4 168 795 1127 1299 1299 1307 61.2% 18 4 12 8 15 1 31 2 24 57 89 115 115 115 49.6% 1390 106 48 98 71 134 67 504 19 343 457 1028 1390 1403 32.9%

Spring 2012 – Evaluation Update Method

Time Frame

Administrative Data – Success Rates and Withdrawal Rates

Post Semester

Course Observations

Pre-Post Semester

Instructor Interviews

Pre-Post Semester

Supplemental Instructor Interviews

Post Semester

Student Focus Groups

Post Semester

Self-Efficacy and Subject Matter Confidence Surveys

Pre-Post Semester

Competing Priorities and Barriers Scales

Pre Semester