The SCALE Efficiency Projects

JALN Volume 2, Issue 2 - September 1998

The SCALE Efficiency Projects Lanny Arvan* John C. Ory** Cheryl D. Bullock** Kristine K. Burnaska** Matthew Hanson** * Sloan Center For Asynchronous Learning Environments, University of Illinois at UrbanaChampaign and **Office of Instructional Resources, University of Illinois at Urbana-Champaign Correspondence on this paper should be sent to: Lanny Arvan, Director SCALE 1406 West Green Street Urbana, Illinois 61801-2291 Phone: 217-333-7054 Email: [email protected] ABSTRACT This paper presents evidence from nine “Efficiency Projects” that were SCALE’s focus in the 1997-98 academic year. The Efficiency Projects were specifically aimed at using ALN to achieve higher student/faculty ratios, without sacrificing instructional quality. The study concentrates on data amassed for the fall 1997 semester. Evidence was collected on the cost side, for ALN development and delivery, and the performance/attitude side, from both student and faculty perspectives. The study supports the view that when a sensible pedagogic approach is embraced that affords the students with avenues to communicate about their learning, ALN can produce real efficiency gains in courses without sacrificing the quality of instruction.

KEY WORDS Efficiency Projects Quality of learning

I. INTRODUCTION This paper presents evidence from nine “Efficiency Projects” that were SCALE’s focus in the 1997-98 academic year. The study concentrates on data amassed for the fall 1997 semester. Evidence was collected on the cost side, for ALN development and delivery, and the performance/attitude side, from both student and faculty perspectives. The Efficiency Projects were specifically aimed at using ALN to achieve higher student/faculty ratios, without sacrificing instructional quality. The higher student/faculty ratios occurred in some cases by increasing the number of students taught, in other cases by reducing the size of the instructional staff. One common feature shared by these projects is class size. All were in large

33

JALN Volume 2, Issue 2 - September 1998

undergraduate classes. Another common feature was the reliance to some degree on automatic, Web-based grading software. Yet there were substantial differences across these projects. The courses were in Chemistry (General Chemistry and Advanced Organic Chemistry for Biology majors), Circuit Analysis (Introductory), Differential Equations, Economics (Microeconomics Principles and Intermediate Microeconomics), Microbiology (Introductory), Spanish (Intermediate Grammar), and Statistics (Introductory for non-technically oriented students). The Principal Investigators (PIs) on these projects differed in their experience in teaching with ALN. Some were among the original SCALE grantees (and among these some had significant relevant prior experience). Others had less experience. Indeed, the Spanish PI was a relative computer novice and was using ALN for the first time. Some of the courses relied heavily on graduate assistants. Others used undergraduate peer tutors. Some of the courses used asynchronous conferencing primarily as a means for providing help to students. Others used synchronous text-based chat for this purpose and used asynchronous conferencing as a means for students to do written work online. In some cases the developers of the online materials were also providing the face-to-face delivery of instruction. In other cases the authoring, presentation, and coaching functions were separated across individuals. Some courses retained the traditional lecture intact. Others substantially reduced face-to-face contact hours. With all this variation, it is probably better for the reader to interpret the results as a collection of case studies rather than as a cross section of evidence on ALN, viewed as a precisely pinpointed approach to online instruction. We do try to draw some general conclusions where we think it appropriate, both about ALN instruction in a large class setting and about using ALN to attain efficiency ends. A critical issue is the extent to which the findings presented in this paper are replicable. A big part of the replicability question is attributing the results to ALN or to the PIs themselves. In the vast majority of the Efficiency Projects, the PIs were early adopters with a great deal of enthusiasm for online teaching. Whether the results translate to mainstream faculty remains an open question. Another factor is the general computing environment. Computer technology permeates daily life at the University of Illinois at Urbana-Champaign (UIUC) and thus it might be equally important to ask whether the results would hold at a campus where computing is less firmly imprinted into the culture. Yet another significant issue is the extent of up-front learning needed, for the support organization as well as for the instructors, as a precursor to any program aimed at utilizing ALN for efficiency ends. Whether others would require the two-year lead for general ALN development, as did SCALE, is an open question. To give the reader more of context for these and related issues, we briefly review how SCALE has grown, from its origins to the present. A. Brief SCALE History SCALE was formed in spring 1995 with a $2.1 million grant from the Alfred P. Sloan foundation and a generous match from the University of Illinois at Urbana-Champaign. The grant covered a three-year period that ended after the spring 1998 semester. The goal was to bring 15 ALN courses online a year. In fact, in the 1997-98 academic year there were approximately 80 courses per semester supported by SCALE. These courses enrolled about 8000 students per semester. The Efficiency Projects represent only a small number of the courses supported by SCALE, but account for about half the enrollments. SCALE’s primary mission was to support ALN course development in an on-campus setting. Initially, Sloan had set four targets for this on-campus ALN to achieve. These were to improve retention, to decrease time to degree, to demonstrate verifiable increases in student learning, and

34

JALN Volume 2, Issue 2 - September 1998

to lower the cost of instruction. Over time, these targets have been modified, based on the experience with ALN and its implementation on the UIUC campus. While there were some Web-based activities in SCALE courses at the outset, the bulk of the initial ALN work entailed use of asynchronous conferencing. At the start there were two asynchronous products supported, the now defunct PacerForum and the still popular FirstClass. Both of these are client-server software. The user must have the client installed on the desktop computer where the user is working. The client allows the user to access the server over the network. It is the server where all the information is stored. In 1996-97 SCALE dropped support of PacerForum but began to support Web-based conferencing, where the Web browser (initially Netscape Navigator, later also Microsoft Internet Explorer) serves as the client. Web-based conferencing allows for a more seamless movement between course-related Web materials and the conferencing environment, a distinct advantage. In 1996-97 SCALE supported the product WebNotes. In 1997-98, SCALE switched to WebBoard and will continue to support use of this product in the upcoming year. In spite of the increasing popularity of the Web, many SCALE faculty continue to use FirstClass. The reasons for this loyalty to FirstClass are many and varied: 1) they have had good success with it in the past, 2) it is what they know and don’t want to have learn something else, and 3) they view the current Web-based alternative as inferior. Increasingly, SCALE faculty have come to put their ALN materials on the Web. In addition to the standard syllabus and lecture notes, simulations (primarily in science and engineering courses) that had previously been delivered via dedicated client software were moved to the Web. Moreover, after the pioneering projects in 1995-96, much effort was put into authoring questions for CyberProf [1] and Mallard [2], products developed at UIUC for allowing students to selfteach via intelligent assessment of short-answer questions delivered through the Web. Without a doubt, Web delivery became an increasingly important component of the teaching strategy in SCALE-supported courses. Apart from an evolution in the technology, there was also a transformation in the pedagogy. Over time, the original grantees came to increasingly trust their ALN teaching approach. ALN became less of an experiment and more an established style, with a heavy emphasis on the assignments that students were to complete. This regularizing of ALN allowed SCALE to provide its support in a consistent, well-prescribed manner. It also allowed grantees who got started in year two of the grant and even more so in year three to get current with ALN teaching in an accelerated manner. They had to learn the ALN software, to be sure, but there was less need to tinker with the pedagogy and wonder if it would work. There has been an independent evaluation team from the start of the SCALE project. That team is headed by John C. Ory and includes Cheryl Bullock and Kristine Burnaska. They produced semester-by-semester evaluations starting in fall 1995 and culminating in spring 1997 [3],[4],[5],[6]. Matthew Hanson joined the evaluation team in summer 1997, to work exclusively on the Efficiency Projects. While the evaluation team has been in frequent contact with SCALE administration and, in particular, the evaluation strategy of the Efficiency Projects was discussed extensively, the actual data collection effort has been the sole province of the evaluation team. This independence helped to minimize the chance of misrepresentation of the findings and to reduce the awkwardness involved in the data collection, particularly in those cases when students or faculty reported that things weren’t going so well. B. Sources of Productivity Improvement Studies of computer technology use aimed at increasing instructional productivity are quite rare. The Rensselaer Studio Courses offer one example [7]. Some work done at Michigan State

35

JALN Volume 2, Issue 2 - September 1998

University by Ed Kashy, Michael Thoennessen, et. al., [8], [9], is closer in spirit to the present study. That is essentially the entire list. That such work is indeed rare is confirmed by Polley Ann McClure [10]. “While there are some cases in which we can document improved educational output as the result of technology intervention, in a brief survey of the literature, I could find no studies documenting improved educational output per unit cost. The educational gains have been at huge cost, in terms of investment in both equipment and software, but more significantly, in faculty and support staff time.” Similarly, David Noble [11], a notable opponent of online education, cites the work of Kenneth Green [12] when arguing, “Recent surveys of the instructional use of information technology in higher education clearly indicate that there have been no significant gains in either productivity improvement or pedagogical enhancement.” That such documentation is so rare suggests two potential explanations: (1) it is not possible to generate productivity increases with computer technology, and (2) it is possible, but the incentives are not right for us to witness them. Robert Koob [13] makes a convincing case for the second hypothesis. Yet affirming that incentives are weak does not in itself prove that computer technology can generate instructional productivity gains. More direct evidence is needed and that provides the raison d’être for our study. Strategic thinking about how instructional technology should be used for advancing productivity ends has clearly outstripped the empirical work in this area. Much of this strategic thinking has come out of Educom’s National Learning Infrastructure Initiative (NLII). Examples include the papers by Carolyn Twigg [14], [15], D. Bruce Johnstone [16], and William F. Massy and Robert Zemsky [17]. The ideas behind the SCALE Efficiency projects have been influenced by this work. Yet it should be understood that making these ideas operational requires compromise, in both the implementation and in the measurement. It is our hope that this paper gives the reader some insight into the type of compromises that are needed to get actual productivity projects underway and the variety of measurement problems that arise as a consequence. Furthermore, there is a fundamental conceptual point that should be considered where the NLII philosophy departs from the ALN philosophy. The basis of the NLII thinking is that educational technology is capital and that any productivity gains must come as capital input substitutes for labor input. While this capital for labor approach is not entirely absent in the ALN approach, it is not the whole story. With ALN, much of the productivity increase comes from labor-for-labor substitution – inexpensive student labor for expensive faculty labor. (The TLT Affiliate of AAHE headed by Steven Gilbert and Stephen Ehrmann, [18], vigorously argues for more of this type of labor-for-labor substitution, but to date they have concentrated their focus on the instructional technology support arena rather than in the online classroom itself.) Viewing the students’time as a productive input, as suggested by Lanny Arvan [19], some of this productivity gain arises from peer-to-peer communication. (Note that we don’t cost-out this student time in the measurement component of this paper, however, some demographic evidence suggests how such a costing out should be done[20].) Additional productivity gains emerge from student interaction with peer tutors who receive remuneration for the help they provide. In the ALN approach, it is critical to view networked computers as, in part, communication tools. This allows the ALN approach to make the instruction more personal while simultaneously increasing productivity. At least, that is the ideal. There has been a change of thinking within SCALE administration about how to deliver on the Sloan objectives. During the first year of the SCALE project, there was an expectation that the desired efficiency outcomes would come as a byproduct of ALN implementation. This due to the enhanced peer-to-peer interaction and the avoidance of wasteful duplication of effort through the instructor answering common student questions once, via posts to a public class conference. It

36

JALN Volume 2, Issue 2 - September 1998

was also expected that efficiency gains could be had in all ALN courses. In fact, most SCALEaffiliated faculty reported increased time involved in instruction as a long-term proposition, because of the increased contact with the students online. Subsequently, some of these instructors have modified their views about the need to be online so frequently for their students to have good access. But the view remains that ALN teaching is arduous. Thus, it became apparent that the byproduct approach would not achieve the desired results. Moreover, it also became clear that efficiency outcomes would be difficult or impossible to attain in small ALN classes. There were two reasons for this that perhaps should have been obvious at the outset of the project but were not. First, if there was substantial up-front development in a small class, such development could not be amortized over a large number of students. Second, in a small class there is very limited opportunity to exploit labor-for-labor substitution. When SCALE administration ultimately contracted for the efficiency projects [21], SCALE targeted large classes only. Another consequence of abandoning the byproduct approach was the need to put in specific incentives to produce efficiency outcomes. After the first year of SCALE, grants to PIs were reduced so that more projects could receive funding. This trend was reversed for some of the Efficiency Projects, which received grants that were as large as those grants given in the first year. Moreover, SCALE was able to obtain assurances from the UIUC administration that any savings produced could be retained within the department where those savings were generated. C. Further Caveats In the main, the SCALE Efficiency projects represent mature ALN development in large classes where the ALN has now been focused on efficiency ends. There are many other ALN courses that SCALE currently supports where no attempt is being made to produce efficiency outcomes. Among these are some large classes. Thus, we are not arguing that large size per se makes a class a good candidate for an efficiency project. For example, SCALE supports an introductory comparative literature course that enrolls about 250 students a semester. The course is taught with a lecture once a week. There are also small sections run by graduate assistants under the supervision of the faculty member who delivers the lecture. The course is writing-intensive and satisfies the campus Composition II requirement. In spite of the course size, the possibility for capital substitution is limited here. Competent evaluators must assess the students’ written work. Computer assessment of the writing is not possible, because the assessment is so contextually based. It can’t be done via a search for key words. This requirement of competent assessment also limits the possibility of labor-for-labor substitution in this course. We think that ALN is improving learning, but we have no way to quantify the learning, so this course is not one of our Efficiency Projects. There are also SCALE-supported courses currently taught in such an inexpensive manner – large lecture with few if any graduate assistants to support the course – that it seems foolhardy to try to further reduce the cost of instruction. We are also not arguing that the SCALE approach can work everywhere, technological considerations aside. The reliance on peer tutors, in particular, requires highly able students who can serve in this capacity and feel they are doing something socially beneficial in the process. The SCALE approach likely can work well at other institutions in the Big Ten and at other similarly regarded public campuses. To what other institutions the approach can be profitably extended is an open question. One further point bears mention here. There has been a negative reaction to using educational technology for efficiency ends, emerging from various pockets of concerned faculty [22], [23]. Much of this reaction relates to the effect on faculty employment. The capital substitution argument would seem to suggest a need for fewer faculty. Certainly there is a fear that this will be the case. Reducing faculty employment is viewed as ‘bad’ in many quarters. It is our view

37

JALN Volume 2, Issue 2 - September 1998

that on the UIUC campus the SCALE Efficiency Projects will have little or no impact on faculty employment, though we do anticipate a big impact from these projects overall. It is graduate student employment that will be affected the most dramatically, if the SCALE Efficiency Projects become more widespread on campus. The reason for this is simple. In the vast majority of the courses that SCALE has been targeting, graduate students do the bulk of the teaching. The course coordination function remains in the hands of a faculty member, even with ALN. The upside of this is to reduce the pressure on graduate student enrollment to staff large introductory undergraduate courses. This should allow graduate student enrollment to better track the new Ph.D. job market in the individual discipline and to better match the quality of the particular degree program. Furthermore, to the extent that the changes in graduate student enrollment can be made without disenfranchising students who are currently enrolled, simply by adjusting the size of entering cohorts, it is not obvious that there is a downside to this approach.

II. A SIMPLE PRODUCTIVITY MODEL All the Efficiency Projects entailed at least some up-front development. This development can be thought of as part learning -- both attaining a comfort level with the software and formulating a successful pedagogic strategy -- and part authoring/online publishing. In collecting the data, development costs were grouped into three categories. First, there are faculty costs (e.g., course buyouts and summer support). Second, there are programming costs (e.g., hourly wages or assistantship support of student programmers). Last, there are equipment costs (e.g., the cost of desktop computers, the pro rata share of server and license costs allocated to the particular project, and the cost of software). Subsequent to the up-front development, each Efficiency Project produced some recurrent benefit. In courses where the overall enrollment remained unchanged, this benefit can be envisioned as a reduction in operating costs, measured on a semester-by-semester basis. Operating costs also include three components. First, there is the pro rata share of the faculty member’s salary (plus benefits) allocated to teaching the ALN course. Second, there is the cost of other course personnel, either graduate students on assistantship or student hourlies. Last, there is the pro rata share of common costs, particularly SCALE support staff. It is helpful to think of the total benefit per semester as the product of two factors: 1) the reduction in operating cost per student and 2) the number of students in the class. This is the entire benefit when overall course enrollment remains unchanged. The benefit calculation is a bit more complicated when ALN allows for an expansion of overall enrollment. Enrollment expansion can occur only if there had been unsatisfied demand for the course, in which case the benefit on the cost side itself has two components and there is a benefit on the demand side as well. These three components are the operating cost reduction on the original class size, the imputed operating cost reduction on the increase in class size, and the benefit that accrues to those students who would have been rationed out had course capacity not expanded. We did not try to measure this third component. We simply note that measurement of the first two components understates the recurrent benefit. A goal from the outset was to measure all costs in dollar terms, to best make comparisons between the various cost components. Thus, with the exception of student hourlies, there was no attempt made to characterize the time entailed in doing the work, say for the instructor authoring the on-line materials. The approach we took differs markedly from time and motion studies. Instead of measuring the time input directly, we measured the dollar amount needed to elicit the

38

JALN Volume 2, Issue 2 - September 1998

requisite time input. In most cases we did this by directly measuring outlays, either in SCALE grants to the PIs, or in actual salary numbers. In some cases we had to make imputations for compensation and where we did that we go to some length to describe the calculations. The most contentious of these imputations was determining the share of faculty salary allocated to teaching a course. In most cases we simply assigned 25% of the nine-month salary to the course, because the typical campus-teaching load is two courses per semester in both the fall and the spring. That we used actual outlays bears mention for several reasons. First, the Efficiency Projects are among a group of successful SCALE projects. A few of the original projects did not succeed. We made no attempt to adjust our development cost measure for the risk of project failure, because there are offsetting biases in measurement. Second, many of the Efficiency Projects were among SCALE’s original projects and have received several rounds of funding. The funding was given out annually and with each grant there was no guarantee of further funding in a subsequent grant. Thus, the measured outlays might differ substantially from the case where a multiyear development cycle was planned from the outset. Third, returning to the question raised in the introduction about whether the results apply to mainstream faculty, there is the issue of how the size of the SCALE grants compare to right level of compensation for the effort; too much, too little, or just right. Last, because others who might embark on such an ALN program now will have the benefit of learning from those who have preceded them, their development costs will likely not be as great as in the case of the SCALE projects. Where possible, a computation was made to determine the number of semesters it would take to recover the up front development cost. This calculation was done twice, once in the case of no discounting and then again when future benefits were discounted. The effect of discounting is to lengthen the period of cost recovery. We used a rather conservative interest rate, 9%, so that we would have reliable bounds on the period of cost recovery [24]. This cost model clearly oversimplifies matters. Authoring does not occur entirely up front. Teaching with ALN is an iterative process. Modifications are made to the on-line materials based on the actual experience of teaching a class. The philosophy behind the way these authoring costs were allocated in this study is as follows. Authoring that takes place during the semester is treated as operating cost. Authoring that occurred in the summer and for which the PI was compensated via a grant from SCALE is treated as up front development.

III. CATEGORIZING THE EFFICIENCY PROJECTS In an attempt to make some relevant cross-project comparisons, we categorize the productivity increase by whether the scheduled contact hour is with a teaching assistant or a faculty member. This distinction is relevant for at least two reasons. First, in terms of converting instructor time to dollars, teaching assistants are paid on a more or less uniform basis. There is much more variation in faculty members' salary, due to variation in rank and variation in compensation across disciplines. Moreover, since faculty obligations are a bundle of teaching, research, and service, with the fraction of the obligation somewhat idiosyncratic, it is hard to parse out the teaching component. The precision of the cost estimates should be understood in that light. Second, this distinction, at least on the UIUC campus, represents the extent to which the instructor must adopt the ALN innovation as terms of employment. Departmental standards determine the graduate assistant load associated with a 50% time appointment, a 33% time appointment, etc. Departments can and do change these standards upon occasion, for reasons quite unrelated to

39

JALN Volume 2, Issue 2 - September 1998

adoption of ALN. Moreover, if some courses are taught with ALN and other not, departments can make the standard flexible to accommodate that distinction. Teaching Assistants with the same fractional appointment may teach three ALN sections but only two non-ALN sections. The point is that departments can implement this, as long as in their judgment the burden on the TAs is roughly equal under either approach and matches the fraction of the appointment. In contrast, such differential loads cannot be imposed upon faculty members without prior consult and approval. When the department that houses the project captures the productivity increase, it occurs either by the instructor teaching more sections or by the instructor teaching more students per section. With the first, students should perceive no difference in quality, as from their perspective the class has not changed. With the second, student perceptions of quality (as well as objective measures of student performance) provide evidence about the consequences on course quality. The SCALE project in introductory Chemistry had the TAs teach more sections. Seven of the other projects achieved the increase in productivity via larger sections. When the instructor captures the productivity increase, the instructor's workload should drop. The SCALE project in Microbiology introduced virtual labs to replace wet labs in some cases. During the weeks when the virtual labs were given, the graduate lab assistants were able to focus their attention on their research rather than their teaching. Though how the productivity increase is captured is an important variable, we chose not to use this dimension to differentiate our projects because there was not enough variation. All but the project in Mathematics utilized on-line quizzing with automated grading to some degree. Students receive the benefits of immediate feedback and repeated tries at the material. A concomitant benefit is that instructors are relieved from the burden of grading. Yet there is more to a large ALN course than automated grading of assignments. These ALN courses have a substantial component of people-to-people interaction online. Some of this is peer-to-peer interaction. The rest is between student and course staff, much of it through scheduled on-line office hours, which are more extensive than their traditional counterpart. The on-line office hours are frequently in the evening, when they are convenient for the students. Office hour staffing is made affordable by having undergraduate peer-tutors, by reducing the number of class contact hours, or some mixture of the two. Though there were some qualitative differences across projects in how these office hours were conducted, all of the projects relied on the on-line office hours extensively. Again, though automated quizzes and on-line office hours are very important components of our Efficiency Projects, there is not enough variation across projects to focus on these components as a means to categorize the projects. For this reason, we searched for another dimension along which the SCALE Efficiency Projects are differentiated. We ended up focusing on whether all the assignments are machine graded (short answer), or if there was still some long answer work graded by TAs. In so doing there are several issues we were trying to address. First, automated grading may seem reasonable to some instructors in some disciplines, but may appear inappropriate elsewhere. As it turned out, in several of the projects the two forms of grading were used in mixed mode with the intent of achieving the best that both have to offer. (Here we are focusing on assignments completed out of class. One could also look at this distinction with exams. Interestingly, some SCALE supported courses that have all short answer assignments have long answer exams while other courses with some long answer assignments have short answer exams.) Second, in some of the courses undergraduate peer tutors have also been utilized as graders. This is controversial. On the one hand, by using undergraduate graders there is an inexpensive supply of grading assistance that the instructor can tap from those students who have taken the course previously. On the other hand, some faculty are suspicious that undergraduates do not have the requisite depth of knowledge of

40

JALN Volume 2, Issue 2 - September 1998

the subject to provide good written feedback to the students. Third, it may be that written work done online and the assessment that goes along with it are simply different from the paper analog. In SPAN 210 much of the written feedback was actually given by other students as they made responses to original posts. The graders' job was much more to acknowledge these efforts by the students and much less to participate in the dialog with additional feedback. In ECON 300, the rapid transmission of the work coupled with the large number of graders, allowed for rapid response to the submission. This, in turn, allowed the teams to resubmit homework in response to the comments of the grader, a practice that is exceedingly rare with paper-based homework.

Productivity Increase/ Grading of Homework Grad Assistants

All Automated Grading

Some Human Grading

BIO 122 (CyberProf) A CHEM 101, 102 (CyberProf) ECON 102 (Mallard)

SPAN 210 (Mallard) B

Faculty

STAT 100 (Mallard) C ECE 110 (Mallard) CHEM 331(WebCT) Table 1. Efficiency Matrix

ECON 300 (Mallard) D MATH 285 (Mathematica)

IV. MEASURING QUALITY OF INSTRUCTION In the contract made with Frank Mayadas, the efficiency project courses were to be redesigned so as to improve the quality of learning or to hold the quality of learning constant. Apparent efficiency gains that resulted in a deterioration of quality were deemed out of bounds. As we subsequently document, these projects have been successful in lowering expenditure per student. Elsewhere, expenditure per student is itself regarded as a quality indicator, with greater expenditure indicating higher quality. (For example, see Money Magazine’s college ratings [25].) Expenditure per student can be viewed as a measure of input quality. We report on our efforts to measure the effect of ALN on the output quality in several Efficiency Projects. To assess changes in course quality due to the use of ALN, in the ideal, we would have the same instructor teach both an ALN section and a non-ALN section of the course and administer a common performance standard for both sections. (Jerald Schutte has done a study along these lines [26]. He finds that the on-line approach significantly outperforms the traditional approach. But he is unable to control for his own teaching effort across the two sections.) In addition, students would be randomly assigned to the sections. We could then look at indicators of student performance and of student satisfaction as measures of output quality. Moreover, we would have an appropriate benchmark to which we could compare the ALN approach. As we described in the introduction, we had to make several compromises in implementing our study. In retrospect, we think an ideal study may be impossible to implement, because of limited resources and the ethical issues such a study raises. The ideal study can be implemented most easily when beliefs about the teaching approach are neutral. The more it is believed that ALN is superior to the traditional approach (or vice versa) the harder it is to implement the study. Instructors do not want to be shown that their teaching is inferior. Students do not want to take the version of the course that will make them less prepared to do well on the exams. And administrators do not want to continue with the traditional approach if ALN appears to afford a

41

JALN Volume 2, Issue 2 - September 1998

productivity advantage. We tried to come as close as possible to the ideal in our investigations of instructional quality. Whenever possible we attempted to collect quality measures that included assessments of student performance. We have more plentiful information about student and instructor attitudes.

V. THE FINDINGS In this section we proceed through the efficiency matrix presenting brief descriptions of the projects, a summary of the cost information, and the available output/quality information. Rather than go through the efficiency matrix in alphabetical order of the cells, we proceed on the basis of the quality of our evidence. A. Cell B of the Efficiency Matrix – SPAN 210 In 1996-97, an Italian professor successfully developed an ALN approach for her ITAL 101 and 102 courses. She designed vocabulary/grammar exercises for the students to complete using Mallard as well as writing assignments done using FirstClass. This professor serves as course coordinator while graduate TAs teach independent discussion sections. SPAN 210, a basic course in Spanish grammar, has a similar structure to ITAL 101 and 102. The idea behind this project was to build on the course development experience of the Italian professor and to thereby use ALN to begin to address the "Spanish Problem" on campus. At UIUC, and at most universities nationwide, the demand for Spanish language courses far exceeds actual enrollment, primarily because the ability to staff these courses is limited. This demand is fueled by the increasing internationalization of our economy. Students who wish to have a minor in international studies need competency in a second language. Spanish is the language of choice. On the UIUC campus, the Spanish problem will be exacerbated by a recently imposed increase in the foreign language requirement. Though much of the demand for Spanish is in the introductory courses, SPAN 210 has also had a chronic excess demand problem. There are students who have wanted to take the course but who have been unable to do so because all the slots were filled. To initiate this Efficiency Project, the Italian Professor searched the Spanish faculty for a willing participant, ultimately enlisting the SPAN 210 course-coordinator, who was drawn into this out of dissatisfaction with the exercises in the textbook she was using. This search occurred in spring 1997 in response to a call from SCALE administration. At the outset of the ITAL 101-102 project, in summer 1996, it was not envisioned that it would lead to a subsequent SPAN 210 project. For this reason, we are not including the development cost of the Italian project in the cost calculation for SPAN 210. In summer 1997 the Spanish professor and a graduate assistant began developing on-line materials for SPAN 210. In fall 1997 two out of nine regular sections of Span 210 were taught with ALN (utilizing both FirstClass and Mallard). Each ALN section was twice as large as a traditional section. The ALN section met only once a week while the traditional section met 3 times a week. This helped to keep the workload for the instructors uniform across sections. The professor also used ALN in a Discovery section. The campus Discovery program includes a set of courses that are for freshmen only, that are taught by tenured or tenure-track faculty, and that have class size capped at 20 students. All ALN and non-ALN sections used similar exams. In fall 1997 the use of ALN allowed the department to increase class size from 19 students to approximately 38 students in each of two sections. The department believed that by using ALN

42

JALN Volume 2, Issue 2 - September 1998

to teach all sections of SPAN 210 in the future they would be able to teach approximately twice as many students, without adding personnel. In fact, in spring 1998, all sections of SPAN 210 were taught with ALN and all have experienced a doubling of enrollment relative to historical norms. It is important to observe that the development period for ALN in Span 210 was only summer and fall 1997, where the Spanish professor authored the on-line material for Mallard. This shorter development cycle can be attributed, in large part, to the Italian professor’s prior development. As it turns out, however, the Spanish professor did not merely translate the questions that had been written in Italian, but rather wrote her own questions that suited her view of how to teach the subject. This wasn’t planned for at the beginning of summer 1997. It turned out that way because the Spanish professor developed an increasing enthusiasm for the enterprise. Nonetheless, we expect to see shorter development cycles in projects that are derivatives of earlier development. If these derivative projects are aimed at efficiency ends, they are likely to produce a dividend quickly. That is the case for the SPAN 210 project. Total development cost (measured by the size of the SCALE grant) was $15,336, divided roughly evenly between faculty summer support and student programming support. Even with discounting, it is clear the first full-scale ALN offering of this course in spring 1998 produced a cost saving that more than covered this development cost. This project has already produce a dividend. It has also paved the way for a further, more ambitious Efficiency Project in the introductory Spanish sequence. The SPAN 210 course is the closest we came to conducting the ideal study of output quality. The two ALN sections were compared to two non-ALN sections used as a control group. These four sections had common exams. We have the results from the two midterms. The ALN sections had approximately twice the number of students as the traditional sections, so in comparing the distributions one should focus on the cumulative distribution functions, not on the absolute number within each category. There was also a common attitudinal survey administered and a focus group for each section. Table 2 shows the results for Midterm 1 and Table 3, the results for Midterm 2 for the ALN and non-ALN sections of Spanish 210. Table 2 shows that the non-ALN section had more students at the extremes of the distribution. This implies the two distributions cannot be ranked via first order stochastic dominance. The non-ALN section had a slightly higher median, in the 91 – 93 range. The ALN section had a median in the 87 – 90 range. Table 3 shows the reverse. The ALN section had more students at the extremes. There were some drops in both sections, more percentage-wise in the non-ALN section. This explains, perhaps, the result at the lower extreme of the distribution. The medians were the same for Midterm 2, in the 87 – 90 range.

43

JALN Volume 2, Issue 2 - September 1998

Midterm 1 ALN non-ALN n % Cumulative n % Cumulative 97 - 100 2 2.56 2.56 3 7.50 7.50 94 - 96 16 20.51 23.08 6 15.00 22.50 91 - 93 12 15.38 38.46 11 27.50 50.00 87 - 90 10 12.82 51.28 10 25.00 75.00 84 - 86 9 11.54 62.82 4 10.00 85.00 81 - 83 8 10.26 73.08 1 2.50 87.50 77 - 80 4 5.13 78.21 2 5.00 92.50 74 - 76 7 8.97 87.18 0 0.00 92.50 71 - 73 5 6.41 93.59 0 0.00 92.50 67 - 70 3 3.85 97.44 2 5.00 97.50 64 - 66 2 2.56 100.00 0 0.00 97.50 61 - 63 0 0.00 100.00 0 0.00 97.50 60 & below 0 0.00 100.00 1 2.50 100.00 N=78 N=40 Table 2. Comparison of ALN and non-ALN Midterm 1 Results in Spanish 210 Midterm 2 ALN % Cumulative 10.67 10.67 14.67 25.33 16.00 41.33 17.33 58.67 12.00 70.67 13.33 84.00 8.00 92.00 0.00 92.00 2.67 94.67 1.33 96.00 1.33 97.33 1.33 98.67 1.33 100.00

non-ALN % Cumulative 5.71 5.71 20.00 25.71 8.57 34.29 22.86 57.14 20.00 77.14 5.71 82.86 8.57 91.43 2.86 94.29 2.86 97.14 2.86 100.00 0.00 100.00 0.00 100.00 0.00 100.00

n n 8 2 11 7 12 3 13 8 9 7 10 2 6 3 0 1 2 1 1 1 1 0 1 0 1 0 N=75 N=35 Table 3. Comparison of ALN and non-ALN Midterm 2 Results in Spanish 210

97 - 100 94 - 96 91 - 93 87 - 90 84 - 86 81 - 83 77 - 80 74 - 76 71 - 73 67 - 70 64 - 66 61 - 63 60 & below

We also perform a comparison of means, under the assumption that all observations occur at the midpoint of the cell. For example, all observations in the 97 – 100 range are treated as occurring at 98.5. Letting x denote the ALN outcome and y the non ALN outcome, we report values of the statistic zm = (xm – ym)/(sx2/Nx + sy2/Ny)1/2

44

JALN Volume 2, Issue 2 - September 1998

Under the null hypothesis that there is no difference in the means for the two classes, zm should have a standard normal distribution. For Midterm 1, we see that the non-ALN sections did score significantly higher than the ALN sections at the 90% confidence level, but not at the 95% confidence level. For midterm 2, there is no significant difference in the scores of the two sections, even at the 99% confidence level. ALN Sections non-ALN Sections mean var N mean var N zm Midterm 85.4 77.76 78 88.43 66.98 40 -1.85 Midterm 87.22 73.61 75 87.31 51.66 35 -0.06 Table 4. Comparison of means between ALN and non-ALN Midterm Scores in SPAN 210

Based on the Midterm 2 performance, it appears that doubling the class size and meeting with the TA only 1 day a week (instead of 3 days a week in the non-ALN sections) has not had an adverse affect on student performance. The attitude surveys indicate that students in the ALN sections had significantly less contact with their peers and with their instructor than students in the traditional section. Interestingly, the students in the ALN sections indicated they had significantly greater access to their instructors. Thus, the reduced interaction with the instructor in the ALN sections must be, in part, a matter of choice by the students. These findings are explained by the following. The traditional sections met for 3 hours a week. The ALN sections met one hour a week. During the other two scheduled hours, the instructor held office hours. (There were also office hours at other times.) This meant the students had no other obligations during at least some of the scheduled office hours and therefore should have been expected to report they had good access to the instructor. Office hours were voluntary. Many students did not avail themselves of this contact opportunity. The focus group discussions indicate that for a grammar course with the exercises in Mallard, many students did not perceive the need to discuss the material with the instructor. This is why there was less contact. The students perceived the course to be self-paced (though it did have the one weekly class session). Apparently, they did the work on their own, rather than in groups. In general, we would like to see a lot of student-to-student contact in an ALN course. But for this type of material, that contact may be unnecessary. There were three summative questions posed in the survey: 1. How difficult was the material? 2. Would you recommend the course to a friend? 3. How much did you learn? The responses to questions (1.) and (2.) did not significantly vary from the ALN to the non-ALN sections. Students found the material moderately easy and more would recommend the course than would not. On question (3.), the ALN students reported learning less than did the non-ALN students. This is somewhat surprising in light of the exam results. From the responses to this question and to the question about Mallard in the focus group, it appears that about 75% of the class liked the Mallard approach and thought they got something worthwhile out of it. But some of the students thought the Mallard exercises dull and would have preferred more human interaction. We suspect it is those students who reported not learning much in the course. It would be very interesting to know 1) whether these students had problems due to computer

45

JALN Volume 2, Issue 2 - September 1998

literacy and 2) how these students actually did on the exams. The surveys were anonymous so we do not have this information. It does raise the question, however, whether the students might be learning, at least as measured by the exams, without their perceiving it.

B. Cell C of the Efficiency Matrix – STAT 100, ECE 100, and CHEM 331 STAT 100 STAT 100 is a course that fills the UIUC’s relatively new General Education quantitative requirement. For most of these students it is the only quantitative course to be taken in college. Several years ago the experienced ALN instructor developed a software package for the course, WinPop, designed to be easy to use by the students (because many of these students were not very computer savvy) and to demonstrate fundamental statistical concepts in a highly visual manner. When SCALE came along, this instructor introduced FirstClass conferencing into the course. (He has since switched to WebBoard.) He also converted his software to Java, so it is visible in Netscape. He has recently started to use Mallard for administering on-line quizzes. By using ALN, this experienced instructor has increased the number of students he teaches in his section of STAT 100. As shown in Table 4, enrollments in STAT 100 continue to increase. The experienced ALN instructor is working with other professors who teach the course to get them to use ALN. The hope is that all professors who teach STAT 100 can raise their enrollment levels by using ALN and subsequently accommodate the expanding enrollments without increasing instructional delivery costs. Semester Number of Students Fall 1995 360 Spring 1996 351 Fall 1996 430 Spring 1997 426 Fall 1997 454 Table 5. Overall Enrollment in STAT 100 by Semester

Table 6 presents information on section-by-section enrollments in STAT 100. (Note that Discovery sections of STAT 100 are not included in the calculations in Table 6, but they do count in the overall enrollments in Table 4.) The ALN section that was taught by the experienced ALN instructor has been the largest section of STAT 100 since fall 1995. For the three semesters, fall 1995 – fall 1996, the ALN section had about 25 more students than the traditional sections. Overall demand ratcheted upwards in fall 1996. This was initially met by adding a traditional section. By spring 1997 it was apparent that this demand increase was permanent. Moreover, the department owed the campus a Discovery section of the course that it had not taught the previous spring. Thus, it had to accommodate over 420 students with only four sections. As an experiment, it was decided to have the entire increased capacity be borne by the ALN section. This experiment proved successful and enrollments in this ALN section remained high in fall 1997. Moreover, some of the other instructors began to experiment with ALN. To keep this experimentation from being too onerous, section size was reduced. This explains the increase in section size in the traditional sections that semester. The long-term plan is to have all sections be ALN and to have only four non-Discovery sections.

46

JALN Volume 2, Issue 2 - September 1998

ALN Section(s) Non-ALN Sections Average Number of Students Average Number of Students 104 (1 Section) 77 (3 Sections) 110 (1 Section) 80 (3 Sections) 101 (1 Section) 76 (4 Sections) 187 (1 Section) 80 (3 Sections) 154 (1 Section) 101 (2 Sections) 49 (2 Sections) Table 6. Average Enrollment Per Section in STAT 100 by Semester

Semester Fall 1995 Spring 1996 Fall 1996 Spring 1997 Fall '97

The large ALN section uses an undergraduate peer tutor at an estimated cost of $1,000 per semester, paid out of funds supplied by SCALE. This constitutes the only difference in operating costs between the big ALN section all other non-Discovery sections. The department provides approximately the same amount of money to be spent on faculty, TA, and grader support in all sections. Thus, it is clear that the big ALN section is more cost-effective than the other sections. We have a modest amount of comparative exam results from the fall 1997 semester. These come from common questions on one midterm. Four of the sections were involved in making these cross section comparisons. These were ALN1, the largest section and taught by the instructor who developed the materials; ALN2, the smallest of the four and taught by an instructor was using ALN for the first time, adopting the approach used in the largest section; and two other sections, non-ALN1 and non-ALN2, that used the traditional approach. ALN1 and non-ALN1 had questions 1 – 4 in common. ALN1, ALN2, and non-ALN2 had questions 5 – 7 in common. We did a comparison of means, on a question by question basis. Table 7 shows that ALN1’s students out performed the others. They had the highest mean scores on each question, significantly higher than non-ALN1’s students on questions 1 and 3, significantly higher than non-ALN2’s students on questions 5 – 7, and significantly higher than ALN2’s students on questions 5 and 7 at the 90% confidence level. The latter suggests that we cannot be sure whether the results are attributable to ALN or instead to instructor-specific effects. Nevertheless, the results should make one optimistic about ALN. It seems that in this class ALN is boosting student exam performance. This is all the more impressive considering that ALN1 is larger and has a lower cost per student. Regarding student attitudes, we have survey data only from the large ALN section. Thus we have no comparative information on student attitudes. The professor used Mallard for on-line quizzes. He used WebBoard for computer conferencing and has developed quite a lot of Web-based, highly graphical statistical material, to illustrate basic principles. In this use of the virtual environment, STAT 100 was quite similar to ECE 110. Moreover, the summative questions in the survey for STAT 100 were identical to those for ECE 110. Interestingly, the student responses were quite similar to those in the engineering course. Over 90% said they found using the Web easy or somewhat easy. Eighty-five percent rated their overall experience as good or better. And not quite 90% said they would probably or definitely take another course that used the Web. It must be emphasized that the STAT 100 students are non-technical (in contrast to the ECE 110 students). Consequently, it might be a reasonable inference that it is the teaching approach coming through in the responses to the summative questions.

47

JALN Volume 2, Issue 2 - September 1998

Question 1 Question 2 Question 3 Question 4

n 134 118 123 108 N=146

Question 5 Question 6 Question 7

n 142 142 119 N=146

ALN1 % 0.918 0.808 0.842 0.74

ALN1 % 0.973 0.973 0.815

var 0.075 0.155 0.133 0.193

var 0.027 0.027 0.151

n 69 65 54 57 N=88

n 42 50 41 N=54

non-ALN1 % var 0.784 0.169 0.739 0.193 0.614 0.237 0.648 0.228

ALN2 % 0.778 0.926 0.759

var 0.173 0.069 0.183

zm 2.707 1.22 3.812 1.471

non-ALN2 N % 44 0.454 72 0.742 64 0.66 N=97

var 0.248 0.191 0.224

ALN1 vs. ALN2 ALN1 vs. non-ALN2 ALN2 vs. non-ALN2 zm zm zm Question 5 3.349 189.6 4.273 Question 6 1.225 106.9 3.225 Question 7 0.84 46.4 1.318 Table 7. Comparison of Student Performance on Common Exam Items Administered in ALN and non-ALN Sections of STAT 100

ECE 110 Because of rapid changes in the field, the Department of Electrical and Computer Engineering has deemed it necessary that students get more hands-on experience in the laboratory beginning their freshman year. To achieve this end, the department has decided to move circuit analysis from the second year to the first year of instruction. The emphasis on hands-on instruction requires some de-emphasis on theory. The new course, ECE 110, takes a more basic approach to the theory than the old course, ECE 270. The PI is the main instructor in the lecture component of ECE 110. He has developed extensive materials for delivery in Mallard. All the homework is on-line and is automatically graded. The PI also makes extensive use of newsgroups/conferencing. In this way he can easily keep up with problems that students may be having with the material. ECE 110 has been taught with ALN from the outset. Thus there is no basis for comparison with a traditional version of the course. The lecture part of ECE 110 is less labor intensive than the analogous part of ECE 270. Indeed, with ECE 270 as the base, all development costs, about $65,000, had been recovered by the end of fall 1997. This should be interpreted cautiously, however. While some of the cost savings are undeniably due to ALN, some of the savings must be attributed to course restructuring. We have absolute attitudinal information from the course survey. Because a newsgroup was used in addition to Mallard, the survey refers to Web use rather than Mallard use. There were three summative questions in the survey. 1. How easy did you find using the Web for purposes of this course?

48

JALN Volume 2, Issue 2 - September 1998

2. How would you rate your overall experience using the Web in this course? 3. Would you take another course using the Web? Over 90% of the class reported that using the Web was easy or somewhat easy. Eighty percent of the class rated it as good, very good, or excellent. And Eighty-eight percent of the class reported they would probably or definitely take another class with the Web. The survey results indicate that the bulk of the students are happy with the way the course is delivered. We cannot tell, however, if that is due to student characteristics, the vast majority being electrical engineering students, or if instead it is due to characteristics of course design. For that reason, the comparison with STAT 100 is helpful. CHEM 331 CHEM 331 is the organic chemistry field course intended for students in the Life Sciences. The project is actually an outgrowth of the PI’s (and the students') displeasure with the use of lectures in the course. The instructor believed she could greatly improve the course by teaching it entirely on-line. The absence of face-to-face contact constitutes a more radical experiment than SCALE’s other projects. There are several potential efficiencies if the project is successful, including larger enrollments, lack of classroom-facility requirements, and additional revenue from extramural student tuition rates. The movement to a totally on-line format was more radical than was taken in the other SCALE courses. The fall 1997 semester was the first offering of the course in this mode. The professor advertised that the course would be taught this way. Nonetheless, she reported that a full Twentyfive percent of the students who signed up for the course did not have the requisite computer literacy. Either the advertising failed or the students ignored it, perhaps because there is only one section of CHEM 331 per semester. It is fair to say that some of the negative responses to the summative questions on the CHEM 331 survey are due to these problems with computer background, rather than with the teaching approach itself. The three summative questions in this survey were 1. Compared to traditional (i.e., non-online) courses, how much did you learn in this course? 2. How would you rate the overall quality of this course? 3. Would you recommend this course to your friends? The results show that 47% thought there was either no difference or more learning in this ALN course than in the traditional course. 38% thought the quality was good or better. And 38% would probably or definitely recommend the course to a friend. Section Analysis A comparison of the STAT 100 and the ECE 110 courses suggest that we are seeing the success of the ALN approach itself. It is unlikely that the high marks on the summative questions in the survey could be explained by a matching of student characteristics to the particular teaching style, given how disparate the audiences for the two classes are. The more mixed responses to the summative questions for CHEM 331 suggest either that some students perceive a benefit to lecture that others, including the instructor, may not acknowledge, or that there needs to be much more help for the students at the outset when an on-campus course is taught totally online.

49

JALN Volume 2, Issue 2 - September 1998

C. Cell D of the Efficiency Matrix – ECON 300 and MATH 285 ECON 300 Among the original SCALE projects, ECON 300 is the only one that from the outset was designed with the goal of achieving cost savings in instruction. After some trial experimentation with ALN, the instructor has gone from teaching a traditional course of 60 to teaching 180 students with ALN. The traditional class size is explained as follows. In the Economics department as elsewhere on campus, it is believed that making eye contact with all students in lecture is critical to being able to offer a high quality course. Countering this, overall enrollments for ECON 300 are around 700 and the department is pressed to find enough instructors to staff the course. The number 60 represents the high end of a balance between these competing needs. The department has had access to amphitheater classroom seating for teaching ECON 300 and that creates an ability to teach relatively large numbers in an intimate setting. Absent this capacity, section size in ECON 300 might have been smaller. Indeed, not all sections use the amphitheater classrooms and these other sections do tend to be smaller. The large ALN section represents an abandoning of the eye-contact model and a general deemphasis of the lecture in favor of on-line activities, with the aim that the overall quality of the course would improve. The pedagogic strategy behind the ALN approach has several components. There is a self-teaching component that utilizes Mallard for on-line quizzes. There are also written problem sets done online in FirstClass. The problem sets are assessed on a team basis. An individual team member submits a proposed solution to a particular homework problem on behalf of the team. The submission is graded rapidly, within 48 hours, and returned online to a team conference. As long as this occurs before the deadline for the problem set, another team member can resubmit the problem, taking account of the grader’s comments. The rapid turnaround time is facilitated by use of undergraduate graders. These same peer tutors also provide office hour help, both face to face and online during the evening. The large class size justifies having many of these peer tutors. An absolute grading scale has been imposed to encourage the students within a team to collaborate. More than half the credit for the course is based on the homework. This is approximately equally divided between the self-teaching work done in Mallard and the group work done in FirstClass. In fall 1997 (and spring 1998) there was another instructor who had his own ALN section of 60 students. This instructor was ‘apprenticing’with the PI and used the previously developed course materials and same on-line pedagogy. The undergraduate TAs for both sections provided common office hours, both face to face and online. The TAs grading time, however, was devoted only to the section to which the TA was assigned. In the cost part of the study on ECON 300, the entire focus is on the original PI. The apprentice instructor is ignored. In the output/quality part of the study, extensive comparisons are made between the two sections, both in student performance and in student attitude. Here we are not measuring how ALN compares to the traditional approach, but rather whether ALN pedagogy can translate well from the creator to another instructor. In Table 8, we present some summary results on the cost estimates for the ECON 300 project. In spring 1996 and again in spring 1997, the course was taught with 150 students. (The ALN version was not offered in fall 1996.) Then in fall 1997, the enrollments in the ALN section were allowed to increase to 180. Since our most precise operating cost information is for fall 1997, we have calculated the numbers in Table 8 using fall 1997 data. The numbers in the first column of Table 8 are based on a pro rata share of the fall 1997 costs rather than on the historical data. If anything, this understates the cost savings, as the peer tutor/student ratio increased from spring 1996 to fall 1997. That there are four rows to the table reflect the issue of how to allocate faculty

50

JALN Volume 2, Issue 2 - September 1998

salary to teaching. Rows one and two reflect a low fraction, one ninth of the annual nine-month salary. With a four course annual load, this leaves more than half the faculty time to be allocated for research and service and in that allocation these alternative uses of faculty time are viewed as orthogonal rather than as complementary. Rows three and four are based on a higher fraction of faculty time, one quarter of the annual salary. This makes sense if there is a strong complementarity between all uses of faculty time and hence it is appropriate to allocate it all to teaching. Another issue reflected in the table is workload. Going from teaching a traditional course of 60 students to teaching an ALN course of 180 increased the workload of the faculty member. In general, to encourage professors to teach larger ALN sections, these sections may have to be given more weight in the calculation of faculty workload. (For ECON 300, the PI believed a multiple of 1.5X would be a fair estimate.) The increased compensation for the large ALN section is reflected in rows two and four of the table. It should be noted that this represents a hypothetical case. No actual compensation of this type was paid. The ECON 300 project was funded in each year of the original grant. Funding was greatest during the first year, when there was both a course buyout for the PI as well as summer support. Total development costs are estimated at $56,224, with around 75% of that figure counting as compensation for the PI. In Table 8 we provide an estimate of the period of cost recovery under each scenario of faculty compensation. To get some reasonable bounds, we do this twice. First, we do this without discounting the future cost savings. These estimates are given in column three of the table. Then we repeat this with discounting, using the 9% as the annual interest rate. As should be obvious from the table, the period of cost recovery is sensitive to how faculty salary is imputed (and to the discount factor). Under the most optimistic scenario, the large ALN section of ECON 300 was already producing a hefty surplus by the end of fall 1997. Under the most pessimistic scenario, we still have to wait a year to recoup all the development costs. Savings per student (150 ) $83

Savings per student (180) $100

In the black by ( i = 0%) Comp = 1/9 salary After spring (ALN and non-ALN) 1998 Comp = 3/18 salary (ALN) $55 $71 After spring Comp = 1/9 salary (non-ALN) 1999 $181 $209 After fall Comp = 1/4 salary (ALN and non-ALN) 1997 Comp = 3/8 salary (ALN) $122 $154 After fall 1997 Comp = 1/4 salary (non-ALN) Table 8 – Per Student Cost Savings and Length of Recovery Period

In the black by ( i = %) After fall 1998 After fall 1999 After fall 1997 After fall 1997

We do not have any ALN versus non-ALN information for ECON 300. We do have substantial information, however, that compares the two ALN sections. We have exam results for common question on two midterms and a final. (Note that on the final there were quite a few students who took a conflict exam. The scores of those students are not included in the sample.) In this case, we are providing evidence on whether the ALN approach transfers readily from the developer of the materials to another instructor. Again we perform a comparison of means. 0n each of the midterms there were 5 common questions on the exams (4 points a piece). On the final, there were 14 common questions.

51

JALN Volume 2, Issue 2 - September 1998

Instructor Experienced with Instructor New to ALN ALN mean var N mean var N Midterm 1 11.95 22.75 170 13.10 22.37 58 Midterm 2 12.53 17.75 166 12.07 29.99 57 Final 30.40 42.79 155 30.37 31.14 49 Table 9 - Comparison of Means on Common Exam Items Administered in ALN Sections of ECON 300

zm -1.60 0.58 0.02

On Midterm 1, the inexperienced instructor’s students actually outperformed the experienced instructor’s students, though the results are not significant at the 90% level. On the second midterm and the final, there is essentially no difference in the performance of the students in the two different sections. This evidence suggests that the ALN approach did transfer, at least as far as its impact on student performance. We also have the results of surveys administered to both sections and group interviews performed by the evaluation team. In the surveys, there were no significant differences between the responses of the students in the two sections on the vast majority of the questions. The experienced instructor’s students did find Mallard somewhat easier than did the students in the other section. But that was the exception. On the following summative questions, for instance, there was no significant difference in the responses of the students. How did the use of Mallard and FirstClass in this course affect: 1. The amount of your learning? 2. The amount of your motivation to learn? 3. Your familiarity with computers? Yet in spite of this, the experienced instructor’s students rated the overall experience significantly higher. This could either be because the ALN approach didn’t transfer completely or because there are instructor-specific effects coming through in the overall rating number. The group interviews produced a good contrast between the two sections on the efficiency-ofinstruction issue. The interviewer asked: One thought is that the use of a program such as Mallard will allow classes to have more students yet maintain the same quality level. For example, one possibility in a class like this is that if the instructor has to grade less homework they will have the necessary extra time to interact with more students. What do you think? In the large section (180 enrolled) taught by the experienced instructor some responses were: • This would be okay. I think I get a lot of contact with the material because of the Mallard homework. The size of this class is okay.

52

JALN Volume 2, Issue 2 - September 1998

• There were only minor problems using ALN; whether or not a course is high quality depends more on the accessibility of TAs and the quality of their help. So, I think it would be okay. TAs just have to be accessible on-line often. In the normal size section (60 enrolled) taught by the instructor new to ALN about half of the students indicated increased enrollment would be a problem. • With higher enrollment, students will be more inhibited to ask questions in class. • Higher enrollment will limit students access to instructor/TA. The stark difference in responses here points to an issue that must be overcome when attempting to attain efficiency in instruction. Efforts must be made to counteract the fear that students have that large classes will be impersonal and ineffective as learning environments. It seems critical that students be offered a communication channel, perhaps multiple communication channels. That does not mean, however, that it must be the instructor who staffs these channels. MATH 285 The PI has been involved in teaching Calculus with Mathematica (C&M) for over a decade. There is now quite a range of math courses taught with the C&M approach on our campus. This approach fully embodies the "guide on the side" model of instruction. Students explore on their own through carefully crafted exercises in Mathematica. There is no lecture and pairs of students typically do homework in tandem. Once a week, after the exploration and right before the homework is due, there is a casual, instructor-led discussion to review what has been learned from the students' explorations and to emphasize the main points of the exercises. During the week, the instructor and undergraduate TAs are available for consultation. Over the past few years, C&M has expanded its operation to include a distance component, NetMath. This was targeted primarily at high school students in rural areas who would not have access to an AP Math course. There are now also adult learners and students from urban high schools. The distance students work the same exercises as the on-campus students. They have access to on-line help through the telephone and more recently through Netscape Conference, application sharing using Timbuktu, and asynchronous communication via e-mail. Undergraduates staff this on-line help. There are two components to the current project. First, these distance-learning help mechanisms have been placed within the on-campus MATH 285 course, turning the on-campus course into a true ALN course. This makes the course more convenient for the students (e.g., they would no longer need the lab for completing their work) while allowing it to accommodate more students. Enrollment in a C&M MATH 285 section has been artificially constrained by the size of the lab. Second, self-pacing will be introduced and students will have a longer time to complete the course. This too will be a convenience for the students, particularly the computer science students who constitute the bulk of the audience for this class. The self-pacing should boost the demand for the course substantially. The increased demand will necessitate more graders and online help. But the incremental cost is low and the cost per student should drop. The burden on the instructor is only the writing and evaluating of the exams. This can be done on a contractual basis, in an overload capacity. In the fall 1997 semester, when this approach was being tried out for the first time, the instructor got course credit for teaching the course and enrollments were only half that of the other C&M

53

JALN Volume 2, Issue 2 - September 1998

sections. As a consequence, cost per student rose. In the spring 1998 semester, however, the course was offered with higher enrollments and the instructor was assigned to the course in an overload capacity. This lowered the cost per student below that of all the other formats in which MATH 285 is taught. We did a survey for both the ALN section and a traditional section late in the fall 1997 semester. The survey had three parts. The first part had common questions. The second part had questions that were intended only for the traditional section. The third part had questions that were intended only for the ALN section. Here, we discuss only the first part and with that only where there were significant differences across sections or where the students made summative judgements. As it turns out, there were significant differences on many of these questions. Students reported greater access to materials in the ALN section. They also reported much greater interaction with their peers and consequently more learning from their peers. In particular, 75% of them reported working with their peers several times a week to complete their assignments, while half the traditional students reported no peer-to-peer interaction of this sort. The ALN students found the instructor less accessible, by a wide margin. On the summative questions, there was no significant difference reported in the amount of learning across sections. The ALN students did report their course was difficult; the non-ALN students reported less difficulty with the material. Nonetheless, the ALN students were more likely to recommend the course to a friend. Our reading of these results is that the ALN course is more intensive than the non-ALN course and that the students by and large like that intensity. But they supply much of it on their own with the help of their peers. The instructor was keenly involved in the course design but is less involved interacting with the students. Section Analysis If instructor productivity is to increase and homework assignments are not short answer, laborfor-labor substitution must be part of the explanation. Both the ECON 300 PI and the MATH 285 PI are staunch advocates of using undergraduates for administering many course functions. The fact that these students have already taken the course means there is little additional training required. Moreover, that the instructor handpicks these students implies everyone is a willing participant in the teaching. That is not always the case with graduate assistants. Some added benefits are that the instructor naturally develops a mentoring relationship with these undergraduate peer-tutors, such mentoring relationships are rare on the UIUC campus, and these students, because they have positions of responsibility, develop on-line communication skills that have value in the job market. At least for these courses and these instructors, competency of assessment by the peer tutors was a not an issue.

D. Cell A of the Efficiency Matrix – BIO 122, CHEM 102 & 102, ECON 102 BIO 122 The BIO 122 course was one of the original SCALE projects. There have been two years of prior development, mostly producing animations of microbial interactions for the Web and quiz content for self-assessment authored in CyberProf. The current project is an outgrowth of two related factors. First, the School of Life Sciences is being split into two separate schools − more or less on the lines of microbiology in one school and macrobiology in the other school. Second, there has been a growing reform movement in the teaching of biology. One precept of the reform movement is to alternate the sequence in which material is presented in the biology curriculum. Traditionally, labs have been used both to perform experiments and to teach the students how to

54

JALN Volume 2, Issue 2 - September 1998

analyze the data that accrues from them. Under the reform view, teaching data analysis should occur prior to performing experiments. The idea behind this project is that data analysis need not be taught in a lab, but rather on-line in a "virtual lab." The conversion from wet labs to virtual labs has the potential to produce substantial savings, not just in BIO 122 but elsewhere as well. Moreover, there are other large introductory courses in the sciences that have no lab component whatsoever, because it would be too costly to provide the wet-lab experience.. Successful use of virtual labs in a high enrollment course in one discipline could very well pave the way toward use of virtual labs in introductory courses in other disciplines that could benefit from this type of enrichment. BIO 122 is a very high enrollment course with about 1500 students a year; the vast majority of them being pre-med students. The PI co-teaches the course with another professor. She lectures for the first half of the course and he lectures for the second half. There are weekly labs. The virtual labs are quite intensive to author. For the implementation in fall 1997, there were about an equal number of virtual labs and wet labs over the part of the course that the PI taught. More of these labs virtual labs are being designed at present. The potential for productivity increase is quite large. The labs are extremely labor-intensive. The introduction of the virtual labs economizes on the number of lab assistants. There is also an obvious economy in lab materials. By design, some of the economy in personnel is captured by the graduate assistants. During the weeks with the virtual labs, they get a break from their assistantship work and can spend that time on their thesis or pre-thesis research. This suggests that one long-term measure of the presence of the efficiency gains is the time to dissertation completion. Since graduate students in microbiology do a lot of teaching, if the virtual lab productivity increases are real, the time to dissertation completion in microbiology should go down. The survey information we have for BIO 122 concerns the course in general, rather than the labs in particular. The three summative questions in the survey were 1. How easy did you find using the Web for purposes of this course? 2. How would you rate your overall experience using the Web in this course? 3. Would you take another course using the Web? Over 90% of the students reported that using the Web for the course was somewhat easy or easy to use. Over 85% of the students reported that their experience with the course was good or better. And over 85% of the students reported that they either probably or definitely would take another course using the Web. The ALN component of the course is quite popular, as the scores indicate. The results here look similar to the results from STAT 100 and ECE 110.

CHEM 101 & 102 This project entails the introductory general chemistry sequence, CHEM 101 and 102. The enrollment for these courses is very large, over 2000 students a year. CyberProf has been adopted as a means for doing on-line homework and as a communication device. While lectures remain the same, weekly quizzes are no longer administered during recitation sections. Instead, the full period is devoted to answering student questions about the homework and to working other problems. Since all the homework is now automatically graded, the TA burden has been

55

JALN Volume 2, Issue 2 - September 1998

substantially reduced. Thus, TAs are now assigned three recitation sections (historically the load has been two sections). More TA help was offered out of the recitation section, both in face-toface hours and on-line. In spite of this increased out-of-class help, there was a substantial saving in personnel costs from having some of the TAs take on three sections instead of the usual load of two sections. In fall 1997 for the first time, Chemistry ran CyberProf off their own servers. They had lots of problems with server configuration and didn’t fully resolve those problems till after mid semester. In essence, they had to use a makeshift system for the first half of the semester. The students’ responses to the summative questions should be interpreted in that light. The three summative questions in this survey are: 1. How easy did you find using CyberProf for purpose of this course? 2. How would you rate your overall experience using CyberProf for this course? 3. Would you take another course using CyberProf? Seventy percent of the respondents reported CyberProf somewhat easy or easy to use. 56% of the students rated their experience good or better. And 56% of the students probably or definitely would take another course using CyberProf. Given the server problems, these marks are certainly encouraging. ECON 102 ECON 102 is one of the highest enrollment classes on campus. Enrollments per year are roughly 3500. The PI has taught ECON 102 lectures with as many as 1400 students. However, due to recent campus restrictions on maximum class size, the enrollment in an ECON 102 lecture has been capped at 750 students. The PI's class meets twice a week ensemble and once a week in recitation sections led by TAs. Last spring, he used Mallard for on-line quizzes and a Web conferencing package written by his former undergraduate programmer. Rather than author the Mallard questions himself, the PI secured a database of questions from the publisher of the textbook he uses and had his assistant put the material in Mallard. The PI authored on-line material (presentation with voice annotation) that would be used by the students in lieu of attending the recitation section. Ultimately, the voice component was dropped in this round because of difficulties in the logistics of implementation. The TAs who had taught these recitation sections were able to offer more help out of class, both face to face and on-line. The PI estimates that he can double the student-to-TA ratio this way and thus require fewer TAs in total. This is consistent with the Economics Department goal to downsize the Ph.D. program. As an added benefit, the reduction in the number of TAs will eliminate some of the variation in individual TA presentations across sections. In spring 1998, the PI used ALN in a few experimental sections, entailing about 200 students. If this proves successful, he plans to use it in all sections of ECON 102 the next time he is slated to teach it, spring 1999. ECON 102 is also one of the original SCALE projects. In each of the three times it has been offered, there have been ALN and non-ALN sections, with common exams. Kristine Burnaska is doing a separate, intensive study, [27], that looks at the various determinants of student performance in ECON 102 and, in particular, compares the performance of students in the ALN

56

JALN Volume 2, Issue 2 - September 1998

sections to those in the non-ALN sections. Because the ALN version of ECON 102 is taught only in the spring, we do not have survey or interview results as part of the this SCALE evaluation. Section Analysis Both BIO 122 and the introductory Chemistry sequence rely on CyberProf a great deal. Yet the students’ responses to the summative questions on the survey for BIO 122 were extremely positive, much more so than in the introductory chemistry courses. This lends credence to the idea that some of the negatives in the Chemistry survey were due to server problems rather than the teaching itself. In all the courses discussed in this section, teaching assistant productivity is increasing via ALN, either because grading has become automated or because presentation has been moved online. That Microbiology allows much of the productivity increase to be captured by the students while both Chemistry and Economics are using the productivity to reduce costs from their end raises an interesting further point. Who captures the gain from the productivity increase – the instructor, the department where the course is housed, or the students in the course, seems to be a matter of choice rather than to be intrinsic to the teaching approach. As it becomes increasingly believed that such productivity gains are feasible, more attention should be paid to the issue of capturing the gains.

VI. GENERAL FINDINGS Four of the courses: SPAN 210, ECON 300, STAT 100, and ECE 110 used Mallard for at least some of the on-line assignments. Two others, the introductory chemistry sequence and BIO 122 used CyberProf in a similar manner. And CHEM 331 used WebCT, a commercially available product that also allows for automated grading of quizzes. There seem to be several benefits from use of this type of software in a large ALN course: 1. Students liked being able to take the quizzes when it was convenient for them rather than at a pre-specified time. 2. Students liked being able to retake quizzes immediately when they got something wrong. They could learn from their own mistakes. Since the course credit they received was based on successful completion of the quizzes only, not the number of tries it took to reach success, this type of learning was encouraged. 3. Students liked that the quizzes had deadlines. The deadlines imposed a certain discipline on them, forcing them to keep up with the course material. There were also some complaints about how the software itself and some about how instructors authored material for it: 1. Instructors would often put in hints to be accessed if students had difficulty with the questions. In some cases students reported the hints were vague and not helpful. In other cases no hints were available. 2. Students reported that they could get the right answer to many questions by guessing rather than by reasoning through the problem. 3. In ECE 110 and in the introductory chemistry courses, there were numerous complaints about delays in getting back a response from the server after submitting an answer. These complaints didn’t come up in the other courses.

57

JALN Volume 2, Issue 2 - September 1998

The first complaint might be dismissed if students were in fact seeking help from classmates, TAs, or the instructor when they were stuck on a question, especially if the instructor had deliberately designed the hints to encourage such dialog. It seems clear, however, that at least some students have an expectation that they should be able to do these on-line quizzes without seeking outside help. The second complaint points to a broader question. Are these on-line quizzes educative or makework? If the quizzes are make work, is this a fundamental problem with this type of courseware? Or is the problem with the way the material has been authored – not being sufficiently attentive to how the students respond when taking the quizzes? If the latter, we are probably understating our development costs in that there needs to be even further rewriting of the on-line material. To the extent that the delays in response could be resolved if there were more servers, the third complaint suggests that we may again be underestimating our fixed costs of ALN instruction. If these delays are inherent in the courseware design, when used in large classes where the question type is such as to impose a heavy computation burden on the server, there is another issue. Could other courseware be written with similar functionality and yet not produce these delays? All the courses used computer conferencing to some degree. The students’ like or dislike of conferencing seemed to relate directly to how much feedback they got from the instructor and/or the TAs. In some cases there was sufficient feedback that class size didn’t seem an issue. In other cases feedback was scarce and there were student complaints about the course being too impersonal. Interestingly, it seems that human feedback per se is the concern. Students did not seem to insist that the feedback come from the professor. Here are some of the faculty responses to the question: What is your philosophy on limiting or not limiting the number of times students can take the ALN quizzes/homeworks? These responses seemingly parallel those of the students in the view of pedagogic benefit but raise other concerns about their limitations, cheating and grade inflation. • There is the question of whether or not this is inflating the grades. The other question is how do you motivate people. These quizzes in a sense force motivation, force the students to do the work. Unless they found someone who would do it for them. But then they had problems taking the test. I’m a little worried that we are not doing enough to discourage cheating. The scores are not on a curve so one good buddy has no incentive not to help his other good buddy or to take money from him to get a substantial part of the grade in the course for his friend. • I let them take them over and over. I think the more subjected they are to the material the better. I’ve also noticed that students are contacting me more about problems that they miss now that they get immediate feedback. Before when students would submit homework and get it back a week later, they didn’t really care any more what the right answer was. Now I get e-mails from students who say they are trying the problem right now and getting the wrong answer, they ask what they are doing wrong. I think this is good because they are thinking about the questions more and why they are getting them wrong. This is a direct result of the immediate feedback from using Mallard as opposed to the delayed feedback of having traditional homework.

58

JALN Volume 2, Issue 2 - September 1998

Also, they are getting subjected to the material more. It doesn’t matter if they are trying it over and over again. The point is that they are looking at it again and again. Doing the work again and again. • My overwhelming motivation to allow them to take them over and over again is that I am subjecting them to the material. If they get it wrong and they look at it again, then they have had more contact than if they simply turned it in and found out later they had missed the question. Grade inflation is not an impossibility. But I think the more correct answer is that they have been subjected more to the material. On the subject of: is it less fair that my students receive a higher grade partially due to the fact they have a limited amount of points that are almost guaranteed than students in other non-ALN sections - I think academic freedom, faculty members ability to decide how to structure their course, is a common standard. My students are subjected more to the material. It’s not that it is unfair.

VII. CONCLUSION We have tried to present the evidence in a balanced light. Certainly, we wish there were more comparative evidence to present, whether it entailed student performance or student attitudes, as we feel more confident weighing in on the “holding quality constant” issue with that type of evidence. The performance evidence that we do have suggests the students in the large ALN classes are not harmed and may even benefit from the ALN approach, relative to their peers in a traditionally taught and traditionally sized class. The comparative attitudinal evidence strongly suggests that what students do in an ALN course is different from what they do in a traditional course. It is by no means a trivial matter to compare course quality when the student activities change. In particular, in both the Spanish Grammar course and the Differential Equations course, the ALN sections entailed a lot more self-study and a lot less interaction with the instructor. Is that good or bad? Does that depend on the students taking the course? The evidence suggests that it is good for at least some students. Where we do not have comparative evidence, the responses to the summative questions suggest that most students like ALN when taught in a large class setting. Furthermore, the above conclusions seem to hold for courses in each of the cells of the Efficiency Matrix. This suggests that there are many possible courses that could be targets of like-minded projects and that there are an array of alternative approaches that can be utilized to achieve the type of results we have found with the SCALE projects. We are buoyed by these results. While not providing absolute confirmation, it supports our view that when a sensible pedagogic approach is embraced that affords the students with avenues to communicate about their learning, ALN can produce real efficiency gains in courses without sacrificing the quality of instruction.

59

JALN Volume 2, Issue 2 - September 1998

REFERENCES 1. 2. 3. 4. 5. 6. 7. 8.

9.

10.

11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21.

22. 23. 24. 25. 26.

27.

(http://cyber.ccsr.uiuc.edu/CyberProf/) (http://www.cen.uiuc.edu/Mallard/) Ory, J. C., Bullock, C.D., and Burnaska, K.K., “SCALE Fall 95 Evaluation Results,” http://w3.scale.uiuc.edu/scale/evaluations/fall95/index.html, 1996. Ory, J. C., Bullock, C.D., and Burnaska, K.K., “SCALE Spring 96 Evaluation Results,” http://w3.scale.uiuc.edu/scale/evaluations/spring96/index.html, 1996. Ory, J. C., Bullock, C.D., and Burnaska, K.K., “SCALE Fall 96 Evaluation Results,” http://w3.scale.uiuc.edu/scale/evaluations/fall96/index.html, 1997. Ory, J. C., Bullock, C.D., and Burnaska, K.K., “SCALE Spring 97 Evaluation Results,” http://w3.scale.uiuc.edu/scale/evaluations/spring97/index.html, 1997. Wilson, J.M.,"The CUPLE Physics Studio," The Physics Teacher, 1994 Kashy, E., Thoennessen, M., Tsai, Y., Davis, N. E., and Wolfe, S. L., “Using Networked Tools to Enhance Student Success Rates in Large Classes,” Proceedings of the Frontiers in Education, http://fairway.ecn.purdue.edu/~fie/fie97/papers/1046.pdf, 1997. Kashy, E., Thoennessen, M., Tsai, Y., Davis, N. E., and Wolfe, S. L., “Application of Technology and Asynchronous Learning Networks in Large Lecture Classes,” 31st Hawaii International Conference on System Sciences, Volume I, Collaboration Systems and Technology Track, page 321, edited by Jay F. Nunamaker, Jr., 1997. McClure, P. A., “‘Growing’Our Academic Productivity,” in Reengineering Teaching and Learning in Higher Education: Sheltered Groves, Camelot, Windmills, and Malls, edited by Robert C. Heterick, Jr., http://cause-www.colorado.edu/information-resources/ir-library/abstracts/pub3010.html, 1993. Noble, D. F., “Digital Diploma Mills: The Automation of Higher Education,” First Monday, http://www.firstmonday.dk/issues/issue3_1/noble/, 1998. Green, K., “The Campus Computing Project,” http://ericir.syr.edu/Projects/Campus_computing/, 1997. Koob, R., “New Funding Paradigms: The Need for New Incentives,” NLII Viewpoint, Vol. 1, Issue 1, http://www.educom.edu/program/nlii/articles/koob.html, 1996. Twigg, C., “Academic Productivity: The Case for Instructional Software,” http://www.educom.edu/web/pubs/pubHomeFrame.html, 1996. Twigg, C., “The Need for a National Learning Infrastructure,” http://www.educom.edu/program/nlii/keydocs/monograph.html, 1994. Johnstone, D.B., “Learning Productivity: A New Imperative for American Higher Education,” NLII Viewpoint, Vol. 1, Issue 1, http://www.educom.edu/program/nlii/articles/johnstone.html, 1996. Massy, W. F., and Zemsky, R., “Using Information Technology to Enhance Academic Productivity,” http://www.educom.edu/program/nlii/keydocs/massy.html, 1995. http://www.tltgroup.org/ Arvan, L. “The Economics of ALN: Some Issues,” Journal of Asynchronous Learning Networks, Volume 1, Number 1, http://www.aln.org/alnweb/journal/issue1/arvan.htm, 1997. http://www.cba.uiuc.edu/~l-arvan/SCALEevalf97/ Arvan, L., “Bottom Up or Top Down, Using ALN to Attain Efficiencies in Instruction,” Proceedings of the Third International Conference on Asynchronous Learning Networks, http://www.aln.org/conf97/slide/arvan/arvan/index.htm, 1997. Berube, M. “Why Inefficiency is Good for Universities,” Chronicle of Higher Education, Vol. 44, Number 29, 1998. Young, J. R., “Technology May Not Be All That Great, Say Professors at 'Second Look' Meeting,” Chronicle of Higher Education, Vol. 44, Number 34, 1998. http://www.cba.uiuc.edu/~l-arvan/SCALEevalf97/Part1-JALN-Efficiency.doc. http://www.pathfinder.com/money/colleges98/collegecat98.html http://www.csun.edu/sociology/virexp.htm. Burnaska, K., “How the Implementation of Technology into Higher Education Affects Students' Motivation, Resource Management Skills, Course Achievement Scores and Class Preparation,” Unpublished Dissertation, 1998.

60