Impacts of college-level courses via Asynchronous Learning Networks: Some Preliminary Results

New Jersey Institute of Technology has been delivering college courses via an Asynchronous Learning Network (ALN) system called the Virtual Classroom for a decade, using various media mixes. Currently, two complete undergraduate degree programs are available via a mix of video plus Virtual Classroom, the B.A. in Information Systems and the B.S. in Computer Science. This paper presents preliminary findings about impacts on students, and touches on some issues and potential impacts for faculty, individual universities, and the structure of higher education. Overall ratings of courses by students who complete ALN based courses are equal or superior to those for traditional courses. Dropout or Incomplete outcomes are somewhat more prevalent, while grade distributions for those who complete tend to be similar to those for traditional courses. For both students and faculty, more startup time devoted to solving the "logistics" of ALN delivery seems to be required at the beginning of courses. ALN delivery is not just a "different" way of doing the same thing, however; it is likely to change the nature and structure of higher education.


INTRODUCTION
The sophistication and flexibility of software structures for supporting distance education vary widely, from posting materials on the World Wide Web or simple electronic mail systems to conferencing systems that have been specially enhanced to support classroom-like experiences, particularly group discussions and joint projects. This paper focuses on what we have learned about teaching techniques that are effective in the networked environment as a result of a decade of experiences at New Jersey Institute of Technology (NJIT) using versions of a computer conferencing system enhanced with features to support collaborative learning, called the Virtual Classroom TM . However, most of the themes emphasized are echoed in reports on similar projects [1,2,3]. The preliminary results reported here are based on the data for the first two years of a three year project, as of the fall of 1995; analysis of the full three-year project data was still underway as of the date of this report.
Generically, the Virtual Classroom (VC) is a teaching and learning environment located within a computer-mediated communication system. Rather than being built of steel and concrete, it consists of a set of group communication and work "spaces" and facilities that are constructed in software. Thus it is a "virtual" facility for interaction among the members of a class, rather than a physical space. Specifically, the Virtual Classroom TM is NJIT's trademarked name for versions of its Electronic Information Exchange System (EIES2) with special software structures designed to support collaborative learning, including structures to order the transcript of discussions, to force active participation, to allocate unique assignment topics, and an electronic gradebook [4,5,6]. Participation is generally asynchronous or "anytime/anywhere": that is, the participants may connect at any time around the clock, and from any location in the world accessible by the Internet or a reliable telephone system, rather than having to be online at the same time. The system stores the entries in a permanent, ordered transcript which keeps the equivalent of "bookmarks" to separate anything that is "new" for each individual from items that have already been seen.
The fact that the educational process is asynchronous means students may engage in more reflective thinking before having to answer or discuss issues, as compared to a synchronous or same-time interaction, and it also means that students can participate at their own convenience, and thus better fit the demands of a college degree program into busy lives.
It should be noted that a VC environment can be used successfully in many different media mixes: • Face to Face (FTF) plus VC: This can vary from adding system use to enrich on-campus courses conducted by traditional means, to distance courses where system use is supplemented by one or two FTF meetings. • VC as the primary means of delivery: This may also use print media in the form of textbooks or course notes. • Multi-media: This offers mixes including VC plus video, or VC plus audio or audiographic media or CD ROM or other special PC-based software. In this format, lecture type material is delivered via a mode that includes audio and video channels, while the class discussions and collaborative student projects take place in a (primarily) textoriented computer conferencing environment.
Some ALN programs were designed to serve on-campus students, and others to serve "distance" students. At NJIT, there is a mix. If students who take most of their courses on campus are permitted to, they will choose to take some portion of their courses via an asynchronous mode such as VC, either because of greater convenience, or because they prefer the mode of interaction.
In the fall of 1995, for instance, about half of the students enrolled in the VC + video undergraduate courses were taking the majority of their degree courses on campus, and half were "distance" students who took the majority of their courses in distance modes. There is also a synergy; if there are two sections, one FTF + VC and the other video + VC, they can be combined in the same online conferences and activities, which seems to motivate students in both sections.

II. COLLABORATIVE LEARNING
Simply making an ALN available and telling students that they can use it to ask questions about the readings or to discuss aspects of the course at any time does not ensure its use. If it is not a "required" and graded, integral part of the course, the majority of the students will never use it at all; and those who start to use it, will generally decide that "nothing is going on there" and stop using it. ALN's can best enrich distance delivery by serving as the means to create the feeling of a true "class" or group of people who are learning together, and by structuring and supporting a carefully planned series of collaborative learning activities which constitute the assignments for the course.
Collaborative learning is defined as a learning process that emphasizes group or cooperative efforts among faculty and students. It stresses active participation and interaction on the part of both students and instructors. Knowledge is viewed as a social construct, and therefore the educational process is facilitated by social interaction in an environment that facilitates peer interaction, evaluation and cooperation [7,8,9,10]. A variety of social-psychological mechanisms make collaborative learning effective, including "self-explanation," "internalization," and "appropriation" [11]: • Self explanation occurs when a more knowledgeable peer explains a topic to another. One receives an explanation and the other benefits by articulating and integrating various pieces of knowledge. • Internalization is the process of learning by verbalizing in a conversation.
• Appropriation occurs as one learns from watching and working with a more skilled partner; apprenticeship is a form of appropriation.
The Virtual Classroom is an environment that was designed to facilitate collaborative learning -among students, between students and instructors, among teachers, and between a class and wider academic and nonacademic communities. It also supports independent learning and generative, active learning techniques that are self-paced by each participant. For distance education students, the increased ability to be in constant communication with other learners is obvious. But even for campus-based courses, the technology provides a means for a rich, collaborative learning environment that exceeds the traditional classroom in its ability to "connect" students and course materials on a round-the-clock basis.
It is important to establish collaborative learning through substantive contributions by students to the class discussion from the very beginning of a course. However, with distance students, there are often problems in obtaining all of the books, videotapes, or other materials by the first week of the course. Therefore, the first week's assignment, in particular, should draw upon the students' own experiences, materials that are posted online, and general knowledge, rather than requiring the reading and synthesis of specific assigned materials.
An example of a collaborative learning strategy applied in the VC that is included in most courses is the "seminar" type of interchange in which the students become the teachers. Individuals or small groups of students are responsible for making a selection of a topic; reading material not assigned to the rest of the class; preparing a written summary for the class of the most important ideas in the material; and leading a discussion on the topic or material for which they are responsible.
Seminar-style presentations and discussions are thus an example of a collaborative learning activity that is often difficult in the Traditional Classroom (TC), but which tends to work very well in the VC environment, even with fairly large classes of undergraduates. Another example is to assign students to identify key concepts or skills in each module of the course, make up a question suitable for an exam to test mastery of this material, and answer each other's questions. Exams then actually include selections from the student-generated questions.

III. EXPERIENCES WITH ALN's AT NJIT
Our first extensive experimentation with the VC began in 1986 and involved comparison to traditional FTF classrooms for a large number of courses over a period of two years in many different disciplines which used this medium of communication, including undergraduate courses in sociology, communication, English composition, management, computer science, mathematics, and statistics. For five of the courses, there was a "matched" section of the same course offered by the same instructor in a traditional classroom and using the VC (as the sole means of delivery, or in combination with a reduced number of FTF meetings). For other courses, there was no "match," and the comparison was subjectively made by the students and instructors to previous, traditional courses.
For the first study and the projects which have followed, we used a "multi-method" approach to evaluation. This includes pre and post-course questionnaires completed by students, direct observation of online activities, interviews with selected students, comparison of test or course grades or other "objective" measures of performance, and regular reports by faculty, which follow a common outline. The results presented here will rely primarily on the post-course questionnaires measuring subjective perceptions of learning. These were administered to all the VC + video sections, plus sections of the same course taught by the same teacher(s) in a different mode; either FTF plus VC, completely FTF, or video only.

A. Virtual Classroom + Video: The Degree Programs
This project was designed to develop, offer, and assess the effectiveness of an undergraduate major in Information Systems delivered via VC plus video, to attain five objectives: • Faster progress towards the undergraduate degree, by facilitating self-paced learning and solving major educational logistics problems; • Improved quality of learning through the increased collaborative learning and facultystudent interaction facilitated by computer conferencing; • Increased access to educational opportunities for working adults or those trying to reenter the work force, particularly women; • Formative and summative evaluation of the effectiveness of this media mix used in different ways, for attaining the above objectives; • Dissemination of the successful techniques and materials to other institutions Some of the courses incorporating video for "lectures" use standard public television courses, such as "Discovering Psychology," produced by PBS. Most video segments are filmed by NJIT in its "candid classroom" and then distributed to remote students on videotape, or via broadcast on a cable channel or satellite. In all video variations, the VC is used for all assignments and additional discussions.
Altogether 26 courses were converted to VC + video format, and some were offered six to seven times thus far. Enrollments have shown a fairly steady upwards trend, aided by a substantial investment of personnel time and money in recruiting efforts (See Figure 1). The CIS department added the additional courses necessary to complete the more technical B.S. in Computer Science, which also increased total student enrollments.

B. Access: Overcoming Logistical Problems
The use of any medium for teaching interacts with characteristics of the students, the institutional context, and the subject matter. Thus, results for NJIT may not be generalizable to different contexts. The majority of NJIT's students are "first generation" college students, who must work while attending school and who commute rather than live on campus. They are faced with overcrowded classes with sections that fill up early and then become closed (due to budget cuts that reduced staff and raised class sizes). They are also faced with degree programs that because of their technological nature, have extensive sequences of courses that build on one another and must be taken in order. If a student is closed out of a course one semester or must withdraw, he or she might lose a whole semester waiting for another opportunity to take this course which is a pre-requisite for subsequent courses.
In addition, many of the courses are offered more slowly than would be necessary for the better prepared or more motivated students. The "mass lectures" that have emerged as a result of budget cuts tend to slow down to the pace of the "lowest common denominator" and frustrate the better students. Both video and VC allow "fast forward" and "replay" self-pacing.
By using a combination of video (tape and/or broadcast) plus VC to deliver courses, students may more easily fit them into busy lives as employees and family members as well as students. Since the vast majority of NJIT students work 20 hours a week or more, and many work 40 hours a week or more (in order to pay their own educational expenses and/or to help support their families) it is very difficult for them to fit courses they need to graduate into their schedules. By offering distance education sections, scheduling difficulties can be alleviated. Students can graduate in fewer calendar years because they can more easily fit in all the courses they need.
In addition to the asynchronous nature of the VC helping to overcome course enrollment and completion difficulties, we are trying several innovations that may better facilitate self-pacing and improved rates of progress towards the degree, while maintaining or enhancing quality. These innovations include double courses offered during the same semester, independent study opportunities during the summer, and "late start" courses.
Many of the courses in the CIS majors form a series of prerequisites and must be taken in order. For example, the calculus perquisites must be taken before the first introductory course(113), which must be taken before 114 (the second part of Introduction to Computer Science); which serves as a pre-requisite to more advanced courses. Students can lose many semesters trying to get through this rigid sequence of courses, one at a time, meanwhile being closed out of some of the courses in the sequence some semesters.
We have redesigned a version of the introductory courses in this series of pre-requisites as an integrated "double course" in the Spring semester each year, for those students who were making up pre-requisites in the fall and were unable to start their major. It covers 113 at twice the normal pace, devoting twice the normal number of hours to this task. By the middle of the semester, some of these "fast track" students may decide that they need a full semester after all; they drop into a slower paced group online. Those who can learn at the accelerated pace complete the first course in the first half of the semester, and then go on to take the follow-on course during the second half of the semester. The participating faculty members coordinate their efforts to provide a "seamless transition" between the two courses. We have found that many students are able to learn at the increased pace because of the intensive interaction and the support available online. Accelerated (7 week) versions of many of the ALN courses are also offered during the summer, for students who need to "make up" a missing course. Every time students successfully complete an accelerated course, they can conceivably cut an entire semester off of the number of semesters needed to finish their degree program.

IV. HYPOTHESES AND FINDINGS ON STUDENT IMPACTS: INTERIM RESULTS
The most basic hypotheses in our initial 1986-1988 study about impacts of the Virtual Classroom on students are listed below; all were supported (see Hiltz, 1994, for complete data and analysis). They are also guiding some of the data analysis for the current Video + VC project; brief descriptions of the measures of variables used and some interim results will be presented below.
H1: Mastery of course material in the VC will be equal or superior to that in the TC.
H2: VC students will report higher subjective satisfaction with the VC than the TC on a number of dimensions, including improved overall quality, whereby the student assesses the experience as being "better" than the TC in some way, involving learning more on the whole.
H3: Those students who experience "group learning" in the VC are most likely to judge the outcomes of online courses to be superior to the outcomes of traditional courses.

A. Problems reported by Students
On the post-course questionnaire, students in all modes were asked a series of questions about the course, the instructor, and their experiences. The relative distribution of responses allows us to compare different delivery modes indirectly. It should be noted that self-report data from questionnaires suffers from some validity issues. They can be biased (either favorably or unfavorably) by how much the student "likes" the professor, and by perceptions of risk that the answers may somehow affect one's grade. We attempted to deal with these issues in two ways. The questionnaires were mailed to all distance students by the assistant project director, and they were invited return them to her; confidentiality was guaranteed. If the questionnaires were handed out in class, a student volunteer was requested to place them all in an envelope and deliver them directly to the assistant project director.
The FTF and video only mode were combined into "no VC" for these analyses, because of the low number of video only responses. Most of the items showed no significant differences among modes; the differences in course and instructor ratings are evidently more affected by factors other than mode.
More students using the VC experienced more logistical problems than should occur in an "ideal" environment. These problems may seem "trivial" or local NJIT problems only; our observation after a decade of experience with ALN is that the specific nature of the technical and logistical problems changes, but that they are persistent.
For example, about 13% of the video + VC students indicated that access to a PC was a serious problem for them (using response 1 or 2 on a 7 point scale, where 1 was "serious problem" and 7 was "not a problem"; see Figure 2). These students should not have signed up for a course which required PC access. All registration materials, for instance, note that access to a "PC and modem are required." It seems that students are oblivious to information they do not want to see, or do not bother to look at the printed course descriptions before self-registering into any open section.

Figure 2: Problems with Access to PC
The most severe access problem is that 40% to 50% of students using VC at NJIT experienced serious problems with busy signals when they tried to dial in (Figure 3). The university does not seem to be able to provide enough modem access to meet the ever expanding demand; there were 48 dial-in ports to campus computing systems the first year of the project, and another 48 ports were added the second year, but within a week, they were saturated, too. One solution would seem to be for students to subscribe to an Internet Service Provider (ISP) for approximately $20 a month, and come in through the Internet, rather than direct dial. However, many students are resistant to spending this additional money, and even if they do, they encounter busy signals from their ISP. Our solution has been to create a "reserved" modem bank that allows only students enrolled in VC courses to use it; and to tell all students that they need to learn how to direct dial and quickly upload and download materials, as an alternative to ISP access.

Figure 3: Problems with Busy Lines or Ports
In terms of disadvantages of learning networks as compared to traditional courses, there are two items on the post-course questionnaire that are worthy of note. Whereas the majority of students meeting FTF courses develop new friendships in class, only a third of the VC + video students report doing so (Figure 4).

Figure 4: Developed New Friendships in Class
It is good to see that a substantial proportion can make new friends via a written mode of communication, but it is obvious that close personal relationships are more difficult. In addition, responding to a different format of question in which VC students were asked to compare their experiences to those they had in traditional courses, VC students report that they are more likely to stop "attending class" when they are busy with other things ( Figure 5). Since the class does not meet at any particular time, it is easy to "postpone" it, and this procrastination all too easily turns into falling seriously behind.

B. Subjective Assessments: Relative Frequency Distributions for Different Modes
In looking at how interesting the course content was, the differences among delivery modes are not significant (p=.21), but those in the experimental Video + VC mode were most likely to strongly agree that the course content was interesting ( Figure 6).

Figure 6: Course Content Interesting By Mode
Students in a mode using VC were slightly more likely to strongly agree that they "participated actively in the class discussion" (p=.06; Figure 7).

Figure 7: Participated Actively in Class Discussion
This is a welcome result, since one of the objectives of the VC is to encourage/require active participation. On the overall course rating, the VC + video students are slightly more likely to rate it as either excellent or very good (p= .10; Figure 8).

Figure 8: Overall Course Rating
A few items yielded statistically significant differences. For example, for the question on whether the course was a "waste of time," the Video + VC mode students were more likely to strongly disagree ( p= .001; Figure 9).

Figure 9: Course a Waste of Time, By Condition
A final section of the post-course questionnaire was administered only to those sections using the VC. The instructions to students were to compare their experiences in the course they had just completed, to their experiences in traditional, FTF college courses. The results of these items are presented in graphical form to show the distributions visually. First we will examine items related to access. The vast majority (71%) feel that ALN's provide better access to their professor ( Figure  10).

Figure 10: Better Access to Professor
This is a very important outcome of ALN and provides one of the major ways in which it can improve overall access to learning opportunities. Of course, it is dependent upon the instructor's making himself or herself available online at least once a day. Despite the reported problems of some in reaching the computer facilities, over 2/3 (69%; see Figure 11) feel that the VC is, overall, "more convenient."

Figure 11: Online Course is More Convenient
Only 15% did not "feel more involved in taking an active part" in the course (Figure 12). The results also indicate that "collaborative learning" did take place and did tend to have its intended motivational and learning consequences. For example, 55% felt more motivated to work hard on their assignments because other students would be reading them (Figure 13), and only 9% disagreed that reading the assignments of other students was useful (Figure 14). Because of the convenient daily access to the class interaction, and the need and motivation to keep up with what their instructor and classmates are contributing, students (as well as faculty) tend to work harder in ALN based courses. For our interim results, the item on total amount of work is phrased as "I didn't have to work as hard for the online class." The majority (67%) disagree with this, while only 13% agree (Figure 15).

Figure 15: Don't Work as Hard For Online Class
One overall measure of student satisfaction is whether students would choose to take another course using an ALN. Only 20% of the students agreed with a statement that they would not do so; 58% indicated that they would ( Figure 16).

Figure 16: I Would Not Take Another Online Class
This is a measure which undoubtedly would improve if logistical problems of reaching the network were lessened. In terms of the total amount that they learned, the modal response is that students are not sure whether they learned more or not than they would have in a traditional course; but 40% felt that they had learned more, where as 21% felt that they had not (Figure 17). The final overall measures of subjective perceptions of the quality of the educational experience are phrased in terms of one to seven Likert-type scale items in which the polar points are 1="definitely yes" and 7= "definitely not." The modal response on whether the VC increased the efficiency of educational delivery is that students are not sure; but 47% feel that it does, as compared to 23% who feel that it does not ( Figure 18).

Figure 18: VC Increased Efficiency of Education
In terms of whether the VC "increased the quality of education," the students are more sure; 58% say yes, while 20% say no ( Figure 19).

C. Outcomes: Grades and Completion Rates
The results for the first two years support the hypothesis of equal or better performance in ALN as compared to traditional modes of course delivery. Two-year preliminary results for all courses combined showed no significant differences, though the withdrawal rate is generally higher than for traditional FTF sections. However, one really needs to look within each course in order to reach more definitive conclusions on this matter, since otherwise mode of delivery is confounded by differences in the level of difficulty and grading tendencies of different courses and instructors. Figure 20 shows the results for the first introductory course (CIS113) for the first two years of the project. The various sections of this course were all taught by the husband and wife team of Fadi and Maura Deek, who follow exactly the same syllabus and grading policies. As for the combined results, there are results statistically significant at the .05 level. However, there is a strong suggestion that video + VC is much more effective than video alone. In fact, we do not intend to offer any more video-only sections, which may be educationally sound but will prevent us getting a large enough "N" to reach statistical significance on these conclusions! There is also a suggestion that a larger proportion of students in this course do well (earning an A or B) in video + VC sections than in traditional sections (50% vs. 31%). The data thus do support the conclusion that ALN is "as good or better" than other modes of course delivery in terms of enabling students to master course materials.

V. IMPACTS ON FACULTY
The following are a few preliminary impressions on impacts on faculty, gathered from completed course reports and discussions in our online "faculty lounge" where the issue of reporting and discussing perceived impacts has been raised. A full set of course reports by individual faculty members may be accessed on the author's Web home page.
The first consideration is impact on faculty workload: Preparing for videotaping is a LOT of work. One spends perhaps 10 hours to prepare the materials for two hours of taping, plus review or rehearsal ahead of time.
Next one must consider the preparation of the ALN online materials, and the moderating of class conferences and daily individual interactions with students through email. The first time a faculty member prepares the materials for weekly modules and moderates class conferences, this part of the workload alone is also considerably more work than just delivering lectures FTF.
Once the course has had its video and basic VC delivery materials developed and the faculty member has gained some experience, then if the class size is approximately 25 or less, the actual amount of work to conduct it is about the same as for a traditional class. However, the amount of work for the moderator or facilitator of a course is directly proportional to the number of students, since there is no limit to the amount of time each student can ask questions and make contributions that need response or grading. For more than about 30 students, class conferences have to be divided to be manageable, thus approximately doubling the faculty workload.
There are also qualitative differences in the way a faculty member functions. For example, at the beginning of the semester, the instructor feels a bit like a "cybernetic cowboy," trying to herd all the students into the class corral. Willy Rossak, in the second week of a VC + Video course, says, "Well, at the moment I feel I am more busy to try to round up my students instead of discussing the course topic. It seems the instructor becomes system administrator for the conference, headhunter, and only then can be a lecturer." Grading becomes a logistical problem Julian Scher says, One unexpected impact was for the mentor to organize himself/herself for the grading process of homework/projects, much different than in the FTF mode. In the traditional class, I walk out of a class with a "batch" of projects/homework to grade, with an obvious batch grading. In video + VC, I gave students the option of submitting projects through email, regular mail or in person. The completed homework arrived at different times and days. And I would feel guilty grading on the "first arrived-first graded" basis due to the desire to maintain consistency.
Impacts of ALN on the nature and size of the faculty teaching load need to be considered by policy making bodies of the university. For example, at NJIT we are trying to get agreement on a policy whereby teaching assistants will be assigned to help with an ALN course when class size goes over a certain limit, such as 30 and/or that class sizes that require the splitting of a course into two or more separate online groups will be counted as "double sections" for course load purposes. There are also ongoing issues related to intellectual property: who owns the rights to the videotapes and the ALN materials created by a professor? Other issues include whether and how faculty should be compensated for their work in preparing course materials for distance delivery, and how to provide adequate support to faculty to master the new skills and techniques they need to be effective in an ALN environment.

VI. LOOKING TO THE FUTURE
When most students have multi-media work stations, the awkward logistics of tape distribution or student recording of broadcasts will probably be supplanted with CD-ROM based or World Wide Web-based digitized video modules. However, for the short term, the VC + video mix seems to work well for a wide variety of courses.
The future of this technology is tied to overcoming some of the difficulties related to the current situation of budget cutting and increased course loads for faculty in higher education. The first difficulty is the initial burden placed upon instructors to completely rethink the nature of their courses and adapt their teaching to a facilitative role. It is also necessary to provide some training for faculty on how to utilize collaborative learning approaches. Because there is an initial increase in workload in terms of creating materials in electronic form, adequate motivations or compensation must be provided, or faculty will feel that ALN will take away from the time devoted to research, which is clearly rewarded in terms of promotion and tenure. Such motivation might come in many forms, including reduced teaching load during the semester that a course is re-engineered for ALN delivery, or clear policies which reward such effort and innovation when promotion and tenure decisions are made.
Institutions may initially think that ALN based courses are a "cheap" way to deliver education. If done with attention to quality, which means that full time faculty should develop and at least be involved in conducting courses, it is actually more expensive. There is much to be done in discovering which tasks can be offloaded from faculty to lower priced teaching assistants, without substantially decreasing the quality of the product delivered. One of the central issues is to decrease the costs of higher education, and a differentiated staffing model may support this better than having faculty do everything, without substantially decreasing quality.
In the long run, ALN's will increase the competition for students and probably lead to a decrease in the total number of colleges and universities in the U.S.. Large institutions and consortia of smaller institutions will develop programs in which students anywhere can enroll, and they will choose this rather than enrolling in a "second rate" local college. ALN's will probably accelerate the trend towards having larger proportions of courses offered by adjunct faculty; already there are instances of faculty members "advertising" their availability to teach Web-based courses for colleges. ALN's will also contribute to the change in the role of colleges and universities, from being primarily residential institutions to take care of 18-21 year olds while they finish growing up, to being centers for a variety of degree programs designed to support students of all ages. Some of these changes may seem desirable and others may seem undesirable. It is up to faculty and to universities to put policies into place that will encourage what are seen as desirable impacts, and discourage the emergence of undesirable trends.