Student Satisfaction and Perceived Learning with On-Line Courses : Principles and Examples from the SUNY Learning Network

A. Course Development Process 1. Step One: Get Started  Visualize course  Assess current instructional practices and relate them to on-line learning  Identify some learning activities and methods of evaluation  Draft profile of course  Successful courses begin with faculty who can articulate description of course  Use a narrative, conversational tone as if addressing a single student to prepare profile of course responding to: ---What will I get out of taking this course? ---What is this study about? ---How is this course organized? ---What exactly will I be doing when I take this course? ---How will you assess my work? ---What constitutes ‘good’ work in the course?  The answer to these questions will become foundation for actual course information and orientation documents.  Document details for course including prerequisites for participation such as additional software or special hardware or other media or tools, if guest speakers will participate

The second section examines issues that contribute to learning effectiveness from a faculty-development and course-design perspective.This section will present the evolution of the four-stage faculty development process and a seven-step course design process that was developed by SLN and comment on lessons learned.
The third section presents results from the SLN Student Satisfaction Survey conducted in spring 1999.This section examines factors from a quantitative analysis that significantly contributes to perceived learning and student satisfaction in on-line asynchronous courses and offers recommendations for course and program design based on these factors.
The fourth section examines learning effectiveness at the level of individual institutions through examples from specific courses.This section will introduce the reader to local implementation of SLN courses at two colleges programs in the SUNY system, the Curriculum Design and Instructional Technology program at the University at Albany (UA) and the Internet Academy (IA) of Herkimer County Community College (HCCC).These case studies present and examine important evidence relevant to learning effectiveness from a single-institution and individual-faculty perspective.
With generous support from the Alfred P. Sloan Foundation, combined with enthusiasm and resources from SUNY System Administration and participating campuses, the SLN has successfully met the challenges of the initial developmental phases that focused on proof of concept and expansion/scalability.The annual growth in courses, from eight in 1995-96 to 1000 in 1999-2000, and annual growth in enrollment, from 119 in 1995-1996 to over 10,000 in 1999-2000, illustrates that the project has far exceeded the original projections.
The SLN started as a regional project in the Mid-Hudson Valley involving eight SUNY campuses.At that time, the development and delivery of asynchronous courses was a new activity for SUNY campuses and faculty.The first courses were offered in the 1995-1996 academic year.
Successful experiences led to an expanded vision and goals for the SLN and the scope and objectives of the project have grown substantially.Where we originally developed courses at the third-and fourth-year leveloffered by two of our institutionswe are now offering courses at all undergraduate levels as well as the graduate level and 42 of our institutions are involved.Our initial developmental phase focused on proof of concept within the SUNY system.This was followed by a phase that focused on proof of scalability that achieved significant growth in course offerings and student enrollments.SUNY's efforts continue to evolve the SLN from a project status to a fully integrated, virtual learning component responsive to the needs of learners in the new millennium.Ultimately, the SLN will represent the entire SUNY through the creation of one virtual campus that will be open seven days a week, 24 hours a day to students across the globe.
The SLN primary mission is to bring SUNY's high quality instructional programs within reach of learners anywhere.Another objective has been to take an efficient approach in supporting the SUNY campuses.
Rather than each campus reinventing the wheel, SLN has developed and implemented the appropriate operational services and support yielding both cost savings as well as the sharing of experience from one campus to another.

I. BACKGROUND INFORMATION FOR PROGRAM
Prior to the SLN program, many SUNY campuses were starting to experiment with asynchronous components to compliment their classroom-based courses.In addition, some SUNY campuses were taking regional approaches to utilizing synchronous, two-way videoconference and some satellite broadcast forms of distance learning courses.

I. INTRODUCTION
Initially the program allocated some compensation from our Sloan grant to the campuses to provide to the faculty in addition to a laptop computer.This is the exception for some selected sponsored degree programs and the local campus handles the majority of individual courses.Some campuses have instituted local approaches to faculty compensation that varies.
The SLN has developed a course-design process to help faculty create instructionally and technically robust learning environments in which to teach and learn.We began the development of our course design process with sound instructional design principles and an existing understanding of distance learning and computer-mediated instruction.Working with hundreds of SLN faculty and students, we have now refined our understanding of on-line teaching and learning and provide our faculty with a comprehensive instructional model that has a thorough framework to guide the design of on-line asynchronous courses.This section will detail our course design process and what we have learned is effective in the design of asynchronous learning environments.

II. BACKGROUND
Beginning in 1994, traditional faculty were hired to create on-line courses for asynchronous delivery into the home via computer.Each faculty member worked with an instructional design partner to implement the course.From fall 1995 through spring 1997, 40 courses were developed and delivered, and the instructional designer conducted interviews, collected empirical data, and made observations.Our objectives during this period were to identify best practices, synthesize scaleable and replicable processes, develop tools and resources, and implement production.

III. RATIONALE
Our objective has always been to develop faculty to teach on-line, and at the same time, insure that they create consistent and effective courses within a specific, limited time frame.This required us to develop a scaleable and replicable process to train large numbers of faculty to produce technically and instructionally sound courses according to what we have learned works best in the design of on-line instruction.Following more than 400 faculty through their full course development and delivery cycles has enabled us to gather a comprehensive understanding of what works in an on-line teaching/learning environment.Our course design process was synthesized from that understanding.Our comprehensive and integrated faculty development and course design processes are the cornerstones of this program and significantly contribute to our success.
Our processes and our understanding in this area have been evolutionary.Access to large numbers of faculty, courses, and students has been the most significant factor in our ability to synthesize a general understanding of effective, on-line teaching and learning, and course design.We are able to collect data, evaluate, and revise specific elements of our program to incorporate our growing understanding.All faculty and course design resources and services reflect our current findings and are refined as our understanding grows.The knowledge we have is explicitly designed into the template application that we created, and is outlined in detail in our course developer handbook.This information is also shared with our entire on-line faculty through our trainings, support, and on-line resources.
We quickly learned that faculty needed guidance and help understanding the options and limitations of this new type of classroom.The need for the multimedia instructional design partner (MID) in our program emerged from our direct work with faculty.We have learned that developing effective on-line instructors and instruction have both technical and instructional aspects that are not necessarily intuitive or analogous to the traditional classroom.For example, there are technical hardware and software issues that require ongoing support in working with faculty.The majority of our faculty requires some kind of technical support and training.Training was required and developed specifically for our program and the applications developed in-house for both faculty and students targeted pointand-click level users.It was clear from the beginning that all faculty required one-on-one support in addition to any group training or documentation they received.The role of the MID evolved to support faculty during their early development with our applications and to help them develop a firm understanding of the options and limitations of this new classroom.
The MID helps faculty design courses and learning activities in a manner consistent with our growing knowledge of best practices.The MID also helps instructors fully understand the limitations students face with potentially slow, remote access and its implications for effective course design.The need to apply a consistent structure to the chunks of a course, and the need to provide detailed explanations, and consistent and redundant instructional cues for students throughout course documents could only be achieved consistently and on a large scale by the MID working one-on-one with faculty.Helping faculty complete the development of their on-line courses prior to the first day of class is another challenge and reason for the role of the MID in the course development process.Without them to ensure that faculty fully develop the materials and activities and test the functionality of their courses, we would not have a consistent way to ensure technically and instructionally sound courses.
We recommend that courses be complete on the day the course starts for several reasons.A complete course gives students the sense of the course as a whole.A stable environment with a consistent design and redundant instructional cues must be designed and tested.Common complaints from faculty include that students will work ahead in the class, or that imposing this constraint prevents the spontaneity or flexibility that exists in the traditional classroom.In the same way that classroom students will rarely read ahead in a book or begin assignments in advance, we have learned that on-line students rarely work ahead of the pace set by the instructor.The advantage to students is that, with the course structure complete, they can get a sense of the topic and of the scope of the activities they will be doing in much the same way as browsing through the course syllabus or leafing through the chapters of a book.We have also learned that the way to insure a flexible on-line classroom is to pre-design a consistent course module structure that contains explanations and shell documents that can accommodate the interests of the students, the spontaneity of the instructor, or that can incorporate current events.A complete course will also allow faculty to concentrate on teaching and managing the course and participating fully with the students rather then trying to plan the next lesson or checking functionality.

IV. METHOD
To begin the process, we ask faculty to think about the development of their courses as a conversion of what they normally do in their traditional classroom rather than a re-creation.Conversion requires that they rethink their learning activities and objectives within the context of the electronic asynchronous learning environment, its options and resources, as well as its limitations, and that they then redesign how they will meet their instructional objectives and how they will assess learning.
For new SLN faculty the first stage in their development as on-line instructors is to get on-line and access the SLN Faculty Developer Gateway (http://SLN.suny.edu/developer).There, they are introduced to the SLN faculty development and course design processes.They participate in a facilitated on-line conference to network with our growing community of on-line instructors and to get the feel for on-line discussion in the asynchronous Web environment.In stage two, faculty begin to conceptualize their courses.They complete an on-line orientation to the Web course environment and they also have the opportunity to observe a variety of live on-line courses that have been selected as models to help them get a sense of the possibilities and to get the look and feel of the on-line classroom.Stage three is the SLN Course Development stage.They are asked to attend three workshops.At the first workshop, faculty receive a customized course template created in Lotus Notes, access to our networked system and on-line resources, and a step-by-step guide for building the components of their course.They are also assigned an instructional design partner to work with throughout their first course-development and delivery cycles and have access to a Help Desk for technology support.Note that it is not until stage three of our faculty development process that faculty are introduced to the technology that they will use to create their course.Our primary focus is on developing and supporting on-line faculty and effective on-line pedagogy, not on the technology.

A. Course Development Process 1. Step One: Get Started
Before beginning work in their course template to design their course, we recommend that faculty begin to visualize their courses in an asynchronous on-line environment.We ask that they assess their current instructional practices and relate them to distance learning principles.We ask them to reflect on what they do in the classroom compared to what they imagine doing in the on-line version of their course.We help them to identify some learning activities and methods of evaluation appropriate to asynchronous learning.We also ask that they draft a profile of their course.Much of the conceptual work in designing their course and our current understandings of effective course design are built into this stage of course development.
We have found that successful courses begin with faculty who can effectively articulate a description of their course.Using a narrative, conversational tone, we ask faculty to prepare a profile of their course that responds to these questions as though a student has asked them, "What will I get out of taking this course?What is this study about?How is this course organized?What exactly will I be doing when I take this course?How will you assess my work?What constitutes 'good' work in the course?"We ask faculty to make their profile user friendly by writing their responses as though addressing a single student.With this we begin to shift faculty from thinking about addressing a classroom of students toward addressing the individual on-line student sitting alone in front of a computer interacting with their on-line course materials and activities.We have found that well-articulated answers to these questions become the foundation for the actual course information and orientation documents that are necessary for students to be well oriented and welcomed into an on-line classroom.
We then ask faculty to document the details of their courses including any prerequisites for participation such as additional software or special hardware or other media or tools, if guest speakers will participate, etc.This step begins to alert faculty to the planning necessary in setting up their courses.Prerequisites must be documented in advance in order to insure that students come to the course prepared and that the technology can accommodate instructor plans.

Step Two: Create an Orientation
We have found certain specific orientation information effective in introducing the student to the on-line learning environment.Students, who are well oriented to the instructor, the course, and the instructor's expectations, will have fewer questions and feel more comfortable.We have identified nine orientation documents that provide students with the walls to their on-line classroom.The purposes of orientation documents are to cover the range of initial information students may need to become familiar with the instructor, the course, and general course-related information.They are: ■ Welcome -Introduces the instructor and the course to the students.We ask faculty to think of it as a letter of introduction.It sets the tone and is the students' first glimpse of the instructor.
■ Contact Information -Details specific information about the course, how to contact the instructor, and the instructor's schedule.
■ Course Overview and Objectives -Describes the course and course objectives in greater detail.
■ Readings and Materials -Details the texts and/or materials to be used in the course.Can list optional/additional reading materials or resources for course.
■ Course Learning Activities -Describes specifically each type of activity that the students will be doing during the course.
■ How You Will Be Evaluated -Details specifically how each activity will be evaluated.
■ My Expectations -Details specifically what the instructor expects from students in terms of participation in the class and/or any other specific expectations the instructor may have for students in their class.
■ Course Schedule -Clearly outlines every activity the student needs to do in the instructor's course, including reading assignments, assignment due dates, scheduled tests and quizzes, special projects, discussions, and group activities.Titles and references to documents and modules in the course must be consistent for the schedule to be effective.
■ Next Steps -Some of the next tasks a student should do might include reading any posted announcements, posting a personal profile, participating in an ice-breaking assignments, etc.

Step Three: Chunk Course into Modules
In designing the modules of a course, the instructor's pedagogical approach, the nature of the content or discipline, and the constraints and features of the on-line asynchronous environment determine how an instructor will chunk his course.We suggest that faculty look at their content, consider how they want to teach it, and see if chunks naturally emerge.We also recommend that faculty look at examples of how others have chunked their courses and provide model courses for observation for this purpose.We have found course structures to be as varied and individual as the instructors themselves.Neither the MIDs nor our SLN course template imposes a pedagogical structure onto instructors or courses.Certain course design structures have emerged as distinct and recognizable across our courses.They include course structures by topic, by task, by chapters in a textbook, by time frames, by steps in a process, by metaphor, and by combinations of these general structures such as time and topic.This is the most important and most difficult step for faculty.It is important, we have found, to allow faculty to create their own course materials and determine the structure of their course.Faculty must have ownership of and investment in their own courses, and ultimately the ability to teach and manage the courses without relying on support.

Step Four: Create Learning Activities in Course Modules
Just as the instructor's pedagogical objectives, the nature of their content, their personal style, and the features and constraints of the Web shaped the module structure of their courses, so, too, will they shape the section structure and specific learning activities for their courses.
We give faculty the following suggestions: ■ List the learning activities that they envision for each of their modules.They then draft a name or title for each activity.
■ Do they foresee students working through the learning activities in a specific order?If so, they draft the list of the learning activities in that order.If not, they list them in a logical order for each module.
■ Does a pattern of activities emerge?For example, activities may logically group by topic, task, or date.
Grouping the activities in a logical and consistent scheme across modules will help the instructor enhance and organize course materials and activities.Consistency in the structure and order of activities across modules also helps students in their understanding and navigation of the course, materials, and activities.
The instructor then creates a draft name for each learning activity that is descriptive and unambiguous.We recommend that they keep the titles short and to the point and that they consider putting due dates, type of task, and a descriptive name in the title.We recommend the use of consistent naming conventions across modules and for similar types of activities.
Once the instructor has decided on the general module framework for the course, the task is to plan out the learning activities within each module.At this stage, sequencing and consistency will be very important.A well-designed course will be consistent and logical in its presentation and organization.For example, a typical module could begin with an overview, followed by some introductory material or lecture.Students are then typically given tasks such as a reading in a textbook, creating a written assignment, and/or participating in an on-line discussion, or directed to complete some on-line or offline project or activity.We ask the instructor to consider the sequence of the learning activities for each module, the quantity of the learning activities for each module, and the pacing of the learning activities for each module.

a. Navigation
Faculty also need to think about how their students will interact with the materials and navigate the course.Any course management tool will have built-in navigational buttons and a Web interface that facilitates students' navigation through all the levels of Web screens.However, an instructor must not assume that students will know what to do and where to go next.Faculty will need to create navigational documents and instructions on their documents that explicitly tell their students where to go next and what to do.
For maximum effectiveness of navigational instructions, they should be consistent.We recommend that they use the same font, put them in the same location on pages, and use consistent wording for the instructions.Instructors can also use the section title and the document title to highlight a type of task, a due date, or a time frame.

b. Evaluation
We also ask faculty to consider carefully how they evaluate students.Timed multiple-choice tests for example cannot be proctored in this environment.Nor can students be observed in person to ascertain certain skills.Working in this environment may require creativity and the design of new evaluation methods.
At this stage, we make the following recommendations to faculty: ■ Review the list of learning activities that they created and take a moment to think about how they plan to assess or evaluate student work, performance, or learning for each activity.
■ Look at the evaluation document they created in their syllabus and orientation area.Have they assigned appropriate values to the types of activities in their course?Do they match the actual activities they have planned?For example, is discussion 60% of the course and only 25% of the grade?
■ How will they evaluate discussion, if it is a component of their course?
■ Review the workload for students and for themselves.How many students are they likely to have?What if they have a very small number of enrollments?What if they have a very large number of enrollments?Will the activities they are planning still work?What alternatives do they have?
■ Give some thought to workload and course management.The more students know about the tasks, activities, expectations, requirements, and how they will be evaluated, the more comfortable and confident they will be participating in the course, and the better able the instructor will be to manage the course.
An Instructional Design Intensive workshop during this step helps faculty identify instructional and technical solutions to create the learning activities in their course and effectively achieve the instructional objectives they have for their course.

Step Five: Walk Through Course
An integral part of the course development process is the evaluation and revision of the course modules as the instructor develops them.If possible and if time permits, they may want to have an outside reviewer such as a colleague or expert in the field, and/or an instructional designer review their course.Reviewers can give very valuable feedback about issues such as content accuracy, technical quality, functionality, and user acceptability and usability, and issues associated with actually implementing and using the instruction.The SLN MID is responsible for this review and we provide a series of technical and instructional "preflight" checklists to facilitate this process.
Whether a reviewer is used or not, it is important for faculty to evaluate and revise or refine the structure, materials, and activities they are designing during the development phase of their courses.The checklists we provide have been designed to help faculty and our MIDs to evaluate, review, and pinpoint areas in their courses in need of revision or further development.A Teaching and Managing Your Course workshop at the end of this step marks the transition from the development phase to the course-delivery-phase.Technical and instructional issues are addressed and we provide a roundtable opportunity for new faculty to meet experienced faculty to discuss their on-line teaching.Giving the experienced group the chance to share their tips, strategies, and recommendations and to allay and fears or concerns new faculty has been very effective and well received by new faculty.

Step Six: Get Ready to Teach
We provide our SLN faculty with a number of recommendations and tips for getting off to a good start.At the beginning of the semester, we encourage faculty to encourage all students to get familiar with the Web environment for their courses.We provide a moderated student-orientation course and encourage faculty to make sure their students prepare for their courses by first going through the SLN student orientation.We also suggest having a few warm-up activities designed in the first module of all courses to get everyone to know each other and to practice using the features specific to the Web-class environment.This allows students to practice doing the kinds of activities they will be doing in the course, and can be designed to break the ice, i.e., introduce the course and the participants in the course to each other and practice certain activities.It also begins to support a sense of class community, something we have found to be a very important part of an effective on-line learning environment.In order to keep the class moving we recommend that the instructor make sure that there is something new for the students at least every two to three days.If students are not moving the discussion along, the instructor might call on specific students to clarify a particular view or to provide support for a view, comment on existing responses, and invite students to respond again.Or put a note in the announcements area encouraging students to participate.If some students continue to remain silent, the instructor can send individual students an E-mail message.(Faculty need to keep in mind that there may be something preventing a student's participation such as, a trip, illness, technical difficulties, etc.)

Step Seven: Evaluate and Revise Course
In anticipation of the evaluation and revision stage, we encourage faculty to keep notes during the teaching phase of their courses.Notes on any issues or problems that emerge as they teach, or that are commented on by students, can help in the evaluation and revision of courses.Thoughts, general or specific, on the design, structure, pacing, and/or sequencing of the courses, or of any of their activities should be documented as the courses are taught.This is the last step in the course developer process.Once instructors conclude the teaching phase of their courses, they should evaluate the courses and their experience and review the notes they made as they taught to assess any improvements and revisions necessary to the structure or activities.
We ask faculty to think about what worked well?What did not?Why?What could be improved?How? Were discussions successful?Were assignments and other activities successful?Were students able to complete all the modules in the course?Did most students complete the course?How was the workload for the instructor and for the students?Was the instructor able to keep up?Was there anything missing?Were there any points in the course where students did not do or understand the activity?The checklists can be used again in this stage to guide or focus summative evaluations of the course materials.

B. Course Management Tips
Faculty should do the following: ■ Log into their courses on a scheduled basisespecially frequently at the beginning of the semester.
Students will be wondering who is out there and the instructor can help by responding right away.This gives students a sense of security and lets them know everything is functioning correctly.Setting and maintaining a regular and consistent log-on schedule is very important and though faculty responsiveness is critical, the expectations and workload should be realistic.
■ Respond to all student E-mail immediately.E-mail should only be used for private communication between student and instructor.If the message is not private in nature instructors should ask the student to post it in the appropriate place in the course.
■ Check for and respond immediately to any student queries in the course itself.
■ Grade and return evaluated assignments to students as quickly as possible.
■ Check to see that students are responding in the appropriate locations in the course and address any problems that may arise immediately.Keeping a course tidy and free from problems, false starts, or empty student documents created by accident keeps the "classroom" running smoothly, cleanly, and free of potential sources of confusion.

A. What We Have Learned and What We Know
On-line courses are, by nature, learner-centered and can have more active participation by all students in the class than in a traditional classroom.Without the structure of weekly classes, students are generally expected to take a more active role in their own learning.A fundamental difference is that instead of simply showing up to make their presence known, in an on-line class students must do something, for example submit an assignment, ask a question, participate in a discussion, etc. Opportunities for these interactions with the course materials, with the instructor, and with other students must be designed into the on-line classroom.
On-line courses differ from traditional classroom courses in several ways.Since students do not have non-verbal cues or the ability to raise a hand to ask questions, learning activities, instructions, and writing must be clear.Faculty must assume nothing and anticipate and address student questions.Faculty that are able to assume the perspective of the student as they design their courses and activities are better able to be sensitive to these issues and to create effective on-line learning environments.
We have learned that an effective learning environment consists of well-organized and complete orientation and syllabus information that begin a course and are essential to help orient the students to the course, the instructor, and to what will be expected.In the design of course materials, faculty need to pay special attention to the tone of their writing and consistency in their module structure, document naming conventions, and instructional cues.Explicit orientations to each module with due dates, time frames, and details about what the module contains, as well as redundant, clear, explicit expectations and instructions are necessary to insure students are at all times well oriented to the content, activities, and tasks in the course.Faculty should design and create as many possibilities for student interaction as possible, both with the instructor and with others in the class.
Our large-scale production required the development of ways to train large numbers of faculty and produce large numbers of courses of consistent quality.Using the MIDs, we avoid cookie-cutter mass production by working with individual faculty and allowing them and their content to drive the design of their courses.And we have the opportunity to influence and share best practices across the design of all courses by the same method.
We ■ Course information that is accessible and redundant ■ Ample opportunities for interaction with the instructor and with others in the course ■ Opportunities to engage and interact with the content activelydirected-learning activities

C. SLN Best Practices: Instructors
Effective on-line instructors ■ Assume nothing and anticipate and address student questions in the design of the course.
■ Are responsive and present in the course.
■ Use directives, first-person voice, and conversational tone.
■ Are sensitive to the student's perspective.
■ Create complete, well explained on-line and off line activities.
■ Encourage a sense of class community and provide community building opportunities and interactions.

D. Course Design Recommendations
Specific examples of some of the course design recommendations include ■ Create a non-graded ice-breaking activity in the first module of the course.Using the mechanisms for conducting an on-line discussion in your course, ask students why they took the course.This will help everyone get to know each other.It provides an opportunity to practice and model a good on-line discussion, and students who enroll late or have technical difficulties will not be so far behind.
■ Encourage a sense of class community and build opportunities for interaction with the instructor and with other students in the course.
■ Consider using a self-test the first week of class as a comprehension check on the orientation and syllabus documents for your course.This can make sure that students read that information and eliminate questions later on in the course.It also introduces the testing capability to students in a less threatening way.
■ ■ Long documents can be broken up into several shorter documents.A good rule of thumb is to not exceed four to five screens for scrolling.On long documents the instructor can inform the student at the top of the pageyou may want to print this out for easier reading.
■ Use heads, subheads, hypertext, and a document hierarchy to break up long paragraphs.But do not break them up so much that it affects the flow or meaning.
■ Put important information at the beginning of a document.
■ Use short descriptive titles for document subjects and module names.Long titles do not fit well on the screen and they lose their purpose.Indicate the type of assignment, due dates, or time frames in the subject lines or module names and use them consistently throughout your course.
■ Use directives, first person, and a friendly conversational tone.This personalizes the course for the student.
■ Do not overuse hypertext to link your course pages or to link to other Web sites.
■ Spell check work.
■ Consider creating a prepared welcome E-mail message that can be forwarded to students as they appear in the course over the course of the first week.
■ Consider sending out an introductory letter to students that specifies the first off-line reading assignments for the first couple of weeks.If they have technical problems they can do the initial reading, know what they should be preparing, and not be so far behind when they finally get on-line.Instructors may also want to design the activities in their course for the first couple of weeks with this in mind.

E. Effective Navigation
We have found the following strategies effective in making sure students will be able to successfully and efficiently navigate the pages and activities in an on-line course: ■ Create Instructional Documents -Instructors should create documents that set up the directions and expectations they have for their various learning activities.
■ Create and Use Instructional Cues -Instructional cues are the instructions and directions that explicitly help students navigate the pages of the course and learning activities efficiently.Instructions are very important in an asynchronous learning environment.Students need to know what to do, where, when, and how.And they need to be able to access information quickly and without difficulty to avoid distraction.For example, if an instructor wants the students to go to the Discussion Area of a course and to respond to a discussion question, they have to tell them to do that and tell them how.
■ Use Module, Section, and Document Titles to Organize and Convey Information about the Activities, Content, and Structure of Your Course -The module, section, and document titles present the organization of the course and all its activities.For purposes of clarity, faculty should consider using titles to specify the type of activity, due date, time frame, etc.The more information that can be put in this framework that the students see from the module view, the more comfortable and confident students will be with what they are to do.

I. RESULTS
In spring 1999, students enrolled in the SLN completed a survey that may be useful in understanding questions related to learning effectiveness in asynchronous on-line courses.In all, 1,406 students completed the survey that represents approximately 42% of enrollees for the spring 1999 semester.The results that stand out most clearly for learner effectiveness are outlined below.
Note: For this section all rating for perceived learning are based on a Likert scale: 1= I learned more than I expected 2= I learned as much as expected 3= I learned less than I expected 4= I learned nothing

A. Interaction with Teacher
Interaction with the teacher is the most significant contributor to perceived learning in these on-line courses.
Students who reported the highest levels of interaction with the teacher also reported the highest levels of perceived learning in the course.

Student Rating of Learning by Interaction with Teacher
How much did you learn?

B. Levels of Participation Compared to Classroom
Students who reported that they participated in their on-line classes at higher levels than in the regular classroom also reported the highest levels of perceived learning.

Perceived Learning by Level of Participation
How much did you learn?

C. Interaction with Classmates
Interaction with classmates is a significant contributor to perceived learning in on-line courses as well.Students who reported the highest levels of interaction with classmates also reported the highest levels of perceived learning in the course.

Perceived Learning by Interaction with Classmates
How much did you learn?

D. Help Desk and Technical Difficulties
Students who reported the highest levels of satisfaction with the Help Desk also reported significantly higher levels of learning than students who rated their satisfaction with the Help Desk as lower.Also, students who reported that technical difficulties impeded their learning reported significantly less learning over all than students who did not report that technical difficulties impeded their learning.

Perceived Learning by Satisfaction with Help Desk
How much did you learn?

E. Motivation
Student motivation for taking courses appears to play an important role in perceived learning.Students who reported that they were taking courses because they were not offered on campus reported significantly lower levels of learning than students who were taking courses because of family responsibilities or because of a conflict with their personal schedule.

Student Ratings of Learning by Reason for Taking the Course
How much did you learn?

F. Gender
Gender appears to play an interesting role in on-line learning.Women reported higher levels of perceived learning than did men.

Student Ratings of Learning by Gender
How much did you learn?

G. Age
Age may also play a part in perceived learning in on-line courses.The youngest students (16-25) reported that they learned the least and that they were the least satisfied with on-line learning.Students in the 36-45 year old range reported that they learned the most and were the most satisfied with on-line learning.

Perceived Learning by Age Range
How much did you learn?

A. Interaction with the Teacher
Interaction with the teacher is the most significant contributor to perceived learning.
Students who do not have adequate access to their instructors feel they learn less.They are also less satisfied with their courses.These measures, while not precise indicators of learning effectiveness are, nonetheless, important in and of themselves.It would seem that, in terms of course design, those courses that include ample opportunity for student-teacher interaction are preferable to those with limited or no interaction.In light of the importance of this variable, another important instructional design feature is the inclusion of documentation outlining reasonable expectations for teacher-student interaction.Clearly, instructors cannot be available 24 hours a day or at the whim of the students.However, if the turn-around time on student requests for assistance is plainly communicated and consistently applied, student disappointment, anxiety, and confusion can be reduced and satisfaction and learning can be increased.

B. Level of Participation
Students who reported that they participated in their on-line classes at higher levels than in the regular classroom also reported the highest levels of perceived learning.Opportunities for high levels of participation are an important course design feature for encouraging learning.Course designs that encourage equitable exchanges of ideas in which the contributions of all students are valued are preferable.Documentation that explains that participation is important and valued and which therefore encourages high levels of participation is another useful on-line design feature.

C. Interaction with Classmates
Interaction with classmates is a significant contributor to perceived learning in on-line courses as well.Students who reported the highest levels of interaction with classmates also reported the highest levels of perceived learning in the course.Opportunity for interaction between classmates is another important course design feature.Documentation that explains that productive student collaboration will be valued in the course is one way to encourage such exchanges.Obviously, creating the forums for such collaboration is also necessary.The methods for building and maintaining student-to-student interaction require careful consideration and a good deal of facilitation, especially early in the course.

D. Help Desk
Students who reported the highest levels of satisfaction with the Help Desk also reported significantly higher levels of learning than students who rated their satisfaction with the Help Desk as lower.Also, students who reported that technical difficulties impeded their learning reported significantly less learning overall than students who did not report that technical difficulties impeded their learning.
Clearly, students feel that technical difficulties can and do impede their ability to learn.It therefore appears to be very important to provide some level of technical assistance to learners in this environment, especially for large programs such as SLN.A course design feature that can be recommended based on these results is documentation that encourages learners to seek help early, before they become so frustrated that they give up.It may also be useful to explain to students that on-line learning requires a certain degree of self-reliance and initiative that may exceed that required in the classroom.

E. Motivation
The students' motivation for taking the course appears to play an important role in perceived learning.Students who reported that they were taking the course because it was not offered on campus reported significantly lower levels of learning that students who were taking the course because of family responsibilities or because of a conflict with their personal schedule.This appears to be a difference between intrinsic and extrinsic motivation.Students who feel that the courses are beneficial because of the flexibility they offer in allowing the completion of goals that are otherwise prohibited may feel they are learning more than students who feel they must take the course because there is no other way to do so.Again, documentation regarding successful on-line learning strategies is useful in helping students decide whether this environment is for them.

F. Gender
Gender appears to play an interesting role in on-line learning.Women reported higher levels of perceived learning than men.This result is especially interesting in light of recent research that reports that in on-line learning (as in the classroom), "Males dominate the conversation, effectively silencing women."(Blum, 1999, p. 10) From our initial examination of the result of this survey (and a pilot survey), small but reliable differences exist suggesting that women feel that they participate at higher levels than in the classroom, that they learn more, that technical difficulties are less likely to impede their learning, that they are more likely to want to continue taking on-line courses, and finally, that they are more satisfied with their specific courses at SLN and more satisfied with on-line learning in general than their male classmates.In summary, the on-line classroom appears to be a very female friendly place.

G. Age
Age may also play a part in perceived learning in on-line courses.The youngest students (16-25) reported that they learned the least and that they were the least satisfied with on-line learning.Students in the 36-45 year-old range reported that they learned the most and were the most satisfied with on-line learning.
Similar results were found in the pilot survey.It may be that age is an indicator of other important characteristics.
Students who are attracted to and succeed in this form of learning tend to share certain traits.Generally speaking, they are voluntarily seeking further education, are motivated, have higher expectations, tend to be older and tend to possess a more serious attitude about their courses (CDLP, 1997).In some ways, age is a proxy for these attributes.
Often, older studentsespecially those with familial obligationsare seeking further education out of necessity, either to keep a job or to get a better one.They tend to have higher expectations, more motivation and a more serious attitude for a number of reasons.If the courses are well designed, it is not unreasonable to expect these students to participate at higher levels and to experience higher levels of satisfaction and learning because of their backgrounds.Once again, it is important to communicate what kinds of students tend to succeed in and enjoy online courses.While age is not in anyway a barrier, motivation and self-reliance may be even more important on-line than in the classroom.

H. Computer Skill Level
An interesting factor that does not seem to matter is reported computer-ability level before the start of the course.In the spring 1999 semester, as in the fall 1998 semester, students' prior computer skill level did not play a significant role in perceived learning.This seems especially curious inasmuch as students who reported their learning was impeded by technical difficulties felt they learned significantly less than those who did not feel this way.Once again, the students with the least prior computer knowledge reported the highest levels of learning.Perhaps the most important implication of this findinglack of prior computer knowledge does not seem to be a barrier to on-line learning.
The next two sections examine learning effectiveness at the level of individual institutions with examples from specific courses.This section introduces local implementation of SLN courses at two college programs in the SUNY system, the Internet Academy (IA) of Herkimer County Community College (HCCC) and the Curriculum Design and Instructional Technology program at the University at Albany.These case studies present and examine important evidence learning effectiveness from a single-institution and individual-faculty perspective.

I. INTRODUCTION
This section will deal with the sequence of events leading to the formation of the IA at HCCC.HCCC is a mediumsized (2,500 students), two-year college in upstate New York.In spring 1997, HCCC decided to join with several other SUNY colleges and participate in the SLN.The academic dean decided that HCCC would begin by offering Internet-based courses leading to the A.A.S. degree in Travel and Tourism, a program that HCCC has offered with great success for many years.In preparation for the fall 1997 semester, three courses were selected, three faculty members were recruited, and a campus support person was designated.As seen in Table 1, the total fall 1997 enrollment was 36 students.In spring 1998, five SLN courses were offered and total enrollment was 55.In the fall 1998 semester we offered eight courses to 94 students, and in spring 1999, there were 12 courses with 206 students.HCCC will offer 26 courses in fall 1999, and anticipates approximately 390 students.

HCCC Courses & Enrollments
During spring 1999, plans were made to expand the Internet-based courses into additional programs and the IA grew out of this effort.The IA was inaugurated at a press conference on May 6, 1999.Students are currently able to be admitted to the college, register for courses, and complete all of the coursework in any one of six degree programs, all without visiting the campus.Plans are to expand this to 10 degree programs in fall 2000.Each program requires between 62 and 65 credits to complete.Course offerings will be programmed in such a way that students may complete their degree programs within two years.The college maintains an IA Website at http://hccc.ntcnet.com/IA.

II. RATIONALE
The IA is thought of as a separate entity within the college.Although the Internet-based courses are scheduled concurrent with the on-campus course calendar, they are intended to appeal to students who find it difficult or impossible to travel to campus for their coursework.Participants in our Internet-based courses completed a survey during the spring 1999 semester.The data suggests that approximately 40% would not be able to take the courses if they were required to attend classes on campus.Additionally, our data suggests that the most important reason why students take Internet-based courses is for convenience.At HCCC, the associate deans who regularly observe classes monitor teaching effectiveness.This same model was applied to the Internet-based classes.The associate deans observed professors as they participated in on-line discussions, graded papers, and conducted the routine tasks associated with on-line teaching.One of the associate deans now teaches an on-line course for the IA, and the other associate dean, who was somewhat skeptical initially, now expresses great enthusiasm for this approach.
Prior to fall 1997, there was no Internet-based instruction at HCCC.In fact, the Internet first became available on campus in fall 1996.The individual who was designated as the support person for the SLN courses had been appointed director of learning systems technology (LST) in fall 1997 and was primarily concerned with two-way, compressed video as the means of providing distant access to campus courses.Then in spring 1999, at an all-faculty meeting, the president of HCCC, Dr. Ron Williams, suggested that we should begin thinking more globally about our Internet-based courses.He went so far as to proclaim his interest in creating "some kind of Internet Academy" with the goal of providing easy access to our programs for disabled and non-traditional learners.A short time later the Director of LST expressed an interest to Dr. Williams in pursuing the idea of an IA, and, with a tremendous amount of campus-wide cooperation and support, the IA now exists.

III. METHOD
For the method section, this case study will focus on the professor at HCCC with the greatest amount of Internet teaching experience.Professor William Pelz, the author of this section, taught two courses, Freshman Seminar (one credit) and Introductory Psychology (three credits) in the fall 1997 semester using courseware called TopClass.The courses were not offered via the SLN but were hosted by SUNY on a SUNY server as part of a university-wide evaluation of TopClass.Introductory Psychology has subsequently been taught four times via the SLN using Lotus Notes.The Freshman Seminar has been taught two more times, also using Notes.In addition, Professor Pelz has developed and taught Abnormal Psychology (twice) and Social Psychology (once).In addition to teaching over the Internet, Professor Pelz has taught Introductory Psychology and Social Psychology over a two-way, compressed video interactive synchronous network.Because of these experiences, he is well positioned to offer opinions concerning courseware, technology and pedagogical issues.

A. Technology and Infrastructure
The courses were developed using either TopClass (1997) or Lotus Notes with custom templates provided by SLN (1998-present).Of the two course-management programs, Lotus Notes was the more mature product and provided the most satisfying experience for both the students and the professor.To a large extent, the ease of use and pedagogical functionality of Lotus Notes is provided through the custom templates developed by SLN programmers.These templates enable the use of class discussions, small group exercises, self-tests and assignments with relative ease, and are flexible enough to allow each professor to exhibit his/her own style of interaction.Although the SLN courses are developed with Lotus Notes, they are viewed by students using their Web browsers.No special software is required.The SLN templates provide a consistent look and feel to all of the courses, making the students' task of mastering the technology much easier.
The infrastructure of servers is provided by SLN.Several servers (the number has grown over the years to five) are maintained and all of the course databases are replicated among these servers frequently.This system provides a great amount of redundancy, so that equipment failures almost never prohibit access to the courses.In addition to servers, SLN provides extensive training and support, both during course development and course administration.There has been no charge for these infrastructure services to date, but that looms as a future possibility.

B. Course Delivery
In all of the courses Professor Pelz has developed, he and not the students determine the pacing of the course.The primary reason for this is to facilitate extensive discussion on course-related topics.Students are willing, many even eager, to discuss relevant, course-related topics asynchronously.The pedagogy requires each student to lead discussions on topics they select from the readings.The other students in the class are required to participate in these discussions by responding to critical thinking questions posed by the student discussion leaders.Discussion continues until it self-extinguishes, and the instructor interjects his comments only as needed to facilitate the integrity and accuracy of the discussion.Professor Pelz attributes much of the student excitement and enthusiasm for his courses to this strategy.Additional methods include essay exams on the readings (which he treats as take-home, open-book exams) and Web-based research papers which require students to locate and sift through numerous Web sites and make decisions on the value of the information they convey.
Student performance is assessed as follows: participation in discussions (quality of questions and responses as well as quantity of contributions)50%, essay exams on the content20%, and research papers30%.No synchronous activities are required.

C. Organization and Evolution
Course development is the responsibility of the professor.However, there is bountiful support from the SLN staff and more recently from local campus expertise.The SLN staff conducts a series of workshops which new course developers are required to attend.In addition, there are separate workshops which experienced Web-instruction faculty are encouraged to attend.There are three workshops for new faculty.They are conducted throughout New York State at sites convenient for all participants.The training begins about six months prior to the course going live, and continues until the course begins.Support is not limited to the workshops.Each professor is assigned a MID who works with him/her on a one-to-one basis throughout the development cycle.Initially, MIDs who work for SLN provide this service.But as participation grows, the campus is expected to provide this support locally.As an example, HCCC appointed Professor Pelz as campus MID when it was decided to launch the IA.It is expected that each campus will do this when the number of courses they offer reaches 15-20 per semester.
The SLN Help Desk provides additional support.At any time, a faculty member can call or E-mail the Help Desk to receive assistance in using the technologysuch as server access, modem setup, E-mail and file attachment issues etc.Help Desk support is also available to students in the courses.
A final support exists in the form of a Developer's Handbook.This reference provides step-by-step instructions and examples of the tasks required to develop and manage a course using Lotus Notes and the SLN templates.The Developer's Handbook is periodically updated and is indexed for ease of use.
HCCC has encouraged and facilitated faculty involvement in Internet-based instruction by providing a budget that addresses faculty needs.When professors volunteer to develop an Internet-based course, and this is never required, they are provided with a computer and printer for their homeseven if they already have home systems.In addition, they receive stipends (approximately $1,000, dependent upon academic rank) for developing and teaching the courses.Also, the college pays to have a second telephone line installed into professors' houses (up to $200), and pays up to $40 per month for telephone and Internet access.This year, the college also moved to provide a campus MID.The college also provides faculty travel expenses to SLN workshops.In exchange, faculty agree to teach each course they develop a minimum of two times.

IV. RESULTS
There are several performance indicators that address the issue of learning effectiveness: ■ Perceived learning by the participating students: The summer 1999 survey results indicate that 94% of students who complete an Internet-based course believe that they learn as much or more as they would in a classroom-based course.
■ Completion rates: Results from the spring 1999 survey reveal that 82% of Internet students complete their courses, and 78% of students taking the same courses in the classroom completed.
■ Perception: Comments which were often made on the student survey suggest that many students perceive the Internet-based courses to be harder than traditional classroom-based courses, yet 70% indicated that they will take more courses on the Internet; and 45% responded that they would like to do all of their coursework on the Internet.

I. INTRODUCTION
We began work on the on-line version of our master's degree in Instructional Technology in spring 1997.Judy Genshaft, the vice-president of Academic Affairs at the University at Albany, brought together several members of the Educational Theory and Practice (ETAP) department, and asked the faculty if they were interested in developing a program that would offer a master's degree entirely asynchronously.
Like most SLN degree programs, UA's was created from an existing face-to-face curriculumthe master's degree in Curriculum Design and Instructional Technology (CDIT).This program was designed both to meet the certification needs of practicing teachers in areas other than those covered by the secondary advanced certification programs and to offer specialized training and retraining in curriculum, instruction in the burgeoning field of educational technology.It was this latter discipline on which UA focused, in part from a shared belief in design as a particularly important path to understanding.
The CDIT degree is a 30-credit, 10-course, master's program, half of which is prescribed, half of which is designed by the student in consultation with an advisor.The intended audience for on-line CDIT courses is people interested in instructional technology but separated in space from our face-to-face classes.It has been found, however, that a large percentage of traditional student populations, practicing teachers, are very appreciative of the chance to complete at least some of their coursework asynchronously.They, it turns out, are often separated from classes by time conflicts.

II. RATIONALE
The primary motivating factor for developing an on-line master's degree program was the chance to experiment with the new medium.For faculty, then, course development was a learning as well as a teaching experience.Online courses also allowed UA to offer students the opportunity to experience advanced educational technologies as they learned about them.
There were no specific goals to be met by the courses other than course objectives common to both on-line and faceto-face versions of the classes.On-line versions of UA's regular end-of-semester student evaluations were undertaken during the first two semesters.The results of these evaluations were slightly higher than those of the traditional versions of the classes.(See Figures 7 and 8.)

III. BACKGROUND INFORMATION
There had been no completely asynchronous courses given at the UA previously, although some very interesting mixed face-to-face and on-line courses had been created for Project Renaissance, and there had been a variety of experiments with satellite and two-way video distance learning.In fact, two other departments turned down the SLN project previously.
Students in the CDIT program have traditionally tended to be mostly newly hired teachers (from all disciplines including the secondary academic areas) who need a master's degree in five years to obtain permanent teaching certification.These are working professionals in their mid-to-late 20s.This trend was accelerated with the introduction of the SLN program.At least one-third of the on-line students are mid-life working professionals in fields other than pre-college teaching who for various reasons want instructional technology courses and/or the master's degree.More than half of on-line students are over 30.Nearly two-thirds are male (in contrast to traditional enrollments).If anything, these students have been better than usual.
Many traditional studentslocal teachersflocked to the on-line courses to fulfill some of their degree requirements.They state that other obligations have made it difficult to attend even one-night-a-week offerings.
Currently, about half of the students have had no ALN experience before enrolling.The other half has taken one or more asynchronous courses, usually within the UA program.Almost half of these students rank themselves as highly computer literate and nearly another half consider themselves to possess intermediate computer skills.Thus, only 10% of the students currently enrolled consider themselves computer novices.Distant students were more likely to be more computer/WWW experienced.
The completion rates for on-line courses have been statistically similar to face-to-face offerings.If anything, they have been slightly higher (Figures 1 and 2).Enrollments in the on-line courses are much higher than in the face-toface versions of the courses (See Figures 3 and 4), which had not been anticipated.

A. Technology and Infrastructure
Courses were developed in Lotus Notes using custom templates provided by SLN (designed to run on both PC and Macintosh platforms).The courses reside on multiple servers maintained by SLN.A decision was made at SLN to keep all courses as simple as possible so that the greatest number of students could access them.Therefore, UA courses are primarily text-based, although many (especially those concerned with media) utilize graphic organizers and images.

B. Content Delivery
These courses, as are all the courses in our program, are delivered according to our regular 14-week semester schedule.Students are strongly encouraged to keep up with assignments that are given on a weekly basis.
Students are given many small assignments of a variety of types (see below) that are assessed on a pass/fail basis for a set number of points.They just have to do it.Sometimes (rarely), extra points are given for especially good assignments and/or points taken off for extremely poor assignments.Students are allowed to redo poor assignments for full credit.Students are primarily assessed on the basis of written assignments, projects, and discussion.On-line discussion is one of the strongest features of asynchronous environments.2 comments @ 1/2 pt ea.

C. Organization and Evolution
Faculty members have sole responsibility for implementing their courses on-line, but they are given assistance as needed from SLN multimedia instructional designers (for instructional design issues) and Help Desk staff (for technical issues).Because the degree program is in instructional technology, less help is needed than for typical faculty members, but help is available when it was needed.Although local support was promised, it has been slow in coming.
ETAP faculty were given monetary stipends, course reductions, laptops, and a budget of $1,000 for additional resources to develop and teach the SLN version of their courses at least once.Monies came from the Sloan Foundation through SLN.Participation in the program was strictly voluntary.The faculty members normally responsible for them developed all courses as on-line versions of face-to-face courses.
Computing in Education (ETAP 426/526) course has been offered four times.The last two semesters, enrollments have been so high that two sections taught by graduate assistants have been added.This fall, three sections were offered.Grading procedures have changed somewhat over time, with new Websites and readings, and revisions made to the instructions to students.

V. RESULTS
For the first two semesters standard student evaluations were usually given at the end of the semester.These Likerttype rankings ask students to agree or disagree (on a five-point scale with five representing "strongly agree") with two statements"This course was excellent" and "This instructor was excellent."The evaluations are traditionally given with the instructor absent from the room.They are machine scored and give a total (averaged) rating for each course.The on-line versions were E-mailed to students and compiled by the SLN staff.
A comparison of evaluations for two courses for which this was done is given in Figures 7 and 8.The comparison is between student evaluations of the on-line versions of each course and student evaluations of the traditional version of the same course given in the preceding year.Student evaluations of the on-line versions of these courses were slightly higher but statistically similar to student evaluations of their traditional versions.Similarly, final grades in the on-line versions of these courses were slightly higher but statistically similar to the traditional versions of the same courses.(See Figure 9.) Finally, as previously noted (Figures 1 and 2), the completion rates for on-line courses have also been slightly higher but statistically similar to face-to-face offerings.One of the critical factors for the success of on-line learning is the valuing of student performance by the instructors.This can take many forms.A particularly good example is on-line discussion.When on-line discussion is valued (graded), authentic (involves real questions), and frequent, when interactions are positive and enthusiastic, students learn more and are happier.Another way of valuing student performance is portfolio assessment.Portfolio assessment respects the learner and gives all students the chance to excel.
Student/teacher and student/student interaction is also critical to successful on-line learning.Frequent, positive, and personal interactions can help bridge the communication gap created when face-to-face courses are moved on-line.Paris is a wonderful city.There is much to learn by exploring its culture and landmarks.This can be done two ways, one as the accidental tourist or by thoughtful and purposeful use of the Paris Metro system.This is not dissimilar to the development and implementation of ALN in our higher education classrooms.Much like the purposeful visitor to Paris, instructors must thoughtfully select and implement the instructional content and teaching methods for use in their technology-enriched environments to ensure a positive effect on student learning.Such is the framework for the State University of New York (SUNY) ALNto provide instructors with guidance and training as they use the vehicle of technology to impact the educational experiences of their students.

ABOUT THE AUTHORS
The workshop context for the SUNY presentation was a forum where academics from varied disciplines and institutions of higher education came together to share best practices of technology and teaching.The specific purpose of the Learning Effectiveness portion, which housed the SUNY presentation, was to explore issues related to which teaching methods and technologies are the most effective in helping students to learn.This paper and presentation furthers this academic discussion by presenting four themes important in studying learning effectiveness within the context of technology-enriched higher education classrooms.
The SUNY Learning Network understands the fundamental principle that effective teaching leads to effective learning.Throughout the training that they provide for faculty they emphasize several principles of good teaching.
Their seven-step course design process begins by asking faculty to look at their current instructional practices, translate their learning activities to the technology-enriched classroom, and make clear both their learning objectives and the methods by which they will be accomplished.
Integrated within all seven steps are other principles of good teaching.They encourage student-faculty contact on a regular, timely, and meaningful basis.Active learning, time-on-task techniques, cooperation among students, and providing prompt feedback are examples of good teaching practice specifically mentioned in the paper.
A subset of this theme is that of tailoring effective instruction to the technology-enriched classroom.Those at the SUNY Learning Network have learned that simply moving course materials on-line without specific consideration of on-line pedagogy has negative impacts on learning effectiveness.The authors note that they "have learned that developing effective on-line instructors and instruction have both technical and instructional aspects that are not necessarily intuitive or analogous to the traditional classroom."They make the compelling case that along with any technical support there must also be ongoing instructional support provided to encourage effective teaching that will then lead to effective instruction in the technology-enriched classroom.This guidance must come from personnel trained in both the technology and the pedagogy of ALNs.
When I worked in industry I learned that quality was equal to consistency.When I entered higher education I learned that one of the primary productsstudent learningis enhanced by the concept of academic freedom.The blending of these two concepts is a common theme in the SUNY paper.The authors note, "We develop ways to train large numbers of faculty and produce large numbers of courses of consistent quality.Using MIDS (Multimedia Instructional Design Specialists), we avoid cookie-cutter mass production by working with faculty and allowing them and their content to drive the design of their courses."Thus, instructors may find themselves spending more REVIEW time wandering when a more focused effort would produce more effective learning.As ALN becomes more and more mainstream to the academy, this theme will become increasingly important.
As an educational psychologist, I know that increasing both the types and frequency of assessments provided to students can have a positive impact on learning effectiveness.For example, assessments can become more authentic in technology-enhanced classrooms.Students in Biology courses may now be asked to dissect a virtual cow's heart or biochemistry students to manipulate complex DNA.Further, while it is impractical for instructors in large courses to give frequent quizzes, technology allows ongoing assessment with frequent feedback provided to the students.
The two case studies in the SUNY article present evidence that learning effectiveness was increased because of the impact that technology had on their ability to assess their students.At Herkimer County Community College, Professor William Pelz felt that his on-line discussions (which were assessed more formally than in-class discussions) increased the students' ability to critically analyze the topic.Professor Pat Swan, at the University at Albany, makes use of technology to have her assessments all portfolio-based.She gives many small assignments that are quite varied.This theme brings forward that ALN is changing the face of higher education by giving instructors more flexibility to use assessments that directly impact learning effectiveness.
By collecting and sharing data, we forward the disciplined inquiry of the relationship of learning effectiveness and ALN.Further, the evaluation data that is collected must move past that of satisfaction and onto outcomes that impact learning effectiveness.
On a more local level, the systematic evaluation of teaching is critical to ongoing, formative improvement of instruction.
Step seven of the SUNY model states, "After you teachevaluate and revise your course."I further encourage instructors to evaluate formatively while they develop their course and during the semester while they are teaching.By collecting information in this formative manner, it can be more easily linked to making learning more effective for the students currently enrolled.
Clearly the study of ALN must include stringent evaluation that collects data to guide and inform us as we change the very face of higher education.I encourage that more disciplined, pervasive, and consistent evaluation be conducted throughout our projects.I further encourage that we move our evaluation questions toward learning effectiveness as well as issues of satisfaction.Obviously, extra funding will be necessary to ensure that these efforts are possible.
The instructors within the SUNY Learning Network provide guidance and direction to maximize the learning effectiveness within the ALN context.Most clearly presented in the SUNY paper and presentation is that effective teaching leads to effective learning.An umbrella that advocates principles of good teaching (such as use of multiple assessments) fostered by the SUNY group, is that of academic freedom.While the SUNY Learning Network has a consistent framework, it is built on the premise that academic freedom is inherent to increasing effective learning.Finally, the SUNY Learning Network encourages evaluation that ultimately adds to the general body of knowledge in best practices and directs improvement efforts within each course.

ABOUT THE DISCUSSANT
Cheryl Bullock is the Head of Measurement and Evaluation for the University of Illinois at Urbana-Champaign.She is also an Adjunct Assistant Professor in the College of Education where she teaches both statistics and Evaluation.Her involvement with the evaluation of on-line instruction began in 1995.Dr. Bullock specializes in the utilization of evaluation results, faculty evaluation, and issues of impact for those with disabilities.She has taught technology enriched courses both on and off-campus and has been the principal evaluator for several on-line educational programs.
Additionally, she has published, trained, and SLN has traditionally assisted campuses to conduct individual course evaluations.Additionally, the SLN office conducted two program-level student surveys and one faculty survey during the 1998-99 academic year.The goal of the student surveys was to gauge the level of student satisfaction with SLN, perceived learning with SLN, and what factors contributed to those results.The results of this survey are presented in this paper.

■
Refer to the Course Navigation Bars, Links, and Buttons -Course pages on the Web will have a navigation bar and links to help students navigate and interact with the pages of the course.Faculty should encourage students to use them by referring to them with instructional cues on their content pages.■ Make Information Accessible -If students have to travel too far to find what they need in their course by having to click too many successive documents or scrolling through very long documents, there is a risk of disorienting and discouraging them.The structure created by descriptively named and well-categorized documents/learning activities also makes an on-line course more accessible.■ Limit the Number of Hypertext Links per Page -If there are links to Web sites outside the course area, make sure students are aware they are leaving and know how to get back.Create links to other modules or to other areas within a module only if necessary.Because of the nature of hypertext it is important to make sure students understand where they are and where their documents are going when creating responses and interacting with your learning activities.Based on our recent surveys we know that faculty and students are very satisfied with the SLN program and with on-line teaching and learning in general.The two best indicators are that our SLN faculty and students persist in our program and are willing to recommend it to their colleagues and other students.Using our process, faculty development, and course design and delivery can be done on a large scale and with consistency in the quality of the teaching experience and environment developed for faculty, and the learning experience and environment designed for students.R E S U LT S O F T H E S P R I N G 1 9 9 9 ST U D E N T S AT I S FA C T I O N S U RV E Y Prescribed courses include two educational foundations coursesone in human learning and one in social thought, a course in instruction, a course in either technology or curriculum, and a course in educational research.Enrollment in the CDIT program is open to any student with an undergraduate degree whose GPA is 3.0 or better.On-line courses are open to all students in the CDIT program.Students may mix and match on-line and face-to-face courses to satisfy the degree requirements.The first on-line courses were offered in the fall 1997.The most technology literate were the first to jump at the opportunity.Carla Meskill created an on-line version of her Language, Literacy and Technology class.Joseph Bowman developed an on-line version of his Web-design courseComputing and Education II.Professor Swan put her introductory Computing and Education I course on-line.It has been offered continuously ever since.In spring 1998, Dr. Meskill and Professor Swan collaborated on the on-line version of Media in Teaching and Learning.In fall 1998, Systematic Design of Instruction (taught by Robert Bangert-Drowns) was added to satisfy the instruction requirement and Teaching in Context (taught by Audrey Champagne) to satisfy the social-thought requirement.In spring 1999, Ted Bredderman developed an on-line version of the required Research Seminar, and Professor Swan recreated her Mass Media and Education course on-line.This fall, the last two courses necessary to completing the degree were added.Meskill and Peter Shea are developing the introductory Educational Research in the SLN format, and Vic Kouba is mounting the last required course, Learning in the Academic Disciplines.
Figure 1 Comparison of Completion Rates for ETAP 426 and 526 (Computing in Education)Traditional and On-line Versions Figure 7 Comparison of Student Rankings for Traditional vs. On-line Versions of ETAP 426/526 provide faculty with abundant tips, recommendations, checklists, best practices, examples, observations, and guidelines on what we know works.Included are lists of things to think about when teaching in an on-line environment, tips for making Web-course materials clearer and more effective, do's for successful Web page presentation, and tips on getting off to a good start.We have compiled lists of tips for effective facilitation of class discussion, course management tips to keep students engaged, and how to deal with inactive students.
B. SLN Best Practices: Course StructureAn effective, well-designed, on-line course has ■ Comprehensive orientation and syllabus documentsexplicit expectations ■ Consistent and complete course chunks/module structure ■ Redundant and consistent instructional cues and detailed explanations ■ Meaningful and consistent course section and document titles to organize and convey information about the activities, content, and structure of course ■ A detailed orientation for each course module ■ Detailed instructions for each learning activity, i.e., expectation, timeframe, navigation, etc.
Create navigational instructions that explicitly tell students where to go next and what to do.Do not assume students will know where to go and what to do next, or for example, what is meant by "discussion."

Assessments for ETAP 522 (Mass Media and Education)
Eric E. Fredericksen is the Assistant Provost for Advanced Learning Technology and a senior manager in Advanced Learning & Information Services, part of the Office of the Provost in the State University of New York System Administration.He is a member of the AL&IS Leadership Group and provides leadership and direction for all of SUNY's system-wide programs focused on the innovative use of technology to support the teaching and learning environment.This includes the SUNY Learning Network-the SUNY System's premiere asynchronous learning program.Fredericksen is also the Co-Principal Investigator and Administrative Officer for a multi-year, multi-million dollar grant on Asynchronous Learning Networks from the Alfred P. Sloan Foundation.He is responsible for the fiscal management, strategic planning, policy development, faculty development, marketing & promotion, student support center, and technical infrastructure.Under his leadership the program has grown from 2 campuses offering 8 courses to 119 enrollments to 42 campuses offering 1000 courses to over 13,000 enrollments in just four years.Glencoe and The Multimedia Sampler for IBM, as well as three on-line courses which are being offered through the SUNY Learning Network, which she has taught for the past two years.She is a project director in the Technology and Literate Thinking Strand of the National Research Center on English Learning and Achievement (CELA) and is currently working on formative and usability evaluation for Project Links at RPI.Dr.Swan serves on the program committees for several local and international instructional technology conferences and is the Special Issues Editor for the Journal of Educational Computing Research.Peter Shea is the Course Management Systems Manager for the 64 campuses of the State University of New York.Formerly Lead Instructional Designer for the SUNY Learning Network (SLN) he has assisted in the design of more than 100 on-line courses and provides training in on-line pedagogy to new instructional designers for SLN.His research interests include student satisfaction and learning in internet-based distributed education and he has published a number of studies on this topic.He has also published and presented the results of research on computer assisted learning both nationally and internationally.Dr. Shea also teaches part-time in the department of Educational Theory and Practice at the University at Albany, State University of New York.