Requirements-Driven ALN Course Design , Development , Delivery & Evaluation

The best path to effective asynchronous learning network (ALN)-based course design, delivery and evaluation is through a requirements-driven methodology that recognizes the uniqueness of ALN-based learning. The methodology calls for the identification of purposeful and functional requirements, the identification of pre-course, early-course, mid-course and end-course activities, course “packaging” and prototyping, and “choreographed” delivery. It also calls for evaluation. The paper presents the methodology in the context of an actual course, a Systems Analysis & Design course offered asynchronously at Drexel University.


I. INTRODUCTION
This paper argues that the best path to an effective asynchronous learning network (ALN)-based course is through a requirements-driven discipline that recognizes the uniqueness of ALN-based delivery.The reason for the emphasis on requirements is simple: without reasonably accurate requirements, definitions and designs, we're likely to develop and deliver courses that might have elegant pedagogical features but little or no relationship to what students want or need.
The proposed discipline requires that we define and model requirements before we develop and deliver ALN-based courses.The assumption is that while conventional face-to-face (FTF) course design could well benefit from the discipline described here, given the nature of FTF delivery it's possible to significantly adapt to unanticipated events during a FTF course to maintain focus on the primary learning objectives.But on an ALN, that luxury is mitigated by asynchronous communications among the instructor(s) and students.Consequently, it's necessary to adopt more rigorous course requirements and design, development, delivery, and evaluation features that recognize the uniqueness of the ALN-based instruction.(See Fig. 1) This paper describes the discipline and illustrates its application to a Systems Analysis and Design course delivered at Drexel University.The overall methodology assumes that front-end requirements analysis is an absolute prerequisite to successful design and development.It also assumes that the only way to improve the design and development process of ALN courses is to engage in systematic empirical evaluation of student judgments about the courses as well as how the courses actually generate desired learning outcomes.
The generic discipline stresses the common denominators of ALN-based design, development, delivery and evaluation.Some of these include the need for ubiquitous access to the network, absolute predictability, the need for network pedagogy anchored in an instructor-set "network personality," and the need to understand requirements prior to course design, all as Figure 1 suggests.

II. REQUIREMENTS
Requirements are the essence of successful course design, development, delivery and evaluation.Our breakdown recognizes purposeful and functional requirements.

A. Purposeful Requirements
The most important question, even before the requirements of specific learning modules, skills, and steps that lead to "competencies" is: why do we want the course in the first place?What longer term learning, understanding, and problem-solving objectives will be served by the course?How does it intersect with existing and planned courses?Most importantly, how will the course fit within whole programs of learning, such as degree and certificate programs?
Without answers to all of these questions, we run the risk of developing and delivering courses that are internally consistent but "disembodied" from larger objectives.Purposeful requirements analysis seeks to identify the reasons why a course exists and justify the additional time and effort necessary to define precisely how it should achieve these objectives.If it's difficult or impossible to identify and validate purposeful requirements then the course should at least be re-examined.
In any case, it's important that the results of the purposeful requirements analysis be documented --as suggested in the following generic template (which is completed for the Systems Analysis & Design course).

Purposeful Objectives (for Systems Analysis & Design):
Abstraction Objectives: To abstract the purpose of technology --and specifically information systems --in the larger business context; the ability to understand the driving forces behind the need for cost-effective information systems ...

Synthesis Objectives:
To integrate and synthesize larger technology issues, constraints and opportunities within the information systems implementation process; to appreciate the costs & benefits of information systems design, development and maintenance; to appreciate the hardware/software/communications intersection ...

Course & Curriculum Linkage Objectives:
To place the course in the larger context of related courses and whole programs; to see the linkages across courses (and disciplines) To understand the systems analysis and design process in the larger systems engineering context; to understand alternative life cycle and system design and development process models; to understand the business value of information systems; to understand the processes by which cost-effective information systems are designed, developed, deployed and maintained ...

Overall Interaction Objectives:
To create an effective virtual learning environment; to demonstrate that topics-driven discussion "windows" and iterative model-supported design assignments support the learning process ...

B. Functional Requirements
Once purposeful requirements have been identified and validated, functional requirements can be detailed.Functional requirements include the specific things the instructors and students expect the course to do.If purposeful requirements represent the 30,000 foot view of the course, functional requirements are at ground zero: they represent the specific things instructors want the students to know and be able to practice by the end of the course.Functional requirements represent the primary source of data for course design (See Table 2).

Functional Objectives: (for Systems Analysis & Design):
Skills Objectives: To understand and be capable of implementing alternative life cycles, alternative requirements analysis and modeling methods, throwaway and evolutionary prototyping, demonstrating prototypes, evaluating prototypes, converting prototype specifications into software specifications, modeling software specifications (via alternative notation techniques and computer-aided software engineering [CASE] tools), managing the life cycle-driven process, and documenting the process ...

Overall Competency Objectives:
To understand how alternative life cycle steps can be combined, modified or eliminated to achieve cost-effective systems analysis and design; to prioritize requirements based on implementation and risk factors; to manage the whole process from ill-defined requirements to software specifications in the context of intersecting skills, disciplines and technologies ...

Communications Objectives:
To be able to communicate asynchronously individually and as a member of a project team .

III. COURSE DESIGN
The essence of the design process is the conversion of requirements into a suite of tasks and activities that together constitute the course.We use a simple template for converting requirements into a set of "pre-course," "early-course," "mid-course," and "end-course" activities and tasks, and ALN interaction, data and software requirements.
This step should ideally be performed by instructors who have taught the course a number of times.While the instructors should work with general course "architects" (those with "domainfree" instructional design experience), the linkages among purposeful and functional requirements, and course learning tasks, should be validated by those with the widest and deepest domain in experience actually teaching the material being converted.This assumption contradicts some instructional design methods and approaches that assume that the conversion of learning requirements into instructional tasks can be performed adequately by those who may never have taught the material to be converted.Our experience suggests otherwise.
We also believe that ALN delivery affects the requirements to tasks conversion process.For example, learning tasks must be informed and contextualized in a virtual learning environment.This means that, for example, the task of converting user requirements into exploratory prototypes requires that students perform the requirements to prototyping conversion process asynchronously and collaboratively via a tool in the network.The task is therefore different and more complex than the same task would be in a FTF learning environment.We developed a template for converting requirements into tasks that recognizes that tasks to be completed in an ALN are different than those completed FTF.The difference here is not just "operational," that is, because success in an ALN depends on one's ability to work the technology; differences can also be traced to, for example, developing an interactive prototype that will be reviewed by the instructor and all of the students in the ALN.
Table 3 presents a snippet of the whole template for the Systems Analysis & Design course illustrating the segmentation of course activity, tasks and requirements.
The methodology requires that we think about the details of instruction within an ALN, not just the details of course delivery where much of the control over form, content, pace and feedback occurs FTF.Syllabus contents must link to purposeful and functional requirements and the activities matrix developed during the course design phase.
These features are important because they address many of the unique requirements of ALNbased teaching, specifically clear statements about what is expected of the students, ways to communicate and access materials, and --especially --the detailed, predictable schedule of topic "windows." Following the development of the syllabus it's necessary to make sure that the whole course is "packaged" properly.This involves converting conventional materials (i.e., papers, textbooks, and presentations), putting them on the network, preparing to package the software applications necessary to support the interaction and communications processes, and making sure that everything works well together.Note that we assume that the preferred location of all course materials is the network and the students' PCs.All materials are thus available anytime and are local and network-accessible to all students and instructors.It's also possible, and often desirable, to add materials to an ongoing course: the quickest and easiest way to do so is to add the material to the network where it can be accessed and/or downloaded by students.
After packaging, it's necessary to "prototype" the course via a simulation of how the course should work.This process involves simulating: • Access to the materials • Asynchronous communications • Threaded discussions • Submission and critique of assignments Simulation is necessary due to the complexity of ALN-based instruction and our relative inexperience delivering ALN courses.
Figure 2 presents a simulation of how assignments, materials and the course "data bases" will interact within a Lotus Notes-based environment.This kind of simulation reduces uncertainty about reliability and robustness and also familiarizes those who will be required to support the course with the interaction process long before the course launches.In our case, the groupware in which we offer courses is based in Lotus Notes.We therefore develop simulations of how the course will actually "perform" from the Lotus Notes application that is the course itself.The simulation is an interactive "walkthrough," where the actual instructors and mock students interact on the network just as they will when the course goes "live." Prototyping via simulation is ultimately a risk analysis and risk management process: if no problems are encountered (a rare occurrence) then the course can go to "production," but when problems are discovered the prototype permits iteration over time to correct the problems.Unlike a FTF course where in-person apologies can be extended when things go wrong, ALN courses and students have limited capacities for forgiveness.

V. COURSE DELIVERY
The delivery process consists of the full-scale "live" implementation of a course choreography that can adapt to some significant number of unanticipated events.It is essential that courses be "choreographed" to stage and anticipate events during the course: if course design is the substance of a course then choreography is its style.
Choreography requires that we think about roles and adaptive procedures, and there are a variety of roles that the players (instructors, support staff and students) play during the course design, development, delivery and evaluation process.Here are the roles:

A. For Instructors
Initiator of discussions

VI. COURSE EVALUATION
Without an evaluation it's impossible to understand the immediate or longer-term effect the course is having on instructors, support staff or students.We have developed a questionnaire that measures student perceptions of how well (or poorly) the course was received.We have also developed a quality assurance process that compares student assignments generated in FTF courses and ones generated via ALN courses.The questionnaire measures perceptions across a variety of areas, some intentionally designed to compare FTF with network-based learning.A snapshot of responses --within the context of some access and interaction data --for the Systems Analysis & Design course appears in Table 4.
In addition to questionnaire data, we've developed a QA approach that compares and contrasts FTF and ALN assignments.We use the following measures: • Quality of requirements models We are convinced that discipline in the form of predictability, consistency and reusability will pay dividends as we extend the reach of ALN education and training.The requirements have already led to the deployment of a methodology for design, developing, delivering and evaluating ALN courses.Over time, additional requirements will be validated and addressed during the design, development, delivery and evaluation processes.Hopefully, our methodology will grow wider and deeper over time.

About the author
Stephen J.

Figure 1 :
Figure 1: The ALN Course Design, Development & Evaluation Process

• Pre-Course Activities • In-Course Activities • Post-Course Activities • Purposeful • Functional • Syllabus Development • "Packaging" • "Prototyping" Program Structure Required & Elective Courses Course Evaluation • Implementation • "Choreography" • Adaptation • Questionnaire Administration • Outcome Assessment
Table 3 suggests how a specific course breaks down, but the methodology itself is generic.
These roles suggest the kinds of behavior required of the players to make courses successful.They also suggest the kinds of simulations that can be run to prepare instructors for what may happen during an ALN course.

for 17 courses and 207 students:
Design courses taught by the same instructor), the ALN students always performed as well and often performed better than their conventional counterparts on all of the measures.Prototype quality was consistently higher in the ALN course than for the conventional course.had more access to the instructor than in "conventional" course delivery 80% felt that conventional courses were more boring than the ALN course 67% felt they had more communication with fellow students than in conventional courses 66% felt they learned more on the ALN-based course than they would have expected to learn in a conventional course 99% felt that seeing the ideas & assignments of others was useful.Table 4: Interaction & sample Subjective Evaluation Data • Computer-aided software engineering (CASE) and other tools can be integrated into a groupware environment Andriole is Chief Technology Officer and Senior Vice President for Technology Strategy at CIGNA Corporation, a global insurance and financial services organization.The work described in this paper is supported by The Alfred P. Sloan Foundation.The Sloan Program Officer is Dr. A. Frank Mayadas.The Sloan Foundation has an on-going research and development program in Asynchronous Learning Networks.We gratefully acknowledge the support of The Sloan Foundation without whose support this work would not have been possible.