ROCHESTER INSTITUTE OF TECHNOLOGY: ANALYZING STUDENT SUCCESS

RIT Online Learning courses have an overall course completion rate of 94%. For lower-division courses, the rate is 92%, undergraduate 93%, and graduate 96%. In this case study. we will share additional measurements we have used to monitor student success and describe strategies we have used to promote online discussion as a key component of effective online courses. We will share results from a large survey of our online students that shows the most interactive courses receive the most positive responses from students. The demographics of our online students have shifted to where almost half our online students are campus-based now. We believe our historical emphasis on interaction in our online courses is serving these new online students equally well, which suggests an important opportunity exists to expand online discussion to more campus-based courses.


I. INTRODUCTION
RIT has an excellent program in online learning offering 51 online programs including 16 Master Degrees, 12 Graduate Certificates, 4 Bachelors of Science programs, and 17 Undergraduate Certificates.We have been involved with online learning for almost 30 years now and were pioneers in the development of effective asynchronous "anywhere, anytime" distance learning format courses.We have always believed our online programs were excellent and since 2002 when we launched the first campuswide course management system we have been transferring our lessons learned in online to the campus classes with Blended Learning initiatives and widespread promotion of our course management system.Our course management system is integrated with the Registrar system so all courses have access to a course presence online, and we now have faculty in 57% of all courses voluntarily using at least one of the online features.In this report we share our insights aspects of our online program that we believe contribute most to our students' success.

II. INSTITUTIONAL MISSION, CONTEXT AND DEMOGRAPHICS
RIT has been offering online courses for almost 30 years, beginning with use of local cable TV to broadcast lectures in 1979 then adding online discussion in 1987, degree completion programs in 1991, and course management systems (CMS) in 1996.

RIT Online Learning Mission Statement
Online Learning supports the academic mission of RIT by helping faculty find innovative and effective ways to use technology and by providing comprehensive support services for students in online courses and programs.
Drawing on expertise in both technology and pedagogy, Online Learning staff assists faculty with all aspects of course design, helps colleges and departments in online program development, examines the effect of technology on teaching and learning, and shares information across the institute about effective practices.
We believe instructional technology can contribute to RIT's academic excellence by fostering greater interaction and collaboration, encouraging active learning strategies, increasing student engagement with course materials, streamlining course administration, and developing students' skills for future learning and work.
RIT Online Learning is a service unit to the Academic Departments and as such is not directly involved with curriculum development or faculty hiring and evaluation.Our influence in course design is based on individual consultations with online faculty, offering a 3-week online course for new online faculty, and sharing our research and project results on effective practices through workshops, presentations, and online documentation.
The National Technical Institute for the Deaf (NTID) population has had a unique and substantial influence on the online program at RIT.There is a strong commitment from our institution to support deaf and hard-of-hearing (deaf/HH) students that goes beyond the ADA Compliance.A significant challenge for us has been to develop courses that are accessible to deaf/HH students.The cost and time constraints of captioning media production has perhaps disproportionately promoted asynchronous text based discussion and collaboration as a primary educational practice in our programs.Our research has shown that all students, but especially the deaf/HH, benefit from increases in this type of interaction.As a consequence, less than half of our courses have audio-based media associated with them.We believe this reflects an appropriate cost/benefit approach to our recommended course design activities.Although our online programs were originally restricted to students who could not otherwise come to campus, most of those restrictions were lifted after 1998 when part-time enrollments declined as a result of a local economic downturn.The excess capacity in online learning became more readily available to campus students and now 58% of our online students are campus based and represent 48% of all online enrollments.

III. STUDENT SUCCESS AS AN INSTITUTION-WIDE PRIORITY
Jeremy Haefner, Ph.D. Provost and Senior Vice President for Academic Affairs has redefined one of the Assistant Provosts positions as Assistant Provost for Student Success.This position is responsible for, among other responsibilities, researching and evaluating all retention efforts at the institute.Retention initiatives now have higher priority and visibility throughout the institute than ever before.
Online Learning, through our research in blended and fully online courses as well as information we glean from our course management system (CMS) actively contributes to the campus discussions on retention.In 2007/2008, while investigating the growing enrollments of campus students in online course we discovered a remarkable correlation in the campus-based subset of online students that now comprise 58% of our online students.We found a 24.5-point increase in graduation rates among the freshmen class of 2001 after 6 years if they took at least one online course during their stay at RIT.For the freshmen class of 2001, 756 students had 1 or more online courses and graduated at 79.2% rate, 1,289 of that same class had no online course and graduated at a 54.8% rate.Further investigation has confirmed this relationship with subsequent freshmen classes and year-to-year retention rates remain consistently better as well.
Although it may be a simple correlation that successful students are attracted to the novelty of an online course, it could also be that the added flexibility of online courses contributes to student success when they are as busy and overloaded as the faculty are.The 2008 NSSE report showed that online courses related positively to active and collaborative learning and suggested that online settings "may offer more opportunities for collaboration" than a campus course [2].We also have some evidence that taking a blended or fully online course can have a transformative effect on students.Further research on the effect of online courses with campus students is in progress.This short video-clip describes how one of our deaf campus students found great success in a blended course that used asynchronous text discussion to support team projects [3].
For further information about how our asynchronous text discussion strategies benefit deaf/HH students, please take a moment to read this short article by our colleague Michael Starenko [4].

IV. RIT'S DEFINITION OF "SUCCESS"
The Institutional Research group at RIT does not report on course completion rates per se, nor does the Office of the Registrar.They do report on DFW (Grade D, Grade F, or Withdrawal) as a measure of course failure and hence success.Online Learning reports on course completion rates for our online courses and the metric is based simply on the student's enrollment status at the end of the course.
Students that enrolled and did not withdraw are counted as completed although some of them will have received D and F grades.
For this report we have further analyzed our online learning enrollment data to provide 4 ways to view student success and completion rates.We have requested campus and online DWF rates from our Institutional Research group and will update this report if we are given permission to include those rates for publication.
Although it is not shown in this table, there has been an increase in course withdrawals across all courses since a 2007 change in the policy extended the deadline for withdrawal from week 6 to week 8 of our 10week quarter.In Online Learning the number of withdrawals jumped from 5.2% to 8.6% of enrollments in the first year of the new policy.This reinforces the points made in the 2008 NSSE report about the risks of focusing on institutional differences.A simple policy change can have dramatic effects on the statistics while the underlying courses are unchanged.

A. Learning Effectiveness
RIT has been part of the Sloan Consortium since its inception, and Online Learning staff have participated actively in many of their activities.Our staff have attended and presented at nearly every Sloan ALN and Online Learning conference.RIT Online Learning has always embraced asynchronous learning and asynchronous text discussion as primary strategies for providing the anywhere, anytime learning opportunities we believe are important to the majority of our online students.We do have programs that are now using synchronous delivery, primarily in a hybrid format of live lecture on campus that is webcast to distant online students, but they do not represent a significant number of enrollments yet.At our core, we have promoted asynchronous text discussion as our primary strategy for engaging students in the classroom, promoting reflective thinking, supporting collaborative learning projects, providing access to the instructor and other students.Our emphasis on interaction relates directly to the NSSE benchmark on Active and Collaborative Learning, Student-Faculty Interaction, and Enriching Educational Experiences.Our commitment to providing comparable services to online learners that their campus counterparts receive relates well to the NSSE benchmark on Supportive Campus Environment.Following are details on how our approach relates to these benchmarks.
RIT Online Learning has embraced online interaction as a cornerstone of course quality since the VAX Notes bulletin board system became available to our faculty in 1987.Fortunately, the early pioneering faculty in online learning instinctively understood the potential for greater interaction in their online courses and pursued the development of new online discussion strategies and online team projects.Their best practices informed us during the rapid development of new courses and programs throughout the 1990's and especially when we migrated to course management systems with even more powerful and easy to use asynchronous discussion tools.
Our second course management system, Prometheus, supported data mining information on the amount of page hits occurring in each course and we derived a metric from that to identify the "most active" courses in our suite of online courses.Those faculty associated with the most active courses were already recognized by our instructional designers as exemplars and pioneers in our program so we instituted a low-stakes faculty recognition program called the "Heavy Hitters" to announce the top faculty each quarter.Our current course management system, Desire2Learn, further allowed us to data mine information about our courses and the "Heavy Hitters" metric now includes all messages posted in the online discussion area, news announcements from the instructor, comments left in the grade book and drop box by the instructor, as well as calendar announcements by the instructor.The largest component of the metric is the online discussion.The metric doesn't include any data about synchronous interaction or email and that is a flaw we would like to address but capturing and measuring that additional data has been problematic.Each quarter we rank all courses at RIT with this metric and post the Top 50 faculty in a News announcement on the course management system and in our Newsletters and web site [5].
Although there is a concern about the "panopticon" effect of data mining and reporting this information at the course level, focusing on only the "Top 50" has proven to be surprisingly successful.Faculty report that they appreciate knowing where they stand on this metric and that their efforts are recognized.When we announce the "Heavy Hitters" each quarter we also take great care to point out that courses on the list are not necessarily excellent courses, but we do notice that most of the names are faculty we have trained, attend our workshops, and are considered among our best faculty by our instructional designers.A few of our presumed excellent faculty have criticized the metric for not recognizing courses that may be excellent but have very little quantity of interaction.One example of these non-listed courses is a very large enrollment online course delivered to Kosovo where our traditional strategies for engaging students in online interaction and working in teams online would be difficult to implement due to cultural differences.Nevertheless, the instructor is well prepared, the course is well designed, but the students spend most of their time interacting with the content individually.
As suggested by the 2008 NSSE report, in 2006/2007 we used the Heavy Hitters metric to investigate the variation within our own online courses.We used the metric to rank order all our online courses and then administered 4 separate but identical surveys to students in each quartile of the ranked courses.The survey included questions about perceptions of learning, perceptions of interaction, as compared to a typical campus course.We were pleasantly surprised to see a strong relationship showing higher levels of interaction related positively to higher levels of positive responses to the survey questions.In 2007/2008 we expanded the project to fall, winter, and spring quarters and made some adjustments in the questions.This data has not been published but has been presented at several conferences.
For fall, winter, and spring quarter of the academic year 2007/2008 the 450 course offerings in online learning were ranked by the interaction metric and then separated into 4 quartiles.This information was data mined from the CMS at mid-quarter so there would be sufficient time to administer a student survey by the end of the quarter.From this data we summarized the following statistics: Obviously the top quartile, Q1, had more frequent "postings" than the other quartiles and the quantity amounted to slightly over 1 per day per person.The top quartile was also much more likely to use the "Groups" feature for small group online discussion and team projects in our CMS and used more of the features built into the course management system.
We then administered 4 versions of the same survey to students in each quartile and cross-tabulated the results to see how student perceptions varied by quartile and volume of interaction.We received a total of 1,237 student responses in the three quarterly surveys.The results were consistent with the previous year's first exploratory survey.Here are several of the questions with the percent of Agree and Strongly Agree responses: The fact that some students responded favorably in even the least interactive courses suggests that in some cases the course management system metric misses other important types of interaction such as email, phone, and web conferencing.And the fact that in the most interactive courses there were some students who did not respond favorably suggests that quantity and type of interaction in some courses may interfere with learning.
This data is still being analyzed.A small proportion of these students are deaf/HH and we are working with NTID researchers to better understand the unique perspectives of these students to interaction in online courses.Although the numbers are small, the deaf/HH and students responses suggest they are even more favorable to higher levels of interaction than hearing students.The asynchronous text discussion levels their playing field and lets them participate more fully than typical interpreter supported classroom sessions.
In order to continuously improve our courses for learner effectiveness, the Online Learning Instructional Design/Technology staff regularly review and critique the online courses.This is also an important part of our Annual Awards process when the best courses are selected for recognition and the Online Learning staff assists the selection committees by screening and then touring the top courses.These investigations have led us to create a Model Course that faculty and their support staff can copy as a starting point for a well-designed online course.Boilerplate and placeholders guide new instructors through a series of effective practices as they prepare their courses.This project was also documented on the vendor's site and highlighted as an effective practice in their newsletter to all users.

B. Scale
There is a range of course designs across the online programs we have helped develop and support.The most expensive options involve any form of audio since that must be transcribed or captioned in addition to any of the other production costs.Since many of our courses are offered just once a year and the average class size is 14 in spring quarter 2008/2009, the course may need to be updated after 3 or 4 years when less than 100 students actually used the media.So we are cautious about production costs and wherever possible try to get faculty to rethink their online courses to include more reflective discussion and more team and group projects.One of our sayings is "The content is in the bookstore, the action is in the interaction."We have been very successful in promoting this format, nevertheless many faculty and possibly certain courses do require videotaped or webcast lectures and we accommodate them.
We developed an Essentials of Online Teaching course for new online faculty in anticipation of a surge in new faculty in one of our colleges.The surge did not materialize, but we have run the course 5 times now and about 50 faculty have experienced it so far.In the event there is a resurgence of interest in distance learning we can accommodate considerably more courses and faculty if the media demand is managed.
Our biggest growth potential, frankly, is in blended and web enhanced courses.About 10% of our campus-based courses are using online discussion and we believe every course could benefit from some outside of class interaction, especially in support of team and group work.Our experienced online faculty tend to incorporate these features more naturally into their campus courses.But there is a large majority of faculty who do not attend our workshops or other events that promote effective online practices and the challenge is how to engage those faculty more effectively.Right now we believe that anything that helps faculty be more productive (save time) is the best hook to grow interest in online learning.Unfortunately, the learning curve is just large enough to inhibit many faculty from incorporating online discussion and other online components to their campus courses.

C. Access
RIT Online Learning offers over 160 fully online courses each quarter in support of the over 50 graduate and undergraduate degrees and certificates we offer.In addition to the "traditional" fully online courses we offer, we have programs at international sites in Kosovo, Croatia, Santo Domingo, and Dubai that rely heavily on our online learning services for infrastructure support.
Some Online Learning courses at RIT have restrictions to students matriculated into an online program to ensure "seats" are available to those who need the online format most.This restriction, if it is used, is lifted just before the beginning of classes so unfilled "seats" are made available to campus students.We had a 13% growth in campus-based enrollments and unique students taking online courses in 2007/2008 over the previous year.
Once students are enrolled in an online course they are automatically enrolled in the Student Community course shell located in our course management system.This Student Community provides all the necessary information they need to prepare for and begin their online learning experience.This approach provides our online students with one place to access their courses and all the relevant support services.The approach of using the course management system as a vector for student orientation has proven successful for us and has been highlighted and presented at our vendor's annual users meeting.
Similarly, the Online Writing Lab is located within our course management system and provides convenient access for students to submit drafts of papers for review and tutoring.On campus students have a walk in center for this support but distant students cannot take advantage of that campus service.The same tutors that support the on-campus lab provide online support.More information about this service is available in an article our colleague Marybeth Koon had published in our vendor's newsletter where it was recently highlighted as an effective practice [6].
The RIT Library has partnered with Online Learning since the beginning to insure distant students could receive the same benefits as their campus counterparts.The library's online databases and electronic reserves have provided convenient access to students anywhere, anytime.Our interlibrary loan service is free to online students.One of the reference librarians is dedicated to supporting online learning faculty and students, as well as providing great support to Online Learning staff.The library technical staff collaborated with our technical staff to develop a link for the top of every course shell in the CMS that directly connects students with specific information about course reserves, discipline based online databases, and contact information or the reference librarian assigned to the discipline.
All media created for Online Learning is transcribed and captioned and that has proven to benefit all students.Hearing and deaf/HH students can use the searchable transcript feature built into our media to use keywords to jump through a lecture to specific points for review.Information and files to support this approach are available for free from our web site [7].
We promote collaborative learning as much or more than any other academic support unit and recognize it is sometimes hard for faculty manage the teams and groups effectively.As part of a senior student software engineering team project we developed an online peer evaluation system that is integrated into our course management system.The convenience of the tool permits faculty to use it for formative evaluations in the middle of a team project while they can still intervene and possibly correct any group dysfunction.It can also be used as a summative evaluation at the end of the project and since it uses the group structures the faculty have already built into the course management system for their class, it is much easier to administer than any other form of feedback [8].
The old style paper course evaluations were not effective in online learning courses.The response rates were typically less than 30% compared to almost 100% in campus classrooms.We developed our own Online Course Evaluation system so online students could more easily complete a course evaluation.The current overall response rate is 52%, but much higher in cases where the instructor promotes the completion of evaluations.The biggest advantage of online evaluations now is the quick summaries and turnaround back to the instructor.The old style paper course evaluations took weeks to summarize and return to faculty and department chairs.The tool has become so useful that it is now used by many oncampus departments and courses [9].

D. Faculty Satisfaction
The RIT Online Learning faculty still includes pioneers from the early days of the VAX Notes bulletin board system used for course interaction.These faculty and some of the newer faculty have been disproportionately more likely to be one of the campus outstanding teacher award winners.In order to recognize these achievements, all faculty who have won a campus or online outstanding teaching award have a banner placed in their course by our staff so their students are aware of the opportunity they have to participate in an outstanding class.The tone set by these outstanding faculty pervades through our workshops and web resources as we glean effective practices and disseminate them across campus.
In the spring of 2007 we administered a faculty satisfaction survey to all online faculty in order to better understand their motivations for teaching online.Peter Shea, from the University of Albany, State University of New York, helped with the study that was based on research he had done with several other schools that was funded by the Alfred P. Sloan Foundation.There were 116 responses from 300 faculty and the general results suggested strong satisfaction about teaching online.See figure 5 for some of the response summaries.A summary of the final report is available on our website.[11] Faculty Satisfaction Survey: RIT Faculty

E. Student Satisfaction
Although the Online Learning department administers the Online Course Evaluation system, we do not have access to the student responses or aggregated summaries.Our perceptions of student satisfaction are informed by anecdotal comments we read in the Student Community areas, phone and email comments to our Support Services desk, and regular surveys of student satisfaction with our services (not directly about specific course experiences).Based on the anecdotal evidence we have long believed that our best courses are better than campus courses, and that most students reporting unsatisfactory course experiences would likely have unsatisfactory experiences in the same course with the same instructor on campus.
The worst student experiences relate to instructors who have minimal presence in the online classroom, followed by slow turnaround on questions and assignments.There is no mystery why teacher presence is an important component of online courses; the mystery is why some faculty are withdrawn online.
Our most recent research project should provide even more evidence how important teacher presence is for online courses.In spring quarter 2008/2009 we collaborated with Dr. Peter Shea at SUNY Albany to administer the Community of Inquiry survey to all our online learning students.The Community of Inquiry model provides a framework for analysis of online interaction as a key component of the educational experience.We administered the survey so that we can compare results between courses based on their quantitative levels of interaction.The survey is being analyzed now and the results will be available in fall 2009 [12].
There have also been in the past occasional complaints about courses that were not well organized, but the Model Course, Essentials Course, and other interventions by our instructional designers has reduced those complaints.The only other clue we have to student satisfaction with the course experience is the dramatic growth in demand from campus students for the fully online format.
With respect to the services we provide our students, we have regularly administered student satisfaction surveys since the early 1990's.These surveys were originally on paper so in 1996 we developed a web based survey system that greatly improved participation and provided very quick turnaround for analysis of trends and issues that could influence our operations the very next quarter.We now administer an annual student survey for all online students and a quarterly new student survey to capture the experiences of our first-time students.These surveys have consistently shown a high level of satisfaction with our services.Below are some questions from the last two surveys that summarize the overall student perception of our online learning programs.The improvements from 2006/07 to 2007/08 can be attributed to the change in demographics (more campus based students), the introduction of the Essentials of Online Teaching for new online faculty, expanded use of the Model Course as a faculty time-saver that has effective practices embedded, and possibly to the continual emphasis on interaction through the Heavy Hitter low-stakes faculty recognition program.However, the low response rate (4%) and N of 138 in 2007/08 survey reduces our confidence in the survey results.

VI. CONCLUSION
RIT Online Learning tries to provide everything an online student needs to be successful.We have adapted to changing demographics and fast changing technology to give our students the best experiences we have control over.We look forward to even better tools to support online learning, but won't lose site of the important role simple asynchronous text messages plays in promoting and supporting student interactions which we believe are at the core of effective instruction.

2007/08 Online Course Interaction Survey
Dr. Richard Fasse collaborates on new technology initiatives, researches the role of interaction in online courses, and strives to continuously improve the quality in the 150+ online courses offered each quarter.His education includes a BS in Business and Computer Science (University of Kansas), an MBA in Information Systems (Penn State), and an Ed.D. in Curriculum and Instruction (University of Rochester).JoeannHumbert is the Director of Online Learning at the Rochester Institute of Technology.She manages all facets of Online Learning: faculty development, course design and development, online student support and the investigation and application of emerging technologies for teaching.In 2008 her Online Learning group received the New Media Consortium's "Center of Excellence Award" recognizing demonstrated excellence and outstanding achievement in the application of technology to learning.In 2009, Joeann received the "Noflett Williams Service to the Field Award" from the National University Telecommunications Network (NUTN) in recognition of her service to the field.Joeann is a member of the Institute's Provost's Learning Innovations Grant Committee.Joeann holds a B.A. in English Literature from Villa Maria College and an M.S. in Instructional Technology from Rochester Institute of Technology.Raychel Rappold is Online Learning's Information Control Analyst, at Rochester Institute of Technology.Raychel has over ten years of experience at Online Learning with student and faculty services.In her current position she oversees the collection and analysis of Online Learning data.She produces reports that assist management with diverse issues such as top-level strategy, planning, forecasting, and resource allocation.Raychel also creates and maintains print and web communications.Raychel graduated from RIT's College of Applied Science & Technology with a B.S., comprised of a dual concentration in Business Management and Social Welfare, and a minor in Psychology.