Evaluation of Online Graduate Epidemiology Instruction and Student Outcomes

In the last two decades, online learning has transformed the field of higher education. Also during this time, institutions of higher education have seen increases in their adult learner populations. The flexibility and accessibility of an online education model is often particularly appealing to adult learners, who bring unique needs, expectations, and learning styles to their educational experiences. Using Kolb’s Learning Style Inventory and Knowles’ andragogy model as theoretical frameworks, this study evaluates an online graduate course in epidemiology in terms of the demographics, learning styles, satisfaction, and achievement of students. Comparing the online course to the same, land-based course that was offered concurrently, we found no differences between students’ learning styles, satisfaction, and overall achievement. However, students in the land-based class were more likely to be matriculated into a degree program (p<0.005), more likely to be full-time students (p<0.002), and more likely to work part-time or not at all (p<0.002). These findings provide evidence that student preferences for method of content delivery are correlated to lifestyle factors and not age, previous experiences, or learning styles.


Introduction
Online learning has become increasingly popular across all levels of education for numerous reasons.An online course typically provides students more flexibility and continuous access to materials than a land-based course, which meets at regularly scheduled times throughout a term.Online, or web-based courses can also accommodate a wider range of learning styles, allowing greater flexibility for both content delivery and learner engagement (Horton, 2000).Technological advances have increased the Internet's capability in terms of providing opportunities to practice problem-solving skills in a real-world context, linking to outside resources that increase the currency and relevancy of course materials, and participating in a dialogue among classmates (Fordis et al. 2005).
The flexibility of online learning is especially valuable to a student population that has grown in the last couple decades: adult learners (Moore & Kearsley, 2005).Adult learners are typically characterized by age and life experience.Almost a decade ago, the National Center for Educational Statistics (NCES), under the U.S. Department of Education, described adult learners as people at least 16 years old (2005); however, adult learners are more commonly identified as people between the ages of 25 and 50 (Kimmel, 2012;Moore & Kearsley, 2005;Osgood-Treston, 2001).As Knowles (1973Knowles ( , 1980Knowles ( , 1984) ) theorized in his andragogy model, adult learners seek information they feel is valuable to them and that can be applied within their lives.They also appreciate self-directed educational experiences, and they prefer to work independently towards their own defined educational and professional needs (Knowles, 1980(Knowles, , 1984;;Kimmel, 2012).This is particularly true of physician-scholars who further their education after completing their residency (Fordis et al. 2005).With busy clinical schedules and clearly defined training goals that must be applicable and relevant to their work, physician-scholars pursuing a postdoctoral degree in research methods often find online education more appealing than traditional face-to-face instruction.

Kolb's Model of Learning Styles
Theories about individualized learning styles arose in the 1970s, with numerous models and inventories emerging since then.Kolb's work on developing a Learning Style Inventory (1976) and Experiential Learning Theory (ELT) (1984) are some of the best-known, and his work has greatly influenced the field of education ever since (Pashler, McDaniel, Rohrer, & Bjork, 2008).Kolb's model breaks learning down into two primary dimensions: first, perception and second, processing.The model further divides these two dimensions into different approaches used by learners.Learners' perceptions, when they are first grasping an experience, typically fall into Concrete Experience or Abstract Conceptualization.Then, Kolb theorizes, after the initial perception, learners transform the experience either one of two ways: Reflective Observation or Active Experimentation (Kolb, 1984;Schaller, Allison-Bunnell, & Borun, 2005).Learners naturally tend to favor one perception approach and one processing technique over another.Based on individual preferences and tendencies, Kolb (1984) describes four different learning styles: converger, diverger, assimilator, and accommodator.Each learning style exhibits its own unique traits in a person.Effective teaching and learning utilizes all four methods at the same time.
The University of Cincinnati (UC) has offered graduate level training in epidemiology since the 1970s, but it piloted its first Introduction to Epidemiology course online in Winter 2011.This arrangement offered a rare opportunity to compare a land-based course with its online equivalent, in terms of student types, experiences, and student success.Incorporating aspects of andragogy and Kolb's Experiential Learning Theory, the purpose of this study was to use student satisfaction and achievement to evaluate the effectiveness of graduate-level training in an online Introduction to Epidemiology course compared to a land-based Introduction to Epidemiology course and examine the differences in demographics and learning styles between students who enrolled in the two courses.

Participants
Graduate students self-selected to enroll in UC's Introduction to Epidemiology landbased or online course (n=68).The courses were taught by the same instructor, covered the same material, and used the same assessments.The online course consisted mostly of asynchronous elearning modules-brief videos that included lecture clips, graphics, and low-stakes quiz questions to check for student knowledge throughout.Other online course activities such as discussion boards, synchronous chat sessions, and papers were used.Both exams (midterm and final) were administered online, and all online course elements were accessible through the Blackboard Learning Management System (LMS).The only differences between the two courses were method of content delivery and teaching assistants.The land-based group included 41 students at the time of pre-test data collection.During the first week of the course, 3 students moved from the land-based course to the online course, 1 student dropped the land-based course, and 1 student moved from the online course to the land-based course.Thus, the land-based students included 38 at the time of post-test data collection.The online course had an initial enrollment of 29 students and ended with 31 due to the shift in students from online to landbased (n=1) and vice-versa (n=3).Demographic information was collected for each student using a pre-test survey.This study was reviewed by the UC Institutional Review Board and determined to be exempt since no identifying information was collected.

Survey Instruments
To evaluate the differences between student backgrounds, experiences, and outcomes, we used a pre-/post-test design.The pre-test survey included questions regarding students' previous experience with computers and online courses, as well as demographic data such as age, ethnicity, race, work status, student status, matriculation status, and highest degree earned.We also ascertained self-reported learning styles using a question from Kolb's (1984) ELT model.Table 1 (previous page) lists the descriptions that students had to choose from on the pre-test instrument.The pre-test survey was administered to the land class (n=41) on the first day of class during the Winter 2011 quarter.The survey was made available online at the same time for the online course (n=29), with 2 follow-up reminder emails being distributed to non-respondents to access the link and complete the survey.
The post-test instrument was comprised of 38 questions for the land class and 50 questions for the online class.The majority of questions on the two post-test surveys asked students to rate their course experiences using a four-point Likert-type scale where 1 indicated strong disagreement, 2 indicated disagreement, 3 indicated agreement, and four indicated strong agreement.These Likert scale questions were broken into eight sections: Course Organization/Design, Course Content, Instruction, Interaction, Evaluation and Assessment, Technology (for the online course only), Satisfaction, and Overall Experience.In general, questions were phrased positively, so low scores indicate negative experiences and high scores indicate positive.For questions that were phrased negatively, we flipped the scores for analysis.Survey questions asked students to rate various aspects of the course and provide written feedback, with additional questions about technological aspects of the course for the online group.The online course had a slightly different version of the survey that included a section regarding the technological aspects of the course.The post-test was administered in hard copy to the students in the land-based course (n=38) and via email for students in the online course (n=31) during the last week of the quarter.All data for the online group were entered into a Research Electronic Data Capture (REDCap) database (http://www.project-redcap.org/),and data for the land-based group were managed using Microsoft Excel.

Data Analysis
We examined demographic differences between the courses using simple descriptive statistics.Student achievement between the land-based and online course were tested using independent sample t-tests for variance.This test was also applied to variables measured using interval level Likert scales.We used SAS for Windows, version 9.3 (SAS Institute, Cary, NC) to carry out all statistical analyses, and a p>.05 significance level was assumed.Because pre-and post-test surveys were de-identified through anonymous submission, we did not have the ability to match pre-test responses with post-test responses or student grades.

Results
Table 2 (previous page) represents the response rate for both sets of surveys.The pre-test response rates for the land course and the online course were 98% and 89%, respectively.Response rates were lower for the post-test survey in both courses.In the land course, 84% of students submitted a post-test survey and only 70% submitted one for the online course.

Characteristics of the Student Population (Table 3)
The pre-test survey instrument was comprised of 17 questions, including demographic information, experience with online courses, and a question related to Kolb's (1984) ELT.Results indicated similarities in the two groups with regard to age, gender, race and ethnicity, previous experience with online coursework, previous experience with Blackboard, and confidence using a computer (Table 3, previous page).Online students were significantly more likely to work full-time (p=0.002), and significantly more likely to be part-time students (p=0.002).The ratio of physicians in the online class (40%) was twice that in the regular class (20%).Also, students in the online course were significantly less likely to be matriculated in a degree program (p=0.005).

Learning Styles (Figure 1)
A majority of students in both courses (37% in the land-based and 35% in the online) identified themselves as divergers.Learning styles were essentially evenly distributed across the remaining three groups and students who didn't answer the question or said they didn't know.Both convergers and assimilators were the next most commonly chosen learning styles in both classes, with 22% of the land-based class and 19% of the online class identifying as convergers and 22% of the land-based class and 19% of the online class identifying as assimilators.Remaining students identified as accommodators (17% of the land-based class and 19% of the online class).Small numbers of students in each class did not answer the question or said they did not know (2% in the land-based class and 8% in the online class).Figure 1 provides the results on identified learning styles in percentages.

Comparison of Course Evaluations
Analysis of the post-test surveys revealed that students in both the land-based and online courses rated their course experiences similarly.We averaged section ratings for both courses and found that all categories except one had average scores within one-tenth of a point of each other (Table 4, next page).Similar or identical averages were seen for Course Organization/Design, Course Content, Instruction, Evaluation and Assessment, and Satisfaction.
The average rating for Overall Experience was equivalent in the two courses: 3.68 on a fourpoint scale.
Student responses to the Interaction section were the only ones that were disparate between online and land-based student groups beyond one-tenth of a point of each other.ratings were lower in the online class with regard to interacting with classmates and using each other as a resource.Although not statistically significant (p=0.43),students in the land-based course indicated more strongly (M=3.33) that they used other students as a resource, whereas students in the online version rated their interaction lower (M=3.15).When asked to rank who they go to first (self, other students, teaching assistant, instructor) when they needed help, the majority of students in both classes indicated high levels of independence, asserting that they went to others as a last resort (56% in the land-based class and 71% in the online class).However, differences emerge in the frequency and type of interactions between the two classes.In the online class, students reported interacting with other students primarily through the course's online discussion board, most often once a week, and 19% of respondents reported they never interacted with other students.In the land-based class, the frequency and variety of contact with other students was much stronger, with 59% of the class reporting they interacted with other students 1-3 times a week, 31% reporting four or more times a week, and only 9% reporting they never interacted with other students.The diversity of interactions in the land-based course was also much greater than in the online course, with land-based students using email, telephone, Facebook, or in-person meetings as methods of communicating and working with classmates.

Student Achievement (Table 5)
In addition to the pre-and post-test surveys, we compared composite scores of student achievement between the two courses.Scores included four homework assignments, a midterm, a term paper, a final exam, and an overall final grade.Students performed similarly in both classes, with the only significant differences arising in homework and final exam grades.Online students were significantly more likely to earn higher scores on homework assignments than their counterparts in the land-based course (p=0.04).However, students in the land-based course were significantly more likely to earn higher marks on the final exam (p=0.03).Differences in student performance were insignificant in all other grade categories, which include the midterm exam, term paper, and overall final grade.

Discussion
Multiple theorists have attempted to characterize the particular needs of adult learners.Knowles (1973Knowles ( , 1980Knowles ( , 1984) developed the andragogy model, which recognizes that adult learners are typically seeking information they feel is valuable to them, both intrinsically and extrinsically.They value knowledge that can be immediately applied in their lives in a relevant way, they view life experience as an important contributing factor to expertise, and they appreciate educational exercises that allow them to be independent and self-directing (1980,1984).Although often described as non-traditional students in higher education, adult learners are redefining the conventional image of a college student.College enrollment numbers among adult learners are expected to increase at almost twice the rate of traditional students (e.g., students under 25 years old) between 2009 and 2017 (National Center for Education Statistics, IES, 2010).
As more adults have chosen to pursue higher education degrees, researchers have worked to refine learning theories and make them more applicable to adults.Adult learners often come to the classroom, whether in-person or virtually, with very different expectations and life experiences.Many adults are accustomed to the standard method of instruction, with an instructor actively delivering content and directing discussions and students more passively listening and responding (Cercone, 2008;Tweedell, 2000).The differences between adult learners and their younger classmates can be especially manifested in an online course, where the method of delivery itself can be new and challenging to older students (Cercone, 2008).However, the desire to work independently and flexibly and to achieve specific, task-oriented goals could also work especially well with adult learners participating in online courses.
Adult learners balance numerous commitments outside of school and that their educational goals are based on well-defined needs (Kimmel, Gaylor, Grubbs, & Hayes, 2012;Osgood-Treston, 2001).Compared to their younger counterparts, adult learners are more selfmotivated to succeed in their studies, and more oriented towards task completion (Merriam & Caffarella, 1999).Our findings within a distance learning context support these assertions, as students in the online class were significantly more likely to be non-matriculated (p<0.005),more likely to be part-time students (p<0.002) and more likely to work full-time (p<0.002).In addition, the ratio of physicians in the online class (40%) was twice that in the regular class (20%).As the disparities between our two student groups indicate, non-traditional students who work full-time and are pursuing graduate coursework for their own knowledge, sometimes not even working towards a formal degree, are more likely to find an online course appealing.In many cases, adult learners have chosen to advance their education so that they can retain a competitive edge in the workplace, especially during times of economic recession (Kimmel, Gaylor, Grubbs, & Hayes, 2012).It is likely that our online adult learners who are working full time are taking classes to improve or change their work situations.In contrast, the students who are pursuing a degree full-time while working part-time or not at all are more likely to enroll in a land-based course, because classes are their main priority and their schedules give them the time and flexibility needed to attend regularly.In addition, students were almost evenly matched across groups in terms of the learning style with which they most identify, meaning students of one learning type or another were not more likely to choose one type of course (online or landbased) over another.These findings provide evidence that student preferences for method of delivery are correlated to lifestyle factors and not age, previous experience, or learning style.
Post-test survey results indicate that student experiences in both courses were very positive, regardless of delivery method.On a 4-point scale, students rated course organization, instruction, and overall satisfaction identically (M=3.8, M=3.68, and M=3.68, respectively).All other scores (course content, evaluation and assessment, and satisfaction) were within one tenth of a point of each other.
Students in the land-based class reported slightly higher levels of interaction with other students (M=3.33)than students in the online class (M=3.15).Students in the land-based class reported going to other students first for help nearly three times as much as the online class (5% in the online class and 13% in the land-based class).In the online class, almost a fifth of the class (19%) reported that they "never interacted with other students."Those who did primarily used the course's discussion board, on average once a week.In the land-based class, the frequency and variety of contact with other students was much greater.Only 9% reported "never interacted with others, and almost a third (31%) of the class reported "interacted with other students four or more times a week.The difference in the frequency of connection between students is even more powerful when considering the means of connection.Students in the land-based course were comfortable communicating with each other over email, telephone, Facebook, and in-person meetings (outside of class), indicating that the levels of support and intimacy between students were stronger.
Student achievement in both courses was similar for the midterm, a term paper, and an overall final grade.Online students earned higher scores on homework assignments than their counterparts in the land-based (p<0.04).This may be attributable to the differences in grading between teaching assistants.Students in the land-based course were significantly more likely to earn higher marks on the final exam (p<0.03),but not the final grade.

Limitations
Our study has several limitations.First, it focuses on a small sample of students in one specific graduate epidemiology course at one university, so the results cannot be generalized.Although we felt it was important to protect student anonymity in the interest of receiving honest feedback, because surveys were submitted anonymously, we were not able to match student pretest results with post-test results or student achievement scores.We were also unable to move pre-test surveys between groups when 3 students switched from the land-based class to the online class and 1 student switched from online to land-based.

Conclusion
Students who were not currently matriculated in a degree program, who were part-time students, and who worked full-time were significantly more likely to take the online version of this course.This indicates that lifestyle factors are the primary contributors to student course decisions, not demographics or learning styles.Student satisfaction was equally positive in both courses, although online students reported less interaction with their peers.Students performed equally well in both the land and online courses.Before generalizing these findings to other graduate level courses, studies with larger, more diverse samples and random group assignment are needed.The ability to match pre-and post-test survey results with individual grades would also provide insight into the connections between student experiences, learning styles, satisfaction, and achievement.

Table 1
-Thinkers and observers -Enjoy learning with lectures and papers

Table 2
Survey Response Rates