Students’ Perceptions of Quality Across Four Course Development Models

Four course development models were compared over a 3-year period, based upon student perceptions of the integration of the Quality Matters (QM) Standards, course structure, and quality. Points of comparison included (a) faculty training, (b) instructional designer supported, (c) additional QM training course, and (d) no training or support used. Students were randomly selected from online courses from each of the categories to receive a survey that measured their perceptions about the courses. Students were asked about the design of the courses in terms of the integration of the QM Standards, structure, and quality. Significant results were found across all eight standards, course structure, and quality for the instructional designer supported course model as compared with the other course design models.


Students' Perceptions of Quality Across Four Course Development Models
With projected declines in freshman enrollment in postsecondary institutions (Selingo, 2012), enrollment services are exploring various approaches to attract nontraditional students.Beyond enrollment figures, the traditional profile of freshman students is also changing as tuition costs increase.To reduce debt after graduation, many students now choose to work full-or parttime while they attend classes.To respond to the needs of these students, 70.7% of postsecondary institutions provide distance-learning courses as an option (Allen & Seaman, 2013).Distancelearning classes enable flexible scheduling for students as they work or care for children.In an evaluation of the impact of distance learning on student success, one university found that the higher the percentage of distance-learning courses, the shorter the time to graduation (Affordability Workgroup 2025Strategic Plan for Online Education, 2016).
Adapting courses from traditional to online formats requires faculty members to shift their pedagogical beliefs, improve technical skills, and adopt different classroom management skills (Allen & Seaman, 2013;González-Sanmamed, Muñoz-Carril, & Sangrà, 2014;Neban, 2014).Faculty members continue to have a negative perception of the quality of online courses, primarily based on the belief that their instructional content is incompatible with online instruction (Neban, 2014).Teaching styles, often developed very early, are difficult to change mid-career.Teaching online challenges faculty members to learn new technology and adjust pedagogy, creating a degree of discomfort in converting courses to online formats (Osika, Johnson, & Buteau, 2009).An additional challenge is the belief that online courses are impersonal and that faculty members will miss student-teacher interactions (Neban, 2014;Osika et al., 2009).
Training faculty members to either design online courses or to understand the online course development process often includes professional development or a collaborative course design process.In a survey of 48 institutions with membership or representation in either the Sloan Consortium or the Western Interstate Commission for Higher Education, 90% of the institutions used a variety of professional development options, including 2-to 5-hour workshops, one-on-one trainings, hands-on trainings, online courses, or one-time training to support faculty members (Meyer & Murrell, 2014).A community of practice was used by 57% of those institutions.Of the course training options, faculty members placed higher value on pedagogical training than on technological training, and webinars were valued the least by faculty members (Meyer & Murrell, 2014).Another approach to teaching pedagogy is through collaborative partnerships with instructional designers and faculty.Within these partnerships, the instructional designer serves multiple roles as the editor and the reviewer of work, the project manager, a coach, multimedia and graphic designer, and help desk functions for students and faculty (Hawkes & Coldeway, 2002).Given faculty's continual concerns and the desire to improve pedagogy in their online classes, this study explores the integration of best practices by faculty into online courses based upon the course development models used.

Review of Related Literature
With the expansion of distance-learning, best practices have emerged.Using the best practices literature, a rubric was developed by MarylandOnline Inc. as a tool to evaluate the quality of online courses.MarylandOnline provided training in the implementation of the rubric guidelines and for course evaluators (MarylandOnline, 2017).Higher education institutions used the rubric to develop training courses, resulting in improved faculty confidence in the use of technology (Hixon, Buckenmayer, Barczyk, Feldman, & Zomoiski, 2011).The institution where this study was conducted adopted the QM Rubric as a guide in the development of online courses.

Quality Matters Quality Assurance Framework
Quality Matters (QM), a nonprofit organization that offers a subscription-based service developed by MarylandOnline Inc., constructed the QM Rubric as a guide for the development of high-quality online courses.The QM Rubric was created by the University of Maryland as a part of a federal grant-funded project (MarylandOnline, Inc., 2014;Shattuck, 2007;Shattuck, 2012).Perhaps one of the most important components of the QM quality assurance model is that it included a faculty-centered, peer-review process through the QM Rubric.The 2014 Quality Matters Rubric 5 th ed.(MarylandOnline, Inc., 2014) has eight general standards (Course Overview, Learning Objectives, Assessment, Instructional Materials, Learner Interaction, Course Technology, Learner Support, and Accessibility).The reviewers received additional guidance through the 43 additional criteria embedded into the eight standards.This rubric was designed to promote continuous course improvement over time by faculty and instructional designers.Below is a description of those standards.
Standard 1: Course overview and introduction.The creators of the rubric included criteria in this section to address the introduction to a course.A "start here" section was encouraged in course development (Lohr, 1998) because it provided an easily accessible course overview complete with schedules and technical requirements.
Standard 2: Learning objectives.Educational research and decades of instructional design practice have led designers and developers to provide learning objectives within each lesson.For this reason, learning objectives were included as part of the QM Rubric.The objectives act as an advance organizer for learners, providing some level of scaffolding for the current lesson.For example, advance organizers allow learners to tie their previous knowledge to new information (Clark & Mayer, 2003).
Standard 3: Assessment.Assessments, which are broad and varied, were included into the QM Rubric to provide an indication of student learning in the course.The guidance in this standard is used by the instructional designer to provide constructive feedback that aids in the design of appropriate assessments and presentation techniques to encourage learning (Lee, Srinivasan, Trail, Lewis, & Lopez, 2011).
Standard 4: Instructional materials.A systematic application of design principles is used to create learning experiences that promote understanding and maximize the strengths of the students in the context of the instruction.The systematic process aligns the assessments to the instructional materials and the learning objectives.Activities are designed to ensure skill development (Dick, Carey, & Carey, 2009;Gagné & Briggs, 1974).Currency of the materials is included in this standard to ensure that a course, developed from 15-to 20-year-old course notes, has progressed and included recent discoveries in the content area.
Standard 5: Learner interactions.The developers of the QM Rubric felt learner interactions were important to reduce student isolation in the online course (Moore, 1989;Moore & Kearsley, 2011;Zhao, Lei, Lai, & Tan, 2005).The learner interactions required in a high-quality course promotes a feeling of belonging to a community of learners who support and motivate each other.
Standard 6: Course technology.Instructional technologies may reduce the transactional distance between the instructor and the students created in distance-learning courses due to the delivery mode.Instructional technology functions best in a transparent and seamless way (Saba & Shearer, 1994).Email is often considered impersonal by students (Biesenbach-Lucas, 2007), whereas a synchronous discussion can feel friendlier and can allows for quick communication and feedback.
Standard 7: Learner support.The QM Rubric developers insisted that student support services be available from within the course so that students can find help (e.g., technical or financial aid support) when necessary.Universities that provide online courses often have learner support centers and services for learners at a distance (Brindley, 2014).
Standard 8: Accessibility.All learners must have access to the course materials to learn, including those individuals with disabilities.The Americans with Disabilities Act, or Section 508, Compliance Regulations state that courses should be accessible by individuals with a variety of disabilities.Universal design also proposed that disabilities occur along a continuum, and any efforts developed to support those with disabilities assist all students to achieve learning outcomes (Rose, Meyer & Hitchcock, 2005;Silver, Bourke, & Strehorn, 1998).
While the QM Standards may seem extensive and a good scaffolding tool to develop highquality courses, the guidelines can be overwhelming to faculty as they begin developing online courses (Chao, Saj, & Hamilton, 2010).Within the rubric criteria are the underlying principles to the design of online courses, but faculty often need pedagogical assistance in the selection and deployment of instructional strategies and assessments.The rubric provides a guide in the development of the course and a place to build the relationship with the faculty.The designer can alleviate concerns the faculty have about the quality of the course (Kumar & Geraci, 2012).Based upon the relationship, the instructional designer guides the faculty into adopting best practices in the rubric.With the instructional designer in support in the design of the courses, student reported better feedback and better instructional practices (Brown, Myers, & Roy, 2003).

Course Structure and Course Quality
Although, the QM Rubric provides good measurements of quality within courses, the rubric does not offer a measurement of the course structure and overall quality of the courses.The QM Rubric focuses upon the lesson structure and the alignment between the learning objectives, instructional strategies, and the assessments.Course structure is how the overall course is organized and whether that organization supports learning.However, quality consists of more than the structure and the lessons.Quality from students' perceptive is about the implementation of the distance-learning course.
Much has been written about course structure.Curriculum, as a field of study, began with the development of the course syllabus, course objectives, and assignments (Tyler, 1949).Much has changed since 1949.Today, alignment of performance objectives connects the course content to the course objectives and to the assessment.The performance objectives determine the types of assessments.The assessments drive the lesson material (Dick et al., 2009;Gagné & Briggs, 1974).In the 1960s, Gagné (1965) proposed conditions of learning, and that lessons should be structured or organized to promote learning.One way to organize the instruction was to start with prior learning and gradually increasing the complexity of the learning tasks (Gagné, 1968;Gagné & Brown, 1961).Later, a modular design was developed for the online learning environments with course material organized around topics (Gagné & Brown, 1961;Simonson, Smaldino, Albright, & Zvacek, 2003).Organization evolved into weekly time-based modules or weekly lessons which contributed to improved student performance (Tenam-Zemach & Lewis, 2014).
Early studies identified overall course quality as easy to rate but difficult to define (Johnson, Aragon, Shaik, & Palma-Rivas, 2000).However, students seem to define quality based on their satisfaction (Moore & Kearsley, 1996;Sun, Tsai, Finger, Chen, & Yeh, 2008).Students appeared to view the course materials and instructor performance as one.Quality in a course goes beyond connecting objectives with assessments and instructional materials.Courses designed on objectives can promote the use of limited designs with video recording and multiple-choice tests (Lowenthal & Hodges, 2015).Structuring of the course as isolated objectives can create a disconnected curriculum design and a focus on program competencies (Krusen, 2015).

Models for Course Development
The development of high-quality courses can be a complex process.Faculty often do not think through the delivery of course material in their face-to-face courses.In face-to-face courses, the faculty often rely upon teaching styles that were developed early in their careers (Osika et al., 2009).Faculty also need to re-envision their courses because they often do not view the content taught as adaptable to online (Neban, 2014).Models for development of courses address these concerns as faculty adopt online as a delivery model for the courses they teach.Often institutions that provides online courses have an approach to assist faculty in the transition to online.Those approaches can be grouped into four different models described below.
Training/professional development model.Training courses which teach faculty how to design online course materials have mixed results.The training course must be carefully designed with the faculty members' expectations in mind.Faculty often have high expectations of the course trainer because the trainer exemplifies the same skills the instructors themselves practice in their classrooms (Terantino & Agbehonou, 2012).For the best results, the instructors of the training course must be prepared, the technology skills should be limited to a few necessary skills, and the guest speakers should be selected to represent different vantage points (Terantino & Agbehonou, 2012).Frequently, training courses require additional time, leading to low participation rates in the course.To mitigate the lack of participation, the extra time commitment needs to be communicated to the faculty members so that they understand the extent of the work required to participate in the course (Cho & Rathbun, 2013).Even with training, faculty members often report lack of confidence in their use of the online technology (Kerrick, Miller, & Ziegler, 2015).
To address concerns about the time commitment required as faculty participate in discussions and complete instructional activities, the training is often presented online or in a hybrid format.Self-paced online courses provide the flexibility to match faculty members' variable schedules and provide the instruction accommodating a range of teaching styles and levels of technology expertise (Rhode & Krishnamurthi, 2016).The implementation of the faculty development for online courses takes extra time compared with face-to-face training.However, once developed, the implementation is frequently scalable (Rhode & Krishnamurthi, 2016).
Online training also has the added advantage of creating learning experiences for faculty members.For example, as faculty interact with the online training material, they experience the challenges as an online student.Through their involvement in the online training, the faculty develop a deep appreciation for precise instructions and immediate feedback.The experience gives the faculty an idea about the amount of effort and time required to successfully complete assignments.The experience changes faculty attitudes in their beliefs about the ability of students to complete assignments and the amount of interaction that occurs in an online course (Gold, 2001).
Instructional designer-supported model.In this model, instructional designers often collaboratively build courses with a faculty member (Hawkes & Coldeway, 2002).Designing online courses is a complex process requiring clear definition of the many tasks involved.Those definitions can be mapped to the roles of the instructional designer and faculty member.The result is creation of courses consistent in the development processes, and reinforcement of the preexisting competencies of the faculty member (Chao et al., 2010).The use of a rubric for the design of a course promote the development of a relationship between the faculty member and the instructional designer.That relationship enables the instructional designer to assuage faculty concerns about the quality of the course under development (Kumar & Geraci, 2012).
The collaborative partnership of the instructional designer and the subject matter expert is ideal.The courses built through the partnership promote interaction between students and faculty, provide many opportunities for students to share ideas, and include multiple active learning activities.Students also reported higher probability of receiving prompt feedback in courses designed with instructional designer support than those designed without a designer (Brown et al., 2003).
The instructional designer-supported model functions better when the institution has course development guidelines.Guidelines, such as those provided by the Quality Matters Rubric, clarify what is required for a successful course delivery, creates consistency across the courses, and promotes a collaborative working relationship between an instructional designer and the faculty member (Chao et al., 2010).
Lone ranger model.In this model, the faculty members designed their courses independently without training or instructional design support (Bates, 2000).The model is frequently used to encourage adoption of new technology to design or deliver instruction.Through a series of small grants, faculty can experiment with the technology.The experiments resulted in strategies and gradual adoption of the new technology (Bates, 2000).
The lone ranger model does have its drawbacks.The laissez-faire approach to development and the experimental origins of the resulting courses caused variability to emerge (Bates, 2000).The approach is an expensive way to develop courses; while, impacting a small number of faculty.The ideas developed, and skills learned often do not transfer to another faculty member (Bates, 2000).Adoption can be slow because faculty serve many roles, functioning as graphic artist, web designer, and instructional designer (Puzziferro & Shelton, 2008).
Combination training and instructional designer-supported model.This approach includes training courses often taught by instructional designers to orient faculty to the instructional design process.The training course is then followed by one-on-one support through the process, with experienced online instructors acting as mentors.The instructional designer supports faculty to ensure their online classes are well-structured and work as the semester begins.Faculty who have participated in this process are more likely to feel prepared to teach their newly designed course (Vaill & Testori, 2012).The process appears to work well.Students reported higher rates of timely feedback and more opportunities to share ideas in courses that used a collaborative design process (Brown et al., 2003).
This model has challenges.To make the collaborative process work, the faculty member is often introduced to guidelines about the development of the courses.The design of online courses can be complex.Faculty can feel overwhelmed about the process (Chao et al., 2010).Within the process, conflicts can arise about the roles of the instructional designer and the faculty member because both members of the partnership understand instructional processes and evaluation (Xu & Morris, 2007).

Context of Study
The university in which this study took place is a large research institution with over 30,000 students attending undergraduate and graduate programs.The university has approximately 1,500 faculty in both tenure-track and nontenure-track positions.The Center for eLearning (CeL), as a centralized service unit, provides training in teaching online and instructional design support for faculty.Approximately 28% of faculty participate in professional development offered by the CeL.Currently, 19% of the 15,000 courses at the university are delivered through an online learning format.Within a 3-year period, four different course development models were used, allowing the evaluation of students' perceptions of course quality across four different design models.

Course Training Model (CT)
To facilitate the development of online courses, the staff at the CeL implemented a strategy which included paying faculty a stipend for participation in a training course which taught them how to design a course.The stipend was paid when the faculty taught the course online the first time.Topics in the course included behavioral objectives, assessments, best practices, delivery of instruction, building community, and disability accommodations.The faculty members were given a sandbox (an empty course shell), that could be used to practice developing a course.The culminating activity of the training course was the development of an online lesson which could be shared with the class.Designers were available for advice on course development.
Within the training, faculty were taught how to design and to teach online courses.Faculty were taught how to write lesson objectives and encouraged to include "start here" videos and to do a syllabus quiz.Other course modules provided information about the types of assessments and instructional materials that could be created.Different types of technology tools were demonstrated for the faculty members.The concepts of learning interaction and accessibility were introduced.The faculty were taught to have a table of contents that included unit, module, or lesson titles with a short description of the topic or the objectives for the lesson.Also, the courses often included the syllabus quiz.The faculty created the actual lesson design.This resulted in variability between courses.

Instructional Designer Supported (DS)
Upon a review of feedback from the faculty and evaluation of students' perceptions of teaching reports, the staff at CeL decided to take another approach to training faculty.The approach was based upon a partnership between the faculty member and the instructional designer.In this model, online courses were developed using the QM Rubric criteria with several refinements.The design partners used rapid prototyping tools to facilitate quick development of online classes.
One development tool was a course template.The template used an attractive design and contained the basic navigational design and support services essential for students.The flexible template accommodated the course organization the instructor wanted, with images and multimedia relevant to each course.The template organized the courses at two levels.The first level was the overall course structure with student support services embedded in the navigation system, a start-here module, and lesson module placement.At the second level, each lesson was organized with an introduction, course objectives, to-do lists of readings, lectures, and a list of assignments (see Appendix A for a snapshot of the lesson template).This common structure provided scaffolding for faculty to understand what to include as a part of their course, at the same time allowing flexibility in the use of pedagogy elements in instructional strategies and assessments.The course templates became the foundation that promoted rapid design, permitting faculty to quickly develop their courses.
The second development tool was a course blueprint in a matrix format.The matrix promoted the course planning and communication about the course content assessments and strategies.Through the matrix, the instructor easily saw the connections between course goals, lesson objectives, instructional strategies, and assessments.The matrix then served as the bridge to the template.The matrix details the objectives, assessments, instructional activities, and resources.The information is transferred to the course template.Both the designer and the instructor updated the course template with completed instructional products.
Another component of the new model promoted a collaborative partnership between the faculty member and the instructional designer.The faculty content expertise complemented the designers' knowledge of the course design processes and technical knowledge.The instructional designers provided as much support and assistance in the development of the content as possible.
Through the instructional designer, the broad knowledge about instructional design and pedagogy was narrowed to the best practices for the development of the content for that course.Faculty were required to produce instructional materials to replicate what would otherwise have been a campus-based lecture.Faculty developed notes, videos, or podcasts to supplement course readings.The additional advantage to the partnership was the opportunity to guide the faculty members in how to both teach their course and to use the technology in their course.Rather than teaching faculty to be an online expert, they became the online expert in their course.

Designed with No Support (NTS)
In NTS model, the faculty members designed their own courses without support or training from CeL.Instead, these faculty used their own learning and teaching experiences to design and to teach the course.Many of the members of this group were innovative and became the leaders of distance learning at the university.Because this group of faculty members were innovators, they developed courses before professional development or instructional designers were available.
Based upon the faculty experience and expertise in teaching online, the courses developed using this model varied in quality.The structure of these courses was dependent upon the instructors' level of knowledge about online instructional pedagogy.The instructional strategies also varied across the courses as well.Faculty in these courses were often experimenting with delivery strategies.Some of the experiments were grounded in research and intuition based on instructors' classroom experiences.

Additional Training to Meet QM Standards (QM)
Online courses developed to meet QM Standards were revised courses formed using either the NTS or the CT course development models.Before submitting an online course for QM review, the faculty members participated in additional training, exposing them to the QM criteria.As part of the course activities an instructional designer reviewed the online course with the faculty member using the QM Rubric.The instructor would then modify the online course based on his or her experiences in teaching the course along with the suggestions for improvement provided by the instructional designer.
Courses developed that met QM Standards and received the certification through the QM outside peer review process were developed using the training course method.Because of the need to teach and to revise the courses, instructors developed few of these courses using the instructional designer-supported method at the time of this study.The characteristics of the lesson structure, overall course organization, and interface variability continued to exist in these courses.To pass the QM review, however, those elements were improved.
Within a 3-year period, courses were developed using the four different models.This unique situation allowed for the comparison of four course development models: (a) course training (CT), (b) instructional-designer support (DS), (c) QM course training (QM), (d) designed with no support (NTS).The motivation for the evaluation was to ensure that dollars spent on the DS model resulted in a better quality of online classes.The second motivation was to evaluate the quality of the courses from the students' perceptions as part of a broader evaluation about quality in online classes to determine whether the students felt the classes were effective in supporting their learning.If classes developed using the CT or the DS models were determined not to be of high quality, then courses would be redesigned based on the students' feedback.Therefore, the following research questions guided this study: (1) Are students' perceptions of course quality equivalent across all development models?
(2) Which course development models did students perceive supported their learning?
(3) Which course development models were perceived as higher quality by learners?
(4) Which course development models were perceived by students as meeting the general QM Standards as described in the survey question for their online courses?

Methods
A questionnaire was developed based on the 43 standards in the 2014 QM Rubric (MarylandOnline, Inc., 2014).To maintain the integrity of the QM Rubric, the questions were modified from the standard to provide relevance to the students or to clarify terms that would not be understood.A Likert-rating scale ranged from 1 =strongly disagree and 5 = strongly agree was used for each of the questions.For example: Standard 7.3.Course instructions articulate or link to an explanation of how the institution's academic support services and resources can help learners succeed in the course and how learners can obtain them.

Student question.
Course instructions explain (or are linked to) academic support services (library, tutoring services, advising, writing center, or labs) and resources are available to help you succeed in the course.
In addition, separate items were developed to measure student perceptions of course quality (CQ) and course structure (CS).Students were asked if the course was structured in a way in which they felt they could learn and whether they felt the course was of high quality.Sampling occurred over an academic year (~12,000 students) across 3 semesters, from Fall 2015 to Summer 2016.The research team at this university kept records of online course development.Groups were based on the design model used to create the online course.
The following hypotheses guided this investigation: (1) The overall mean score differed across the different course-development models.
(2) The mean score for each standard differed across course development models.
(3) The mean score for the general QM Standards differed across course development models.
(4) The mean scores for course quality and structure differed across course development models The course roster from each course section was used to generate a student email distribution list.A total of 9,998 students were emailed a consent form and were told which course was to be evaluated.This sampling occurred after the midterm exams but before final exams.The timing of the request was designed to prevent biased responses based upon the grade received.Quantitative analyses were performed to ascertain the students' perceived differences of quality among the groups.A response rate of 3.24% resulted in a total of 324 (n = 324) responses to the survey: 33 from the QM group (n = 33), 115 from the CT group (n = 115), 98 from the DS group (n = 98), and 78 from the NTS group (n = 78).

Results
In evaluating the data for Hypothesis 1, differences were noted based upon the development model.The mean for the designer-supported (DS) group was the highest (M = 196.95).Further, this group displayed the lowest standard deviation (SD = 28.41) and the lowest standard error (E = 2.87).The no training support group (NTS) had the next highest mean (M = 182.28);this group also had the next lowest standard deviation (SD = 35.33)and the next lowest standard error (E = 3.76).The means of the Quality Matters (QM) and course training (CT) groups were slightly lower and nearly equal (M = 181) and (M = 180.6).The standard deviations were slightly higher and almost the same, 37.15 and 37.36.These results suggested that students' perceptions of the overall quality of QM and CT professional development models were similar.
An ANOVA compared the group mean scores and revealed that students' perceptions differed among groups.With respect to Hypothesis 1, the ANOVA found that the null hypothesis of equal means among all four groups must be rejected, F=(3, 320) = 4.80, (p = .003).The DS group scored higher than the QM, CT, and NTS groups on all standards, course structure, and course quality (see Table 1).The DS mean score for each standard was higher than the other course development models (see Table 2).To identify significant differences in the means per standard and to address Hypothesis 2, an ANOVA was run for each standard.The results revealed the null hypothesis must be rejected because differences were noted for every standard, course quality (CQ), and course structure (CS).To examine the cause of those significant differences, pairwise comparisons were conducted.As shown in Table 3, different results emerged from comparing pairs of course development models for each standard using the Bonferoni correction method.The results indicated that differences existed across the QM Standards, CS, and CQ.Therefore, Hypotheses 3 and 4 were accepted.The mean of the DS group was statistically significantly higher than that of the QM group for Standard 5, as well as for the students' perceptions of CS and CQ (p < 0.05).The mean score for CS for the DS group was statistically significantly higher than that of the QM and CT groups.The means of these two groups were statistically equivalent for the remaining standards.The students' perceptions of the CS were statistically nonsignificant across the remaining course development models.Finally, there were no statistically significant differences between the mean of the QM group and the mean of the CT group across all standards (p > 0.05).Similar results were observed between the QM and the NTS groups.

Discussion
As a result of conducting the study, significant differences were found between the course design models of course training (CT), instructional designer-supported (DS), no training support (NTS), and Quality Matters training (QMT).At the institution where this study took place, faculty control of the curriculum was highly valued.Within each of the course development models, the faculty member selected the content and the delivery method of the course materials.In the CT courses, the faculty member also designed the organizational structure of the course.In the DS model, the templates were provided; however, the faculty member controlled the deletion or addition of elements to the courses.In the DS model, the designer suggested strategies that complemented course content.The differences in the models were reflected in the students' perceptions.The courses developed with the assistance of an instructional designer were of significantly greater quality and had a better course structure.Students scored courses developed using the designer-supported model (DS) higher on all Quality Matter Standards.These courses employed the talents of both a faculty member and an instructional designer, the best of both worlds.An instructional designer provides pedagogical and technical expertise to support the faculty members as they implement their vision of the course.
Students identified the differences in the courses by the development models in the quality, structure, and all QM Standards.The DS group scored better from the students' perceptions across all standards, course structure (CS), and course quality (CQ), including the QM-certified group.However, this should not be completely unexpected since instructional designers used a course template designed with the QM Standards for the instructional design-supported courses.Also, instructional designers have an awareness of the standards.Thus, they built upon the quality level provided by the QM Rubric.Specifically, instructional designers were very important for this process as those courses were perceived to have significantly better course activities, student interaction, accessibility, and usability.While QM Standards are supportive in guiding the development process, the study shows instructional designers are an important part of the process.Future research should explore how the instructional designer amplifies the use of those standards, CS, and quality.Surprisingly, those courses developed without training (NTS) still performed relatively well with students scoring this group of courses as better than the QM and CT groups based upon the standards for learning objectives, instructional materials, learner interaction, and course technology.The concept of training faculty in the development of courses should have improved their ability to implement the QM Standards in the courses.The students in this study indicated that courses developed without training were just as likely as courses with faculty trained to design their own courses to include the QM best practices.Perhaps training courses for CT and QM overemphasized a few instructional strategies and instructional technology.A possible explanation for the higher scores in these areas could be the NTS faculty.The professors in this group were the distance-learning leaders who developed these courses before the CeL was in existence.Therefore, they would have incorporated instructional strategies they were exposed to at conferences or content journals.The NTS group struggled with learner interaction, indicating a skill that needs to be developed through training or support.Learner interaction appeared to be very important to this group of students.The NTS faculty could benefit from guidance in packaging the courses into a student-friendly interface.
Students perceived some strengths in the QM-certified courses over the CT and NTS courses in course overview, learner support, assessment, and quality, but not DS.This would indicate that the additional training provided by the staff improved these areas.However, the QMcertified courses scored lowest in instructional materials, learner interaction, course technology, accessibility, and course structure.Possibly, the limitations of the first course training were reinforced in the second course.The training of the second course further constricted the strategies used by providing too much structure in meeting the QM Standards.For these courses, a QMapproved template was not in place which would have addressed the course structure issues.The additional training did not lead to a reflective process to improve the courses.With the NTS group scoring the lowest in the course overview, training was beneficial in teaching faculty the importance of adding a course overview resulting in the higher score for the CT faculty group.
This study has several limitations.One was in the design of the survey itself.The question about quality should have been asked first.As the students answered the questions about the course structure, they may have been guided into believing those elements were the only ones to consider in the evaluation of quality.Another drawback was the students' ability to judge some elements of quality.For example, they may not be able to judge the accuracy of the content of the course.
The Quality Matters Rubric itself has limitations.The alignment of the learning objectives, instructional material, and assessment is an exercise within itself and is highly valued in the rubric.However, the best written objectives do not necessarily result in good instruction.The use of performance objectives can result in fewer examples of creativity in the delivery of the instruction, a finding identified by Lowenthal and Hodges (2015).With emphasis placed on the writing of objectives in the training course, it is possible the courses became disconnected, as described by Krusen (2015).
The QM Rubric identified evidence of some best practices, but not necessarily other variables of quality.Although the rubric evaluated types of interaction including learner-instructor interaction, the rubric cannot measure the quality of instructor presence as the course was implemented.These considerations can improve students' perceptions of quality within a course (Baran & Correia, 2014).Students may perceive quality based upon the quality of instructor interaction and relevance of the instructional material to their own educational goals rather than those outlined in the learning objectives for the lesson.Finally, the rubric itself does not evaluate the faculty expertise in the content area (Krusen, 2015).

Conclusions
Designer-supported courses provided personalized, one-on-one consultations with an instructional designer.During these consultations, instructional designers focused on the alignment of performance objectives to course activities.This allowed for a well-crafted course that reflected the instructor's vision and included a strong sense of teacher presence.For example, in the Standard 3 comparisons shown from students' perceptions, the designer-supported courses had significantly better assessments.After taking a training course, a faculty member may be able to align the course with the instructional methods and assessments.However, they are not as skillful in articulating that alignment throughout the course through lesson structure, text formatting, or word choice.
On several of the criteria, faculty without training scored better than those with training.It is suspected that those faculty who had taught campus-based courses had good strategies for teaching that content.Therefore, it is likely that some of those faculty were somewhat successful in translating their classroom-based strategies into the online environment without training or support.These faculty.as leaders in the field, would have attended conferences discussing best practices and implemented those strategies in their courses.Training courses often focus on assessment and instructional technologies embedded in the learning management system, which may have limited faculty creativity in the course training group.
This study gives additional insight into the importance of instructional designers in the design of online courses (Brown et al., 2003).The nature of the consultation meetings could have led to the higher scores over the course training model.The strategies discussed in consult meetings are immediately relevant to the faculty as they apply the skills in the design of their course.A deeper understanding of the impact of the instructional and assessment strategies in their courses resulted in the higher student perception scores.Through the consultation process, faculty members were willing to try different strategies, enhancing the perception of quality and the identification by students of the QM Standards in their courses.
The course training model approach at this institution had several challenges in its implementation.The course development training was a one-size-fits-all and one-stop approach to teaching faculty how to design courses.To finish the course, the faculty spent time completing and turning in assignments rather than designing an online course.Without a quality evaluation before teaching a course and receiving a stipend, wide variability resulted in the quality of the courses developed.Some courses were not well organized, with low levels of student interaction or engagement.The instructional materials developed utilized limited technology options based upon the technical skills of the faculty members designing the courses.With course development in the hands of the faculty members, who had many other responsibilities, very few of the courses were developed in a timely manner.Faculty incorporated the QM Standards taught in the course; however, they lacked the experience of an instructional designer.The result was the culmination of trial-and error-approaches by the faculty occurring in every course to determine what would work well for each instructor.
Training courses are commonly used to promote the development of courses.Training provided generic procedures, tools, and instructional strategies.Instructional designers who supported faculty combine two sets of skills, those of the faculty member and those of the designer.The best courses are developed through the partnership.All institutions may not have the budgetary means to institute the instructional design model.This would suggest that training needs to be carefully designed to honor the faculty members' knowledge and to maintain flexibility so that best practices are deployed within the courses.
Since this study, a system of professional development has been developed and implemented.With the instructional design support at the core, additional components have been added to enhance the faculty's experience and to provide additional support.Those components included the community of practice, a certification workshop, with ongoing professional development sessions, and open lab support.The system is flexible, allowing faculty to enter professional development from multiple points.The instructional designer then guides the faculty into using the additional resources to improve their online teaching practice (Golden & Brown, 2016).The next step for this institution would be evaluating the impact of the system of ongoing professional development and the impact upon student perceptions of the courses to see if more interactions create higher quality of courses.
The findings of this study can be extended in several ways.Replication of the study at another institution would assist in generalization of the study findings.A deeper evaluation of student success variables could determine if the standards in the QM Rubric improve course design, reducing the frustration of the students in online courses.For example, the students may be less likely to drop or withdraw from the course.Finally, it is possible results would be different using a different course evaluation tool, such as the OSCQR Course Design Review Scorecard that is part of the Online Learning Scorecard Quality Scorecard Suite.