Student Perceptions of the Impact of Quality Matters–Certified Online Courses on Their Learning and Engagement

Quality Matters (QM) is one of the most widely adopted sets of standards for best practices in online courses to promote student learning. In this study, we examined student perceptions of the impact of QM-certified courses on students’ learning and engagement. Fifty graduate students enrolled in online courses completed a survey developed based on the QM rubric items. The QM framework includes 43 individual standards clustered into eight general categories. Among the eight categories, students rated Course Activities and Learner Interaction to have the highest impact on both student learning and engagement. The exploratory factor analysis revealed that 12 factors explained 88.89% of the variance of the impact on learning, and eight factors explained 85.72% of the variance of the impact on engagement. Clear expectations loaded as the highest factor for both learning (eigenvalue = 12.85) and engagement (eigenvalue = 21.58). These results will help instructional designers and online instructors understand the impact of individual QM design standards from students’ point of view to design online courses that effectively contribute to their learning and engagement.

Student perceptions of the quality of their online courses and their satisfaction with them are important since these perceptions have a direct impact on their learning and motivation (Davies, Howell, & Petrie, 2010). Student perceptions can provide an understanding of the effectiveness of student success in online courses. Researchers have investigated quality in online learning by studying factors related to student concerns (Aman, 2009;Ralston-Berg & Nath, 2008;Swan, Matthews, Bogle, Boles, & Day, 2012;Young, 2014) and found a positive relationship between learners' perceptions of course quality with the courses designed using QM guidelines. Although student perceptions of quality and satisfaction are associated with QM-certified courses, research related to student perceptions about how QM standards impact their learning and engagement in online QM-certified courses is limited. Therefore, it is important to understand students' perceptions of how specific QM standards impact their learning and engagement. In addition, it would be helpful to see how well the student-impact ratings for the QM items align to the eight QM standards. With the massive growth in the number of university students taking online courses (Allen & Seaman, 2016), online course quality in terms of improving student engagement and learning is an important goal. This research will identify student perceptions of the impact of QM standards on their learning and engagement in the QM-certified courses and recommend best practices in online course design using QM guidelines. Ally (2004) defines online learning as "the use of the Internet to access learning materials; to interact with the content, instructor, and other learners; and to obtain support during the learning process, in order to acquire knowledge, to construct personal meaning, and to grow from the learning experience" (p. 7). On the other hand, hybrid/blended learning refers to the blending of face-to-face and online learning "where the mix between classroom and online instruction can vary based on consideration of differences in course content and the level of student comfort with online learning" (Potter, 2015, p. 4). The focus of QM is to promote student learning in online and hybrid/blended courses, and the specific criteria evaluated under the rubric were selected as those characteristics shown to positively influence student learning (Legon & Runyon, 2007). Research suggests that well-designed online courses correlate positively to student attainment of learning outcomes, increased student-content interaction, ease of course navigation, decrease in student questions regarding course expectations, and higher student satisfaction (Alizadeh, Mehran, Koguchi, & Takemura, 2019). Swan et al. (2012) revised a course based on the QM standards and found that students achieved better learning outcomes. Young's (2014) study found that learners in the hybrid/blended courses designed to meet QM standards were more motivated, had more positive attitudes, and reported gaining knowledge to a greater extent. Studies also show that students achieve higher grades and have greater interaction with course materials after redesign of courses using QM standards (Hollowell, Brooks, & Anderson, 2017;Swan et al., 2012;Young, 2014). With the positive impact that QM-designed courses have on student learning, it is important to understand how students perceive the impact of individual QM standards on their learning in QM-certified online courses.

Student Engagement in QM-Certified Courses
Student engagement refers to the "extent to which students actively engage by thinking, talking, and interacting with the content of a course, the other students in the course, and the instructor" (Dixson, 2015, p. 2). Researchers affirm the importance of student engagement in online learning because it provides evidence of the effort required for students' cognitive development and their ability to create knowledge in a way that leads to a high level of student success (Banna, Lin, Stewart, & Fialkowski, 2015;Meyer, 2014). Research shows that student engagement, students' GPA, and course retention are improved when courses apply QM standards (Knapp & Paull, 2013;Runyon, 2006). Runyon (2006) explored the impact on student grades of learning activities that were improved to meet QM standards. Results showed that learners engaged more with the course content, and their grades improved as a result of the more interactive activities designed based on the QM guidelines.
In another study, Hixon, Buckenmeyer, and Barczyk (2015) explored student perceptions of the quality of course design and found that clear instructions on how to get started in a course, where to find various course components, and the design features that required students to interact with others were ranked high. They concluded that instructors should make sure that the course design features that facilitate interaction are relevant, appropriate, and well-structured.
Intentionally designing courses to increase students' interaction can positively impact students' overall learning in online courses (Knapp & Paull, 2013). The QM standards include general and specific standards related to learner engagement, but specifically provide a rubric with eight general standards and 43 specific standards to assess the quality of the course design for online courses. Although research suggests that QM-designed online courses can benefit students in terms of improving their engagement, satisfaction, and learning outcomes, there is still little information about how students perceive the impact of specific QM rubric standards on their engagement.

The QM Rubric Standards
The QM program is one method for assessing quality in online courses employed by many institutions of higher education. The QM program uses a rigorous peer-review process and rubric based on standards of best practice, research, and instructional design principles that are well established in online education (Legon, 2015;Shattuck, Zimmerman, & Adair, 2014). To measure the validity of the QM rubric, Legon (2006) compared it with a set of standards endorsed by the Council for Higher Education Accreditation and the eight regional accrediting agencies. He found the rubric to be consistent with published accreditation standards for online education and concluded that it "can demonstrate an institution's commitment to quality assurance of its online offerings and its success in achieving a well-defined standard for course design" (p. 9). In another study, Ralston-Berg (2014) explored student perceptions of course design features that indicate quality and compared student ranking of quality elements with QM-rubric item numerical ratings and found a correlation between learners' responses and standards of quality identified in the QM rubric. The QM rubric includes 43 specific standards clustered into eight general standards that are used to evaluate the design of online or blended courses (Legon, 2015). Every standard is assigned different point values, ranging from three points for essential, two points for very important, and one point for important elements. The QM rubric provides a quantitative measurement for evaluating the quality of the online course. Following are the eight general standards and their objectives: 1. Course Overview and Introduction: This standard ensures that the overall course design is clear to students and helps them understand how to get started in the course. Researchers have found that students perceive clear instructions on how to get started in a course and where to find various course components as the most important feature of a course design (Hixon et al., 2015).

Learning
Objectives: This standard ensures that learning objectives are clearly stated from a student perspective, are measurable, are properly aligned, are easy to understand, and help students focus their efforts in the course. Learning objectives define the course outcomes the instructor expects students to achieve. Research shows that the use of learning objectives in courses results in more efficient use of instructional time and, therefore, improves learning outcomes (Swan et al., 2012).
3. Assessment and Measurement: This standard ensures that assessments are aligned with the learning objectives, are consistent with course activities and resources, and clearly explain how the course grades are calculated. Learning objectives provide a method by which to tailor assessments to individual assignments and activities. Conveying clear grading expectations to students helps them focus their efforts, produce higher quality work, better grades, and less anxiety about their course assignments (Reddy & Andrade, 2010). Based on the research findings of Jonsson (2014), student performance greatly improved as a result of clarified expectations through the use of rubrics.
4. Instructional Materials: This standard ensures that instructional materials are comprehensive to achieve course objectives or competencies, and thoughtfully selected to support student learning outcomes. Based on her research, Murphy (2000) noted that instructional materials for online courses are effective when they are written with precise objectives and learning activities that are intertwined within the learning units.

Course Activities and Learner
Interaction: This standard ensures that forms of interaction incorporated in the course motivate students to attain course objectives and promote learning. Research shows that intentionally designing courses for increasing students' engagement can positively impact students' academic achievement (Knapp & Paull, 2013). Legon and Runyon (2007) found that courses designed with a QM rubric resulted in improving student learning outcomes, increasing student-content interaction, and decreasing student questions regarding course expectations.
6. Course Technology: This standard ensures that the course navigation and technology support student engagement and are used to achieve learning objectives. Any technology tools selected to be used by students should align with learning objectives by effectively supporting instructional materials, learning activities, and assessment instruments.
7. Learner Support: This standard ensures that the course provides links to resources for students to access institutional academic policies, technology support, and student support services essential to their success. This helps direct learners to services that can assist them in meeting course expectations and lead them through the institution. Young and Norgard (2006) suggest that technical support is vital to student satisfaction with online courses.
8. Accessibility and Usability: This standard ensures that the course provides documentation on the accessibility of the course materials, tools, and activities for all students. Meeting this standard helps focus learners' attention on the things that are most relevant and reduce time wasted in trying to decide what needs to be accomplished (Legon, 2015).
The QM rubric is based on the best practices and research on the preferences of online students and adults (Legon, 2015). This study builds on the current QM standards by identifying online students' perceptions of the impact of specific QM standards on their learning and engagement. Since student learning and engagement are the key focuses of QM program, such research is of vital importance to determine the impact of the individual rubric standards more precisely and provide evidence of how students perceive the effectiveness of these standards in improving their learning and engagement in online courses. Figure 1 provides a conceptual framework of how online course design is critical for student learning, engagement, and retention and transfer. In this study, we focus on learning and engagement. Figure 1. Framework of associations between course design, engagement, and student learning.

Purpose of the Study
Although researchers have been studying student perceptions of the outputs of QM standards, research on student perception of the impact of specific QM standards on their learning and engagement in QM-certified courses is limited. Therefore, the purpose of this study was to examine student perceptions of QM standards on their engagement and learning in QM-certified courses. The following research questions guided our study: 1. What are online student perceptions of the impact of QM standards in QM-certified courses on their learning?
2. What are online student perceptions of the impact of QM standards in QM-certified courses on their engagement?
3. How well do the student-impact ratings for the QM items align to the eight QM standards?

Participants
A cross-sectional survey was disseminated to 100 students in QM-certified classes offered in an instructional technology program at a university in the southeastern United States. This survey was sent only to one program at a single institution since our intent was to survey students in programs that have received QM certification for program design. This program-design certification ensures that all courses in the program are QM certified, which supports measurement of the impact of QM-certified courses. During the period when this survey was distributed, there were only four programs globally that had received program-level certification or program-design certification. Fifty graduate students who took online courses during spring, summer, and fall 2017 semesters completed the survey. The response rate was 50%. There were 36 (72%) female and 14 (28%) male respondents. The age of the participants ranged from 23 years old to greater than 53 years old. Sixty-six percent of the respondents were students seeking a master's degree. The

Procedure
This study used a descriptive, survey-based research design in which both closed-ended and open-ended questions were asked. Content validity was established by using the QM rubric to create the survey items. The instrument is described in the Instrument section below with items listed in Table 1. The questionnaire was created by the authors and disseminated through an Internet survey tool called SurveyShare to all of the nonredundant registered students of the eight QM-certified courses in the Instructional Systems Technology program across three semesters so that each student was requested to complete the survey once. Each semester captured only the new students. A personalized email with a link to the survey was sent to each student through the mail merge option in Gmail. Two reminders approximately two weeks apart were sent to students who had not yet completed the questionnaire. No incentive was offered.

Instrument
The 42 QM standards within the eight categories were used to create this survey. Item 7.3 (academic support services and resources) was inadvertently left out of the survey. Each QM standard was rated on a scale from 0 to 3 (no impact, a little impact, some impact, a lot of impact) on two the constructs of learning and engagement. Students were given the following prompt: "Please think about each standard and rate how much impact this standard has on your online learning (engagement)." Cronbach's alpha, an estimate of the internal consistency of student responses across all items of the survey (.98) and separately for each construct (.94 for learning and .97 for engagement), was high. There were also two open-ended questions: (a) "Which strategy(ies) impacted you the most for your learning?" and (b) "Which strategy(ies) impacted you the most for course engagement?"

Data Analysis
Data from each semester were combined into one SPSS 23 database. Descriptive statistics were used to report the participants' perception of the impact of the QM survey on their engagement and learning in order to answer Research Questions 1 and 2. The percentage of responses for each response option for each QM item is provided as well as a grand percentage for each standard. The mean and standard deviation are provided to support the comparative analysis of individual items. An exploratory factor analysis was conducted to examine Research Question 3 regarding how well student perceptions of the items within the standards aligned to the items of the eight standards as assigned by QM.
Responses to the open-ended survey questions were analyzed using Miles and Huberman's (1994) constant comparative method. First, the data were coded by segmenting and assigning labels to the text passages. Then, similar responses were categorized into the eight prestructured QM categories to gain further insights into the survey results. The frequencies of the responses were also noted to identify categories that had the greatest explanatory potential. Once all of the responses were coded, each category was then reanalyzed to determine the relationships between the codes to provided further explanations of student perceptions of the standards that impacted their learning and engagement the most.

Research Question 1
The percent of responses for each item and grand percentage for each QM standard for learning is presented in Table 1. A majority of students rated each QM standard item as impacting their learning A Lot. For learning, the top two QM standards were Course Activities and Learner Interaction (78.50% A Lot) and Learning Objectives (75.20% A Lot). Within the Course Activities and Learner Interaction standard, the highest-rated item was 5.1: "The learning activities promote the achievement of the stated learning objectives or competencies" (96% A Lot and Some, mean 2.78). Within the Learning Objectives standard, the highest-rated item was 2.2: "Module/unit learning objectives or competencies describe outcomes that are measurable and consistent with the course-level objectives or competencies" (82.0% A Lot). The lowest rated QM standard to impact learning was Learner Support (42.65% A Lot). In response to the open-ended question, students recorded the strategies that impacted their learning the most (see Table 2). The strategies were coded using the QM standards. Many students (40%) noted Course Activities and Learner Interaction as the most effective strategies. Students thought that they got the most out of the peer discussions because they helped them understand other students' points of view and reasonings. Students also believed that course activities with hands-on-learning projects impacted their learning by simulating real environments. Some students mentioned that Instructional Materials (22%) were helpful in providing the opportunity to learn content in a variety of formats (i.e., readings, videos, discussions, projects, etc.) to support module learning goals. Note. n = 50. Participants may have mentioned multiple strategies.

Research Question 2
The percent of responses for each item and grand percentage for each QM standard for engagement are presented in Table 3. In general, students rated the QM items as having slightly less impact on engagement than on learning. The top two QM standards for impact on engagement were Course Activities and Learner Interaction (79.08% A Lot) and Course Technology (70.61% A Lot). Within the Course Activities and Learner Interaction standard, the highest-rated item was 5.2: "Learning activities provide opportunities for interaction that support active learning" (82.98% A Lot). Within the Course Technology standard, the highest-rated item was 6.2: "Course tools promote learner engagement and active learning" (83.67% A Lot). The lowest rated QM standard to impact engagement was Learner Support (42.71% A Lot). Similar to the ratings of impact on learning, the lowest-rated QM standard and item on impact on engagement rating is quite high, falling in the "some impact" range. In response to the open-ended question, students reported the strategies that impacted their engagement the most (see Table 4). The strategies were coded using the QM standards to further explain the survey results. Several students' responses (30%) were coded under Course Activities and Learner Interaction as the most effective strategies to impact their engagement in the course.
Students explained that interacting with other students and getting feedback from the instructor kept them engaged through the course. In addition, discussion forums and collaborative problemsolving using case studies encouraged engagement in courses. Some students (14%) mentioned that having a variety of instructional materials enhanced their course engagement, such as videos, readings, and PowerPoints. Students also asserted that e-lesson in the course helped them to stay engaged with the course content. Note. n = 50. Participants may have mentioned multiple strategies

Research Question 3
In order to answer the third research question, two exploratory factor analyses (EFA) were conducted using principal components. A preliminary review of the correlation matrix for the factors revealed no correlations reaching the threshold of .35 indicating an orthogonal solution (Tabachnick, Fiddell, & Ullman, 2007). Therefore, a varimax rotation was used. The Guttman-Kaiser rule (Guttman, 195;Kaiser 1960Kaiser , 1970 of eigenvalues greater than 1 and the scree plot were used to identify the number of factors. Factor loadings of .35 were used to identify the viable items for each factor. The first EFA identified 12 factors that explained 88.89% of the variance of the impact on learning (see Table 5). Some items were complex, loading on more than one factor, and several factors had only one or two items associated with them. The second EFA identified eight factors that explained 85.72% of the variance of the impact on engagement (see Table 7). Again, some items were complex, loading on more than one factor, and several factors had only one or two items associated with them.
We reviewed the factor loadings in an attempt to identify the common element for each of the factors. Tables 6 and 8 list the factor loadings and common theme for each of the factors for learning and engagement, respectively. Items that loaded on more than one factor were assigned to the factor with the highest factor loading. Three factors consisted of only one item (1.2, 1.4, and 6.4) and were eliminated from further consideration, so the number of factors was further reduced from 12 to nine factors for learning. These factors included (a) expectations clearly stated, (b) instructional materials, (c) measurement, (d) how course components fit together, (e) tools and objectives, (f) interaction, (g) assessment, (h) usability, and (i) alignment.  As with the learning EFA analysis, items that loaded on more than one factor for the engagement EFA were assigned to the factor with the highest factor loading. Two factors consisted of only one item (6.3 and 4.5), so they were eliminated from further consideration. The process of assigning the items to the factor with the highest loading and eliminating the two factors with only one item resulted in four identified factors for engagement. They included (a) expectations clearly stated, (b) course objectives, (c) learner centered, and (d) currency.

Discussion
Our study found that Course Activities and Learner Interaction was rated the highest among the eight categories when instructional technology students were asked about the impact of QM standards in QM-certified IST courses on their learning and engagement. This supports other research studies that have found that courses designed with the QM rubric resulted in improving student-content interaction (Fedynich, Bradley, & Bradley, 2015;Legon & Runyon, 2007). Students are satisfied with online experiences when they perceive that the instructor provides opportunities to interact in different ways (Fedynich, Bradley, & Bradley, 2015;Walker, 2016). Course Activities and Learner Interaction was also rated the highest based on students' open-ended comments. Students' open-ended comments further revealed that the peer discussions and handson projects were very helpful both for engagement and learning in online courses. This is consistent with research studies that conclude that discussions are effective instructional tools to engage learners in online courses (Darabi, Liang, Suryavanshi, & Yurekli, 2013). Similarly, hands-on projects have been found beneficial for engagement and learning in online courses (Chi & Wylie, 2014). Knapp and Paul (2013) concluded that being intentional in designing courses to increase students' engagement can positively impacts students' overall academic achievement.
Learning Objectives was the second highest among the eight categories when students were asked about the impact of QM standards in QM-certified IST courses on their learning. Researchers have stated that informing the learners of the objectives at the beginning of instruction clarifies what is expected of them and assists them in guiding their learning (Reiser & Dick, 1996). Studies on online learning have confirmed the importance of setting clear expectations in online courses (Kearns & Mancilla, 2017;Swan et al., 2012). In addition to establishing expectations in the learner, learning objectives also provide the foundation for designing assessments. Swan and colleagues (2012) found that students' performance in online graduate courses improved because the QM revision led instructors to focus on objectives and the mapping of objectives to learning outcomes. These results suggest that students like having clear and measurable learning objectives that help them be engaged within the course and lead to improved learning. Though Learning Objectives was rated the second highest of the closed-ended survey items, only a few students noted its importance to impact their learning and engagement in the open-ended comments. This may be due to the fact that they already rated the learning objectives in the closed-ended items higher. It is not that they do not value the importance of learning objectives, but they choose to include other items in their open-ended comments. The two themes rated highest based on the open-ended comments were (1) learning activities and interaction and (2) instructional materials.
Course Technology received the second-highest percent of students rating it as impacting their engagement a lot. Online course technology provides opportunity for the learners to engage with the instructor, other students, and with the content in online courses. Martin and Bolliger (2018) in their study found that students considered several engagement strategies that used course technologies as important. Some of these included students working collaboratively using online communication tools, instructors' use of the learning management system for activities such as sending announcements or posting online grading rubrics, or using a forum to ask the instructor questions. In addition, students were engaged when content was delivered in different formats, such as text, video, audio, games, and simulations. Instructors are able to create online courses with content in various modalities due to the availability of the technologies. Among the various items on the survey, tools that provide active learning and are readily available were important to the students.
While students had rated Course Activities and Learning Interaction and Learning Objectives as the highest-rated categories in the closed-ended items, Instructional Materials was rated as the second-most-important theme in open-ended items for both learning and engagement (see Tables 2 and 4). Students commented that having a variety of learning tools, such as videos, readings, and PowerPoints, in the QM-certified online course enhanced their engagement with the content. This implies that presenting instructional material in different formats creates a rich learning environment that can sustain engagement and deeper understanding of the course content. Song, Singleton, Hill, and Koh (2004) also found that graduate students benefited from having a variety of activities, including chat rooms, discussion boards, and a few synchronous meetings online, as they helped develop a sense of community in an asynchronous course. QM standards 4.1 and 4.2 emphasize the importance of having both a clear use and purpose for the instructional materials that contribute to the achievement of learning objectives. Instructional materials for online courses are effective when they are written with precise objectives, learning activities, and assessments that are intertwined within the learning units (Kearns & Mancilla, 2017). Alizadeh et al. (2019) suggested that although developing effective online instructional materials and resources is time-consuming, it is a valuable process for student satisfaction in online courses.
An EFA was used to examine the alignment of student perceptions of impact of the QM items to the QM standards to which they had been assigned. Clear expectations loaded as the highest factor for both learning and engagement. The initial factor analysis identified 12 factors that impacted on learning and eight factors that impacted on engagement that were able to be further reduced by combining items that loaded on more than one factor. Although the factors identified through the EFA grouped some of the items in the same standard together, they did not align well to the eight QM standards for impact on engagement or learning. This indicates that students may perceive the QM items differently than the intended standard to which they are assigned. Students may also perceive items differently when thinking about their impact on learning and engagement. Items were grouped into fewer factors for engagement (four factors) than learning (eight factors). Shattuck (2015) suggested that narrowing the search for the impact of a QM review to specific groups of standards, such as course alignment standards or learner engagement standards, can be productive. Although a focus on individual standards in the QM rubric can be a precise way of measuring impact, clustering objectives striving for a common outcome may provide a better account of the cross impacts of multiple standards (Legon, 2015).

Conclusion
This study has implications for instructional designers and online instructors on student perceptions of the QM standards in QM-certified online course design to support learning and engagement. Since Course Activities and Learner Interaction was the most important standard to impact student learning and engagement, instructors using QM guidelines can focus on activities that encourage students to become active learners. For example, instructors can incorporate collaborative learning activities, such as problem-based learning or case-based learning, that can create community of learners through peer-to-peer interactions. Students can interact with the course content as well as each other through discussion boards, group work, peer review, and so on that encourage students to work together and be active participants in the learning process. In addition, students ranked high the necessity of having clearly aligned learning objectives with the course activities. In this regard, instructors can use the syllabus or course introductions to specifically explain how different course activities are aligned with the learning objectives and how the course activities will help students achieve their learning goals. Instructors can also create e-lessons at the beginning of every module to introduce weekly activities and clearly explain the learning objectives that indicate what students will learn and be able to do after successfully completing the assignments every week. Finally, instructors can present instructional materials in a variety of formats using different course technologies to meet the needs of different learning styles. For example, instructors can use digital media (audio or video), such as YouTube, podcasts, TED Talks, screencasts, PowerPoints, Prezi, and so on, to create a rich learning environment that promotes learner engagement and active learning. In addition, they can use diagrams, tables, pictures, games, simulations, and so on to make content visually appealing and sustain motivation and deeper understanding.

Limitations and Future Research
There were some limitations in this study. First, the sample size was small and was drawn from a single program at a single university. The perceptions drawn from this sample may not be representative of the population of online learners. Additional research with different samples of participants is needed to corroborate these findings. Second, all data were self-reported due to the nature of the study. Participants were enrolled in QM-certified courses. It is possible that the student perceptions varied because they were not familiar with the QM standards or because they knew they were enrolled in a QM-certified course. Third, these findings may have response bias. The response rate was 50%. Students who chose to participate in the survey might have answered differently than students who chose not to participate in the survey. Future researchers could replicate the factor analysis of this study in preparation for confirmatory factor analysis. It would be worthwhile to follow up quantitative studies via interviews or focus groups that can add to the existing body of research.

Author Note
This work was supported by the Scholarship of Teaching and Learning Grant at University of North Carolina Charlotte.