Community College Faculty Perceptions of the Quality Matters ™ Rubric

This paper reviewed the factors that make up quality assurance including course design, content, delivery, and institutional support, as well as infrastructure in relationship to professional development impact on teaching practice. Building on the assumption identified in literature is the concept of course design being the most critical component impacting both student learning and faculty teaching. Course design affects student learning, faculty satisfaction with the course, establishes a teaching presence, and influences the transactional difference that occurs between the students and the instructor. Using the premise of the critical nature of course design, this study reviewed how the use of faculty professional development through a Applying the Quality Matters Rubric (APPQMR) workshop using the Quality Matters (QM) rubric as a framework to impact course design created specific faculty perceptions and affected teaching practice. Six themes identified from the research related to faculty’s perceived value and rigor of the QM rubric and training are discussed in the conclusion section.

alike. There are a number of factors that make up quality assurance, such as course design, content, delivery, institutional support and infrastructure, and student achievement (U.S. Government Accountability Office, 2011). To achieve accreditation, the USGAO report noted that institutions employ a range of course design principles and performance assessments to measure online course quality within a program. Each component is essential; however, course design is critical because it affects student learning and outcomes. Garrett et al. (2019) also found an emphasis on course design in the CHLOE 3 report, asserting that "the implementation of quality standards has progressed substantially in institutions engaged in online education, with the deepest penetration in online course design" (p. 32). CHLOE is an annual research report that examines the changing landscape of online education based on survey responses of U.S. higher education chief online learning officers. According to the findings, 85% of community colleges have adopted external quality standards for course design. Quality Matters (QM), is a widely recognized nonprofit organization that provides research-based best practices in online course design (Quality Matters, 2017).
The Quality Matters Higher Education Rubric, 5th edition (hereafter QM rubric), offers a model to help assure quality course design. Design establishes a sense of teaching presence in a course and influences the amount of transactional distance between students and instructors. Adair and Shattuck (2015) described how student engagement improved after applying the QM rubric to course design, resulting in higher grades, completion rates, and improved student satisfaction ratings in a number of research studies.
Online instruction and traditional classroom instruction differ considerably, requiring online instructors to design courses differently than they experienced during their own education (Mehta, Makani-Lim, Rajan, & Easter, 2017). Faculty members are subject-matter experts and often lack the skills required to effectively design online instruction unless they have received training in instructional design (Gregory & Martindale, 2017;Moore & Kearsley, 2012). Thus, online faculty members require various forms of support and resources to prepare them for success in the online educational environment. QM offers several workshops, most notably Applying the Quality Matters Rubric (APPQMR).
The purpose of this research was to explore the influence of QM's flagship workshop, APPQMR, on community college faculty perceptions of the QM rubric and their course design skills. Research was conducted in two phases at two QM subscribing institutions in the southeast U.S. Mezirow's (1997) theory of transformational learning is the guiding theoretical framework to this study, which suggests that faculty members, as adult learners, can change their perspectives following a disorienting dilemma such as the APPQMR professional development experience can provide. Perspective transformation may result progressively as participant's opinions are reinforced or challenged, new perspectives are introduced, and through critical reflection. Perspective changes can be experienced across three dimensions: psychological (autonomous reasoning), convictional (experience-driven belief systems), and behavioral (outward actions).
The following research questions guided the study and are later answered and discussed in terms of perspective transformation's three dimensions: (1) How does successful completion of the APPQMR professional development training effect participants' perceptions about the QM rubric?
(2) What are the challenges and successes that faculty experience as a result of APPQMR?

Professional Development and Course Design
The literature suggests that a strong correlation exists between faculty professional development and course design quality (Bigatel & Williams, 2015). McQuiggan (2012) found that faculty who thoughtfully engaged in professional development not only improved their online teaching skills but were also more likely to fundamentally change their teaching philosophies and practices. Likewise, Koepke and O'Brien (2012) found through faculty surveys and interviews that participation in instructor training resulted in significant changes to their pedagogical beliefs and teaching practices. In a six-case qualitative study, Johnson (2015) found that faculty members who completed training developed new skills and changed their course design practices. Johnson et al. (2012) examined an institution's three-day summer intensive workshop and found that 100% of participants (N = 24) experienced greater comfort creating online courses after training was completed.
It has been estimated that almost 50% of higher education institutions rely on QM for their online faculty development (Herman, 2012). Kearns and Mancilla (2017) surveyed over 22,000 individuals who had completed at least one QM workshop between 2012 and 2015. Of the 2,148 people who responded, 92% had completed APPQMR. The researchers found that participation in APPQMR influenced faculty member instructional practices in their online and face-to-face teaching. Hollowell, Brooks, and Anderson (2017) found that faculty participation in APPQMR led to significantly higher informal course review scores than before training. They also found that students' final exam scores and overall course averages increased as QM review scores increased. Despite these findings, research is lacking about community college faculty perceptions and the results of QM training.

Setting
The population of this study included 470 full-time and adjunct faculty members at two Southern Association of Colleges and Schools Commission on Colleges (SACSCOC) accredited community colleges in the southeastern U.S. who use the QM rubric for internal quality control initiatives. Thirty-nine usable subjects were collected and included in the study.
Both institutions mandate that distance education courses meet QM standards as outlined in the rubric. Both colleges have departments that support online and blended course development, although the number of staff members varies between them. The staff in these departments provide support to faculty who are responsible for the design and development of online and blended courses. The full-time equivalency (FTE) rates are approximately 4,000 (institution A) and 5,000 (institution B).

Design
An explanatory sequential mixed methods study (Creswell, 2014) was used to examine the influence of the APPQMR workshop on community college faculty members' perceptions of QM rubric and to understand the challenges and successes community college faculty experienced after completing the APPQMR workshop. This two-phase study began with the collection of quantitative data from community college faculty members who were using the QM rubric to design and develop online and blended courses. Via a researcher-created online Quality Matters Rubric (QMR) survey, participants self-reported their completion of the APPQMR training (i.e., yes, no) and rated the QM rubric on perceived usefulness (PU) and perceived ease of use (PE). Using a causal comparative design and a Mann-Whitney U test, the difference in faculty perceptions of the QM rubric based on participation in the APPQMR training was explored.
In phase two, a basic qualitative research design was employed as a means to explore participants' experiences and feelings (Merriam & Tisdale, 2016;Patton, 2015). Qualitative data was collected from faculty members who reported completing the APPQMR training using a researcher-designed semi-structured interview protocol. Interview questions were based on the original research questions and on the results of phase one. Data were coded and analyzed in accordance with grounded theory procedures, including (a) open coding for concept identification, (b) coding for concept development and elaboration, and (c) coding for context, process, and integration (Corbin and Strauss, 2015, pp. 322-323). Themes emerged through constant comparison and inductive analysis (Patton, 2015). Interviews were used to explain the survey results and further examine faculty members' experiences in APPQMR to garner an understanding about training elements that helped or hindered their perceptions of and ability to apply the QM rubric.

Quantitative Phase
In the quantitative phase of the study, a convenience sampling was drawn from approximately 470 full-time and adjunct faculty members across the two participating institutions. During spring 2018, an email was sent to all faculty via their institutional email requesting their participation in the online QMR survey. Forty-six (9.78%) faculty responded to the online survey and 39 responses were complete.
The faculty participants were balanced in gender (female, n = 21, 51.2% and male, n = 18, 48.8%). Most of the participants were Caucasian (n = 38, 97.4%). All participants held a master's degree or higher, which is consistent with the standard educational requirement to teach at the community college level (SACSCOC, 2006). Participants taught across a variety of academic disciplines, such as applied arts, social and behavioral science, business, humanities, mathematics, and natural sciences. Seventeen faculty completed the APPQMR workshop, and 22 had not participated in or completed the APPQMR workshop. Participants' demographic and experience data disaggregated by whether or not they completed APPQMR is presented in Table 1.

Qualitative Phase
Following the quantitative data collection and analysis, eight of the participants who completed the survey and participated in an APPQMR workshop were selected via purposeful sampling with maximum variation based on college affiliation, general demographic information, and experience level. Participants were selected who reported both positive and negative perceptions. The eight selected participants were contacted via email, asked to complete an informed consent, and scheduled for a 30 to 45-minute video-conferencing interview. Interviews were conducted over a five-week period in spring 2018 using the semi-structured protocol. Open coding was used after each interview to identify and explore main ideas in the data as it was presented. Initial codes were categorized into groups through constant comparison between interviews, allowing for constant reflection and refinement of codes and categories to combine similar ideas. Themes emerged within the data as categories were organized by research questions. Following the interviews, their transcription, and development of preliminary findings, the participants received a debriefing statement and a written summary of the preliminary findings. Table 2 summarizes the participant characteristics and perception scores based on the QMR survey. Though the combined perception range is 11 to 55, the combined perception scores of interview participants ranged from 25 to 55. Participants with the lowest scores generally had the fewest years of experience and were from institution B.

Quantitative Phase
Thirty-nine valid survey responses were used for the quantitative data analysis. The Mann-Whitney U test was conducted as the nonparametric alternative to the independent-samples t-test. Results of the test demonstrated that there were no differences in faculty perceptions of the QMR survey based on their completion of the APPQMR workshop, U = 141.00, z = -1.309, p = .001. Community college faculty who completed APPQMR training had similar perceptions about the QMR (Mdn = 22.71) as the faculty who did not complete the training (Mdn = 17.91).

Qualitative Phase
Eight faculty members with diverse demographics and QMR perspective scores from across the two community colleges were interviewed and results were analyzed. Results of the analysis indicated that APPQMR training held practical significance in its influence on faculty perspectives of the QM rubric across psychological, convictional, and behavioral dimensions. Six themes emerged from the analysis to answer the central research questions about training's influence on faculty perceptions of the QM rubric and their experience in APPQMR. These themes help to further illuminate the significance of training.

Theme 1: The QM rubric is a useful, though not perfect, framework for course design.
Faculty increased their knowledge of the QM rubric through participation in APPQMR training, resulting in favorable perceptions of the instrument. Six of the eight participants perceived the rubric as a useful, though not perfect, framework for course design. Even faculty who had low perception scores recognized the utility of the rubric. Participants found value in how the rubric helped them to improve course organization, navigation, and clarity of purpose from a studentcentered perspective.
Despite the mostly positive perceptions, five of the faculty members noted that the QM rubric could be improved in some way. Primary concerns included a perceived subjectivity in the standards, the rubric's failure to evaluate content, and an overall inability to ensure a high-quality course upon delivery.
Theme 2: Applying QM standards to course design is rigorous and inspiring. The rubric contains 43 individual standards and is accompanied by detailed annotations. Faculty did not view the rubric as "easy to use" in terms of its content. Rather, they viewed the rubric as rigorous and professional. One faculty member expressed feeling overwhelmed by the QM rubric prior to training, but later described the standards as an extremely helpful guide. Six of the eight interview participants described making notable changes to the design of their online courses after QM training. However, the time involved with applying QM standards to online courses was found to be a deterrent to making course improvements for one of the interviewees.
Theme 3: Learning to apply QM standards through training was more rigorous and time-consuming than anticipated. The APPQMR workshop is designed to encourage reflection while exposing participants to concepts related to the QM rubric and online teaching challenges. Faculty were challenged by the amount of time involved with completing some of the learning activities and disagreed with QM's estimation that the workshop should require 14 to 16 hours over two-weeks to complete. For instance, one participant remarked that the training took her twice as much time to complete the workshop as anticipated. This participant felt the time and rigor of training might hinder others from participation. Another participant, though a proponent for the training, claimed to tell his colleagues to schedule their training during a slow time due to the rigorous expectations.
Theme 4: The most challenging and influential aspects of APPQMR were learning objective alignment and content accessibility. Each of these aspects represent a portion of the QM rubric. Learning objective alignment is the cornerstone of the QM rubric. Alignment is evaluated in nine of the 43 individual standards and is the one concept that is cross-referenced in four of the eight general standards. The APPQMR workshop focuses upon alignment as a core goal because a course cannot pass a QM quality review unless all alignment related standards are met. Accessibility is one of the eight general QM standards and is comprised of five specific standards that address issues such as navigation, format, readability, and usability of course materials for all students.
Seven of the eight faculty interviewed discussed how learning objective alignment impacted their thinking and course design strategies. Six of the faculty found value in the concept of alignment and recognized its importance. They viewed alignment as a tool to help improve course organization and clarity. However, two felt that the process of alignment was cumbersome and disadvantageous to students. In this view, the inclusion of learning objectives and the articulation of alignment confuses students by focusing on educator-oriented information and creates unnecessary complexity, a detraction from the course content.
Accessibility was another challenging and often mentioned aspect of the QM training experience. Half of faculty interviewed acknowledged the importance of course content accessibility for students with disabilities but felt underprepared or unable to successfully implement accessibility guidelines or to check learning objects for accessibility. These participants reported that their response to accessibility challenges is to eliminate learning objects such as visual illustrations, charts, graphs, and video, despite the detriment to student learning. The faculty felt that the APPQMR workshop did not contain enough instruction on accessibility and expressed a desire to learn more about technology tools and processes to more efficiently ensure course accessibility.

Theme 5: The APPQMR workshop has the potential to be a transformative experience.
Six of the eight participants developed a belief through training that they could plan and implement course design changes to improve course quality.
The workshop experience provided many of the faculty with a disorienting dilemma by challenging them to consider a student-centered design approach. As a result, faculty perceived the QM rubric as a useful tool for improving course organization and structure. The learning experience also provided participants with a rationale to support their beliefs about online course design and using the QM rubric to support the design of their courses. Further, half of those interviewed expressed a belief that that the workshop could provide a transformative experience for all faculty, including traditional brick-and-mortar classroom instructors.

Theme 6: When QM is mandated, both social influence and facilitating conditions influence faculty's perceptions and use of the QM rubric.
Though unintended during the design of the study, the two community colleges presented opposing cases in terms of institutional approach to QM implementation. Differing approaches may have had an impact on resulting faculty perceptions. Both institutions mandate QM compliance in online course design; however, the initiative to adopt QM was a faculty-led decision at institution A, whereas the initiative was administratively driven at institution B.
Interviewees from institution A expressed a strong sense of social influence throughout the faculty and reported having access to a robust training and support network (which are forms of facilitating conditions). Four of the five participants identified a faculty member who was instrumental in the QM initiative. Participants also indicated that the QM initiative continues to be faculty-driven, as evidenced by the faculty provided workshops and training opportunities available at the college and peer mentoring. Additional facilitating conditions included financial incentives, technology tools and support, and one-to-one assistance from instructional designers and experienced online instructors.
In contrast, the interviewees representing institution B felt forced by college administration to comply with QM standards and expressed frustration with the process due to insufficient facilitating conditions, namely training and design support. These faculty members held more negative perceptions about QM than the faculty from institution A. Faculty members expressed frustration that their administrators had used QM punitively and to discourage online course development in the past. Instead of viewing the rubric as a course improvement tool, they felt burdened to comply with its standards when they teach online. Notwithstanding the historical influences and a stated need for more support and training, interviewees believed that faculty perceptions of QM have started to improve due to recent administrative changes.
Results showed that social influence and facilitating conditions both had an impact on faculty perceptions of QM at the two institutions. Results also indicated how participation in QM training can be used to counteract negative impressions.

Results
Results of this study showed that although there was no statistically significant difference in faculty's perceptions of QM between training participants and nonparticipants, there are several practical implications that training and support can provide. Six unique themes emerged from interviews with faculty who completed APPQMR. Themes centered around the value and rigor of the QM rubric and training. Results also helped to illuminate the faculty experience in APPQMR. These research findings are explored further in the following section.

Discussion
It is clear from the results of this study that creating a culture of support for online course developers-including various forms of training and development-can positively impact faculty's perceptions, which will positively impact teaching quality and student success in a distance education environment. Theory and research suggest that faculty experience level and demographics may provide explanation for the ambiguous results found in the literature and for the nonsignificant results found in this study.
Though there is no statistically significant difference in faculty's perceptions of the QM rubric between those who completed APPQMR and those who did not, the interview data demonstrated that those who participated in training found it useful in changing both their perceptions and behaviors. APPQMR is one form of training that can positively affect faculty's course design knowledge and skills.
Faculty professional development on course design has also been shown to improve student success throughout the literature (e.g., Bento & White, 2010;Hollowell et al., 2017;Rockinson-Szapkiw, Wendt, Wighting, & Nisbet, 2016). Additional research is needed to integrate the understanding of faculty development and student learning outcomes success.
Results of the basic qualitative research conducted in phase two provided some explanation about how participation in the training influenced faculty's perceptions of the usefulness and ease of use of the QM rubric. It also illuminated additional insight regarding potentially insignificant results, including mixed QM experiences. Results may be explained in part due to varying social influences and facilitating conditions between faculty at two community colleges. Two distinct themes emerged from this analysis that imply the practical significance of training. First, seventyfive percent of faculty who participated in the APPQMR training found the QM rubric to be a useable and helpful guide as they design distance education courses. Second, trained faculty believe that applying QM standards to their course design is a rigorous process.

APPQMR Content
A primary goal within the APPQMR professional development workshop was to understand and apply the foundational concept of learning objective alignment. It is not surprising then that the learning objective alignment content was found to be the most influential aspect of training. All but one of the interviewed participants discussed learning objectives and alignment. Some discussed it in a positive light while others discussed it negatively. Three out of four participants expressed that the content challenged them to think critically about their course design choices. As educators, they felt that the focus on alignment of learning objectives to course materials, activities, and assessments provided them with a useful outline from which to build a course. Faculty purported to gain new knowledge about the content that changed the way they think about course design. Likewise, the faculty in Mercer's (2014) study identified alignment between learning objectives and assessment as their biggest "takeaway" (p. 152). However, the faculty in Mercer's study also found the subject of learning objective alignment to be the most challenging. A few participants in this study expressed similar sentiments. Twenty-five percent of faculty members, however, failed to recognize the value and relevancy of learning objectives and alignment, arguing that the content is too educator oriented. The other seventy-five percent of faculty did not express concerns about the learning objective and alignment content. Additionally, faculty acknowledged that designing course materials to be accessible to all students is a worthy and important goal for distance education. Yet they felt ill-prepared to do so after training. They cited a lack of technology skills and resources to effectively and efficiently apply accessibility principles to their course design and they expressed a desire to improve their skills in this area.
Best practices in faculty professional development highlights that effective professional development includes ongoing support; thus, APPQMR may be most successful if course developers are supported by college instructional designers when applying the QM rubric (Roehrs et al., 2013). Dempsey and Liu (2017) also recommended providing faculty support with certain aspects of the QM rubric implementation, especially regarding technology and accessibility.
Results, therefore, provide greater insight into faculty's experience with specific aspects of the APPQMR workshop. The results of the research study suggested that faculty discovered learning to apply the QM rubric through training provided a disorienting dilemma, as found in transformational learning literature (Mezirow, 1997) that resulted in the acquisition of knowledge and confidence to plan and make a change.

Practical Implications
To determine the practical implications of professional development, we must look past providing only technology training and into the organizational cultural that provides ongoing support for learning technology, implementing technology, and providing significant faculty support for adoption within their online course structures. Creating a culture of support for online course developers-including various forms of training and development-will affect some faculty's perceptions and ultimate success in teaching online.
From an organizational perspective, the findings of this study illustrate that faculty's perceptions of the QM rubric's usefulness and ease of use can be influenced, in part, through participation in the APPQMR training. These perceptions are also impacted by social influences and facilitating conditions within the local setting where faculty engage with distance education quality assurance initiatives. Therefore, we suggest higher education institutions may need to develop a comprehensive plan that addresses the many aspects of quality assurance, including, but not limited to, faculty professional development, course design support, and evaluation of online course delivery. Previous studies by Bogle, Cook, Day, and Swan (2009) and Swan, Matthews, Bogle, Boles, and Day (2012) serve as excellent examples of combining QM design standards with Garrison, Anderson, and Archer's (2000) community of inquiry (CoI) framework to ensure the quality of course delivery. In this model, both frameworks were leveraged to combine the benefit of QM's explicit course design guidelines with the CoI constructs of social presence, teaching presence, and cognitive presence during the redesign of a program's core courses. The university provided professional development and support for both frameworks and the findings demonstrated significant improvements in student learning outcomes. The combination of training, social influences (including a QM expert and an instructional designer), and facilitating conditions (in the form of ongoing, individualized support) can serve as essential elements within an institution's quality assurance initiative for distance education.
Perhaps the greatest implication we see is the need for effective communication with faculty throughout the adoption of quality assurance tools and processes. Distance learning quality assurance initiatives will not be effective without faculty buy-in (Ragan & Schroeder, 2014;Wingo, Ivankova, & Moss, 2017), and high-quality courses begin with high-quality faculty to design and deliver them (Chen, Lowenthal, Bauer, Heaps, & Nielsen, 2017). This research has shown that faculty buy-in can be influenced, at least in part, through participation in training and development. As faculty members are faced with a disorienting dilemma through the knowledge, feedback, and interaction that training affords, they are challenged to reflect on their own pedagogical beliefs and practices and make behavioral changes. However, not all faculty who participate in training will experience this level of transformative learning.

Limitations
This was an investigative study limited to the APPQMR faculty professional development opportunity at two community colleges in the southeastern U.S. where there is an institutional mandate to use QM standards for course design. Results may not be generalizable to other institutions due to the limited focus or small sample size. Replication across a more diverse population or among different institutions and regions would improve generalizability. Further, a more robust experimental design could be used to allow for more control over the variables.

Conclusion
Faculty perceptions of an initiative like QM are important, because perceptions impact their intent to adopt and support any new initiative. The goal of this explanatory sequential mixed methods study was to examine the influence of the APPQMR workshop on faculty's perceptions about the QM design standards and their ability to design and develop distance education courses at the community college level. No statistically significant difference in perceptions was found between faculty who participated in APPQMR training and those who did not. However, several themes emerged through qualitative analysis of interviews that indicate participation in the workshop can be significant and influential for faculty learning how to apply the QM rubric to the design of their online and hybrid courses. The rigor of the training and of the QM rubric challenged faculty, but also provided them with a usable (but not perfect) framework for course design. Moreover, participants can experience a transformative learning experience through training that results in changed perceptions and course design skills. Results of the study provided descriptive information about faculty's experience in the APPQMR workshop, including illuminations of the most influential and troublesome aspects of the workshop, which were alignment of learning objectives and accessibility, respectively. It also provided some contextual explanation of the impact social influence and facilitating conditions have on faculty perceptions of QM. Both factors played a significant role in the formation of faculty perceptions of QM at the institutions where compliance was mandated. Additional research exploring the intersection of faculty professional development and student success in courses that are redesigned by faculty who are trained and supported in course design would be beneficial.