Ensuring Online Learning Quality: Perspectives from the State University of New York

As participation in online learning continues to expand, higher education institutions must implement policies and procedures to ensure quality at the course, program, and institution levels. In this paper, the authors describe a process that the State University of New York (SUNY) System implemented, utilizing the OLC Quality Scorecard, to help individual campuses examine the quality of their online offerings and develop strategies to support continuous improvement.

Participation in the process is free and voluntary and is generally initiated and/or supported by the chief academic officer on a particular campus. The IR Process is designed to engage a campus team made up of campus academic leadership, as well as representatives from all functional and infrastructure areas identified in the scorecard standards. This aspect of the process is essential for campus impact, credibility, and systemic change to occur. The campus agrees to commit to the 3-step engagement (see Figure 1) and is required to engage a campus leadership team comprised for representatives from across campus. After the three meetings, campuses are asked to complete an Implementation Plan, which addresses any deficiencies and outlines plans for improvement. The plan is signed by the campus president and submitted to Open SUNY.

Figure 1. 3-Step Engagement Process
Since launching the IR Process in April 2013, 53 SUNY campuses have expressed interest in or started the process, 33 have completed the third session, and 19 have submitted their Implementation Plan. The online learning leaders at the campuses that have engaged in the process believe it is valuable for enhancing cross-campus awareness of online learning quality, benchmarking their current practices with established standards, and documenting plans for continuous improvement.

Online Learning Evaluation and Assessment
Although the entire IR Process can be considered an assessment in and of itself, the second half of this paper will focus specifically on the standards in the Evaluation and Assessment section of the OLC Quality Scorecard, which help campuses establish procedures necessary for continually ensuring online learning quality after the IR Process is complete.

Standard 9.1-The program is assessed through an evaluation process that applies specific established standards.
For the purposes of the IR Process, the program is defined as the campus's distance learning initiative to include the delivery and support of both online courses and programs. The process addresses the institution's capability to ensure quality in online learning across the campus, not just for particular academic programs. SUNY campuses are expected to have a regular process or mechanism for assessing the outcomes of their distance learning initiative against an established quality standard and the goals they have for that initiative.
At most institutions, there are assessment processes for academic programs, academic departments, and nonacademic administrative units. In New York, guidance for this process is provided by our regional accreditation body, Middle States Commission on Higher Education, in Standards V and VI (MSCHE, 2014). Where those processes are in place for ongoing functional/departmental reviews, they can be leveraged to fulfill the requirement for this OLC Quality Scorecard standard. In fact, we encourage institutions to build upon existing processes in order to minimize the perception of new or additional work and to strengthen the existing processes.
One of the challenges with Standard 9.1 is the uncertainty or question around the basis for the evaluation-against what standards, goals, or models should institutions assess their online learning initiative? There are a couple of options here that we suggest campuses consider. An important priority for SUNY is scaling online learning with quality, so we have established a quality framework in SUNY that is based on the OLC Quality Scorecard as a first option. The second can come from the campus if there are clearly articulated goals for the online learning initiative that can serve as the basis for assessment and be folded into a regular evaluation process. This can be complementary to the use of the OLC Quality Scorecard and does not have to be one or the other.

Standard 9.2-A variety of data (academic and administrative information) are used to regularly and frequently evaluate program effectiveness and to guide changes toward continual improvement.
Online learning quality must be evaluated using a variety of data from multiple perspectives. Institutions are to develop a holistic review plan that incorporates multiple aspects of quality such as online course and program enrollment trends, student learning outcomes, student success standards (i.e., grades, course completion rates, retention rates, and degree completion rates), and measures of student and faculty satisfaction. In addition to evaluating the online course and programs themselves, it is important to examine the effectiveness of support services and technology related to online learning. Institutions must use a broad lens to determine what data is meaningful, and then create a plan for collecting and using it.
In order to ensure that any data collected is valuable, institutions should also establish goals for the process, benchmarks for comparison, and mechanisms for reporting findings and making improvements. Goals could be set based on prior evaluation and assessment efforts or existing institutional priorities; they serve to clarify how the institution views online learning quality. Benchmarks, either data from previous years or data from peer institutions, can provide context for setting quality standards. Lastly, the findings from these processes should be continually communicated back to stakeholders so the data can be used to make positive changes. The point of collecting a variety of data is to use it for improvement, not to just write reports about it or file it away.
Of course, to make all of this possible, it is necessary to have appropriate organizational structures and institutional supports in place. In addition to an office of institutional research and/or an assessment professional, many institutions have an online learning professional and/or an online learning committee with representation from areas across campus that can help to oversee the online learning quality review process. The degree to which online learning is recognized as an institutional priority and adequately supported is also crucial. When areas in need of improvement are identified through the evaluation and assessment process, senior-level administrators must understand and support the recommendations in order to bring about change.

Standard 9.3-Intended learning outcomes at the course and program level are reviewed regularly to ensure alignment, clarity, utility, appropriateness, and effectiveness.
At many institutions, the processes used to ensure quality student learning outcomes in online courses and programs may not differ from the processes used for face-to-face courses and programs. However, if those processes do not work well for evaluating student learning outcomes for online courses and programs, institutions need to make sure that new processes are developed and followed. It is also important for instructors to be intentional and creative when designing their courses to ensure that students have opportunities to demonstrate particular learning outcomes that may seem to be difficult to assess in an online environment, such as oral communication. Depending on the particular learning objectives, this could be a challenge and may need to be addressed through support from instructional designers and/or the use of innovative technology. Nevertheless, it is crucial that institutions ensure that they have a regularly conducted process for reviewing student learning outcomes in online courses and programs and that they document and utilize the finding to promote continuous improvement.
For example, Herkimer College has employed the following strategies to ensure that student learning outcomes for online course and programs are reviewed regularly: • A campus-wide assessment process is in place to ensure alignment of learning outcomes from course to course. Instructors meet regularly to review the common course objectives and discuss strategies to align assessments to evaluate student learning outcomes. The alignment of learning outcomes and assessments is integrated into the course management system and provides empirical data to inform the twice yearly Assessment Day discussions.
• All course outlines are reviewed and/or revised every two years to assure relevance and comprehensiveness of content, appropriateness of course learning objectives, and assessment of student learning outcomes.
• Quantitative results of student learning outcomes assessments are reviewed every semester by the faculty and modifications to the instructional design, content, and/or assessment strategies are proposed for subsequent semesters.
• Programs are reviewed and revised every five to seven years by the program faculty and Associate Dean. This process requires input from graduates, current students, instructors, institutional research, and, where appropriate, program advisory committees.
• The Curriculum Committee reviews and approves all course and program revisions. This assures that changes are reflected in all college documents and forms.
• All online programs have been reviewed to adhere to SUNY's new Seamless Transfer initiative and are fully compliant. This assures that transfer students from, and graduates of, a Herkimer program have achieved the learning outcomes required for successful continuation of their education or entry into their chosen career.
• Observation of the online classes occurs on a regular schedule. Some faculty have volunteered to have their courses observed on a more frequent basis by the Internet Academy. Observers are experienced online faculty and administrators, who use the same consistent rubric to monitor how effectively the course addresses the achievement of the stated student learning outcomes.
• At the conclusion of each full semester, all full-time faculty meet for a "Closing the Loop" workshop (previously referred to as "Assessment Day") at which they compare their course student learning outcomes assessments and discuss strategies for improvement.
• The college has also created the "Herkimer College Assessment Handbook," which is an online vehicle for instructing the processes of assessment, data collection, and results dissemination for course, program, and institutional objectives and learning outcomes assessment.

Standard 9.4-A process is in place and followed for the assessment of support services for faculty and students.
Assessing the support services offered to both faculty and students is critical to positively impacting the learning outcomes within face-to-face and online courses and programs. Without knowing whether the support services are meeting the needs of their users, it is impossible to determine if the best possible support is being provided to enhance the online learning environment. "Student support services would include library services, tutoring, bookstore, counseling, advising, online student orientations, financial aid, and cashier services. Faculty support services would include technical support, course development support, professional development activities, and ongoing support during the teaching process" (Shelton et al., 2014, p. 89). Surveys are often employed as the primary mechanism for assessing support services, but focus groups and individual interviews can be used as well. Surveys can either be short and targeted toward users of individual services, or longer and distributed to a sample of all students/faculty to obtain their perspectives about multiple services. In many cases, surveys may already exist to evaluate some of these services, but institutions may need to disaggregate the data between face-to-face and online faculty and students. Institutions must determine which methods work best for obtaining the data needed to evaluate the quality of these services and make any necessary changes. After these methods are identified, they should be included as part of the broader online learning quality review process.
At Finger Lakes Community College (FLCC), each service area (defined as any office that is not an academic department) is required to participate in a "Service Area Assessment" process on a five-year rotating schedule, which is coordinated by the Office of Assessment, Planning, and Continuous Improvement. This directly connects to online learning in that the Office of Online Learning goes through this process, but also because online students depend upon many of the services areas who also go through this process. Student success is not housed in one office; it is a campus-wide effort. The Service Area Assessment process begins with trainings for the service areas on the improvement process and the assessment language that is utilized throughout. After successful completion of training, each service area begins by crafting a mission and a vision statement. Upon completion of the mission/vision statements, the service area director, working with members of their department and in consultation with the executive sponsor, crafts three to five goals to cover the next three years. The created goals are required to have, where applicable, data to support the goals as well as any supporting documentation. Once the mission/vision and goals have been approved by the Vice President and the Chief Assessment Officer, the service area begins crafting their program review packet.
The program review is where the mission/vision, goals, and resources are outlined and explained in-depth so that when external reviewers come to campus, they have a solid understanding of the current state of the program. The program review includes all aspects of the service area and its operation. The program review document can range anywhere from 30-100 pages, beginning with an executive summary and including a field analysis, history, report on departmental personnel, financial considerations, facilities, and culminating in outcomes. This comprehensive review of the department not only serves to offer a snapshot of each department to those not intimately involved in daily operations, but also serves to create a true understanding of and alignment to each department's functions. It also establishes a plan for the assessment and evaluation of each service area and, depending on the department, documents how the data will be disaggregated to examine the experiences of online students. Following the creation of the program review, the service area has an external review with members of the field who work at other, similar, institutions. After the external review the service area presents their program review and the external review findings to the Board of Trustees, which offers the opportunity to showcase the service area as well as to point out any areas in which the service area could use greater support.
The service area assessment process at FLCC is truly about continuous improvement. When service areas function at their highest levels, it makes it easier for students to achieve success. There are numerous advantages to the service areas as they go through this process including face time with the Board of Trustees, the ability to determine areas for improvement, the ability to spotlight fantastic work by staff members, and to create a greater understanding of how all service areas fit into the greater umbrella of student success.

Standard 9.5-A process is in place and followed for the assessment of student retention in online courses and programs.
Student retention in online courses and programs must be monitored in order to ensure student success. Institutions should analyze a variety of metrics including course completion rates, course grades, and student retention from semester-to-semester and year-to-year. The online quality review plan should specifically indicate goals or benchmarks for these measures that are meaningful for the institution. For example, aggregated online course completion rates could be compared to aggregated face-to-face course completion rates at that institution, or, instead, they could be compared to an online benchmark from a peer institution. The individual metrics must be placed in context to have any meaning. It also may be useful to disaggregate the data to analyze courses by subject or level and programs by major/CIP code. Alternatively or additionally, the data could be disaggregated by student characteristics (i.e., race, gender, age, etc.) to determine if any targeted intervention strategies are necessary. Since prior research has found differences for student success in online courses based on students' demographic characteristics and academic program (Bull, 2015), these types of analyses could promote practices to enhance equity.
Since the inception of online courses at Herkimer College in 1997, the institution has compared the course completion rates and academic achievement (grades) of students taking courses taught by the same instructors in both the online and face-to-face learning environments. The aggregated completion rates for face-to-face courses have been consistently higher than completion rates for online courses, but the differences are statistically significant in only about half of the individual course comparisons. The significant differences have been addressed in two ways: 1. Herkimer's award-winning "Course Refresh Initiative" was a 2-year project that provided retraining and course revision assistance to all full-time and 90% of the adjunct online instructors.
2. Herkimer has a "Closing the Loop" session twice each year where instructors meet to share course completion and success information and discuss strategies for improvement.
In terms of academic achievement (grades), aggregated online course achievement has only been statistically significantly lower than face-to-face course achievement four times out of the 36 semesters reported; the two aforementioned initiatives have addressed this issue as well.
Once processes for collecting and analyzing student retention data are in place, it is then necessary to establish mechanisms for enhancing student retention. One effective practice for enhancing student retention seems to be the implementation of an early alerts system, which can help to streamline communication strategies with students, develop student success networks, and provide ways to identify barriers to student success. Finger Lakes Community College has had great success in increasing their online course completion rates over the last three years by implementing an early alert system and delivering interventions in the first seven days. Students who are flagged by faculty as lacking in participation within the first week of a course are contacted via email and a personalized telephone call. This has led to an increase in course completion rates of 8% from fall 2014 to spring 2017. Similarly, at Monroe Community College (MCC), an early alert system was put into place and the result was a retention rate through census of 89% in the spring of 2017. Understanding the reasons for failures to complete, as well as the addition of the early alerts process, made a significant difference in online student success. MCC is also in the process of using current data to create a more robust predictive analytics model. This model, based on both student activity and demographic histories, will help identify potential long-term success potential in students.

Standard 9.6 -A process is in place and followed for the assessment of recruitment practices.
Marketing online programs is a new and growing field that demands that new strategies be implemented. No longer can colleges depend on traditional marketing methods. Online learning has broken the mold of limitations to geographic constraints and offers opportunities for students to effectively study at a chosen college or university without the need to actually relocate to that location. It is also important to make sure that students being recruited understand the course and program requirements and that students are not admitted or enrolled who do not meet those requirements.
Campuses do not always think about recruitment as a factor in ensuring quality in online learning; however, there are implications for resource investment and ultimately for course and program retention. Assessment of recruitment practices includes both understanding whether or not recruitment practices are effective in generating new students or enrollments, as well as whether or not the recruited students can successfully complete the online courses or programs in which they are enrolled. This includes being able to track and compare the outcomes of recruitment practices through the recruitment funnel-from inquiry to applicant to enrollment-and being able to differentiate recruitment of online students from recruitment of campus-based students. It is also important to differentiate student outcomes, such as course completion, program retention, and graduation rates, based on recruitment practice for both online and campus-based students. This is sometimes done within the overall enrollment function of the college, but can also be done separately within a unit responsible for online learning.
Monroe Community College (MCC) has a long history of offering online courses and has many programs that are available online. Most recently, the campus has started to think more strategically about increasing enrollment in online programs and how they can reach new markets with those programs. The process they followed for those new programs is outlined below: 1. Identify the target market. Most degree programs now require a justification as to market demand for the degree. Colleges need to get more specific with this requirement by including market demand in a larger geographic area and being able to identify areas where student demand for the degree matches the need. MCC recently created two new online degree programs: Math and Sports Management. First, they identified areas in the state where math would be in high demand and landed on geographic regions around the SUNY university centers, specifically around Stony Brook University on Long Island. The potential audience for math degrees and programs were similar to the engineering programs at Stony Brook. They were able to geotarget students in a 10-mile radius that fit the demographic profile of their successful math students. As a result, MCC is offering more online math classes during key semesters than any other subject.
2. Campuses should measure program inquires, or leads, all the way through the enrollment funnel. Normally, the only way to determine the source of the lead was by direct questionnaire. It is now possible to align return on investment based on lead generation. Campuses can also dig deeper and analyze the demographic return versus the target in very specific ways. This was impossible with the more traditional marketing methods.
3. Lastly, use data to drive future enrollments. The "big data" generated by online marketing programs gives MCC the ability to even more specifically geotarget populations. Additionally, they are able to use more psychographic targeting strategies by aligning message delivery to the psychographic profiles of those students who traditionally respond to specific messages. It is critical that successful campaigns are analyzed and "deep data mined" so that the psychographic profile of the successful students can be aggregated and thereby targeted.

Standard 9.7-Program demonstrates compliance and review of accessibility standards (Section 508, etc.)
Access to higher education includes online courses and degree programs. The American Disabilities Act (ADA) of 1990, Section 504 of the Rehabilitation Act (1973), and the 1998 Amendment to Section 508 of the Rehabilitation Act articulate the foundational requirements for making online education accessible to all students. This standard aims to ensure accessibility compliance for students with disabilities in all aspects of the online program, including student support services, and online course materials, resources, and platforms.
Open SUNY developed an online course design rubric and process that addresses both the instructional design and accessibility of an online course. The Open SUNY Course Quality Review (OSCQR) rubric and process is aimed to assist online instructional designers and online faculty with improving the quality and accessibility of their online courses, while also providing a systemwide approach to collect data that informs faculty development and supports large-scale online course design review and refresh efforts systematically and consistently. The OSCQR rubric and process are currently being used by 56 SUNY institutions and the rubric was adopted by the OLC in November 2016 as their online course quality scorecard. The OSCQR Accessibility standards are based on the recommendations of SUNY's Office of General Counsel and address the legal considerations required to be compliant with Section 508 of the Rehabilitation Act (1973), the New York State Enterprise IT Policy NYS-P08-005, Section 504 of the Rehabilitation Act (1998), and the Americans with Disabilities Act.
At Niagara County Community College, the institution identified a need to enhance compliance with accessibility as an outcome of participating in the Open SUNY IR Process. As a result, they implemented a compliance program through a SUNY grant to ensure compliance with accessibility standards. The compliance program was completed over a one-year period and continues to be used for course reviews and new online course development. Some of the strategies employed include the following: • Implementation of Compliance Team consisting of eLearning staff and an online faculty member.
• Creation of resources posted to NCCC Online Learning Faculty Center website and Blackboard community.
• Creation of a compliance checklist used to review all online courses. The Compliance Team reviewed one online course for each online faculty member. The faculty members were provided with detailed resources to make updates.
• The team held "30 Minute Mentor Monday" sessions throughout the semester based on results from review findings. Each session revolved around a specific need such as making documents compliant (e.g., Word documents, PDF documents, captions for videos, PowerPoints, etc.). Sessions were recorded and sent to all online faculty.
• Creation of Accessibility Training in Blackboard. Online faculty were asked to complete the training and take a short quiz to complete the process.
• Implementation of Blackboard Ally campus wide to evaluate course content. A team was assembled that included accessibility services staff, faculty and instructional designers. This team was tasked with the development of communication and training materials.

Standard 9.8-Course evaluations collect feedback on the effectiveness of instruction in relation to faculty performance evaluations.
Course evaluations represent an opportunity for assessing quality in the online course from the students' perspective and provide feedback that can be helpful as an input into the performance evaluation process for faculty who teach online. For this to be meaningful and have an impact on quality, two conditions must be met. First, there should be a regular process for soliciting student input on course evaluations. This means course evaluations should be conducted for online courses on a regular basis and should have a high enough response rate to represent the perspectives of the students in the online courses. Second, there must be specific questions on the course evaluation that gathers relevant insights from students on the instruction provided by the faculty. The questions can vary from campus to campus or between academic units but should align with the expectations set for faculty who teach online within a particular campus or academic unit. The results of course evaluations can be looked at separately for individual courses or in aggregate if an instructor is teaching multiple courses within the evaluation period. What is important is that there is a deliberate strategy for collecting student input on the effectiveness of instruction in online courses that it is systematic across the institution, and that the feedback serves as one input into the continuous improvement of online teaching.

Standard 9.9-A process is in place and followed for the institutional assessment of faculty online teaching performance.
On most campuses, there is a process for the institutional assessment of faculty performance, and that can vary significantly from campus to campus or even between academic units for an individual institution in terms of how regularly it is done, what types of activities comprise the assessment and how the feedback or outcomes from such a process help to ensure quality online teaching. For any faculty assessment process to really impact the quality of online teaching, it must include a component for online teaching performance and be based on clear expectations for faculty who teach online courses. Expectations can be outlined in faculty position descriptions, faculty handbook, policy documents for online learning, or other formal documents related to faculty performance or online learning.
Assuming expectations are in place for online teaching performance, there should be a consistent process or mechanism for assessing faculty performance that includes a variety of inputs. Examples include, but are not limited to, student course evaluations, as discussed in the previous standard, peer evaluation by other faculty, self-assessment, and online course observations. The performance evaluation process should result in an individual development plan for the faculty based on all the inputs and against the established standards or expectations. Use of a quality review rubric can be helpful to ensure that performance standards are assessed consistently and fairly for all faculty who teach online.
Niagara County Community College has implemented two strategies to ensure the quality of their online courses and one specifically to assess faculty online teaching performance as listed below: Quality Peer Review Project: NCCC implemented an online quality peer-review initiative to ensure quality course design and compliance of online courses using the Open SUNY OSCQR rubric and process. The NCCC Quality Review and Refresh process is a faculty-driven, collegial process, which is voluntary and not a part of the faculty evaluation process. These reviews help maintain quality and compliance through this process of continuous improvement. NCCC currently has 51 faculty that teach online and 100% of those that have taught more than one semester have completed this review and refresh process. Additional information on the NCCC Quality Review Project can be found on the faculty website at: https://nccconlinelearning.com/nccc-quality-review-project/ Online Course Observations: In the fall 2016 semester, NCCC started online course observations as a part of the official faculty evaluation process. Through this process, 33 faculty had an online course observed. They will continue observing five faculty each fall and spring semester. One of the action items from the IR process was to include online courses and those teaching online in the faculty evaluation process.
1. The Online Learning Advisory Council was tasked to come up with a process and rubric for the completion of the online course observations. They learned about a rubric created by faculty at Penn State that based it is observation around the "Seven Principles for Good Practice in Higher Education." 2. The council modified the rubric to provide examples, terminology, and resources specific to NCCC and presented the recommended observation process and rubric to their campus VP of Academic Affairs and her Academic Council. The process and rubric were approved as the official faculty evaluation process for online courses.
3. The VP of Academic Affairs hired a consultant to conduct the observations with the faculty member. The consultant is an experienced online educator and administrator and was well received by those faculty scheduled for a course observation because of such experience. The consultant and faculty meet virtually and the faculty member provides a course tour so the consultant can get the required information to complete the course observation rubric and provide notes and recommendations. Once completed, the faculty member signs off on each principle. They can agree or disagree and add a rebuttal for each of the seven principles. Once the faculty member signs the document, copies of the observation go to the VP of Academic Affairs, the Coordinator of Online Learning, and the appropriate division chair. The eLearning Coordinator or VP of Academic Affairs meets with the faculty member to discuss the observation and create any necessary action plans.
4. There are 25 faculty members who had one of their courses go through both the quality peer review and had an online course observation. All 25 courses show an increase in successful course completion rates. The average increase for all 25 courses from fall 2016 through spring 2019 is 6.22%.

Standard 9.10-A process is in place and followed for the assessment of stakeholder (e.g., learners, faculty, staff) satisfaction with the online program.
It is important to regularly solicit feedback and suggestions and assess stakeholder satisfaction with online programs, broadly speaking, at an institution. Student, faculty, and staff perspectives should be considered when determining the overall effectiveness of an online program and planning improvements. Some of these efforts might be covered by practices employed to address Standard 9.4 depending on how the assessment mechanisms are designed. A regular survey cycle is recommended, but it also may be useful to solicit informal feedback on an ongoing basis. Sometimes the findings may reveal that some aspects of the delivery of the online program need to be improved, whereas in some cases, a low level of satisfaction may indicate that individuals are simply not aware of existing policies, procedures, or resources and a marketing campaign might be needed instead. Thus, even though this satisfaction assessment effort pertains to the overall online program, the assessment instrument needs to be designed in a way that collects information specific enough to produce actionable findings.

Standard 9.11-Course evaluations collect student feedback on quality of online course materials.
This standard represents another place where course evaluations can be a helpful tool for ensuring quality, specifically in regard to the appropriateness of electronic materials used in online courses. Similar to 9.8, there should be a regular process in place for conducting course evaluations for online courses and there must be questions that address instructional materials within the course website and as part of the learning activities. Examples include online content (text, video, etc.), online activities (quizzes, discussions, assignments, etc.), and any other materials used in the online course. Feedback from students on the relevance and quality of materials as well as how well they supported or contributed to their learning outcomes would be helpful to the instructor and would provide a basis for continuous improvement of the course materials in future course offerings. While it is helpful to look at evaluations on an individual course basis, it might also be helpful to look in aggregate across all online courses to identify opportunities for professional development or where policy changes might be helpful to ensure quality. Niagara County Community College collects student feedback on online course materials through their online course evaluation process using the Smart Evals online course evaluation software. Smart Evals is a program deployed through Blackboard so that all students can easily access the evaluations and provide feedback anonymously at the end of each semester. For online courses, there are specific questions to get feedback on course management and course design. Faculty can also add their own questions to get feedback to help with future course improvements. Faculty obtain access to the Smart Evals results at the end of the semester and use the tool to make continuous improvements to their online courses. If the faculty member is an adjunct, the program coordinator can review the evaluations and provide support and recommendations for course improvements. Full-time faculty evaluations are monitored by the office of Academic Affairs and are included in the promotion and tenure and full-time faculty evaluation cycle, set forth by our Chief Academic Officer.

Impact
The primary goal of the Open SUNY IR Process is to systematically increase the capacity of our SUNY campuses to ensure quality and success in online learning. In Spring 2017, we conducted an impact study, considering of focus groups, with representatives from 12 campuses that completed the IR Process to understand the extent to which the process was beneficial to their online learning initiatives. We learned that as a result of completing the Open SUNY IR Process, many of our campuses have made changes to resource allocations, organization or governance structures, policies and procedures, and/or technology infrastructure. Our campuses also explained that the IR Process, especially the Implementation Plan, was useful for Middle States accreditation reports, strategic planning, and other campus planning processes. The Open SUNY IR Process had benefits for the individuals involved as well as for the institutions. Participants in the study noted that the process furthered their own personal knowledge of what is necessary to facilitate quality online learning, gave them the opportunity to learn about perceptions of online learning from faculty/staff across campus, and enhanced their understanding of policies and procedures within different offices. They also explained that the process helped to inspire institutional culture change as it pertains to online learning. It encouraged a wide-range of campus constituents to develop a more comprehensive view of online learning quality and determine what actions and/or resources are needed to make improvements.
The Open SUNY IR Process, and the OLC Quality Scorecard Evaluation & Assessment Standards in particular, provide campuses with a systematic framework to ensure quality online learning. After the Open SUNY IR Process is completed, these particular standards help to ensure continuous improvement. It is difficult to parse out the impact of any one particular evaluation and assessment practice, but the implementation of a comprehensive set of practices embedded across campus indicates the value placed on ensuring online learning quality.

Conclusion
When Open SUNY was launched in 2014, one of the main priorities was to ensure quality of online learning at the course, program, and institution level. Using the OLC Quality Scorecard, the Open SUNY IR Process, has helped to support positive campus culture change regarding online learning at numerous SUNY campuses. As described in this paper, the Open SUNY IR Process is an assessment mechanism by itself, but the Evaluation & Assessment Standards promote the development of sustainable, ongoing, and comprehensive continuous improvement efforts. We are confident that campuses that have completed the Open SUNY IR Process are continually striving to offer optimal online learning experiences for their students.