THE SLOAN-C PILLARS AND BOUNDARY OBJECTS AS A FRAMEWORK FOR EVALUATING BLENDED LEARNING

The authors contend that blended learning represents a boundary object; a construct that brings together constituencies from a variety of backgrounds with each of these cohorts defining the object somewhat differently. The Sloan-C Pillars (learning effectiveness, access, cost effectiveness, student satisfaction, and faculty satisfaction) provide a foundation for the evaluation of asynchronous learning networks that works equally well for the evaluation of blended learning environments. The Pillars and a simplified model of a learning system, focus on inputs, processes, and outputs, and provide the framework for a case study of blended learning design and evaluation in a 500-student section of an Introductory Psychology course. Results of a multi-method evaluation of this course indicated very high levels of both learning effectiveness and student satisfaction. The article concludes with the suggestion that blended learning may represent a black swan, a high-impact, unpredicted, and rare event that highlights the limitations of our ability to reliably predict the future in any arena, including online learning environments.

However, establishing an effective classification scheme for blended learning is important because without such metrics gauging its impact becomes difficult, if not impossible. Fortunately there is work in this area. For instance, Graham [2] approaches the problem by organizing blending from a value added perspective. He examines instructional blends and finds organic categories that explain how they interact with the learning environment. His enabling blend responds to the current student generation in higher education and its needs for access, convenience, and flexibility. His enhancing blend recognizes the potential for blended learning as a means to augment and improve learning arrangements while transforming blends change, in a fundamental way, the teaching and learning process and the pedagogy underlying them.
Picciano, [3] in proposing his multimodal model of blended learning, suggests that the motivation for developing the modality lies in its potential to: assist students, support faculty, maximize facilities, and respond to social needs. He goes on to argue that certain strategic and logistical elements become vital to the process: infrastructure, faculty development, instructional design, and student support. Through this foreshadowing he proposes his blending with purpose model that generates its components from the multiple elements: course content, social and emotional development, the dialectic and questioning, higher order student assessment, collaboration, co-creation of content and reflective practice.

II. BOUNDARY OBJECTS
Both the Graham and Picciano evaluation models [2,3] resonate with the work of the sociologist Susan Leigh Star on boundary objects: elements or constructs that bring constituencies together but find each cohort defining the object differently. Consider this definition by Bowker and Starr [4].
"[boundary objects]…are those objects that both inhabit several communities of practice and satisfy the informational requirements of each of them. Boundary objects are thus both plastic enough to adapt to local needs and constraints of the several parties employing them, yet robust enough to maintain a common identity across sites. They are weakly structured in common use and become strongly structured in individual-site use. These objects may be abstract or concrete… Such objects have different meanings in different social worlds but their structure is common enough to more than one world to make them recognizable, a means of translation. The creation and management of boundary objects is a key process in developing and maintaining coherence across intersecting communities." The Sloan-C Pillars [5] constitute another framework that can be considered a boundary object because of their appropriation by different communities for the purpose of evaluation and assessment. Although initially developed as a basis for assessment of Asynchronous Learning Networks (ALNs) [6], the pillars have also been used with traditional on-campus courses [7], and are increasingly being used as a framework for evaluating blended learning environments [5,6]. This paper will describe and provide a case example of how the Sloan-C Pillars can be used as a framework for evaluating blended learning environments.

III. EVALUATING LEARNING SYSTEMS
Both traditional and online learning systems have their own traditions of evaluation and assessment. Perhaps the simplest model of a learning system contains inputs, processes, and outputs (see Figure 1). An input to a learning system is something that is put into it in order to achieve a particular outcome.
Some input examples include things like the characteristics or prior knowledge of the learners, the attitudes of the faculty, or the resources available for course redesign. Processes are actions within the system that are designed to bring about certain changes or outcomes. Examples of outcomes might be grades or graduation rates. While many of the inputs and desired outcomes are shared between the online and on-campus communities, processes used to reach the outcomes tend to vary widely.
Blended learning incorporates processes of both online and face-to-face instruction. The combining of these defines new processes themselves that can often require more complex ways of thinking about evaluation and assessment. For example, in a face-to-face classroom setting, faculty can assess student satisfaction by observing students' in-class engagement and non-verbal cues. In an online setting student engagement might be assessed by observing the number and quality of posts to a discussion board or students' promptness in completing assignments. In a blended learning environment, faculty can leverage the "best of both worlds" by combining such face-to-face observations with robust tracking tools that are part of many course management systems. For example, faculty might track student submissions of weekly online quizzes and send e-mails automatically to students who fail to complete such assignments in a timely manner. In conjunction with face-to-face observations of these same students in the classroom setting, faculty are more fully equipped to evaluate and assess student progress in the course.

IV. THE SLOAN-C PILLARS
The Sloan-C Pillars provide a framework for considering inputs, outputs, and processes involved in the quality implementation of asynchronous learning networks (ALNs) [6]. This framework can also be used to evaluate the implementation of a blended learning system. The five pillars (see Figure 2) are (1) learning effectiveness, (2) access, (3) cost effectiveness, (4) student satisfaction, and (5) faculty satisfaction. These pillars are categories for helping us to think about important inputs, processes, and outputs that impact blended learning environments. The framework is a good fit for assessing implementation because the first three pillars in the framework correspond to the three primary reasons why people adopt blended learning: (1) to improve learning effectiveness, (2) to increase access and flexibility, and (3) to increase cost effectiveness [2]. Figure 3 is a template provided on the Sloan-C wiki that shows the five pillars each divided into inputs, processes, and outputs. This template can be used to begin documenting what variables should be measured in each dimension to effectively assess the quality of a blended learning implementation. Additionally, common measures between blended, online, and face-to-face modalities provide a platform for comparison and continual improvement.

A. Measuring Outputs
Identifying and measuring the outcomes of a blended learning system is the most important task. While both direct and indirect measures of an outcome can be collected, there is an increasing push by accrediting agencies for universities to gather direct evidence of student learning (Middle States Commission on Higher Education, 2003, and SACS). For example, when measuring learning effectiveness, a direct measure of learning could be scores on an exam, a course project, or a performance assessment. Indirect measures of learning could be course evaluations, percent of time spent in active learning, or a measure of student engagement. Typically the most important direct outcomes of learning effectiveness are the growth in knowledge, skills, and dispositions.

B. Measuring Inputs
Measuring inputs to a blended learning system can also be important, especially if outcome comparisons will be made with online or face-to-face courses. Inputs related to learning effectiveness might include student characteristics, learning styles, technology competencies, etc. Knowing the inputs will help in understanding whether or not meaningful comparisons can be made on learning outcomes. Also, measuring inputs can provide needed information for designing appropriate processes. For example, if you know that many students enrolling in blended learning courses have very low technology skills, you can compensate for that up front with tutorials and other help.

C. Evaluating Processes
The processes in a blended learning system entail the design and implementation of the course or program itself. Similar to all complex learning environments, it is important to have a mechanism for formative evaluation built into the system so that continual and recursive improvements can be made. Rarely will a blended learning design be perfect in the first iteration. As instructors select the blend that meets their purposes (e.g., learning, access, and/or cost effectiveness) they should collect formative data so that improvements can be made that are based on data-driven evidence that facilitates incremental improvement.
The remainder of this paper will use a case study to demonstrate how assessment and evaluation of blended learning in a large enrollment course has led to continual improvements.

A. Description of the Course
Blended learning is a viable pedagogical option for faculty regardless of course enrollment. Although most blended learning courses typically have enrollments well below 100 students, this case study will demonstrate the effectiveness of a blended learning approach to the teaching of an Introductory Psychology course at San Diego State University (SDSU) with an enrollment of 500 students. Such largeenrollment introductory courses have become increasingly common at SDSU and other large state universities, as these institutions struggle with the convergence of growing enrollments and shrinking budgets. Introductory Psychology at SDSU has been taught in a 500-student lecture format since the Fall 2004 semester. The blended learning Introductory Psychology course was first offered during the Fall 2006 semester.
The Sloan-C Pillars provided the foundation for redesigning and evaluating this large-enrollment blended learning course. Specifically, the impetus for redesigning the course in a blended learning format was the desire to improve Learning Effectiveness (LE). As the redesigned course evolved, however, the benefits for students with respect to both Access (A) and Student Satisfaction (SS) also became quite clear.
The blended learning Introductory Psychology course at SDSU represents an enhancing blend within Graham's [2] model, with a focus on blended learning as a means to augment and improve learning arrangements for students. The redesigned course replaces 45% of traditional face-to-face instruction (lecture, demonstrations, and "clicker" activities) with synchronous sessions delivered via the Wimba Live Classroom, a web-conferencing tool for education that is integrated into the SDSU Blackboard system. Synchronous online sessions make this course design relatively unique, as most blended learning courses employ asynchronous online activities [8]. Typically these synchronous online sessions, "attended" live by approximately 30% of the 500 students enrolled in the course, involve a variety of activities, including mini-lectures (less than 15-20 minutes at a time), demonstrations, videos, and polling questions to assess student learning and student opinions. All synchronous online sessions are archived (recorded) and made available to students for the remainder of the semester.
The face-to-face class meetings for the blended learning Introductory Psychology course include numerous course design strategies to engage the learner and increase active learning in the large lecture hall. For instance, a student response system (a.k.a. "clickers") is utilized extensively in these face-to-face class meetings. "Clicker" activities range from (1) ConceptCheck questions designed to assess student learning of lecture content to (2) demonstrations designed to bring course content to life in the classroom setting (e.g., demonstrations of phenomena related to memory) to (3) the use of Mazur's [9] Peer Instruction approach to the assessment of student learning in the classroom. This Peer Instruction approach to the use of "clickers" involves the following sequence of activities: (1) pose a conceptual multiple choice item with at least two commonly selected answers, (2) reveal the students' responses to the item, but do not reveal the correct response, (3) instruct the students to collaborate with their classmates in order to arrive at the correct response, (4) re-ask the same multiple choice item, and (5) reveal the students' responses, along with the correct response. These various "clicker" activities provide a rich way to index Learning Effectiveness (LE)-well-designed "clicker" assessments are reflective, contextual, and authentic. Student feedback regarding the use of "clickers" in these ways in this Introductory Psychology course has been extremely positive.

B. Evaluation Protocol
In conjunction with the initial roll-out of this blended learning Introductory Psychology course during the Fall 2006 semester, a formal evaluation protocol was developed to assess the effects of this course redesign on a number of variables, including student learning (LE), student access (A), and student satisfaction (SS) [10]. Thus, the Sloan-C Pillars provided the foundation not only for the redesign of this course but also for the evaluation of this redesign.  (6) course grades. This formal evaluation protocol has been renamed the SDSU Evaluation Toolkit and will be shared with other campuses within the California State University (CSU) system.
In terms of the simplified model of a learning system outlined above, much of the focus of the evaluation of this blended learning Introductory Psychology course was on outputs or course outcomes. Data from three of the six parts of the evaluation protocol are reviewed here: (1) the IDEA Student Ratings of Instruction survey, (2) end-of-semester departmental course evaluations, and (3) course grades. In each case, comparisons are presented between the blended learning Introductory Psychology course described above and a traditional, face-to-face version of the same course taught by the same instructor. This traditional, face-to-face course covered the same content as the blended learning course, with the major difference being the replacement of approximately 45% of normal "seat time" with the online activities described above.

C. IDEA Survey Data
Data from the IDEA Students Ratings of Instruction survey point directly to both Learning Effectiveness (LE) and Student Satisfaction (SS), permitting comparisons between the blended learning (BL) and traditional, face-to-face (F2F) sections of Introductory Psychology.

D. Departmental Course Evaluation Data
Data were also collected via standard end-of-semester departmental course evaluations administered by the SDSU Department of Psychology. These data assess directly the Sloan-C Pillars of Learning Effectiveness (LE) and Student Satisfaction (SS).    Interestingly, however, when an even longer viewpoint is taken of these course grade comparisons, evidence of the convergence between the BL and F2F sections noted above for both the IDEA ratings and the departmental course evaluation ratings does emerge. Figure 5 presents the latest summary data for all four semesters during which both the BL and F2F sections of Introductory Psychology have been offered. These data paint a more equal picture of the grades achieved by students in these two courses.

F. Lessons Learned
The experiences of the instructor for Introductory Psychology at San Diego State University highlight a number of significant lessons learned for the adoption of a blended learning approach to teaching a large enrollment course. First, it is abundantly clear that the Sloan-C Pillars provide an indispensable framework for both redesigning and evaluating large-enrollment blended learning courses. The Pillars focus on (1) learning effectiveness, (2) access, (3) cost effectiveness, (4) student satisfaction, and (5) faculty satisfaction; all of which represent targets for course redesign efforts and after such redesign efforts have taken place, targets for formal evaluation of these efforts. Second, assessment and evaluation of blended learning can lead to continual improvements within blended learning courses. Much of the data reviewed above point to continued refinements over several semesters that appear to improve student ratings of learning effectiveness and student satisfaction. Third, in a related way, data from such evaluations of blended learning inform the iterative process that is course (re)design. Armed with these sorts of data, instructors and other important stakeholders can target for continuous improvement parts of their current course designs that are not effectively meeting goals related to the Sloan-C Pillars. Finally, the experiences at San Diego State with this Introductory Psychology course point to the conclusion that blended learning occupies the "Sweet Spot" for maximizing these Sloan-C Pillars -learning effectiveness, access, cost effectiveness, student satisfaction, and faculty satisfaction. Designed well and evaluated thoroughly, blended learning can indeed offer the best of both worlds -the face-to-face and online learning environments.

VI. CONCLUSION
The Sloan-C Pillars, when viewed as boundary objects for evaluation bridge the online and face-to-face environments by integrating multiple learning domains and changing the nature of evidence by expanding assessment options. The pillars respond to the current realization that valid evaluation and assessment evidence is evolving from a posture of objectivity to one of reflectivity; from non-contextual frames to contextual examples and from constructed protocols to ones that reflect organic authenticity. Blended learning requires the interaction of multiple communities of practice (face-to-face, asynchronous learning environments, instructional design and pedagogy, learning theory, and assessment, for example). The pillars intersect with all these cohorts and serve as the mediating elements for new and evolving educational environments.
Student satisfaction is an excellent prototype showing how the pillars become boundary objects in blended learning, providing value added community intersections. Certainly satisfaction responds to students becoming active participants in the evaluation of their educational experiences and their own assessment of learning. They express their satisfaction or lack of it in a number of ways; for instance, responding to evaluation forms that index multiple characteristics of their courses and instructors. On these surveys their comments provide the basis for an evaluative narrative. Rating courses and instructors is by no means the only way students express themselves about quality. They vote by choosing to register for or avoid courses taught by certain instructors. They publish their own evaluations through a number of channels including student government and social organizations. In addition, every campus has a functioning "gossip" network where instructors' reputations are passed along multiple student cohorts. Most recently, the emergence of ratemyprofessors.com takes the expression of student evaluation of instruction to another level. For better or worse, that website creates a world wide forum for students to evaluate courses and instructors and communicate their ratings to an audience of astounding size. Course evaluations radiate to social networking sites, such as Facebook and MySpace. Students are able to "Twitter" friends on their cell phones, telling them what is happening in their courses in real time.
YouTube serves as a student satisfaction forum where students have posted videos of instructors in the act of teaching. Although this paragraph focuses on student satisfaction, it seems clear that each pillar may be deconstructed into a boundary object.
Another reasonable assumption is that the concept of blended learning itself can be thought of as a boundary object that is used by many different communities in slightly different ways. For example, from the perspective of the distance learning community, blended learning is a way of describing the addition of face-to-face (f2f) encounters, like a program orientation or a student teaching experience that compliment the core instruction that is done online. For the traditional campus-based community, the concept of blended learning often implies the movement towards the effective use of technology in teaching, accompanied by a reduction of in-class seat time. In the corporate training world, blended learning can denote the innovative use of many different kinds of technologies and learning modalities, including face-to-face classroom instruction and self-paced independent online learning. Despite some of the definitional fuzziness (that drives academics crazy), there is enough plasticity in the blended learning concept that it can be of benefit to multiple communities.
Boundary objects, communities of practice and blended learning share an important interactive relationship. Communities of practice have individuals who reside at the core of the practice, as well as individuals who function at the boundary of the practice. This seems to be the case in blended learning and its evaluation. Importantly, these boundary individuals have broad interests-participating in multiple communities, but not locating themselves at the core of any of them. They play an extremely important role, however, in the development of emerging educational trends, such as blended learning; serving to transfer and pollinate ideas across communities, eventually evolving into boundary objects themselves. Those within the field of instructional technology seem particularly suited to this role because they interact regularly with multiple communities of practice, especially in blended learning.
Finally, to put this study in perspective, one must understand that blended learning's impact on higher education may well be what Taleb [12] identifies as a black swan: a life-transforming event, unpredicted, that results from the continuing uncertainty in our environment. The fundamental point is that the effect of asynchronous learning networks was not predicted by any empirical models. Early adopters will offer historical perspectives attempting to backfill a narrative about the planned and systemic nature of online learning; however, its beginnings were spontaneous, unpredictable, and chaotic. (Nobody predicted Google. Nobody predicted Harry Potter). Consider Steven Johnson's characterization of a black swan that is relevant to blended learning.
"History has its epic thresholds where the world is transformed in a matter of minutes-a leader is assassinated, a volcano erupts, a constitution is ratified. But there are other, smaller, turning points that are no less important. A hundred disparate historical trends converge on a single, modest act-some unknown person unscrews the handle of a pump on a street in a bustling city-and in the years and decades that follow, a thousand changes ripple out from that simple act. It's not that the world is changed instantly; the change itself takes many years to become visible. But the change is no less momentous for its quiet evolution." [13].
Like all black swans, the impact of blended learning is potentially monumental-permanently changing how students interact with higher education and the manner in which they express their satisfaction with it. As this current black swan plays out there may be another looming on the horizon that is equally unforeseen. However, for the present, blended learning has established a culture of sustainability in higher education, providing accessibility to the most diverse student population in history; with the unanticipated side effects of raising students' expectations and their standards for learning as well. As a result the Sloan-C pillars are emerging as an authentic evaluation and assessment framework because they resonate with both the face to face and online modalities-maximizing their potential for effective teaching and learning.