Using Importance-Performance Analysis to Guide Instructional Design of Experiential Learning Activities

Designing experiential learning activities requires an instructor to think about what they want the students to learn. Using importance-performance analysis can assist with the instructional design of the activities. This exploratory study used importance-performance analysis in an online introduction to criminology course. There is limited research on experiential learning in online courses as well as empirical data to assist with the instructional design of the experiential learning activities. The primary goal of this article is to demonstrate the use of importanceperformance analysis to guide the instructional design of experiential learning activities.


Introduction
Experiential learning is a philosophy that draws on the work from prominent 20 th century scholars such as John Dewey, Kurt Lewin, Jean Piaget and others.This is a learning experience or process that allows students to apply classroom concepts to real-world situations (Kolb & Kolb, 2005).Experiential learning can take many forms including service learning, wildernessbased adventure programs, and professional development activities (Carver, 1997).Traditional experiential learning typically follows Kolb's experiential learning theory (Kolb & Lewis, 1986).This theory provides a holistic model for the learning process and a multilinear model of adult development (Kolb, Boyatzis, & Mainemelis, 2001).Kolb's experiential learning theory is based on a four-stage learning cycle: concrete experience, reflective observation, abstract conceptualization, and active experimentation.Effective learning has been observed when learners progress through the cycle.
The benefits of experiential learning have been well established (Waldner, McGory, & Widener, 2010), as research has shown that students in experiential learning courses had more positive course evaluations (Markus, Howard, & King, 1993).Other research supports positive beliefs and values toward service and community (Cohen & Kinsey, 1994), higher academic achievement (Boss, 1994), and positive impact on personal, attitudinal, moral, social, and cognitive outcomes (Giles & Eyler, 1994).
Higher education enrollment is increasing according to the Institute of Education Statistics (IES).There are approximately 17 million students enrolled in higher education institutions in the United States; that number is predicted to jump to over 20 million by 2021.Online enrollment has increased from 1.2 million students in 2002 to 7.1 million students in 2012 (Allen & Seaman, 2014).Online course offerings are also growing to accommodate a diverse student audience.According to the National Center for Educational Statistics from 2000-2012, distance education program enrollment went from 2% in 2000 to 16.3% in 2012 (The Condition of Education, 2014).
As more courses and programs offer online options, it also becomes important to integrate and research experiential learning in online courses.Waldner, McGorry, and Widner (2012) noted very limited research with regards to online experiential learning.The few empirical studies that have been conducted on online experiential learning have identified common challenges and best practices but have conducted case study or descriptive research without reporting results.In addition, many of these studies failed to compare what is important to students to how they perceived their performance.

Kolb's Experiential Learning Theory
John Dewey developed a philosophy of education that describes the process of infusing experiences into the learning environment.This philosophy has become the foundation for experiential learning theory.Through an extensive review of Dewey's philosophy, Giles and Eyler (1994) identified nine components to consider for experiential learning: continuity of experience, citizenship, interaction, inquiry, reflection, educative products, concrete and abstract knowledge, the great community, and democracy.By implementing experiential learning theory, students experience both process and outcomes (Carver, 1997).Students are able to apply knowledge and skills gained in the classroom through real-world experience.One of the key factors in Dewey's philosophy is creating individual learning experiences through reflection (Deans, 1999).
Kolb further developed the experiential learning theory based on Dewey's philosophy and Piaget's psychology for experiential learning implementation.Kolb's experiential learning theory includes four learning activities: concrete experience, reflective observation, abstract conceptualization, and active experimentation (Petkus, 2000;Kolb & Lewis, 1986).Experiential learning theory is a holistic model of learning processes and a model for adult development.
The experiential learning theory is based on a learning cycle where two activities are for grasping experience: concrete experience and abstraction conceptualization.The other two activities are for transforming that experience: reflective observation and active experimentation.
Through experiential learning, the learner experiences all four activities.Concrete experience is the basis for reflective observation (Kolb & Kolb, 2009).These reflections are conformed into abstract conceptualizations from which students may draw new conclusions.These conclusions are then integrated into active experimentation.Together, these activities when integrated together create a positive experiential learning experience for students, community partners, and faculty (Kolb & Kolb, 2009).
Experiential learning involves teaching and learning strategies that can include community service or activities, hands on experience, and critical reflection (Waldner et al., 2010) to course concepts (McGorry, 2012).Research has supported the benefits of experiential learning.Eyler, Giles, Stenson, and Gray, (2001) identified 135 articles on experiential learning where 132 articles reported positive or neutral consequences.Faculty benefits included satisfaction in student learning and research; student benefits included critical analysis, application of knowledge in practical settings, and improved course satisfaction (Eyler et al., 2001).

Online Experiential Learning
Online experiential learning is experiential learning that is incorporated into courses delivered in an online format.Malvey, Hamby, and Fottler (2006) defines it as . . .an electronic form of experiential education and [it] incorporates electronically supported experiential learning.It is delivered online and uses the Internet and state of the art technologies that permit students, faculty, and community partners to collaborate at a distance in an organized, focused, experiential service learning activity, which simultaneously promotes civic responsibility and meets community needs (p.187).Waldner et al. (2012) conducted a thorough review of experiential learning literature in an online environment using a three-pronged approach.The approach searched for literature using the peer review database ProQuest Central, an Internet search using Google and Google Scholar, and a search of journals dedicated to experiential learning.Only 18 journal articles and one book were found to be directly related to online experiential learning.The result of this extensive literature review established that there is little empirical research related to online experiential learning.
The studies in online experiential learning yielded the following results regarding best practices: • Technology • Communication • Course design Waldner et al. (2012) stated that specific technology should be identified to facilitate communication between the instructor, students, and potential community partner(s), and clear communication expectations should be established prior to course delivery and be included in the course design for students.Along with technology and communication expectations, course design should be taken into account since the integration of online experiential learning can increase instructor workload (Waldner et al., 2012).All parties including outside agencies, instructors and students, must be trained on the communication technology and have the technology tested prior to implementation.The instructor should partner with instructional designers for course design and technology support (Waldner et al., 2010).Course design should be "solid".Instructors should evaluate courses according to course design rubrics that have been established and validated to ensure there is a connection between the activities and the course goals (Waldner et al., 2012).The course should include clear instructions and explanations of reflective assignments, gather student input on the course and activities, and ensure communication expectations are included within the course design (Waldner et al., 2010;Waldner et al., 2012).In both cases, the students highly valued solving authentic real-world problems as noted on end-of-course evaluations.
McGorry (2012) conducted a case study research with two online marketing classes and two traditional marketing classes.This was a comparison study to determine if online experiential learning delivered the same benefits as traditional experiential learning based on self-reported perceptions of experiential learning outcomes using the Service Learning Benefit scale (SELEB).SELEB measures student perceptions of experiential learning activities.From this study, the author concluded there was no significant difference between the online group and the traditional group with regards to student perceptions but further studies should explore student grades and perceptions (McGorry, 2012).

Student Perceptions and IPA
Student perceptions often have been used to gauge perceived achievement instead of measuring actual achievement (Karns, 2005).Students frequently are asked questions regarding whether the experiential activity deepened their learning, yet they typically are not asked what they feel is important prior to the experiential learning activity.Markus et al. (1993) researched student perceptions towards experiential learning and then compared the mean grade between the control class and the class that integrated experiential learning.Overall the mean grades in the experiential learning class were significantly higher.
Importance-performance analysis (IPA) has been applied in higher education in various contexts.It has typically been utilized in education for marketing and tourism courses (Oh, 2001).IPA has also been used for student evaluation of teaching and course design (Huybers, 2014;Lewis, 2004) The importance survey is designed to be implemented at the beginning of the semester and could also be considered a pre-survey.The performance survey is deployed at the end of the semester is considered a post-survey.For the purposes of this paper, the surveys will be described in terms of importance and performance (pre-/post-).The SELEB importanceperformance survey consist of four factors: practical skills, interpersonal skills, citizenship, and personal responsibility.Twenty variables are associated with the four factors as seen in Table 1 (Toncar, Reid, Burns, Anderson, & Nguyen, 2006).The central distribution for importance and performance are calculated (means or median) and the score is used to calculate the axis for a two-dimensional matrix called an action grid (Siniscalchi, Beale, & Fortuna, 2008).Figure 1 depicts what Ortinau, Bush, Bush, and Twible (1989) have defined for each quadrant: Data in the upper right quadrant (high importance/high performance) is "keep up the good work"; data landing in the upper and lower left quadrants (low importance/high performance and low importance/low performance) suggests overkill or low priority.Data landing in the lower right quadrant (high importance/low performance) indicates importance outweighs the ability.Importance-performance surveys have been used to guide changes in service industries but in this case, importance-performance is use to analyze the outcomes of the design of the experiential learning activity.After a review of the literature, there is a clear gap on effectively integrating online experiential learning into online asynchronous courses and whether students perceive online experiential learning to be beneficial to their learning based on the importance of the skills learned.Also note that none of the online experiential studies discussed the actual results of surveys delivered nor included the surveys in the appendix.

Research Strategy
This quasi-experimental study explored students' perceptions regarding online experiential learning in terms of what they perceived to be important and how they performed.The study utilized an importance-performance analysis (IPA) method.

Participants
This study used an Introduction to Criminology course with majors and non-majors, which was delivered in fall 2014 during a fifteen-week semester.This is an undergraduate course at a mid-sized southeastern university in the United States.The asynchronous criminology course had 29 students.Students were recruited to participate in this study on a voluntary basis at the beginning of the semester.Students who chose to participate in the study, were assigned a random identification number which allowed the researcher to anonymously match survey data with grades for statistical analysis.Fifteen students completed both the importance and performance survey with a 51.7% response rate.This response rate falls in line with the recommended response rate for online surveys (Baruch & Holtom, 2008).Student demographics showed that 14 students age 18-30 and one student 31-50.There were 7 males and 8 females.There were a variety of student classifications including 2 freshmen, 3 sophomores, 6 juniors, and 4 seniors.Twelve students stated they were comfortable using computers and 3 who identified themselves as advanced.A majority of the students had taken online classes with 8 having taken 5-10 courses.Two students had taken 1-2, four had taken 3-4, and one had taken more than 10 online courses.

Materials and Procedure
The SELEB survey was used to gauge student perceptions of the experiential learning and questions relating to demographic data were added to the electronic survey.The SELEB scale is a self-report measure of student perceptions of the benefits of service learning.According to Toncar et al. (2006), the SELEB survey has been identified as a valid and reliable tool for assessing outcomes of student perceptions, value of experiential learning efforts from the student viewpoint, and evaluate the extent to which the experiential learning activity contributed to the learning objectives of the course.The Cronbach's alpha of the SELEB scale was calculated at .82 (McGorry, 2012).Toncar, et al.'s evaluation (2006) suggested the SELEB scale should be used to assess the benefits of experiential learning.
The importance SELEB scale includes a 12-item Likert scale on the importance of incorporating various items such as civic duty, connecting theory to practice, and workplace skills in course activities.This is given to the students at the beginning of the semester.The scale is 1 -7 with 1 being not important and 7 being very important.The performance SELEB scale is a Likert 20-item scale with 1 indicating the project was not applicable and 7 being very applicable.The 20-item performance SELEB is deployed to the students at the end of the semester once the experiential learning activity had concluded.The instructor included a summary of the study on Blackboard ©, the Learning Management System (LMS) at the university, for the students.The 12-Likert scale importance SELEB survey and questions pertaining to demographic information was given electronically to the students at the beginning of the semester.The survey included class status, age, sex, major, previous experience with online learning, and classification of "on-campus" or "distance education" student.At the end of the semester, the 24-Likert questions performance SELEB was deployed electronically to the participants that gauged student perceptions towards the criminal justice in action activity, which was the experiential learning activity that spanned the semester, at the end of the semester.

Criminal Justice in Action
For the experiential learning project, students were asked to identify a location in their community to visit and observe.The students were required to conduct two one-hour-long observations.Prior to community observations, students wrote an essay which included expectations and intentions based on the class materials.Students also completed the importance-SELEB scale to assess their perceptions towards experiential "community-based" learning.Once students completed their observations, they were paired with another student in the class to create a final presentation of their findings and explain how the observations related to the class materials.The students created a blog, a letter to the editor, or a voice-over PowerPoint presentation.Finally, students were asked to compose a critical reflection essay on their experience and complete the performance-SELEB survey to assess their perceptions towards the "community-based" experiential learning.

Data Analysis
IPA was used to analyze student perceptions and performance.This data was used to construct a two-dimensional matrix where importance is depicted on the y-axis and performance is depicted on the x-axis.The gridlines for the IPA graphs are determined by the overall mean of Importance = 6.02) and Performance ( = 4.85).
A paired-samples t-test was conducted to evaluate the impact of the experiential learning assignment using importance-performance scale.There was a significant difference between importance and performance for all associated variables except for the Applying Knowledge to the Real World.There was a statistically significant decrease in the performance scores.Table 2 displays the mean scores for the four main factors of the SELEB survey.There was a decrease in the mean from importance to performance.A paired-samples t-test was conducted to analyze the four main factors comparing the importance to the performance.Practical skills resulted in a decrease from importance (M = 6.033,SD = .949)to performance (M = 4.748, SD = 1.858), t (14) = -3.335,p = .005(two-tailed).The mean decrease in the practical skills scores was -1.293 with a 95% confidence interval ranging from .458 to 2.112.The eta squared statistic (.44) indicated a moderate effect size.Interpersonal skills resulted in a decrease from importance (M = 5.952, SD = 1.579) to performance (M = 4.446, SD = 1.544), t (13) = -4.415,p = .001(two-tailed).The mean decrease in the interpersonal scores was -1.506 with a 95% confidence interval ranging from .769 to 2.242.The eta squared statistic (.60) indicated a moderate effect size.Citizenship skills resulted in a decrease from importance (M = 6.044,SD = .871)to performance (M = 4.60, SD = 1.929), t (14) = -3.3511,p = .003(two-tailed).The mean decrease in the citizenship scores was -1.444 with a 95% confidence interval ranging from .562 to 2.326.The eta squared statistic (.47) indicated a moderate effect size.Personal responsibility skills resulted in a decrease from importance (M = 5.892, SD = .1.583) to performance (M = 4.446, SD = 1.544), t (13) = -4.206,p = .001(two-tailed).The mean decrease in the personal responsibility scores was -1.446 with a 95% confidence interval ranging from .703 to 2.189.The eta squared statistic (.58) indicated a moderate effect size.   2 illustrates the associated variables from the importance and performance SELEB scale on an importance-performance quadrant.The x and y axis is determined by determining the overall mean score for both importance and performance (Huybers, 2014).The overall importance mean was 6.022 and overall performance was 4.854.Applying real world skills landed in the upper right quadrant, which suggests that this activity met both the importance and performance for students.Workplace and leadership skills fall between the upper left (concentrate here) and lower left (low priority) quadrants.All other skills fell in the low priority quadrant.
Figure 2. Results from the importance-performance mean.

Discussion and Conclusion
The results in this experiential learning activity demonstrated that students were able to apply classroom knowledge to real world activities.Although there was a decrease in the mean for all associated variables of the SELEB survey, this was an introductory class that included both majors and non-majors.The instructor's main objective in this introductory course was to connect theory with practice; thus the experiential learning activity was successful.Many of the studies conducted in online experiential learning thus far have neither included the results of their surveys nor used an instrument designed for experiential/service learning.The results of this exploratory study can assist with the design of experiential learning activities depending on the goals of the instructor and the activity/course.Since this course was an introductory course that included majors and non-majors, students may not have found value in the other associated variables.The goal of the instructor was for students to connect classroom concepts to the real world and this was successful.

Limitations and Future Research
The use of importance-performance analysis can be an important evaluation tool to guide the design of experiential learning whether in online or face-to-face environments.This study had several limitations.One limitation was the sample size.This study applied to only one introductory class and students had the option to participate, which likely reduced the number of participants.Sample size could be increased by using multiple sections of the same course.Another limitation was the fact that the course was an introductory course that included both majors and non-majors; therefore, students may not have perceived the experiential learning activity to be beneficial.Perhaps future studies could compare majors and non-majors to determine if perceptions change.Further studies should be conducted across disciplines to make the findings more generalizable.

Figure 1 .
Figure 1.Importance-performance analysis (IPA) Table 1 displays the survey items and how each are associated with each main factor.There are four factors associated with the twenty variables.

Table 2
Mean comparison of the SELEB factors

Table 3 t
-test importance-performance of SELEB factorsTable4depicts the analysis of the two-tailed t-test for the variables of the four main factors.Each item is labeled which is depicted in Figure2on the IPA quadrant.There is a decrease in the mean scores from importance to performance.