Studying Cultural Relevance in Online Courses : A Thematic Inquiry

This study investigated a set of online reading courses at the undergraduate and graduate levels in a university’s college of education. The courses were thought to integrate culturally relevant pedagogy (CRP), and the study’s intent was to determine the extent to which they actually did so. The goals of the study were to determine whether CRP was emphasized in course descriptions, informational materials, activities, assessments, and rubrics, and to determine whether there was alignment between CRP-related descriptions, activities, assessments, and rubrics. Findings indicate that CRP was not strongly emphasized or well aligned in most courses. Indeed, across most or all courses, we found the following: a lack of emphasis on democratic classrooms; a lack of alignment between CRP objectives, assignments, and rubrics; a lack of evidence that depth of CRP knowledge was required across courses; problems with course design; and a lack of emphasis on identity and cultural competence. The study’s findings highlight the importance of a fully developed syllabus for adjunct/part-time faculty who rely heavily on the syllabus; emphasize the importance of strong syllabi for curriculum and course integrity (particularly when programs move fully online); and indicate that alignment between assessment and objectives and a detailed syllabus are critical in creating culturally relevant online courses. This study also introduces a tool for evaluating the extent to which CRP is emphasized in a course, which may be valuable to colleges wishing to implement CRP more deeply and consistently. In addition, the study’s evaluation process demonstrates that faculty working in teams increases the success of program revision.


Review of Related Literature
Research indicates that culturally relevant teaching is critical for diverse students to succeed.Underserved populations continue to be marginalized when culturally relevant pedagogy (CRP) and teaching are not implemented with fidelity in K-12 classrooms (Bartolome, 1994;Ladson-Billings, 1995a).As our K-12 student population becomes more diverse at a rapid rate, curricular changes are needed at the teacher preparation program level to ensure we are preparing culturally relevant teachers (Ladson-Billings, 1995a, 1995b, 1998).Teacher candidates' experiences in college courses must prepare them for what they will face throughout their teaching careers.When the needs of the classroom are not being heard and addressed in teacher preparation programs, the gap does not widen slightly but exponentially (Darling-Hammond, 2012).Materials and readings used in the teacher preparation coursework play a critical part in developing CRP.
As teacher programs increasingly provide online course options, determining if online courses provide the same learning activities as face-to-face courses is critical (Darling-Hammond, 1997).To complicate matters, it is difficult to ensure that CRP is integrated into online courses with fidelity due to the wide range of structures, platforms, and resources that are embedded within these courses (Dell, Dell, & Blackwell, 2015).Furthermore, in order to discover if teacher preparation programs are embedding CRP throughout their programs, investigations into course activities, assessments, and objectives are imperative.
Below, we review the literature on CRP in more detail and then discuss the importance of curricular alignment and danger of curricular drift in the CRP space.We then discuss how these issues play out in the online context.

Culturally Relevant Pedagogy
CRP is rooted in the belief that learning is socially and culturally mediated (Groulx & Silva, 2010).There has been over 30 years of research that has focused on culturally relevant teaching (e.g., Gay, 2000;Ladson-Billings, 1992, 1995;Nieto, 1999Nieto, , 2004)).Ladson-Billings' (1994) definition of culturally relevant teaching as a "pedagogy that empowers students intellectually, socially, emotionally, and politically by using cultural referents to impart knowledge, skills, and attitudes" (pp.17-18) is particularly helpful in framing studies focused on culturally relevant teaching.Paris (2012) also asserted that in order to implement a CRP, educators must first believe that all students can succeed, maintain student-teacher relationships, and at the same time understand the complexities associated with student success.The emphasis on shifting beliefs in teachers is critical; believing students can succeed must be accompanied by a willingness to teach in ways that are appropriate for all students.At the core of this belief is a healthy student-teacher relationship based upon mutual respect (Paris, 2012).
In her work, Ladson-Billings (1995a) has identified three key components for teaching to be culturally relevant: (1) students must experience academic success in that classroom; (2) students must "develop or maintain" cultural competence; and (3) "students must develop a critical consciousness through which they challenge the status quo of the current social order" (p.160).To determine if a lesson being taught in a K-12 classroom promotes the three components, Ladson-Billings (1995b) states that "culturally relevant teachers encourage [d] a community of learners rather than competitive, individual achievement" (p.480).In this collaborative space, educators promote shared questioning, engaging activities, open dialogue, and meaningful assessments.
Culturally relevant teachers promote "smart classes" as opposed to "smart children," allowing children who had not previously identified as academically successful to shift their identity and create a classroom of active learners (Ladson-Billings, 1995b, p. 480).
The discrimination inherent in the U.S. system of education, including the deficit-based perspective used across systems of secondary and postsecondary education, were of particular importance in framing this study (Beauboeuf-Lafontant, 1999;Macedo & Bartolome, 2014).While standards are often framed in asset language, the systemic lack of asset approaches by teachers in schools with marginalized groups has caused opportunity gaps (Ladson-Billings, 1998;Paris, 2012).Deficit beliefs and teaching practices are in direct conflict with the foundations that must be in place for culturally relevant teaching to occur (Ladson-Billings, 1995a;Paris, 2012).One of those foundations is cultural integrity; teachers must allow students to maintain cultural integrity in their classrooms (Ladson-Billings, 1998).One example of allowing children to maintain cultural integrity is language use in the classroom.While insisting on grammatically correct written English is appropriate for some assignments, it is also appropriate to create a classroom where students may use their home languages in various ways (Flores & Rosa, 2015;Ladson-Billings, 1995a).
It is critical for preservice and in-service teachers to understand the necessity of using culturally relevant practices, particularly in ensuring success for historically marginalized groups.Howard (2003) states that "teachers must be able to construct pedagogical practices that have relevance and meaning to students' social and cultural realities" (p.197).Constructing pedagogical practices that have relevance to students does what Ladson-Billings (1995a) called providing a vehicle for learning.This means allowing students to navigate their own educational path.In part, this includes a democratic classroom approach (Howard, 2003;Ladson-Billings, 1995a).Democratic classrooms require that teachers use the principles of democracy-namely freedom, equity, and justice-as foundational principles in the running of the classroom (Kesici, 2008).Student choice in the method for completing the assignment and/or proving mastery of the standards is a popular tool in democratic classrooms (Liston & Zeichner, 2013).However, allowing menus or other choice tools for students to complete activities does not go far enough to provide equity and justice in the classroom.This requires a truly reflexive teacher respecting the values and traditions of each student (Liston & Zeichner, 2013;Paris, 2012).Teacher educators play an important role in equipping teachers with the skills and tools for reflection that will allow them to use culturally relevant practices.To ensure that teacher educators are indeed doing so, colleges of education may develop CRP-related course objectives.However, as we discuss below, CRP-relevant course objectives may or may not translate into CRP practice within the course.

Curricular Alignment and Drift
Curriculum development is a key factor in the success of any academic program.Curriculum alignment describes the alignment between the course objectives, lesson objectives, and assessments in a course (Zemksy, 2014).Curriculum drift describes the process of "drift" that can occur when multiple faculty make different curriculum choices in teaching a course with the same standards, objectives, and assessments.Curriculum drift is a process in which the curriculum returns to a pre-innovative state, and faculty return to teaching what they know (Wilson, Rudy, Elam, Pfeifle, & Straus, 2012).It is the gap between the curriculum and what is actually taught (van de Mortel & Bird, 2010).Curriculum drift explains how an apparently quality, cohesive curriculum can appear vastly different when looking at numerous syllabi from the same course.Wilson et al. (2012) argued that curriculum drift can be unpredictable, yet when teacher education programs understand the forces that promote curricular drift, this may lead to the development of preventable strategies.Zemksy (2014) argued that a program's curriculum is the best place to start looking for solutions to problems in the program.There are several factors related to quality curriculum development.Some aspects of a quality curriculum are alignment, coherence, and cohesion (Johns-Boasts, 2014).These components of curriculum cannot exist individually, "rather, between each there exist strong relationships, which means when one aspect is changed it affects and brings about change in the other two aspects" (Johns-Boasts, 2014, p. 7).For this reason, understanding the components is a first step in evaluating a curriculum.One way to evaluate the curriculum is to ask the following questions: 1. Is there alignment between activities and assessments in a course?2. Is there alignment between standards and activities in a course?The answers to the above questions factor into the quality of the curriculum development (Johns-Boasts, 2014).
Research emphasizes that the importance of the intersection of learning, implementation, and design cannot be understated (Penuel, Fishman, Cheng, & Sabelli, 2011).The implementation of teaching pedagogies, such as CRP, cannot be guaranteed if alignment of curriculum is not done properly.Additionally, a lack of curriculum alignment may be the underlying reason a pedagogy is not evident.In other words, in the program design, activities that promote CRP development may have been included, but the design may not have been translated into the learning of the teacher candidates (Penuel, Fishman, Cheng, & Sabelli, 2011).This could be from a lack of coherence in the syllabus provided or a lack of coherence across the program (Johns-Boasts, 2014).
If coherence is not achieved in the online course or program, it is likely that alignment is also not evident in face-to-face versions of those courses (Dell, Dell, & Blackwell, 2015).Fink (2003) stated that in designing any course, situational factors (i.e., number of students, when and where offered, prior knowledge) must be considered before factors such as backward design of curricular elements, goals, objectives, assessments, and activities can be accomplished.After situational factors are complete, what Fink (2003) called the "initial phase" begins.This is where examination of alignment occurs.
Program revisions in teacher preparation programs are common practice.Recent research stresses the need for more stringent protocols to ensure standards from the accreditation body, program level, and college level align.As teacher preparation programs feel the pressure to place more courses and programs online, curriculum drift is happening at a higher rate due to lack of coherent programming and oversight in departments and colleges (Kraglund-Gauthier, Chareka, Orr, & Foran, 2010;Moore, Sener, & Fetzner, 2009).Additionally, there is literature suggesting that "widespread production models" used in online course development often do not use a collaborative approach (Oblinger & Hawkins, 2006).Widespread production models typically happen when faculty develop courses based on prefabricated online course shells, created by one faculty member without the benefit of a curriculum team (Oblinger & Hawkins, 2006).
It is also important to note that there are direct connections between cultural relevance and curriculum design (Hastie, Martin, & Buchanan, 2006;Stoddart, Bravo, Mosqueda, & Solis, 2013;Villegas & Lucas, 2002).In order for courses designed for preservice and in-service teachers to be culturally relevant or teach cultural relevance-and they should do both-curriculum design (including alignment of tasks, assessments, and rubrics) must be included.Teachers often teach the way they were taught themselves (Darling-Hammond, 1997;Ladson-Billings, 1998b).In order to ensure the equitable access of all K-12 learners, teacher preparation programs need to explicitly teach CRP and provide opportunities for purposeful practice of these skills and strategies (Darling-Hammond, 2010;Ladson-Billings, 1995b).Therefore, it is important that teacher preparation programs ensure that CRP is embedded and taught with fidelity (Darling-Hammond, 2010;Ladson-Billings, 1995b).
While the connection between how CRP is taught and how those learned skills are applied has been well documented, there is a gap in the literature on whether or not in-service teachers are able to generalize those CRP strategies in their own classroom (Howard, 2003;Lucas & Villegas, 2013;Villegas & Lucas, 2002).Research shows that teacher education increasingly requires inquiry into various sets of standards and often involves balancing conflicting accrediting agencies (Darling-Hammond, 1997;Darling-Hammond, 2012;Smartt & Reschly, 2007;Wang & Chen, 2009).While standards may dictate a reflexive teaching approach, a teaching accrediting agency may have a check-the-box approach that limits teacher reflection (Darling-Hammond, 1997).In order to bridge these demands, new materials and methods must be embedded within the curriculum to meet accreditation requirements.More importantly, new methods must meet the needs of the increasingly diverse population of students (Darling-Hammond, 2012).

Curriculum in Online Courses
Online courses have become a popular option for both undergraduate and graduate students.The percentage of students taking at least one online course in higher education is about 28%, and that percentage has grown steadily over the past 13 years (Allen & Seaman, 2016).The enrollment in online programs continues to increase at a higher rate than other models (i.e., faceto-face and hybrid).As the rate of enrollment continues to grow, it is essential that faculty take a critical lens to investigate the pedagogy and instructional methods used in online learning.Allen and Seaman (2016) also reported that only 29.1% of higher education leaders said that their faculty "value the legitimacy of online education" (p.21).Faculty reported not feeling confident in the ability of online education to have the same impact as face-to-face instruction, and faculty felt less confident delivering their content in the online environment (Allen & Seaman, 2016).Furthermore, in many cases teacher educators are expected to design and deliver online instruction, although they many not possess an understanding of what constitutes effective online teaching.Although numerous articles focus on facilitating online learning, there are a limited number of studies that examine the implication of technology on our teacher preparation programs (Hannon & D'Netto, 2007;McLoughlin, 2001;Shin & Lee, 2009).
The "technology-led curriculum design" common in online platforms often places preference on "values of access and management of content" rather than on pedagogy or inquiry (Hannon & D'Netto, 2007).Online learning gives students a sense of urgency and immediacy not present in traditional face-to-face courses; the changes in their demands as learners and their communication expectations have been noted by multiple researchers (Hannon & D'Netto, 2007;McLoughlin, 2001).These questions have led to extensive exploration of online learning and cultural relevance (Shin & Lee, 2009).For instance, instructors in a face-to-face setting are easily able to facilitate and model CRP by creating a classroom community questioning and critiquing inequity in schools and society.This becomes more difficult and complex to do in an online environment, in particular if faculty are not yet adept at creating an online classroom community where those conversations are easily facilitated (McLoughlin, 2001).
As early as 1998, Bonk and Cunningham emphasized the need to communicate with learners in new ways "as the menu of technology choices in schools, workplaces, and on college campuses escalates" (p.27).As new learning tools and assessments were created to better reach students in an online environment, many researchers explored the opportunity to simultaneously make courses more culturally relevant (Lauzon, 1999(Lauzon, , 2000;;McLoughlin, 2001).Several important themes emerged: online learning platforms and delivery, assessment and task alignment in online courses, and differences in emphasizing the pedagogy of online learning and the pedagogy of content areas (McLoughlin, 2001).The importance of alignment of tasks and assessments with the objectives of a course has been studied extensively (Fink, 2003;Penuel, Fishman, Cheng, & Sabelli, 2011;Wiggins & McTighe, 2005).When curriculum design or redesign occurs, the objectives and assessments are often a focus of the initial work (Cobb, Confrey, diSessa, Lehrer, & Shauble, 2003).
While alignment is a critical component of online courses, and tasks and assessments must allow students to master the course objectives, this is no different from traditional courses.Wiggins and McTighe (2005) are cited in literature on online course development; good course design is widely accepted as a key component of a quality online course.Unfortunately, while Softchalk (2018), Quality Matters (2018), and hundreds of university guidelines exist for best practices in online course design, there is little in the literature explicitly tying good course design in traditional courses to good course design for online learning.Adding tools to online courses without training for faculty on how to best integrate those tools into their teaching and promote an online community will continue to hinder the quality of online courses and programs (Wiggins & McTighe, 2005;Penuel, Fishman, Cheng, & Sabelli, 2011).With the recent push for technology integration in colleges of education, it seems unfortunate that this gap in the literature exists (Harris, Mishra, & Koehler, 2009;Rogers, 2000).This is particularly true given that understanding how course design applies to what is actually taught within the course is imperative.Finally, as online learning grows, investigating what is being taught in those courses becomes increasingly critical.One study that examined successful online courses used rigorous learning outcomes, effective use of technology to enhance pedagogy, and active student engagement throughout the course as indicators of successful courses (Shin & Lee, 2009).This study seeks to expand upon that work by beginning with an investigation of several online courses investigating their use of CRP across learning outcomes and through multiple engaging assignments.

Problem Statement and Research Tradition
The research questions guiding this study were the following: 1. Is culturally relevant pedagogy emphasized in the course descriptions and other course informational materials?
2. Is culturally relevant pedagogy emphasized in the course activities, assessments, and rubrics?
3. Is there an alignment between the descriptions and activities, assessments, and rubrics?
Our research questions were driven by the desire to determine alignment between descriptions of the use of CRP and evidence in courses of the use of CRP.Additionally, our research is meant to provide a replicable process for peer review and improvement of CRP integration into coursework.While the goal is not for this study to be generalized, it is our hope that this study will encourage others to take a critical lens to curriculum alignment and the incorporation of CRP in online courses.
The research questions focused on examining CRP in several components of the syllabi: goals and objectives, standards, activities, and assessments.Additional questions were added to investigate course alignment.Based upon the program revision literature, it seemed likely that the objectives and standards would emphasize CRP, but the assessments would not-an indicator of a lack of course alignment (Fink, 2003).While there are key terms that might appear that would make CRP obvious to researchers ("social justice"; "cultural integrity"; "asset based"), it seemed more likely that a thematic analysis looking for patterns of terms would be necessary to really identify culturally relevant components of the syllabi.Thus, qualitative inquiry and its capacity to generate deep, rich descriptions (Creswell, 2015) was appropriate.This is a case study (Stake, 1998) using grounded theory only as a lens for analysis.This methodology allowed for rich descriptions and thematic analysis while also allowing researchers and colleagues less familiar with the qualitative tradition to make use of the data and findings.The components of grounded theory used-such as the cycle of constructing themes and discussing between investigators before themes are agreed upon-provided a shared language for analysis.This is further described in the data analysis section.

Context of the Study
This study took place at a southeastern university offering 82 undergraduate degrees, 39 master's degrees, eight specialist's degrees, and 10 doctoral degrees.While the university is only 50 years old, it has experienced rapid growth.Similar growth has occurred with online learning coursework and programs.In 2010, there were 1,057 online courses offered by this university; in 2014, that number jumped to 3,950.In the college of education, the first online course was offered in 2008.In 2015, there were 184 online courses approved for delivery.Increasingly, faculty are responsible for the transition from a face-to-face offering to an online course offering.That leaves the curriculum work and alignment to faculty teaching in those programs.While there is some training and assistance available throughout the process, that training is primarily focused on technological components of online learning.The role of the instructional designer in the college is to ensure that the course follows a set of standards created by Quality Matters (QM).QM is a faculty-centered, peer review process that allows institutions of higher education to evaluate the design of an online course.Although the rubrics used in this review process are grounded in research and are intended to provide quality online instruction, there is still a lot of room for faculty developers to interpret the content as they see fit.In addition, if the QM reviewer is unfamiliar with curricular content, there is no guarantee that the evaluation of the course will be driven by an understanding of content.
The courses in this study span three departments; thus, the three researchers represented each department to ensure knowledge of content area and accrediting bodies for each department.While these accrediting agencies and guidelines are helpful and necessary for course development, there is little consistency across departments to ensure that rubrics, syllabi, assignments, and assessments are developed and updated using a systematic and democratic approach.There were six courses included in this study, and they are housed or "owned" by those three different departments.All of these departments use different program standards for their various degrees and their associated Specialized Professional Associations (SPA) standards.SPA standards are typically associated with particular content; in this case the SPA for all courses is the International Literacy Association (2018), an accrediting agency for courses focused on reading development in P-12 children.The commitment to cross talk and collaboration is an integral part of this study.Working collaboratively across P-12 environments is a requirement of the SPA; it is also of the utmost importance if the results of this study lead to curriculum alignment or professional development.
The courses in this study were chosen based upon their applicability to the SPA noted above; the courses are all in reading education.Additionally, the courses were chosen based upon intentional sampling; all courses had either been taught or developed by a member of the research team.This allowed for a small sample size.Researchers intentionally chose to focus on fewer courses in order to ensure that thorough coding could occur in every component of the course.As no student data was being collected, this study was categorized as exempt following review by the Institutional Review Board at our university.This approach allowed us to take a deep dive into the syllabi, although it should be noted that some of the syllabi used in this study were "passed down" from other faculty.Pseudonyms will be used throughout the discussion when referring to specific courses.The specific undergraduate reading courses are as follows: an undergraduate reading assessment course (RD 1), a graduate reading assessment course (RD 2), a graduate content area reading course (RD 3), an introduction to reading theory (RD 4), a young adult literature course (RD 5), and literacy for P-12 students with disabilities (RD 6).

Analysis Process
This study was analyzed using elements of grounded theory (Glaser, 1992).Researchers were "concerned with generating and plausibly suggesting . . .many properties and hypotheses about a general phenomenon" (p.438).For this case study, the phenomenon being investigated was CRP and its prevalence in an online course as evident in the syllabus.The authors created a coding instrument specifically for this study.Themes and patterns were pre-identified to create a coding instrument based upon the work of Bartolome (1994) and Ladson-Billings (1995a, 1998), and all courses were evaluated based upon the coding instrument.In this way, the research veered from the grounded theory method, as it was only used for analysis and was not meant to be generalized or create a theory (Corbin & Strauss, 2008;Glaser, 1965;Glaser, 1992).However, while the researchers were "concerned with many hypotheses synthesized at different levels of generality" (Glaser, 1965, p. 438), they were being more prescriptive in their coding than may generally be seen in qualitative case studies (Creswell, 2015).
The initial coding instrument, Evidence/Contra-Evidence of Cultural Relevance Holistically in Courses (Table 1), looked at CRP themes across courses, assignments, assessments, and rubrics to provide a holistic view of the reading program.Researchers were asked to provide comments and/or evidence to support their claim.It is important to note that if one assignment was not coded with a CRP theme, it did not prevent the course as a whole from being labeled as culturally relevant.The researcher looked at one course at a time and decided if the code was evident or not evident in the syllabus and assignments.To ensure interrater reliability, after initial coding another researcher looked at the same course and, using a clean copy of the same coding instrument, looked for the listed components.For codes that were not in agreement, the research team looked at the information holistically and decided together if there was enough evidence to warrant a code.
After an initial round of coding, the researchers used a second instrument based on the works of Bartolome (1994) and Ladson-Billings (1995a, 1998).This second instrument, Evidence of Culturally Relevant Course Components, took a deeper dive into the course content, specifically capturing if a theme was evident multiple times throughout a course.Table 2 provides the themes according to the courses and was created as a checklist so that researchers were able to look at a specific activity, assignment, assessment, or rubric and check off a specific theme if it was present.This allowed for a fuller picture of what pieces of the course were culturally relevant.Table 2 provides some examples of activities, assignments, assessments, and rubrics that researchers were able to check off as having CRP components.
The Evidence of Culturally Relevant Course Components coding instrument followed the initial coding instrument's protocol.The lead researcher reviewed items throughout the course and used the checklist to indicate if CRP was present.Then a second researcher took a blank checklist and reviewed course items, without any knowledge of lead researchers.Once coding was completed by both parties, the researchers compared coding sheets and discussed agreements and disagreements.Like the first coding sheet, consensus was found for all coding items.Triangulation occurred through initial coding, discussions with partners, and application of agreed upon tables to the courses for verification and agreement.An unanticipated result of the coding instrument is the ability to use it across programs to investigate curriculum alignment and CRP.

Results
In "themes spanning courses," there were five components agreed upon to create the coding instrument.One component was a negative: It should not appear in a culturally relevant course.That example, "attempting to insert culture into education," appeared in two out of six courses.The other four components were positive examples and should appear; three out of four appeared in at least one course.However, for two out of four components, only one course showed evidence (see Table 1).Those two components were "insists that students experience academic success" (Ladson-Billings, 1995a, 1998) and "insists that students develop and/or maintain cultural competence" (Ladson-Billings, 1995a, 1998).

Evidence/Contra-Evidence of Cultural Relevance Holistically in Courses
Themes Spanning Courses Attempting to insert culture into education as opposed to inserting education into culture (Ladson-Billings, 1995) RD3: "diverse" is used throughout without asking candidates to explore or clarify RD4: parent education workshop is superfluous and doesn't tie to the course Using a humanizing pedagogy that respects and uses the reality, history, and perspectives of students as an integral part of educational practice (Bartolome, 1994) RD2: case study assignment examines child's family and context for learning RD5: novels are selected based upon choice in the classroom community RD6: evidence provided of appropriate design for learning to reach all learners in every activity Insists that students experience academic success (Ladson-Billings, 1995, 1998) RD6: evidence of student learning must be provided in every activity and is included in the rubric for those activities Insists that students develop and/or maintain cultural competence (Ladson-Billings, 1995, 1998) RD6: understanding the identities of students with disabilities is critical to succeed in this course; candidates must describe reaching all students in multiple activities and their rubrics Insists that students develop a critical consciousness through which they challenge the status quo of the current social order (Ladson-Billings, 1995, 1998) n/a When looking at the results from the first coding instrument, the first theme "attempting to insert culture into education as opposed to inserting education into culture," is found in two of the six courses.This theme provides evidence that a course does not incorporate CRP with fidelity.The examples found included using the term diverse without ever defining it-thus inserting the instructors' definitions as opposed to collaboratively constructing a definition.Additionally, an example was the parent education workshop.The workshop did not tie directly to the standards, and the workshop directions did not ask candidates to explore the cultures of the community before designing the workshop.Thus, the workshop would be what the candidates felt was important as opposed to being designed around the needs of the community.In themes two through four, RD6 appears multiple times.The final theme, "insists that students develop a critical consciousness through which they challenge the status quo of the current social order," is not evident in any of the six courses.One interesting note is that RD6, a course on students with disabilities and literacy instruction, met many criteria.The course standards and objectives began with knowing each learner as an individual, which aligned beautifully with CRP.
The second coding instrument, "themes in sections of courses," allowed our research team to investigate five components of CRP.In all five cases, the goal was for courses to include the component; none were examples specifying something that was not CRP.When using the second coding instrument, only two out of the five themes ("teachers attend to students' needs while allowing them to choose how their cultural excellence occurs" and "students' culture is used as a vehicle for learning") appeared in any of the courses.As Table 2 illustrates, only "teachers attend to students' academic needs while allowing them to choose how their academic excellence occurs" spans all six courses.It is also important to note that while many activities and course objectives displayed that important tenet of CRP, in no instance was that sentiment also evident in the rubric for the activity.(Ladson-Billings, 1995, 1998) n/a Students' culture is used as a vehicle for learning (Bartolome, 1994;Ladson-Billings, 1995) RD1: portfolio; course objectives RD2: objectives and inclusion of cultural insights in case study assignment Students develop sociocultural competence (Ladson-Billings, 1998) n/a Students critique cultural norms, values, and institutions that produce and maintain social inequities (Ladson-Billings, 1998) n/a In Table 2, sociocultural competence is a key component noted multiple times.This term was first unpacked by the researchers to ensure shared understanding.The agreement was that students must understand the norms, values, and institutions that produce and maintain social inequities, but they must also understand (and critique) those norms as tools to further develop sociocultural competence (Ladson-Billings, 1995a, 1998).Sociocultural competence and critiquing cultural norms are critical components of Ladson-Billings's (1998) definitions of CRP.Thus, these components need to be explicitly taught, purposefully practiced, and continuously reflected in course content, directions, activities, and rubrics.Sociocultural competence is the ability to understand and use a language appropriately, but it is also understanding the cultural structures within that language.RD1 was a good example of a course where a bit more depth-that is, a reflection requirement to accompany the portfolio focusing on tenets of CRP-would allow for a rich, culturally relevant course.The portfolio required candidates to include lesson plans illustrating how they addressed the learning styles of each student, how they differentiated for students, and how they assessed students using choice and both formative and summative assessments.These assessments allow teachers to investigate through informal and formal means students' mastery of course standards and objectives.However, in general, the coding instrument revealed that few of these courses incorporated CRP in a deep and consistent fashion.A deeper analysis of how and why the courses did not meet our coding criteria suggested six themes that were common across most or all courses: a lack of emphasis on democratic classrooms; a lack of alignment between CRP objectives, assignments, and rubrics; a lack of evidence that depth of CRP knowledge was required across courses; problems with course design; and a lack of emphasis on identity and cultural competence.

Democratic Classrooms
One of the most striking findings was the lack of democracy included in these online courses.A goal of the program is to "provide [teachers] with the knowledge, skills, and dispositions to plan, implement, and assess high-impact instruction that meets the needs of all students, regardless of abilities or language proficiencies."Additionally, this reading program (like reading programs across the country) adopted updated literacy standards in order to "be just as forthright about the resources and adjustments that will be needed to ensure that all childrenstruggling learners, gifted students, dual language learners -reach . . .goals" (International Literacy Association, 2016, para. 15).As all standards and objectives in the courses in the program align to those goals, it was expected that high-impact practices, including creating democratic spaces for learning, would be incorporated.The lack of democratic classroom evidence is a layered problem.The assignments and rubrics in the courses show no evidence of creating a democratic classroom for candidates; the rubrics and activities are set up with little evidence of choice in assignments and no evidence of "attend[ing] to students' academic needs while allowing them to choose how their academic excellence occurs" (Theme 1 from Table 1).Allowing students to choose how their own academic excellence occurs is an example of respecting the varied strengths of those students, which is culturally relevant (Ladson-Billings, 1995a, 1998).
Furthermore, there was no assignment in any course investigated in which candidates were asked to create a democratic classroom for their students.While there were activities in which candidates asked students to make choices (i.e., the novel choice assignment in RD5), there were no examples of instructors allowing their teacher candidates to choose activities to prove mastery or any directions asking them to include evidence of such choice in their activities.Creating and providing examples of democratic classrooms in online courses is incredibly important for our courses and programs to reflect CRP.

Alignment
RD6 was the only course investigated in which true curriculum alignment occurred.In that course, course objectives explicitly stated that candidates "collaborate with and communicate data outcomes and instructional decision-making with families, colleagues, and administrators" and behave as an "advocate" and "culturally responsive practitioner."In each module, candidates must administer varied and appropriate reading assessments with students, including instructions on how to determine which assessments are most appropriate for the target students.Candidates are asked to collaborate with peers and defend their instructional choices throughout the course, and all language in the instructions and objectives are included in the rubrics.While this course was listed as meeting Theme 4, "insists that students develop or maintain cultural competence," this was determined because researchers felt that understanding the needs of students with disabilities qualified (Ladson-Billings, 1998).After analyzing the data, this course was identified as having the most aligned curriculum.However, suggestions for improving the cultural relevance of the course would be to ask candidates to describe how the target student's community, family, and environment contribute to the student's strengths in reading or in other areas of school.
Another important finding related to alignment was the benefit of the coding instrument.While the coding instrument was initially developed to ensure interrater reliability in this study, we found that this instrument was also incredibly helpful at highlighting inconsistencies in courses.For example, many courses had course objectives that could have led to culturally relevant teaching, and yet assignments and rubrics did not meet those objectives explicitly.The coding instrument itself is worthy of future use in program evaluation.

Depth of CRP Knowledge
The lack of alignment between the objectives, assignments, and rubrics in the courses was most problematic as the researchers analyzed in terms of depth of knowledge.Numerous times, comments were made such as "the objectives list a really great possibility, but that doesn't show up in the activities" or "the rubric doesn't seem to assess what is emphasized in the instructions."The concern is that the objectives may lead to assignments that are culturally relevant, but language in a rubric might minimize the culturally relevant components of the assignment.For this reason, it was critical that courses contain more than "check-the-box" components noted in Table 2.When the objective emphasized cultural relevance, but an assignment was a check-the-box activity requiring little thought or engagement, this disconnect was most obvious to researchers.Examples included "describe literacy assessment and instructional approaches that support all learners, including struggling readers, English language learners, students from a variety of cultural backgrounds, and students with cultural needs" and "integrate individual, family, cultural, and/or community assets into literacy assessment and instruction."Those objectives exemplify CRP.However, they are not clearly met by the activities, assessments, or rubrics.This disconnect keeps candidates from mastering the depth of knowledge needed to enter the field using CRP.

Course Design
The findings in this study led us to believe that the lack of CRP in these courses was primarily due to a lack of alignment.As noted, five out of the six courses included objectives that directly connect to culturally relevant teaching.What is not noted is that in all six courses, multiple objectives, standards, and assessments readily lend themselves to the inclusion of CRP.There are a number of possibilities.The standards and objectives could be aligned to the SPA listed above, without the inclusion of activities that meet the demands of that SPA.It is also possible that some of that culturally relevant teaching and unpacking of standards is being done in face-to-face courses and has not been explicitly listed in course materials for online delivery.It is also possible that by investigating online courses that may still rely heavily on synchronous sessions, faculty are teaching online courses as face-to-face courses.The researchers acknowledge that without interviewing other faculty teaching these courses it is impossible to determine precisely where the breakdown occurs between what is listed in the standards and objectives and what is being assessed.

Identity and Cultural Competence
While reflections were included in a few assignments, the reflections rarely included descriptions of identity and how it impacted teaching and learning.It also was not clear in any course how teachers were using their cultural competence or if they had indeed gained cultural competence.An example of this is the case study assignment in RD2.Candidates listed the cultural information about the child; however, it was unclear what was meant by that.Questions such as the following could have strengthened and explicitly embedded culturally relevant information: • What funds of knowledge did the child possess?
• What were the cultural competencies used by the teacher's own funds of knowledge in order to determine the best methods in teaching the student?• How did the practitioner's thinking shift?
• Did the child's identity as a reader change over the course of their work together?
Researchers felt this was a missed opportunity for teacher candidates and their students.Researchers also wondered if faculty reflected on how they address their teacher candidates' funds of knowledge.It is imperative that faculty reflect upon that shift when working with teacher candidates who come in with different assumptions and backgrounds.The work of asking our preand in-service teachers to use educative curriculum materials (Davis & Krajcik, 2005;Drake, Land, & Tyminski, 2014) changed the way we pictured our approach to increasing our culturally relevant teaching and, thus, creating more culturally relevant courses.

Discussion
As discussed above, the key findings of the study were a lack of emphasis on democratic classrooms; a lack of alignment between the objectives, assignments, and rubrics; a lack of evidence that depth of knowledge was required across courses; problems with course design; and a lack of reflexivity.The study looked at courses that had been in existence for some time; these courses will be updated as new standards are implemented.This leads to an opportunity to use the study to improve these courses.Thus, the findings may have a local impact.Additionally, these findings may help other colleges of education investigate their online course offerings; this may lead to changes in the field as faculty strive to increase the cultural relevance of their courses and pedagogy.

Local Implication and Impacts
The faculty researchers implemented changes in the courses we were assigned to teach the following summer to ensure CRP was embedded throughout.Assignments like the case study, portfolios, and intervention reports (shown in the tables above) now require candidates to reflect upon their own culture, their own funds of knowledge, and those of their students before and after any other information is gathered (Esteban-Guitart & Moll, 2014;Gonzales, Moll, & Amanti, 2006).Essentially, each teacher candidate administers a pre-and postcourse survey to themselves and the students they serve.This is a first step in ensuring two things: the alignment is closer to the course objectives, and the cultural relevance of the course is improved.The survey questions asked in the pre-and postcourse survey allow the teacher candidate to think through their beliefs before and after working with students.Additionally, this work will allow further research at the end of those courses to see if the change to syllabi, assessments, and rubrics aligns our work more closely with the tenets of CRP.There should be a change in teacher candidates' responses to the Likert questions from not familiar with topics or not able to implement to very familiar and could teach others to implement.We also plan to offer the coding instrument in survey form to assess our teacher candidates' efficacy on CRP.We will then have program data to show if they are improving in their self-perceptions of CRP.
As we went through the process of this investigation, we realized that surveying teacher candidates in the "old" courses and the "new" courses is critically important.While course evaluation data exists for multiple semesters on each of these six courses, that data is irrelevant when using the coding instrument.The course evaluation is a university-wide measurement, and teacher candidates respond generally to prompts such as "this course contributed to my knowledge" and "this instructor contributed to my knowledge."In order to measure the impact of the course on a candidate's sense of efficacy in practicing CRP, we will need to implement the survey mentioned above.Additionally, the layered approach we all agreed upon as a research and design team would mean administering a similar survey to the children our candidates work with.In the future, faculty, teacher candidates, and K-12 students need to be asked questions such as the following: • Did they feel that the instructor cared about their culture?
• Did they feel that the instruction allowed them to be themselves?• Was their opinion valued?
The findings of this study align with the literature on CRP and the need to improve teacher candidates' sense of efficacy in culturally relevant teaching (Adams, Rodriguez, & Zimmer, 2017;Ladson-Billings, 2004;Moulding, Stewart, & Dunmeyer, 2014).Additionally, while this is important program data, it is also a way for faculty to engage in the scholarship of teaching and learning and see if the changes implemented had an impact on the learning of our teacher candidates.This type of work enables meaningful program revisions that are consistent and data driven while also improving the cultural relevance of our programs and teaching.
This study raised an important question of whether or not our programs were actually engaging students in CRP.The results from this study prompted us to be more vigilant in our courses about the kind of work that we have students engage in; this includes some of the face-toface courses that we teach.Another question that came to mind after analyzing the data was, How do we fix this?One of the things that the researchers had to consider was conversations with other faculty, since numerous faculty are involved with these courses.These conversations sparked interest around the college of education, and the researchers had the opportunity to present this work in a college-wide meeting.There was also interest in collaborating on how to remedy some of the issues that the researchers saw.

Lessons for the Field
As stated previously, one of our key findings was a problem in course design.The online courses left quite a lot of room for interpretation; this was particularly noticeable in the lack of alignment described previously.In a face-to-face course, perhaps the instructions given by faculty filled that gap.We felt there were missing pieces of information-an objective was clear, but the attached assignment did not specifically measure those objectives.Our suggestion to faculty is to make explicit connections in the syllabus and course materials in online courses.In this study, the courses investigated were master courses shared by multiple instructors.As a syllabus is shared, it is critical for as much detail to be included as possible.The lesson for the field from this particular finding would be to create an online syllabus for online courses.These syllabi would include specific objectives and assignments with each lesson/module; these should all be aligned with course objectives and any standards to which the course adheres.If a university allows more instructor flexibility, or if online courses are the purview of the faculty members, the syllabus still ensures some alignment across all objectives and assignments.
The lack of syllabi like these was problematic to us for several reasons.First, if a new faculty member or adjunct faculty receives only the syllabus and rubrics in a course, how can the incorporation of CRP be ensured?Additionally, if the course materials online do not support CRP, an additional examination of those objectives is warranted.Are the objectives requiring teacher mastery of CRP being met if the course materials and assessments are not explicitly measuring CRP?In the case of this study, there were program assessments mandated in two of the courses (RD2, RD3).The provided program rubrics do not emphasize CRP.Thus, even if the course is culturally relevant, revision of those program rubrics to include culturally relevant components would be advisable.According to the literature, this indicates that an intersection is not occurring and, thus, the design is flawed (Fink, 2003;Penuel, Fishman, Cheng, & Sabelli, 2011).
Similarly, there are findings that may be helpful for reading programs at the undergraduate and graduate levels in the field.Our coding instrument is unique and valuable.It allows faculty to work collaboratively within and across programs to investigate the cultural relevance of their courses.This tool could be used in various ways (beyond reading courses) in colleges of education.Another suggestion for reading programs based upon our findings is to include democratic classroom examples in online courses and to require candidates to design activities leading to democratic classrooms in courses where such work is appropriate (Marri, 2005;Subba, 2014).Creating a democratic classroom in the online environment would include options for discussions, an online learning environment that values the participants and their varied views and beliefs, and options in the formats assignments take based upon the needs of each candidate (Brookfield & Preskill, 2012;Schallert et al., 2015).This also fosters the use of democratic classroom communities by the candidates when they leave the program, a key component of culturally relevant teaching (Ladson-Billings, 1998, 2004).Asking teacher candidates to create assignments that allow both teacher and student to learn is imperative.The layered approach we found missing in this study and our work is also necessary; the faculty need to model what we are asking teachers to use with their K-12 students.Finally, that would allow for teacher candidates to delve more deeply into areas of interest or to fill knowledge gaps.We found that depth of knowledge was not emphasized across courses.Allowing teacher candidates to learn about the funds of knowledge struggling readers bring to the classroom was an objective emphasized across courses, yet assignments led to superficial examination of this topic (Gonzales, Moll, & Amanti, 2006).This is an area of improvement for our program that may need to be addressed at other colleges as syllabi are examined in this way.

Limitations
In another setting-in particular, a setting where online course development was undertaken-the outcomes would likely differ from those in this study.This study was intentionally limited in scope to allow for depth of inquiry and thick descriptions.In future studies, it might be helpful to connect the specific courses investigated to program goals for the purpose of ensuring course and program integrity.The limitation of the small number of courses was not applicable to our use of this study for program improvement; researchers were able to use this data to make immediate course and program changes.This limitation also highlights the limitation of using our own courses-courses we teach frequently-for investigation.We determined as a research team that investigating our own courses also allowed bias to be used positively in terms of being more critical.While our familiarity certainly impacted the results, we used the team approach to limit this bias.In each case, only one researcher was familiar with a course being investigated.We all had two courses we knew quite well and four we were not familiar with.This minimized the bias inherent in this work.Further, the limitation was also an asset.We were able to speak with depth on topics in the course(s) we knew well and had recently taught.
Another important limitation may be unique to studies of online courses in teacher education programs.While our coding instrument includes "teachers" and "students," and while all three researchers visualized the terms applying to our teaching candidates ("teachers") and their K-12 students ("students"), this was not clear when coding was discussed.As codes were shared and analyzed by multiple researchers, it was evident that in all instances, researchers were thinking students applied to both candidates and their K-12 students.In other words, it should have been clear before coding began if the analysis should be limited to what CRP was evident in the course in terms of requirements of faculty to teacher candidates, if the analysis should be limited to what CRP teacher candidates were asked to incorporate for their students, or if a layered approach was more appropriate.The study began by assuming that it was important to investigate how candidates were asked to use cultural competence and CRP in the program in online courses.
Finally, a limitation of the study is that we were investigating online courses to improve our own program, yet we are limited in the amount of autonomy we have to change certain rubrics.While we may change our assignments and may suggest that program assessment become more culturally relevant, there is not yet a college-wide protocol for sharing materials.Faculty may feel that we are implying a loss of academic freedom if we suggest that all faculty should use the same materials with their syllabi; this is true in our college even in online courses.Thus, this will impact three departments and could be work others want to engage in, but for the immediate future it will impact only our own teaching and the impact we have in these particular courses on teacher candidates.

Conclusions
This study allowed three faculty members to look at online courses in a way we had not before.While we set out to see if CRP was embedded in those courses, we found out much more about the curriculum design process-and what was missing from it-by having conversations with colleagues.This work allowed us to create a framework, using our coding sheet, that could be shared across programs in order to investigate courses in new ways.It also led to the creation of a team of researchers who are able to systematically look at groups of courses and create an iterative process of (re)design, implementation, and investigation.That process spans three departments and multiple programs and is thus beneficial far beyond the students in these six courses.
When the study began, it seemed likely that several elements of CRP were missing from the online courses based upon the researchers' teaching experiences in those courses.In addition, based on the coding instruments, it seemed that alignment work was needed between assignments and rubrics to ensure CRP was reflected and modeled.As the review of literature indicated, aligning tasks and assessments is critical for student achievement; in this case it seemed more critical since the program seeks to produce culturally relevant teachers.If instructors do not model how their own course rubrics align with assignments, it is difficult for teacher candidates be held accountable to model this with their own K-12 students.With the push for more online courses within teacher education programs, it is imperative that we model and incorporate CRP into online courses.
3. Is there alignment across courses, indicating cohesion?4. Is each component of the course clear enough to indicate coherence? 5. Is each course coherent, leading to a coherent program?

Table 2 .
Evidence of Culturally Relevant Course Components