FACE-TO-FACE AND ONLINE PROFESSIONAL DEVELOPMENT FOR MATHEMATICS TEACHERS : A COMPARATIVE STUDY

The study compared the effects of a professional development course delivered in an online and a face-toface format. The effects examined included changes in teachers’ pedagogical beliefs, instructional practices, and understanding of teaching number-sense and related mathematical concepts. The study randomly assigned participants to either the online or the face-to-face format and employed the same instructors, reading material, and instructional activities for both formats of the course. Both formats of the course were also delivered over the same eight-week period and required participants to invest approximately the same amount of time each week engaging in learning activities. Both formats of the course showed significant impacts on teachers’ mathematical understanding, pedagogical beliefs, and instructional practices. Consistent with prior research on online versus face-to-face instruction, the positive outcomes were comparable across both formats. Interestingly, teachers who participated in the online course reported that they were more willing to take courses in the future online than did teachers in the face-to-face condition. Further research is needed to determine whether this finding is limited to selfselected teachers, the specifics of this course, or other factors that limit generalizability.


I. INTRODUCTION
Online professional development provides several advantages over face-to-face sessions.In an online course, teachers have more flexibility and greater choice as to when they participate and engage in learning.Teachers in remote areas or who work in small schools can also gain access to professional development courses that would otherwise be expensive or impractical to deliver in a face-to-face environment.Through online courses, it is also easier to connect teachers across schools and districts, thus widening perspectives and fostering professional connections that would not occur otherwise.Nonetheless, since its introduction more than a decade ago, educational leaders have questioned whether professional development delivered in an online environment is as effective as traditional face-to-face sessions in increasing teachers' content and pedagogical knowledge and in improving their instructional practices.
The study presented here builds on past research that examines the effect that different design features of online professional development has on teachers' knowledge, pedagogical beliefs, and instructional practices [1,2].These previous studies systematically varied the level of interactions that occurred within different versions of a professional development course.One version included asynchronous, facilitated discussions among participants as a central, required part of the course while in the other version each participant worked independently, without a means of communicating with their peer participants.These studies also manipulated the amount and types of interactions participants had with the course instructors.
At one extreme, participants had access to two instructors, one with content area expertise and the other skilled in facilitating online discussions.Participants in this strongly facilitated online course received considerable amounts of feedback and had interactive discussions with the instructors throughout the course.At the other extreme, participants worked through the course at their own pace with minimal interaction with a single facilitator who handled technical and logistical questions but did not have content expertise.In this version of the course, participants were instructed to complete assigned readings, classroom activities, and writing assignments on their own and received a simple statement that their writing assignment was received and met the course requirements, with no constructive feedback or follow-up questions.
In both of these previous studies, the purpose of manipulating the level of interaction among participants and the amount and type of interactions with the course instructor(s) was to examine whether these design features influenced the intended outcomes of the course.All versions of the course were found to have comparable positive effects on teachers' pedagogical beliefs, instructional practices, and knowledge of teaching algebra (the focus of the course).Surprisingly the positive effects did not differ significantly among the different versions of the course.
As described more fully below, the study presented here shifts the focus of our research from manipulating aspects of an online course to directly comparing the effects of online versus face-to-face courses in which the goals, content, and activities are kept as comparable as possible.In the sections that follow, we provide a brief review of prior research on online professional development that informed the design and focus of the study presented here.We then describe the course and the design of the face-to-face and online versions of the course.We then describe the methodology employed to recruit and assign teachers to groups, as well as the data collection instruments employed for this study.Finally, we present the study findings and explore the implications these findings may have for the design and delivery of online professional development.

II. SUMMARY OF PAST RESEARCH
Since the early 1980's, there has been a lack of consensus about whether online instruction affects learning as well as face-to-face instruction.Believing that technology is a neutral carrier of content, Clark [3,4] argues that the use of a given technology for instructional purposes should not affect learning outcomes.Other researchers, however, argue that computer-based technologies have potential to be transformative [6,7,8].In the context of professional development for K-12 teachers, the use of technology to delivery professional development courses in an online forum offers several potential advantages over face-to-face instruction.
In contrast to the spontaneous verbal communication of face-to-face learning environments, asynchronous learning environments in which communication is primarily written may provide conditions that promote reflective inquiry [9,10,11].Threaded discussions available in online environments differ from face-to-face discussions in that they enable exchanges across time and space, provide a permanent record of interactions, and allow participants more time to reflect on a given topic before responding [9].While there are elements of real-world conversations in threaded discussions, there is nothing analogous to threaded discussions in the real world [12].
In a 5-year longitudinal case study, Delfino and Persico experimented with different techniques for teacher professional development including a face-to-face approach, an online approach, and a blended approach [13].Although the general objectives of the face-to-face and online courses were the same, Delfino and Persico found the online approach, which was free from time constraints and maintained a record of written interactions, promoted critical thinking and in-depth reflection on content.Similarly, when comparing discussions of graduate students in an online environment with those in a face-to-face environment, Meyer found that the use of threaded discussions increased the amount of time learners were engaged in discussions related to class objectives [14].Meyer indicates that learners in an online medium spent more time reflecting on course material, asking questions, and engaging in discussions about the course content.Given the speed with which face-to-face discussions occur and evolve, Meyer also suggests that online threaded discussions may benefit learners who require additional time and reflection to construct knowledge.
Despite the potential benefits that online discussions may have for supporting student learning, Garrison argues that when online learning simply recreates the conditions of a traditional classroom, it fails to exploit the unique strengths of online learning [9].While researchers have reached a broad consensus on the general components of high-quality K-12 teacher professional development in face-to-face contexts, many questions remain about the design and delivery of effective online professional development [15].
Several recent studies have focused on the characteristics of existing OPD programs, but little research has examined specific OPD formats and their impacts on teacher practice and student learning [16,17,18].Ginsburg, Gray, and Levin reviewed the quality and effectiveness of more than 40 online professional development programs, judging them against what is known about similar evidence for effectiveness and quality in traditional face-to-face professional development [17].Overall, Ginsburg et al. found that the OPD sites lacked rigorous independent assessments of the impact of OPD on teacher or student outcomes.Similarly, in a review of the literature on professional development for teachers, Lawless and Pellegrino concluded that success of professional development is typically judged from anecdotal evidence or the results of surveys in which teachers indicate their satisfaction with the experience or their perception of its usefulness to their work [16].Consequently, researchers know what teachers think about their professional development activities, not what they actually learn, how it changes their pedagogies, or how it impacts student learning [16].
Two recent studies, however, systematically varied specific characteristics of a professional development course and examined the impact on learning outcomes.In both studies, the learning goals, the course materials, and the assignments participants were required to complete were identical.In the first study, the course was delivered in two formats.For the first format, the course was led by a course facilitator who actively participated in discussions, guided participants through the course material in a paced manner, and encouraged interactions among participants through a threaded asynchronous discussion forum.For the second format, participants did not have access to a threaded discussion board, were unable to communicate with other participants via email, had limited discussions with the course instructor, and completed the course material at their own pace [1].
The second study contrasted four conditions.One was a highly supported condition, with a math education instructor, an online facilitator, and asynchronous peer interactions among participants as they worked through the course together.Another was a self-paced condition, in which participants did not have access to a math education instructor, could not communicate with other participants through online discussions or email, and simply submitted assignments to the instructor who did not provide any specific feedback to participants.A third condition included threaded discussions but were led by an online facilitator who did not have expertise in math education instruction.The fourth condition allowed participants to communicate online with an instructor with expertise in math education instruction, but participants were not able to communicate with each other [2].Both studies found that all versions of the course had positive effects on teachers' beliefs about math instruction, on their practices (as reported by themselves and their students), and on their understanding of mathematics instruction as measured by a mathematics test.Surprisingly, both studies also found that the positive effects did not differ among the conditions.Regardless of the level of facilitation, the level of expertise of the instructor, the pacing of the course, or the level of interaction among participants, the course had the same positive effects on teachers' pedagogical beliefs, instructional practices, and increased knowledge of teaching mathematics [1,2].This finding parallels findings from several studies that have compared the effects of traditional face-to-face instruction versus online instruction.Although small, this body of research consistently reports that no significant difference in learning outcomes occur between courses delivered in face-to-face or online modes [19,17,20,21].The study presented here extends the two previous studies that compared the effects of various online conditions, and seeks to expand the body of research that compares the effect that the mode of delivery, that is online versus face-to-face, has on the intended learning outcomes of the course.

III. BUILDING A SYSTEM OF TENS COURSE
The course employed for this study was titled Building a System of Tens.The course was originally developed by Deborah Shifter and her colleagues and is Part I in a series called Developing Mathematical Ideas (DMI).For the purposes of this study, the face-to-face version of this course was adapted by the Education Development Center to create an online version.
For the study presented here, both the online and face-to-face versions of the course required teachers to read the same material, experiment with the same activities in their classroom, and complete the same assignments within an eight-week period.In addition, to control for the potential effect of instructor quality, the courses were run by the same set of facilitators, each of who facilitated one face-to-face session and one online session.The intended learning outcomes of the course were also the same for both versions.These outcomes included the following: Increased understanding of number sense.Increased ability to identify misconceptions and errors that students apply when working on problems specific to number sense.Increased ability to identify and apply instructional strategies that hold potential to help improve student understanding of these concepts.Increased use of instructional methods that allow students to develop their thinking, share their understandings, and discuss their classmates' problem-solving strategies.
What differed between the two versions of the course was the mode in which the course was delivered.In the version we term "online", participants logged on to the course website to access course materials.The course website's discussion board provided the platform for participants to communicate with other course participants and the course facilitator.The online course followed a weekly schedule and participants were expected to spend three hours each week interacting online with their peers and completing course activities.Each online cohort had two course facilitators: one content expert and one online facilitation expert.The content facilitator provided individualized, instructional feedback on assignments and other direct instruction as needed.The content facilitators where experienced mathematics educators who had been trained in the DMI approach by the original course authors.The primary duties of the online facilitation expert were to send welcome and twice-weekly reminder e-mails, track participation, and provide technical assistance when the need arose.Course participants were given the email addresses and phone numbers of their course facilitators.The content facilitator followed the same syllabus employed by the face-to-face participants.Participants completed assignments weekly and submitted them to their content facilitator.These participants downloaded all course material and viewed videos online.
In the second version, which we term "face-to-face", participants attended weekly class sessions throughout the 8-week duration of the course.Each class lasted for approximately three hours.The face-to-face cohort had one course facilitator trained and experienced in DMI.Participants were given the email address and phone numbers of their course facilitators.While some participants contacted their facilitator via phone and email, the vast majority of the interactions occurred during class periods.The course facilitator followed a syllabus that was identical to that used for the online version.The facilitator provided individualized, instructional feedback on assignments and other direct instruction as needed.Participants were expected to complete weekly assignments and bring these assignments to class.Participants spent much of the class sessions working in small groups with other teacher participants.Some teachers independently exchanged email addresses with one another and communicated outside of class.
It should be noted that participants in both versions of the course were awarded graduate course credit and a stipend of $200 for participating in the course and completing all of the data collection instruments described below.

IV. STUDY DESIGN
The study presented here was designed to compare the learning outcomes of the same course delivered online or face-to-face.To limit class size, a total of six sections of the course were delivered, three online and three face-to-face.In addition, three facilitators with content-area expertise were hired to run the sessions.To control for the effect that a facilitator may have on the learning outcomes of the course, each facilitator ran one online version and one face-to-face version.
Participation in the study was limited to teachers in grades one through five who were currently teaching mathematics.Because one version of the course required teachers to meet face-to-face, three geographic areas were targeted, namely schools within easy driving distance of Bourne, MA, Newton, MA, or Manchester, NH.Principals of schools within easy driving distance of the targeted geographic areas were asked to invite teachers in grades 1 through 5 to participate in the professional development course.One hundred and fifty teachers who met the following criteria were selected to participate in the study: sCurrently teach math to elementary level students, and; Have a working email address, and Have access to a classroom of students with whom to work.
Teachers were stratified by geographic location and gender, and then randomly assigned to one of the two treatment groups: face-to-face or online.The face-to-face cohorts contained 21 teachers in Manchester, 16 teachers in Bourne, and 18 teachers in Newton.The online cohorts contained 28 teachers in Manchester, 35 teachers in Bourne, and 32 teachers in Newton.Note that the online versions were assigned a larger number of teachers because prior research indicated that approximately 40% of teachers taking the online courses did not complete the course.While we did not have data on the drop-out rate for face-to-face courses, we assumed that the drop-out rate would be smaller for face-to-face.Thus, to yield approximately the same number of participants in both settings at the end of the study, a larger percentage of teachers were randomly assigned to the online courses.

V. DATA COLLECTION INSTRUMENTS
The study presented here focused on the effect that the two versions of a professional development course had on the intended outcomes of the course.To collect data about the intended outcomes, six instruments were employed.These instruments included a background survey, a pedagogical beliefs and practices survey, a measure of teachers understanding of teaching the base ten number system, a student survey, a teacher log, and a course evaluation.As shown in Table 1, the background survey, pedagogy survey, teacher log, and the mathematics assessment were administered prior to the start of the course.The pedagogy survey, mathematics assessment student survey, teacher logs, and course evaluation were administered after the course was completed.Below each of the instruments is described briefly and pdf versions of the instruments are available at http://www.bc.edu/research/intasc/researchprojects /optimizingOPD/OPD.shtml.

A. Background Survey
The background survey was designed to collect demographic information and information about each teacher's prior experiences with professional development and technology use.The majority of the items on the background survey were closed-response.In a few instances, participants were requested to type in numeric values.In addition, the background survey contained items that were used to form four scales.These scales and their associated reliabilities included Prefer to Work on a Team (.78), Prefer to Learn on Own (.69), Proactive Regarding Own Professional Development (.56), and Access to and Use of Technology (.70).Each of these scales was standardized to have a mean of zero and a standard deviation equal to one.

B. The Pedagogy Survey
The pedagogy survey collected information about teachers' pedagogical beliefs and instructional practices, and was administered prior to and following the completion of the course.All items were closed-ended and asked teachers to either report the frequency with which they employed specific instructional techniques or to indicate the degree to which they agreed or disagreed with a statement that focused on the value of a given instructional practice.The vast majority of items employed for this survey were taken from two previous studies that focused on the relationships between pedagogical beliefs and practices and the use of instructional technology in the classroom [22,23].The specific scales that were formed and the associated reliabilities included

C. Math Assessment
The math assessment was designed to collect information about teachers' understanding of the base ten number system and related concepts covered in the professional development course.The assessment was administered twice, once in the first week and once in the final week of the course.The assessment included a set of multiple-choice items and open-response items that presented teachers with a sample of student work for a given problem specific to the base ten number system.The teacher was then presented with two sets of open-response items.The first set of three open-response items presented examples of student work and asked the teacher to explain the method the student used to acquire their answer and to evaluate whether the method makes mathematical sense.The second set of items presented the teacher with work performed by two students and asked the teacher the following questions: a) Explain what the students might have been thinking.Why does this method give the wrong answer?and b) As their teacher, what is an approach that you might take to help students understand why this method doesn't work?Include examples to clarify your approach.Using a scoring guide that was shown to yield reliable scores (inter-rater reliability of .80 or higher), two readers independently scored teacher responses and, when discrepancies occurred, the readers discussed their scores before reaching a consensus score.

D. Teacher Logs
The teacher logs were designed to capture information about teachers' day-to-day pedagogical practices.The teacher logs were administered twice, once during the first week of the course and once during final week of the course.During each administration, teachers completed three logs.Each log consisted of a series of instructional strategies similar to those explored in the course.For each strategy, teachers were asked to indicate whether the strategy was: (1) not used at all; (2) a minor component of the lesson; (3) a major component of the lesson; or (4) the most important component of the lesson.The ratings provided for each activity were then averaged across the logs recorded for each week.

E. Student Survey
To triangulate data provided by teachers via the pedagogy survey and the teacher logs, a survey was administered to students in the teacher's mathematics classroom.The study survey was administered twice, once prior to the start of the course and once following the completion of the course.The survey asked students to indicate the frequency with which they engaged in specific learning activities (e.g., performing worksheets individually, working with partners to solve problems, sharing solutions with their class, etc.) and how often their teacher employed specific instructional strategies (e.g., asking students to explain how they solved a problem, showing students how to solve a problem, asking students to respond to each others work, etc.).All items were forced-choice and were used to measure the frequency with which students and their teachers engaged in specific instructional and learning activities.The scale employed for these items ranged from: (1) almost always; (2) most of the time; (3) once in a while; and (4) never.Hence, a low score for a given item indicated more frequent use of the given instructional strategy.

F. Course Evaluation
The course evaluation was designed to collect information from teachers about the positive and negative aspects of the course.The course evaluation instrument asked participants to rate the value and quality of various aspects of the course including the reading material, the assignments, the facilitator, feedback on assignments, etc.

VI. FINDINGS
The primary research question examined in this study asked whether the mode of delivering the courseonline versus face-to-face-affected teachers' mathematical understanding, their pedagogical beliefs, and their instructional practices.To this end, the majority of analyses conducted for this study focused on comparing the effect of the two versions of the course on these three outcomes.But, to provide a better understanding of characteristics of the study participants, descriptive statistics were calculated for several items on the background survey.We begin, then, with results of the background survey.

G. Characteristics of Study Participants
The majority of teachers who participated in and completed the study were female (95%).The vast majority of teachers also identified themselves as white (93%).The age of participants was distributed fairly evenly across age groups, with 6% 25 years or under, 11% 26-30 years, 11% 31-35 years, 13% 36-40 years, 12% 41-45 years, 19% 46-50 years, 15% 51-55, and 14% older than 55.There were wider differences in the number of years participants reported that they had worked as a teacher.Six percent had taught for less than one year, 26% for 1-5 years, 18% for 6-10 years, 21% for 11-15 years, 8% for 16-20 years, 9% for 21-25 years, and 12% for more than 25 years.The vast majority of teachers reported that they did not have either a minor or a major in mathematics (93%), but did have full credentials for teaching in an elementary school (84%) and had a Masters degree (66%).Sixty-two percent of teachers also reported taking 1-5 professional development courses related to mathematics during the previous three years.
A t-test was conducted on each of these background variables and on the four background scales to examine whether the treatment groups differed with respect to their demographics and prior educational and work experiences.No statistically significant differences between the groups were found for any of the background variables.

H. Completion Rate
Of the 150 participants who agreed to participate in the study, 24 opted not to participate once they were notified of the treatment to which they were assigned.No data was collected from teachers who opted not to participate in the study.However, it is interesting to note that several of the teachers who opted not to participated noted that they were hoping to take the course online.
The actual number of teachers who began the study was 126.Of those, an additional 45 teachers either did not complete the course or did not complete the data collection instruments.Thus, of the teachers who initially agreed to participate in the study, 54% completed all components of the study.Of those teachers who actually began a course, 64% completed.While these completion rates are lower than we anticipated, they were consistent with past research on online professional development.As an example, the drop out rates for the two studies upon which this research builds were 46 and 44% [1,2].
Table 2 displays the drop-rates for each session of the course.For two of the three face-to-face sessions, the drop-out rate was noticeably lower than the online courses.However, the third face-to-face cohort experienced the largest drop-out rate of all sessions.Across cohorts, participants who did not complete the course were sent a survey inquiring about the reasons for their non-completion.Twenty-one drop-outs responded.Thirty percent of those drop-outs who responded indicated that personal reasons, such as family health issues or divorce, made it too difficult to participate in the course.Twenty-four percent of respondents indicated that the demands for the course were too high and an additional 19% stated that the time commitment was larger than they had anticipated.
Chi-square analyses were performed to examine differences in background characteristics for students who dropped-out and those who participated.Statistically significant differences were detected for four items.First, while 33% of those who completed the course reported that they had taken a course on the base ten number system previously, 67% of drop-outs had done so.This may indicate that the content of the course was more familiar for those who dropped out than it was for those who persisted.Second, a higher percentage of teachers who persisted through the course stated that they tend to complete tasks and assignments before deadlines.Third, a higher percentage of teachers who persisted through the course stated that they strongly value learning by seeing visual images such as pictures or movies.Finally, teachers who dropped-out of the course reported that they had taken more professional development courses related to teaching mathematics over the past three years as compared to those teachers who persisted.It is important to note that no statistically significant differences were found for characteristics such as preference to work on a team, preference to learn on own, being proactive regarding own professional development, and access to and use of technology.Finally, of those participants who dropped out, none of the background characteristics differed significantly between those who were in the face-to-face or the online version of the course.

I. Quality of the Course
Overall, participants believed that both versions of the course were well developed, well delivered, and valuable.Across all items on the survey that asked participants to rate the value of specific components of the course and the course overall using a four point scale that ranged from not valuable to very valuable, the mean fell between the valuable and very valuable.As a few examples, when asked to rate the value of the case studies, the mean rating was 3.67 with 4 representing "very valuable."When asked to rate the value of the expertise and feedback from the facilitator, the mean rating was 3.45.And when asked to rate their learning in the course compared to their previous face-to-face professional development opportunities, the mean rating was 3.82.
To examine whether participants in the online and the face-to-face groups had similar views about the course, a series of t-tests were conducted for each item on the end-of-course evaluation.After adjusting for multiple comparisons, a statistically significant difference was found for five items.Specifically, as one might expect, teachers in the online version of the course reported experiencing more technical problems while taking the course than did teachers in the face-to-face group (t=3.20,p<0.01).Teachers in the face-toface version also reported a higher value for performing the assigned math activities (t=4.72,p<0.01), sharing ideas with other participants (t=3.18,p<0.01), and receiving feedback from their facilitator (t=3.27,p<0.01).For all three of these items, however, the mean rating for both groups was above 3, indicating that both groups believed each of these were valuable.The final difference occurred on an item that asked, "If given a choice, I would prefer to take a course like this face-to-face as opposed to online."The scale for this item ranged from 1 (strongly disagree) to 4 (strongly agree).The mean rating for teachers in the face-to-face group was 3.41, while the mean rating for the online course was 2.40 (t=4.41,p<0.01).Clearly, those teachers who experienced the course in a face-to-face environment indicated that they would prefer this version in the future.In contrast, teachers who experienced the online version seemed less enthusiastic about taking a similar course in a face-to-face environment.

J. Changes in Pedagogical Beliefs
The Pedagogy Survey was designed to collect information about teachers' pedagogical beliefs.The survey was administered pre-and post-course, and the data was used to examine changes in teachers beliefs and practices that followed their participation in the course.It should be noted that the scale scores ranged from 1 (low value) to 4 (high value).Table 2 displays the mean scale scores for each treatment group for the preand post-course administrations.The column labeled Change displays the change in mean score between the pre-course post-course administrations.For the teacher confidence teaching math, student-centered beliefs, and comfort recognizing students' problem solving strategies scales, teachers scale scores increased significantly.While there was a slight decrease in teacher-centered beliefs, this change was not statistically significant.There were no statistically significant differences between the face-to-face and online groups for any of the pedagogical belief scales.

K. Changes in Instructional Practices
Teachers completed three teacher logs prior to the course and three logs following the completion of the course.As described above, the logs asked teachers to indicate the extent to which each type of activity was a component of the logged lesson.Table 3 displays the mean ratings given by teachers across lessons logged prior to and following the course.The column labeled post-pre indicates the change in the mean rating between the pre-course logs and the post-course logs.Within the face-to-face group, the extent to which these twenty-seven instructional strategies were elements of lessons changed significantly for fifteen strategies.Two of these strategies-having students do worksheets and stopping class to address discipline-saw declines in their occurrence.The remaining thirteen strategies experienced increases.Among the strategies that experienced increases were: developing and adapting instructional material; asking students to make conjectures, draw pictures, write about math, work with manipulatives, or solve problems related to everyday life; leading whole class discussions and asking follow-up questions to students' responses; and asking students to work in pairs or groups to solve problems.For the online group, significant changes were found for sixteen strategies.Half of these strategies were the same as those that saw changes in the face-to-face group.Only one strategy experienced a significant decrease-having students work independently.The remaining strategies experienced increases.
Independent sample t-tests were conducted to compare the amount of change that occurred for each item between the two groups.After adjusting for multiple comparisons, a significant difference was found for only one item-having students work independently.For this item, the face-to-face group experienced a small increase in reported use while the online group reported a significant decrease.The fact that a significant difference was found for only one of twenty-seven items suggests that the two versions of the course had similar effects on the use of many instructional practices explored during the course.

L. Student Survey Results
Similar to the teacher logs, analyses of the student survey focused on responses to individual items.For each item, the mean response for students within each class was used to estimate the frequency with which a given instructional or learning strategy was employed within the teacher's classroom.For each treatment group (face-to-face and online), the mean score for each item was then calculated across all teachers during each administration period (pre-course and post-course).Recall that the scale was structured such that a low score (1) represented frequent use of the strategy while a high score (4) indicated that the student reported never being exposed to the strategy.Table 4 displays the change in mean response within each group.Within the face-to-face group, statistically significant changes occurred for three items.Specifically, students reported increases in the frequency with which teachers asked them to come up with new ways to solve math problems and to write in words about their approach to solving a problem.Students also felt more strongly that their teacher tried to understand the student's approach to doing math problems.
For the online group, statistically significant changes occurred for five items.Students of online teachers reported a decrease in the frequency with which teachers had them work on math worksheets, but increases in the asking students to come up with new ways to solve problems, explaining how they got answers to problems, and working on extended math projects.Students also felt more strongly that their teacher tried to understand the student's way of doing math problems.
To examine whether the amount of change differed between the two groups, an independent samples t-test was performed for each item.After adjusting for multiple comparisons, no significant differences among groups were found.

M. Mathematics Understanding
To examine the effect that participation in the online professional development course had on teachers' knowledge of teaching mathematics, teachers completed a mathematics test that contained 15 multiplechoice items and 5 open-response items.Table 5 displays the mean total score for each treatment group for the pre-and post-course test administration.
Within each group, total test scores increased significantly.For the face-to-face group, scores increased by 1.86 points (on a scale of 1-20) while scores for the online group increased points.An independent samples t-test, however, indicated that the score increases did not differ significantly between the two groups (p = 0.48).

VII. SUMMARY AND DISCUSSION
The study presented here compared the effects that a face-to-face and an online version of the same course had on the intended outcomes of that course.For both versions, participants engaged in the same reading material, learning activities, writing assignments, and instructional activities in their own classrooms.The length of the course was also the same across both conditions (8 weeks) and the level of participation and interaction among participants was designed to be equivalent.The fundamental difference between the two versions was the medium in which participants interacted with each other and their facilitator, namely faceto-face or online.
The intended learning outcomes of both courses were identical.Specifically, the course was intended to affect teachers' pedagogical beliefs, their instructional practices when teaching mathematics, and their understanding of teaching the base ten number system.To examine the effect that this online professional development course had on these three outcomes, several data collection instruments were administered.A mathematics test was used to measure changes in teachers understanding of teaching the base ten number system.A survey was used to measure teachers' pedagogical beliefs.And the combination of a teacher survey, student survey, and instructional logs were used to measure teachers' instructional practices.
As described in greater detail above, both conditions of the course altered teachers' beliefs about teaching mathematics, changed their instructional practices, and, to a lesser extent, increased their understanding of teaching the base ten number system.On average, there were significant increases in teachers' confidence teaching mathematics, strength in their belief in student-centered instructional practices, and confidence analyzing students' problem solving strategies.As reflected in both the teacher's instructional logs and the student surveys, there were also several significant changes in teachers' instructional practices.Among these were asking students to develop new ways to solve math problems, trying to understand students' approaches to solving math problems, asking students to describe in writing how they solved math problems, having students work on extended math problems, and having students work in pairs or groups.Finally, a significant increase occurred on a test that measured teachers' understanding of teaching the base ten number system.Each of these outcomes is not surprising, particularly since teachers, on average, rated the quality of the course as high and reported that many components of the course provided valuable learning opportunities.
While a few differences in the magnitude of change were found between the two versions, overall the effects of the course were similar across the two conditions.Specifically, the improvement in teacher test scores did not differ significantly between the face-to-face and online versions.Similarly, there were no statistically significant differences in the amount of change in teachers' pedagogical beliefs.Among the twenty-seven instructional strategies teachers were asked to log, a significant difference was found for only one strategy-having students work independently.And, for the sixteen items on the student survey, there were no significant differences.Thus, overall the data suggests that, while both versions of the course affected the intended learning outcomes, the effects were similar across the two online and face-to-face versions of the course.This finding is consistent with previous research that generally reports similar effects for face-to-face and online professional development courses.
While the effects of the course were similar across delivery modes, it is interesting to note that there was a significant difference in opinions about whether, after completing the course, teachers preferred to participate in future professional development in face-to-face environment rather than online.While teachers who were in the face-to-face group expressed a strong desire for the face-to-face mode, those teachers who were actually able to experience an online version were less enthusiastic about taking future courses in a face-to-face mode.Since there were no differences in teachers' prior experiences taking online courses, this difference in opinion about taking future courses online or face-to-face may suggest that as teachers become more familiar with online professional development, they are more open to taking future courses online.
Nonetheless, these findings must be placed in the context of the many limitations of this study.First, the study focused on only one course delivered to elementary school mathematics teachers.Had a different course that focused on different types of teachers or different content been employed, the outcomes may have been different.
Second, both versions of the course experienced considerable attrition.On average, 36% of the teachers who began a course did not finish the course or the required data collection instruments.The fact that 36% of teachers did not complete the course may mean that the course either did not meet their needs or was too challenging for them.Although the characteristics of teachers who dropped out of the course did not differ between the two versions, it is plausible that had these teachers persisted, different findings may have resulted.
Third, the course employed for this study was well designed and employed high quality reading materials and learning activities.In addition, the course lasted for eight weeks.While the quality of the course was not compared to other professional development courses, it is likely that many other courses are of shorter duration, employ lower quality reading material, or ask participants to engage in less effective classroom activities.For a course that is shorter in length or that employs materials of a lower quality, the mode of delivery may have a larger impact on the outcomes of the course.
Finally, all of the participants in this course were volunteers.Although a small stipend was offered to compensate participants for the time required to complete the data collection instruments, the teachers who participated and completed the course were likely highly motivated individuals who were sincerely interested in developing their mathematics teaching skills.In many cases, however, participation in professional development is required by a school or district.In such cases, some teachers may be less motivated or engaged in the learning.If presented with an online version of the course, some of these teachers might be unmotivated to log into the course, complete the reading and activities, or make a minimal effort when completing assignments.The resulting effects of the course might then be smaller.
Despite these limitations, this study provides further evidence that an online course designed as an interactive learning experience can have very positive effects on teachers knowledge, pedagogical beliefs, and instructional practices, and that the magnitude of these effects are similar to those that occur in a faceto-face environment.

Table 3 : Summary of Teacher Log Mean Scores by Treatment Group
**Change between groups statistically significant at p=.05.