Understanding Teachers ’ Cognitive Processes during Online Professional Learning : A Methodological Comparison

This study examined the effectiveness of three types of think aloud methods for understanding elementary teachers’ cognitive processes as they used a professional development website. A methodology combining a retrospective think aloud procedure with screen capture technology (referred to as the virtual revisit) was compared with concurrent and retrospective think aloud procedures. Elementary teachers from a large metropolitan area were assigned to one of the three think aloud conditions (N = 45). Participants in the concurrent condition verbalized their thoughts while simultaneously navigating a professional development website for 20 minutes. Participants in the retrospective condition verbalized their thoughts following their 20-minute website navigation without any aids. Finally, participants in the virtual revisit condition verbalized their thoughts while viewing a screen recording of their website navigation. Think aloud protocols were analyzed to determine the frequency of cognitive processes verbalized by participants in each condition. The findings of this study indicated significant differences in the types of verbalizations produced by participants across the three think aloud conditions. In addition, findings reveal benefits and limitations of employing each type of think aloud method in the context of a professional development website.


Understanding Teachers' Cognitive Processes during Online Professional Learning: A Methodological Comparison Introduction
Elementary teachers are a necessary foundation for building successful programs in the classroom (Gambrell & Anders Mazzoni, 1999;Pressley, Mohan, Raphael, & Fingeret, 2007).Successful programs begin with a repertoire of pedagogical knowledge, pedagogical content knowledge, and research-based instructional practices.This repertoire of information can be delivered to practicing elementary teachers through various professional development opportunities (Cervetti, Kulikowich, Drummond, & Billman, 2012;Desimone, 2009;Kao, Wu, & Tsai, 2011).
One facet of teacher professional development is online learning, which occurs when professional knowledge is constructed from multiple modes of digital information-photographs, videos, and interactive tools, to name a few (Mayer, 2002).Online learning is a favored approach to professional development because it creates accessible opportunities; online learning takes place within platforms that deliver information in a means that removes time, place, and situational barriers (Kanuka & Nocente, 2003).Online learning opportunities have also been shown to have positive effects on and even change teachers' pedagogical and content knowledge, classroom practice, and student outcomes (Weschke & Barclay, 2011).As elementary teachers increasingly turn to the Internet for their professional learning (Charalambousa & Ioannou, 2011;Kao et al., 2011), it is essential to examine how they use and learn from online resources and professional development websites.
Most studies that have examined online teacher learning have gathered data through surveys, questionnaires, and interviews (Duncan-Howell, 2010;Hur & Brush, 2009;Kao et al., 2011).These methods offer information about teachers' attitudes towards online professional learning; however, data generated from these methods is limited to participants' recollection of past events.A method that tracks teachers' cognitive processes as they make online choices is necessary to provide further insight into how teachers use and learn from online environments.The think aloud methodology is an approach that can track teachers' cognitive processes during decision-making activities.While think aloud research is extensive, studies that compare the effectiveness of different think aloud methodologies for understanding teachers' cognitive processes as they navigate online resources are limited (Kuusela & Paul, 2000;van Gog, Paas, van Marrienboer, & Witte, 2005).The purpose of this comparative study was to examine the effectiveness of three types of think aloud methods for understanding elementary teachers' cognitive processes as they used a professional development website.A methodology combining a retrospective think aloud procedure with screen capture technology (referred to as the virtual revisit) was compared with concurrent and retrospective think aloud procedures.
A detailed discussion of the think aloud methodology sets the foundation for this paper.The current study's methods are then presented, followed by the results and a discussion of the significant findings.The study's limitations and educational implications conclude this paper.

Literature Review
Thinking aloud has historical roots in introspection analysis, a form of data collection aimed at investigating psychological claims and theories of mind during the eighteenth century (Boren & Ramey, 2000;Ericsson, 2002).The cognitive revolution of the 50s and 60s produced alternative types of verbal reports of thinking to gather information about cognitive structures and processes (Ericsson, 2003).Today, the think aloud method most widely employed is based on the techniques of protocol analysis by Ericsson andSimon (1984, 1993).As described by Ericsson and Simon (1984), thinking aloud captures cognitive processes in real time and verbal reports "provide the most informative data available on thinking during cognitive tasks" (Ericsson, 2003).Cognitive processes underlying decisions and behaviors are usually "hidden from direct observation" (Gaissmaier, Fifc, & Rieskany, 2010, p. 141).However, the think aloud method makes monitoring cognitive processes possible-the think aloud generates direct data about the ongoing cognitive processes that occur during task performance (Jaspers, Steen, van den Bos, & Geenen, 2004).Ericsson and Simon (1984) describe three levels of verbalizations that can occur during the think aloud method.The first two levels require information processing in the participant's short term memory and the third level requires additional cognitive resources and retrieval of information from long term memory (Olmsted-Hawala, Murphy, & Hawala, 2010).While Ericsson and Simon (1984) state that Level 3 verbalizations or higher cognitive processes are less reliable because they involve access to long-term memory, usability researchers suggest that this type of data provides useful information about online learning, website user goals, and online behaviors (Boren & Ramey, 2000;Guan, Lee, Cuddihy, & Ramey, 2006;Olmsted-Hawala et al., 2010).
Usability researchers most often employ the concurrent and retrospective think aloud methods to gain insight into web seeking behaviors and to evaluate a website's content and ease of use (Aranyi, Schaik, & Barker, 2012;Barzilai & Zohar, 2012;Branch, 2006;Kuusela & Paul, 2000).During the concurrent procedure participants verbalize their thoughts aloud while they simultaneously complete a task.Verbal reports that result from the concurrent procedure generate data about the website user's navigational experience.For instance, Aranyi and his colleagues (2012) conducted an exploratory study of interaction experience with a news website.The concurrent think aloud yielded five categories of experience based on the participants' evaluative statements: impression, content, layout, information, architecture, and diversion (Aranyi et al., 2012).Similarly, Barzilai and Zohar (2012) utilized the concurrent procedure to examine epistemic thinking in action.Data was collected to shed light on the relationship between sixth grade students' knowledge construction and their online practices.Analysis revealed a positive relation between students' online strategies and their epistemic cognition (Barzilai & Zohar, 2012).Damico and Balidon (2007) also employed the concurrent procedure to examine how elementary students engage with an educational website.Findings from their study highlight how elementary students evaluated claims and evidence of online educational resources (Damico & Balidon, 2007).
The retrospective procedure is also referred to as post-task testing, retrospective report, and think after.Retrospective think alouds alone are used less often in the fields of online learning and website usability since they require participants to think aloud after a task has been completed.An international survey found that just 5% of think aloud studies (not limited to website usability) employed the retrospective technique, whereas 89% used the concurrent think aloud, and 6% used an alternative think aloud (McDonald, Edwards, & Zhao, 2012).
One of the main reasons why retrospective think alouds are used less often is due to the fact that the procedure relies on the ability to recall decisions after a task has been completed.As participants recall their decisions, information may be incomplete and include errors, omissions, and substitutions (Branch, 2006).For instance, a comparative study that examined retrospective and concurrent verbal protocol analysis in the context of a decision-making task found retrospective reports more prone to errors of omission whereas concurrent reports contained more relevant information about the decision making process (Kuusela & Paul, 2000).
While retrospective procedures are limited by the fact that they may be incomplete and include errors, omissions, and substitutions (Branch, 2006), they have the advantage of freeing cognitive resources by thinking aloud after the task has been completed-retrospective think alouds do not interfere with task performance (McDonald et al., 2012).Concurrent think alouds, on the other hand, can interfere with task performance since participants verbalize their thoughts while they simultaneously complete a given task-participants engage in two different processes at the same time.When two processes occur simultaneously there is an increase in cognitive load-"the level of mental energy required to process a given amount of information" (Ping Lim, 2004, p. 17).As a result of a higher cognitive load, task completion may be compromised during the concurrent procedure and resulting think aloud reports are often procedural in nature (McDonald et al., 2012;van Gog, Kester, Nievelstein, Giesbers, & Paas, 2009).Findings of such studies suggest that alternative think aloud methods warrant attention.
An alternative to the concurrent and retrospective procedures is the virtual revisit think aloud method.The virtual revisit is a variation of the retrospective think aloud method and allows participants to review and comment on a visual recording of how they interacted with a particular website.The goal of the virtual revisit is to aid recall of original events and thought processes by using a screen-capture recording of participants' navigational experiences.Similar to cued retrospective reporting where participants are given instructions to report retrospectively on the basis of a record of observations (van Gog, Paas, van Marrienboer, & Witte, 2005), the virtual revisit think aloud combines a retrospective think aloud with screen capture technology to aid recall of original events and thought processes.
Despite the limitations of the concurrent procedure, it has been widely used in usability research, mostly as a means to evaluate a given website-participants verbalize their thoughts about the ease of use and accessibility of information.While evaluative data contributes to the refinement of professional development websites, user experience is a complex process and usability research should go beyond evaluating websites to include a range of cognitive processes and learning strategies (Dillon, 2001); the virtual revisit think aloud has the potential to avoid the limitations of the concurrent and retrospective procedures.In addition, few studies have been undertaken to compare the relative utility of different think aloud procedures (Kuusela & Paul, 2000;van Gog, Paas, van Marrienboer, & Witte, 2005).The current study addresses these gaps in the literature by examining the utility of three think aloud methods during online professional learning.
The following research questions guided the study: 1.To what extent do participants' verbalizations differ across the three think aloud methods? 2. What are the benefits and limitations of employing each type of think aloud in the context of online learning?

Methods
This research was undertaken within the context of the development and refinement of a professional development website.The website is a multimedia evidence-informed literacy professional development website that provides free professional learning resources for elementary teachers and educators.The website is highly complex and interactive, and includes virtual tours of exemplary classrooms (PreK-6), video clips of expert teachers explaining and demonstrating effective educational practices, detailed lesson plans, photographs of teaching materials, exemplars of student work, and links to related research articles.

Participants
Forty-five practicing elementary teachers from a large metropolitan area participated in this research over an eight-month period.All participants completed informed consent forms and volunteered to participate in this study.

Data Sources
Demographic questionnaire.A demographic questionnaire was administered to participants to obtain data on a range of relevant factors based on the literature related to teacher development and online learning (e.g., age, gender, years of teaching experience, education, extent of involvement with various web-based technologies).Participants were asked to complete the questionnaire online prior to a one-on-one meeting.The questionnaire was administered through an online survey program.After participants completed the questionnaire, they were placed in one of three think aloud conditions (concurrent, retrospective, and virtual revisit).The conditions are described below.Stratified random assignment was employed to ensure that certain demographic features were represented within each group as equally as possible.Stratification variables were selected based on the literature on web navigation and teacher professional development.Research has found that gender (Page, Robson, & Uncles, 2012;J. Pearson, A. Pearson, & Green, 2007), age (Laberge & Scialfa, 2005), subject matter knowledge and experience (Laberge & Scialfa, 2005), and computer self-efficacy (Page et al., 2012) influence the perception and navigation of websites; therefore, comparable aspects including gender, age, years of teaching experience, current teaching grade, comfort with technology, and frequency of Internet use for professional purposes, were selected as the key variables used for the group assignment.
Table 1 (next page) summarizes the demographic characteristics for the participants across the conditions.As can be seen from the table, participants in the three groups were quite similar with respect to gender, age, years of teaching, current grade, comfort with technology, and frequency of Internet use.In other words, the stratified random assignment was successful.Think aloud.Participants completed a think aloud during a one-on-one meeting.

Concurrent think aloud.
Participants in the concurrent condition verbalized their thoughts for 20 minutes while simultaneously completing a website task.

Retrospective think aloud.
Immediately following a 20-minute website task, participants in the retrospective condition recalled and verbalized their thought processes without any aids.
Virtual revisit think aloud.Immediately following a 20-minute website task, participants in the virtual revisit condition reviewed their online choices virtually and verbalized their thoughts while viewing the 20-minute screen recording of their explorations.

Screen capture technology.
During participants' navigation of the website, each visual step was captured with Camtasia Studio, a screen-recording computer software program developed by TechSmith (Uppal, 2011).

Procedure
One-on-One Meeting.The one-on-one meetings followed a sequence of events and lasted approximately 45 minutes.

Website task and think aloud. The following website task instructions were presented to all participants:
Your task is to use a professional development website as you normally would when seeking information online for your teaching practices.
While the website task instructions were consistent across the conditions, the think aloud instructions varied for each condition.Participants in the concurrent condition were given the think aloud instructions before they completed the website task, whereas participants in the retrospective and virtual revisit groups were given the think aloud instructions after they completed the website task.The purpose of informing participants in the retrospective and virtual revisit conditions of the think aloud instructions after their navigation was to reduce reactivity-"influences of the verbalizations on the decision process" (Ranyard & Svenson, 2010, p. 119)-as much as possible.The following passage outlines the think aloud instructions.The underlined portions state the different think aloud instructions given for each condition.
In this study we are interested in what you think about when you explore a professional development website.In order to do this, I am going to ask you to think aloud (concurrent condition: as you explore the website; retrospective condition: about your exploration of the website; virtual revisit condition: while you view a recording of your exploration of the website).What I mean by think aloud is that I want you to tell me everything that you are/were thinking from the time you begin/began exploring the website until the end of your exploration.I would like you to talk aloud constantly.I don't want you to try to plan out what you say or try to explain to me what you are saying.Just act as if you are alone in the room speaking to yourself.It is most important that you keep talking.
While think aloud studies most often employ a specific task, an open-ended task was used in this study to reflect as naturally as possible how teachers use and learn from professional development websites.To reduce disruption to the participants' cognitive processes, prompts, redirections, and interventions were kept to a minimum during the process of verbalizing (Jaspers, 2009).Participants who were silent for a period of 30 seconds were only told to "keep talking."This prompt was only given to one participant in the concurrent condition.

Data Analysis
Audio recordings were transcribed verbatim resulting in 45 think aloud transcripts.Word counts were calculated for each of the three conditions (concurrent, retrospective, and virtual revisit).As shown in Table 2 there are clear differences in the average number of words participants generated in the three think aloud conditions.The coding scheme used to code all 45 transcripts was generated based on several studies relating to website usability (Aranyi et al., 2012;Cooke, 2010;Damico & Balidon, 2007;van Gog et al., 2005;Tan & Wei, 2006;Zhao & McDonald, 2010), teacher planning and decision making (Kansanen et al., 2000;Moos, 2014), and Ericsson and Simon's levels of verbalizations and suggested statements (1984).The final coding scheme used to code all 45 transcripts includes 11 categories.Table 3 summarizes the coding scheme and offers a description of each category and an example from the think aloud transcripts.The categories listed in Table 3 are types of cognitive processes.Cognitive processes underlie the study of decision-making during professional learning experiences and can be categorized as higher order processes and lower order processes.Higher order processes, such as reasoning, involve access to "thematically related information in long-term memory" (Horz & Schnotz, 2010, p. 238).Information stored in memory "is interrelated and rearranged, and extended to achieve a specific purpose" (Lewis & Smith, 1993, p. 136).Lower order processes, such as procedural knowledge, are normally executed in an automated way and are "only marginally influenced by intentional processes" (Horz & Schnotz, 2010, p. 238).In the current study, higher order cognitive processes refer to planning, connecting, reasoning, reflecting, and evaluating, whereas lower order cognitive processes refer to diversion, understanding, describing, and reading.The higher order and more complex cognitive processes "involve the manipulation of information" (McLoughlin & Mynard, 2009, p. 148), whereas the lower order cognitive processes demand only "mechanical application of previously acquired information" (Lewis & Smith, 1993, p.133).
The organization of the categories into higher and lower order cognitive processes is consistent with Krathwohl's revised taxonomy of educational objectives (2002).Krathwohl (2002) organizes six major categories and 19 sub-categories of the cognitive domain hierarchically and discusses how they differ in complexity.Krathwohl (2002) distinguishes between higher and lower order cognitive processes: the more complex categories (e.g., create) are higher on the scale, whereas the less complex categories (e.g., recalling) are lower on the scale.This distinction was taken into consideration during the development of the coding scheme for the current study.
Prior to coding, the think aloud transcripts were first segmented or "unitized" into thought units-each utterance was deemed a separate segment or thought unit if it conveyed relevant information and was preceded and followed by a pause and a change of ideas (Lincoln & Guba, 1985).According to Ericsson and Simon (1993) this procedure, in which protocols are unitized into phrases or segments provides more reliable findings.A second researcher was trained on dividing the think aloud transcripts into segments or thought units to establish interrater agreement.The second researcher was not involved in the research project and had no specific interest in the outcomes (van Someren, Barnard & Sandberg, 1994).This was necessary to provide results that were as objective as possible (van Someren et al., 1994).Training involved a review of unitizing, a demonstration of segmenting the transcripts into thought units, and a practice trial of segmenting a portion of one of the transcripts into thought units.As described by van Someren et al. (1994), during think aloud protocol analysis "coders need to be trained in the use of the coding scheme" and the context should be considered when interpreting individual phrases (p.128).
Following the training session, two researchers segmented 10% of the total transcripts into thought units.The percentage of agreements was calculated (agreements/agreements + disagreements).The unitizing reliability check on 10% of 45 transcripts indicated high reliability, with an inter-rater agreement of 95.3% (Guetzkow's U = .012;Guetzkow, 1950).Due to the high inter-rater agreement and to the submission that unitizing involves subjective interpretation and contextualization (Lomard, Snyder-Duch, & Campanella Bracken, 2004), the remaining transcripts were unitized by the primary researcher, who had a thorough understanding of the research topic.
Once all 45 transcripts were unitized, the total thought units were calculated.Similar to the differences in word counts, there are clear differences in the average number of thought units generated by participants in each think aloud condition (see Table 4).Similar to the unitizing procedure, a second coder who was not involved in the research project and had no specific interest in the outcomes was trained on the coding scheme to establish inter-rater agreement.Following the training session, 10% of the transcripts were coded by the primary researcher and second coder.Disagreements were discussed and resolved until an inter-rater agreement of 97% was reached.Cohen's Kappa was performed to determine consistency among raters and was found to be 0.98, CI (0.978-0.996).Due to the high inter-rater agreement and the assertion that researchers themselves may serve as coders (Lombard, Snyder-Duch & Campanella Bracken, 2002), the remaining transcripts were coded by the primary researcher.
Thought units were tallied to provide frequency counts for each category.These frequency counts were then transformed into percentages based on the total number of thought units across categories.Coding resulted in a total of 4,521 thought units.The frequencies of thought unit were converted to percentages for each participant.The percentage of thought units in each condition were then analyzed quantitatively.Analyzing the percentage of thought units, as opposed to the frequency of thought units, produces a more accurate representation of the cognitive processes (Rosenzweig, Krawec, & Montague, 2011).This allows for a more accurate comparison of thought units across conditions.The word counts and total number of thought units were also analyzed quantitatively.Descriptive statistics, ANOVA, and Pearson correlations were performed on the proportion of thought units, total word counts, and total number of thought units across the three conditions (concurrent, retrospective, and virtual revisit).Table 5 displays the distribution of the frequency and percentage of thought units across the three conditions.

Results
A one-way between subjects ANOVA was performed to compare the percentage of each category verbalized by participants in each think aloud condition.The analysis revealed a significant main effect for planning, F(2, 42) = 7.05, p = .002,η 2 = .251;connecting, F(2, 42) = 8.35, p = .002,η 2 = .251;reasoning, F(2, 42) = 22.01, p < .001,η 2 = .512;and reflecting, F(2, 42) = 5.36, p = .008,η 2 = .203.ANOVA results also indicated significant main effects for describing website features, F(2, 42) =11.05, p < .001,η 2 = .345;and for reading, F(2, 42) =43.81, p < .001,η 2 = .676.There were no significant main effects for the other five categories.Table 6 displays the summary statistics for the main effects and the means and standard deviations for each thought unit variable.The means indicate the average proportion of each type of thought unit across participants.Post hoc comparisons using Tukey HSD revealed that participants in the retrospective and virtual revisit conditions verbalized a significantly greater proportion of planning thought units than participants in the concurrent condition (p < .05).In addition, participants in the retrospective condition verbalized a significantly greater proportion of connecting thought units than participants in the concurrent condition (p = .001),and participants in the virtual revisit condition verbalized a significantly greater proportion of reasoning thought units than participants in the other two conditions (p < .05).Furthermore, post hoc tests using Tukey HSD revealed that participants in both the retrospective and virtual revisit conditions verbalized a significantly greater proportion of reflecting thought units than participants in the concurrent condition (p < .05).Finally, participants in the concurrent condition verbalized a significantly greater proportion of describing website features thought units and reading thought units than participants in the other two conditions (p < .05).
With respect to the remaining thought units, findings indicated no significant differences between the three conditions for evaluating, diversion, understanding, and describing procedural behaviors.This suggests that regardless of the type of think aloud employed, participants will verbalize a relatively equal number of thoughts related to evaluating the website, to their confusion and understanding of the web-based tools and information, and to descriptions of their own actions and online behaviors.
A one-way between subjects ANOVA was also performed to compare the frequency of words and thought units verbalized in each think aloud condition.The analysis revealed a significant main effect for both variables across the three conditions: F(2, 42) = 130.42, p < .001,η 2 = .861for word count; and F(2, 42) = 90.76,p < .001,η 2 = .812for number of thought units.As Table 7 shows, participants in the retrospective condition verbalized the fewest number of words (M = 658.80,SD = 229.00)and thought units (M = 30.73,SD = 13.15).Participants in the concurrent condition verbalized more than twice the number of words as participants in the retrospective condition (M = 1676.13,SD = 394.89)and more than four times the number of thought units than participants in the retrospective condition (M = 147.40,SD = 35.83).Participants in the virtual revisit condition verbalized the most number of words (M = 2637.87,SD = 359.94).However, participants in the virtual revisit condition verbalized less thought units than teachers in the concurrent condition (M = 123.27,SD = 20.58).This finding suggests that thought units verbalized by participants in the virtual revisit condition contained more words than thought units verbalized by participants in the concurrent condition.Post hoc comparisons using Tukey HSD revealed significant differences in the frequency of words (p < .01)and the frequency of thought units (p < .05) between all three conditions.In contrast to the participants in the concurrent and virtual revisit conditions who were asked to verbalize their thoughts for 20 minutes, participants in the retrospective condition were not given a time constraint to verbalize their thoughts.Although an unlimited amount of time was given to participants in the retrospective condition, these participants verbalized their thoughts for an average four minutes 30 seconds.Therefore, it is not surprising that participants in the retrospective condition verbalized significantly fewer words and thought units than the other two conditions.A more interesting finding is the difference in word counts and number of thought units between the concurrent and virtual revisit conditions.Participants in the virtual revisit condition verbalized significantly more words than participants in the concurrent condition; however, the reverse is true for the number of thought units.This finding indicates that thought units produced by participants in virtual revisit condition contained a greater number of words.This suggests that the thought units verbalized by participants in the virtual revisit condition were more complex than the thought units verbalized by participants in the concurrent condition.For example, as a participant from the concurrent condition views the homepage for the first time she describes her procedural behavior: "I'm going to have a look at the How To Videos."A participant from the virtual revisit condition who also views the homepage for the first time goes further to provide a reason for her behavior: "I'm always interested in ways to increase my students' background knowledge of vocabulary and comprehension so I go back and forth between comprehension and vocabulary before I narrowed it down and selected vocabulary."Both of these thought units were verbalized during the participants' initial view of the home page.However, the thought unit verbalized by the participant in the virtual revisit condition is more complex in that it provides a reason for her navigational choice.While participants in the concurrent condition verbalized on average more thought units, the verbalizations were less likely to include reasons for their decisions.Examples of thought units produced by participants in the concurrent condition include: that's interesting; let's see what that is; there's a word wall; that's like what we did in kindergarten.
Pearson correlations.Pearson correlations were computed to determine three relationships: between the cognitive processes, between the frequency of word counts and thought units, and between the cognitive processes and word counts and thought units (see Table 8).According to the results, higher order cognitive processes (planning, connecting, reasoning, and reflecting) were positively correlated to other higher order cognitive processes, and negatively correlated to lower order cognitive processes (diversion, understanding, describing website features, and reading).Similarly, lower order cognitive processes were more likely to be positively correlated to other lower order cognitive processes and negatively related to higher order cognitive processes.For instance, reading was negatively correlated to planning (r = -.42,p < .01),connecting (r = -.39,p < .01),reasoning (r = -.50, p < .01),reflecting (r = -.55,p < .01),evaluating website content (r = -.31,p < .05),and evaluating user experience (r = -.37,p <.05).In contrast, reading was positively related to diversion (r = .46,p < .01),understanding (r = .30,p < .05),and describing website features (r = .45,p < .01).These findings corroborate the above ANOVA results and suggest that participants in the concurrent condition, who were more likely to read text during the think aloud, were less likely to verbalize thought units related to higher order cognitive processes.
Pearson correlations also revealed a positive correlation between the frequency of words and the frequency of thought units (r = .70,p < .01).This finding indicates that as the number of words increased the number of thought units increased.Results also revealed a negative correlation between higher order cognitive processes and the number of thought units verbalized.This suggests that teachers who verbalized thought units related to higher order processes were more likely to verbalize fewer thought units than teachers who verbalized thought units related to lower order processes.

Discussion
The findings of this study indicated significant differences in participants' verbalizations across the three think aloud conditions.One possible reason for these differences relates to cognitive load.Cognitive load refers to the amount of mental effort in working memory: cognitive load "represents the load that performing a particular task imposes on the cognitive system" (Pass & Van Merrienboer, 1994, p. 122).Cognitive load may have impacted participants in the concurrent condition who were required to use a higher level of mental energy to process the given information as they simultaneously completed the website task (Ping Lim, 2004).The cognitive load on working memory may have diminished the quality of their verbalizations.This is consistent with Ericsson and Simon (1993) who state that concurrent reporting may become difficult to maintain under high cognitive load conditions.Cognitive load research also indicates that during multimedia tasks, concurrent reporting will interfere with information processing and limit the extent of the thought units (Nielsen, Clemmensen, & Yssing, 2002;van Gog et al., 2009).As the participants in the concurrent condition verbalized their thoughts while simultaneously completing the website task, the cognitive demand on their working memory increased: their brains prioritized information processing over verbalizations.Therefore, participants in the concurrent condition verbalized fewer thought units related to higher cognitive processes (Cooke, 2010).
In contrast, participants in the retrospective and virtual revisit conditions had the advantage of thinking aloud after the website task had been completed; thus, the cognitive load on their working memory was lighter during their think aloud than those in the concurrent condition.As a result of a lighter cognitive load, participants in the retrospective and virtual revisit conditions had more cognitive resources available during the think aloud; therefore, they could focus their mental energy on constructing and verbalizing complex ideas.
For example, planning is a complex cognitive activity and involves visualizing the future, producing and generating new information, and "putting elements together to form an original product" (Krathwohl, 2002, p. 215).Planning is also an indicator of teachers' intentions for practice (Clark & Peterson, 1986;Krathwohl, 2002).The planning thought units verbalized by participants in the retrospective and virtual revisit conditions contained complex information that demonstrate intentions for practice.For instance, a video on social studies and writing integration led a participant in the retrospective condition "to think of a different idea that [she] could bring into [her] class which includes doing cross-curricular work with probability and ancient civilizations and trading cards and games."Similarly, a teacher in the virtual revisit condition described how she could modify a lesson to include cross-curricular integration: "Building interviewing and report writing skills, I know that there is a lot that I can do with this in terms of reading and writing and oral and drama and themes like social justice and history and so on."These examples demonstrate complex verbalizations; teachers are planning as they use the professional development website-they are visualizing the future and beginning to generate new ideas.
Another possible reason for the findings can be drawn from research on information processing and memory recall.Information processing is enhanced when new incoming information is connected to prior knowledge and previous experiences (Mastin, 2010;Weber, Corrigan, Fornash, & Neupauer, 2003).Information processing is also enhanced when new material is interesting to the learner (Garner & Gillingham, 1991).The more deeply new information is processed (i.e., through connections and interest), the more likely it will be recalled.In contrast to participants in the concurrent condition, participants in the retrospective condition were able to process information during the website task on a much deeper level because they did not have the same cognitive demands of verbalizing their thoughts while simultaneously completing the website task.They were able to verbalize significantly more connections with past experiences and interests than participants in the concurrent condition.For instance, a participant in the retrospective condition connected components of the website to her current reading program: After visiting a virtual tour, I was interested in going through all the different parts of the balanced literacy and I clicked on a couple of things that interested me and that I'm working on, like comprehension skills and fluency and word building.
In addition, participants in the retrospective condition did not have access to a visual cue or memory aid.It is possible that in the absence of a visual cue, participants in the retrospective condition verbalized more connecting thought units because they recalled meaningful memories established during the website task.For instance, a participant in the retrospective condition connected website content on hand writing to a student in her class: "I saw something about hand writing which made me think about a student that I have in grade one who…it's quite a struggle for her to read what she's writing."The information that this participant recalled was meaningful to her because it directly related to the needs of a student in her classroom.participant in the virtual revisit condition was 40 minutes.For some researchers, this may be costly.Moreover, the time required to employ the virtual revisit think aloud may limit participant involvement.
Collectively, the findings indicate that the virtual revisit can avoid the limitations of the concurrent and retrospective procedures and provide thorough and descriptive thought units and insights into how teachers use and learn from a professional development website.

Study Limitations
There were three main limitations to this study: (1) factors that may have caused reactivity, (2) the use of one professional development website, and (3) the possibility of researcher bias.First, factors may have caused reactivity during the think aloud procedure.Reactivity occurs when task performance is altered as a result of an awareness of the study task.Reactivity may have occurred as a result of participants' awareness that they were completing a task in the presence of the primary investigator.A "motivational shift" in which the participants anticipate exposure of their think aloud protocol may have occurred when the participants were informed of the think aloud procedure (Russo, Johnson, & Stephens, 1989).Another factor that may have caused reactivity is hearing one's own voice.The additional aural stimulation may have interfered with the concurrent navigation (Russo et al., 1989).In general, reactivity was reduced as much as possible during the one-on-one meeting by: staying neutral during the task and think aloud, keeping verbal and nonverbal cues to a minimum, and providing participants in the retrospective and virtual revisit conditions with the second part of the instructions after they had completed the website task.However, the factors described above should be considered when interpreting the findings.
Secondly, the current study was context-specific and used one professional development website.Future research comparing the three think aloud conditions should be conducted with additional websites and online resources.Conducting similar studies with alternative professional development websites will enhance the credibility and transferability of the results.
Finally, it is difficult to eliminate researcher bias.Steps were taken to reduce researcher bias as much as possible: the use of relevant literature to develop the coding scheme, unitizing the transcripts based on Lincoln and Guba's (1985) procedures, using a second coder procedure, and staying as close to the data as much as possible during the analysis and interpretation of the findings.However, this limitation must be considered when interpreting the findings.To avoid researcher bias, future research could involve a team of researchers with varying backgrounds, particularly during data analysis.

Significance and Educational Implications
The significance of this research is that it compares the effectiveness of two traditional think aloud methods-concurrent and retrospective-with a think aloud method combining a retrospective procedure with screen capture technology (the virtual revisit).While think aloud research is extensive, studies that compare the effectiveness of different think aloud methodologies for understanding cognitive processes as website users navigate online resources are scarce.Based on the findings of this study, there may be potential benefits and limitations to employing each type of think aloud method.In addition, the virtual revisit think aloud, a relatively underused type of think aloud method, appears to be an effective method for examining teachers' cognitive processes as they use a professional development website.The virtual revisit think aloud method produces thorough verbalizations that incorporate reasons behind the decision-making process.
Another significance of the current study is that the participants were practicing teachers with between one and over 25 years of classroom experience.Much of the research examining teachers' beliefs and attitudes towards learning in online environments has studied preservice teachers.Studying the cognitive processes of practicing teachers has the potential to contribute to the understanding of teacher professional development and teacher cognition.
Furthermore, the virtual revisit think aloud method also has potential that transcends the specific domain in which it is applied in the current study.Virtual revisits with think aloud could be applied to examine cognitive processes of participants in research involving online learning and website use in many domains in education.If researchers provide feedback to educational website developers based on the virtual revisit methodology, improvements can be made to the design and content of their sites.

Conclusion
Teachers learn in many different aspects of practice, including their classrooms, their school communities, professional development courses, and online environments (Borko, 2004).To understand teacher learning, it must be studied within these multiple contexts and it must be studied with an effective methodology that provides rich and thorough data about the reasoning process.Gaining greater insights into teachers' cognitive processes as they navigate online environments can lead to the reconsideration of the design of online learning environments so that they "are more conducive to informal learning…so that they further develop the ability of professionals to solve problems and learn independently" (Lohman, 2006, p. 144).

Table 2
Average Word Counts across the Three Conditions

Table 3
Coding Scheme-Categories Used to Code the Transcripts

Table 4
Average Number of Thought Units across the Three Conditions

Table 5
Frequencies and Percentages of Thought Units

Table 6
ANOVA Summary Statistics for Thought Units

Table 7
ANOVA Summary Statistics for Word Count and Number of Thought Units

Table 8
Pearson Correlations