The Impact of Attitudes, Beliefs, and Cognitive Reflection on the Development of Critical Thinking Skills in Online Students

Learning and development of critical thinking (CT) skills in higher education is essential for academic achievement. The following experiment is the first to examine the effect of online student’s perceptions and attitudes towards CT across dimensions of confidence, valuing, misconceptions, cognitive reflection, and authors writing. Furthermore, a CT intervention was developed, and the effects of the intervention examined with an aim to help students improve their grade point average. The analyses demonstrated that student’s confidence and cognitive reflection predict academic achievement. Moreover, the online CT intervention was associated with improved students’ CT attitudes, skills, and academic performance. Significant interactions were observed between time (preand post-intervention) and intervention in cognitive reflection, confidence, beliefs, and attitudes related to CT, and student grade point average (GPA, as a measure of student’s performance on online modules). It was concluded that the CT can be taught and that an intervention based on “how to think” rather than a “what to think” mixed approach can help online students develop CT, strengthen their confidence in CT and help students improve their academic performance in an online setting.

Critical thinking (CT) is a core skill in higher education and most educators emphasize the importance of fostering students' CT skills to prepare them for "real-world" challenges (Facione, 1990;Halpern, 2014;Puig et al., 2019). CT skills are considered essential to learning in higher education (Thompson, 2011) not only because they facilitate high academic performance, but because they are associated with higher rates of employability, stronger public engagement, and firm financial position (Facione & Facione, 2001;Osborne et al.,2013). However, although learning and developing CT skills represent an important area of pedagogy, there remains a need for further research in online environment (Hussin et al., 2019;MacKnight, 2000).
Online programs in higher education are no different from on-campus programs in terms of promoting CT, and yet unfamiliar teaching techniques in online environment can cause difficulties regarding the best methods for fostering CT (Hussin et al., 2019). Attempts have been made to promote CT using online discussion boards and text-based communication (e.g., Belcher et al.., 2015) and course curriculum changes (e.g., Nold, 2017). Although positive evidence towards promoting critical thinking in online learning has been found (e.g., Arend, 2009;Lunney et al., 2008;Swart, 2017), most studies present significant limitations, such as not measuring critical thinking at pre-and post-intervention, not outlining clear and replicable procedures, or focusing strictly on discussion boards. There is also a tendency to focus on specific assessment-focused aspects of learning rather than promoting CT skills more generally, which may be transferable to real-life (Maurino, 2007). Furthermore, only limited research exists in online environments, which tests the effectiveness of teaching approaches such as discussion boards, focus group, problem-based and task-based strategies in promoting CT (Guiller et al., 2008;Parker et al., 2015;Richardson & Ice, 2010). As such, a consensus on how to define, measure, and nurture CT skills through educational effort in online environments is yet to be achieved. Facione (1990, p. 2) defined CT as "a purposeful, self-regulatory judgment that results in interpretation, analysis, evaluation, and inference as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based". This definition captures the multifaced nature of CT and may explain the difficulties that educators face when teaching and promoting CT interventions (Tsui, 2002). Systematic reviews consistently indicate that teaching CT should adopt "how to think" instead of "what to think" approaches to advance teaching strategies that may influence the development of CT skills (Cloete, 2019;Puig et al., 2019). This is particularly important because the teaching of CT skills in higher education has been identified-globally-as an area requiring improvement with students reporting difficulty in understanding and demonstrating CT in their assessments  and many teachers often lack understanding of what CT encompasses and how to teach it (Janssen et al., 2019). For example, Duro et al., (2013) explored students' and lecturers' understanding of critical thinking and found a mismatch between students' understanding of CT and lecturers' expectations. They recommended structured interactive CT exercises to enhance students' critical metacognitive processes for the development of strong arguments.

Review of Literature
Previously, CT has been described as a metacognitive process, believed to be pivotal in logical thinking, decision-making, and problem-solving (Halpern, 2003). This metacognitive process is associated with the analytic system posited by dual process theorists in cognitive psychology (Bonnefon, 2016;Kahneman, 2011). Dual process theories of thinking and reasoning propose a qualitative contrast between "Type 1" automatic, fast, and implicit processes (e.g., intuitions or gut-feelings) and "Type 2" analytic processes that are conscious, effortful, and self-regulatory. Type 2 processing is associated with executive function and working memory capacity but also other factors that are important in CT such as aptitude for reflective judgments, beliefs and confidence, and propensity to avoid miserly processing of information by relying on Type 1 processes (Toplak, West, & Stanovich, 2014). Thus, it is expected that CT entails cognitive and metacognitive components to enable learning and application of CT skills.
Recent systematic reviews (Abrami et al., 2015;Dwyer et al., 2014;Puig et al., 2019;Ross et al., 2013) have identified a range of measures related to aspects of CT such as dispositions (e.g., the California Critical Thinking Disposition Inventory [CCTDI], Facione et al., 2001), problem solving (e.g., the California Critical Thinking Skills Test [CCTST], Facione et al., 1992) and CT-related abilities to process information in a logical manner (e.g., Watson-Glaser Critical Thinking Appraisal [WGCTA], Watson & Glaser, 1964). These measures were widely used to assess students' CT; however, there were concerns raised about their psychometric properties as the validity and reliability of these measures is difficult to establish Bernard et al., 2008). Furthermore, these measures do not cover some of the very important elements of CT such as metacognitive thinking and cognitive aptitudes (Liu et al., 2014).
The importance of metacognitive thinking was highlighted as an important factor of CT (Dwyer et al., 2014), whereby metacognitive thinking entails self-regulation of thoughts by using executive function, such as attention, memory, and higher-order cognitive skills for application of the CT skills. As such, application of CT skills depends on cognitive reasoning processes and metacognitive monitoring where confidence and beliefs in CT knowledge, and cognitive reflection may help students to engage in effortful thinking and to prevent errors in reasoning. For example, students' positive attitudes and beliefs about CT significantly correlates with their ability to override false beliefs, the ability to assess arguments strengths and are highly significant predictors of students' grade point average (GPA) (Stupple et al., 2017). Furthermore, an ability to override incorrect responses, by engaging in more effortful and actively open-minded thinking strongly correlates with beliefs and attitudes in CT and academic performance (Frederick, 2005;Heijltjes et al., 2015;Klaczynski, 2014;Stanovich, 2011;Stanovich et al., 2016;Stupple et al., 2017). Thus, the measures related to attitudes and beliefs in CT and cognitive reflections could be used to teach students about common misconceptions about CT and how to challenge those misconceptions by facilitating reflection on their CT and by engaging them in analytic thinking. Moreover, these can help students to engage deeply with the learning and teaching material presented to them and help identify themselves as the authors of their assessments-rather than superficially seeking course credit and passively regurgitating material for grades rather than knowledge.
Indeed, attitudes and beliefs in CT are important for creation of authorial identity that is in turn important in assessment writing. Authorial identity epitomizes the attitudes and beliefs that a writer has of themselves as an author and the way they express themselves in terms of the construct and critical analysis in their writing (Elander et al., 2010). Authorial identity is moreover related to the development of CT, particularly in terms of developing writing skills (Cheung et al., 2015;Elander et al., 2010). Cheung et al., (2015) demonstrated correlations between beliefs and attitudes about CT and authorial identity suggesting an important link between the development of CT skills and the development of writing skills through education. Indeed, the application of their learned academic literacy is not only essential to the construction of a good argument (Elander et al., 2010) but also may lead to prevention of unintentional errors such as plagiarism (Abasi et al., 2006). Thus, examination of students' sense of ownership of written work and authorial ways of thinking may help the development of pedagogic interventions. Furthermore, teaching students about authorial identity may help strengthen their confidence, attitudes, and beliefs about their CT skills related to the critical evaluation of the written argument.
Considering the multifaceted nature of CT, it is understandable why educators face difficulties when teaching and promoting CT interventions. A CT intervention was developed (consisted of four parts) with an aim to include several important aspects of CT (Ennis, 2016). The first part of the intervention provided material and guidance for students to gain some basic understanding of general CT skills (e.g., Facione, 2000) and instructions on how to recognize and apply them in real life situations. The second part focused on the importance of metacognition (e.g., Stanovich, 2018) and highlighted the importance of "sharpening" metacognitive skills by thinking of common errors in reasoning and avoiding biases. The third part focused on the real-life examples from Twitter, fake news and biases, fallacies, and heuristics based on the dual-processes theories (Evans & Stanovich, 2013). The last part covered general instructions on how to approach assessments and develop basic strategies when preparing assessment. Thus, the overall goal was to help students become critical thinkers by using "how to think" rather than "what to think" approach based on the evidence that teaching strategies that encourage, stimulate, and facilitate students' acquisition and transfer of thinking skills are essential for CT development (Beyer, 2008;Halpern, 1993).
The primary goal was to assess students' critical thinking ability, attitudes, and beliefs on critical thinking. The Cognitive Reflection Test CRT (CRT; Toplak et al., 2014) was chosen because it measures analytic thinking which theoretically underpins CT (Halpern, 2014). The Critical Thinking Toolkit (CriTT; Stupple et al., 2017) and the Student Attitudes and Beliefs about Authorship Scale (SABAS; Cheung et al., 2015) were chosen because they measure attitudes (e.g., aptitudes, beliefs, and confidence related to CT and academic writing) that are important in higher education and the development of the CT skills (e.g., Dwyer et al., 2014). Secondly, it was assessed whether the intervention improved CT in students. GPA was used to examine whether the intervention improved GPA and as a direct measure of the correlation between CT and student academic achievement . The intervention design was based on mixed approaches (for a review, see Tiruneh, Verburgh, & Elen. 2014) whereby the importance of metacognition, real-life examples related to cognitive errors and biases, fake news, and general assessments preparation were covered. This was to address the complexity and breadth of CT development and help students to effectively communicate their critical thinking.
It was predicted that students' attitudes, beliefs, and confidence would predict willingness to engage with the workshops. Second, it was also predicted that positive attitudes, beliefs, confidence, and higher cognitive reflection would positively predict variance in students' GPA. Lastly, it was predicted that the students who engaged with the workshops would score higher on CRT, CriTT, SABAS, and GPA after the workshops than students that did not engage.

Participants
To determine target sample size, an a priori power analyses using G*Power (version 3.1.9.2) was conducted. Based on the suggestion related to the common effect size in educational research (Hattie, 2008), for a small effect size and a standard alpha level of .05, a minimum of 187 participants would be required to have 80% power in correlational analyses. The second a priori power analysis was conducted for the group comparisons analysis for a small effect size of 0.3, alpha adjusted level of .001 1 for multiple comparisons, a minimum of 54 participants would be required to have 80% power in the mixed ANOVA analyses. We aimed to recruit upwards of 200 participants to account for incidents of missing data and participant withdrawals. A total of 191 university students (Mage = 28.01 years, SD = 8.62; 62.23 % female) responded to an online advertisement distributed via email lists and module announcements, these participants were entered in the correlational analysis. For the intervention, participants who responded the advertisement expressing their interest to take part in the study were randomly allocated to either a control waiting list or intervention. A total of 58 participants (Nintervention = 37, Ncontrol = 21) completed all required parts of the study and were entered in the group comparison analysis (mixed ANOVAs). All participants were enrolled in an online British Psychological Society-accredited Master's in Psychology degree at the authors' institution. Inclusion criteria required that participants to be fluent in English, aged 18 years or over, and currently studying in an online capacity at the University. Although the intervention was made available to all students, participants reporting current diagnoses of psychiatric, affective, or neurological disorder likely to impact one's ability to learn or critically analyse were asked not to take part in pre-or post-measures. Participants provided written informed consent in accordance with approved university research ethics protocols and British Psychology Society ethical guidelines by ticking a box on both the first and last pages of online survey.

Materials
The CriTT (Stupple et al., 2017). The CriTT comprises 27 items that measure student perceptions of and attitudes towards critical thinking across dimensions of confidence, valuing, and misconceptions (e.g., "Critical thinking is essential in higher education"), using a 10-point scale. Each item is rated using a scale anchored from "1 -Strongly disagree" to "10-Strongly Agree" This scale was originally tested with 133 students and showed high reliability (Confidence (Cronbach's α = .92); Valuing (Cronbach's α = .79); Misconceptions (Cronbach's α = .60)). For this research, only the total CriTT scores were calculated and analyzed, with high scores indicative of greater levels of more positive perceptions of critical thinking. The scale also showed high reliability in our study (Cronbach's α = .89). (Toplak et al., 2014). The CRT comprises seven items that measure one's ability to resist and override intuitive responses by engaging analytic ability (e.g., "A bat and a ball cost $1.10 in total. The bat costs a dollar more than the ball. How much does the ball cost?"). Here, one's intuitive response is to state that the ball costs $0.10 (totalling $1.20), when the correct answer is $0.05. Each item is rated using as either correct or incorrect, with higher scores indicative of greater levels of cognitive reflection This scale was originally tested with 160 students and showed high reliability (Cronbach's α = .72). The scale also showed high reliability in our study (Cronbach's α = .79).

The Seven-item CRT
The SABAS (Cheung et al., 2015). The SABAS comprises 17 items that measure beliefs and attitudes about academic writing (e.g., "I am able to document my ideas clearly in my writing") using a 6-point scale anchored from "1 = Strongly Disagree" to "6 = Strongly Agree." High scores indicated stronger beliefs about and more positive attitudes towards academic writing. This scale was originally tested with 445 students and showed high reliability (Cronbach's α = .89). The scale also showed high reliability in our study (Cronbach's α = .88).
Intervention. The intervention was designed and created by the research team for the purpose of this research using built-in screen capture software for a Macbook Pro (macOS Mojave, v.10.14.6), and was embedded within Qualtrics survey software to monitor compliance and to record responses. The intervention was approximately one hour in length (4 parts of 15 minutes) and focused on the enhancement of critical thinking through: observations, inferences, and assumptions (e.g., Halpern, 2003), fallacies, biases, and heuristics in reasoning and critical thinking (e.g., Kahneman & Tversky 1984;Tversky & Kahneman, 1983), argument evaluation (e.g., Stanovich & West, 1997), and contrasts in evidence comparison (e.g., Dunn et al., 2008). The intervention combined written and spoken materials and asked participants to complete tasks throughout. The survey could not move onto the next section until a task specific amount of time had passed.

Procedure
On expressing interest to take part in the research, participants accessed the baseline online survey, whereby they entered their demographic information and completed the CriTT, CRT, and SABAS. Participants were randomly allocated to either an active (intervention) or waiting list group and were emailed information on this placement by a researcher (JM), who would remain blind to the subsequent data collection, analysis, and interpretation. Over a fourweek period, participants in the active group received weekly intervention materials via email and for completion that week. Each element of the intervention took approximately 15 minutes to complete. Following the final element of the intervention, all participants were sent a followup survey, asking them to complete the CriTT, CRT, and SABAS. After completion of the study, links to the intervention were made available and debriefing material were provided to all participants. GPAs were recorded before and after completion of the semester. The range was from 0 to 100 and consists of the average grades in the modules each student had completed at the time of the study. Analytic strategy and scoring Data were log transformed (LG10) if they didn't meet normality criteria (e.g., CriTT and SABAS subscales for logistic and multiple regression). Initial analyses used logistic regression to check whether student's willingness to engage with intervention could be predicted based on their scores in CRT, CriTT, and SABBAS. Next, a multiple regression was conducted with three predictors: the CRT (Toplak et al., 2014); the CriTT (Stupple et al., 2017) and the SABAS (Cheung et al., 2015) with GPA at the beginning the semester as the outcome variable. Finally, four mixed ANOVA (N = 58) analyses examined the effect of the intervention (before and after semester) and intervention attendance (students attended or did not attend intervention) on GPA, CRT, CriTT, and SABAS.

Regression scores before semester
A logistic regression (Enter method) tested baseline differences in CRT, CriTT subscales (Confidence in CT, Valuing CT, and Misconceptions), and SABAS subscale (Authorial Confidence, Valuing Writing, and Identification with Author) as predictors of participation in the critical thinking intervention. The overall fit of the model is assessed using the log likelihood statistics (Table 1). The model was significant (p = .001) and indicated that students who took part in the intervention had lower confidence in their CT (CriTT Confidence) and authorial identity (SABAS Authorial Confidence). A multiple regression (Enter method) tested the relative predictive strength of CRT, CriTT subscales (Confidence in CT, Valuing CT and Misconceptions), and SABAS subscale (Authorial Confidence, Valuing Writing and Identification with Author) for GPA baseline scores. After controlling for age and sex, data indicated that the seven predictors combined reliably accounted for 4% of the variability in GPA. The Beta for both CRT and CriTT Confidence scores showed a positive correlation. This indicated that the higher scores on CRT and CriTT Confidence were associated with greater GPA. The remaining variables were not significant predictors of GPA. A Factorial Mixed Measures Design was used to examine the effect of time (before and after semester) and workshop-attendance on student GPA. The interaction effect between time and workshops attendance on GPA was also examined.

Table 2 Multiple Regression Analysis of Cognitive Reflection Test, Critical Thinking Toolkit and Student Attitudes and Beliefs about Authorship as predictors of student's Grade Point Average
Data were analyzed using a 2 (Time) × 2 (Workshop attendance) ANOVA. There was a significant interaction between time and workshop attendance showing that workshop attendance improved after semester scores F(1, 56) = 58.51, p < 0.001, ƞp 2 = 0.51 such that GPA after the semester increased for the students that attended workshop whereas GPA did not increase for the student that did not attend workshop (Table 3). However, a significant main effect of Time F(1, 56) = 4.35, p > 0.01, ƞp 2 = 0.07 and the main effect of workshop were not significant F(1, 56) = 0.21, p = 0.65, ƞp 2 = 0.004. Overall, the results indicated that the workshop intervention increased GPA for students that attended the workshop. Next, a Factorial Mixed Measures Design was used to examine the effect of time (before and after semester) and workshop attendance on student CRT scores. The interaction effect between time and workshop attendance on CRT scores was also examined.
Data were analyzed using a 2 (Time) × 2 (Workshop's attendance) ANOVA. There was a significant interaction between time and workshop attendance F(1, 56) = 11.55, p = 0.001, ƞp 2 = .17 where CRT scores after the semester increased for the students that attended workshop while CRT scores did not increase for the student that did not attend workshop (Table 4). There was a significant main effect of Time F(1, 56) = 18.51, p < 0.001 , ƞp 2 = 0.25. Student's CRT scores were higher after they completed the semester than before they started the semester. However, the main effect of workshop was not significant F(1, 56) = 1.93, p = 0.17, ƞp 2 = 0.03.
Overall, the results indicated that the workshop attendance increased CRT for students that attended the workshop. Furthermore, CRT scores were overall lower before than after semester. Next, a Factorial Mixed Measures Design was used to examine the effect of time (before and after semester) and workshop attendance on student CriTT scores. The interaction effect between time and workshop attendance on CriTT scores was also examined.
Data were analyzed using a 2 (Time) × 2 (Workshop attendance) ANOVA. There was a significant interaction between time and workshop attendance F(1, 56) = 27.51, p = 0.001, ƞp 2 = .33 where CriTT scores after the semester increased for the students that attended workshop while CriTT scores did not increase for the student that did not attend workshop (Table 5). However, the main effect of time F(1, 56) = 0.19, p = .66, ƞp 2 = 0.003 and the main effect of workshop were not significant F(1, 56) = 0.35, p = 0.55, ƞp 2 = 0.006. Overall, the results indicated that the workshop attendance increased CriTT for students that attended the workshop. Last, a Factorial Mixed Measures ANOVA examined the effect of time (before and after semester) and workshop attendance on student SABAS scores. Data were analyzed using a 2 (Time) × 2 (Workshop's attendance) ANOVA on students' SABAS scores related to attitudes and beliefs about academic writing. There was no interaction between time and workshop attendance F(1, 56) = 0.76, p = 0.38, ƞp 2 = 0.01. The main effect of time F(1, 56) = 1.82, p = 0.18, ƞp 2 = 0.03. and the main effect of workshop were also not significant F(1, 56) = 0.01, p = 0.97, ƞp 2 < 0.001 (Table 6).

Discussion
The current study replicated previous findings (Stupple et al., 2017) that CT abilities, attitudes, and confidence predict students' GPA at baseline level. As predicted, attitudes, beliefs, and confidence related to both CT and academic writing predicted students' willingness to engage with the workshops. Moreover, it was observed that students who engaged with CT intervention exhibited improved GPA, CT skills, and CT attitudes scores. Results are discussed in detail in the following sections.
Intervention participation and engagement. The results from a logistic regression indicated that students who took part in the intervention had lower confidence in their CT (CriTT Confidence) and authorial identity (SABAS Authorial Confidence). This suggests differences in confidence levels between the intervention and control groups at baseline even though participants were randomly assigned to groups. These results can be interpreted in combination with the mixed ANOVAs. The lack of significant main and interaction effects of authorial confidence indicates that the intervention might have failed to address a lack in authorial confidence. However, even though students in the intervention group started with lower confidence in critical thinking, this was significantly improved with the intervention (see discussion of intervention effects). Such findings require further examination and represent tentative evidence in the predictive value of the CriTT in identifying students who lack confidence in their CT skills (Stupple et al., 2017), considering that the students who lacked confidence chose to sign up for the workshops. This coincides with evidence that identifying, challenging, and building stronger confidence in attitudes and beliefs about CT is an important facet for the development of students' CT skills (Celuch et al., 2009).
Differences in GPA at baseline. The current findings indicate that attitudes and beliefs towards CT (CriTT) and cognitive reflection (CRT) predicted differences in GPA at baseline. Specifically, it was found that students who are more confident about CT and scored higher on cognitive reflection had achieved higher grades in modules completed before the intervention. These findings are in line with the predictions and give support to previous literature showing that confidence and attitudes towards CT and cognitive reflection predict academic performance (Stupple et al., 2017). This indicates that the development of CT skills depends on dispositional attitudes that have an impact on student's confidence to develop and demonstrate their CT skills (Ennis, 1985). Moreover, this further implies that to successfully develop CT interventions, it is not only important to understand how students think and reason, but also what they believe and how they structure their belief system (Lamont, 2020).
In line with previous research the current findings indicate that there is a strong relationship between CT and cognitive reflection (Kember et al., 2000;Kraft, 2002;Kuiper, 2002). Based on the results from this study and previous research, it is reasonable to assume that cognitive reflection-the ability to analyze and critically evaluate information and arguments-provides students with the capability not only to engage in learning strategies but also to use executive functions to be more reflective and organized in preparation for their assessments (Dwyer et al., 2014;Phan, 2006). Thus, reflective students are more likely to overcome uncertainties, to critically evaluate their argument, and to monitor and self-regulate their thinking. Considering the above, the various versions of CRT currently available may represent a potent measure of cognitive reflection that can be applied in higher education. The ability to measure the tendency to override initial biases in uncertain conditions and predict variance in students' GPA can be an important tool in understanding aptitude and identifying where to target tailored support for student who most need it (Simonovic et al., 2018;Stupple et al., 2017;Toplak et al., 2014).
Intervention effects. The most notable contribution of the present study is the indication of a strong relationship between online CT intervention and students' CT attitudes, skills, and academic performance. Significant interactions were observed between time (pre-and postintervention) and intervention in CRT, CriTT, and GPA in students that took part in the intervention. Participants who took part in the intervention improved their scores in these three aspects significantly more than participants who did not take part in the intervention. Where there was a non-significant main effect of intervention, this might be explained by metaanalytic findings suggesting improvements in CT typically vary throughout temporal stages of educational courses (Huber & Kuncel, 2016). However, it remains unclear to what extent any gains are sustainable and so further exploration is required. Nevertheless, an explicit discussion of CT is important, which is one of the strengths of the present study. Abrami et al.'s (2008) meta-analysis with 117 studies indicated that CT interventions with explicit instructions and where it was part of the course objectives had the strongest effects, whereas immersion interventions (when CT content is simply embedded in the course and not part of the objectives) had the lowest effects. There is a strong relationship between our intervention and GPA increase, but only in students that were engaged in our intervention. The results of this study are encouraging given the suggestion that one hour intervention can produce significant effects, albeit only in students that were motivated to engage. Thus, the results indicate that motivation to engage in CT is also the key to success. Therefore, this can easily be included as part of a program of learning without adding a significant workload to students.
CT is rewarded in HE and has been consistently a moderate predictor of student achievement, as observed by Fong et al., (2017) in a meta-analysis with 23 studies, which assessed student achievement in different ways (e.g., retention in community college, degree attainment, and course completion or achievement related outcomes such as grades, GPA, or tests). Although some researchers suggest that CT skills might be decreasing in university students (Huber & Kuncel, 2016), our current findings inform that such skills are still relevant for academic achievement and need to be fostered in HE. Even though, the debate whether and how to improve CT skills is still ongoing (e.g., Puig et al., 2019), our results indicate that CT skills in online students can be improved by using a mixed method approach and providing students not only a specific instruction related to the assessments, but also teaching students "how to think" in more general terms about CT and the importance of metacognitive awareness about their thinking.

Limitations and Future Direction
Limitations of our research, and future directions are as follows. First, participants selfselected to take part in the study that explicitly informed that the intervention might have the potential to improve CT skills. Although a possible bias, this was circumvented as much as logistically and ethically possible using a waiting-list procedure whereby students in the comparison group also expressed their interested in taking part in the intervention-suggesting equivalent motivation between the two groups (although this was not formally assessed). Online students can lack motivation to succeed compared to students that attend face-to-face course (Stark, 2019). Thus, considering that motivation may positively influence CT (e.g., Riggs, 2014), and that lack of motivation could be one of the explanations for participant's lack of engagement with interventions, it is important that future studies assess students' motivation. It is also important to note that the students who engaged with all elements of our task were a subset of online learners who were motivated to enhance their thinking and learning skills.
Second, academic achievement was only measured as a function of GPA. Butler et al. (2017) suggested that students with higher CT skills also report more positive life events compared to students with lower CT skills. Therefore, future studies should consider the benefits of a CT intervention beyond academic achievement, including real-life events. Finally, even though the intervention was designed with an aim of enhancing general CT skill, it is noted that CT skills may be transferable between contexts; however, the current study only considered a short period of time. Thus, it is not certain if the learned skills are transferable and what the lasting effects of the intervention are.
There are further limitations with the use of the CRT. Some items are increasingly wellknown (e.g., the bat and ball problem). Most CRT questions are mathematical and there are some gender differences in performance (e.g., Campitelli & Gerrans, 2014). Thus, some caution should be exercised when using variants of the CRT when predicting grades, particularly among math-anxious individuals (Morsanyi et al., 2014). There are, however, an increasing variety of cognitive reflection tests available that have varying difficulty levels and reduced reliance on mathematical ability (Thomson & Oppenheimer, 2016), and as such, with careful item selection and variation they can provide a useful tool in higher education settings.

Conclusion
To our knowledge, this was the first study to examine the effect of an explicit critical thinking intervention with online HE students. The results of this study indicate that CT skills can be enhanced with brief online workshop interventions, and that cognitive reflection, attitudes, and beliefs play an important part in the development of students' CT skills both through orienting toward opportunities to develop these skills and academic outcomes. Furthermore, it was demonstrated that a short, mixed-method intervention can improve students' GPA. Although there are some limitations to this study, the results are encouraging for offering opportunities to students to develop CT skills in addition to modules and class workload.

Declarations
All authors contributed to the conceptualization, design, and writing of the paper. The author(s) declare no potential competing interests with respect to the research, authorship, and/or publication of this article. This work was supported by internal funding (Undergraduate Research Scholarship Scheme; £2,200) from University of Derby.
Data are available on request from b.simonovic1@derby.ac.uk The study was approved by the University of Derby Undergraduate Research Scholarship Scheme Ethics Committee. Consent to participate and consent for publication were obtained from participants.