The Scale of Online Course Anxiety: Assessing College Students’ Anxiety in Online Courses

The purpose of this study was to develop an instrument to measure student online course anxiety, a factor that detrimentally affects student learning in the online environment. Based on Keegan’s theoretical framework that identified fundamental differences between online education and traditional education, the instrument of Scale of Online Course Anxiety (SOCA) was developed and tested with a sample of 170 students from a 4-year higher educational institution. The total score and the four subscale scores show high reliability. Confirmatory Factor Analysis exhibited solid goodness of fit between SOCA items and the factor structure hypothesized in previous literature. Evidence of divergent validity shows SOCA differentiates the state anxiety and trait anxiety as expected. Limitations and possible topics for future research are also discussed.

initial registration of the threat stimulus, (b) activation of a primal threat mode, and (c) evocation of elaborative and reflective thinking. Expanding beyond Beck's earlier work, other scholars (Salkovskis,1985;Salkovskis & Warwick, 1986) emphasized that by removing the responsible cognitions, impinged individuals could reduce or even remove the threat that produces anxiety.
From the cognitive perspective on anxiety, Spielberger (1966) proposed a conceptual framework that differentiates state-and trait-anxiety, suggesting that they need to be assessed separately. According to Spielberger (1966), trait anxiety is a stable condition related to the personality, which stimulates certain responses to threatening situations. On the other hand, state anxiety is an individual's perception of harm or threatening situations and is exhibited as a transient emotion. Each type of anxiety is considered unidimensional, and an individual's global level of anxiety needs to be studied and assessed separately (Spielberger, 1966;Muris et al., 1998). Spielberger's view of anxiety has been broadly accepted by researchers in multitudinous areas such as medicine, psychology, and education. Using the instrument developed by Spielberger and his colleagues, the State-Trait Anxiety Inventory (STAI, Spielberger et al., 1983), many researchers have delved into different "states" (i.e., types) of anxiety, which has long been an essential focus within the context of education, including the anxiety related to language (Horwitz et al., 1986;Young, 1991;Al-Shboul et al., 2013), the anxiety in statistics (Baloğlu et al., 2011;DeVaney, 2010), and test anxiety (Cassady, 2004;Conneely & Hughes, 2010) to name just a few. Like any other "state," online learning holds its own characteristics and may be perceived as threating and raise anxiety. Considering the above, the following section will focus on the anxiety of online learning.

Online Learning Anxiety
In the new era of online education, researchers have attempted to apply traditional anxiety assessment tools to assess anxiety in the online setting. For instance, with a sample of 69 students, Stowell and Bennett (2010) alternated the order of two examinations to overcome the order effect and administrated two examinations in the online and the traditional classroom conditions. Students' anxiety in the two conditions was measured with the Academic Emotions Questionnaire (AEQ; Pekrun et al., 2002). The study showed that the online environment affects student academic performance differently depending on their original anxiety level: for those who were anxious about learning, the online situation detrimentally affects their performance. In another study, DeVaney (2010) compared the statistics anxiety of 120 graduate students enrolled in traditional on-campus or online statistics courses. Their statistics anxiety was measured through the Statistics Anxiety Rating Scales (STARS; Cruise et al., 1985) and attitude towards statistics through the Survey of Attitudes toward Statistics (STATS-28; Schau et al., 1995) in pre-and post-test conditions. Comparing statistics anxiety as well as attitudes toward statistics between the online and on-campus student groups, the researcher found higher levels of anxiety towards statistics for students in the online setting. Furthermore, he also found that students in the online courses held less favorable attitudes toward statistics as reflected in their emotional experience and perceived difficulty of the course. This study revealed the potential challenges that online courses might pose for students in a statistics class. There were still other researchers studying anxiety on a "macro-level" by looking into student anxiety as a composite status of anxiety stemming from various sources on the online educational platform. For example, Bolliger and Halupa (2012) followed a series of doctoral courses delivered online and gauged students' satisfaction and anxiety where students' anxiety was assessed through computer anxiety, internet anxiety, and online anxiety. It was found that these sources of anxiety have a negative correlation with students' satisfaction with their online learning experience. Similar to Bollliger and Halupa's (2012) research, computer anxiety (Rosen & Weil, 1995;Saadé & Kira, 2007), and internet anxiety (Montelpare & Williams, 2000) were frequently studied together with test anxiety, competence, and satisfaction in research on online anxiety. Research investigating means to alleviate the anxiety for online students was also a vital topic, such as the study by Abdous (2019) that investigated how an online learning orientation could better prepare students and result in lower level of anxiety. His research utilized a one-item scale on 4,000 students and compared the relationship across different demographics and characteristics. His study showed that a preparation session before the online course significantly alleviated student's anxiety levels.
As much as we admire the researchers' efforts to study online anxiety, we would like to indicate three limitations of the current research. First, adopting instruments developed to measure anxiety in traditional educational settings may not capture the uniqueness of the online course setting. When assessing "state-specific" anxiety, the characteristics of the state must be presented saliently in the instrument to elicit respondents' true feelings toward the state. The assessment based on developed inventories for measuring anxiety could be an accurate measure as a trait-anxiety, but usually not specific to the "state" of the online course. Second, although the various types of anxiety, such as computer anxiety and internet anxiety, are essentially related to online course anxiety, online course anxiety is a specific type of anxiety. Different from casually surfing the internet or acquiring knowledge in a self-learning manner through the internet, online course anxiety is the experience of a particular population of students who are engaged in learning in courses formally offered by higher education institutions where their performance will be evaluated and bear significant consequences. Throughout our search, the "online" is rarely treated as a "state" but overlooked as an environment that has been studied in conjunction with other types of anxiety, such as test-anxiety in an online setting (Alibak et al., 2019), language anxiety in an online setting (Martin & Valdivia, 2017), and anxiety relating to online collaborative projects (Hilliard et al., 2020). If we want to shed light on online course anxiety as a factor affecting student learning in online courses in the higher educational setting, we need an instrument that characterizes the uniqueness of online courses compared to the traditional courses, and elicits students' negative feelings, such as worry, fear, and stress, when taking online courses. We propose that this need will be satisfied by the Scale of Online Course Anxiety (SOCA) we aimed to design.

The Uniqueness of Online Courses
As noted in Koerner and Dugas (2006), anxiety is best understood within the context, which in this study, is the online setting. To capture the uniqueness of the online course, it is essential to define online learning first. Although the ambiguity in terminology across online learning, distance learning, and e-learning had long been an issue, there was some common ground for researchers in the field Singh & Thurman, 2019). The metaanalysis study by Singh and Thurman (2019) provided a scope for us to select our theoretical framework. Systematically reviewing how online learning has been defined in peer-reviewed journal articles from 1988 to 2018, Singh and Thurman (2019) summarized the evolution of the definition of online learning as (a) the technology is the "most abundant and clearly defined element" (p. 295); (b) the later definitions "include interactivity as a key element of online learning" (p. 300); (c) the later definitions delve into the topic of lack of communication; (d) the time element is a possible aspect when synchronous or asynchronous are compared as attributions of online learning; and (e) the educational context element is utilized for "distinguishing between open learning environments and formal online learning ones" (p. 299).
Upon reviewing several theoretical frameworks that have been widely adopted in the field of online learning, we found: Moore and Kearsley's (2011) framework based on his theory of Transactional Distance (Moore, 1993), which focuses on the role of dialog, structure, and autonomy played in the novel educational setting (i.e., online); Keegan's (1980Keegan's ( , 2013 framework that compares traditional and distance education from the aspects such as separation, communication, technology; Community of Inquiry framework (Garrison et al., 2000) which illustrates online learning as a process of developing a community, which revolves around social presence, teaching presence, and cognitive presence. Given these options, we decided to utilize Keegan's (1980Keegan's ( , 2013 framework as a guide when designing the structure and content of SOCA in the current study. The reason being that his work "corresponds to how online education is being conceptualized in the present day" (Lee, 2017, p. 16), as well as the dimensions in his framework closely aligned with the summarization of Singh and Thurman's (2019) metaanalysis on concurrent studies in the field.
Defining distance education as "an institutionalized offering through public or private providers" (Keegan, 2013, p. 45), Keegan (1980Keegan ( , 2013 was one of the first scholars who systematically compared online education and traditional education to identify the unique characteristics of online education (i.e., a major form of distance education nowadays). Keegan (2013) identified five dimensions in which the online education environment inherently differs from the traditional one: (a) the quasi-permanent separation of teacher and learner; (b) the quasipermanent absence of learning groups; (c) the role of the educational organization; (d) the place of the technological medium (media), and (e) the provision of two-way communication. During the last two decades that witnessed the growth of technology and development of online education, Keegan's dimensions that differentiate the online education setting from the traditional education setting have been validated and strengthened by other researchers (Bernard et al., 2004;DeVaney, 2010;Lee, 2017;Picciano, 2002).
The first dimension of "quasi-permanent separation of teacher and learner" is arguably one of the essential characteristics that distinguish the online from traditional education (Keegan, 2013). The "quasi-permanent" part of the dimension indicates the separation between students and the instructor, and can vary from "nil, to voluntary, to compulsory" (Keegan, 2013, p. 45). For the convenience of further discussion, we name the dimension "separation from the instructor." The second dimension of "the quasi-permeant absence of learning group" describes the connections with peers by students or the instructor. Unlike a traditional classroom where a community of learners is physically available, such a community is only possible either by students' initiatives or the instructor's design. For our purposes, we name the dimension "separation from peers." The third dimension of "the role of the educational organization" pertains to the important role that public and private educational organizations play in accrediting scattered, individual, and private learning activities to formal and institutionalized learning. However, because SOCA focuses on personal perceptions and feelings within the higher education context, other forms of educational organization (e.g., MOOC) are beyond this study's scope. Therefore, this dimension was not included in our instrument.
The fourth dimension of "the place of the technological medium" emphasizes the vital and dominant functionality that technology serves in the design, implementation, and delivery of online courses. Video, online textbooks, online forums, online course portals, to name a few, are widely utilized in online education but are almost non-existent in traditional classrooms. Learners may see technology, a new component of the learning environment, as a threat, challenge, or advantage depending on their savviness in technology, and experience online courses differently. We name this dimension "technological challenge." The last dimension of "two-way communication" emphasizes that students "should be able to initiate dialogue and not be just the recipient of it" (Keegan, 2013, p. 46). As the different forms of dialogue between instructor and students are essential to students' learning, the ability to initiate dialogue by students, especially in the novel online environment, is essential to lessening their anxiety levels. We name this dimension "lack of two-way communication." As psychologists postulate that anxiety can be caused by being away from a familiar place into a new place where help may be unavailable (Grupe & Nitschke, 2013), we postulate that the online course anxiety is caused by the fundamental differences between the traditional classrooms with which students are familiar and the online environment that is novel to them. Keegan's four dimensions of isolation from instructor, isolation from peers, role of technology, and two-way communication, are deemed as sources of online course anxiety. Thus, we adopted these dimensions as subscales of SOCA to measure the state-anxiety of the online course.
The study was designed with the purpose of developing an instrument (i.e., SOCA) to assess the degree of online course anxiety experienced by students in a higher education environment. The researchers aimed at answering the following research questions: 1. Is the factor structure derived from Keegan's (2013) theoretical framework supported by the collected sample? 2. Is the SOCA a valid and reliable instrument for measuring online course anxiety for college students?

Methods
The instrument development followed Hinkin's (1998) framework and followed the steps of (a) Item generation; (b) Questionnaire administration; (c) Initial item reduction; (d) Confirmatory Factor Analysis; and (e) Convergent/Discriminant Validity check.

Item Generation
The items in SOCA were originated in two parts. One of the co-authors of the study taught an online statistics course for years and asked students to write essays by the end of each semester to reflect their leaning experience in the online course. Over the years, more than 100 essays were cumulated. Although the essays were not explicitly designed for developing the online anxiety scale, they were utilized as the source of items of the scale to enhance ecological validity (Brewer & Crano, 2000) as these items can be traced back to students' reflections of their learning experience right after an online course. The researchers analyzed the content of the essays to identify the themes of the qualitative data and aligned them with Keegan's (2013) four dimensions (Table 1), and the coding process followed three-step approach of open coding, axial coding, and selective coding (Strauss & Corbin, 1998). Items were created from student input, and sometimes students' original phrases were used. Then the researchers added items to enhance reliability of the subscales that did not have sufficient items. In the end, an initial pool of items for SOCA that included 35 items was created. For consistency of participants' responses to the constructs, a 5-point Likert scale with scores from 1 to 5 corresponding to the five options of "strongly disagree," "slightly disagree," "neutral," "slightly agree," and "strongly agree" was used. Another thing I would suggest is to make an appointment with the professor if you are having a hard time. That is something I did and it really did relieve me from the stress.
Peer sep [In previous online courses] we had webinars and class activities that all of us participated and we used to interact with each one of us …, which was helpful. I missed this in this online class… and working with my classmates could really ease things up.
Tech I found myself doing the assignment last minute on the day the Aplia (online learning environment) homework system was malfunctioning, which freaks me out…and my grade certainly reflected this.
Lack comm I just feel more comfortable when talking to the instructor (in a face-to-face environment).
Note. Teacher sep = the separation from instructors. Peers sep = the separation from peers. Tech = the technological challenge. Lack comm = the lack of two-way communication.

Content Validity and Equivalence
Prior to the administration of the instrument, the researchers established agreement on items for content validity. As suggested in previous studies (Gravetter & Forzano, 2018;Hardesty & Bearden, 2004), face validity was judged based on whether the items appear to be suitable for its aims. Three reviewers were invited for their feedback regarding the face validity of the instrument. The wording of a few items was changed based on the feedback provided by the reviewers to make the items easy to comprehend. Then, the Content Validity Index (CVI; Lynn, 1986;Zammanzadeh et al., 2015) was examined via I-CVI (Item-CVI) and S-CVI/UA (Scale-CVI/Ave). Given there were only three reviewers examining the items, we chose the rather conservative criterion of I-CVI over .83 (Lynn, 1986) and S-CVI/UA over .9 for the items to be included in the instrument (Lynn, 1986;Polit & Beck, 2006). Items causing low I-CVI and/or S-CVI/UA were removed or rewritten after inspection. For the equivalence, the inter-rater reliability was checked as three reviewers categorized items into each pre-defined construct, and Fleiss' Kappa was calculated with the criterion of over .8, indicating great agreement (Gwet, 2014;Sim & Wright, 2005). Lastly, the pool of 24 items that met or exceeded the aforementioned criteria (I-CVI = 1; S-CVI/UA =1; Flesiss' Kappa = .83) were distributed to prospective participants for data collection.

Instrument Administration
The administrated SOCA (Appendix A) contains four factors, including (a) the separation from the instructor (5 items); (b) the separation from peers (5 items); (c) the technological challenge (7 items), and (d) the lack of two-way communication (7 items).
The instrument was administered to college students in a higher education institution in the southwest area of the United States. The instrument was distributed to 207 students enrolled in 6 different online classrooms representing different content areas. The instrument was implemented via Qualtrics at the beginning of the semester. All the participating courses were asynchronous online courses. Acknowledging the differences between asynchronous and synchronous online courses, we decided to focus on the asynchronous online course in the current study and investigate the anxiety in synchronous online courses in future research. Realizing the experience of taking an online course might change the anxiety the participants experienced over time, we set up a valid response period to control the confounding variable. That is, any survey that was returned later than the first three weeks of the semester were excluded. In the end, a total of 170 usable responses were collected, yielding an 82% response rate. The students recruited were from two different colleges with a roughly 50/50 split. Among all the participants, 73.5% of them were female. The courses were open to both undergraduate students (n = 63) and graduate students (n = 107). The average age of the participants is 34.8 (SD = 9.9), and the average number of online courses that the participants have taken is 9.0 (SD = 7.6), which included accredited online course experience in their associate degree up to the current program. Additionally, the data neither suffered from missingness item-wise (< 1.2%) nor participant-wise (< 1.5%), with rates that were considered inconsequential given the 5% threshold suggested by Schafer (1999). We utilized Multiple Imputation (MI), which is known as an efficient way to recover the missing data (Enders, 2010). The MI was conducted through mice package (Van Buuren & Groothuis-Oudshoom, 2011) in R. Among the 24 items, responses to items 8 and 29 were reverse-coded to align with the rest of the items of the instrument so that the higher the scores, the higher the anxiety level for all items.

Data Analysis
Following the instrument administration, the researchers proceeded with statistical analyses to test the instrument's psychometrics, and the reciprocal procedures of revise and retest were incorporated throughout the analysis.
Conventionally, Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were utilized over a randomly half-split sample to explore the nature of latent constructs (i.e., in EFA) and ascertain it in CFA based on the findings in EFA (Kline, 2015). Given that the item generation of SOCA was profoundly driven by Keegan's (2013) theoretical framework, the main scope of the analysis for this study it to confirm the reliability, validity, and alignment of factor structures of the instrument. Therefore, the researchers choose only to fit a CFA model with four aforementioned factors underlining the proposed 24 items through lavaan package (Rosseel, 2012) in R.
Firstly, the item reliability was tested by examining the standardized factor loadings. According to Hair and his colleagues (1998), a standardized factor loading over .7 is considered good reliability, whereas over .5 is acceptable as the cut-off. The researchers decided to remove items with standardized factor loading lower than .5, then proceed to the higher cut-off if necessary. The deletion of items was executed in a stepwise fashion as the model fit were inspected at each step. To ensure the confirmatory nature of the model, the item deletion was limited to be less than 20% (i.e., 5) of the total number of items (Hair et al., 1998). Then, the CFA model fit was examined through a variety of fit indices such as a) comparative fit index (CFI; Bentler, 1990), Tucker-Lewis index (TLI; Tucker & Lewis, 1973), root mean square error of approximation (RMSEA; Steiger & Lind, 1980), and standardized root mean square residual (SRMR; Muthén & Muthén, 2017). Per fitting criterion, the researchers adopted Hoyle and Panter's (1995) suggestion of CIF and TLI equal to or greater than .90 as acceptable fit, as well as Browne and Cudeck's (1992) suggestion of RMSEA/SRMR equal to or less than .08 as acceptable fit. As for a more ideal fit, we also considered the higher CFI/TLI value that over .95 and lower value of RMSEA/SRMR under .06 (Hu & Bentler, 1999;Tucker & Lewis, 1973). Next, the construct validity which included the convergent validity, and the discriminant validity were also examined. The Average Variance Extracted (AVE) were calculated for each construct, as well as the total model, and a threshold of over .5 was utilized as an indication of adequate convergence (Hair et al., 1998). The factor correlations were calculated and presented, and factor correlations less than .85 were considered as having an adequate level of discrimination (Kline, 2005).

Results
After fitting the data to a 4-factor CFA model guided by Keegan's (2013) theory and removing the items with low standardized factor loading in a stepwise fashion, the final hypothesized factor structure is presented in Figure 1.

Figure 1 Hypothesized CFA Model
Item 8 and item 1 were deleted in a stepwise fashion according to their low standardized factor loadings (  For the item reliability, and standardized factor loadings were significant at .001 alpha level (Table 3). Overall, 15 standardized factor loadings were over the .7, and only two (item 8 = .56 and item 21 = .59) were under .6 yet still over the acceptable cut-off of .5. The item level reliability of SOCA was therefore achieved. The standardized loadings for "separation from instructors" (mean = .78, median = .78) and "separation from peers" (mean = .75, median = .80) indicated slightly higher reliability at the item level, while "technological challenge" (mean = .70, median = .69) and "lack of two-way communication" (mean = .73, median = .75) were slight lower. Respectively, the AVE for each factor was .62 (separation from instructors), .59 (separation from peers), .50 (technological challenge), .54 (lack of two-way communication), and .55 (overall). All the AVE values were at or over .5, which indicated that SOCA exhibits an adequate level of convergent validity. The factor correlations were presented in Table 4, and all were less than .85, as well as in the expected directions, thus providing support for the discriminant validity. In sum, the factor structure of SOCA, which stemmed from Keegan's (2013) theoretical framework, was supported by the CFA and SEM results. And psychometrically, the SOCA was found to be a valid and reliable instrument to measure students online course anxiety.

Discussion
This study has a noticeable limitation of sample size that we wish to address in future research. Although the sample size (n = 170) in this study fell within Boomsma's (1985) rule-ofthumb of a minimum sample size of 100 to 200, it violates other suggested numbers such as N:q rule of 10 observations (Kline, 2015) or 5 observations (Bentler & Chou, 1987) per parameter estimated. For this study, we proposed the factor structure by having literature-grounded and theory-driven rationale; however, in light of the sample size, we have decided only to focus on the psychometric characteristics of the SOCA and have not incorporated variables that were potentially suitable for an invariance testing.
For future research, the invariance tests could be conducted to evaluate the generalizability of SOCA in instances such as the form of the course (e.g., fully online or hybrid online), the form of the instruction types (e.g., synchronous and asynchronous), and the demographic of students (e.g., undergraduate students and graduate student). We believe the SOCA could benefit from a larger size and broader spectrum of participants in a future study, thus ensuring higher power and better generalizability of the scale. Moreover, considering the negative relationship between anxiety and performance of students in online courses (Hauser, Paul, & Bradley, 2012;Stowell, & Bennett, 2010), creating the instrument is only the first step in a line of research that could improve the quality and outcome of online education. Using SOCA, we plan to identify factors in the design, implementation, and delivery of online courses in correlational studies, which will guide us in experimental research to find solutions to reduce anxiety and enhance student learning in the relatively new online environment.
Overall, the study contributes a much-needed instrument to measure online course anxiety (SOCA), which is theoretically sound and demonstrated to be reliable and valid. Starting with a psychological postulation that state-anxiety is caused by novelty, uncertainty, and unfamiliarity (Grupe & Nitschke, 2013), we adopted Keegan's (2013) model as a framework that identifies the fundamental differences between online and traditional courses, including separation from instructor, separation from peers, technological challenge, and lack of two-way communication. We developed items for the instrument from students' input of their experiences in online courses to ensure the ecological validity, while statistical analysis on the psychometric features of the instrument, including the analyses of factor structures, reliability, and validity, provided us with satisfactory evidence of soundness. When introducing the new instrument to fellow researchers who share our concern/interest in online education, we hope the instrument will be applied and tested further in future research. We believe such an instrument is needed for researchers who are interested in online education, especially those who are interested in ameliorating the negative feelings, such as anxiety, students experience when taking online courses.

Declarations
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. The author received approval from the ethics review board of Texas Tech University, USA for this study. The author received no financial support for the research, authorship, and/or publication of this article.