Correlational Analysis of Student Visibility and Performance in Online Learning

This study examines the relationship between student visibility and learning outcomes in a graduate-level online course. Visibility in this study refers to students’ cognitive, social, and emotive presence [1, 2] in various communication settings, such as posts on the discussion board, contributions in live chats, email messages, online profiles, and inputs via any other means of communication. A visibility score is determined for each student, and the Spearman r correlational tests are used to detect any significant correlation between visibility and learning outcomes (grades). In addition, two surveys were distributed to the students at the end of the course: (a) Survey on Self-Perception on Learning Experiences provides a context for understanding student performance; and (b) Survey on Useful Aspects of Socializing Online asks students to rank the importance of eight types of online activities, such as sharing information, solving problems, and making friends. Both surveys probe into students’ perceptions and social context, which often have great impact on students’ online presence.


I. INTRODUCTION
In a study of computer conferencing in higher education, Garrison, Anderson, and Archer [1] created a conceptual framework/model that specifies crucial elements for generating a successful teaching and learning experience.Their Community of Inquiry model states that learning occurs in a community of teachers and students through the interaction of cognitive, social, and teaching presence.Synthesizing current arguments and theories on learner engagement in virtual learning environment, Wang and Kang [2] recently proposed a Cybergogy for Engaged Learning Online model, which also has three overlapping domains (cognitive, emotive, and social), each with a set of critical factors (see Figure 1).Wang and Kang argue that, in order to better engage distributed students in online courses, the critical factors in each domain need to be well attended to encourage students' cognitive, emotive, and social presence in an online learning environment.Although this model [2] still needs to be tested through systematic research, research in the past decade (e.g., [3,4]) has commonly supported the importance of social factors (interaction, social network, and learning communities) in online learning and learning outcomes.In particular, socializing is considered essential for a fun and successful learning experience in technology-mediated learning situations such as online learning and blended learning [5].

Online Learning Environment
A handful of studies have examined the relationship of socializing to learning outcomes, with differing results.Whereas Aviv, Erlich, Ravid, and Geva [6] found that social networks positively affected learning outcomes in a study on knowledge construction in asynchronous online discussions, other studies (e.g., [7,8]) found that witness learners or lurkers who refrain from visible interactivity still meet learning objectives or achieve their self-established goals.Although the study conducted by Fritsch [8] was less conclusive, participants reported that witnessing forum postings contributed more to their personal success in the seminar than actually posting.Invisibility, therefore, cannot be directly associated with either less involvement with the course or less overall achievement [7].
These conflicting arguments, along with the growing body of research addressing diversity issues, such as ethnicity, culture, and learning styles, create a dilemma for faculty designing and conducting online courses requiring online interaction.Although not intended to solve the problem, this study aims to contribute to the current discussion by addressing the relationship between students' "online visibility" (i.e., cognitive, social, and emotive presence) and learning outcomes.
Referring to the Community of Inquiry Model [1] and the Cybergogy for Engaged Learning Model [2], visibility in this study refers to students' cognitive presence (exploration, integration, resolution), social presence (interaction, community, support), and emotive presence (self-confidence, competence, emerging emotions) in a learning community.Whereas many physical cues are missing in a digital environment, new elements of learning behavior become visible.These elements of learning behavior include the open sharing of personal concerns and emotions (social and emotive), and the reconsideration of views through revising or deleting posts from the discussion board (cognitive) [9].
By contrast, some self-regulated activities-such as surfing, browsing, and searching for informationremain invisible.A few studies (e.g., [7,10]) have attempted to assess visibility in online courses and conferences, but their approaches are rather narrowly-defined and not transferable to different settings (such as a typical 16-week online courses offered at institutions of higher education).Picciano [10], for instance, assesses students' social presence by counting students' posts on the discussion board.Beaudoin [7] determined students' visibility in online conferences by the number of words they logged in at one of the two conferences: high visibility (>1000 words), low visibility (no log-on in one of the online conferences), and no visibility (no log-on in either online conference).
Building on the strength of the visibility measurement used in the Picciano [10] and Beaudoin [7] studies, this study intends to create a more expansive and generic means of assessing online visibility.Here visibility is operationally defined as participants' contribution to visible areas of the course, such as chat rooms, discussion boards, email messages, and personal profiles.Thus, students' visibility in this study was measured with the communication evidence students produced in these areas, including chat transcripts, discussion board posts, and email messages.
Known as communication artifacts, some of this evidence reveals each student's thinking process in comprehending course materials and completing assignments.Thus, the study of visibility will also indicate how thoughts affect learning.The representations of thoughts through letters and texts also have many pedagogical consequences, such as higher motivation, better sharing of resources and information, and the increased feeling of belonging to a learning community.By examining communication artifacts, this study will make initial inroads into the thinking behind the interactions that take place in online learning environments, and thus better inform instructors to guide online interaction.This study is innovative also because it approaches online participation from new perspectives-the study of "online visibility" instead of "invisible" course usage or access.

II. RESEARCH METHODS
This study uses a single case with embedded cases design [11], with the online course as the single case and the students as individual cases.This case study examines visibility and learning outcomes in a graduate-level course that teaches students how to design technology-enhanced teaching and learning materials.The class had 14 participants (8 females, 6 males) distributed in two countries and five different states in the USA.Eight of the 14 students were familiar with course setting and technology tools.These participants were teachers or trainers in K-12 schools, higher education institutions, private corporations, or government agencies.Their common interests in technology and course delivery drew them to this elective course.
We purposely chose this sample because of the similarity of students' professions and their experience with technology and online learning.Also, 12 of the 14 students are enrolled in the same certificate program and have all fulfilled the course' pre-requisite.Additionally, the grading variance in this course enables the occurrence of any correlation between visibility and learning outcomes.
The course was primarily conducted via Blackboard, which provided a variety of communication tools, such as email, discussion boards, file exchange (Digital Drop box), chat rooms, and a group conferencing system.In addition, the instructor held several webcast and chats outside Blackboard.This chat system (known as CourseWebtalk) included streaming video and audio.The chats allowed students to interact with the instructor and others through both text chat and video/audio.Students had the choice of being seen and heard online or be identified through their text responses on screen.

Figure 2. Screen Shot of Course WebTalk
Although collaboration was not required, the instructor intentionally encouraged teamwork.Unstructured collaborative learning began with students sharing their thoughts, knowledge and resources, and continued through all the stages of the course.
Student performance was assessed through three writing assignments (15% of the grade), a mid-term (25% of the grade), and a capstone final project (50% of the grade).Class participation counted towards 5% of the grade but was excluded from overall grades in the correlational analyses.Students worked individually on writing-instruction analysis and lesson plan critique, and exams that were short scenariobased essays.For the final project, students were encouraged to work collaboratively in teams of two or three to prepare a program of systematically designed, aesthetically pleasing and wholly usable instructorled, media-enhanced lessons.Thus, writing and exams assessed students' critical thinking skills and problem-solving abilities; the final project assessed their abilities in design, communication, and application of theories in practice.
courses.In addition, Blackboard generated statistics about students' course usage, including their access to the discussion board, course documents, tools area, email, and collaboration.
To help better understand students' different levels of course participation, an online survey on selfperception (see table 2) was given to students at the end of the semester.The survey elicited their perceptions of the online discussions, asked for their self-rated participation levels, perceived sense of community, and feedback concerning their satisfaction with teamwork.The goal was to gain "insiders'" perspectives on their learning experiences.Insider perspectives were not related to visibility and learning outcomes, but they could provide a context for understanding student performance in this qualitative study.
Another survey on Useful Aspects of Socializing Online (see table 3) was distributed soon after the perception survey.The survey explored how student perceptions on the usefulness of socializing online affected their visibility in online discussions.Participants ranked the following eight aspects of socializing online: talking about readings and assignments, sharing exchanging resources and information, logistical planning for projects, solving problems as a team, learning to better use the course tools, getting to know others and learning from them, providing and receiving emotional support to feel connected or share personal concerns, and making friends.
These surveys are important and valuable, because they probe deeply into students' perceptions, which often have great impact on students' observable interactivity levels [12] and thus their visibility.

B. Data Analysis
Quantitative content analysis [13] of communication artifacts was used to determine student visibility.The analysis involved coding all communication artifacts to obtain counts of visibility indicators (i.e., threaded discussion posts, turn-taking in chats, email entries, length of personal profiles, and instant messages) for purposes of statistical analysis.A systematic coding procedure was established, including defining important terms, specifying the element of analysis, and formulating coding categories.Two independent coders followed this procedure to establish inter-rater reliability (around 90%).

Defining Important Terms a. Visibility
A visibility score was determined for each participant, based on the number of posts each made to the discussion board, the number of times each spoke or wrote in live chats, the number of the email messages each sent to the instructor, and the length of personal profiles.

b. Learning Outcomes
Assignments in this course are aligned with the instructional content and objectives.To ensure grading fairness and reliability, a graduate assistant re-grades the writing assignments, the exams, and the final project, and then collaborates with the instructor to determine final grades.Findings from previous studies (e.g., [10]) lead to the hypothesis that visibility might affect learners' performance with only certain tasks, such as tasks that can benefit from group efforts, peer review, or a collaborative approach, as opposed to exam performance.Thus, in addition to the overall grades, this study also analyzed the impact of visibility on different types of assignments, including writing (15% of the grade), exams (25% of the grade), and a capstone final project (50% of the grade)-three grading categories that are closely related to course objectives.

c. Element of Analysis
Every input or entry from the students-such as turn-taking in chat rooms or posts on the discussion board or email messages-was considered one element and counted as one point towards their visibility scores.This elements approach was first used by Chappel and colleagues [14] in conducting fast coding of online learning behaviors.Elements in their study refer to learning behaviors situated in a holistic context, such as organization, facilitation, and dissemination.The reasons for counting the number of entries instead of words in each entry are twofold: (a) the length of an entry is very much dependent upon the context of the discussion; and (b) some students might be more succinct than others in expressing ideas and thoughts.This coding technique is supported by some well-established studies.For instance, Garrison and his colleagues [1, p. 102] argued that "different units of analysis might be used to focus on different phenomena within the same study." In addition, a random sampling count of chat and discussion board entries showed small variation in the length of each entry.The length of discussion board posts ranged from 30 to 40 words (M=35 words), and the length of each chat entry ranged from 20 to 30 words (M=25 words).Entries that fell within these lengths counted towards one point.Longer entries (above 70 words on the discussion board and above 50 words in chat) were noted and counted as two or three points in the visibility score.In addition, the length of students' online profiles varied a great deal (from 0 to 592 words, M=300 words, SD=102); thus the length of their profiles is used as a visibility indicator.Profiles that have fewer than 300 words counted as one point, and longer profiles counted more than one point in the total number of visibility score, based on their deviation from the mean score.Team discussion data are excluded from this count because teamwork was optional and only four of the 14 students chose to do teamwork.Because teams were required to post their online communications in the group spaces of Blackboard, each team generated a great amount of communication evidence, which no doubt increased team participants' visibility.The modality of work is therefore an external factor that should be discounted to ensure fair comparison of visibility.

Statistical Analysis
The quantitative data, visibility score, grades, and Likert Scale survey ratings were systematically entered into statistical software Analyse-It™ for statistical analysis.The following three analyses were conducted: (a) a correlational analysis of students' course usage and their grades; (b) a correlational analysis of online visibility and learning outcomes (overall grades and grades for three different types of assignments-writing, exam, and final project); (c) a descriptive analysis of students' perceptions of their online learning experiences and of socializing online.

Content Analysis of Narrative Data
To provide a social context of the students' online activities, narrative data (the open-ended comments on the survey forms) were content analyzed for themes and patterns.Again, two independent coders coded the sample narrative data independently and compared their analyses to generate themes and patterns.The inter-rater reliability of coding is about 90%.

A. Correlational Analysis of Course Usage and Grades
Course Because all participants were familiar with Blackboard prior to the course, it is assumed that very few of these access hits were for the purpose of learning how to use the tools.Instead, it is assumed that all course access was mainly for the purpose of interacting and learning course content.
Of all the course usage indicating visibility, the discussion board got the most hits (38.1%), followed by hits to communication areas (1.7%), and email (1.0%).Access to the Groups area was also high (22.9%),but this portion was excluded from analysis because only four of the 14 participants participated in teamwork.
The mean of course usage (hits or visits to all areas of the course) was 1243, SD=1374, range=5008, which formed a positively skewed distribution.By contrast, the course grades (m=86, SD=8.4,range=32.25)showed a negatively skewed distribution.At a significant level of 0.05, Spearman Correlational test results (r=0.73,n=12, df=10, 2-tailed p=0.007) indicated the highly significant positive correlation between course usage and grades.

B. Correlational Analysis of Visibility and Learning Outcomes (Grades)
The mean score of student visibility was 2361, with a large standard deviation of 1749.The Median was 1744.The Shapiro-Wilk normality test (p=0.001)showed the frequency distribution as positively skewed.The Spearman r correlational test (r=0.7,p=0.01, n=12, df=10) indicated significant correlation between students' visibility online and their overall grades.The same test (r=0.35,p= 0.2607, n=12, df=10) revealed no significant correlation between visibility and exam scores (r=0.35,p= 0.2607) and visibility and writing assignment (r=0.15,p= 0.6307).But there was some correlation between visibility and performance on the final project (r=0.55,p=0.0653).The final project was weighted at a greater percentage of the final grade than the rest of the assignments.Nine of the 14 students responded to the survey.About the necessity of attending live chats, 75% enjoyed attending the synchronous sessions, while 38% felt that they could learn as well without participating in online discussions.As to self-perceived visibility level, 63% considered themselves active in online discussions, while 50% thought they could have participated more.

Survey
All respondents felt comfortable with the manners of discussion (e.g.netiquette, respectfulness, etc.) that this class demonstrated.They also felt comfortable in expressing opinions different from the instructor's.Also, 88% felt that negotiation and debate were acceptable ways of communicating with classmates and the instructor.75% felt that their inputs were valued by their classmates.88% felt that they were part of a learning community and were willing to maintain its growth.90% did not notice any gender differences in the class discussions.
Table 3 shows participants' responses to questions about the usefulness of online socializing activities.Questions ) Talk about readings and assignments 0% (0) 46% 8%  Participants perceived logistical planning for projects (31% of all responses) the most useful aspect of online interaction, followed by sharing resources and information (23% of all responses), and enhancing proficiency with computer tools and the Internet (23%).Making friends or making contact with other students (36%) is perceived as the least useful aspect of online interaction, seconded by getting to know others and learning from them (18%).

Themes From Students' Narrative Comments a. Varied Preference for Communication Tools:
Email and threaded discussion on Blackboard are less controversial.Most students used both email and threaded discussions for communicating with peers and the instructor, but respondents differed in their opinions of chats.One respondent wrote, "I'm not sure I've felt compelled to use all the tools as I feel our class gives me the communication opportunities I need."But another student thought that chats can be futile if the number of attendants is over four.Some only used whatever they needed; however, "I think the fact that the tools are there is extremely helpful," write one student.

b. Negative Feelings about Online Communication:
Some felt that online communication does not really support human touch.Feelings of frustration with online interaction might have attributed to some students' low visibility.Some were also concerned about the social impact of online communication.As one student commented, "online activities are a good (but not great) supplement to the things listed above, but what disturbs me most is that many students come to rely too much on them.Increasingly, these students do not want to work problems out in person....they would rather hit the send button on an email where they will not have to confront others.This is a bad thing for students and society."

IV. CONCLUSION
First, the Spearman Correlational test result revealed a significant positive correlation between course usage and grades.This conforms to a common observation about student performance: those who work harder usually accomplish more.Second, the Spearman r correlational test indicated significant correlation between students' visibility online and their overall grades.The same test showed no significant correlation between visibility and exam scores, visibility and writing assignments, and some correlation between visibility and performance on the final project.These results suggest that visibility seems to affect students' performance on certain tasks that require more social presence, such as communication and collaboration with others (group cohesion).This finding is consistent with the existing knowledge base on visibility and learning outcomes (e.g., [10]).
It seems that these students were striving for more visibility.Although the minority opinions should not be discounted, it is legitimate to conclude that a great majority felt they were part of a learning community and were comfortable with its discourse styles.It is therefore reasonable to assume that sense of community and discourse styles should not have discouraged them from being interactive and visible online.
Students also found that the logistical planning for projects is the most important activity when interacting online.By contrast, making friends with others was rated the least helpful aspect, presumably either because this group of students was more task-oriented than socially oriented, or because even socially oriented students viewed the online chat rooms as an undesirable or ineffective way to socialize.It is also reasonable to assume that adult learners who have social networks of their own are less enthusiastic about building virtual social networks.In addition, adults who choose to work in groups, generally and online, may be people who already are 'visibility conscious,' externally oriented, and strong in terms of Gardner's interpersonal intelligence frames [15].
Student perceptions no doubt affect their learning behaviors and levels of participation in online activities.Social context, in addition, "greatly affects the nature of learning activities and outcomes" [1, p. 91].Therefore, although the survey results cannot be considered the cause of students' visibility, they reveal to some extent why some students were more active than others.For instance, the online discussion and activities might be of interest to some, but not to others.To increase students' visibility, online activities should be prioritized to suit students' needs, focusing more on project work, information exchange, learning computer tools and communication skills, and building social networks.
The significant correlation between visibility and students' overall grades confirmed the importance of being "visible" and social in online courses.But the nature and extent of their online socialization needs to be further investigated.Therefore, this study has paved the way for two follow-up analyses we plan to conduct: a) Using the Community of Inquiry Coding Template [1] and fast coding approach [14] to differentiate the frequencies of students' cognitive, social, and emotive presence, such as information exchange, social talk, group cohesion, and emotional expressions (emotions).This differentiation will further inform the distribution of individual's presence.
b) Conducting a social network analysis [6] of the social relations formed among the 14 students, and the effect of social relations on student performance.This analysis will probe deeper into the effect of social relations on student performance.
This study is a prototype in nature, and the size of the population is its most constraining limitation.Therefore, findings from this study will be continuously confirmed through continuous study of more online classes and degree programs.It should also be noted that this study did not take into account age, gender, native language, and participants' prior online learning experiences.Future studies will also be more expansive and will consider other factors such as gender, ethnicity, cultural difference, and individual's learning preferences.

Figure 1 .
Figure 1.Cybergogy for Engaged Learning: Increasing the Level of Presence

Table 1 . Overall Course Usage Statistics (major hits areas out of the 22 areas): excluding instructor activities
usage indicated how often students accessed the course management tool, Blackboard, and how many times they visited the 22 areas of the course-primarily announcement, discussion board, course content area (course information, course document, assignments, textbooks, external links), tools area (the portal to drop box, personal information, calendar, grades, student manual, tasks), communication areas (the portal to announcement, e-mail, collaboration, roster, group pages and discussion boards), and collaboration (the portal to lecture hall and office hours).