Designing an Online Discussion Strategy with Learning Analytics Feedback on the Level of Cognitive Presence and Student Interaction in an Online Learning Community

This study investigated the impact of using a discussion strategy with learning analytics on the level of student cognitive presence and interaction. The study used a quasi-experimental design with control and experimental groups. The experimental group applied open-ended discussion and elaborated feedback with learning analytics while the control group applied open-ended discussion and elaborated feedback without learning analytics. A mixed-method approach was used in this study. Data were collected through content analysis, social network analysis (SNA), and interviews. The results showed that the level of cognitive presence in the experimental group increased more than the control group. SNA revealed that students in the experimental group developed more cognitive learning ties with their peers during the process of developing cognitive presence. Interview data showed that students found that the discussion strategy with learning analytics made them aware of their level and quality of interaction and their role in building knowledge in an online learning community. In addition, they felt that the discussion strategy with learning analytics increased their motivation to participate in the discussion. This study provides recommendations on how students can enhance their cognitive presence and learning experience in an online learning community.

Recent years have witnessed heightened research interest in the relationship between online discussions and quality of learning. Online discussions can provide students with opportunities to build knowledge collaboratively by fostering critical discussions. Garrison, Anderson, and Archer (2001) developed the community of inquiry model (CoI) to improve students' engagement in an online learning environment. Cognitive presence is considered crucial to enhancing the depth of a discussion. Studies show that social interaction and the density of cognitive learning ties play a role in the cognitive process (Alwafi, Downey & Kinchin, 2020). Therefore, both cognitive presence and social interaction need to be considered in any assigned online discussion. Previous literature found that a teacher's use of discussion strategies such as open-ended and elaborated feedback can influence the level of cognitive presence and social interaction (Van Der Kleij et al., 2015;Lee & Recker, 2021). Prior studies also have found that incorporating learning analytics into learning activities can improve students' reflection and awareness of academic outcomes (Koh et al, 2019;Arnold, 2012). Several studies recommend incorporating learning analytics as feedback to increase students' awareness of knowledge building, cognitive presence, and learning behaviour (Kovanović, 2017). Based on a literature review, this study anticipated that a teacher's use of discussion strategies with learning analytics feedback would enhance the level of student cognitive presence and social interaction. Therefore, this study employed an experimental design to investigate the impact of using a discussion strategy with learning analytics on the level of cognitive presence and the development of cognitive learning ties in students.

Literature Review Cognitive Presence and Online Discussion
The community of inquiry model (CoI) was developed by Garrison, Anderson, and Archer (2001) in response to the increased focus on enhancing the quality of critical discussion in online learning environments. This model was used to evaluate the quality of interaction as well as to enhance online engagement to reach higher-level learning. CoI concentrated the development process of deep learning and the role of students in the process of knowledge building through different types of presence: cognitive, teaching, and social (Garrison & Anderson, 2003). Cognitive presence can be defined as "the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse" (Garrison & Arbaugh, 2007, p. 161). One of the key elements of CoI is cognitive presence, which focuses on students' engagement in the knowledge-building process through discussion and interaction in an online learning environment. Cognitive presence is categorised into four phases: 1. The triggering event phase involves contributions that question or identify the problem. 2. The exploration phase focuses on searching for and sharing information. 3. The integration phase focuses on constructing meaning from the information developed in the exploration phase. 4. The resolution phase focuses on examining and evaluating the solution or idea. (Garrison et al, 2000) Studies of online learning have found some contributory factors related to discussion design that plays a role in enhancing the level of cognitive presence in an online environment. Gašević et al. (2015) found that discussion and course design affect the level of cognitive presence. For example, discussion strategies can enhance the quality of the discussion and student participation (Lee & Recker, 2021). Ertmer et al. (2011) investigated the influence of different kinds of questions on student engagement and found that open-ended questions increased participation in an online learning environment. Another factor that may affect the level of cognitive presence is the role of teachers. An et al. (2009) found that instructor facilitation affects student participation. Also, Zhu (2018) found that teacher facilitation in an online discussion can increase the level of cognitive presence and lead to higher-order thinking. Studies have also found that teacher feedback has an influence on cognitive presence. Van Der Kleij et al. (2015) found that elaborated feedback had a positive impact on student learning and engagement. Therefore, this study suggests that using open-ended questions and elaborated feedback may contribute to a positive learning experience.

Cognitive Presence and Social Interaction
Studies about online learning argue that engagement in an online discussion should not only focus on the quality of posts and level of cognitive presence but also on the students' social structures, interactions, networking, and the distribution of students' ties in the network (Alwafi et al., 2020). Developing cognitive presence requires students to establish interactive relationships with others. Social network analysis can be used as a method to understand students' interaction and process of building knowledge in an online environment (Rienties et al., 2012). Although social presence can provide indicators about how the learner recognizes the presence of other learners in the interaction, the social network can provide insight on the structure. Network approach can provide insight about the impact of instructional strategy on students' cognitive presence in an online learning environment (Rolim et al., 2019). The main elements of SNA are node and link. The node represents social entities such as individuals or organisations while the link represents the relations between the social entities. SNA has been used to examine the dynamics of knowledge building, cognitive presence processes, and group interaction (Alwafi et al., 2020). SNA can identify active participants in online discussion and examine the density of their interactions. Studies on social interaction recommend examining the process and structure of knowledge building in an online learning environment (Shea et al., 2010;Alwafi et al., 2020).

Learning Analytics and Student Participation in an Online Learning Activity
Learning analytics can be described as gathering, analysing, and reporting data related to learners' activities in an online learning environment (Siemens & Gašević, 2012). Learning analytics can be used as feedback to enhance the student learning process (Koh et al, 2019). Studies have found that learning analytics encourage student reflection, increase understanding and recognition of the learning process, and improve academic outcomes and achievements (Arnold et al, 2012). Designing an online discussion with learning analytics feedback can create an effective learning environment that enhances the quality of the discussion. Several studies recommend incorporating learning feedback to increase students' awareness of knowledge building, cognitive presence, and learning behavior (Kovanović, 2017). However, previous studies on the use of learning analytics do not investigate the use of discussion strategy with learning analytics, or, more specifically, the use of learning analytics with elaborated feedback to enhance the quality of the online discussion and social structure of the knowledge building process.
Hence, whether the use of discussion strategies with learning analytics impacts the level of cognitive presence and social interaction in students was the question at the centre of this study. It employed an experimental design to investigate the impact of open-ended questions and elaborated feedback with learning analytics on the level of cognitive presence and the development of cognitive learning ties.

Research Hypothesis and Questions
Based on a literature review, this study expected that the use of discussion strategies with learning analytics feedback would enhance the level of student cognitive presence and social interaction. Therefore, the following research questions were formulated: RQ1: What is the impact of a discussion strategy with learning analytics feedback on the level of cognitive presence?
RQ2: What is the impact of a discussion strategy with learning analytics feedback on social interaction? RQ3: What are students' perceptions of the use of learning analytics as a method of feedback on their cognitive learning process?

Methodology Participants
In Spring 2020, the 41 participants in this study were enrolled in an online course focused on the issues related to e-learning tools as part of a master's programme in e-learning at a university in Saudi Arabia. All participants in this study were female and their average age was 29.

Research Design
This study used a pre-test/post-test control and experimental group design. This design allowed for the exploration of the differences between the open-ended discussion supported by elaborated feedback-based learning analytics and the learning environment without the learning analytics feedback.
Participants were randomly allocated to either an experimental or control group. Twentyone students were allocated to the experimental group and twenty students were assigned to the control group. This study had two iterations: one before the intervention and one after the intervention. In the first iteration, students were engaged in an online learning activity that involved open-ended discussion supported by elaborated feedback without applying feedbackbased learning analytics in both groups to measure the level of cognitive presence and pattern of interaction. In the second iteration, students in the experimental group were engaged in an online learning environment supported by feedback-based learning analytics. The open-ended and elaborated feedback focused on asking students to offer some clarification, justification, or evidence for their answer such as "What is your evidence?" and "Can you explain to us how you reached this conclusion?" In terms of learning analytics, tracked data included number of posts, word count submitted, number of students interacted with and number of reciprocal ties, and duration of participation in the online learning environment. This feedback was sent to students via weekly email.
The control group did not have learning analytics feedback. After the second iteration, students measured their cognitive presence and patterns of interaction. At the end of the experiment, the experimental group was interviewed to explore their perception of, and experience with, an online learning environment supported by feedback-based learning analytics.

Online Learning Activities
This study focused on a two-credit e-learning course that lasted for 16 weeks. The course focused on current problems in e-learning design and solutions for design implementation. Students were involved in several learning activities through the discussion forum that concentrated on solutions to current issues in e-learning environments and the effective design of e-learning environments. Study activities had no relationship to, or bearing on, student course grade.
The experimental group in the second iteration received feedback-based learning analytics every week. Feedback included information related to the number of posts, word count submitted, number of students interacted with and number of reciprocal ties, and duration of participation in the online learning environment. Instructors provided elaborated feedback with learning analytics in weekly emails.

Content Analysis of Students' Posts
To examine the level of cognitive presence in an online learning activity (RQ1), the study analysed students' posts. This study used the cognitive presence coding schema proposed by Garrison et al. (2001). The coding schema consist of four categories: Triggering, Explanation, Exploration, and Resolution. Content analysis was performed manually.
To test the reliability of content analysis, inter-rater reliability was applied. Two coders experienced in content analysis analysed the coding sample independently. The inter-rater reliability between coder 1 and 2 was 0.78, between coder 1 and 3 it was 0.80, and between coder 2 and 3 it was 0.82. The value of inter-rater reliability represents excellent agreement (Krippendorff, 1980).

Student Interaction with Others Using Social Network Analysis
SNA was applied to examine the form of social communication among students (RQ2). A social network consists of nodes (actors) and ties among actors. In this study, the interaction in an online learning community was translated into a social network by observing who replied to others posts. The social network data collected in this present study for the SNA involved all student interaction (posts) in online discussion forms. The social network data centered on the flow of interactions, in terms of sent and received posts.
In this study, both whole network analysis and ego network analysis were used. Two measures of SNA were applied to determine the level of students' interaction in developing cognitive learning ties: whole network density and ego network density (size). The whole network density measured the overall level of interaction among students in an online learning activity. The network density can be calculated as the number of all actual links divided by the number of all possible links. The ego network density was measured for each student to examine the number of actors connected by the ego network (Reinties et al., 2012).

Interview
Semi-structured interviews were conducted at the end of study to explore students' perceptions of open-ended questions supported by learning analytics feedback. Interviews were conducted with six students individually. Interview questions were designed to understand their experiences in engaging in this learning environment. Specifically, students were asked about their perceptions of the use of learning analytics as a method of feedback on their cognitive learning process. Sample questions included: "How did the feedback that you received help you in the course?"; "How did the feedback that you received affect your engagement and contribution in the discussion form?"; "How did the feedback that you received affect your social interaction with your peers?" All interviews were conducted online synchronously and took around 15-20 minutes. Thematic analysis was applied to the interview transcripts using the six steps of thematic analysis developed by Clarke and Braun (2006). These steps begin with an overview of the gathered data, followed by the coding process, creation, and revision of themes, and providing name to the themes (Clarke & Braun, 2006). Two experts studied the interview questions to assess the trustworthiness of the interview process. Also, a member check was done by sending the interview transcript to the interviewees to check the clarity and accuracy of the interview.

Statistical Analysis
Data normality was examined by visually reviewing the curve of normal distribution and by analysing the skewness and kurtosis value of the dependent variables. The data fell within acceptable limits ± 1.96 (Hair, Black, Babin & Anderson, 2010). An independent t-test was used to measure differences between the groups for cognitive presence and ego network size, and a paired sample t-test was applied to measure differences within the groups. The UCINET v6.658 and NetDraw v2.163 software tool was used to analysed social network data.

Research Question 1: Change in Cognitive Presence
In terms of the volume of student posts, as seen in Table 1, both groups made a similar level of posts in all categories in the first iteration. The highest percentage of posts were in the explanation category, followed by exploration, trigger, and resolution. Table 1 shows that there was an increment increase in all categories of cognitive posts for the experimental group in the second iteration. However, the control group only saw an increase in explanation and trigger posts. Table 2 examines changes in cognitive presence before and after the intervention for both the control and experimental groups. The results of a paired-sample t-test show a significant increase between the first and second iteration in the mean of all categories of posts in the experimental group, while in the control group there was only a significant change in the mean of explanation and trigger. In terms of group differences, the independent t-test shows that there was a significant increase in all categories of cognitive posts between the control and experimental groups in the second iteration, with the experimental group posting more (see Table  1).

Research Question 2: Change in Pattern of Interaction
The second research question examined the effect of using learning analytics as feedback on patterns of interaction in online learning environments. Table 3 shows no obvious change between the density of the whole network for the control and experimental groups in the first iteration. In the second iteration, the experimental group saw an increase in the value of the density but only a slight increase in the density of the network for the control group.  Figure 1 shows the experimental and control group networks in different iterations. It appears that the cognitive network of the experimental group became denser over time as the number of links among students increased. However, the number of links among students in the control group did not show an obvious change from the first to the second iteration. To examine the difference between the networks within the groups, the study used a permutation test called a paired sample t-test, which was appropriate for examining the whole network data. As seen in Table 3, results showed that the density of the whole network changed significantly from the first to second iteration, while the control group did not change in the value of the density over time.

Figure 1 Visualisation of the Experimental and Control Group Networks
In terms of the ego network size, results showed significant differences between the control and experimental groups in the first iteration (See Table 5). However, in the second iteration, the experimental group showed more significant growth in the mean of the network size than control group. This means that students in the experimental group engaged more with their peers in developing cognitive presence than in the control group in the second iteration.

Research Question 3: Students' Perceptions of Their Learning Environment Supported by Learning Analytics Feedback
In response to RQ3 that focused on examining students' perceptions of their learning environment through learning analytics feedback, an interview was conducted with students in the experimental group. Interview data were coded, and two main themes were developed: an increase in motivation and raising awareness of engagement.
Throughout the interview, all students claimed a positive experience when using learning analytics as feedback in the online learning environment. One of the main themes that emerged from the interview was that learning analytics feedback increased students' motivation to participate in the online discussion. Students acknowledged that the learning analytics feedback encouraged them to participate in the online discussion. For example, one student said that "the statistical data showed me my participation in the discussion and encouraged me to participate in each online activity." The second theme that emerged from the interview was raising awareness of engagement. Students found learning analytics feedback enhanced their engagement in an online learning environment. For example, one student said that "the descriptive data about the number of my posts let me think and evaluate my level of participation and reinforce me to post more." Also, students found that learning analytics increased their awareness of the number of peers they connected with directly. For example, one student said that "the numbers of my contacts in the discussion forms promote me to develop my connection and not focus on small number of peers."

Discussion
This study attempted to respond to gaps in the literature related to teachers' uses of discussion strategies with learning analytics and how they impact the level of cognitive presence and interaction in students. The first research question was open-ended and focused on examining the impact of elaboration feedback with learning analytics on the level of cognitive presence. This study found that using learning analytics-based elaboration feedback increased the level of cognitive presence. Moreover, it allowed students to engage in higher-order thinking. Students in the experimental group saw increases in the exploration and resolution categories more than the control group. Students in the experimental group also engaged in exploration more than explanation. One possible explanation for this result is that learning analytics data may make students aware of their contribution in the discussion forum and allow them to evaluate themselves. Koh, Jonathan, and Tan (2019) found that learning analytics can increase critical thinking. This implies that students may find that learning analytics feedback helps them to reflect on their participation and improve the quality of their contribution in the discussion forum.
The second research question examined the impact of learning analytics feedback on patterns of interaction. Findings indicated that the whole density of the network increased over time and the network size of students in the experimental group increased after the intervention. Qualitative data provided an explanation for this development. As shown in the interview, students found that learning analytics made them aware of their connections. Recent studies show that teachers' awareness of their networking and connections with others can play a crucial role in improving and developing their network (Van Waes et al., 2019). Verbert, Duval, Klerkx, Govaerts, and Santos (2016) argue that learning analytics can help students become aware of their learning behaviour which consequently leads them to improve it. The findings of this study suggest that learning analytics enhance students' social learning and networks.
The third research question focused on understanding students' perceptions of their learning environment supported by learning analytics feedback. Students found that the discussion strategy with learning analytics feedback made them aware of their level and quality of interaction and their role in building knowledge in an online learning community. In addition, they felt that the environment increased their motivation to participate in the discussion. This finding is consistent with studies (e.g., Wise, Zhao, & Hausknecht, 2014) on learning analytics which found that students value learning analytics as they make students aware of their progress and motivate them to participate. This implies that designing online discussions with learning analytics feedback can create interactive learning environments that maximise student engagement and motivation.

Limitations and Future Direction
This study has several limitations that need to be considered. First, the number of participants was relatively small. Future research should be replicated with a larger number of participants. Second, this study only examined students' perceptions. Future research should interview teachers to understand the benefits of using learning analytics feedback in online discussions from different perspectives. Another limitation is related to the study sample itself, since most participants were teachers and therefore may have prior experience with the discussion forum with their own students. Thus, replicating this study with non-teacher undergraduates who might not face the experience of using online discussion might provide different results. Therefore, future study can replicate this study to other populations of online learners. Finally, this study only focused on the impact of learning analytics feedback on students' cognitive presence and their cognitive learning ties. The relationship between students' network positions, centrality, and the types of cognitive presence posted were not investigated. Future research should examine the centrality of the individual network and the depth of the discussion.

Conclusion
This study examined the impact of teachers using a discussion strategy with learning analytics on the level of cognitive presence and interaction in students. The study found that the learning environment supported by learning analytics increased the level of cognitive presence in online discussions as well as the density and cognitive learning ties among students. The study also found that the discussion strategy with learning analytics increased students' awareness of their level and quality of interaction, their role in building knowledge in an online learning community, and their motivation to participate in the discussion. Findings from this research have practical implications for enhancing the design of online discussions. Learning analytics can incorporate teachers' feedback during participation in an online learning environment. The learning analytics feedback should include different kinds of learning analytics data such as level of participation, quality and type of contribution, and social network data. This information can be used to guide students' learning behaviour and make students aware of their cognitive and social learning development in an online learning environment.

Declarations
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
The author(s) assert that approval from an ethics review board (IRB) was obtained but declined to include the name of the board that reviewed study.
The authors received no financial support for the research, authorship, and/or publication of this article.