Using Online Tools to Develop Higher Order Learning Among Tertiary Students

It is widely recognised that the development of higher order thinking skills is a fundamental goal of higher education. There are a variety of online tools that assist the development of student higher order thinking. In this paper, a process of scaffolding for the writing of higher-order questions enabled through peer learning activities is explored. Data collected over two years across five cohorts shows that there was an overall statistically significant improvement in the number of higher-order questions produced by students at the end of each unit. The findings reveal a viable peer teaching tool that can easily be embedded into existing programmes to develop the necessary critical thinking skills for higher education students. use

This article discusses the use of an online software platform that enables the student development of the multiple-choice questions in an initial teacher education course delivered at an online university in Australia. Specifically, the article investigates the importance of teacher scaffolding. The ongoing participation in the student-teacher relationship enabled the development of quality questions that students subsequently authored. Additionally, the importance of peer explanations associated with their questions provided continued student progress in their ability to write high quality questions. While there are investigations into the quality of student-produced questions reported in the literature, the current analysis addresses gaps in studies relating to initial teacher education literature as well as the student-teacher partnership required to develop higher-order thinking skills and production of quality questions.
There are many online tools to assist students in their online learning and examples such as self-quizzing facilitate the development of higher-order thinking skills as it has been shown that these specific skills positively correlate with academic success (Kenney, 2020). This study presents one online software platform that teacher education providers can easily use to expedite this process. The publicly available software requires students to author multiple choice questions from course content that encourages students to engage in the course content in greater depth. Multiple choice questions are then shared and discussed among peers. This approach encourages collaboration, co-constructivism of learning, and a rich engagement in the course material (Denny et al., 2011). While it has been noted that the potential challenges to multiplechoice writing are that creating good items can be confusing to the novice, this article provides an outline of the efficacy of the online platform as a useful learning tool that included a scaffold for learners to overcome this potential issue. To maximise student learning, scaffolding for students about how to take ownership of the course content by writing a good multiple-choice question, as well as higher-order questions, will be shown to be fundamentally important. It will also show how socially constructed learning to produce quality outcomes can be developed. Therefore, instead of lecturers transmitting information and content, students are engaged in the learning process to meet their learning outcomes (Rivers et al., 2017). The role of learning communities to support the development of student initial learning in writing multiple-choice questions in the online space will be discussed. Data show that online tools assisted the development of writing higher-order questions across five cohorts of students in an initial teacher education course. The findings will outline that the benefits of using online tools to encourage the process of active learning-shifting students from passive receivers to active learners-is a worthwhile addition in higher education.

Foundational Ideas about Higher Order Thinking
Higher order thinking skills are considered an essential component in our ever-changing world for our future citizens. Higher-order thinking skills are necessary to facilitate the transition of students' knowledge and skills into responsible action and to meet this challenge, the development of critical thinking is required (Choi et al., 2017). The significance of developing higher order thinking skills in students is well documented. Research has shown significant differences in average student performance when higher-order instruction is used (Caulfield-Sloan & Ruzicka, 2005;Jones, 2015) and Nevid et al., (2017) have shown that writing-to-learn assignments applying the concepts of higher-order thinking improved subsequent examination performance. The critical thinking necessary to respond to higher-order instruction is one of the main components in encouraging students to take responsibility for their decisions, think logically and problem-solve (Liu et al., 2014). There is also evidence to support the view that critical thinking skills can further assist in career development and promotion (Casner-Lotto & Barrington, 2006). Moreover, the significance of developing the skill of higher-order questioning in students is that it both provides a platform for critical thinking and provokes student interest and deeper thinking (McCollister & Sayler, 2010).
Scaffolding (Wood et al., 1976) allows for the preparation for students of higher-order questioning, as students move from a lower level of understanding to a higher level of questioning and knowledge. It involves a focus on the moment-by-moment exchanges between the teacher and students in a teaching and learning sequence, or as Parkin and Harper (2018) articulate, "scaffolding is like teaching someone to ride a bike" (p.36). In the online environment, scaffolding has been found to have a large and statistically significant effect on learning outcomes (Doo et al., 2020). Scaffolding nurtures students understanding and helps students feel more comfortable learning new material, which is likened to the lecturer modelling what is needed or providing expectations of what is required (Robinson et al., 2017).
Scaffolding is often used to transition student learning towards greater cognitive complexity, as described in Bloom's taxonomy (Bloom, 1956). The taxonomy describes how a student should move in a step-by-step process (McNeil, 2011). The revised Bloom's taxonomy pyramid (Anderson et al., 2001), adapted to reflect 21 st -century learning, moves students from the development of basic recall and memory to the understanding of knowledge and the application of that knowledge. Analysis of their learning is then able to be achieved at the next step in the hierarchy. Evaluation and finally creation, where students put together the elements of their knowledge and understanding of the material, can then be performed by moving through the more cognitively challenging phases of the taxonomy.

Figure 1
Revised Bloom's Taxonomy: Cognitive Domain (Anderson et al., 2001) cited in Douglas, Wilson, and Ennis (2012). Support for, and expectations of, student learning through the provision of a feedback rubric using this taxonomy has proven to be successful. Feedback is given in partnership and requires reciprocal engagement by the student and in this model, lecturers and students have as much responsibility as the other (Rivers, et al., 2017). Other research provides evidence that feedback promotes academic success. Giacumo et al. (2013), for example have shown that students given a writing assignment without a rubric did not perform as well in the production of higher-order thinking than did the students who were given a rubric where higher-order thinking categories were listed as prompts. To assist in the quantitative rigour of such an approach, Omar et al. (2012) had developed a classification system to facilitate the measurement of higher-order questions in exam essays using Bloom's taxonomy. However, the usefulness of Bloom's taxonomy to give feedback has also been called into question, especially concerning its limited evaluation of the criteria levels (Biggs & Collis, 2014). The Structure of Observed Learning Outcomes (SOLO) taxonomy was developed is an instrument that allows for the evaluation of the quality of a student's work retrospectively in a systematic way, thus addressing the limitations of accurate measurement in Bloom's framework (Biggs & Collis, 1982). The taxonomy is also known to support students to reflect on their thinking while providing feedback and feedforward concerning learning (Chan et al., 2002). The structure of the SOLO taxonomy is outlined below.

Figure 2
SOLO Taxonomy (Biggs & Collis, 1982) http://www.johnbiggs.com.au/academic/solo-taxonomy/ The teaching of higher-order thinking according to the taxonomies is designed to develop student learning. Moreover, the educational benefits from engaging students in the creation of question development from course content themselves have been deemed successful regarding learning outcomes . One method of engaging students in the course material has been in the production of multiple-choice question development. Multiple-choice formats, however, have long been criticised for their inability to tap into higher-order thinking because traditionally, the questions themselves have only been developed at knowledge-level thinking (Hancock, 1994). Multiple-choice questions have also been criticised for their overuse in online learning as a "traditional" assessment tool (Dumford et al, 2018) although by "traditional" the authors refer to the use of a summative approach to determining students' knowledge of the course content (Rambe, 2020). They remain, however, arguably the most popular type of assessment in education and have been reported as an activity that promotes learning and can be used in formative assessment allowing immediate feedback (Butler, 2018). Further, Butler (2018) stipulates multiple-choice questions that demand high levels of student critical thinking as vital, and research has shown that multiple-choice questions as an assessment assist in providing higher mastery and retention of course content (Mingo et al., 2018).
The synthesis level of "creation" in Bloom's taxonomy can only be assessed using multiple-choice questions, for example, by the production of new questions. Additionally, evaluation can only occur in the description of the answer by the author. The limitations of traditional multiple-choice development can be overcome by introducing platforms to assist student partnerships in learning, teaching, and assessment.

Online Tools for the Development of Higher Order Thinking
While there are many online tools to assist students in their development of higher order thinking (Kenney, 2020), this study was particularly interested in PeerWise as a tool to facilitate peer-to-peer interactions and collaboration with course material. PeerWise is an online software tool that requires students to engage in the creation, sharing, responding and discussion process with other students. It requires students to use these skills in the development of multiple-choice questions, written from course content. PeerWise was designed by Denny et al., in 2008, offering an innovative approach to teaching and learning by encouraging students to engage with the course material in a novel way. The system was specifically designed to promote higher order thinking by providing a cost-free platform that allowed students to specify an explanation of their multiple-choice answer and critically evaluate others' answers. The inclusion of the comments area allows students to give feedback about other peers' questions, ask questions for clarification and for the author to respond, which allows for the clarification and further development of the initial model answer (Galloway & Burns, 2015). This element of peer assessment and feedback takes students from being recipients of their educational process to being working partners, creating, assessing and making critical judgements. PeerWise uses a pedagogical methodology of constructive evaluation that allows students themselves to develop higher order questions that are driven by their own learning rather than teacher-led learning (Luxton-Reilly & Denny, 2010). Constructive evaluation shifts the learner from being a passive consumer of course content to participants in knowledge sharing and production.
Research conducted into the efficacy of online multiple-choice tools has produced two consistent results. The first key finding is that students are inclined to contribute more questions, answers. and give comments to peers than is asked of them as a minimum requirement, suggesting high levels of interest and engagement in the course material. Alongside these outcomes, a further finding has indicated that there is a correlation with higher summative assessment scores and students with higher levels of scores (Bates et al., 2012;Casey et al., 2014;Denny et al., 2010;Feeley & Parris, 2012).
There are, however, fewer studies that address the quality of student-generated contributions in multiple-choice online tools. In one study,  investigated the quality of student-generated questions Specifically, they examined the errors detected and subsequently corrected by peers. The definition of quality was whether the questions were an effective and efficient tool to help peers learn. Ratings were given from 0 to 5. While subjective, it was found that students write and use high-quality questions more often than low-quality questions.
Furthermore, Bates et al. (2014) found that 75% of chemistry student-generated questions were developed at a standard beyond simple recall using a revised Bloom's taxonomy (Anderson et al., 2001) to map question quality. Anderson et al. (2001) suggested a simplification of Bloom's taxonomy which groups the upper three levels into a single level. In  study, academic staff interaction was passive after introducing students to the assignment with scaffolding activities. Again, using the revised Bloom's taxonomy by Anderson et al. to map question quality, Galloway and Burns (2015) investigated the quality of physics studentgenerated content. Comparably, the findings indicated 86% of the participants provided examples of high-quality questions (defined as more than simple factual recall). Scaffolding was provided for students before the task only.
An additional outcome of Bates and Galloway's work was the evidence of a strong link between the scaffolding resources that both supported and guided students and the writing of high-quality questions and explanations. This is supported by other work such as Preus (2012), who concluded that with adequate support strategies and teaching resources, all students including those with learning difficulties could achieve improved levels on Bloom's taxonomy. This research contributed to the study in using online multiple-choice tools and the application of scaffolding students to write higher-order questions according to Bloom's taxonomy. It achieves this aim by investigating the use of ongoing scaffolding throughout the unit of teaching as well as providing feedback for improvement, rather than providing students with an understanding of scaffolding as a single event. Further, scaffolding the writing process towards higher-order questions in this way has been applied in an online context (Preus, 2012).
It becomes increasingly apparent, given the research outlined above, that the efficacy of scaffolding and feedback appears to require strong and ongoing relationships between students and their lecturers. Regular and specific feedback encourages active engagement and is required to move student learning towards becoming "partners in the learning journey through learning to ask questions and evaluate conclusions and answers" (Rivers, et al., 2017, p. 4).

Theoretical Framework
The model is underpinned by the theoretical framework of social constructivism (Vygotsky, 1978), which purports that knowledge is assembled through social interplay. The construction of meaning is developed in the interaction with others and is critical to the learning process. Discussion and social interaction enable students to arrive at a level of deeper understanding of their shared meanings of the material. While support systems for online learners have long been a concern for the online teaching space and the development of essential skills (McLoughlin & Marshall, 2000) as traditional platforms of social interaction occur face-toface (Luxton-Reilly & Denny, 2010), social learning opportunities can occur online when students come together in virtual learning environments (Richardson et al., 2017). Thus, in alignment with social constructivist theory, scaffolding, or to be more precise using Vygotsky's terminology-operating within the zone of proximal development (Obukhova & Korepanova, 2009)-is used as a mechanism to provide students with the capabilities from a teacher or other peer to help them perform a task that would not typically be accomplished independently.
The current article describes the use of student-generated content within the context of multiple-choice higher-order assessment questions where scaffolding is consistently provided for students and interaction encouraged between peers. In the online teaching course, students were invited to participate in learning communities in which they were able to share ideas, ask questions, and respond to peer comments. The communities included the participation by the lecturer. Reciprocal feedback was enabled accepting students as partners in the content to explore. Details of the method that was used for the study are described below.

Method
The current research explored the role of scaffolding to assist in the authoring higherorder questions. The research questions were: 1. How can students be encouraged to interact with the teaching material presented in an online higher education programme? 2. How can students develop higher order thinking skills in an online higher education programme? 3. How successful was the online tool used in the study in developing higher order thinking skills?

Participants
Participants were pre-service teacher education students enrolled in an online university course that used an online software tool, across four cohorts over two years. These students lived in a variety of locations around Australia as well as overseas. Students were between 21 and 52 years of age. Seventy percent of the participants were women and 30% were male. All were enrolled in a post-graduate pre-service teaching degree and 44% were enrolled as primary school pre-service teaching students and 66% were enrolled as secondary school pre-service teaching students. Ethics approval was obtained to use students' data from the university's Human Ethics Committee. Students gave consent to participate in the collection of the summary of results. For the purposes of the research focus, results were analysed across each cohort, allowing the participants to be anonymous.

Procedures
The delivery of the classroom management course consisted of eight weeks in total. Students were enrolled in communities of practice and each community received weekly resources to scaffold the writing of questions, explanations, and evaluations/comments. Bloom's taxonomy (revised) (Anderson et al., 2001) provided a framework to guide higher-order question writing while the SOLO taxonomy (Biggs & Collis, 1982 provided a framework to help assess and describe the growing complexity in question writing and understanding of the unit content. Each week, an updated scaffold was posted to include examples of questions, explanation, and critical evaluation and comments-all mapped to the frameworks. Students also participated in weekly community forum discussions where they were encouraged by the lecturer to ask and respond to questions from each other relating to the course content. The lecturer participated in the forums to confirm the accuracy of the content when required.
The multiple-choice activity formed 30% of the overall unit grade, and over the eightweek unit, and student submissions were required to be submitted every two weeks, making a total of four submissions per student. For each submission, students were required to: 1. Design two higher-order multiple-choice questions based on the related readings, resources, and materials 2. Write a detailed explanation for each of the two questions to justify the correct answer and to explain and/or justify each distractor option, using the related readings, resources, and materials to support their explanations. 3. Answer at least four multiple-choice questions created by peers. Students were encouraged to answer as many questions as possible beyond the required four to help deepen their understanding of the content and to improve their multiple-choice question writing and answering skills.

Critically evaluate two questions created by their peers.
Student evaluations and analyses informed by any relevant literature from the unit and beyond with in-text referencing were completed. The primary focus of the evaluation/analysis was concentrated on the content of the questionas indicated in the marking rubric. As a secondary focus, students were encouraged to evaluate the structure and/or quality of peers' question/s. Students were also required to support each evaluation with multiple sources (i.e., at least two references) derived from the unit materials and beyond to be eligible for full marks under the criterion. Students were given marks for the correct submission of each section. The higher the score indicated more higher-level thinking questions authored by students. A score of 30 showed students writing questions that were all higher order.
The dataset comprised six cohorts of students (N = 625), with a wide range of student numbers in each cohort ( ℎ 1 = 189, ℎ 2 = 29, ℎ 3 = 88, ℎ 4 = 181, ℎ 5 = 36, ℎ 6 = 102). This made comparisons between all cohorts to assess the consistency of performance within a given assessment task difficult, particularly for cohort 2 due to its small number of members. It would be expected that marking would be consistent within cohorts 1, 3, 4 and 6 as these contain a sufficient data sample.
An examination of the data distributions for all assessment tasks showed a ceiling effect for all tasks, with a substantial number of students achieving full marks for the task from all cohorts. For this reason, all data analysis will use non-parametric methods, using IBM SPSS Statistics V22.0. In order to determine changes between the assignment scores for each student across the cohorts, a Kruskal-Wallis test was first employed to establish if there were any significant differences in the marking between the cohorts. A Friedman's test was then conducted to identify a difference between the four assignments on the whole dataset, and a Wilcoxon Signed Ranks Test was used as a post-hoc test to identify changes in performance between successive assignments.

Results
An assessment of the marking on each individual assessment task for cohorts 1, 3, 4 and 6 was completed to determine if the marking of the tasks were consistent across these cohorts. A Kruskal-Wallis test was employed to establish if there were any significant differences in the marking between the cohorts, with the results shown in Table 1. The results in Table 1 indicate however, significant variations in the marking indicated for all assessment tasks and suggests that there is some lack of consistency in marking within tasks that might need to be addressed. Table 2 shows the mean rank comparisons reported in the Kruskal-Wallis tests for each of the assessment tasks. An examination of student data from Table 2 showed that while there appears to be some variation in the consistency of grading within the cohorts, there was an overall statistically significant improvement in student results, where an increase in higher-order questions across their respective trimester can be observed. From Week 3 onwards, high quality questions, explanations and critical evaluations and comments were evident. Questions reflected a combination of thinking levels with growing complexity in understanding and linking of evidence-based research to future practice. There appears to be substantial variation in the mean rankings in the various assessment tasks, shown in Tables 3, and indicated in an analysis of the Wilcoxon Signed ranks Test (Table 4) and summary of rank changes (Table 5). For example, cohort 3 has the highest mean rank for assignments 2, 3 and 4, suggesting that cohort 3 may have been marked more liberally. Due to the variation identified in the cohorts for each of the assignments, an analysis at the cohort level does not appear to be justifiable.

Discussion
The preservice teacher education course in classroom management began in Trimester 1, with the scaffolded teaching of higher order thinking skills for students using the platform of an online multiple-choice tool until Trimester 3 two years later, when the unit was restructured. In this article, student construction of higher-order multiple choice questions and the thinking that underpinned the authoring of those questions is examined. The purpose of comparing assessment marks for each student as they progress through the unit was to verify the usefulness of an online tool as valuable in enabling the scaffolding of higher order thinking and for students to take greater responsibility for their learning. The results, in response to the research questions, indicated that the online software tool, assisted students to interact with course material and enable them to produce quality multi-choice questions. Secondly, the results also indicated that student cohorts produced a larger number of higher order questions across the unit. From these results, it can also be concluded that online tools to facilitate the development of higher order thinking are successful, although scaffolding was also provided to assist students in the development of these skills.
The multiple-choice tool is a web-based system that supports student learning in a variety of ways in part because students are asked to actively engage in the course content . Traditional, transmission-style methods of online teaching practices are often limited to recorded lectures and the reading and revision of texts, causing higher education pedagogy to come under scrutiny (Gilboy et al., 2015). Scaffolding relies on learner-centred instruction and encourages students to take responsibility for engaging in the lesson, challenge their thinking, and enable problem-solving (Bergmann & Sams, 2012). Scaffolding also aligned with the framework of social constructivism, used as a pedagogical approach in the unit. For these reasons, an online multiple-choice tool, alongside learning communities of practice, is considered an excellent online approach to learning.

Limitations and Future Research
As previously noted, there were variations in the consistency of grading within the cohorts. This discrepancy can be addressed by introducing robust moderation practices that provide quality assurance across assessments (Crimmins et al., 2016). The inconsistencies were however somewhat surprising, given the use of the SOLO taxonomy to provide a rubric to guide the marking, proposed as a solution to a similar problem by Hardy et al. (2014). A further limitation of the study was that while conclusive outcomes were evident in the quantitative analysis, it would be valuable to investigate student perceptions of the specific nature of the benefits of peer interactions in order to provide for a richer assessment of the usefulness of the approach described in this paper. Understanding the nuances of what best works in the learning process would also benefit future research directions.
Recommendations for future teaching practices include the continued application of online tools, the addition of ongoing scaffolding, and the support of peer community forums to encourage the development of higher order thinking skills for pre-service teacher education students. Replication of the study comparing the quantity and quality of various applications of scaffolding and across a comparison group would be useful to provide a more comprehensive understanding of the role that teachers play in the online learning platform.

Conclusion
This study provides evidence to support the use of software tools that can serve to assist in the planning and delivery of online teaching. Tools such as the platform used in this study are cost-free applications that can be embedded within higher education prgrams to faciliate students' interaction with the course material. Additionally, online multiple-choice tools served as an interesting means by which to include multiple-choice quizzes into a unit. Morevover, this study showed that students did improve in their ability to produce higher order questions over the teaching period. This result provides support for such platforms to be used in higher education teaching programmes.

Declarations
The author declares no conflicts of interest. Ethics approval for the study was granted by the University of New English, Australia Institutional Review Board. The author declares no funding for this work.