Student Preferences for Learning Resources on a Land-Based Postgraduate Online Degree Program

Creating engaging online resources is an important part of the rapidly changing discipline of e-tutoring. There is increasing use of a wide range of media for online training but only a limited number of studies assessing their effectiveness. This study involved an educator working collegiately with cohorts of online students studying a specialist land-based postgraduate degree program ( n = 79). The opinions of these mature online students, on current and potential learning resources, informed two interventions that provided novel online resources to the course. Student opinion on these new resources was captured and subjected to thematic analysis. The results identify that these students’ favored resources were online lectures, course notes, primary literature

order to develop mastery in their professional role. It is worth noting that these postgraduate students are often already the local expert. The development of internet associated technologies (IATs) has supported the improvement of such courses. Here we report on a review of the resource preferences for a group of such students. The students, drawn from six countries, are work-based and are enrolled on one of very few specialist master's level programs in the field. Although they all have sufficient English skills to access the course, individual levels of English ability and opportunities to use English are variable. For most of the students the choice of distance learning results from a lack of viable alternatives rather than a commitment to distance learning per se. Although not unique, this type of program and student cohort characteristics are in need of further, more specific research. In this paper we focus on students' perspectives on online learning resources.
For online vocational courses to be effective, ongoing review of their online learning resources is essential (Anderson, 2008). Such reviews need to engage with students' perspectives and consider the impact of a range of online learning resources in relation to the students' study behaviors (Palloff & Pratt, 1999;Means et al., 2009;Redmond et al., 2018). Here we report on a two-stage design to elicit students' perspectives on the use they make of the present resources and what other resources they would use. In stage one, four cohorts of students (n = 79) completed a questionnaire and a sample (n = 15) were interviewed to ascertain students' views on the present resources. In stage two, we designed two interventions in relation to two different submodule learning units. The first offered the same content in multiple, different online formats. The second offered different content in different online formats. Students used these resources as a normal part of their learning and were invited to comment on the usefulness of the resources in a postintervention questionnaire.
Following a review of the extant literature related to online learning, we will set out more formally our research design. We will then review the data from stage one and then stage two of the study before identifying some key implications for course leaders and learning designers for these types of programs and students.

Review of Related Literature The Experience of Studying Online
If you are part of that minority that chooses to take your degree online, what is the learning experience like? Much research has reported that online study is often seen as a poorer form of delivery of HE courses than on-campus study, both by students and tutors (e.g., Picciano, 2002;Vonderwell, 2003;Song et al., 2004;Muilenberg & Berge, 2005;Weller, 2007;Cole et al., 2014;Gillett-Swan, 2017). A key factor contributing to this perception, of online study being a 'lesser experience', is the isolation of the online learner (Selwyn et al., 2006). Another major consideration is that regular communication between student and tutor is considered critical for the success of an online course (Beaudoin, 2002;Beuschel et al., 2003;Augar et al., 2006). Previous research, particularly in the 1990s and early 2000s, highlighted that online courses can suffer from high drop-out rates when compared to their on-campus equivalents (Fisher, 2003;Palloff & Pratt, 2003;MacDonald, 2006). Withdrawals from programs are associated not only with feelings of isolation and lack of communication with tutors, but also whether the course is relevant to the learner and whether student support systems are put in place (Lee et al., 2011).
Specialist literature on e-learning seeks to address these issues of isolation, lack of communication and subsequent student drop-out. For example, several authors point to the potential for online students to have greater autonomy for their studies (e.g., Lockwood & Gooley, 2001;MacDonald, 2006;Smith, 2008) and the flexibility of asynchronous online delivery is highlighted as giving more choice and more control of their learning to the online student (Inglis et al., 2002;Gillani, 2003;Conrad & Donaldson, 2004). In addition, some authors speculate that online learning, if developed in a suitable way, can be more student-centered than typical oncampus teaching at a university (Forsyth, 2001;Richardson, 2006;Weller, 2007). Some key characteristics of an effective online student and her/his online learning community that emerge from educational literature are presented in Figure 1.  Palloff & Pratt (2003). Amendments are displayed in black text.

Creating Effective Learning Resources for Online Courses
There are a considerable number of instructional texts, aimed at tutors and course managers, on the creation of engaging online learning materials (e.g., Palloff & Pratt, 2003;Race, 2005;Bonk & Zhang, 2008;Garrison & Anderson, 2011). Unfortunately, due to the fast pace of change in software and virtual learning environment (VLE) capabilities, some sections of such texts become rapidly dated. This fast-developing method of course delivery presents a considerable number of challenges to both the provider and to the cohorts of students engaged with online learning (McVay-Lynch, 2002). For instance, the last decade has seen a move towards compatibility of online resources with mobile technology which results in students having their university courses 'in their pocket' (Sharples et al., 2006;Bell, 2008). Creating bespoke resources, ensuring there is formative assessment, and providing motivational rewards for undertaking these formative assessments is recommended (Gillani, 2003). Seale et al. (2007) see the challenge of creating new online learning resources as three-fold, they should be designed for learning, highly accessible & highly reusable. In addition, Martin and Bolliger (2018) emphasise that interactions between the learner, other learners, course resources and tutors is of critical importance for good student engagement.
There are several studies that found the use of audio/visual learning resources for online learners to be effective, especially the use of instructional video and online lectures (MacPherson & Nunes, 2004;Mitra et al., 2010;Borup et al., 2011;Carmichael et al., 2017;Crook & Schofield, 2017;Scagnoli et al., 2017). Although now a popular delivery format in online courses, moving away from a high level of reading content, Pomales-Garcia and Liu (2006) warn against providing overlong videos or lectures. Their research highlights that online learners were less likely to complete modules with resources that took longer for them to work through. For some online provision, the use of synchronous webinars has become more common. These allow students to interact whilst an online lecture is being delivered (Moore & Kearsley, 2012). It is a format which comes closest to simulating an interactive lecture hall environment. In addition to these audio/visual offerings, some researchers have assessed the impact of podcasts as online learning materials (Richardson, 2006;Salmon, 2008;Lawlor & Donnelly, 2010). Lawlor and Donnelly (2010) found that podcasts were extensively used by a proportion of postgraduate students taking an online course, making them a valuable form of differentiation.
Although there is a consensus that students gain from a planned program of high quality, well-tested online learning resources, Bonk (2001) found, through surveying online tutors, that less than 40% of sampled online courses contained the interactive elements that the tutors themselves stated would be valuable for their students. In addition, Kinash et al. (2015) state that there is a lack of empirical evidence in relation to the effectiveness of the technologies used for online teaching. They concluded, from their meta-analysis of online student experiences, that this is a key knowledge gap. This study explores aspects of this identified "knowledge gap" within the specific context of an educator providing bespoke online learning resources to students on a specialized postgraduate course. This action research study was framed around two interrelated research questions:

RQ1
Of the current online resources on a selected postgraduate course, which did the students use most frequently, and for what reasons?

RQ2
Of the newly generated online resources created for the two interventions in this study, which did the students use more frequently, and for what reasons?

Methods
Participants in this study were enrolled on a specialist online master's degree course in arboriculture and urban forestry, delivered by Myerscough College on behalf of the University of Central Lancashire (UCLan). In addition to present students, a cohort of ex-students who had just completed the qualification were invited to be involved with the interview phase of the study. Details of the student cohorts are provided in Table 1. An initial online questionnaire (created within SurveyMonkey ® and provided in Appendix 1) asked participants about their views of the learning resources they had previously had access to on the course (Table 2). Table 2 Current Learning Resources used in this Postgraduate Course.

Learning Resource
Form of resource Accessibility

Online lectures
Slideshows with accompanying audio, most with a script to view. These were created in Adobe Presenter ® , OfficeMix ® , or Powerpoint ® .
Not downloadable-could only be viewed when the student was logged into the VLE.

Academic papers
Journal papers-typically in PDF format-selected by the tutors to be the most relevant for the topic being covered in that teaching session.
Downloadable & could be viewed on a range of devices.

Knowledge review quizzes
Created within the VLE, these quizzes focused on reviewing knowledge gained from reading the academic paper(s) highlighted in each session.
Not downloadable-could only be undertaken when the student was logged into the VLE.

Tutor's viewpoints
A concise, illustrated article by the tutor on a contentious and contemporary topic-typically in PDF format.
Downloadable & could be viewed on a range of devices.

Further reading
A range of documents, mainly PDF and Word ® documents.
Downloadable & could be viewed on a range of devices.

Tutor's own papers
Some module tutors have authored their own research papers. Where these are relevant to modules, they are provided to students (typically as PDF files).
Downloadable & could be viewed on a range of devices.

External links
Links to other websites and external learning resources selected by the tutor.
Would initially have to be accessed via the VLE but then can be saved and accessed independently.

Discussion board
In-built discussion board within the VLE, where students and tutors can create, read and answer discussion threads.
Could only be accessed via the VLE.

Announcements
A messaging system used by tutors to contact the whole of an online class of students, in-built to the VLE.
Announcements are viewable within the VLE but are also sent out to student email addresses.
This initial questionnaire finished with a question to ascertain students' preferences for new learning resources that the tutor could create for them. This was a closed question of predefined options scored by participants using a Likert scale. Further, fifteen students across the four cohorts participated in an online, semi-structured interview to obtain more in-depth views on their learning experience on the course and their reasons for favoring some learning resources over others. The framework of questions asked in these interviews is provided in Appendix 2.
Data from both these processes fed into the creation of new learning resources for two distinct interventions during the delivery of the course program. The first intervention involved supplying the same content in a range of different formats (online lecture, course notes, video, streamed video, and podcast) for students to trial. The second intervention involved supplying a wider range of formats for learning materials with each new resource distinct from any other in terms of content (Table 3). The rationale behind this approach was to seek to separate the students' preferences for different media formats from the effectiveness of unique learning objects due to their form and content. These two interventions were followed up with a final questionnaire (created within SurveyHero ® and supplied in Appendix 3) sent to all students who had participated in the intervention (n = 63), which asked them to evaluate their experience of the new online resources provided via the two interventions. A timeline for the key actions in this study is provided in Figure  2. Figure 2. Timeline illustrating the four key phases of this study: an initial questionnaire, semi-structured interviews, two interventions, and a final questionnaire.

Data Collection and Analysis
This self-reporting approach to data capture was considered effective in answering our research questions. The respondents in this case were all mature learners in professional roles who had a good rapport with the lead researcher. The discussions during the interviews where consistent with the responses in the questionnaire, with previous module evaluations and reflections of the course team. Answers were also consistent with known VLE metrics of students' use of existing resources. The research itself was articulated to participants as part of the resource development for the course. Care was taken to ensure that participants were aware that there was no "correct" answer and that involvement in the study would not impact on their learning or assessment. The study gained ethical approval at UCLan.
The responses from the semi-structured interviews and open questions within the questionnaires were first coded then themed (Burton et al., 2008). The theming of responses was reiterated three times to achieve conjoining of similar themes and to identify emergent and more specific themes that were initially placed within broader themes (Tracy, 2013). Relevant insightful quotes were selected to provide supporting evidence for the thematic analysis (Galletta, 2013). The closed, Likert scale questions were analyzed utilizing nonparametric, one-way analysis of variance.

Initial questionnaire
The initial questionnaire received forty responses across the three cohorts of current students, representing a response rate of 54.8%. Figure 3 provides the outcomes when students were asked which current online resources they found most efficient and effective for their learning. A Kruskal-Wallis and post-hoc Mood's Median test identified that there were significant differences in the rating of these resources by these respondents (H = 69.21; df = 6; p < 0.001). Knowledge review quizzes scored the lowest in terms of preferences, so, without highlighting this to the participants, these quizzes were omitted from the learning resources created for the two interventions. As we note shortly, students did not mention missing these resources during the intervention.
An open question about student preferences in terms of existing resources elicited a clear preference for the online lectures (Table 4), but the creation of course notes was the highest-scoring preference for new learning materials. This latter preference is probably best explained by this course being predominantly delivered via online lectures at that time. Therefore, the addition of course notes had the potential to diversify the learning materials. In addition, some students stated a preference for more reading materials rather than for further audio/visual resources to be created. They cited that their high reading speeds made this form of learning resource efficient for their learning and that they could read away from their computer. "It gives me an introduction into the topic/subject matter and helps explain key elements." "This kind of resource is a more interesting learning material, which summarize the content of the lesson." "It cannot be found elsewhere." "Mimics a real lecture. I learn best from either physically doing something (difficult with an online course!) or verbal communication. Online lectures are as close as one can get to real life university."

Research links
Key information Specific Assignmentrelated Accessibility Up-to-date "It provides a strong basis to the study of the particular topic. Other references and searches can then be undertaken." "It can be accessed anytime, anywhere." "Very informative."

More usable
Assignmentrelated "Tailored to the specific module and provides a good overview and insight into the key elements that are being explored."

Further Reading
1 Scientific evidence "Good for knowing the latest research." Note. Coded themes are ordered such that the most frequently-occurring themes are at the top of each list.

Semi-structured interviews
The interview process gave rise to much "rich data" and only a small proportion can be reported here. Three main learning resources were mentioned as the students' most-favored resources: the online lectures (n = 13), tutor's viewpoints (n = 8), and the academic papers (n = 6).
The knowledge review quizzes were mentioned as helpful by two interviewees and no other online resources were mentioned (i.e., external links and further reading).
Key themes relating to online learning resources are provided here, with example quotes from interviewees.
• A limited use of the online discussion board: "I didn't tend to contribute, but I read all the contributions to the discussion board. It was interesting to see what the other students were thinking. Most of the student contributions were well-written and they had quite interesting perspectives. I could not often add to what was being said." • A preference for online lectures: "I mostly use online lectures, tutor viewpoints and related research. Most of all I favor online lectures as they guide you through the topic and they are easier to consume than written text, especially scientific articles." • Low usage of knowledge review quizzes: "Quizzes-I use the least. When I am time-poor, these are not essential." • Mixed views on suitable formats for future learning resources: "I already listen to podcastsusually when I am doing something mundane like the washing-up-so I can take it in. If you do sit down to study, though, you want the audio/visual-something to look at as well as to listen to. Videos could also be good, for clarifying things further."; Respondent B: "I am not used to using podcasts-they are not needed for me-I wouldn't use them. The slides and online lectures work well for me. I can see video being of some use-for example, to look at hazardous trees" (Respondent A).

Final questionnaire
The final questionnaire was completed by eighteen students on the course program. Their responses on preferred learning resources from Intervention One are provided in Table 5. In this intervention the same content was provided in several different formats.
In the first intervention, the online lectures were most-favored and received no critical comments. The podcast received the most criticism as a learning resource. Downloadable video received the most mixed reviews. It required the largest data allowance on the students' devices, but some respondents felt that it gave the best viewing performance. Some of these differences can be traced to the level of English language of the students. Students whose reading and comprehension ability was higher than their oral comprehension preferred written texts. It was also clear that in some contexts certain resources could more easily be used, surreptitiously, in "work time," and hence were preferred. Accessible Efficient Supporting "I used less than five minutes to read the new learning resource PDF. While I watch the video, although I change the speed to 2x, I still have to use fifteen minutes. Reading is always more efficient for me." "As a foreign student, understanding of English might be difficult at some point. The course notes will be a great help to understand both presentation and video." Online Lecture

10
Effective Like attending Uni Note-taking Stimulating Tutor emphasis "It is the closest imitation mode to being in the class physically." "I preferred the audio/visual presentation as this is the most like a lecture and got me in the mindset of studying."

2
Accessible Convenient "I enjoyed the ability to listen to the podcast whilst doing my dayto-day work."

(No specific positive comments)
"I think the audio/visual presentations were of equal merit. They were clear and well-structured."

7
Accessible IT compatibility Technically superior "Smooth watch experience." "My first choice would be the MP4, which had the best resolution and works on all devices."

Negative Responses Frequency Coded themes Selected student comments
Course Notes 2 Not essential "I avoided downloading the course notes. I prefer using audio/visual and making my own notes which I find easier to reference for assignments." Online Lectures

(No comments) No comments received
Podcast 6 Lesser resource No images Not effective "No pictures and no words to read." "Podcast-not funny." "Had no need to listen to them on the move/remotely." Streamed Video

2
Poor screen size Poorer format "Just preferred resources that are better for using."

3
Data uploading Not essential "Time consuming to download, used much of my internet data allowance." "Didn't use the downloadable videos -didn't see the advantage when the audio/visual presentation works fine."  A very similar pattern of feedback was received for both interventions: that online lectures were most favored (supporting the findings of MacPherson & Nunes, 2004;Mitra et al., 2010;Borup et al., 2011;Carmichael et al., 2017;Crook & Schofield, 2017;Scagnoli et al., 2017) and that the podcasts were the most criticized (Lawlor and Donnelly, 2010). Views were mixed on the use of outdoor videos to assess urban trees in the second intervention, some students wanted higher production values in terms of technical content and sound recording. Similar comments were received about the streamed videos, where lower quality was a key limitation.
No respondents commented on the absence of the knowledge review quizzes that had been consistently provided in all previous eleven teaching sessions of this module. This strongly suggests that these formative assessments were not a highly valued learning resource for these students.

Conclusions
The findings of this study should be understood in the specific context of a specialist online course on a technical topic at postgraduate level. It is considered likely that the ages, previous educational backgrounds, and English language ability of the students who participated in this study will have had a strong bearing on the results reported here.
The initial questionnaire in this study identified three current resources that were significantly more favored than the others: online lectures, academic papers, and tutor's viewpoints. These resources were conceived to be key elements of the teaching provision for this MSc course program by its tutors-so this instructional design was strongly supported by the students' views (n = 40). A continuing preference for these resources was shown after both interventions (n = 18), identifying that successful course delivery should involve a mix of audio/visual resources with ample written resources. The responses received provided a clear answer to our two research questions. These postgraduate students appreciated the mix of learning resource types made available to them but showed strong preferences toward the use of online lectures, reading primary literature and having bespoke course-related notes or articles.
In terms of creating an effective online learning community (Figure 1), this study elicited responses in a number of key areas that may be relevant to other practitioners. At this postgraduate level, critical thinking is a key expectation of students' work. Students valued the "Tutor's viewpoints" in the course materials because this learning resource always provided a critical stance on key topics from which students could develop their own views and opinions. Provided in written form (PDF files), it was the critical content that the students valued; there were no comments on this being a less valuable learning object because of its medium. A number of responses on a range of learning resources provided, showed there was a clear focus on relevant content over medium. Students valued the flexibility of their studies but only a few expressed an interest in the use of the podcasts. Those that did emphasized that they "freed them from the screen." The majority of these students, however, did not like this medium. We would recommend that it should only be provided as a means of minor differentiation, providing the same content as, say an online lecture or course notes in a format that suits only a minority.
Other suggested key attributes of an effective online course, in terms of its learning materials, are that they should be learner-focused, interactive and collaborative (Figure 1). The most interactive elements of the current course were considered by the tutor to be the review quizzes and the discussion board but these were not favored by the students. They often stated a strong, individualistic focus on gaining new knowledge for themselves rather than on interaction or collaboration. The quizzes in their current format were clearly ineffective learning resources. They were both rated low on the initial questionnaire and were not missed by students when they were omitted from the second intervention. This may be because they were provided as knowledge reviews on specific reading material. We are intending to trial alternative interactive quizzes with different foci to see if it was their original specificity that was off-putting to students. Likewise, adjusting the "rules" of the discussion board may lead to a better level of interaction. For example, we are considering allowing anonymous postings or pseudonyms for students so they feel less daunted about using this collaborative tool or by making engagement with the discussion board compulsory (Malkin et al., 2018).
An online course designer or tutor should act to empower their students by creating flexible, interactive, attractive, and content-rich learning resources that leads to stronger engagement by the students on the course (Redmond et al., 2018). This study has reported on one: an iterative cycle of learning object creation, appraisal, and user feedback. The knowledge acquired has provided a more nuanced understanding of the ways in which individuals, on this program, value and utilize the resources made available to them. Our findings have resonated with tutors delivering other online postgraduate programs to mature, work-based learners. The broad learning from this work is twofold. Firstly, that care is needed in transferring general research in online learning to specialist, atypical, groups of learners. Secondly, that an iterative cycle of reviewing resources brings educational and financial benefits to tutors delivering online courses. As a result of this work time and effort has been more effectively directed towards the generation of appropriate and engaging online learning resources.