Moving Hands-On Mechanical Engineering Experiences Online: Course Redesigns and Student Perspectives

Hands-on lab experiences are essential for enabling students to be successful engineers, especially those who identify as kinesthetic learners. This case study describes how a Mechanical Engineering Practice course sequence was redesigned during the COVID-19 emergency transition to remote learning and examines how students responded to these changes. The remote course included videos of Graduate Teaching Assistants conducting data acquisition phases of the practice session to replace hands-on experiments. To understand student perspectives and performance, researchers reviewed approximately 400 reflective essays from Spring 2020, and compared assignment submissions between Fall 2019 and Spring 2020. Results suggest that some students perceived the loss of hands-on activities as detrimental to their learning and it was not comparable to face-to-face counterparts. Furthermore, students felt forced to develop self-directed learning skills. However, in contrast to student comments in reflective essays, comparisons of assignment submissions suggested that students in Spring 2020 did not receive lower grades or have a reduced demonstration of conceptual knowledge obtained in the course.


Moving Hands-On Mechanical Engineering Experiences Online: Course Redesigns and Student Perspectives
A hands-on, engaging, lab-based curriculum provides a solid foundation of application for theoretical concepts and is critical for future engineers to be successful (Basey et al., 2008;Clemons, 2004;Hofstein & Lunetta, 2004;Lucas, 2003;Pusca et al., 2017;Sprenger, 2006). Although online versions of lab-based classes exist, they are not as widespread as in-person labs in engineering programs due to the unique challenges facing online engineering courses, including high quality remote hands-on activities (Bourne et al., 2005;Junaidu, 2008). Past publications have explored online delivery of labs, including remote or simulation activities, compared to face to face (F2F) delivery. However, no consensus exists as to the most effective method of content delivery (Brinson, 2015;Ma & Nickerson, 2006). The literature does show, however, that remote or virtual labs in which students can interface with experimental hardware are equal in merit to hands-on labs, whereas simulations focusing on simulated data and computer-based models are inferior in students' perception, likely due to a missing link between theory and application (Corter et al., 2007;Lindsay & Good, 2005;Ma & Nickerson, 2006;Sauter et al., 2013;Scanlon et al., 2004;Triona & Klahr, 2003). The challenge, however, is in delivering material appropriately and ensuring that students are learning and retaining content.
Online classes are typically developed over months or years to promote the best delivery of content in a virtual environment. The onset of the COVID-19 pandemic in March 2020 forced universities to transition their content delivery from F2F to virtual in a matter of days. Though challenging, this situation provides opportunities to learn new strategies for delivering courses (Crawford et al., 2020;Manthalkar et al., 2020). This applies, in particular, to hands-on, lab-based courses typically reserved for F2F instruction, i.e., the style of instruction of most undergraduate engineering programs.

Context of the Study
This study takes place in a Mechanical Engineering-Engineering Mechanics (MEEM) Department, which is part of a U.S. public Midwestern institution (approximately 7,000 students) with high research activity offering primarily F2F instruction. This case study focuses on four practice-based courses entitled Mechanical Engineering Practice (MEP) 1 -4. MEP 1, 2, and 4 are hardware-based, whereas MEP 3 is simulation-based and is in a flipped-classroom approach. Since the focus of this research was hands-on lab courses using physical hardware, MEP 3 was not considered. Typically taken by students during their second and third years, the enrollment in these required courses ranges between 60 and 200 students per course, depending on the semester. The premise of these courses is to provide students practical exposure to mechanical engineering content via hands-on projects to develop and improve their critical thinking, communication, teamwork, and application skills while solidifying their foundational academic knowledge and developing a more intuitive understanding of the material (Barr, 2017;Miller et al., 2014;van Susante et al., 2016). These courses are challenging due to their practical nature, highlighting actual engineering work without a textbook-style correct answer, and are critical for preparing successful engineers. See Table 1 for course descriptions and details. Three-credit, third-year MEEM students, hardware-based, one project focused on characterizing dynamics and vibrational characteristics of bicopter including PID control (vibration analysis, impact testing, modal analysis, operating deflection shapes, acoustics and sound, Simulink modeling, Kalman filtering, flow momentum & energy use, optimization, thrust and flowfield measurements)

Switch to Remote Learning
The university rapidly transitioned (5 days) in week 9 of the 14-week semester to fully remote learning. The most significant challenge in this transition for the MEP sequence was labs' hands-on nature as a critical learning platform. Due to instructor preference and limitations, changes were minimal to MEP 1 except for select videos produced by GTAs on labs and instructors producing videos of lecture content. In MEP 2 and 4, changes were more substantial. Practice sessions were converted to asynchronous videos of GTAs conducting and explaining lab activities. Videos included a webcam stream observing hardware behavior and screen sharing of the LabVIEW program used for control and data acquisition, along with audio GTA discussion and explanations. See Figure 1 for a screen capture from an example video for MEP 4, involving controlling a bicopter trajectory. Students completed a quiz on the video content, implemented either as a one-time quiz or a mastery quiz. Upon completing the quiz, students downloaded the necessary data to complete the assignments. MEP 2 practice sessions were still held remotely in Zoom, and attendance was expected but not enforced. The courses also used virtual office hours (GTA and instructor), along with Canvas (Learning Management System) features including discussion boards, announcements, hints and common sources of error outlined in pages, and modules with sequential requirements. Assessments, including group and individual assignments, were mostly unchanged during the transition to remote instruction, as was the prevalence of group work and team collaboration on assignments, though now conducted using remote tools.

Screen Capture of MEP 4 Lab Video.
Note: Left shows webcam stream of bicopter in test cage, right shows LabVIEW program interface being used to fly the bicopter and acquire data. This view represents what students would physically see during in-person labs.

Methods
At the end of each MEP course students put together a portfolio consisting of 3-4 assignments and a culminating reflective essay (1-2 pages). Questions typically ask students to reflect on what was learned from the assignments, incorporation of feedback, and other aspects such as their perceptions of their communication or teamwork skills. However, the COVID-19 pandemic led to new questions (Table 2) asking students to reflect on how the shift to virtual instruction impacted their learning during the spring 2020 semester. These answers were then analyzed using a qualitative case study approach (Case & Light, 2011;Koro-Ljungberg, 2008) and content analysis, exploring the language used by the students to describe their experiences (Hsieh & Shannon, 2005;Neuendorf, 2017), to answer the question of how the rapid transition to remote learning challenged students in the hands-on environment and impacted their conceptual understanding of the material relative to a F2F environment. Student responses revealed common themes, including personal challenges, lessons learned, resource limitations, future considerations, and hands-on limitations in online learning environments, the last being the focus of this current paper. No statistical analysis was undertaken on the comments. Instead, a qualitative approach using content, also referred to as thematic, analysis was undertaken to analyze essay responses to achieve a deeper understanding than can be gained by merely counting responses (Boyatzis, 1998;Creswell & Creswell, 2017;Merriam, 1998;Weiss, 1995). Applying rigorous thematic analysis to qualitative student reflective portfolios ensures the results are more acceptable as engineering educators rely on deductive and empirical research methods (Baillie & Douglas, 2014). Table 2.

Original Reflective Essay Questions
Added Questions due to COVID Transition What assignments did you choose to include in your portfolio and why? Explain why you selected the two assignments of your choosing to include in your portfolio. Consider what you learned from these assignments, what steps you took to retain that learning for future application, and how you incorporated your instructor/GTA's feedback to improve the current work.
In what ways has the shift to online-only instruction impacted your learning?
Which lesson or assignment in this course has been the hardest for you so far? What steps did you take to help you master the material presented? What courses outside ME Practice helped you understand the concepts in this class?
What has been your biggest challenge these last seven weeks?
Removed Question in Spring 2020 (Note: This question is different for each MEP course; the example presented is from MEP 2) How would you rate your level of effort in this course compared to other courses you took this semester? Consider how many hours per week you spent on the course compared to other courses (the College of Engineering recommendation is the allocate two to three hours outside of class for each credit hour).
How can the university (faculty, staff, GTAs, community) help you continue to progress toward your educational and career goals when you return to campus in the fall?
In addition to focusing on student reflection relevant to hands-on limitations in the remote environment, students' conceptual understanding of the material between the online and the F2F environment was evaluated. The authors informally solicited feedback from GTA's by asking them to compare performance on assignments over weeks 9-14 in fall 2019 to the same assignments in spring 2020. GTAs were also asked for their perceptions on changes in student performance and interactions.

Results and Discussion Portfolio Comments Review
Portfolio comments were synergized using selected quotes from students. Two common themes arose, the first being student perception that online activities and learning were not comparable to F2F and the second, that students, perhaps unintentionally, were forced to develop self-directed learning skills.
Regarding the first and most prominent theme from the students' reflective essays, students felt that the online activities and learning were not comparable to F2F. For example, students commented on the challenge of using pre-recorded videos instructing them on how to complete lab activities. Although these may have been the only option due to the abrupt transition, videos are not substitutes for hands-on engagement and interaction with hardware. One student commented, "it is much harder to remember the specific details when watching a recording of a lab rather than doing the lab in person, as more of a kinesthetic learner." Students used language like "impossible," "missing out," or "not the same" when referencing online relative to F2F learning. Others commented that their learning was more challenging or compromised due to this experience, such as "there is simply no way to realize the educational value of hands-on labs in an online setting." Another student summed it up this way: "What keeps me engaged sometimes is the enthusiasm of the professors, social interactions with friends relating to the classes, and even just having a separate learning environment other than my bedroom." However, several students stated that it was "helpful being able to watch and rewatch the lab session while completing the desired report," illustrating that these videos do have supplemental value in educating the students, though not as significant as the in-person experience. These comments indicate that students need hands-on interaction and engagement to learn the content.
The second key theme was that students were forced to develop self-directed learning skills, which are imperative for success in their academic career and beyond. Regardless of the challenges faced, some positives came out of the experience, including comments about being forced to learn and improve communication, time management, motivation, engineering professionalism, and accountability. In particular, one student commented, "engineers have to adapt and overcome. This situation is no different. We can adapt to the new learning style and move forward," recognizing additional learning opportunities that will enhance their skill set for their future career. Furthermore, students also commented on being taught to be self-dependent and learn more on their own, "study smarter," and make "leaps in my understanding of engineering design thinking," which will help prepare them for their future endeavors.

Differences in Conceptual Understanding of Material -F2F (Fall 2019) versus Remote (Spring 2020)
Students consistently commented that they felt they learned less, that online learning is inferior to F2F and could never be comparable, and that their retention of the material will suffer for future courses. This led the authors to compare Fall 2019 to Spring 2020 performance in the MEP courses to understand reduction in knowledge due to the remote transition. In MEP 1, the GTA noted that students in the online environment had their learning and understanding detrimentally impacted, as evidenced by their final group reports where they have the options of which project to present. In past years, the bridge truss project was a popular choice with 31% of teams presenting on it; however, no student groups in the spring presented on that project, which was completed during the online portion of the course. It is hypothesized that students were not as confident in the material and had a reduced understanding of the project, and therefore chose a project where they had performed the work in the lab. Additionally, most of the group reports for MEP 1 occurred during the remote learning experience, and therefore, students struggled to work together efficiently as a team and produce a quality report demonstrating their understanding of the content. Of the 7 assignments submitted during the course's online portion, student averages on 3 of these assignments were lower, by between 6 to 16%.
Conversely, in MEP 2, the GTA noted that they did not see any significant change in understanding of material between the two semester offerings. Student grades semester to semester were consistent in overall distribution, with similar averages and errors on group (lab-based) assignments, all ranging between 93 -100%, over the assignments considered during the remote education period. However, the GTA commented that discussion was more limited in assignments from the spring remote offering than when students physically performed the practice sessions. Students' explanations relied on comments from practice session videos instead of thinking critically and hypothesizing about what happens and why.
In MEP 4, a random sampling of 20 student reports each semester yielded consistent trends regarding mistakes and frequency. An average of 30% of students exhibited errors in data analysis or conceptual explanations, consistent between remote and F2F classes. Although students reflected that their comprehension and learning had suffered, that perception was not substantiated by their work product. Additionally, students were provided with correct data in Spring 2020, meaning fewer opportunities existed for error. For example, students were provided with data including the setpoint PID controller gains to enable the bicopter to fly a desired trajectory in the test cage (See Figure 1). This ensured that students knew their data was correct in the virtual environment, however, in the F2F environment, in lab, students experiment with PID gains until they achieve the desired trajectory, with this experimentation and failure providing opportunities for additional learning.
The trend that in earlier MEP's (MEP 1) conceptual understanding was hindered more than in later MEP's lends evidence to students' increased maturity and skills as they progress through the curriculum. Furthermore, no significant changes were implemented in transitioning MEP 1 to a virtual environment due to instructor preference. This lack of new tools and techniques to enable students to learn the hands-on material in a new environment may have hindered their knowledge acquisition.

Lessons Learned & Best Practices
The rapid transition to remote learning due to the COVID-19 pandemic was a large, unprecedented, and unexpected challenge. This was particularly difficult for the hands-on classes as part of the MEP sequence, lending to extreme measures to deliver these courses online. However, several potential solutions exist to build hands-on skills in online learning environments. Many of these ideas were proposed in students' reflective essays, particularly in MEP 4. Along with videos, example recommendations include the following: • Having GTAs perform the lab live during lab sections to provide a time for immediate feedback and allow students to view real-time the changing parameters and impacts on system behavior; • Constructing models or simulations of the physical hardware; • Interacting (remotely) with the hardware, including using strategically placed webcams, to understand, "if I change this, then this happens," providing instant reinforcement and understanding of the physical system behavior, limited to systems that do not require human interaction; or • Use of at-home lab-based kits (Deboer et al. 2017) or "an adapted lab kit that could be used to complete modified lab assignments at home." In addition to replicating the hands-on experience in a virtual environment, other aspects of learning must be considered, including interaction and peer-to-peer learning to provide a "social community" to enhance learning experiences and knowledge retention (Terkowsky et al., 2013). This strategy is currently implemented in F2F MEP courses by providing group work and collaborative teamwork, both during practice sessions and outside in regards to data analysis and deliverable development. This continued into remote learning; however, better attention and instruction must be provided by GTAs' and instructors to teach students the best means of online collaboration using the technology tools available. These critical lessons learned regarding methodologies to enhance remote engagement, including synchronous participation and remote control of the laboratory equipment via Remote Desktop and Discord servers, will be implemented in the Fall 2020 semester. Student perceptions from the same reflective essay under this planned implementation can then be characterized for additional insight.