Functionality, Impact, and Satisfaction of a Web-Based and Mobile Application Support Program for Students with Autism Spectrum Disorder

This paper evaluates the satisfaction and impact of a self-directed, web-based or mobile app, student support program designed to facilitate self-determination of secondary or postsecondary students with Autism Spectrum Disorder (ASD) in the United States. Five professionals and four successful college graduates with ASD evaluated the program and completed an intervention satisfaction rating and an ePlatform performance questionnaire. Program content and ePlatform function was modified based on reviewers’ feedback. Following modification, the program was piloted with three high school seniors and one freshman at a two-year college with ASD. Results pointed toward strengths and limitations of the program protocol and future directions.

This paper contributes to current literature by (a) evaluating program satisfaction of usability, content, presentation, suitability, and exercises; and (b) examining the impact of the program with potential or newly enrolled post-secondary students with ASD. The program was designed as a supplemental program to be offered within the spectrum of services provided to students. As noted by Shmulsky et al. (2015) transitional programs and coordination of other student support services may be critical to retention and success. If this program proves to have promise, it may offset potential challenges and provide direction for individualized, specialized college campus supports. As noted by Betts et al. (2013) the internet has transformed our ability to increase access to higher education, but that does not equate to accessibility supports needed for success in higher education.

Content Development
The use of evidence-based content is the foundation of self-directed, digital interventions (Ritterband et al., 2009). As such, content sought to increase the self-determination of students with ASD via three evidence-based units. Unit 1 introduces career/major decision-making, Unit 2 discusses the path to graduation, and Unit 3 describes social navigation of the college campus (See Appendix A). The content is framed within a self-determination model because self-determination is positively related to academic performance, college adaptation, and self-advocacy (Field et al., 2003;White et al., 2016). One study points to the potential benefit of transition programs with a focus on self-determination. Kelly (2008) evaluated the effects of a goal planning meeting intervention on self-determination of ten high-school graduates, some of who had college experience. Intervention steps included identifying goals, developing a multi-media presentation and leading an individual education plan (IEP) planning meeting. Results indicated an increase in some self-determined behaviors for some participants. However, satisfaction of participants was only mildly positive. Hoffman and Field's (2006) model of self-determination inspired the content. Qualitative studies designed to understand the experiences of post-secondary students with ASD in relation to self-determined behavior informed program development as well (Bellon-Harn et al., 2020;Ward & Webster, 2018). Each unit frames content to increase the student's ability to know themselves and their environment, to facilitate planning and action, and to evaluate plans and actions. Text, videos, and reflective activities are included in each unit. Videos provide models of college students completing activities related to text content and reflective activities since video-modeling holds promise for its use within interventions (Ke et al., 2018). Following a video model, students are encouraged to reflect on their knowledge, attitudes, and behavior. Metaphor or other figurative language is not included to facilitate understandability and actionability (Kalandadze et al., 2018).

Self-Directed Approach
Key elements contributing to effective digital programs include a self-directed approach with guided support (Baumeister et al., 2014;Morrison et al., 2012). Self-directed features are linked to positive outcomes of digital interventions with individuals with moderate to severe intellectual disability and/or ASD (Ayres et al., 2013). Knowles et al. (2015) describes components of adult learning that may be critical in a self-directed digital learning environment for young adults. Such principles are frequently used to guide online learning (Allen, 2016). First, adults need to know why they need to learn something. An overview describing how the information and activities will support college success clarified the program's purpose. Second, adults need to maintain responsibility for their learning. The content and app features were designed to promote independence and used guided support (e.g., email, chat function) for students to obtain additional information as needed. Such communicative functions may facilitate the impact of digital interventions (Webb et al., 2010). Third, adults want to learn meaningful content that applies to real-life situations. As such, content included real-world examples, including extensive video modeling, and self-directed learning activities that are found to promote adult learning outcomes. Finally, adult learners are responsive to internal motivators and perceived competence, which are intertwined (Elliot & Dweck, 2013). Digital interventions that result in high levels of internal motivation and perceived competence are likely to facilitate learning and behavioral change (Wantland et al., 2004). Content was developed to promote motivation and perceived competence through relevant self-directed activities, encouragement and praise for completion of learning paths (LP), freedom of access to information, and systematic planning of information.

Study Purpose
To determine if such a program can mitigate the potential negative impact of current and looming accessibility issues with an intended audience, the content and delivery mode must be refined through an iterative research process (Amiel & Reeves, 2008). The development of the program followed a continuous improvement cycle to expose flaws and improve accordingly. Initial revisions of the content and delivery system were addressed with the ePlatform developers and the authors. The success of digital interventions may be limited by lack of attention during the developmental process. Further refinement requires an examination of the degree to which potential consumer groups are satisfied with the content and delivery mode and perceive the content as valuable and useful (Fleuren et al., 2014;van Gemert-Pijnen et al., 2011). This is a critical step since limited uptake from the end-user is often associated with digital programs (Eysenbach, 2005). A relatively small number of users can provide critical data on the usability and content that inform revisions to a program. Research indicates that five users are sufficient (Nielson, 2012). However, other research argues increasing to ten users improves the confidence of the results (Alroobaea & Mayhew, 2014;Borsci et al., 2013). Evaluations of education programs report between five and twenty testers (Navarrete & Luján-Mora;Nguyen et al., 2018).
In an effort to understand the satisfaction, value, and usefulness of potential consumers, experts with established backgrounds in college student support and disability services and individuals with ASD who had either successfully earned a college degree or were in upper division coursework evaluated the program. In an effort to determine the potential impact of the program, potential or newly enrolled students with ASD completed the program.

Aim 1
To address the first aim, two groups were involved in the evaluation of the program (IRB-FY18-95). Group 1 included five experts with established backgrounds in college student support and disability services. Group 2 included four individuals with ASD who had either successfully earned a college degree or were in upper division coursework successfully working toward a college degree. Reviewers were invited to review the program by the first author. Reviewers progressed through the program (see Protocol and Process Flow below) in the same way potential or enrolled students with ASD navigated the program. Additionally, reviewers completed an ePlatform performance questionnaire to describe strengths and weakness regarding their progression through the digital components.

Aim 2
In order to address the second aim, an exploratory multiple case study design was used. This design allows for the exploration and understanding of complex issues in real life contexts (Yin, 2003). Further, an exploratory multiple case study design can provide useful information about the individual response to the program since the approach is not a controlled intervention context. Finally, since there is no data using this or other programs of this nature, this design can provide critical information regarding whether or not the program warrants further experimental trial beyond this pilot study. Use of quantitative and qualitative data, as in this multiple case study, can provide an opportunity to explain both within-program responses and potential outcomes (Tellis, 1997).
Students were recruited through social media (i.e., Facebook and Twitter) and the Disability Resource Center on the campus of the university of the first author (IRB-FY18-95). Inclusion criteria included: (a) aged 18 years and over; (b) high school graduate; (c) accepted to and/or enrolled in a two-or four-year college as a freshman or sophomore; (d) the ability to read and type in English; (e) no barriers to using a computer (e.g., no significant fine motor control or visual problems); (e) internet and email access and the ability to use it; (g) commitment to completing the program; (h) self-report of ASD; and (i) a threshold score of 6 or more on the Autism Quotient-10 (AQ-10). One student was recruited who met the criteria. Four students contacted the first author; however, they were not eligible or decided not to participate. Participants were also recruited through counselors at local high-schools (IRB-FY19-146) using the same inclusion criteria except that students needed to be high school seniors accepted to and/or in the process of applying to a two-or four-year college. Five graduating seniors diagnosed with ASD were recruited. In sum, six students were enrolled. Of the six, four students completed the program.

Protocol and Process Flow
First, potential students were invited to register in the ePlatform and to complete an informed consent, a demographic questionnaire, and the AQ-10 ( Allison et al., 2012). If students were eligible, phone calls were made to ensure they understood the program and were committed to completing the program. Then, they were invited into the app and asked to complete the Arc Self-Determination Scale (ARC; Wehmeyer, 1995) and the Multidimensional Scale of Perceived Social Support (MSPSS; Zimet et al., 1988). Once the students completed the pre-program questionnaire, they initiated the program. At the end of each LP, students completed the valueusefulness Intrinsic Motivation Inventory (IMI; Ryan, 1982) and the Perceived Competence Scale (PCS; Williams & Deci, 1996). The value/usefulness subscale is a seven-item measure of selfregulation with respect to activities individuals experience as useful or valuable for themselves (Deci et al., 1994). The PCS is a four-item questionnaire, and is a valid instrument designed to assess constructs from self-determination theory. Following program completion, students completed the ARC, MSPSS, and an intervention satisfaction rating to examine satisfaction of program usability, content, presentation, suitability and exercises. The satisfaction rating (Beukes et al., 2016) consists of 15 five-point Likert-type scaled questions ranging from low (i.e., 1) to high (i.e., 5). Four open-ended questions are related to the best aspects of the program, time spent on each chapter, potential improvements, and suggestions for further development.

Program Delivery and Data Collection
An ePlatform was used to deliver the program, which allowed students to engage in a secure learning community and researchers to collect data. The ePlatform was hosted in a Health Information Trust Alliance (HITRUST) certified, HIPAA compliant environment that is 24x7x365 managed by ClearDATA on Amazon Web Server (AWS). Since 2014, ClearDATA Networks has successfully achieved Common Security Framework (CSF) Certified status from the HITRUST. The functionalities of the ePlatform included the following: § Study information: Invitation, information, and electronic consent form. § Learning management system: Interactive program. § Automated messaging system: Ability to send automated messages. § Chat function: Secured chat function in which students can send messages to clinicians. § Questionnaires: Function that can administer questionnaires and record responses.  Mobile view of the program.

Data analysis
Aim 1 (i.e., satisfaction of the usability, content, presentation, suitability, and exercises) was answered using descriptive statistics based on ISR responses and descriptive responses to open-ended questions. Aim 2 was answered using descriptive statistics based on ISR responses and descriptive responses to open-ended questions. Responses to within-program measures were described. Pre-and post-responses to outcome measures were described.

Aim 1
The program was highly rated and comments regarding the program content were uniformly positive. The mean scores for usability, content, suitability, presentation, and exercises ranged from an average score of 4 to 4.9 (i.e., agree to strongly agree). The lowest average score was related to whether or not the content was presented in a well-structured manner (e.g., adequate presentation of diagrams, text, pictures, and videos). The time required to complete the program was approximately six hours. Eight reviewers identified the videos as the best part of the program, but some noted video and audio functionality as an improvement area. One reviewer stated that the logical framework for tasks was the best part. Three mentioned that more direction on how to complete the tasks would have been helpful. Another suggested that some chapters included too much text. Several reviewers noted that the IMI and PCS resulted in a final score based on their responses, which was not intended to be part of either questionnaire. Reviewers reported that the design elements allow for easy navigation and appreciated the color scheme. Concerns were noted regarding the size of video and text on the mobile app.
Two reviewers used the mobile app to complete the program. Seven used their computer. No difficulty completing the intervention on either was reported. Descriptions of the ePlatform were positive and ePlatform performance was noted as efficient. No issues with enrollment (i.e., user registration, consent process, email), connectivity of ePlatform through both web and/or mobile usage, or the synchronization of study milestones (e.g., presentation of questionnaires, push notifications) were noted. However, two reviewers noted problems with the Chat function as a component of the automated messaging system. One reviewer suggested that there were many questions and recommended the inclusion of a rationale for participants. When asked to describe experiences with clicking or tapping on tabs to access questionnaires or the content chapters, reviewers did not note any issues using the questionnaire tab. However, several experienced some difficulty identifying the "Learning" tab as the start for the program. One reviewer stated that they would like an option to review their responses. Modifications were made including more direction to complete tasks added, removal of redundant, complex language, removal of embedded clauses, correction of questionnaires, ensuring videos and audio worked, and correction of the chat function.

Aim 2
Participants included three high school seniors planning on enrolling in college after graduation and one student enrolled part-time at a two-year college. All participants were White, non-Hispanic with a reported diagnosis of ASD. Sam was 19 years of age and received a score of 9 on the AD-10. Curt was 18 years of age and received a score of 6 on the AD-10. Dawn was 18 years of age and received a score of 6 on the AD-10. Ron was 19 years of age and received a score of 6 on the AD-10. Curt, Dawn, and Ron used their computer to complete the program, and Sam used the mobile app. Sam spent 2 hours, Curt spent 6 hours and 15 minutes, Dawn spent 10 hours and 15 minutes, and Ron spent 6 hours to complete the program. The student rated the program high. The mean scores for usability, content, suitability, presentation and exercises ranged from an average score of 4.5 to 4.9 (i.e., agree to strongly agree).

Pre-post ARC responses
Percentage of positive responses on the ARC was calculated for autonomy, self-regulation, psychological empowerment, and self-realization (see Figure 2). A total self-determination score was calculated as well. Sam's total score was 41 at pre-program and 47 at post-program. Overall, Sam presented with lower self-determination scores than other participants and minimal change in performance was noted. Curt's total score was 49 at pre-program and 58 at post-program. The most notable change occurred in self-regulation. Overall, Curt presented with low selfdetermination; however, positive change in performance occurred after the program. Dawn's total score was 70 at pre-and post-program. Overall, Dawn exhibited strengths at both pre-and postprogram, particularly in autonomy. Ron's total score was 73 and 71 at pre-and post-program respectively. Overall, Ron exhibited notable strengths in self-regulation, psychological empowerment, and self-realization.

Pre-post MSPSS responses
The MSPSS rates statements related to social support on Likert Scale from 1 (very strongly disagree) to 7 (very strongly agree). Sam's average score was 5.2 at pre-program and 4.6 at postprogram. Curt's average score was 4.8 at pre-program and 7.0 at post-program. Dawn's average score was 3.2 at pre-program and 6.6 at post-program. Ron's average score was 5.8 and 5.2 at preand post-program respectively.

Figure 2
Pre-post ARC scores on autonomy, self-regulation, psychological empowerment, self-realization, and total self-determination.

Within-Treatment IMI Responses
Sam rated a majority of the items on the value and usefulness subtest following Unit 1 and Unit 2 generally as "somewhat true." His ratings were higher for Unit 3 (See Table 1). He wrote that the Learning Path 1 would be useful for finding a career, important to do because it can help "learn about my interests," and the activity could help him "learn about myself." He did not respond to the fill-in-the-blank for Unit 2. His responses to Unit 3 included that the information and activity would be useful for "learning to talk to other students," important to do because it can help "improve his social skills."  Note. Scores are on Likert Scale of 1-7; 1 = not at all true; 7 = very true Q1 = I believe these activities could be of some value to me; Q2 = I would be willing to do this again because it has some value to me; Q3 = I believe doing these activities could be beneficial to me; Q4 = I think these were important activities.
Curt rated a majority of the items on the value and usefulness subtest as "true" and "very true." He responded that LP1 would be useful for understanding how to communicate with college staff, important to do because it can help him be successful in college classes, and the activity could help him understand what to do to be successful in college. He responded that Unit 2 would be useful for knowing what to ask when he has to talk to someone at college, important to do because it can help him with college and his education, and the activity could help him "make meetings at college better." Finally, he wrote that Unit 3 would be useful for "everything," important to do because it can "help me socialize," and doing the activities could help him "get a job." Dawn rated all of the items on the IMI as "true." She responded that Unit 1 would be useful for "understanding," important to do because it can "help people in my age understand colleges," and the activity could help her understand about careers and college. No responses were given for the other items.
Ron rated all of the items on the IMI as "true." He responded that Unit 1 would be useful for exploring careers, important to do because it can "give a person a view on what their interests are and what type of field that can be best for them." He responded that Unit 2 would be useful for time management, important to do because it can "give a person clear ideas on how to manage time," and the activity could help him to "learn how to manage my life in college." He responded that Unit 3 would be useful for how to interact with other students and teachers, important to do because it can "give an opportunity to make new friends and get help if you need it," and the activity could help him to interact with students and teachers in college.

Within-Treatment PCS Responses
Ratings on the PCS indicted that overall Sam rated items as "somewhat true" (See Table  2). However, he did feel able to meet the challenges of performing well in Unit 1. He did not feel that it was true that he could achieve his goals after Unit 3. Curt rated items as "somewhat true" to "very true." His lowest rating (i.e., 3) was related to achieving his goals after Unit 3. Dawn perceived herself as competent on all items after each learning path as noted by a majority of ratings as "true." Ron perceived himself as competent on all items after each learning path as noted by all of his ratings as "very true." Note. Scores are on Likert Scale of 1-7; 1 = not at all true; 7 = very true; Q1 = I feel confident in my ability to learn this material; Q2 = I am capable of learning the material in this course; Q3 = I am able to achieve my goals in this course; Q4 = I feel able to meet the challenge of performing well in this course

Discussion
Identification of effective practices to support PSS-ASD is critical. The purpose of this study was to evaluate the satisfaction and impact of a program designed to facilitate selfdetermination of secondary or post-secondary students with ASD. First, the study sought to evaluate satisfaction of the usability, content, presentation, suitability, usability, and exercise materials among expert reviewers.

Aim 1
Based on reviewers' report, the time required to complete each chapter appears to be appropriate (i.e., 6 hours), which appears to be appropriate and is consistent with time required to complete programs in other ePlatforms (e.g., Vlaescu et al., 2015). Nine of the ten reviewers identified the videos as the best part of the intervention. This is positive in light of the fact that evidence shows video-modeling is an important component for interventions with young adults with ASD. One reviewer stated that the logical framework for tasks was the best part. Overall, the mean scores for usability, content, suitability, presentation and exercises were strong, and the responses indicated that the flow of information was good and following the self-determination framework may be beneficial.
In light of the fact that that this program was designed to offset potential challenges and provide opportunities to pursue specialized supports on a college campus as needed, the chat function is a critical element. The chat function was modified to ensure functionality. More direction regarding how to use the chat function was included. Some studies have reported users identifying login issues with the ePlatform (Vlaescu et al., 2015). However, no login problems were reported about the ePlatform. Overall, usability testing and program revision helped modify the program to improve the users' experience.

Aim 2
Second, this study sought to evaluate the impact of the program with potential or newly enrolled post-secondary students with ASD. Recruitment and retention in intervention studies for individuals with ASD is a challenge (Lord et al., 2005;Raffaele et al., 2018). However, few studies describe recruitment challenges. In this study, a total of 6 participants were enrolled following comprehensive promotion of the study on social media, through the campus Disability Resource Center, and through counselors at local high schools. Of the six, one was not retained, and no explanation was provided after the first author attempted to contact the participant. As recommended in Haas et al. (2016), focus on recruiting through direct referrals and snowballing and including neurotypical PSS who would benefit from the program may be beneficial. Haas et al. (2016) further recommended that recruitment activities be tailored to appeal to specific participant categories. This study attempted to tailor the message across varied methods (e.g., social media, support groups, service providers); however, challenges were noted in recruiting a large, representative sample.
Results of this study contribute to previous work indicating the potential utility of a selfmanagement web-based or mobile app program on self-determination and goal-setting (Ayers et al., 2013). Results of the ARC indicated that the participants presented with varying levels of selfdetermination. Both Dawn and Ron presented with strengths in self-determination at pre-program, which may have contributed to their ability access and apply program information. As noted, their ratings on the IMI subtest and PCS were uniformly strong, suggesting they approached information and activities with motivation and competence. It may be that because their scores were higher scores at pre-program, the program was not sufficient to impact change in self-determination. Dawn and Ron spent 6 hours total time on the program, which is consistent with the time reported by the reviewers.
Curt demonstrated notable change on the ARC total self-determination score (i.e., pre = 41; post = 58). His scores on the IMI subtest and PCS were uniformly strong as well. Curt spent 10 hours on the program, which is longer than anticipated. He was motivated and perceived himself as competent, which may have contributed to his ability to access and apply the information. Sam presented with a comparable, albeit lower score than Curt, and no notable change was observed at post-program. Overall, his rating of the IMI and PCS were lower than all participants. Of note, he only spent two hours on the entire program. Taken together, lower perceptions of the experience as valuable and usefulness and lower perceived competence may have contributed to time spent on the program and his ability to access and apply information presented in the program. Based on this small sample, it may be that more information regarding motivation and perceived competence is needed in the prescreener to identify individuals for whom a self-managed program may be most beneficial.
Overall, the participants rated the program as high. This is consistent with satisfaction reported in other student support programs (Anderson et al., 2019), but not the transition program with a focus on self-determination (Kelly, 2008). Additionally, all participants appeared to manage the ePlatform well. One participant, Sam, experienced some difficulty registering and accessing the program after completing pre-program questionnaires. However, this was resolved. The chat function is a potential strength of digital programs and was identified as an area of revision in Aim 1 results. In Aim 2, it is notable that the participants did not use the chat function. The program was designed as a supplement to the spectrum of services offered on a college campus, in particular transition programs. In its full implementation, other direct supports may be concurrent with the program, and guided support through the program may not be necessary. On the other hand, it may be that post-secondary students with ASD approach web-based or mobile app programs in a qualitatively different manner than other populations, which is not surprising in light of the diverse, variable findings across interventions (Anderson et al., 2019;Kuder & Accardo, 2018). Effectiveness of this program may not be linked to the guided support as described in Ritterband et al. (2009).
The program was designed to facilitate college success through a self-determination framework, including socialization on the college campus. However, the nature of how social supports are offered is not established in part because students vary on whether or not they value social supports. White et al. (2016) found that parents valued social opportunities more than PSS-ASD or school personnel, and Jackson et al. (2018) reported that only 7 of 59 college students surveyed sought support for social skills. Although college students with ASD may have differing views on the role of socialization in their college experience, there is no argument that programs need components related to social navigation within the college context. As noted, Anderson et al. (2019) reported that academic difficulties frequently included group work and in-class social skills. This program embedded content related to these areas. Both Curt and Dawn demonstrated positive changes on the MSPSS. This may be linked to multiple activities orienting the student to current supports, potential supports, and self-awareness of socialization. On the other hand, Sam decreased his perceptions of social support and Ron remained unchanged. More research is needed to understand how the program impacted socialization or perceptions of socialization. As noted by Anderson et al. (2019) consideration of the wants and the needs of post-secondary students with ASD may be critical.

Limitations and Future Studies
The results of this study provided confidence and direction for further evaluation. The small, diverse sample does not allow for any specific conclusions to be drawn. Larger scale experimental research with better descriptions of participant characteristics (e.g., comorbid conditions) is needed. Further, more research is needed to determine the profile of the student who may benefit most from a digital program. The program was not examined in the context of a spectrum of services. Intervention research examining the contribution of the program in conjunction with other support services should be examined. Other limitations include that outcome measures in this study were based on self-reports, which may not be reliable. Outcome measures that include on-campus performance measures should be included (e.g., number of services accessed; types of learning strategies used; number of courses passed; number of assignments completed). Further, different recruitment procedures should be considered including direct meetings with referral sources and opening the inclusion criteria to include various populations to avoid tokenism. Consequently, the impact of the program could be evaluated across groups (e.g., neurotypical, PSS-first-generation student).