VRainSUD: Content validation of a Cognitive Training Program using the Delphi Method

VRainSUD: Content validation of a Cognitive Training Program using the Delphi Method | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article VRainSUD: Content validation of a Cognitive Training Program using the Delphi Method Tânia Caetano, Maria Salomé Pinho, Hugo Freire, Dany Mota, Eduardo Ramadas, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3869938/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Substance abuse has undeniable effects on a number of cognitive dimensions, namely memory and executive functioning. In turn, deficits in these dimensions have been associated with poorer treatment outcomes for Substance Use Disorders (SUD). The main goal of this study was to validate VRainSUD, a cognitive training program composed of a virtual reality platform and a mobile follow-up application that proposes to reduce cognitive deficits in individuals with SUD. The Delphi method was used to reach a consensus on each of the program’s characteristics (e.g., structure, cognitive training tasks). A panel of experts was invited to participate in the content validation by answering two rounds of questions (scale and free-text boxes) regarding the program. Changes were made between the rounds according to the results of the first round. The consensus was defined as at least 70% of the experts agreeing on the validity of a characteristic. From 39 invited experts, 11 completed the first round, and 6 completed both rounds. The structure of the program reached a consensus on the first round, along with most tasks (with the exception of two mobile application tasks). Significant changes were made following the feedback received, namely turning the program personalizable, and adding positive feedback to the tasks in order to improve adherence and engagement. The Delphi method was a valuable tool to improve VRainSUD. The experts’ opinions not only allowed for the validation of main features, but also informed important additional changes. Substance use disorders cognitive training virtual reality VRainSUD Delphi method content validation Introduction Substance use disorders, brain changes and cognitive deficits The global prevalence of substance use disorders (SUD) has increased significantly over the last decades (Degenhardt et al. 2018 ). According to Our World in Data, it is estimated that around 1.4% (Ritchie and Roser 2018a ) and 0.9% (Ritchie and Roser 2018b ) of the world population have an alcohol or drug use disorder respectively. The disease burden of these disorders is high. The latest data reveal that each year more than 300.000 direct deaths can be attributed to SUD (Ritchie and Roser 2018a , b ). With indirect deaths, the number more than doubles. In 2019 (Roser et al. 2021 ), SUD had a disease burden of 35.13 million disability-adjusted life years (DALYs), higher than natural disasters, conflict and terrorism, and interpersonal violence (e.g., murder). Given its growing prevalence and significant disease burden, it is not surprising that treatment for SUD has been the object of considerable interest from the scientific community. However, relapse rates remain high with more than two thirds of individuals returning to substance use within one year of finishing treatment (Brandon et al. 2007 ). The consistent relapse rates have been attributed to the chronicity of SUD, which itself is at least in part explained by neural changes that are shown to persist long after substance use stops. The development and maintenance of SUD seem to be mostly explained by three key neurocircuitry domains, which map onto the three functional stages of addiction (Koob and Volkow 2016 ): the basal ganglia (binge/intoxication), the extended amygdala (withdrawal/negative affect), and the prefrontal cortex (preoccupation/anticipation). The basal ganglia are responsible for the rewarding effects of substance use as well as for the formation of habitual substance taking. The extended amygdala controls the feelings of unease, anxiety, and irritability that accompany withdrawal. Finally, the prefrontal cortex is implicated in executive function, namely the ability to make decisions and inhibit behavior (i.e., substance use). According to Koob and Volkow ( 2016 ), the preoccupation and anticipation stage of addiction, explained by the changes in prefrontal cortex as well as in the insula and allocortex, could be a main element of relapse in humans. Imaging studies have found deficits in executive function associated with a decrease in frontal cortex activity that impacts working memory, decision-making, inhibitory control, and self-regulation (Volkow et al. 2011 ). Considering the influence that deficits in these cognitive domains seem to have on the relapsing nature of SUD, an effective intervention to mitigate them could be an important addition to treatment. A promising possibility is cognitive training. Cognitive training programs for Substance Use Disorders The effectiveness of cognitive training is still a controversial subject. Many studies (e.g., Sala and Gobet, 2019 ) have reported that effects resulting from cognitive training may be short-term or training specific, not translating into an improvement of general cognitive functioning in a given domain. However, some authors suggest that the lack of “far-transfer” effects may be due to a failure to broaden its concept as well as an almost exclusive reliance on primary outcomes (e.g., Brooks et al., 2020 ). Nonetheless, even amid the absence of consensus regarding its effectiveness, the state of the art indicates that cognitive training programs can lead to significant improvements in a number of populations, particularly in mild cognitive impairment (Brum et al. 2009 ; Mendoza Laiz et al. 2018 ; Zhang et al. 2019 ), parkinson’s disease (Leung et al. 2015 ; Gavelin et al. 2022 ), traumatic brain injury (Hallock et al. 2016 ; Vander Linden et al. 2019 ), as well as in patients living with HIV (Wei et al. 2022 ). We have recently conducted a systematic review looking into the effective of cognitive training on memory, executive functioning and processing speed in individuals with SUD (Caetano et al. 2021 ). From the 26 studies included in the review, 18 found some type of cognitive improvement, even though two of these saw only marginally significant effects. The majority of the included studies focused on computerized cognitive training programs and only five used a paper-and-pencil modality. Even though it is clear that these types of programs are increasingly using new technologies (e.g., smartphone applications), none of the studies included in the review utilized one of the most recent – virtual reality (VR). Virtual reality interventions and cognitive training Virtual reality can be described as a simulated environment with scenes and objects that appear to be real and with which the user can interact as if they were present (Biocca 1992 ). By providing an immersive and, at times, naturalistic environment, virtual reality seems to enhance motivation and engagement (Makransky et al. 2019 , 2021 ). The research into the use of virtual reality technology in healthcare interventions is still recent, but shows promising results. In a recent systematic review, Rowland et al., ( 2022 ) found that virtual reality was an efficacious treatment modality for emotional disorders, namely anxiety disorders and post-traumatic stress disorders. Another review looking specifically into the use of virtual reality in the treatment of addictive disorders (Segawa et al. 2020 ), also reported positive results. These findings are supported by a systematic review of reviews conducted by Cieślik et al. ( 2020 ), which provided evidence for the positive impact of VR interventions in psychiatric disorders. While these and other studies focused on psychological interventions, such as cognitive and behavioral therapy and exposure therapy (Bordnick et al. 2011 ; Hone-Blanchet et al. 2014 ; Segawa et al. 2020 ; Rowland et al. 2022 ), there have been also studies interested in the use of VR for cognitive training in these populations. Jahn et al. ( 2021 ) conducted a systematic review of randomized controlled trials studying fully immersive VR cognitive training in patients with neurological (e.g., mild cognitive impairment) and psychiatric disorders (e.g., schizophrenia). They found that cognitive training through VR led to significant improvements in some cognitive domains, specifically executive function, and attention. However, despite considering the results as promising preliminary evidence for the effectiveness of VR cognitive training interventions, the authors also highlighted the need for larger and higher quality studies. No study focusing on SUD or other addictive disorders was included in the review. To our knowledge, only two studies have looked into the effectiveness of a VR cognitive training intervention in individuals with SUD. Man ( 2018 ) investigated the impact of a VR-based vocational training system in the cognitive performance, vocational outcomes, and work-related self-efficacy of young ketamine users. The results indicated that the VR group showed significant improvements in attention and memory, both maintained after 3 months. Participants from that group also presented a higher employment rate, even though the difference was not significant. Gamito et al. ( 2021 ), studied the effect of the Systemic Lisbon Battery (SLB; Gamito et al., 2016 ), VR-based serious games platform, in individuals with alcohol use disorder undergoing inpatient treatment. Results indicated that the SLB positively impacted attention and cognitive flexibility. Even though these studies categorize their interventions as cognitive training, both appear to be more focused on improving functioning on an everyday context, something more akin to cognitive rehabilitation. The presented example tasks mirror real-world activities, such as choosing clothes to wear, going to the supermarket or, in the VR-based vocational training system studied by Man ( 2018 ), doing the work of a salesman on a clothing shop. They do not target specific cognitive domains as much as complex daily chores. VRainSUD VRainSUD is a cognitive training program (CTP) developed as an add-on intervention to be used with patients already receiving treatment for SUD, with the goal of mitigating cognitive deficits caused by long-term substance use and, hopefully, contribute to overall treatment success and a reduction of the relapse rates. It includes two modalities for two different stages of treatment: 1) a fully immersive VR platform with training exercises targeting specific cognitive domains, to be administered by psychologists or neuropsychologists (with training in the area of ​​cognitive training) to patients undergoing treatment; 2) a mobile follow-up application also with cognitive training exercises, for Android and iOS, to be used by the patients (who previously used the VR platform) at their discretion after the end of treatment. The VR platform aims to improve memory, executive functioning, and processing speed, while the follow-up application intends to maintain potential cognitive gains achieved through the VR platform during treatment. This CTP was created taking into account an exhaustive search of cognitive training programs and cognitive training tasks for psychiatric disorders in general and SUD in particular. Additionally, the clinical experience of the professionals working in the inpatient treatment center where the CTP, if proven effective, will be integrated into the treatment program was also considered. Their input was invaluable for the creation of a program to which the patients will adhere to and feel motivated to complete. For example, while both of the studies presented before (Man 2018 ; Gamito et al. 2021 ) used ecologically sound tasks focused on daily activities (e.g., going to the supermarket), the professionals with years of experience with individuals with SUD who were questioned, before the creation of the CTP, do not consider that this type of task would be an added value and did not believe that this type of task would attract the population with SUD. As such, we opted for more attractive and gamified domain-specific tasks. Despite the extensive research that was conducted, we believe that content validation is essential to arrive at the best possible program. To that end, our goal was to utilize the Delphi method to gather expert opinion on and validate the following dimensions of the CTP: 1) structure of the program (e.g., organization, number, and duration of sessions; type and number of tasks per session); 2) cognitive training tasks of the VR platform; 3) cognitive training tasks of the mobile follow-up application. Methods Study Design The Delphi technique is an established research method that involves rounds of survey questions sent out to a panel of experts on a given subject and that, according to Häder (2014, as cited in Niederberger & Spranger, 2020 ), can be used to achieve one or more of the following goals: 1) find consensus among experts; 2) aggregate ideas; 3) make future predictions; 4) determine experts’ opinions. It is founded on the believe that the opinions of many outweigh those of a single individual, even if the individual is the foremost expert in their field. In the present study, an online platform was used to anonymously collect the survey responses from an international group of experts. Similarly to the original Delphi method (Hasson et al. 2000 ), this study used a mixed-methods approach, with both rounds including Likert scales for quantitative analysis of consensus and free-text boxes to gather the experts’ opinions and allow for the sharing of new ideas about how to improve the program. All respondents gave their informed consent before participating in the study. Participants The participants were professionals with academic and/or practical experience in SUD and cognitive training. Inclusion criteria There is no agreed method to consider someone an expert in a given field. To facilitate the selection process, we used a simple point evaluation system. We included all professionals with a final evaluation of at least 2 points. Regarding academic experience, we considered the following factors: At least one publication on the field of cognitive training (1 point) & at least one publication on the field of SUD (1 point) OR at least one publication on the field of cognitive training in SUD (2 points). Concerning practical experience, we considered experience in the development of a cognitive training program (1 point). Moreover, we also considered if the professional had clinical experience in the application of cognitive training. However, we chose not to include this dimension in the quantitative analysis given how difficult it was to find the necessary information to confirm this type of experience. Recruitment A sample size of 8 to 15 respondents has been recommended for Delphi studies, given that smaller samples reduce reliability and bigger ones do not appear to add much value (Keeney et al. 2010 ). For this study, we aimed to recruit 10–12 participants. Potential participants were identified through a literature review, existing academic contacts, and a web search for cognitive training programs (for SUD or other conditions) and the respective scientific teams. After being evaluated in accordance to the point system previously described, professionals who met the inclusion criteria (2 points or more) were invited to participate by email. In the initial questionnaire, they were asked to recommend other professionals who they considered to be experts in the field. Recommended professionals who met the inclusion criteria were contacted and invited to join the expert panel. A total of 39 professionals were invited to participate in the expert panel. Survey Design Even though classic Delphi studies use a qualitative approach in the first round, with the goal of allowing a broader understanding of the expert’s opinion on a given topic, this would not have been adequate for the current study. At the time of validation, VRainSUD was already on a beta phase and the main purpose was to get feedback on very specific characteristics such as the cognitive training tasks and the structure of the program. This could only be achieved through directed questions. As such, we used a modified study design that included both Liker-scale questions (from 1-strongly disagree to 5-strongly agree) and free-text boxes for qualitative feedback. The draft of the first-round survey was reviewed by several academics and clinicians involved in the project to different degrees. Taking into account their feedback, the following changes were made: 1) more background information on VRainSUD was added; 2) information on the cognitive training tasks was simplified and demonstrative GIFs were included; 3) the structure of the survey was redesigned in order to reduce its length and avoid repetition. Procedure The Research Ethics and Deontology Committee (CEDI) of the Faculty of Psychology and Educational Sciences of the University of Coimbra approved the research project in which this content validation study is integrated. The link for the first-round survey was sent to the participants by email, along with a brief explanation on how to access it and the planned deadline (1 month). The questionnaire was delivered using google forms, allowing the participants to complete it electronically while ensuring anonymity. In the first part of the questionnaire, participants were asked to give their electronic consent and provide some basic sociodemographic and professional information. We collected the participants’ sex, age, educational background, and profession as well as country and city of work. Moreover, we also inquired about the participants’ practical experience in cognitive training. The Likert-scale responses from the first round were analyzed to understand which questions had reached consensus and which needed to be addressed before being included in the second round. The responses from the free-text boxes were first transcribed verbatim and then subject to qualitative content analysis. They informed additional changes that could be made to improve the structure of the VRainSUD program. The link to the second-round survey was sent approximately two weeks after the first round closed. In the first part of this second questionnaire, a summary of the results from the first round was presented, along with additional information requested from some of the participants (e.g., regarding the target population). Participants were given two weeks to complete the second survey. Analysis Quantitative analyses were conducted using IBM SPSS version 28. A database was created with every sociodemographic and Likert-scale questions/items (e.g. “The program using the VR platform consists of 12 individual sessions, held 3 times a week, for 4 weeks, with a duration of 30 minutes per session.”). Categorical variables were encoded (e.g., 0-female and 1-male) and ordinal scale variables (Likert-scale questions/items) used the following correspondence: 1-strongly disagree; 2-disagree; 3 – neither disagree nor agree; 4-agree; 5-strongly agree. Consensus in a given question/item was defined as at least 70% of responses being of agreement (4- agree or 5-strongly agree; (Sumsion 1998 )). Descriptive statistics were used to characterize the sample and assess consensus on both rounds of the survey. The Mann-Whitney U Test was used to evaluate the significance of the differences between the means. Qualitative content analysis was conducted to assess the free-text box responses. The feedback from the experts was organized by categories (e.g., program structure, session zero, performance sequence, and others), facilitating its interpretation and the recognition of recurring themes or patterns. This organization also resulted in the structure of the second-round questionnaire. Results Email invitations to participate in the expert panel were sent to a total of 39 professionals. This included 35 who were identified by the research team and 4 who were recommended by one of the experts. Of these, 11 (28.2%) completed the first round. Six (15.4%) of those who responded to the first round also completed the second round. Of the participants included in the first round (n = 11), 63.6% were men (n = 7) and 36.4% were women (n = 4). The average age of participants is 45 years. Regarding educational background, 3 indicated clinical neuropsychology (27.3%), 2 psychology (18.2%), 1 cognitive neuroscience and psychology (9.1%), 1 experimental psychology (9.1%), 1 clinical psychology (9.1%), 1 cognitive psychology (9.1%), 1 cognitive psychology and cognitive neuroscience (9.1%) and 1 smart healthcare (9.1%). Concerning their profession, 2 reported being researchers and 9 professors. Of the 11 experts, only one responded that they did not have practical clinical experience in cognitive training (having only research experience). The average practical clinical experience of the 10 experts was 14.25 years. Regarding the country where they worked, 2 indicated the United States of America (18.2%), 1 United Kingdom, 1 South Africa (9.1%), 1 Australia (9.1%), 1 Austria (9.1%), 1 Iran (9.1%), 1 Canada (9.1%), 1 India (9.1%), 1 Portugal (9.1%) and 1 Korea (9.1%). Finally, as for the city of work, 1 mentioned Liverpool (9.1%), 1 Cape Town (9.1%), 1 Bengaluru (9.1%), 1 Melbourne (9.1%), 1 Tehran (9.1%), 1 San Francisco (9.1%), 1 Kelowna (9.1%) and 1 Busan (9.1%). Two experts (19.2%) decided not to report their city of work. Sociodemographic data were not collected in the second-round questionnaire. Round 1 An overview of the results from round 1 is presented in Table II. VR Platform In the round 1 questionnaire, the experts were presented with various general aspects of the VR platform, namely program structure, session zero and performance sequence, as well as the VR cognitive training tasks and a set of structured sessions (which from session 7 onwards would be administered sequentially to all patients). Several suggestions were taken into account following the 1st round questionnaire. General Aspects of the VR Platform. In the first round, regarding general aspects of the VR platform, the consensus was reached on all topics included, namely on the structure of the program (72.2% of consensus; the number of weeks, sessions per week, duration of the sessions), session zero (90.9% of consensus) and performance sequence (72.7% of consensus). Even though there was consensus on all these aspects of the platform, some experts suggested that more sessions should be done per week or that the duration of the intervention should be extended in order to increase the total hours of cognitive training ( “Consider higher dosage, i.e., daily cognitive training for 4 weeks or longer duration 8 weeks.”; "It suggested a longer intervention in time, even if only with two weekly sessions.” ). Still regarding the general aspects of the VR Platform, another expert suggested: “I suggest to reduce the features that may induce stress on participants, particularly for the first sessions. For example time pressure, lack of positive feedbacks, graphical designs. Having sounds that induce a feel of achievement and enjoy is highly suggested (you might already have these feature).” VR Platform’s Cognitive Training Tasks. Consensus (more than 70% of the answers were Agree or Strongly Agree) was also reached on all the VR cognitive training tasks (described in Table I). One of the experts mentioned: “ I like the task design. Particularly the inhibitory control task and working memory tasks (I have more familiarity with lab-based versions of these tasks, so it's easier to see how they would be implemented in VR ”. Another expert suggested: “…to have brief dynamic feedback for the Labyrinth navigation task, since the participant may proceed and then realize he/she might went wrong and the time is up… Overall, the tasks seem amazing.”. Cognitive Training Sessions with VR Platform. In the 1st round, the VR Platform sessions were presented in a structured way with a specific task sequence selected for each session. However, the consensus was not reached on the structure of the sessions and on the fact that the sessions would be repeated after the seventh. One of the experts suggested “to put working memory related tasks as the first training task for each session, since it demands more cognitive energy and keep those tasks which are more gamified … for the last part of the session” . Likewise, most experts suggested that sessions be personalized for each patient. One suggested “consider performance-based training vs. everything sequentially, i.e., train in tasks that patient exhibits the worst performance at baseline” , while another considered it would “be more appropriate an evolution considering the previous neuropsychological assessment and contingent on the performance of the tasks would” . Mobile Follow-up Application In the round 1 questionnaire, all information about the program structure, how to complete the tasks, as well as a description of all the tasks included in the mobile application was made available to the experts. Several suggestions from experts to improve the mobile application in question were also considered. General Aspects of the Mobile Follow-up Application. Although all general aspects of the follow-up mobile application obtained a consensus of more than 70%, some experts suggested some improvements and made some considerations. One of the experts mentioned: “30 minutes a day can be quite intensive for participants; it would be nice if you could reduce time commitment while maintaining the same active ingredients. It would also be important to consider the context in which participants will train, e.g. supervised or unsupervised, …” . Another commented that “It would be interesting to have assessment after 3 month/6 months of interval to measure the true transfer effect of CTP on cognitive functioning”. Other suggestions for improvement from another expert were also taken into account: “I think repeating the training tasks during follow-up could be less intense in terms of the frequency of sessions (i.e., two times per week).” Mobile Follow-up Application’s Cognitive Training Tasks. Consensus (more than 70% of the answers were Agree or Agree Completely) was reached on the cognitive training tasks 1, 3 (which became task 2), and 4 (which became task 3). We did not reach a consensus on tasks 2 (which became task 1.1.) and 5 (which became task 4). For task 4 (before task 5) we considered the following suggestions for improvement: “I suggest that instead of the instruction appear objects with the confounding characteristic, for example, red square, red circle. Instead of finishing the task if you make a mistake, I suggest displaying the information that you made a mistake and allowing you to keep trying until you get the rule right, a bit like in WIT or Wisconsin.” and “Task 5 has many cognitive functions involved; (motor) inhibition may not very well targeted”. A description of the follow-up mobile application cognitive training tasks is available in Table I. Round 2 An overview of the results from round 2 is presented in Table III. VR Platform In the second-round questionnaire, all items that did not reach consensus in the first round were considered, as well as items that, although they had reached consensus, would benefit from and could be adjusted according to the experts’ suggestions. General Aspects of the VR Platform. Regarding the VR platform, even though consensus had been reached in the first round on the structure of the program (72.7%) and on the performance sequence (72.7%), we chose to make changes based on the experts' suggestions. Therefore, these dimensions were once again evaluated in the second-round questionnaire. Based on the experts' suggestions, we extended the inpatient treatment duration from 6 to 8 weeks, which allowed us to increase the cognitive training program by another 2 weeks (6 weeks instead of 4). The first week of treatment will be used for the pre-test and establishment of a baseline, the cognitive training will happen between the second and the seventh weeks (inclusive), and the last week will be used for the post-test. Previously, the program had 6 distinct sessions that were repeated sequentially. Based on the experts' feedback, it was decided that the cognitive training will be personalized, and more focus will be given to the cognitive domains in which the participant presents the most difficulties at baseline. Consideration will also be given to prior neuropsychological assessment and task performance as the sessions progress. Since the tasks administered will be based on the participant’s individual needs, there will be no repetition of structured sessions (previously, from the 7th session, the patients repeated the same structured sessions that they had already performed in the first six training sessions). Other suggestions for improvement were also considered, namely regarding the order of the tasks within each session. The first task of every session will be a working memory task while the more gamified tasks will be left for the last part of the session. Mobile Follow-up Application For the follow-up mobile application, the second-round questionnaire considered aspects that had not reached consensus among the experts, as well as other suggestions for improvement. General Aspects of the Mobile Follow-up Application. Even though the consensus was reached on task 2 (before task 3; 72.7%), on the program structure (81.8%), and on the task completion (72.7%), with the goal of improving the mobile application, some of the experts' suggestions were introduced. Since most experts mentioned that 30 minutes a day could be quite intense for the participants, the advised training frequency will be reduced. A directive training approach will be adopted with each participant being asked to complete 20 minutes of training, twice a week (based on the follow-up individual sessions). The psychologist will instruct the participant to do two specific tasks in each of these 20 minutes training sessions. Mobile Follow-up Application’s Cognitive Training Tasks. Task 2 will no longer exist, becoming a subtask of task 1. After task 1 is finished, task 1.1. (previously task 2) will automatically appear. In this subtask, the patient must identify objects presented in the previous task (task 1), selecting “Yes” for the objects they remember. Even though consensus had been reached on task 2 (before task 3), considering some comments from the experts, this task was replaced with a modified (non-VR) version of the VR task targeting the same cognitive domain (which reached a consensus greater than 90%). Thus, task 2 became remembering and reproducing a number sequence. Some numbered squares appear in the scenario, and after a few seconds, the numbers in question are hidden. The participant must select the squares where the numbers were, following the order in which they appeared initially. The difficulty increases as more numbered squares appear and the subject will have to memorize a longer sequence. Based on the experts' suggestions, task 5 (now task 4) was reformulated. Thus, instead of the instruction indicating the objects that the subject will have to avoid, objects with a confusing characteristic will appear, for example, a red square, or red circle. Finally, it was decided that all tasks in the mobile application will have a set time instead of coming to an end when a participant makes a mistake. The participant will be informed of the mistake but will be allowed to continue trying to successfully complete the task until the established time comes to an end. Differences between rounds A summary of the changes made between the two rounds is presented in Table IV. The mean difference for each item repeated on both rounds was calculated by subtracting the mean of the first round from the mean of the second. Afterwards, the sample’s distribution was assessed using the Kolmogorov-Smirnova test and a lack of normality was verified. As such, the non-parametric test for independent samples, Mann-Whitney U Test was chosen to test the significance of the differences between the two rounds. The one-tailed test was selected in a specific direction because the N (= 11) of first round is greater than the N (= 6) of second round. In table IV, mean differences between the rounds are presented. No difference was statistically significant. Discussion The main goal of this study was to validate VRainSUD, a cognitive training program developed to mitigate the cognitive deficits of individuals receiving treatment for SUD. The focus was on two major dimensions of the program: the structure (e.g., number, frequency, and duration of sessions; type and number of tasks per session) and the cognitive training tasks (of the VR platform and the mobile follow-up application). With the exception of two cognitive training tasks in the mobile follow-up application, both of these two dimensions reached the required level of consensus in the first round. Despite this, significant changes were made taking into account the feedback from the panel of experts. Initially, VRainSUD was designed to follow a specific sequence of sessions with a pre-determined combination of cognitive training tasks, with the goal of giving similar training time to the various cognitive domains being targeted (i.e., memory, executive functions, processing speed). However, several experts indicated that a personalizable program would not only be more versatile but potentially also more effective. As such, the structure was altered, while allowing for the personalization of each session according to the baseline deficits as well as the progress of a given patient. The number of sessions itself was increased by maintaining their frequency (3 sessions per week) but extending the duration of the program. Considering that VRainSUD was developed with a specific setting in mind, this factor was conditioned by the characteristics of the inpatient treatment in question. However, we were able to increase the duration of the treatment from 6 to 8 weeks, allowing for 6 weeks of cognitive training (instead of the initial 4 weeks). Even though this change is significant, some experts considered that an even higher number of cognitive training hours could be beneficial. The optimal number of training hours will likely vary according to the baseline deficits and progression of each individual using the program. Future studies using VRainSUD without our current constraints (i.e., other settings), could investigate the relative effectiveness (and adherence) of different program durations. In comparison with the VR platform, the mobile follow-up application (e.g., structure and tasks) received more criticism and was subject to more changes. This could be in part explained by the fact that the mobile application was in an earlier stage of development at the moment of validation. Moreover, it is important to note that despite the increased technical difficulty of the VR platform, the mobile follow-up application may actually be more complex in order regards, given that it will be used without constant supervision. As such, many of the experts’ suggestions were focused on not only improving the cognitive training tasks themselves but also increasing the patients’ adherence to this part of the program. Both of these factors can have a significant impact on the program’s effectiveness. Regarding the tasks, one was added to the previous one, another was replaced by a modified version of a VR training task, which had already received a high consensus and yet another was altered. Equally important, significant changes were made to improve adherence and motivation. The frequency and duration of suggested training sessions was reduced, and the training sessions will be incorporated into the individual sessions the patients will have twice a week with a clinical psychologist (part of an aftercare program offered to those who completed in-patient treatment). Once again, we should note that the structure of this part of the program is constrained by the already existing treatment in which it will be integrated. Future studies could experiment with additional structural changes, but a directive approach (versus self-management of training time) should always be preferred. The mean difference between the levels of consensus of the two rounds was greater for the structure of the VR platform and one of the tasks of the mobile follow-up application. This result is relevant since these are also the two dimensions that suffered the most changes following the feedback received in the first round. Despite this, the differences still weren’t statistically significant, which may be explained by the small sample size. Even though the focus of the validation was on the structure and tasks of the CTP, the inclusion of free-text boxes on both rounds allowed the experts to comment on/suggest other, more specific, features. It was not possible to accommodate all the suggestions, but we did incorporate a majority of them. Still looking to improve motivation and adherence, a positive feedback system was added so that a patient is validated (through sounds and visual stimuli) when they successfully complete a task and/or move on to another level of difficulty. A study by Burgers et al. ( 2015 ) looking into the impact of feedback on motivation in a brain-training game (a type of cognitive training), reported that positive feedback promotes long-term motivation. In a more recent study with a mental arithmetic task, Peifer et al. ( 2020 ) found that the relationship between positive feedback and flow and performance seems to be mediated by self-efficacy. Certain changes were also made with the goal of preventing/reducing frustration. For the VR tasks, a patient will only move to another level of difficulty if they successfully complete the previous one (starting with a baseline level). Each task has many different exercises for each level. Finally, on the mobile follow-up application, a set duration time was added to all the tasks. When a patient makes a mistake, they receive feedback on this mistake but are allowed to continue training (versus having to start over from the beginning). The present study has some limitations that should be taken into account when interpreting the results. The sample of experts used was relatively small, especially considering that only around half of the original panel completed the second round. Even though too big samples are not useful when using the Delphi method, we do believe that it would have been beneficial to have a slightly higher number of experts and a better turnout rate at the second round. This limitation could be explained by the fact that the subject under validation is very specific and therefore, the pool of available experts is itself limited. Another important limitation to consider is the current state of the art on VR and its use for cognitive training in general, and in individuals with SUD in particular. Since VR and its use in health interventions is such a recent innovation, even highly qualified and experienced professionals do not have all the answers regarding the best way to utilize it in order to optimize the intervention’s effectiveness. As with every new technology or scientific development, it will take time and many more studies to truly understand and make use of all of VR’s potential. While not widely used in the field of clinical or neuropsychology, the Delphi method was a valuable tool in the development of VRainSUD. It allowed not only for the validation of certain key dimensions of the program, such as its structure and cognitive training tasks, but also for its overall improvement. Moreover, the experts’ feedback was essential for taking into account factors such as adherence, motivation, and engagement, which have their own impact on the program’s effectiveness. In view of its contribution, we consider that content validation using the Delphi method should be more often utilized when designing new cognitive or psychosocial interventions. Declarations Author Contribution Conceptualization, T.C., M.P. and M.D.; methodology, T.C., M.P. and M.D.; investigation and formal analysis, T.C., E.R. and J.L.; resources, T.C., H.F. and D.M.; software, H.F. and D.M; writing – original draft preparation, T.C.; writing – review and editing, T.C., H.F. and J.L.; supervision, M.P. and M.D.; project administration, T.C.; validation, T.C. All authors contributed to manuscript revision, read, and approved the submitted version. Acknowledgment A special thank you to all the experts who dispended some of their time participating in the expert panel for this Delphi validation study. Their feedback was invaluable. References Biocca F (1992) Communication Within Virtual Reality: Creating a Space for Research. J Commun 42:5–22. https://doi.org/10.1111/j.1460-2466.1992.tb00810.x Bordnick PS, Carter BL, Traylor AC (2011) What virtual reality research in addictions can tell us about the future of obesity assessment and treatment. J Diabetes Sci Technol 5:265–271. https://doi.org/10.1177/193229681100500210 Brandon TH, Vidrine JI, Litvin EB (2007) Relapse and relapse prevention. Annu Rev Clin Psychol 3:257–284. https://doi.org/10.1146/annurev.clinpsy.3.022806.091455 Brooks SJ, Mackenzie-Phelan R, Tully J, Schiöth HB (2020) Review of the neural processes of working memory training: Controlling the impulse to throw the baby out with the bathwater. Front Psychiatry 11:1–15. https://doi.org/10.3389/fpsyt.2020.512761 Brum PS, Forlenza OV, Yassuda MS (2009) Cognitive training in older adults with Mild Cognitive Impairment: Impact on cognitive and functional performance. Dement Neuropsychol 3:124–131. https://doi.org/10.1590/s1980-57642009dn30200010 Burgers C, Eden A, Van Engelenburg MD, Buningh S (2015) How feedback boosts motivation and play in a brain-training game. Comput Human Behav 48:94–103. https://doi.org/10.1016/j.chb.2015.01.038 Caetano T, Pinho MS, Ramadas E, et al (2021) Cognitive training effectiveness on memory, executive functioning, and processing speed in individuals with substance use disorders: A systematic review. Front Psychol 12:. https://doi.org/10.3389/fpsyg.2021.730165 Cieślik B, Mazurek J, Rutkowski S, et al (2020) Virtual reality in psychiatric disorders: A systematic review of reviews. Complement Ther Med 52:. https://doi.org/10.1016/j.ctim.2020.102480 Degenhardt L, Charlson F, Ferrari A, et al (2018) The global burden of disease attributable to alcohol and drug use in 195 countries and territories, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. The Lancet Psychiatry 5:987–1012. https://doi.org/10.1016/S2215-0366(18)30337-7 Gamito P, Oliveira J, Brito R, et al (2016) Evaluation of cognitive functions through the systemic lisbon battery: Normative data. Methods Inf Med 55:93–97. https://doi.org/10.3414/ME14-02-0021 Gamito P, Oliveira J, Matias M, et al (2021) Virtual reality cognitive training among individuals with alcohol use disorder undergoing residential treatment: Pilot randomized controlled trial. J Med Internet Res 23:1–10. https://doi.org/10.2196/18482 Gavelin HM, Domellöf ME, Leung I, et al (2022) Computerized cognitive training in Parkinson’s disease: A systematic review and meta-analysis. Ageing Res Rev 80:. https://doi.org/10.1016/j.arr.2022.101671 Hallock H, Collins D, Lampit A, et al (2016) Cognitive training for post-acute traumatic brain injury: A systematic review and meta-analysis. Front Hum Neurosci 10:. https://doi.org/10.3389/fnhum.2016.00537 Hasson F, Keeney S, McKenna H (2000) Research guidelines for the Delphi survey technique. J Adv Nurs 32:1008–1015. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x Hone-Blanchet A, Wensing T, Fecteau S (2014) The use of virtual reality in craving assessment and cue-exposure therapy in substance use disorders. Front Hum Neurosci 8:1–15. https://doi.org/10.3389/fnhum.2014.00844 Jahn FS, Skovbye M, Obenhausen K, et al (2021) Cognitive training with fully immersive virtual reality in patients with neurological and psychiatric disorders: A systematic review of randomized controlled trials. Psychiatry Res 300:113928. https://doi.org/10.1016/j.psychres.2021.113928 Keeney S, Hasson F, Mckenna H (2010) The delphi technique in nursing and health research Koob GF, Volkow ND (2016) Neurobiology of addiction: a neurocircuitry analysis. The Lancet Psychiatry 3:760–773. https://doi.org/10.1016/S2215-0366(16)00104-8 Leung IHK, Walton CC, Hallock H, et al (2015) Cognitive training in Parkinson disease: A systematic review and meta-analysis. Neurology 85:1843–1851. https://doi.org/10.1212/WNL.0000000000002145 Makransky G, Andreasen NK, Baceviciute S, Mayer RE (2021) Immersive virtual reality increases liking but not learning with a science simulation and generative learning strategies promote learning in immersive virtual reality. J Educ Psychol 113:719–735. https://doi.org/10.1037/edu0000473 Makransky G, Terkildsen TS, Mayer RE (2019) Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learn Instr 60:225–236. https://doi.org/10.1016/j.learninstruc.2017.12.007 Man DWK (2018) Virtual reality-based cognitive training for drug abusers: A randomised controlled trial. Neuropsychol Rehabil 8:1–18. https://doi.org/10.1080/09602011.2018.1468271 Mendoza Laiz N, Del Valle Díaz S, Rioja Collado N, et al (2018) Potential benefits of a cognitive training program in mild cognitive impairment (MCI). Restor Neurol Neurosci 36:207–213. https://doi.org/10.3233/RNN-170754 Niederberger M, Spranger J (2020) Delphi Technique in Health Sciences: A Map. Front Public Heal 8:1–10. https://doi.org/10.3389/fpubh.2020.00457 Peifer C, Schönfeld P, Wolters G, et al (2020) Well Done! Effects of Positive Feedback on Perceived Self-Efficacy, Flow and Performance in a Mental Arithmetic Task. Front Psychol 11:. https://doi.org/10.3389/fpsyg.2020.01008 Ritchie H, Roser M (2018a) Alcohol Consumption. In: Our World Data. https://ourworldindata.org/alcohol-consumption . Accessed 10 May 2022 Ritchie H, Roser M (2018b) Opioids, cocaine, cannabis and illicit drugs. In: Our World Data Roser M, Ritchie H, Spooner F (2021) Burden of Disease. In: Our World Data. https://ourworldindata.org/burden-of-disease . Accessed 20 May 2022 Rowland DP, Casey LM, Ganapathy A, et al (2022) Psychosocial Intervention Emotional Disorders. 31:1–20 Sala G, Gobet F (2019) Cognitive training does not enhance general cognition. Trends Cogn Sci 23:9–20. https://doi.org/10.1016/j.tics.2018.10.004 Segawa T, Baudry T, Bourla A, et al (2020) Virtual Reality (VR) in Assessment and Treatment of Addictive Disorders: A Systematic Review. Front Neurosci 13:. https://doi.org/10.3389/fnins.2019.01409 Sumsion T (1998) The Delphi technique: An adaptive research tool. Br J Occup Ther 61:153–156 Vander Linden C, Verhelst H, Deschepper E, et al (2019) Cognitive training benefit depends on brain injury location in adolescents with traumatic brain injury: A pilot study. Eur J Phys Rehabil Med 55:585–594. https://doi.org/10.23736/S1973-9087.18.05548-X Volkow ND, Wang GJ, Fowler JS, et al (2011) Addiction: Beyond dopamine reward circuitry. Proc Natl Acad Sci U S A 108:15037–15042. https://doi.org/10.1073/pnas.1010654108 Wei J, Hou J, Mu T, et al (2022) Evaluation of Computerized Cognitive Training and Cognitive and Daily Function in Patients Living with HIV: A Meta-analysis. JAMA Netw Open 5:1–17. https://doi.org/10.1001/jamanetworkopen.2022.0970 Zhang H, Huntley J, Bhome R, et al (2019) Effect of computerised cognitive training on cognitive outcomes in mild cognitive impairment: A systematic review and meta-analysis. BMJ Open 9:. https://doi.org/10.1136/bmjopen-2018-027062 Tables Table I. Description of CTP tasks 1. Virtual Reality (VR) Platform Tasks Task Task Name Cognitive domains Objective Description Task 1 Labyrinth Navigation Logical/Deductive Reasoning, Planning, Processing Speed Reach the labyrinth's exit. During the time designated for the task in question, several labyrinths appear to the subject. He must use the provided auxiliary map to successfully navigate each labyrinth and reach the exit. The auxiliary map also allows the subject to visualize his current position. Task 1.1. (before Task 2) Remembering Space Long Term Memory Select “YES” for the objects memorized in the task “Labyrinth Navigation” and “No” for the objects that have not appeared before. During the “Labyrinth Navigation” task, various objects will appear in the labyrinths, which the subject will have to memorize. In this task, the subject is presented with these and other objects that have not appeared before and should select "YES" only when he identifies objects that were present in the “Labyrinth Navigation” task. Task 2 (before Task 3) Number Sequence Cognitive Flexibility, Logical/Deductive Reasoning, Processing Speed Complete number sequences. The subject must understand the logic of the presented sequence and complete it with the numbers he has at his disposal. Task 3 (before Task 4) Passing Balls Cognitive Flexibility, Inhibitory Control, Processing Speed, Working Memory Only hit the allowed balls. In this task, the subject has at his disposal a racket, which will allow him to hit the balls that appear in the task scenario. Instructions regarding which balls should not be hit will alternate throughout the task. The instructions for the balls to avoid appear for a few seconds and then disappear, so that the subject must pay attention to the change and memorize it. The longer the subject manages to only hit the allowed balls, the more points he adds to the total score (the higher the total score). Task 4 (before Task 5) Cube Memory Working Memory Remember the location of numbers in sequential order. In the scenario, some numbered cubes will appear, which rotate after a few seconds, hiding the number in question. The subject must select the cubes that had numbers, following the order by which they appear. Task 5 (before Task 6) Working Maze Working Memory Reach the labyrinth's portal (which can only be found if the subject follows all directions correctly). At first, the subject visualizes a sequence of directions (arrows like → ↑ → ← ) on the floor of the maze. He will have to memorize them and navigate the maze according to them. When advancing through the maze, the subject can no longer see the directions. 2. Mobile Follow-up Application Tasks Task Task Name Cognitive domains Objective Description Task 1 Remember the Object Working Memory Only select objects that have not been selected previously. The subject will be selecting objects and will have to memorize them. As more objects appear, he will have to continue selecting only objects that have not been previously selected. Task 1.1 (before Task 2) Remembering Space Long Term Memory The subject will have to indicate with “Yes” or “No” if he had seen the object before. In this subtask, the patient must identify objects presented in the previous task (task 1). Task 2 (before Task 3) Name changed to Squares of Memory (before Treasure Hunt) Working Memory Remember the location of numbers in sequential order. Some numbered squares will appear in the scenario, and after a few seconds, the numbers in question will be hidden. The participant must select the squares that had numbers, following the order by which they appeared initially. Task 3 (before Task 4) Number Sequence Cognitive Flexibility, Logical/Deductive Reasoning, Processing Speed Complete number sequences. The subject must understand the logic of the presented sequence and complete it with the numbers he has at his disposal. Task 4 (before Task 5) Colors and Shapes Cognitive Flexibility, Inhibitory Control and Processing Speed Select objects according to the instructions provided. The subject should select only objects that meet the instructions provided. Objects with a confusing characteristic will appear, for example, red square, red circle. If the patient makes a mistake, the participant will be informed of the mistake but will be allowed to continue trying to find the correct rule until the time set for the task comes to an end. Table II. Appraisal of the Cognitive Training Program (CTP) in Round 1 Item Statements Round 1 (n = 11) Consensus, n (%) Mean (SD) 95% CI 1. Virtual Reality (VR) Platform 1.1. General Aspects of VR Platform 1.1.1. Program Structure The program using the VR platform consists of 12 individual sessions (held 3 times a week, for 4 weeks, with a duration of 30 minutes per session). 72.7% 3.91 (0.70) 3.44 – 4.38 1.1.2. Session Zero A session will be held before the 1st Session (Session zero), during which instructions will be provided for the use of VR glasses and the patients will be allowed to explore the platform and clarify any questions that may arise. The objective of this session is to ensure that all patients have an adequate mastery of the virtual environment before the start of the training sessions. 90.9% 4.73 (0.65) 4.29 – 5.16 1.1.3. Performance Sequence All participants will perform the same execution/performance sequence. The tasks must take place sequentially according to the pre-defined logic. In total, the VR platform comprises 6 distinct tasks. 72.7% 4.00 (0.77) 3.48 – 4.52 1.2. Description of VR Platform Tasks 1.2.1. Task 1 Labyrinth Navigation 72.7% 3.73 (1.01) 3.05 – 4.41 1.2.2. Task 2 (changed to Task 1.1. an round 2) Number Sequence 72.7% 3.73 (0.79) 3.20 – 4.26 1.2.3. Task 3 (changed to Task 2 an round 2) Remembering Space 72.7% 3.73 (0.79) 3.20 – 4.26 1.2.4. Task 4 (changed to Task 3 an round 2) Passing Balls 90.9% 4.18 (0.60) 3.78 – 4.59 1.2.5. Task 5 (changed to Task 4 an round 2) Cube Memory 90.9% 4.27 (0.65) 3.84 – 4.71 1.2.6. Task 6 (changed to Task 5 an round 2) Working Maze 81.8% 4.00 (0.89) 3.40 – 4.60 1.3. Sessions of VR Platform Tasks 1.3.1. Session 1 Session 1 is composed of tasks 1 (Labyrinth Navigation, 10 min.), 2 (Number Sequence, 15 min.) and 3 (Remembering Space, 5 min.). 45.5% 3.45 (0.52) 3.10 – 3.81 1.3.2. Session 2 Session 2 is composed of tasks 4 (Passing Balls, 15 min.), and 5 (Cube Memory, 15 min.). 45.5% 3.45 (0.52) 3.10 – 3.81 1.3.3. Session 3 Session 3 is composed of tasks 1 (Labyrinth Navigation, 10 min.), 4 (Passing Balls, 15 min.), and 3 (Remembering Space, 5 min.). 45.5% 3.45 (0.52) 3.10 – 3.81 1.3.4. Session 4 Session 4 is composed of tasks 6 (Working Maze, 15 min.) and 5 (Cube Memory, 15 min.). 54.5% 3.55 (0.52) 3.19 – 3.90 1.3.5. Session 5 Session 5 is composed of tasks 1 (Labyrinth Navigation, 10 min.), 6 (Working Maze, 15 min.), and 3 (Remembering Space, 5 min.). 45.5% 3.45 (0.52) 3.10 – 3.81 1.3.6. Session 6 Session 6 is composed of tasks 1 (Labyrinth Navigation, 10 min.), 4 (Passing Balls, 15 min.), and 3 (Remembering Space, 5 min.). 54.5% 3.55 (0.52) 3.19 – 3.90 1.3.7. Sequentially repeated sessions From the Session 7 on (regarding the VR platform) the sessions will repeat sequentially. 54.5% 3.55 (0.52) 3.19 – 3.90 2. Mobile Follow-up Application 2.1. General Aspects of Mobile Follow-up Application 2.1.1. Program Structure It is recommended that the subject uses the mobile follow-up application for approximately 30 minutes a day, three times a week, which makes 24 sessions over a period of 8 weeks. 81.8% 4.09 (1.22) 3.27 – 4.91 2.1.2. Task Completion The mobile follow-up application consists of 5 tasks. In each 30-minute session, the subject is expected to alternate between the various tasks in order to provide training in the various cognitive domains. 72.7% 3.73 (1.10) 2.99 – 4.47 2.2. Description of the Mobile Follow-up Application Tasks 2.2.1. Task 1 Remember the Object 81.8% 3.82 (1.08) 3.09 – 4.54 2.2.2. Task 2 (changed to Task 1.1. an round 2) Remembering Space 63.6% 3.55 (1.21) 2.73 – 4.36 2.2.3. Task 3 (changed to Task 2 an round 2) Treasure Hunt 72.7% 3.91 (0.70) 3.44 – 4.38 2.2.4. Task 4 (changed to Task 3 an round 2) Number Sequence 81.8% 3.91 (0.54) 3.55 – 4.27 2.2.5. Task 5 (changed to Task 4 an round 2) Colors and Shapes 54.5% 3.82 (0.87) 3.23 – 4.41 Table III. Appraisal of the Cognitive Training Program (CTP) in Round 2 Item Statements Round 1 (n = 6) Consensus, n (%) Mean (SD) 95% CI 1. Virtual Reality (VR) Platform 1.1. General Aspects of VR Platform 1.1.1. Program Structure Instead of 12 training sessions (3 times per week, for 4 weeks, 30 minutes each session), we changed to 18 training sessions (3 times per week, for 6 weeks, 30 minutes each session). The total number of sessions went from 12 to 18. Note: Taking into account the total duration of the inpatient treatment and the treatment as usual (TAU) in the center where the program will be implemented, which has other daily therapeutic activities, it is not feasible to increase the duration of the training sessions or their frequency. However, we managed to extend the inpatient treatment duration from 6 to 8 weeks, what allows us to increase the cognitive training program another 2 weeks (6 weeks instead of 4). The first week of treatment will be used for the pre-test, the cognitive training will happen between the second and the seventh weeks (inclusive) and, the last week will be used for the post-test. 100% 4.67 (0.52) 4.12 – 5.21 1.1.2. Performance Sequence Previously, the program had 6 distinct sessions that repeated sequentially. Based on the experts' feedback, it was decided that the training will be personalized, and more focus will be given to the cognitive domains in which the participant presents the most difficulties at baseline. Consideration will also be given to prior neuropsychological assessment and task performance as the sessions progress. 83.3% 4.33 (1.21) 3.06 – 5.60 1.2. Description of VR Platform Tasks 1.3. Sessions of VR Platform Tasks 1.3.1. Sessions Based on the experts' suggestions, the tasks administered will be based on the participant’s individual needs, so there will no repetition of structured sessions (previously, from the 7th session, the patients repeated the same structured sessions that they had already performed in the first six training sessions). 100% 4.67 (0.52) 4.12 – 5.21 Although the training will be personalized, other suggestions for improvement will also be considered, namely regarding the order of the tasks within the same sessions. The first task of every session will be a working memory task. On the other hand, the more gamified tasks will be left for the last part of the session. 83.3% 4.00 (0.63) 3.34 – 4.66 2. Mobile Follow-up Application 2.1. General Aspects of Mobile Follow-up Application 2.1.1. Program Structure Since most experts mentioned that 30 minutes a day could be quite intense for the participants, it was decided to reduce the training frequency throughout the follow-up mobile application. Also in accordance with the feedback given, a directive training was chosen (based on follow-up consultations) twice a week. The psychologist will instruct the participant to dedicate 20 minutes, two times a week (total of 40 minutes) to complete the tasks. The psychologist will instruct the participant to do two specific tasks in each of these 20 minutes training sessions. 100% 4.50 (0.54) 3.93 – 5.07 2.1.2. Task Completion All tasks in the mobile application do not end immediately if the participant makes a mistake. There will be a set time for each task, so each task only ends when this time comes to an end. 100% 4.17 (0.41) 3.74 – 4.60 2.2. Description of the Mobile Follow-up Applications Tasks 2.2.1. Task 1.1. (before Task 2 ) Task 2 will no longer exist (so it will become part 2 of task 1. Task 2 will be automatically linked with task 1. That is, after task 1 is finished, task 2 automatically appears (which we will call task 1.1.) In this subtask, the patient must identify objects presented in the previous task (task 1), selecting “Yes” for the objects they remember. (To remind: In task 1, the participant selects objects and will have to memorize them. As more objects appear, he will have to continue to select objects that he has not previously selected.) 83,3% 4.17 (0.75) 3.38 – 4.96 2.2.2. Task 2 (before Task 3) Task 3 consisted of the following: Connect two points (a boat to a treasure) without falling into the existing traps. It has been changed and replaced by the following task (similar to one of the VR platform tasks that achieved a high consensus – greater than 90%): Remember the location of numbers in sequential order. Some numbered squares will appear in the scenario, and after a few seconds, the numbers in question will be hidden. The participant must select the squares that had numbers, following the order by which they appeared initially. The difficulty increases as more numbered squares appear and the subject will have to memorize a longer sequence. 100% 4.50 (0.55) 3.93 – 5.07 2.2.3. Task 4 (before Task 5) Based on the experts' suggestions, task 5 was reformulated as follows: Instead of the instruction indicating the objects that the subject will have to avoid, objects with a confusing characteristic will appear, for example, red square, red circle. On the other hand, instead of the task ending if the patient makes a mistake, the participant will be informed of the mistake but will be allowed to continue trying to find the correct rule until the time set for the task comes to an end. 100% 4.17 (0.41) 3.74 – 4.60 Table IV. Summary of Consensus in the two rounds Item Round 1 (n = 11) Round 2 (n = 6) Mean difference p Consensus, n (%) Mean (SD) 95% CI Consensus, n (%) Mean (SD) 95% CI 1. Virtual Reality (VR) Platform General Aspects of VR Platform Program Structure 72.7% 3.91 (0.70) 3.44 – 4.38 100% 4.67 (0.52) 4.12 – 5.21 -0.76 0.368 Session Zero 90.9% 4.73 (0.65) 4.29 – 5.16 - b - b - b — — Performance Sequence 72.7% 4.00 (0.77) 3.48 – 4.52 83.3% 4.33 (1.21) 3.06 – 5.60 -0.33 0.713 1.1. Description of VR Platform Tasks Task 1 72.7% 3.73 (1.01) 3.05 – 4.41 - b - b - b — — Task 1.1. (before Task 2) 72.7% 3.73 (0.79) 3.20 – 4.26 - b - b - b — — Task 2 (before Task 3) 72.7% 3.73 (0.79) 3.12 – 4.26 - b - b - b — — Task 3 (before Task 4) 90.9% 4.18 (0.60) 3.78 – 4.59 - b - b - b — — Task 4 (before Task 5) 90.9% 4.27 (0.65) 3.83 – 4.71 - b - b - b — — Task 5 (before Task 6) 81.8% 4.00 (0.89) 3.40 – 4.60 - b - b - b — — 1.2. Sessions of VR Platform Tasks Session 1 45.5% 3.45 (0.52) 3.10 – 3.81 - b - b - b — — Session 2 45.5% 3.45 (0.52) 3.10 – 3.81 - b - b - b — — Session 3 45.5% 3.45 (0.52) 3.10 – 3.81 - b - b - b — — Session 4 54.5% 3.55 (0.52) 3.19 – 3.90 - b - b - b — — Session 5 45.5% 3.45 (0.52) 3.10 – 3.81 - b - b - b — — Session 6 54.5% 3.55 (0.52) 3.19 – 3.90 - b - b - b — — Sequentially repeated sessions 54.5% 3.55 (0.52) 3.19 – 3.90 - b - b - b — — Sessions based on individual needs - a - a - a 100% 4.67 (0.52) 4.12 – 5.21 — — Order of the tasks - a - a - a 83.3% 4.00 (0.63) 3,34 – 4,66 — — 2. Mobile Follow-up Application 2.1. General Aspects of Mobile Follow-up Application Program Structure 81.8% 4.09 (1.22) 3.27 – 4.91 100% 4.50 (0.55) 3.93 – 5.07 -0.41 0.562 Task Completion 72.7% 3.73 (1.10) 2.99 – 4.47 100% 4.17 (0.41) 3.74 – 4.60 -0.44 0.368 2.2. Description of the Mobile Follow-up Applications Tasks Task 1 81.8% 3.82 (1.08) 3.09 – 4.54 - b - b - b — — Task 1.1. (before Task 2) 63.6% 3.55 (1.21) 2.73 – 4.36 83.3% 4.17 (0.75) 3.38 – 4.96 -0.62 0.713 Task 2 (before Task 3) 72.7% 3.91 (0.70) 3.44 – 4.38 100% 4.50 (0.55) 3.93 – 5.07 -0.59 0.368 Task 3 (before Task 4) 81.8% 3.91 (0.54) 3.55 – 4.27 - b - b - b — — Task 4 (before Task 5) 54.5% 3.82 (0.87) 3.23 – 4.41 100% 4.17 (0.41) 3.74 – 4.60 -0.35 0.220 a These items were not available in round 1. b These items were not available in round 2. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3869938","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":267599535,"identity":"d6ebbe06-605c-4487-95e6-02570e72e949","order_by":0,"name":"Tânia Caetano","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/klEQVRIiWNgGAWjYJCCAwwMzDC2DRAzNh7Ap5wHoQWsLQ2kpYGgFgYkLYdh9uIG9uxnDx748cdanr+9/+DngorzdmvbDwNtqbGJxmkLT17Cwd62dMMZZw4zS884czt525lEoJZjabkNOB2WY3CAt+FwAsONZAZp3rbbyWYHgFoYGw7j1sL/xuDgnz+HE+TvP2b+zdt2Ltns/EMCWiRyDA7zsB1OMLjBzAa05YCd2Q1Cttx4Y3BYFuiXjWeSzax5ziQnmN0A2pKAxy/s/TnGH98AQ0zu+MHHt3kq7OzNzqc/fPChxganFgyQCFaZQKxyELAnRfEoGAWjYBSMDAAA/x1jyQ1YN04AAAAASUVORK5CYII=","orcid":"","institution":"University of Coimbra, Sciences of University of Coimbra","correspondingAuthor":true,"prefix":"","firstName":"Tânia","middleName":"","lastName":"Caetano","suffix":""},{"id":267599536,"identity":"bce1957d-47e1-497e-be44-722f4f42da8c","order_by":1,"name":"Maria Salomé Pinho","email":"","orcid":"","institution":"University of Coimbra, Sciences of University of Coimbra","correspondingAuthor":false,"prefix":"","firstName":"Maria","middleName":"Salomé","lastName":"Pinho","suffix":""},{"id":267599537,"identity":"90508b10-4e7f-4cc2-880d-18d89c24fece","order_by":2,"name":"Hugo Freire","email":"","orcid":"","institution":"Merkle","correspondingAuthor":false,"prefix":"","firstName":"Hugo","middleName":"","lastName":"Freire","suffix":""},{"id":267599538,"identity":"2481148e-6160-439d-ac8d-aac75b4983f5","order_by":3,"name":"Dany Mota","email":"","orcid":"","institution":"Polytechnic of Leiria","correspondingAuthor":false,"prefix":"","firstName":"Dany","middleName":"","lastName":"Mota","suffix":""},{"id":267599539,"identity":"f463a3cd-37df-428a-8e90-15a10c597dd1","order_by":4,"name":"Eduardo Ramadas","email":"","orcid":"","institution":"VillaRamadas International Treatment Centre","correspondingAuthor":false,"prefix":"","firstName":"Eduardo","middleName":"","lastName":"Ramadas","suffix":""},{"id":267599540,"identity":"ef9d6551-80a4-441a-a8e6-e849d8a2b751","order_by":5,"name":"Jessica Lopes","email":"","orcid":"","institution":"VillaRamadas International Treatment Centre","correspondingAuthor":false,"prefix":"","firstName":"Jessica","middleName":"","lastName":"Lopes","suffix":""},{"id":267599541,"identity":"50a7b8b4-902e-4442-b18c-b25e68200441","order_by":6,"name":"Maris dos Anjos Dixe","email":"","orcid":"","institution":"Polytechnic of Leiria","correspondingAuthor":false,"prefix":"","firstName":"Maris","middleName":"dos Anjos","lastName":"Dixe","suffix":""}],"badges":[],"createdAt":"2024-01-16 13:29:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3869938/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3869938/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":49845753,"identity":"477b69cd-1977-4ce2-ae37-35ef6e72bd08","added_by":"auto","created_at":"2024-01-19 02:42:31","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":640094,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3869938/v1/3da6fa3c-666b-4832-b5a5-00e92d661da2.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"VRainSUD: Content validation of a Cognitive Training Program using the Delphi Method","fulltext":[{"header":"Introduction","content":"\u003cdiv id=\"Sec2\" class=\"Section2\"\u003e \u003ch2\u003eSubstance use disorders, brain changes and cognitive deficits\u003c/h2\u003e \u003cp\u003eThe global prevalence of substance use disorders (SUD) has increased significantly over the last decades (Degenhardt et al. \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). According to Our World in Data, it is estimated that around 1.4% (Ritchie and Roser \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2018a\u003c/span\u003e) and 0.9% (Ritchie and Roser \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2018b\u003c/span\u003e) of the world population have an alcohol or drug use disorder respectively. The disease burden of these disorders is high. The latest data reveal that each year more than 300.000 direct deaths can be attributed to SUD (Ritchie and Roser \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2018a\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003eb\u003c/span\u003e). With indirect deaths, the number more than doubles. In 2019 (Roser et al. \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2021\u003c/span\u003e), SUD had a disease burden of 35.13\u0026nbsp;million disability-adjusted life years (DALYs), higher than natural disasters, conflict and terrorism, and interpersonal violence (e.g., murder).\u003c/p\u003e \u003cp\u003eGiven its growing prevalence and significant disease burden, it is not surprising that treatment for SUD has been the object of considerable interest from the scientific community. However, relapse rates remain high with more than two thirds of individuals returning to substance use within one year of finishing treatment (Brandon et al. \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2007\u003c/span\u003e). The consistent relapse rates have been attributed to the chronicity of SUD, which itself is at least in part explained by neural changes that are shown to persist long after substance use stops.\u003c/p\u003e \u003cp\u003eThe development and maintenance of SUD seem to be mostly explained by three key neurocircuitry domains, which map onto the three functional stages of addiction (Koob and Volkow \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2016\u003c/span\u003e): the basal ganglia (binge/intoxication), the extended amygdala (withdrawal/negative affect), and the prefrontal cortex (preoccupation/anticipation). The basal ganglia are responsible for the rewarding effects of substance use as well as for the formation of habitual substance taking. The extended amygdala controls the feelings of unease, anxiety, and irritability that accompany withdrawal. Finally, the prefrontal cortex is implicated in executive function, namely the ability to make decisions and inhibit behavior (i.e., substance use).\u003c/p\u003e \u003cp\u003eAccording to Koob and Volkow (\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2016\u003c/span\u003e), the preoccupation and anticipation stage of addiction, explained by the changes in prefrontal cortex as well as in the insula and allocortex, could be a main element of relapse in humans. Imaging studies have found deficits in executive function associated with a decrease in frontal cortex activity that impacts working memory, decision-making, inhibitory control, and self-regulation (Volkow et al. \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e2011\u003c/span\u003e). Considering the influence that deficits in these cognitive domains seem to have on the relapsing nature of SUD, an effective intervention to mitigate them could be an important addition to treatment. A promising possibility is cognitive training.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eCognitive training programs for Substance Use Disorders\u003c/h2\u003e \u003cp\u003eThe effectiveness of cognitive training is still a controversial subject. Many studies (e.g., Sala and Gobet, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) have reported that effects resulting from cognitive training may be short-term or training specific, not translating into an improvement of general cognitive functioning in a given domain. However, some authors suggest that the lack of \u0026ldquo;far-transfer\u0026rdquo; effects may be due to a failure to broaden its concept as well as an almost exclusive reliance on primary outcomes (e.g., Brooks et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). Nonetheless, even amid the absence of consensus regarding its effectiveness, the state of the art indicates that cognitive training programs can lead to significant improvements in a number of populations, particularly in mild cognitive impairment (Brum et al. \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2009\u003c/span\u003e; Mendoza Laiz et al. \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Zhang et al. \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2019\u003c/span\u003e), parkinson\u0026rsquo;s disease (Leung et al. \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Gavelin et al. \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), traumatic brain injury (Hallock et al. \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2016\u003c/span\u003e; Vander Linden et al. \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2019\u003c/span\u003e), as well as in patients living with HIV (Wei et al. \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eWe have recently conducted a systematic review looking into the effective of cognitive training on memory, executive functioning and processing speed in individuals with SUD (Caetano et al. \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). From the 26 studies included in the review, 18 found some type of cognitive improvement, even though two of these saw only marginally significant effects. The majority of the included studies focused on computerized cognitive training programs and only five used a paper-and-pencil modality. Even though it is clear that these types of programs are increasingly using new technologies (e.g., smartphone applications), none of the studies included in the review utilized one of the most recent \u0026ndash; virtual reality (VR).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eVirtual reality interventions and cognitive training\u003c/h2\u003e \u003cp\u003eVirtual reality can be described as a simulated environment with scenes and objects that appear to be real and with which the user can interact as if they were present (Biocca \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1992\u003c/span\u003e). By providing an immersive and, at times, naturalistic environment, virtual reality seems to enhance motivation and engagement (Makransky et al. \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2019\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). The research into the use of virtual reality technology in healthcare interventions is still recent, but shows promising results.\u003c/p\u003e \u003cp\u003eIn a recent systematic review, Rowland et al., (\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) found that virtual reality was an efficacious treatment modality for emotional disorders, namely anxiety disorders and post-traumatic stress disorders. Another review looking specifically into the use of virtual reality in the treatment of addictive disorders (Segawa et al. \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), also reported positive results. These findings are supported by a systematic review of reviews conducted by Cieślik et al. (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), which provided evidence for the positive impact of VR interventions in psychiatric disorders. While these and other studies focused on psychological interventions, such as cognitive and behavioral therapy and exposure therapy (Bordnick et al. \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2011\u003c/span\u003e; Hone-Blanchet et al. \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2014\u003c/span\u003e; Segawa et al. \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Rowland et al. \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), there have been also studies interested in the use of VR for cognitive training in these populations.\u003c/p\u003e \u003cp\u003eJahn et al. (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) conducted a systematic review of randomized controlled trials studying fully immersive VR cognitive training in patients with neurological (e.g., mild cognitive impairment) and psychiatric disorders (e.g., schizophrenia). They found that cognitive training through VR led to significant improvements in some cognitive domains, specifically executive function, and attention. However, despite considering the results as promising preliminary evidence for the effectiveness of VR cognitive training interventions, the authors also highlighted the need for larger and higher quality studies. No study focusing on SUD or other addictive disorders was included in the review.\u003c/p\u003e \u003cp\u003eTo our knowledge, only two studies have looked into the effectiveness of a VR cognitive training intervention in individuals with SUD. Man (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) investigated the impact of a VR-based vocational training system in the cognitive performance, vocational outcomes, and work-related self-efficacy of young ketamine users. The results indicated that the VR group showed significant improvements in attention and memory, both maintained after 3 months. Participants from that group also presented a higher employment rate, even though the difference was not significant. Gamito et al. (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2021\u003c/span\u003e), studied the effect of the Systemic Lisbon Battery (SLB; Gamito et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2016\u003c/span\u003e), VR-based serious games platform, in individuals with alcohol use disorder undergoing inpatient treatment. Results indicated that the SLB positively impacted attention and cognitive flexibility.\u003c/p\u003e \u003cp\u003eEven though these studies categorize their interventions as cognitive training, both appear to be more focused on improving functioning on an everyday context, something more akin to cognitive rehabilitation. The presented example tasks mirror real-world activities, such as choosing clothes to wear, going to the supermarket or, in the VR-based vocational training system studied by Man (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2018\u003c/span\u003e), doing the work of a salesman on a clothing shop. They do not target specific cognitive domains as much as complex daily chores.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eVRainSUD\u003c/h3\u003e\n\u003cp\u003eVRainSUD is a cognitive training program (CTP) developed as an add-on intervention to be used with patients already receiving treatment for SUD, with the goal of mitigating cognitive deficits caused by long-term substance use and, hopefully, contribute to overall treatment success and a reduction of the relapse rates. It includes two modalities for two different stages of treatment: 1) a fully immersive VR platform with training exercises targeting specific cognitive domains, to be administered by psychologists or neuropsychologists (with training in the area of ​​cognitive training) to patients undergoing treatment; 2) a mobile follow-up application also with cognitive training exercises, for Android and iOS, to be used by the patients (who previously used the VR platform) at their discretion after the end of treatment.\u003c/p\u003e \u003cp\u003eThe VR platform aims to improve memory, executive functioning, and processing speed, while the follow-up application intends to maintain potential cognitive gains achieved through the VR platform during treatment.\u003c/p\u003e \u003cp\u003eThis CTP was created taking into account an exhaustive search of cognitive training programs and cognitive training tasks for psychiatric disorders in general and SUD in particular. Additionally, the clinical experience of the professionals working in the inpatient treatment center where the CTP, if proven effective, will be integrated into the treatment program was also considered. Their input was invaluable for the creation of a program to which the patients will adhere to and feel motivated to complete. For example, while both of the studies presented before (Man \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Gamito et al. \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) used ecologically sound tasks focused on daily activities (e.g., going to the supermarket), the professionals with years of experience with individuals with SUD who were questioned, before the creation of the CTP, do not consider that this type of task would be an added value and did not believe that this type of task would attract the population with SUD. As such, we opted for more attractive and gamified domain-specific tasks.\u003c/p\u003e \u003cp\u003eDespite the extensive research that was conducted, we believe that content validation is essential to arrive at the best possible program. To that end, our goal was to utilize the Delphi method to gather expert opinion on and validate the following dimensions of the CTP: 1) structure of the program (e.g., organization, number, and duration of sessions; type and number of tasks per session); 2) cognitive training tasks of the VR platform; 3) cognitive training tasks of the mobile follow-up application.\u003c/p\u003e"},{"header":"Methods","content":"\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eStudy Design\u003c/h2\u003e \u003cp\u003eThe Delphi technique is an established research method that involves rounds of survey questions sent out to a panel of experts on a given subject and that, according to H\u0026auml;der (2014, as cited in Niederberger \u0026amp; Spranger, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), can be used to achieve one or more of the following goals: 1) find consensus among experts; 2) aggregate ideas; 3) make future predictions; 4) determine experts\u0026rsquo; opinions. It is founded on the believe that the opinions of many outweigh those of a single individual, even if the individual is the foremost expert in their field.\u003c/p\u003e \u003cp\u003eIn the present study, an online platform was used to anonymously collect the survey responses from an international group of experts. Similarly to the original Delphi method (Hasson et al. \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2000\u003c/span\u003e), this study used a mixed-methods approach, with both rounds including Likert scales for quantitative analysis of consensus and free-text boxes to gather the experts\u0026rsquo; opinions and allow for the sharing of new ideas about how to improve the program.\u003c/p\u003e \u003cp\u003e All respondents gave their informed consent before participating in the study.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eParticipants\u003c/h2\u003e \u003cp\u003eThe participants were professionals with academic and/or practical experience in SUD and cognitive training.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eInclusion criteria\u003c/h2\u003e \u003cp\u003eThere is no agreed method to consider someone an expert in a given field. To facilitate the selection process, we used a simple point evaluation system. We included all professionals with a final evaluation of at least 2 points.\u003c/p\u003e \u003cp\u003eRegarding academic experience, we considered the following factors: At least one publication on the field of cognitive training (1 point) \u0026amp; at least one publication on the field of SUD (1 point) OR at least one publication on the field of cognitive training in SUD (2 points).\u003c/p\u003e \u003cp\u003eConcerning practical experience, we considered experience in the development of a cognitive training program (1 point). Moreover, we also considered if the professional had clinical experience in the application of cognitive training. However, we chose not to include this dimension in the quantitative analysis given how difficult it was to find the necessary information to confirm this type of experience.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eRecruitment\u003c/h2\u003e \u003cp\u003eA sample size of 8 to 15 respondents has been recommended for Delphi studies, given that smaller samples reduce reliability and bigger ones do not appear to add much value (Keeney et al. \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2010\u003c/span\u003e). For this study, we aimed to recruit 10\u0026ndash;12 participants.\u003c/p\u003e \u003cp\u003ePotential participants were identified through a literature review, existing academic contacts, and a web search for cognitive training programs (for SUD or other conditions) and the respective scientific teams. After being evaluated in accordance to the point system previously described, professionals who met the inclusion criteria (2 points or more) were invited to participate by email. In the initial questionnaire, they were asked to recommend other professionals who they considered to be experts in the field. Recommended professionals who met the inclusion criteria were contacted and invited to join the expert panel.\u003c/p\u003e \u003cp\u003eA total of 39 professionals were invited to participate in the expert panel.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eSurvey Design\u003c/h2\u003e \u003cp\u003eEven though classic Delphi studies use a qualitative approach in the first round, with the goal of allowing a broader understanding of the expert\u0026rsquo;s opinion on a given topic, this would not have been adequate for the current study.\u003c/p\u003e \u003cp\u003eAt the time of validation, VRainSUD was already on a beta phase and the main purpose was to get feedback on very specific characteristics such as the cognitive training tasks and the structure of the program. This could only be achieved through directed questions. As such, we used a modified study design that included both Liker-scale questions (from 1-strongly disagree to 5-strongly agree) and free-text boxes for qualitative feedback.\u003c/p\u003e \u003cp\u003eThe draft of the first-round survey was reviewed by several academics and clinicians involved in the project to different degrees. Taking into account their feedback, the following changes were made: 1) more background information on VRainSUD was added; 2) information on the cognitive training tasks was simplified and demonstrative GIFs were included; 3) the structure of the survey was redesigned in order to reduce its length and avoid repetition.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eProcedure\u003c/h2\u003e \u003cp\u003e The Research Ethics and Deontology Committee (CEDI) of the Faculty of Psychology and Educational Sciences of the University of Coimbra approved the research project in which this content validation study is integrated.\u003c/p\u003e \u003cp\u003eThe link for the first-round survey was sent to the participants by email, along with a brief explanation on how to access it and the planned deadline (1 month). The questionnaire was delivered using google forms, allowing the participants to complete it electronically while ensuring anonymity.\u003c/p\u003e \u003cp\u003eIn the first part of the questionnaire, participants were asked to give their electronic consent and provide some basic sociodemographic and professional information. We collected the participants\u0026rsquo; sex, age, educational background, and profession as well as country and city of work. Moreover, we also inquired about the participants\u0026rsquo; practical experience in cognitive training.\u003c/p\u003e \u003cp\u003eThe Likert-scale responses from the first round were analyzed to understand which questions had reached consensus and which needed to be addressed before being included in the second round. The responses from the free-text boxes were first transcribed verbatim and then subject to qualitative content analysis. They informed additional changes that could be made to improve the structure of the VRainSUD program.\u003c/p\u003e \u003cp\u003eThe link to the second-round survey was sent approximately two weeks after the first round closed. In the first part of this second questionnaire, a summary of the results from the first round was presented, along with additional information requested from some of the participants (e.g., regarding the target population). Participants were given two weeks to complete the second survey.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eAnalysis\u003c/h2\u003e \u003cp\u003eQuantitative analyses were conducted using IBM SPSS version 28.\u003c/p\u003e \u003cp\u003eA database was created with every sociodemographic and Likert-scale questions/items (e.g. \u0026ldquo;The program using the VR platform consists of 12 individual sessions, held 3 times a week, for 4 weeks, with a duration of 30 minutes per session.\u0026rdquo;). Categorical variables were encoded (e.g., 0-female and 1-male) and ordinal scale variables (Likert-scale questions/items) used the following correspondence: 1-strongly disagree; 2-disagree; 3 \u0026ndash; neither disagree nor agree; 4-agree; 5-strongly agree. Consensus in a given question/item was defined as at least 70% of responses being of agreement (4- agree or 5-strongly agree; (Sumsion \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e1998\u003c/span\u003e)). Descriptive statistics were used to characterize the sample and assess consensus on both rounds of the survey. The Mann-Whitney U Test was used to evaluate the significance of the differences between the means.\u003c/p\u003e \u003cp\u003eQualitative content analysis was conducted to assess the free-text box responses. The feedback from the experts was organized by categories (e.g., program structure, session zero, performance sequence, and others), facilitating its interpretation and the recognition of recurring themes or patterns. This organization also resulted in the structure of the second-round questionnaire.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003eEmail invitations to participate in the expert panel were sent to a total of 39 professionals. This included 35 who were identified by the research team and 4 who were recommended by one of the experts. Of these, 11 (28.2%) completed the first round. Six (15.4%) of those who responded to the first round also completed the second round.\u003c/p\u003e \u003cp\u003eOf the participants included in the first round (n\u0026thinsp;=\u0026thinsp;11), 63.6% were men (n\u0026thinsp;=\u0026thinsp;7) and 36.4% were women (n\u0026thinsp;=\u0026thinsp;4). The average age of participants is 45 years.\u003c/p\u003e \u003cp\u003eRegarding educational background, 3 indicated clinical neuropsychology (27.3%), 2 psychology (18.2%), 1 cognitive neuroscience and psychology (9.1%), 1 experimental psychology (9.1%), 1 clinical psychology (9.1%), 1 cognitive psychology (9.1%), 1 cognitive psychology and cognitive neuroscience (9.1%) and 1 smart healthcare (9.1%).\u003c/p\u003e \u003cp\u003eConcerning their profession, 2 reported being researchers and 9 professors. Of the 11 experts, only one responded that they did not have practical clinical experience in cognitive training (having only research experience). The average practical clinical experience of the 10 experts was 14.25 years.\u003c/p\u003e \u003cp\u003eRegarding the country where they worked, 2 indicated the United States of America (18.2%), 1 United Kingdom, 1 South Africa (9.1%), 1 Australia (9.1%), 1 Austria (9.1%), 1 Iran (9.1%), 1 Canada (9.1%), 1 India (9.1%), 1 Portugal (9.1%) and 1 Korea (9.1%). Finally, as for the city of work, 1 mentioned Liverpool (9.1%), 1 Cape Town (9.1%), 1 Bengaluru (9.1%), 1 Melbourne (9.1%), 1 Tehran (9.1%), 1 San Francisco (9.1%), 1 Kelowna (9.1%) and 1 Busan (9.1%). Two experts (19.2%) decided not to report their city of work.\u003c/p\u003e \u003cp\u003eSociodemographic data were not collected in the second-round questionnaire.\u003c/p\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eRound 1\u003c/h2\u003e \u003cp\u003eAn overview of the results from round 1 is presented in Table II.\u003c/p\u003e \u003cp\u003e \u003cb\u003eVR\u003c/b\u003e \u003cb\u003ePlatform\u003c/b\u003e\u003c/p\u003e \u003cp\u003eIn the round 1 questionnaire, the experts were presented with various general aspects of the VR platform, namely program structure, session zero and performance sequence, as well as the VR cognitive training tasks and a set of structured sessions (which from session 7 onwards would be administered sequentially to all patients). Several suggestions were taken into account following the 1st round questionnaire.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGeneral Aspects of the VR Platform.\u003c/b\u003e In the first round, regarding general aspects of the VR platform, the consensus was reached on all topics included, namely on the structure of the program (72.2% of consensus; the number of weeks, sessions per week, duration of the sessions), session zero (90.9% of consensus) and performance sequence (72.7% of consensus). Even though there was consensus on all these aspects of the platform, some experts suggested that more sessions should be done per week or that the duration of the intervention should be extended in order to increase the total hours of cognitive training (\u003cem\u003e\u0026ldquo;Consider higher dosage, i.e., daily cognitive training for 4 weeks or longer duration 8 weeks.\u0026rdquo;; \"It suggested a longer intervention in time, even if only with two weekly sessions.\u0026rdquo;\u003c/em\u003e). Still regarding the general aspects of the VR Platform, another expert suggested: \u003cem\u003e\u0026ldquo;I suggest to reduce the features that may induce stress on participants, particularly for the first sessions. For example time pressure, lack of positive feedbacks, graphical designs. Having sounds that induce a feel of achievement and enjoy is highly suggested (you might already have these feature).\u0026rdquo;\u003c/em\u003e\u003c/p\u003e \u003cp\u003e \u003cb\u003eVR Platform\u0026rsquo;s Cognitive Training Tasks.\u003c/b\u003e Consensus (more than 70% of the answers were Agree or Strongly Agree) was also reached on all the VR cognitive training tasks (described in Table I). One of the experts mentioned: \u0026ldquo;\u003cem\u003eI like the task design. Particularly the inhibitory control task and working memory tasks (I have more familiarity with lab-based versions of these tasks, so it's easier to see how they would be implemented in VR\u003c/em\u003e\u0026rdquo;. Another expert suggested: \u003cem\u003e\u0026ldquo;\u0026hellip;to have brief dynamic feedback for the Labyrinth navigation task, since the participant may proceed and then realize he/she might went wrong and the time is up\u0026hellip; Overall, the tasks seem amazing.\u0026rdquo;.\u003c/em\u003e\u003c/p\u003e \u003cp\u003e \u003cb\u003eCognitive Training Sessions with VR Platform.\u003c/b\u003e In the 1st round, the VR Platform sessions were presented in a structured way with a specific task sequence selected for each session. However, the consensus was not reached on the structure of the sessions and on the fact that the sessions would be repeated after the seventh. One of the experts suggested \u003cem\u003e\u0026ldquo;to put working memory related tasks as the first training task for each session, since it demands more cognitive energy and keep those tasks which are more gamified \u0026hellip; for the last part of the session\u0026rdquo;\u003c/em\u003e. Likewise, most experts suggested that sessions be personalized for each patient. One suggested \u003cem\u003e\u0026ldquo;consider performance-based training vs. everything sequentially, i.e., train in tasks that patient exhibits the worst performance at baseline\u0026rdquo;\u003c/em\u003e, while another considered it would \u003cem\u003e\u0026ldquo;be more appropriate an evolution considering the previous neuropsychological assessment and contingent on the performance of the tasks would\u0026rdquo;\u003c/em\u003e.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eMobile Follow-up Application\u003c/h2\u003e \u003cp\u003eIn the round 1 questionnaire, all information about the program structure, how to complete the tasks, as well as a description of all the tasks included in the mobile application was made available to the experts. Several suggestions from experts to improve the mobile application in question were also considered.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGeneral Aspects of the Mobile Follow-up Application.\u003c/b\u003e Although all general aspects of the follow-up mobile application obtained a consensus of more than 70%, some experts suggested some improvements and made some considerations. One of the experts mentioned: \u003cem\u003e\u0026ldquo;30 minutes a day can be quite intensive for participants; it would be nice if you could reduce time commitment while maintaining the same active ingredients. It would also be important to consider the context in which participants will train, e.g. supervised or unsupervised, \u0026hellip;\u0026rdquo;\u003c/em\u003e. Another commented that \u003cem\u003e\u0026ldquo;It would be interesting to have assessment after 3 month/6 months of interval to measure the true transfer effect of CTP on cognitive functioning\u0026rdquo;.\u003c/em\u003e Other suggestions for improvement from another expert were also taken into account: \u003cem\u003e\u0026ldquo;I think repeating the training tasks during follow-up could be less intense in terms of the frequency of sessions (i.e., two times per week).\u0026rdquo;\u003c/em\u003e\u003c/p\u003e \u003cp\u003e \u003cb\u003eMobile Follow-up Application\u0026rsquo;s Cognitive Training Tasks.\u003c/b\u003e Consensus (more than 70% of the answers were Agree or Agree Completely) was reached on the cognitive training tasks 1, 3 (which became task 2), and 4 (which became task 3). We did not reach a consensus on tasks 2 (which became task 1.1.) and 5 (which became task 4). For task 4 (before task 5) we considered the following suggestions for improvement: \u003cem\u003e\u0026ldquo;I suggest that instead of the instruction appear objects with the confounding characteristic, for example, red square, red circle. Instead of finishing the task if you make a mistake, I suggest displaying the information that you made a mistake and allowing you to keep trying until you get the rule right, a bit like in WIT or Wisconsin.\u0026rdquo;\u003c/em\u003e and \u003cem\u003e\u0026ldquo;Task 5 has many cognitive functions involved; (motor) inhibition may not very well targeted\u0026rdquo;.\u003c/em\u003e\u003c/p\u003e \u003cp\u003eA description of the follow-up mobile application cognitive training tasks is available in Table I.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003eRound 2\u003c/h2\u003e \u003cp\u003eAn overview of the results from round 2 is presented in Table III.\u003c/p\u003e \u003cp\u003e \u003cb\u003eVR\u003c/b\u003e \u003cb\u003ePlatform\u003c/b\u003e\u003c/p\u003e \u003cp\u003eIn the second-round questionnaire, all items that did not reach consensus in the first round were considered, as well as items that, although they had reached consensus, would benefit from and could be adjusted according to the experts\u0026rsquo; suggestions.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGeneral Aspects of the VR Platform.\u003c/b\u003e Regarding the VR platform, even though consensus had been reached in the first round on the structure of the program (72.7%) and on the performance sequence (72.7%), we chose to make changes based on the experts' suggestions. Therefore, these dimensions were once again evaluated in the second-round questionnaire. Based on the experts' suggestions, we extended the inpatient treatment duration from 6 to 8 weeks, which allowed us to increase the cognitive training program by another 2 weeks (6 weeks instead of 4). The first week of treatment will be used for the pre-test and establishment of a baseline, the cognitive training will happen between the second and the seventh weeks (inclusive), and the last week will be used for the post-test. Previously, the program had 6 distinct sessions that were repeated sequentially. Based on the experts' feedback, it was decided that the cognitive training will be personalized, and more focus will be given to the cognitive domains in which the participant presents the most difficulties at baseline. Consideration will also be given to prior neuropsychological assessment and task performance as the sessions progress. Since the tasks administered will be based on the participant\u0026rsquo;s individual needs, there will be no repetition of structured sessions (previously, from the 7th session, the patients repeated the same structured sessions that they had already performed in the first six training sessions). Other suggestions for improvement were also considered, namely regarding the order of the tasks within each session. The first task of every session will be a working memory task while the more gamified tasks will be left for the last part of the session.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003eMobile Follow-up Application\u003c/h2\u003e \u003cp\u003eFor the follow-up mobile application, the second-round questionnaire considered aspects that had not reached consensus among the experts, as well as other suggestions for improvement.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGeneral Aspects of the Mobile Follow-up Application.\u003c/b\u003e Even though the consensus was reached on task 2 (before task 3; 72.7%), on the program structure (81.8%), and on the task completion (72.7%), with the goal of improving the mobile application, some of the experts' suggestions were introduced. Since most experts mentioned that 30 minutes a day could be quite intense for the participants, the advised training frequency will be reduced. A directive training approach will be adopted with each participant being asked to complete 20 minutes of training, twice a week (based on the follow-up individual sessions). The psychologist will instruct the participant to do two specific tasks in each of these 20 minutes training sessions.\u003c/p\u003e \u003cp\u003e \u003cb\u003eMobile Follow-up Application\u0026rsquo;s Cognitive Training Tasks.\u003c/b\u003e Task 2 will no longer exist, becoming a subtask of task 1. After task 1 is finished, task 1.1. (previously task 2) will automatically appear. In this subtask, the patient must identify objects presented in the previous task (task 1), selecting \u0026ldquo;Yes\u0026rdquo; for the objects they remember. Even though consensus had been reached on task 2 (before task 3), considering some comments from the experts, this task was replaced with a modified (non-VR) version of the VR task targeting the same cognitive domain (which reached a consensus greater than 90%). Thus, task 2 became remembering and reproducing a number sequence. Some numbered squares appear in the scenario, and after a few seconds, the numbers in question are hidden. The participant must select the squares where the numbers were, following the order in which they appeared initially. The difficulty increases as more numbered squares appear and the subject will have to memorize a longer sequence. Based on the experts' suggestions, task 5 (now task 4) was reformulated. Thus, instead of the instruction indicating the objects that the subject will have to avoid, objects with a confusing characteristic will appear, for example, a red square, or red circle. Finally, it was decided that all tasks in the mobile application will have a set time instead of coming to an end when a participant makes a mistake. The participant will be informed of the mistake but will be allowed to continue trying to successfully complete the task until the established time comes to an end.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003eDifferences between rounds\u003c/h2\u003e \u003cp\u003eA summary of the changes made between the two rounds is presented in Table IV.\u003c/p\u003e \u003cp\u003eThe mean difference for each item repeated on both rounds was calculated by subtracting the mean of the first round from the mean of the second. Afterwards, the sample\u0026rsquo;s distribution was assessed using the Kolmogorov-Smirnova test and a lack of normality was verified. As such, the non-parametric test for independent samples, Mann-Whitney U Test was chosen to test the significance of the differences between the two rounds. The one-tailed test was selected in a specific direction because the N (=\u0026thinsp;11) of first round is greater than the N (=\u0026thinsp;6) of second round.\u003c/p\u003e \u003cp\u003eIn table IV, mean differences between the rounds are presented. No difference was statistically significant.\u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe main goal of this study was to validate VRainSUD, a cognitive training program developed to mitigate the cognitive deficits of individuals receiving treatment for SUD. The focus was on two major dimensions of the program: the structure (e.g., number, frequency, and duration of sessions; type and number of tasks per session) and the cognitive training tasks (of the VR platform and the mobile follow-up application).\u003c/p\u003e \u003cp\u003eWith the exception of two cognitive training tasks in the mobile follow-up application, both of these two dimensions reached the required level of consensus in the first round. Despite this, significant changes were made taking into account the feedback from the panel of experts.\u003c/p\u003e \u003cp\u003eInitially, VRainSUD was designed to follow a specific sequence of sessions with a pre-determined combination of cognitive training tasks, with the goal of giving similar training time to the various cognitive domains being targeted (i.e., memory, executive functions, processing speed). However, several experts indicated that a personalizable program would not only be more versatile but potentially also more effective. As such, the structure was altered, while allowing for the personalization of each session according to the baseline deficits as well as the progress of a given patient.\u003c/p\u003e \u003cp\u003eThe number of sessions itself was increased by maintaining their frequency (3 sessions per week) but extending the duration of the program. Considering that VRainSUD was developed with a specific setting in mind, this factor was conditioned by the characteristics of the inpatient treatment in question. However, we were able to increase the duration of the treatment from 6 to 8 weeks, allowing for 6 weeks of cognitive training (instead of the initial 4 weeks). Even though this change is significant, some experts considered that an even higher number of cognitive training hours could be beneficial. The optimal number of training hours will likely vary according to the baseline deficits and progression of each individual using the program. Future studies using VRainSUD without our current constraints (i.e., other settings), could investigate the relative effectiveness (and adherence) of different program durations.\u003c/p\u003e \u003cp\u003eIn comparison with the VR platform, the mobile follow-up application (e.g., structure and tasks) received more criticism and was subject to more changes. This could be in part explained by the fact that the mobile application was in an earlier stage of development at the moment of validation. Moreover, it is important to note that despite the increased technical difficulty of the VR platform, the mobile follow-up application may actually be more complex in order regards, given that it will be used without constant supervision. As such, many of the experts\u0026rsquo; suggestions were focused on not only improving the cognitive training tasks themselves but also increasing the patients\u0026rsquo; adherence to this part of the program. Both of these factors can have a significant impact on the program\u0026rsquo;s effectiveness.\u003c/p\u003e \u003cp\u003eRegarding the tasks, one was added to the previous one, another was replaced by a modified version of a VR training task, which had already received a high consensus and yet another was altered. Equally important, significant changes were made to improve adherence and motivation. The frequency and duration of suggested training sessions was reduced, and the training sessions will be incorporated into the individual sessions the patients will have twice a week with a clinical psychologist (part of an aftercare program offered to those who completed in-patient treatment). Once again, we should note that the structure of this part of the program is constrained by the already existing treatment in which it will be integrated. Future studies could experiment with additional structural changes, but a directive approach (versus self-management of training time) should always be preferred.\u003c/p\u003e \u003cp\u003eThe mean difference between the levels of consensus of the two rounds was greater for the structure of the VR platform and one of the tasks of the mobile follow-up application. This result is relevant since these are also the two dimensions that suffered the most changes following the feedback received in the first round. Despite this, the differences still weren\u0026rsquo;t statistically significant, which may be explained by the small sample size.\u003c/p\u003e \u003cp\u003eEven though the focus of the validation was on the structure and tasks of the CTP, the inclusion of free-text boxes on both rounds allowed the experts to comment on/suggest other, more specific, features. It was not possible to accommodate all the suggestions, but we did incorporate a majority of them.\u003c/p\u003e \u003cp\u003eStill looking to improve motivation and adherence, a positive feedback system was added so that a patient is validated (through sounds and visual stimuli) when they successfully complete a task and/or move on to another level of difficulty. A study by Burgers et al. (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2015\u003c/span\u003e) looking into the impact of feedback on motivation in a brain-training game (a type of cognitive training), reported that positive feedback promotes long-term motivation. In a more recent study with a mental arithmetic task, Peifer et al. (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) found that the relationship between positive feedback and flow and performance seems to be mediated by self-efficacy.\u003c/p\u003e \u003cp\u003eCertain changes were also made with the goal of preventing/reducing frustration. For the VR tasks, a patient will only move to another level of difficulty if they successfully complete the previous one (starting with a baseline level). Each task has many different exercises for each level. Finally, on the mobile follow-up application, a set duration time was added to all the tasks. When a patient makes a mistake, they receive feedback on this mistake but are allowed to continue training (versus having to start over from the beginning).\u003c/p\u003e \u003cp\u003eThe present study has some limitations that should be taken into account when interpreting the results. The sample of experts used was relatively small, especially considering that only around half of the original panel completed the second round. Even though too big samples are not useful when using the Delphi method, we do believe that it would have been beneficial to have a slightly higher number of experts and a better turnout rate at the second round. This limitation could be explained by the fact that the subject under validation is very specific and therefore, the pool of available experts is itself limited. Another important limitation to consider is the current state of the art on VR and its use for cognitive training in general, and in individuals with SUD in particular. Since VR and its use in health interventions is such a recent innovation, even highly qualified and experienced professionals do not have all the answers regarding the best way to utilize it in order to optimize the intervention\u0026rsquo;s effectiveness. As with every new technology or scientific development, it will take time and many more studies to truly understand and make use of all of VR\u0026rsquo;s potential.\u003c/p\u003e \u003cp\u003eWhile not widely used in the field of clinical or neuropsychology, the Delphi method was a valuable tool in the development of VRainSUD. It allowed not only for the validation of certain key dimensions of the program, such as its structure and cognitive training tasks, but also for its overall improvement. Moreover, the experts\u0026rsquo; feedback was essential for taking into account factors such as adherence, motivation, and engagement, which have their own impact on the program\u0026rsquo;s effectiveness. In view of its contribution, we consider that content validation using the Delphi method should be more often utilized when designing new cognitive or psychosocial interventions.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eConceptualization, T.C., M.P. and M.D.; methodology, T.C., M.P. and M.D.; investigation and formal analysis, T.C., E.R. and J.L.; resources, T.C., H.F. and D.M.; software, H.F. and D.M; writing \u0026ndash; original draft preparation, T.C.; writing \u0026ndash; review and editing, T.C., H.F. and J.L.; supervision, M.P. and M.D.; project administration, T.C.; validation, T.C. All authors contributed to manuscript revision, read, and approved the submitted version.\u003c/p\u003e\u003ch2\u003eAcknowledgment\u003c/h2\u003e \u003cp\u003eA special thank you to all the experts who dispended some of their time participating in the expert panel for this Delphi validation study. Their feedback was invaluable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBiocca F (1992) Communication Within Virtual Reality: Creating a Space for Research. J Commun 42:5\u0026ndash;22. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/j.1460-2466.1992.tb00810.x\u003c/span\u003e\u003cspan address=\"10.1111/j.1460-2466.1992.tb00810.x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBordnick PS, Carter BL, Traylor AC (2011) What virtual reality research in addictions can tell us about the future of obesity assessment and treatment. J Diabetes Sci Technol 5:265\u0026ndash;271. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/193229681100500210\u003c/span\u003e\u003cspan address=\"10.1177/193229681100500210\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrandon TH, Vidrine JI, Litvin EB (2007) Relapse and relapse prevention. Annu Rev Clin Psychol 3:257\u0026ndash;284. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1146/annurev.clinpsy.3.022806.091455\u003c/span\u003e\u003cspan address=\"10.1146/annurev.clinpsy.3.022806.091455\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrooks SJ, Mackenzie-Phelan R, Tully J, Schi\u0026ouml;th HB (2020) Review of the neural processes of working memory training: Controlling the impulse to throw the baby out with the bathwater. Front Psychiatry 11:1\u0026ndash;15. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fpsyt.2020.512761\u003c/span\u003e\u003cspan address=\"10.3389/fpsyt.2020.512761\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrum PS, Forlenza OV, Yassuda MS (2009) Cognitive training in older adults with Mild Cognitive Impairment: Impact on cognitive and functional performance. Dement Neuropsychol 3:124\u0026ndash;131. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1590/s1980-57642009dn30200010\u003c/span\u003e\u003cspan address=\"10.1590/s1980-57642009dn30200010\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBurgers C, Eden A, Van Engelenburg MD, Buningh S (2015) How feedback boosts motivation and play in a brain-training game. Comput Human Behav 48:94\u0026ndash;103. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.chb.2015.01.038\u003c/span\u003e\u003cspan address=\"10.1016/j.chb.2015.01.038\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCaetano T, Pinho MS, Ramadas E, et al (2021) Cognitive training effectiveness on memory, executive functioning, and processing speed in individuals with substance use disorders: A systematic review. Front Psychol 12:. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fpsyg.2021.730165\u003c/span\u003e\u003cspan address=\"10.3389/fpsyg.2021.730165\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCieślik B, Mazurek J, Rutkowski S, et al (2020) Virtual reality in psychiatric disorders: A systematic review of reviews. Complement Ther Med 52:. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.ctim.2020.102480\u003c/span\u003e\u003cspan address=\"10.1016/j.ctim.2020.102480\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDegenhardt L, Charlson F, Ferrari A, et al (2018) The global burden of disease attributable to alcohol and drug use in 195 countries and territories, 1990\u0026ndash;2016: a systematic analysis for the Global Burden of Disease Study 2016. The Lancet Psychiatry 5:987\u0026ndash;1012. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/S2215-0366(18)30337-7\u003c/span\u003e\u003cspan address=\"10.1016/S2215-0366(18)30337-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGamito P, Oliveira J, Brito R, et al (2016) Evaluation of cognitive functions through the systemic lisbon battery: Normative data. Methods Inf Med 55:93\u0026ndash;97. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3414/ME14-02-0021\u003c/span\u003e\u003cspan address=\"10.3414/ME14-02-0021\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGamito P, Oliveira J, Matias M, et al (2021) Virtual reality cognitive training among individuals with alcohol use disorder undergoing residential treatment: Pilot randomized controlled trial. J Med Internet Res 23:1\u0026ndash;10. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2196/18482\u003c/span\u003e\u003cspan address=\"10.2196/18482\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGavelin HM, Domell\u0026ouml;f ME, Leung I, et al (2022) Computerized cognitive training in Parkinson\u0026rsquo;s disease: A systematic review and meta-analysis. Ageing Res Rev 80:. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.arr.2022.101671\u003c/span\u003e\u003cspan address=\"10.1016/j.arr.2022.101671\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHallock H, Collins D, Lampit A, et al (2016) Cognitive training for post-acute traumatic brain injury: A systematic review and meta-analysis. Front Hum Neurosci 10:. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fnhum.2016.00537\u003c/span\u003e\u003cspan address=\"10.3389/fnhum.2016.00537\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHasson F, Keeney S, McKenna H (2000) Research guidelines for the Delphi survey technique. J Adv Nurs 32:1008\u0026ndash;1015. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x\u003c/span\u003e\u003cspan address=\"10.1046/j.1365-2648.2000.t01-1-01567.x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHone-Blanchet A, Wensing T, Fecteau S (2014) The use of virtual reality in craving assessment and cue-exposure therapy in substance use disorders. Front Hum Neurosci 8:1\u0026ndash;15. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fnhum.2014.00844\u003c/span\u003e\u003cspan address=\"10.3389/fnhum.2014.00844\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJahn FS, Skovbye M, Obenhausen K, et al (2021) Cognitive training with fully immersive virtual reality in patients with neurological and psychiatric disorders: A systematic review of randomized controlled trials. Psychiatry Res 300:113928. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.psychres.2021.113928\u003c/span\u003e\u003cspan address=\"10.1016/j.psychres.2021.113928\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKeeney S, Hasson F, Mckenna H (2010) The delphi technique in nursing and health research\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKoob GF, Volkow ND (2016) Neurobiology of addiction: a neurocircuitry analysis. The Lancet Psychiatry 3:760\u0026ndash;773. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/S2215-0366(16)00104-8\u003c/span\u003e\u003cspan address=\"10.1016/S2215-0366(16)00104-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeung IHK, Walton CC, Hallock H, et al (2015) Cognitive training in Parkinson disease: A systematic review and meta-analysis. Neurology 85:1843\u0026ndash;1851. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1212/WNL.0000000000002145\u003c/span\u003e\u003cspan address=\"10.1212/WNL.0000000000002145\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMakransky G, Andreasen NK, Baceviciute S, Mayer RE (2021) Immersive virtual reality increases liking but not learning with a science simulation and generative learning strategies promote learning in immersive virtual reality. J Educ Psychol 113:719\u0026ndash;735. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1037/edu0000473\u003c/span\u003e\u003cspan address=\"10.1037/edu0000473\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMakransky G, Terkildsen TS, Mayer RE (2019) Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learn Instr 60:225\u0026ndash;236. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.learninstruc.2017.12.007\u003c/span\u003e\u003cspan address=\"10.1016/j.learninstruc.2017.12.007\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMan DWK (2018) Virtual reality-based cognitive training for drug abusers: A randomised controlled trial. Neuropsychol Rehabil 8:1\u0026ndash;18. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1080/09602011.2018.1468271\u003c/span\u003e\u003cspan address=\"10.1080/09602011.2018.1468271\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMendoza Laiz N, Del Valle D\u0026iacute;az S, Rioja Collado N, et al (2018) Potential benefits of a cognitive training program in mild cognitive impairment (MCI). Restor Neurol Neurosci 36:207\u0026ndash;213. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3233/RNN-170754\u003c/span\u003e\u003cspan address=\"10.3233/RNN-170754\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNiederberger M, Spranger J (2020) Delphi Technique in Health Sciences: A Map. Front Public Heal 8:1\u0026ndash;10. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fpubh.2020.00457\u003c/span\u003e\u003cspan address=\"10.3389/fpubh.2020.00457\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePeifer C, Sch\u0026ouml;nfeld P, Wolters G, et al (2020) Well Done! Effects of Positive Feedback on Perceived Self-Efficacy, Flow and Performance in a Mental Arithmetic Task. Front Psychol 11:. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fpsyg.2020.01008\u003c/span\u003e\u003cspan address=\"10.3389/fpsyg.2020.01008\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRitchie H, Roser M (2018a) Alcohol Consumption. In: Our World Data. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ourworldindata.org/alcohol-consumption\u003c/span\u003e\u003cspan address=\"https://ourworldindata.org/alcohol-consumption\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed 10 May 2022\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRitchie H, Roser M (2018b) Opioids, cocaine, cannabis and illicit drugs. In: Our World Data\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRoser M, Ritchie H, Spooner F (2021) Burden of Disease. In: Our World Data. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ourworldindata.org/burden-of-disease\u003c/span\u003e\u003cspan address=\"https://ourworldindata.org/burden-of-disease\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed 20 May 2022\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRowland DP, Casey LM, Ganapathy A, et al (2022) Psychosocial Intervention Emotional Disorders. 31:1\u0026ndash;20\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSala G, Gobet F (2019) Cognitive training does not enhance general cognition. Trends Cogn Sci 23:9\u0026ndash;20. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.tics.2018.10.004\u003c/span\u003e\u003cspan address=\"10.1016/j.tics.2018.10.004\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSegawa T, Baudry T, Bourla A, et al (2020) Virtual Reality (VR) in Assessment and Treatment of Addictive Disorders: A Systematic Review. Front Neurosci 13:. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fnins.2019.01409\u003c/span\u003e\u003cspan address=\"10.3389/fnins.2019.01409\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSumsion T (1998) The Delphi technique: An adaptive research tool. Br J Occup Ther 61:153\u0026ndash;156\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVander Linden C, Verhelst H, Deschepper E, et al (2019) Cognitive training benefit depends on brain injury location in adolescents with traumatic brain injury: A pilot study. Eur J Phys Rehabil Med 55:585\u0026ndash;594. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.23736/S1973-9087.18.05548-X\u003c/span\u003e\u003cspan address=\"10.23736/S1973-9087.18.05548-X\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVolkow ND, Wang GJ, Fowler JS, et al (2011) Addiction: Beyond dopamine reward circuitry. Proc Natl Acad Sci U S A 108:15037\u0026ndash;15042. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1073/pnas.1010654108\u003c/span\u003e\u003cspan address=\"10.1073/pnas.1010654108\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWei J, Hou J, Mu T, et al (2022) Evaluation of Computerized Cognitive Training and Cognitive and Daily Function in Patients Living with HIV: A Meta-analysis. JAMA Netw Open 5:1\u0026ndash;17. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1001/jamanetworkopen.2022.0970\u003c/span\u003e\u003cspan address=\"10.1001/jamanetworkopen.2022.0970\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang H, Huntley J, Bhome R, et al (2019) Effect of computerised cognitive training on cognitive outcomes in mild cognitive impairment: A systematic review and meta-analysis. BMJ Open 9:. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1136/bmjopen-2018-027062\u003c/span\u003e\u003cspan address=\"10.1136/bmjopen-2018-027062\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTable I. Description of CTP tasks\u003c/p\u003e\n\u003ctable border=\"1\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\" width=\"934\"\u003e\n \u003cp\u003e\u003cstrong\u003e1. Virtual Reality (VR) Platform Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003e\u003cstrong\u003eTask\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003e\u003cstrong\u003eTask Name\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003e\u003cstrong\u003eCognitive domains\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003e\u003cstrong\u003eObjective\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003e\u003cstrong\u003eDescription\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eLabyrinth Navigation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eLogical/Deductive Reasoning, Planning, Processing Speed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eReach the labyrinth\u0026apos;s exit.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eDuring the time designated for the task in question, several labyrinths appear to the subject. He must use the provided auxiliary map to successfully navigate each labyrinth and reach the exit. The auxiliary map also allows the subject to visualize his current position.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 1.1. (before Task 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eRemembering Space\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eLong Term Memory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eSelect \u0026ldquo;YES\u0026rdquo; for the objects memorized in the task \u0026ldquo;Labyrinth Navigation\u0026rdquo; and \u0026ldquo;No\u0026rdquo; for the objects that have not appeared before.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eDuring the \u0026ldquo;Labyrinth Navigation\u0026rdquo; task, various objects will appear in the labyrinths, which the subject will have to memorize. In this task, the subject is presented with these and other objects that have not appeared before and should select \u0026quot;YES\u0026quot; only when he identifies objects that were present in the \u0026ldquo;Labyrinth Navigation\u0026rdquo; task.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 2 (before Task 3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eNumber Sequence\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eCognitive Flexibility, Logical/Deductive Reasoning, Processing Speed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eComplete number sequences.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eThe subject must understand the logic of the presented sequence and complete it with the numbers he has at his disposal.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 3 (before Task 4)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003ePassing Balls\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eCognitive Flexibility, Inhibitory Control, Processing Speed, Working Memory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eOnly hit the allowed balls.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eIn this task, the subject has at his disposal a racket, which will allow him to hit the balls that appear in the task scenario. Instructions regarding which balls should not be hit will alternate throughout the task. The instructions for the balls to avoid appear for a few seconds and then disappear, so that the subject must pay attention to the change and memorize it. The longer the subject manages to only hit the allowed balls, the more points he adds to the total score (the higher the total score).\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 4 (before Task 5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eCube Memory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eWorking Memory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eRemember the location of numbers in sequential order.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eIn the scenario, some numbered cubes will appear, which rotate after a few seconds, hiding the number in question. The subject must select the cubes that had numbers, following the order by which they appear.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 5 (before Task 6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eWorking Maze\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eWorking Memory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eReach the labyrinth\u0026apos;s portal (which can only be found if the subject follows all directions correctly).\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eAt first, the subject visualizes a sequence of directions (arrows like \u0026rarr; \u0026uarr; \u0026rarr; \u0026larr; ) on the floor of the maze. He will have to memorize them and navigate the maze according to them. When advancing through the maze, the subject can no longer see the directions.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\" width=\"934\"\u003e\n \u003cp\u003e\u003cstrong\u003e2. Mobile Follow-up Application Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003e\u003cstrong\u003eTask\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003e\u003cstrong\u003eTask Name\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003e\u003cstrong\u003eCognitive domains\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003e\u003cstrong\u003eObjective\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003e\u003cstrong\u003eDescription\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eRemember the Object\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eWorking Memory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eOnly select objects that have not been selected previously.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eThe subject will be selecting objects and will have to memorize them. As more objects appear, he will have to continue selecting only objects that have not been previously selected.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 1.1 (before Task 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eRemembering Space\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eLong Term Memory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eThe subject will have to indicate with \u0026ldquo;Yes\u0026rdquo; or \u0026ldquo;No\u0026rdquo; if he had seen the object before.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eIn this subtask, the patient must identify objects presented in the previous task (task 1).\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 2 (before Task 3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eName changed to Squares of Memory (before Treasure Hunt)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eWorking Memory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eRemember the location of numbers in sequential order.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eSome numbered squares will appear in the scenario, and after a few seconds, the numbers in question will be hidden. The participant must select the squares that had numbers, following the order by which they appeared initially.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 3 (before Task 4)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eNumber Sequence\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eCognitive Flexibility, Logical/Deductive Reasoning, Processing Speed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eComplete number sequences.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eThe subject must understand the logic of the presented sequence and complete it with the numbers he has at his disposal.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"69\"\u003e\n \u003cp\u003eTask 4 (before Task 5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"108\"\u003e\n \u003cp\u003eColors and Shapes\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"154\"\u003e\n \u003cp\u003eCognitive Flexibility, Inhibitory Control and Processing Speed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"189\"\u003e\n \u003cp\u003eSelect objects according to the instructions provided.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"414\"\u003e\n \u003cp\u003eThe subject should select only objects that meet the instructions provided. Objects with a confusing characteristic will appear, for example, red square, red circle. If the patient makes a mistake, the participant will be informed of the mistake but will be allowed to continue trying to find the correct rule until the time set for the task comes to an end.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable II. Appraisal of the Cognitive Training Program (CTP) in Round 1\u003c/p\u003e\n\u003ctable border=\"1\" width=\"101%\"\u003e\n \u003ctbody\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 70px;\" rowspan=\"2\" width=\"20%\"\u003e\n \u003cp\u003eItem\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 70px;\" rowspan=\"2\" width=\"44%\"\u003e\n \u003cp\u003eStatements\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" colspan=\"3\" width=\"35%\"\u003e\n \u003cp\u003eRound 1 (n = 11)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 35px;\" width=\"14%\"\u003e\n \u003cp\u003eConsensus, n (%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"10%\"\u003e\n \u003cp\u003eMean (SD)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"10%\"\u003e\n \u003cp\u003e95% CI\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35.4839px;\"\u003e\n \u003ctd style=\"height: 35.4839px;\" colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1. Virtual Reality (VR) Platform\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 35px;\" colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.1.\u003c/strong\u003e\u003cstrong\u003eGeneral Aspects of VR Platform\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.1.1.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Program Structure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eThe program using the VR platform consists of 12 individual sessions (held 3 times a week, for 4 weeks, with a duration of 30 minutes per session).\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.91 (0.70)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.44 \u0026ndash; 4.38\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 88px;\"\u003e\n \u003ctd style=\"height: 88px;\" width=\"20%\"\u003e\n \u003cp\u003e1.1.2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Session Zero\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 88px;\" width=\"44%\"\u003e\n \u003cp\u003eA session will be held before the 1st Session (Session zero), during which instructions will be provided for the use of VR glasses and the patients will be allowed to explore the platform and clarify any questions that may arise. The objective of this session is to ensure that all patients have an adequate mastery of the virtual environment before the start of the training sessions.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 88px;\" width=\"14%\"\u003e\n \u003cp\u003e90.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 88px;\" width=\"10%\"\u003e\n \u003cp\u003e4.73 (0.65)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 88px;\" width=\"10%\"\u003e\n \u003cp\u003e4.29 \u0026ndash; 5.16\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 61px;\"\u003e\n \u003ctd style=\"height: 61px;\" width=\"20%\"\u003e\n \u003cp\u003e1.1.3.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Performance Sequence\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"44%\"\u003e\n \u003cp\u003eAll participants will perform the same execution/performance sequence. The tasks must take place sequentially according to the pre-defined logic. In total, the VR platform comprises 6 distinct tasks.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"14%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"10%\"\u003e\n \u003cp\u003e4.00 (0.77)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"10%\"\u003e\n \u003cp\u003e3.48 \u0026ndash; 4.52\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 35px;\" colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.2.\u003c/strong\u003e\u003cstrong\u003eDescription of VR Platform Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 35px;\" width=\"20%\"\u003e\n \u003cp\u003e1.2.1.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"44%\"\u003e\n \u003cp\u003eLabyrinth Navigation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"14%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"10%\"\u003e\n \u003cp\u003e3.73 (1.01)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"10%\"\u003e\n \u003cp\u003e3.05 \u0026ndash; 4.41\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.2.2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 2 (changed to Task 1.1. an round 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eNumber Sequence\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.73 (0.79)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.20 \u0026ndash; 4.26\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.2.3.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 3 (changed to Task 2 an round 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eRemembering Space\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.73 (0.79)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.20 \u0026ndash; 4.26\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.2.4.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 4 (changed to Task 3 an round 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003ePassing Balls\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e90.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e4.18 (0.60)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.78 \u0026ndash; 4.59\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.2.5.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 5 (changed to Task 4 an round 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eCube Memory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e90.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e4.27 (0.65)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.84 \u0026ndash; 4.71\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.2.6.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 6 (changed to Task 5 an round 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eWorking Maze\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e81.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e4.00 (0.89)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.40 \u0026ndash; 4.60\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 35px;\" colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.3.\u003c/strong\u003e\u003cstrong\u003eSessions of VR Platform Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.3.1.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Session 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eSession 1 is composed of tasks 1 (Labyrinth Navigation, 10 min.), 2 (Number Sequence, 15 min.) and 3 (Remembering Space, 5 min.).\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e45.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.45 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.10 \u0026ndash; 3.81\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.3.2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Session 2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eSession 2 is composed of tasks 4 (Passing Balls, 15 min.), and 5 (Cube Memory, 15 min.).\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e45.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.45 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.10 \u0026ndash; 3.81\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.3.3.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Session 3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eSession 3 is composed of tasks 1 (Labyrinth Navigation, 10 min.), 4 (Passing Balls, 15 min.), and 3 (Remembering Space, 5 min.).\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e45.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.45 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.10 \u0026ndash; 3.81\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.3.4.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Session 4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eSession 4 is composed of tasks 6 (Working Maze, 15 min.) and 5 (Cube Memory, 15 min.).\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e54.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.55 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.19 \u0026ndash; 3.90\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.3.5.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Session 5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eSession 5 is composed of tasks 1 (Labyrinth Navigation, 10 min.), 6 (Working Maze, 15 min.), and 3 (Remembering Space, 5 min.).\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e45.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.45 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.10 \u0026ndash; 3.81\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.3.6.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Session 6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eSession 6 is composed of tasks 1 (Labyrinth Navigation, 10 min.), 4 (Passing Balls, 15 min.), and 3 (Remembering Space, 5 min.).\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e54.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.55 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.19 \u0026ndash; 3.90\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e1.3.7.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Sequentially repeated sessions\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eFrom the Session 7 on (regarding the VR platform) the sessions will repeat sequentially.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e54.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.55 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.19 \u0026ndash; 3.90\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 35px;\" colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eMobile Follow-up Application\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 35px;\" colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e2.1.\u003c/strong\u003e\u003cstrong\u003eGeneral Aspects of Mobile Follow-up Application\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 61px;\"\u003e\n \u003ctd style=\"height: 61px;\" width=\"20%\"\u003e\n \u003cp\u003e2.1.1.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Program Structure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"44%\"\u003e\n \u003cp\u003eIt is recommended that the subject uses the mobile follow-up application for approximately 30 minutes a day, three times a week, which makes 24 sessions over a period of 8 weeks.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"14%\"\u003e\n \u003cp\u003e81.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"10%\"\u003e\n \u003cp\u003e4.09 (1.22)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"10%\"\u003e\n \u003cp\u003e3.27 \u0026ndash; 4.91\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 61px;\"\u003e\n \u003ctd style=\"height: 61px;\" width=\"20%\"\u003e\n \u003cp\u003e2.1.2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task Completion\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"44%\"\u003e\n \u003cp\u003eThe mobile follow-up application consists of 5 tasks. In each 30-minute session, the subject is expected to alternate between the various tasks in order to provide training in the various cognitive domains.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"14%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"10%\"\u003e\n \u003cp\u003e3.73 (1.10)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 61px;\" width=\"10%\"\u003e\n \u003cp\u003e2.99 \u0026ndash; 4.47\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 35px;\" colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e2.2.\u003c/strong\u003e \u003cstrong\u003eDescription of the Mobile Follow-up Application Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 35px;\"\u003e\n \u003ctd style=\"height: 35px;\" width=\"20%\"\u003e\n \u003cp\u003e2.2.1.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"44%\"\u003e\n \u003cp\u003eRemember the Object\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"14%\"\u003e\n \u003cp\u003e81.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"10%\"\u003e\n \u003cp\u003e3.82 (1.08)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 35px;\" width=\"10%\"\u003e\n \u003cp\u003e3.09 \u0026ndash; 4.54\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e2.2.2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 2 (changed to Task 1.1. an round 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eRemembering Space\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e63.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.55 (1.21)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e2.73 \u0026ndash; 4.36\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e2.2.3.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 3 (changed to Task 2 an round 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eTreasure Hunt\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.91 (0.70)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.44 \u0026ndash; 4.38\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e2.2.4.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 4 (changed to Task 3 an round 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eNumber Sequence\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e81.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.91 (0.54)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.55 \u0026ndash; 4.27\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr style=\"height: 48px;\"\u003e\n \u003ctd style=\"height: 48px;\" width=\"20%\"\u003e\n \u003cp\u003e2.2.5.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task 5 (changed to Task 4 an round 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"44%\"\u003e\n \u003cp\u003eColors and Shapes\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"14%\"\u003e\n \u003cp\u003e54.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.82 (0.87)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"height: 48px;\" width=\"10%\"\u003e\n \u003cp\u003e3.23 \u0026ndash; 4.41\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable III. Appraisal of the Cognitive Training Program (CTP) in Round 2\u003c/p\u003e\n\u003ctable border=\"1\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" width=\"15%\"\u003e\n \u003cp\u003eItem\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" width=\"54%\"\u003e\n \u003cp\u003eStatements\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"3\" width=\"30%\"\u003e\n \u003cp\u003eRound 1 (n = 6)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003eConsensus,\u003c/p\u003e\n \u003cp\u003en (%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003eMean (SD)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e95% CI\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1. Virtual Reality (VR) Platform\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.1.\u003c/strong\u003e\u003cstrong\u003eGeneral Aspects of VR Platform\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"15%\"\u003e\n \u003cp\u003e1.1.1.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Program Structure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"54%\"\u003e\n \u003cp\u003eInstead of 12 training sessions (3 times per week, for 4 weeks, 30 minutes each session), we changed to 18 training sessions (3 times per week, for 6 weeks, 30 minutes each session). The total number of sessions went from 12 to 18. Note: Taking into account the total duration of the inpatient treatment and the treatment as usual (TAU) in the center where the program will be implemented, which has other daily therapeutic activities, it is not feasible to increase the duration of the training sessions or their frequency. However, we managed to extend the inpatient treatment duration from 6 to 8 weeks, what allows us to increase the cognitive training program another 2 weeks (6 weeks instead of 4). The first week of treatment will be used for the pre-test, the cognitive training will happen between the second and the seventh weeks (inclusive) and, the last week will be used for the post-test.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.67 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.12 \u0026ndash; 5.21\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"15%\"\u003e\n \u003cp\u003e1.1.2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Performance Sequence\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"54%\"\u003e\n \u003cp\u003ePreviously, the program had 6 distinct sessions that repeated sequentially. Based on the experts\u0026apos; feedback, it was decided that the training will be personalized, and more focus will be given to the cognitive domains in which the participant presents the most difficulties at baseline. Consideration will also be given to prior neuropsychological assessment and task performance as the sessions progress.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e83.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.33 (1.21)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.06 \u0026ndash; 5.60\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eDescription of VR Platform Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.3.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eSessions of VR Platform Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" width=\"15%\"\u003e\n \u003cp\u003e1.3.1.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Sessions\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"54%\"\u003e\n \u003cp\u003eBased on the experts\u0026apos; suggestions, the tasks administered will be based on the participant\u0026rsquo;s individual needs, so there will no repetition of structured sessions (previously, from the 7th session, the patients repeated the same structured sessions that they had already performed in the first six training sessions).\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.67 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.12 \u0026ndash; 5.21\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"54%\"\u003e\n \u003cp\u003eAlthough the training will be personalized, other suggestions for improvement will also be considered, namely regarding the order of the tasks within the same sessions. The first task of every session will be a working memory task. On the other hand, the more gamified tasks will be left for the last part of the session.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e83.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.00 (0.63)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.34 \u0026ndash; 4.66\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; \u003cstrong\u003eMobile Follow-up Application\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e2.1.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eGeneral Aspects of Mobile Follow-up Application\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"15%\"\u003e\n \u003cp\u003e2.1.1.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Program Structure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"54%\"\u003e\n \u003cp\u003eSince most experts mentioned that 30 minutes a day could be quite intense for the participants, it was decided to reduce the training frequency throughout the follow-up mobile application. Also in accordance with the feedback given, a directive training was chosen (based on follow-up consultations) twice a week. The psychologist will instruct the participant to dedicate 20 minutes, two times a week (total of 40 minutes) to complete the tasks. The psychologist will instruct the participant to do two specific tasks in each of these 20 minutes training sessions.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.50 (0.54)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.93 \u0026ndash; 5.07\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"15%\"\u003e\n \u003cp\u003e2.1.2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; Task Completion\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"54%\"\u003e\n \u003cp\u003eAll tasks in the mobile application do not end immediately if the participant makes a mistake. There will be a set time for each task, so each task only ends when this time comes to an end.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.17 (0.41)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.74 \u0026ndash; 4.60\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e2.2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eDescription of the Mobile Follow-up Applications Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"15%\"\u003e\n \u003cp\u003e2.2.1. Task 1.1. (before Task 2 )\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"54%\"\u003e\n \u003cp\u003eTask 2 will no longer exist (so it will become part 2 of task 1. Task 2 will be automatically linked with task 1. That is, after task 1 is finished, task 2 automatically appears (which we will call task 1.1.) In this subtask, the patient must identify objects presented in the previous task (task 1), selecting \u0026ldquo;Yes\u0026rdquo; for the objects they remember. (To remind: In task 1, the participant selects objects and will have to memorize them. As more objects appear, he will have to continue to select objects that he has not previously selected.)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e83,3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.17 (0.75)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.38 \u0026ndash; 4.96\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"15%\"\u003e\n \u003cp\u003e2.2.2. Task 2 (before Task 3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"54%\"\u003e\n \u003cp\u003eTask 3 consisted of the following: Connect two points (a boat to a treasure) without falling into the existing traps. It has been changed and replaced by the following task (similar to one of the VR platform tasks that achieved a high consensus \u0026ndash; greater than 90%): Remember the location of numbers in sequential order. Some numbered squares will appear in the scenario, and after a few seconds, the numbers in question will be hidden. The participant must select the squares that had numbers, following the order by which they appeared initially. The difficulty increases as more numbered squares appear and the subject will have to memorize a longer sequence.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.50 (0.55)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.93 \u0026ndash; 5.07\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"15%\"\u003e\n \u003cp\u003e2.2.3. Task 4 (before Task 5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"54%\"\u003e\n \u003cp\u003eBased on the experts\u0026apos; suggestions, task 5 was reformulated as follows: Instead of the instruction indicating the objects that the subject will have to avoid, objects with a confusing characteristic will appear, for example, red square, red circle. On the other hand, instead of the task ending if the patient makes a mistake, the participant will be informed of the mistake but will be allowed to continue trying to find the correct rule until the time set for the task comes to an end.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.17 (0.41)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.74 \u0026ndash; 4.60\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable IV. Summary of Consensus in the two rounds\u003c/p\u003e\n\u003ctable border=\"1\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" width=\"24%\"\u003e\n \u003cp\u003eItem\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"3\" width=\"29%\"\u003e\n \u003cp\u003eRound 1 (n = 11)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"3\" width=\"29%\"\u003e\n \u003cp\u003eRound 2 (n = 6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" width=\"8%\"\u003e\n \u003cp\u003eMean difference\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" width=\"8%\"\u003e\n \u003cp\u003e\u003cem\u003ep\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003eConsensus, n (%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003eMean (SD)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e95% CI\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003eConsensus, n (%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003eMean (SD)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e95% CI\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"9\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1. Virtual Reality (VR) Platform\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"9\" width=\"100%\"\u003e\n \u003cp\u003e\u003cstrong\u003eGeneral Aspects of VR Platform\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eProgram Structure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.91 (0.70)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.44 \u0026ndash; 4.38\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.67 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.12 \u0026ndash; 5.21\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e-0.76\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e0.368\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eSession Zero\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e90.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.73 (0.65)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.29 \u0026ndash; 5.16\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003ePerformance Sequence\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.00 (0.77)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.48 \u0026ndash; 4.52\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e83.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.33 (1.21)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.06 \u0026ndash; 5.60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e-0.33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e0.713\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"7\" width=\"83%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.1.\u003c/strong\u003e\u003cstrong\u003eDescription of VR Platform Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.73 (1.01)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.05 \u0026ndash; 4.41\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 1.1. (before Task 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.73 (0.79)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.20 \u0026ndash; 4.26\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 2 (before Task 3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.73 (0.79)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.12 \u0026ndash; 4.26\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 3 (before Task 4)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e90.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.18 (0.60)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.78 \u0026ndash; 4.59\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 4 (before Task 5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e90.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.27 (0.65)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.83 \u0026ndash; 4.71\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 5 (before Task 6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e81.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.00 (0.89)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.40 \u0026ndash; 4.60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"7\" width=\"83%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.2.\u003c/strong\u003e\u003cstrong\u003eSessions of VR Platform Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eSession 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e45.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.45 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.10 \u0026ndash; 3.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eSession 2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e45.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.45 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.10 \u0026ndash; 3.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eSession 3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e45.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.45 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.10 \u0026ndash; 3.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eSession 4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e54.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.55 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.19 \u0026ndash; 3.90\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eSession 5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e45.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.45 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.10 \u0026ndash; 3.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eSession 6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e54.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.55 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.19 \u0026ndash; 3.90\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eSequentially repeated sessions\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e54.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.55 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.19 \u0026ndash; 3.90\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eSessions based on individual needs\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.67 (0.52)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.12 \u0026ndash; 5.21\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eOrder of the tasks\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e83.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.00 (0.63)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3,34 \u0026ndash; 4,66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"7\" width=\"83%\"\u003e\n \u003cp\u003e\u003cstrong\u003e2.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eMobile Follow-up Application\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"7\" width=\"83%\"\u003e\n \u003cp\u003e\u003cstrong\u003e2.1.\u003c/strong\u003e\u003cstrong\u003eGeneral Aspects of Mobile Follow-up Application\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eProgram Structure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e81.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e4.09 (1.22)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.27 \u0026ndash; 4.91\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.50 (0.55)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.93 \u0026ndash; 5.07\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e-0.41\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e0.562\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask Completion\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.73 (1.10)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e2.99 \u0026ndash; 4.47\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.17 (0.41)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.74 \u0026ndash; 4.60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e-0.44\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e0.368\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"7\" width=\"83%\"\u003e\n \u003cp\u003e\u003cstrong\u003e2.2.\u003c/strong\u003e\u003cstrong\u003eDescription of the Mobile Follow-up Applications Tasks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e81.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.82 (1.08)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.09 \u0026ndash; 4.54\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 1.1. (before Task 2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e63.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.55 (1.21)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e2.73 \u0026ndash; 4.36\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e83.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.17 (0.75)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.38 \u0026ndash; 4.96\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e-0.62\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e0.713\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 2 (before Task 3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e72.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.91 (0.70)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.44 \u0026ndash; 4.38\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.50 (0.55)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.93 \u0026ndash; 5.07\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e-0.59\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e0.368\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 3 (before Task 4)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e81.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.91 (0.54)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.55 \u0026ndash; 4.27\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e-\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24%\"\u003e\n \u003cp\u003eTask 4 (before Task 5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e54.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.82 (0.87)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.23 \u0026ndash; 4.41\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10%\"\u003e\n \u003cp\u003e4.17 (0.41)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9%\"\u003e\n \u003cp\u003e3.74 \u0026ndash; 4.60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e-0.35\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8%\"\u003e\n \u003cp\u003e0.220\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003csup\u003ea\u003c/sup\u003e These items were not available in round 1.\u003c/p\u003e\n\u003cp\u003e\u003csup\u003eb\u003c/sup\u003e These items were not available in round 2.\u003c/p\u003e\n"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Substance use disorders, cognitive training, virtual reality, VRainSUD, Delphi method, content validation","lastPublishedDoi":"10.21203/rs.3.rs-3869938/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3869938/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSubstance abuse has undeniable effects on a number of cognitive dimensions, namely memory and executive functioning. In turn, deficits in these dimensions have been associated with poorer treatment outcomes for Substance Use Disorders (SUD). The main goal of this study was to validate VRainSUD, a cognitive training program composed of a virtual reality platform and a mobile follow-up application that proposes to reduce cognitive deficits in individuals with SUD.\u003c/p\u003e \u003cp\u003eThe Delphi method was used to reach a consensus on each of the program\u0026rsquo;s characteristics (e.g., structure, cognitive training tasks). A panel of experts was invited to participate in the content validation by answering two rounds of questions (scale and free-text boxes) regarding the program. Changes were made between the rounds according to the results of the first round. The consensus was defined as at least 70% of the experts agreeing on the validity of a characteristic.\u003c/p\u003e \u003cp\u003eFrom 39 invited experts, 11 completed the first round, and 6 completed both rounds. The structure of the program reached a consensus on the first round, along with most tasks (with the exception of two mobile application tasks). Significant changes were made following the feedback received, namely turning the program personalizable, and adding positive feedback to the tasks in order to improve adherence and engagement.\u003c/p\u003e \u003cp\u003eThe Delphi method was a valuable tool to improve VRainSUD. The experts\u0026rsquo; opinions not only allowed for the validation of main features, but also informed important additional changes.\u003c/p\u003e","manuscriptTitle":"VRainSUD: Content validation of a Cognitive Training Program using the Delphi Method","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-19 02:34:23","doi":"10.21203/rs.3.rs-3869938/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4276f1c8-f343-49f3-9385-41f770ad76ed","owner":[],"postedDate":"January 19th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-06-12T09:23:19+00:00","versionOfRecord":[],"versionCreatedAt":"2024-01-19 02:34:23","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3869938","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3869938","identity":"rs-3869938","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}