Evaluating Human-Centred Design for Public Health A Case Study on Developing a Healthcare App with Refugee Communities.

Evaluating Human-Centred Design for Public Health A Case Study on Developing a Healthcare App with Refugee Communities. | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Evaluating Human-Centred Design for Public Health A Case Study on Developing a Healthcare App with Refugee Communities. Rebeccah Bartlett, Jacqueline Boyle, Jessica Simons Smith, Nadia Khan, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-226630/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 14 You are reading this latest preprint version Abstract Background: Australian women from migrant and refugee communities experience reduced access to sexual and reproductive healthcare. Human-centred design is an ethical and effective approach to developing health solutions with underserved populations that are more likely to experience significant disadvantage or social marginalisation. This study aimed to evaluate how well Shifra, a small Australian-based not-for-profit, applied human-centred design when developing a web-based application that delivers local, evidence-based and culturally relevant health information to its non-English speaking users. Methods: This study undertook a document review, survey and semi-structured interviews to evaluate how well Shifra was able to achieve its objectives using a human-centred design approach. Results: A co-design process successfully led to the development of a web-based health app for refugee and migrant women. This evaluation also yielded several important recommendations for improving Shifra’s human-centred design approach moving forward. Conclusions: Improving refugees’ access to sexual and reproductive health is complex and requires innovative and thoughtful problem solving. This evaluation of Shifra’s human-centred design approach provides a helpful and rigorous guide in reporting that may encourage other organisations undertaking human-centred design work to evaluate their own implementation. Keywords: human-centred design; design thinking; refugee health; evaluation Plain language summary: Australian women from non-English speaking migrant and refugee communities face reduced access to sexual and reproductive healthcare and many then go on to experience poor health outcomes as a result. There is an urgent need for new approach to improve access to healthcare for underserved communities, one that centres these women in the process of finding, developing and disseminating the solutions themselves. Human-centred design is an ethical and effective methodology to working with communities to develop these health solutions. This study aimed to evaluate how well Shifra, a small Australian-based not-for-profit focused on improving access to healthcare for refugees and new migrants, undertook human-centred design approach when developing a Smartphone app that deliver local, safe and culturally relevant health information to non-English speaking Australians. The authors interviewed refugees, health and social sector experts and computer programmers involved in creating Shifra to evaluate how well they used human-centred design to achieve its goals. This evaluation found that Shifra’s approach was successful whilst also highlighting several important recommendations for improving collaborative efforts with refugee communities. These findings could help other projects also seeking to undertake an authentic community co-design process. Health Economics & Outcomes Research Health Policy sexual and reproductive healthcare healthcare access refugee communities Figures Figure 1 Introduction During 2017, Australia became home to 16,757 refugees. 1 Women and children, who comprised 87% 1,2 of these new arrivals, face significant health challenges, including limited access to quality sexual and reproductive health (SRH) services. 3 , 4 Poor SRH care has intergenerational consequences, affecting health and psychosocial outcomes for both mothers and their children. 5 Conversely, access to quality SRH services improves a number of health outcomes in women including prevention and management of high-risk pregnancies, reduction in unplanned pregnancies and abortions, reduction in obstetric complications, decreased anaemia and improved nutrition for both mother and baby. 6 , 7 Despite these benefits, within Australia, women from migrant and refugee communities report less SRH awareness and experience reduced access to SRH-specific care as well as culturally-relevant support that could assist them to make evidence-based decisions about their own health and service utilisation. 2 A new approach to improve access to healthcare for underserved communities, particularly for women from refugee and migrant backgrounds, is needed, one that centres these women in both the process of finding, developing and disseminating the solutions themselves. Addressing public health problems through human-centred design (HCD) is an ethical and effective approach to developing solutions with underserved populations that are more likely to experience significant disadvantage or social marginalisation. 8 , 9 HCD utilises multidisciplinary teams to approach the problem-solving process through three distinct phases: Inspiration, Ideation and Implementation. 10 HCD utilises “techniques which communicate, interact, empathise, and stimulate the people involved, obtaining an understanding of their needs, desires, and experiences, which often transcends that which the people themselves actually realised”. 11 Human-centred design, design thinking, co-design, co-production and co-creation are all terms that are often used interchangeably despite having nuanced differences in application and outcome. Each of these approaches focus on addressing complex problems and designing solutions with the end user communities (i.e. beneficiaries). 12 , 13 Design thinking is a specific set of stages within the HCD approach which help to guide problem solving teams through the whole experience as it diverges and converges the Inspiration, Ideation and Implementation phases in an iterative manner. Co-design, short for collaborative design is the process of design thinking steps that includes generative research (i.e. learning from end users) and development design (i.e. creating solutions with end users). 14 This second, development design stage, is often termed co-production and together with co-design, these two stages form co-creation. Studies have demonstrated promise that using HCD when developing health interventions can improve health outcomes for diverse populations 15 – 17 ; that solutions developed using this approach result in increased uptake of services 18 ; produce higher quality products and interventions; and that these products and interventions increased beneficiary satisfaction. 19 However, a scoping review analysing 21 different studies for use of HCD in global health across various geographies and populations was unable to draw definitive conclusions about the effectiveness, because of the heterogeneity of implementation, application areas and contexts. 20 There remains a lack of understanding regarding how best to achieve and evaluate a successful HCD-driven solution. 20 Thus, there is a growing need to evaluate the implementation of programs that apply HCD principles. Implementation evaluations not only assess a program’s deliverables against intended goals but also identify the strengths and weaknesses of a the implementation process, informing replication and efforts to scale. 21 To date, there have been limited studies or reports detailing an implementation evaluation of an entire HCD project. Most studies address only one aspect of the design process such as planning, 18 , 22 prototyping 23 or assessing stakeholder engagement 24 but do not evaluate the entire approach. This study aimed to evaluate the HCD approach that , a small -based not-for-profit focused on improving access to healthcare for refugees and new migrants, undertook in developing a web-based application to deliver local, evidence-based and culturally relevant SRH information to its users. Context The web-app (herein simply referred to as an “app”), provides high quality, rights-based information on family planning, pregnancy and newborn health, sexuality and sexual health, as well as mental health, family violence and adolescent health. Written and video resources provide information on accessing health services in Australia and cover topics such as healthcare rights and responsibilities, accessing translating and interpreting services, public and private insurance, as well as clinic locations. 25 The app was originally designed for English and Arabic-speaking communities living in and is in the process of being translated into other languages. As is committed to working in partnership with refugee and migrant communities to create products that are both ethical and sustainable 25 , the team chose to apply a HCD approach, using design thinking methods to prototype and eventually develop its digital health intervention. Co-designers for the app included refugee end users, subject matter experts (SMEs) from different, partner organisations that focus on health for multicultural communities, user experience (UX) students and computer programmers. Local partnerships were integral to completing many of the design steps. In 2017, collaborated with an undergraduate UX class from the . A semester-long process to design a digital health solution to bridge the gap in refugees’ access to SRH services resulted in five prototypes. ’s founder () then selected two designs to combine and develop further in conjunction with the co-designers to incorporate end users’ values and cultural beliefs. 26 The team used a combination of design thinking approaches developed by IDEO, Stanford’s d-School and Mummah et al (2016). 10 , 23 IDEO’s Field Guide to Human Centred Design 10 and the Stanford d-School’s Process Guide influenced the development of the Empathise, Define, Ideate, Prototype and Test steps 27 however the external assessor renamed the Test step “Launch and Share”, to ensure appropriate dissemination of any product or early research findings as per Fig. 1 . 23 The details of activities in each design thinking step used in developing the app are described in Table 1 , below. Given the importance of evaluating public health interventions yet the lack of robust methodology surrounding those that are co-designed, the team planned for process evaluation to be undertaken regularly and as objectively as possible. Table 1 's Human-Centred Design Process Stage Activity Empathise & Define Empathy sessions between Arabic-speaking refugees, refugee advocates and healthcare workers took place with final year UX students from University throughout the first half of 2017. Researchers undertook CBPR activities and group surveys to explore barriers and enablers to accessing healthcare for women from these refugee backgrounds. Partner organisation, , referred four refugee end users and three others were recruited using snowball-sampling techniques. also assisted in connecting the CBPR researchers to women interested in helping the team their understanding of the healthcare journey of different women from within these communities in . Ideate & Design UX students designed a solution based upon the insights gained during these empathy sessions. These designs were ideated then iterated with end users and other key stakeholders over several sessions throughout University’s first semester in 2017. Prototype After selecting the winning design, ’s founder participated in a hackathon hosted , a not-for-profit company that connects business analysts, programmers and UX designers with social impact organisations for weekend long prototyping meetups. Several computer programmers worked on at the event and continued to develop the technological component of the app until the next hackathon five months later. The beta version of this app was developed, tested and iterated with Arabic-speaking refugees over the next 5 months. SMEs vetted health information and simplified content into plain language for accuracy and accessibility. After development of the initial prototype, a more advanced version was user tested with two different groups of co-designers. First, Arabic-speaking women tested the beta version of the app through a partnership with a local adult education program and neighbourhood house, located in ’s inner-city suburbs where one in four people are from migrant or refugee backgrounds. 28 Again, the app’s content was user tested with SMEs for accuracy and accessibility. Online SMEs from around the world completed functionality and basic content testing via a Qualtrics online survey software survey and a group of local SMEs then met in person to review and edit the health content, ensuring it was evidence-based before simplifying the information further into plain English prior to Arabic translation. The seven women who attended user testing sessions and the all SMEs were recruited using snowball-sampling techniques. Launch & Share launched the beta version of the app in August 2017 with an event attended by co-designers, supporters, and funders. Methods Evaluation Questions The evaluation was designed and conducted by an external assessor () to reduce bias focused on the following three questions: To what extent did complete all the steps of the design thinking process shown in Fig. 1 ? To what extent did the final app incorporate the contributions of all co-designers? To what extent were the co-designers satisfied with the process? The first question assisted ’s team to understand how faithful to design thinking principles the initial co-design sessions were. The answer to this question could help the team improve future co-design endeavours. The second question was intended to determine the extent of co-designer involvement, a hallmark of the HCD approach. Finally, third question would not only assess co-designer satisfaction but also inform decision-making about how to structure future design sessions. Ethics Approval University provided ethics approval prior to data collection (Project ID number 13811: Evaluating the process and product of ’s mHealth intervention). As the external evaluator was from The University of , the Institutional Review Board from also reviewed the evaluation plan and determined no additional ethics approval requirements for this project ( Study #18-1449). Data Collection Approach The primary data for the first question (completion of designing thinking steps) involved a thorough review of all of ’s organisational documents including student design reports, community based participatory research (CBPR) results, meeting notes from prototyping events, Qualtrics data from user testing, emails between staff and the computer programmers regarding app updates and requests, as well as launch event information. Since there is a notable lack of validated tools evaluate HCD projects a maturity rubric was designed to synthesize the findings from the document review. This rubric was developed through consultation with two experts in the field of implementation science (), two reproductive and indigenous health experts (), one HCD expert () and one participatory research experts (). Several iterations of feedback from the expert panel were used to improve the usability, completeness and level of detail of the rubric. While the rubric and design steps used (Fig. 1 ) appears linear, the design process is fluid and the steps listed did not necessarily occur in a stepwise fashion. The rubric assesses the level of completion of each step of the design thinking process on a scale from 0 (non-existent) to 3 (full completion) with a maximum score possible of 12 (Table 2 ). Table 2 Human-Centred Design Scoring Rubric Empathise and Define No Attempt: (0 points) End users were not engaged, nor were attempts made to understand their lived experience. Minimally Attempted: (1 point) End users were engaged through interviews and felt respected. Moderately Attempted: (2 points) End users were engaged, felt respected, and were compensated for their participation in co-design sessions. (Compensation may be financially via cash or gift cards, transportation costs to get to sessions, provision of childcare during sessions, or other means.) Empathy exercises were undertaken to understand lived experience of end users. Satisfied: (3 points) End users in co-design sessions were engaged, felt respected, were compensated and were representative of the whole target population. Empathy exercises were undertaken to understand the lived experience end users. Ideate and Design Not Attempted: (0 points) Solution was decided on ahead of time and did not emerge in response to what was learned from end users. Minimally Attempted: (1 point) Learnings from empathy exercises were compiled. Pre-determined solution was minimally modified in response to what was learned from end users. Moderately Attempted: (2 points) Learnings from empathy exercises were compiled, brainstorming solutions was done in teams, and end user insights guided design of solution. Group consensus was obtained on the problem to be addressed/solved. Satisfied: (3 points) Learnings from empathy exercises were compiled, brainstorming solutions sessions were done in teams, end user insights guided the creation of the solution, additional information was gathered from end users if necessary. Group consensus was obtained on the problem to be addressed/solved. Group consensus was obtained regarding which solutions to prototype solve the identified problem. Prototype Not Attempted: (0 points) No attempt was made to obtain feedback about prototypes or minimum viable product (MVP) from end users. Minimally Attempted: (1 point) Single prototype made. End user feedback solicited. Feedback not incorporated into final version of solution. Moderately Attempted: (2 points) Multiple iterations of prototypes and/or MVPs created. Prototypes or MVPs tested with end users. End user feedback incorporated into subsequent iterations. Satisfied: (3 points) Multiple iterations of prototypes and/or MVPs created and tested with end user population. End user feedback incorporated into subsequent iterations of solution. Solution validated with subject matter experts and/or existing literature. Launch and Share Not Attempted: (0 points) Product was launched. No user feedback obtained. No dissemination completed. Minimally Attempted: (1 point) Product launched. User testing completed to understand users’ experience and satisfaction. Moderately Attempted: (2 points) Product launched. User testing completed to understand users’ experience and satisfaction. User feedback incorporated into plans for future iterations. Process or product results shared with program staff. Satisfied: (3 points) Product launched. User testing completed to understand users’ experience and satisfaction. User feedback incorporated into plans for future iterations. Process or product results shared with program staff, co-designers, and wider community. Total Score (max score of 12): A survey (Table 3) and a semi-structured interview guide (Table 4) modelled on the IDEO Field Guide to Human Centred Design and mHealth evaluation guidelines 10,29,30 were created to answer the second and third evaluation questions as well as to obtain clarification and confirmation of the data obtained through the document review. Table 3 Participant Survey Feedback Form At the co-design session … Strongly Disagree Disagree Neither disagree nor agree Agree Strongly Agree N/A 1 The group work felt collaborative 2 My contribution was valued 3 People attended who do not usually have representation (i.e. people with refugee and migrant backgrounds, people with diverse sexual orientation, people experiencing health accessibility issues) 4 The voices of refugee and/or migrant end users were heard 5 The roles and responsibilities of my participation were clearly defined 0 0 3 6 3 0 6 I understood both the processes and the language used 7 My time participating was compensated appropriately 8 There was a commitment by staff to develop consensus on what the end product should include 0 0 3 2 6 1 9 All co-designers were kept informed of any changes 10 I felt respected by staff 11 I felt respected by all co-design partners 12 staff offered me an opportunity for skill development and capability building 13 staff offered me co-design training and resources 14 There were strategies to involve people with different communication needs 15 There was enough time to allow relationship building 1 2 3 2 4 0 16 staff made attempts to reduce any power imbalance (e.g. between health professionals and refugees) 17 I felt safe sharing my opinions 18 Refugee and/or migrant end users helped shape the common agenda 19 I would participate in another co-design session with staff 20 I would encourage others to participate in a co-design session with staff 21 I see my ideas and contributions reflected in the final website/app 0 0 2 4 5 1 Comments/Feedback: These questions explored end user representation, co-designers’ understanding of the co-design sessions and design thinking methods used, issues around communication (i.e. language barriers, role clarification and understanding HCD goals), and co-designers’ levels of satisfaction, using a Likert scale (Table 2 ) regarding their involvement in the process. Table 4 Semi-Structured Interview Guide 1 Could you elaborate on how you felt your contribution to the co-design sessions was (or was not) recognised and valued? 2 Were there stakeholders who were not represented at the co-design sessions? If so, who else should have been invited? 3 Was there any confusion regarding your role, the processes or any language barriers at the co-design sessions that could have been made simpler/easier to understand? If so, what were these? 4 Were there ways that the team or other participants made you feel respected or disrespected/unwelcome during this co-design experience? 5 Do you feel that any power imbalances between co-design participants were addressed e.g. do you feel that your opinion mattered and that it was safe for you to communicate your thoughts and experiences if you wanted to? 6 How could this co-design experience be improved in the future? 7 Do you have any other feedback you wish to share? Survey and Interview procedures The external evaluator () engaged three groups of co-designers to complete surveys and semi-structured interviews. This included four refugee end users, three UX designers or computer programmers (UX/programmers) and six SMEs (including one funding representative) totalling 13 respondents. All SMEs and UX/programmers were fluent in English even if it was not their native language. End users considered themselves “conversational-level” English speakers. 10 of the respondents identified as women. All co-designers had at least a bachelor’s-level education. compensated end users for their time participating in the evaluation through store-bought gift cards. The co-designers who participated in the evaluation represented the larger co-design groups involved in creating the app in relation to gender identity, education level, and English proficiency. Interviews with each participating co-designer were consented to in advance and then again in person when they were completed at a location of the co-designer’s choosing. All Arabic-speaking end users declined the use of an interpreter. The survey was administered before the interview, with three exceptions: one SME declined to answer the survey, and two phone interviewees completed the surveys after the interview. The surveys were analysed using Qualtrics online survey software. Interviews were recorded and transcribed. Transcripts were de-identified, coded inductively using a hierarchical framework using NVivo 12 software by the external evaluator and a research assistant (), herself a daughter of immigrants, who requested to work on this project because of its migrant women’s health focus. Results Successes scored 9 out of 12 for fidelity to the design thinking process with a notable need for improvement around the ideation stage (Table 5). Table 5: HCD Evaluation Score Empathise and Define Comments Points Moderately Attempted: (2 points) End users were engaged, felt respected, and were compensated for their participation in co-design sessions. (Compensation may be financially via cash or gift cards, transportation costs to get to sessions, provision of childcare during sessions, or other means.) Empathy exercises were undertaken to understand lived experience of end users. · The organisational documents provided evidence of compensation, end user interviews and ethnographic work, CBPR and empathy exercises but there was no documentation regarding whether co-designers felt respected or recognised for their contribution. Instead, this information came from the surveys. · All co-designers selected ‘agree’ or ‘strongly agree’ to feeling respected and if they would encourage others to participate in a co-design session with the team. The representativeness of the end user co-designers was also not apparent in any of the organisational documents but emerged as a theme during the interviews. · Co-designers’ views on how representative the end users were varied by the type of co-designer group the participant came from. · SMEs and UX/programmers generally agreed that there was enough end user representation at the co-design sessions. As one computer programmer said: “We focused bringing on more and more people from the refugee and migrant community which is really good … I don’t think we had a shortage of that diversity … in terms of cultural background it was quite well represented.” The end users themselves felt that there were groups within the Arabic-speaking population that were unrepresented. Some of the suggested groups include individuals who did not attend university, Arabic speakers with no or low English proficiency, middle and late middle age individuals, people with different levels of proficiency with mobile technology, and refugees who had just arrived to Australia compared to refugees who have been living in Australia for some time. A more representative end user population would have accrued a higher score. 2 Ideate and Design Points Minimally Attempted: (1 point) Learnings from empathy exercises were compiled. Pre-determined solution was minimally modified in response to what was learned from end users. An extensive document review verified that the team did not have a pre-conceived idea of what the prototype would be, and that the app’s features and structures arose in response to insights gained from the end user co-designers. The ideation stage took place within the design student teams and was tested at intervals with end users. Surveys and interviews with the co-designers revealed that one quarter of all survey participants marked ‘neither agree nor disagree’ to whether confirmed group consensus for either the problem statement or the solutions to be prototyped. The remaining three quarters indicated they ‘agree’ or ‘strongly agree’ that did in fact, achieve this. Whilst the issue of consensus did not emerge as a theme across all interviews, one computer programmer articulated the problem this way: “I feel like sometimes there was a bit of disconnect between what [ staff] wanted and maybe necessarily what the refugees wanted … they wanted a resource where they could find health information, locations specifically of hospitals, GPs, pharmacies … the actual health information, they would rather go directly to the source, rather than … reading it online.” Given the conflicting information received from the surveys and the interviews and the lack of documentation, the evaluator reported finding difficulty scoring in this area. A lower score of 1 out of 3 was given to draw attention to this issue in the future. More thorough record keeping during this design stage will shed light on this process and the methods used in future HCD projects. 1 Prototype Points Satisfied: (3 points) Multiple iterations of prototypes and/or MVPs created and tested with end user population. End user feedback incorporated into subsequent iterations of solution. Solution validated with subject matter experts and/or existing literature. End users from different backgrounds tested and contributed to iterations of the app on multiple occasions and continue to do so to this day. SME co-designers and their feedback lead to tangible changes in the app appearance, language accessibility and functionality. Organisational documents alone verified these requirements received 3 out of 3 in this stage. 3 Launch and Share Points Satisfied: (3 points) Product launched. User testing completed to understand users’ experience and satisfaction. User feedback incorporated into plans for future iterations. Process or product results shared with program staff, co-designers, and wider community. held a product release in August 2017 which included co-designers, partners, and funders and presented preliminary findings from the CBPR projects. received 3 out of 3 in this stage. 3 Total Score (max score of 12): 9 Feedback obtained through both the interviews and surveys verified that the team did complete all the steps of the design thinking approach. All survey respondents selected ‘agree’ or ‘strongly agree’ to statements assessing the collaborative nature of the group work and that they felt safe sharing their opinions (Table 3). Over 90% reported that they would participate in another co-design session and that they would recommend participation to a friend or family member. In the interviews, a feeling of enjoyment from participating in the co-design sessions clearly emerged. All 13 evaluation participants reported feeling valued, appreciated, and/or respected during the co-design sessions. When we went to … the students, they were very enthusiastic. They take our notes and they try to discuss with us … And as users … they take our notes and they try to improve. And we share, really. We share as a big group, as a teamwork. And we share our ideas together. The surveys and interviews revealed important learnings for the team when using HCD within this particular context. Communication First was the issue of communication. Three of the four end users mentioned that language was a barrier, despite assistance from other community members who acted as interpreters during the co-design sessions and their own self-assessment as being proficient in English. This likely exacerbated the second area for improvement: role clarification. This was a need reported by co-designers from all groups. An end user expressed how she came to understand her role in the co-design session: After some time, I could realise what’s going on, and understand what I had to do. It wasn’t clear in the beginning. Like when I went there, I didn’t know why I’m going there. I just know that I want to be part of this, this is what I really wanted to do, and yeah, after some time I could understand what’s going on, but nobody explained me how. Fragmentation of involvement There was fragmentation of co-designers’ experiences during the sessions. Many spoke about not understanding the project’s entire process, wanting to be more involved but not receiving further invitations, or the need to build on previous sessions with end users. While end users were involved at every step of the co-design, different individuals participated at different points and in different ways. Very few end user co-designers were a part of the process from beginning to end. This led to feelings of disconnection and confusion, as one end user put it: First, I was really interested, but after some time when I found like nobody’s calling you back, so I said no I don’t want to waste my time on this. But it’s something really helpful and I really like the idea of helping new arrivals from refugee and migrant backgrounds. SMEs who facilitated meetings between staff and end users also felt the desire to be more involved throughout the entire HCD project. One SME commented: I think it would be good to have a follow up consultation on working on actual usage … because we haven’t sort of touched base again with those women to say, have you used it? … it was almost still in the design stage, and things hadn’t quite been finished. So, I think we could have a follow up that says, this is the latest version of the product, let’s have a play around with it, what do you think now? I think that would be really timely. A third area for consideration in future co-design session is the importance of diversity and representation within end-user groups. The SMEs and UX/programmer generally agreed that there was sufficient end user representation at the co-design sessions: We focused bringing on more and more people from the refugee and migrant community which is really good…I don’t think we had a shortage of that diversity … in terms of cultural background it was quite well represented. The end users themselves, however, felt that there were groups within the Arabic-speaking population that were unrepresented. Some of the suggested groups include individuals who did not attend university, Arabic speakers with no or low English proficiency, middle and late middle age individuals, people with different levels of proficiency with mobile technology, and refugees who had just arrived to Australia compared to refugees who have been living in Australia for some time. The risks of only collaborating with end users who are university-educated, recently resettled and English-proficient were summarised by one end user: Maybe we will use this program [] and maybe never we will use this program because we can search … what we need by Google … but that program [], it’s good for different level of the people. Increased representation from within the end-user group should be considered for future HCD endeavours. Finally, a survey question assessed whether co-designers felt that there was enough time for relationship building during the co-design sessions. End users responded most negatively to the statement, with half stating that they ‘strongly disagree’ or ‘disagree’ that there was enough time for this. Discussion This evaluation found that a co-design process was successfully applied to the development of a web-based app for refugee and migrant women in reproductive health. This evaluation also yielded several important recommendations for improving ’s HCD approach moving forward, findings that can be applied to other projects seeking to undertake an authentic community co-design process. First, with so many people of diverse backgrounds contributing to the project, clear communication about roles and expectations is critical. More attention to facilitator training and identification of session goals and following up with consistent communication and seeking end user and SME feedback would help to reduce future confusion. 10 Second, it is important to set realistic expectations and role clarifications with co-designers. Evaluation participants frequently mentioned their role and the purpose of the project as a whole. 27 Design is a non-linear and creative process, which can inadvertently contribute to confusion about the co-designer’s purpose and the project’s goals. 31 Care should be taken in advance to explain this and answer questions from participants not familiar with the concept. Third, it is important not to view all end users as interchangeable. 10 UX/programmers and some SMEs saw refugees at all the meetings and viewed that as enough end user participation. End users however, felt there were other voices from their community that needed engagement. For example, the fact that all the end users spoke some English meant that co-design sessions could proceed without certified interpreters, but it also meant that the voices and experiences of refugees with low English proficiency were missing. There will naturally be trade-offs in any public health project since limited resources are an unchanging reality, but several process adjustments could address this issue. This evaluation demonstrates that the practice of engaging end users across all empathy, design, and prototyping stages and into product development is possible. 9 31 Collaborating with multiple migrant and refugee advocate organisations to use diverse sampling techniques will help to engage a more representative sample in the future co-design sessions. Finally, by setting aside adequate time to develop collaborative relationships amongst all co-design groups the HCD process is an opportunity to give power and control back to the end user population for whom one is designing the health intervention. 32 Placing greater effort into building relationships as a part of the co-design session is especially important with ’s target refugee population. 25 When properly implemented, the intent of HCD is to provide public health organisations a pathway to sharing (and where applicable, handing over) power in order to achieve true citizen participation and control. 24 Failure to apply HCD principles in an authentic or purposeful way usually results in tokenism, and development of solutions that are unsustainable. 32 It can also “exacerbate social exclusion and destroy trust systems” when done poorly. 33 A project cannot utilise HCD without a power dynamic shift that ensures the end user, not the UX designer, computer programmer or community organisation, is in the primary decision-making role. 11 Future Considerations After reviewing all three data sources, there are several important considerations when planning for evaluation of any HCD-driven projects. First, while this version of the rubric was helpful in gaining a deeper understanding of ’s HCD approach, several iterations will increase the tool’s usefulness moving forward. Including operational definitions of each of the three stages would clarify the expectations, especially for anyone who is trying to use the rubric to guide future HCD-driven projects. Additionally, some of the requirements were impossible to assess using existing documentation alone. For example, the evaluator was unable to ascertain whether end users felt respected from organisational documents alone and all three data sources were required to complete the rubric. Co-designers’ perceptions of whether they felt respected and valued during co-design sessions should be included all future feedback forms, surveys and interview guides. Despite this, using multiple data sources (document review, surveys, and interviews) should have helped to verify answers. Instead, each data source provided unique and isolated findings and there proved inconsistency between interviews and the survey responses concerning communication. For example, while the majority (75%) of co-designers responded ‘agree’ or ‘strongly agree’ to the survey statement about whether roles and responsibilities had been clearly explained, many anecdotes arose during the interviews around the fact that co-designers did not actually understand their role in the overall project or at specific co-design sessions. Similarly, one of the main themes of the semi-structured interviews was how fragmented the co-designers felt their involvement was though there was no way to verify this finding in the document review or with the surveys. One way to address this problem is by considering evaluation methods alongside HCD planning meetings. Collecting co-designer surveys throughout the project can help understand their experiences ‘in the moment’ and offers an opportunity for a more agile response if needed, whilst also offering an opportunity to compare experiences and feedback later following end of project co-designer evaluations. did not have any documentation regarding training materials on design thinking facilitation or how facilitators had introduced the co-designers to various methodologies at different points in the app’s development, making difficult to verify these findings. Future documentation on roles, design thinking goals and checking in with co-designers frequently would benefit all involved. Strengths and Limitations This paper adds to the existing literature regarding the rigorous use of HCD in public health. This implementation evaluation provides an important guide to future knowledge about purposefully working with end user communities to design better health interventions. Engaging in evaluation work increases the transparency of organisations and helps them demonstrate their commitment to the HCD ethos. This paper and the rubric are helpful tools for organisations attempted to evaluate their use of design thinking methods, and can assist them to plan, prepare for and execute co-design successful sessions. Additionally, the evaluation included multiple sources of data including surveys, interviews, and records. While ’s data collection methods will be improved as a result of this evaluation, multiple sources of data give a more full, rich, and accurate picture of co-designers’ experiences and the methods themselves. 34 The richness of the data obtained is especially important since HCD is nebulous by nature and there is a lack of demonstrated, rigorous evaluation. 20,35 Finally, the utilisation of an external assessment increased the objectivity of this evaluation. 36 There were several limitations to this evaluation study. First, the convenience sampling of co-designers introduced selection bias into the results. It is possible that the co-designers who did not respond to a request to participate in the evaluation would have provided different answers and perspectives. Second, the evaluation took place 9-16 months after most of the co-design sessions, making results vulnerable to recall bias. Several interviewees mentioned at different points that the sessions had happened so long ago it took effort to remember and answer the questions. Third, a yet to be validated rubric guided part of the evaluation. Despite expert feedback and iteration on this rubric before its application, there needs to be more use of the checklist to ensure that it is both reliable and generalisable. Fourth, within the research team, only the research assistant () had any personal connection to the migrant or refugee experience. Future research on migrant or refugee communities must include funding to support people from migrant and refugee backgrounds to play an active and ideally, lead, role in designing, conducting, analysing and reporting on said research. Finally, evaluating HCD, design thinking and co-design efforts in public health is challenging due to its abstract, creative, and iterative nature and because results are specific to the local context. There are no definitive guidelines providing specific parameters for assessing a HCD-driven project and these terminologies are frequently interchanged despite their differences in application and outcome. The methods identified in this paper are a first attempt to benchmark this innovative approach and will need to be refined in the future. Conclusion Improving refugees’ access to sexual and reproductive health is complex and multidimensional and requires innovative and thoughtful problem solving. HCD is one way to address complex problems in an ethical and effective way and it is how chose to approach the development of its solution to this problem among Arabic-speaking refugees in , Australia. The surveys and interviews revealed that end user, SME, and UX/programmer co-designers enjoyed participating in the co-design sessions, felt respected and welcomed, and saw their contributions reflected in the final product. Opportunities for growth include engaging a more diverse end user population and communicating expectations and results more clearly during and after the co-design sessions. A comprehensive process evaluation will benefit the field by providing an example of how to assess an organisation’s ability to follow all the HCD steps and will advance knowledge on the effectiveness of HCD in developing solutions that are aligned with the needs of the target audience. This evaluation of ’s HCD approach provides a helpful and rigorous guide in reporting that may encourage other organisations undertaking HCD work to evaluate their own implementation. Such organisations should explore in advance, how they plan to evaluate not only the design steps but also co-designers’ perceptions around their role and the contributions they made to the end product. Determining how to assess satisfaction with both process and product needs thoughtful consideration to ensure co-designers and evaluators are reflecting and measuring the same outcome respectively. Finally, utilising mixed methodologies has the potential to reveal inconsistent answers across the different sources of data being examined so care needs to be taken to ensure evaluative and probing questions in the semi-structured interview process add clarity and reduce confusion. Declarations Ethics approval and consent to participate Project ID number 13811: Evaluating the process and product of Shifra’s mHealth intervention Consent for publication All authors consent to publication Availability of data and materials Upon request Competing interests Conflict of Interest: First author (RB) founded the Shifra app, however an external evaluator (JSS) was used to limit bias in evaluating Shifra’s HCD process otherwise no authors report competing interests Funding Not applicable Authors' contributions JSS undertook the initial research NK assisted with some data assistance RB draft this paper based on the research and managed all edits and version control JB and TR reviewed the paper RR provided supervisory guidance Acknowledgements The authors recognise , and for their contribution to the evaluation plan and rubric design of this paper. Authors' information (optional) Not Applicable References Historical Settlement Reports. (2018) In: Australian Government; 2018. Sexual and Reproductive Health Data Report. Multicultural Centre for Women's Health;2016. Austin J, Guy S, Lee-Jones L, McGinn T, Schlecht J. Reproductive Health: A Right for Refugees and Internally Displaced Persons. Reproductive Health Matters. 2008;16(31):10-21. Casey SE, Chynoweth SK, Cornier N, Gallagher MC, Wheeler EE. Progress and gaps in reproductive health services in three humanitarian settings: mixed-methods case studies. Confl Health. 2015;9(Suppl 1 Taking Stock of Reproductive Health in Humanitarian):S3. Habibov N, Fan L. The Effect of Maternal Healthcare on the Probability of Child Survival in Azerbaijan. BioMed Research International. 2014;2014:1-6. Cohen S. The Broad Benefits of Investing in Sexual and Reproductive Health. . Guttmacher Institute;2004. Barot S. Sexual and Reproductive Health and Rights Are Key to Global Development: The Case for Ramping Up Investment. Guttmacher Institute;2015. Roy MJ, Donaldson C, Baker R, Kerr S. The potential of social enterprise to enhance health and well-being: a model and systematic review. Soc Sci Med. 2014;123:182-193. Iipito Mendonca HNKaVZ, I. . Youth empowerment: the role of service design and mobile technology in accessing reproductive health information. Paper presented at: Proceedings of the 13th Participatory Design Conference: Short Papers, Industry Cases, Workshop Descriptions, Doctoral Consortium papers, and Keynote abstracts2014. IDEO.org. The Field Guide to Human Centered Design. In:2015: http://www.designkit.org/resources/1 . Giacomin J. What Is Human Centred Design? The Design Journal. 2015;17(4):606-623. Kindig S. Solving problems with human-centred design. In: Gates Cambridge; 2015. Brown TaW, J. Design Thinking for Social Innovation. In . Vol 8(1)31-35: Standford Social Innovation Review; 2010. Sanders EBN, Stappers PJ. Co-creation and the new landscapes of design. CoDesign. 2008;4(1):5-18. Koehly LM, Morris BA, Skapinsky K, Goergen A, Ludden A. Evaluation of the Families SHARE workbook: an educational tool outlining disease risk and healthy guidelines to reduce risk of heart disease, diabetes, breast cancer and colorectal cancer. BMC Public Health. 2015;15:1120. Parmar V. Disseminating maternal health information to rural women: a user centered design framework. Paper presented at: AMIA Symposium2010. Catalani C, Green E, Owiti P, et al. A clinical decision support system for integrating tuberculosis and HIV care in Kenya: a human-centered design approach. PLoS One. 2014;9(8):e103205. Itad. Evaluation of The Hewlett Foundation’s Strategy to Apply Human-Centered Design to Improve Family Planning and ReproductiveHealth Services in Sub-Saharan Africa. Hewlett Foundation 2018. Steen M, Manschot, M, and De Koning, N. Benefits of co-design in service design projects. International Journal of Design. 2011;5(2):53-60. Bazzano AN, Martin J, Hicks E, Faughnan M, Murphy L. Human-centred design in global health: A scoping review of applications and contexts. PLoS ONE. 2017;12(11):e0186744. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6(2):134-147. Palmer VJ, Piper D, Richard L, et al. Balancing Opposing Forces—A Nested Process Evaluation Study Protocol for a Stepped Wedge Designed Cluster Randomized Controlled Trial of an Experience Based Codesign Intervention. International Journal of Qualitative Methods. 2016;15(1). Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (Integrate, Design, Assess, and Share): A Framework and Toolkit of Strategies for the Development of More Effective Digital Interventions to Change Health Behavior. J Med Internet Res. 2016;18(12):e317. Hendricks S, Conrad N, Douglas TS, Mutsvangwa T. A modified stakeholder participation assessment framework for design thinking in health innovation. Healthc (Amst). 2018;6(3):191-196. . About Us. 2018; www..app , 2018. Hussain S. Toes that look like toes: Cambodian children's perspectives on prosthetic legs. Qual Health Res. 2011;21(10):1427-1440. d.School S. An Introduction to Design Thinking Process Guide. . https://dschoolold.stanford.edu/sandbox/groups/designresources/wiki/36873/attachments/74b3d/ModeGuideBOOTCAMP2010L.pdf . Diversity and Inclusion. 2020; https://www.yarracity.vic.gov.au/the-area/diversity-and-inclusion . Marcolino MS, Oliveira JAQ, D'Agostino M, Ribeiro AL, Alkmim MBM, Novillo-Ortiz D. The Impact of mHealth Interventions: Systematic Review of Systematic Reviews. JMIR Mhealth Uhealth. 2018;6(1):e23. Bloomfield GS, Vedanthan R, Vasudevan L, Kithei A, Were M, Velazquez EJ. Mobile health for non-communicable diseases in Sub-Saharan Africa: a systematic review of the literature and strategic framework for research. Global Health. 2014;10:49. Brown TaK, B. . Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation In: Harper Collins eBooks; 2009. Arnstein SR. A Ladder Of Citizen Participation. Journal of the American Institute of Planners. 1969;35(4):216-224. Burkett I. An Introduction to Co-Design. In: Centre for Social Impact,. Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009;30:175-201. Donetto ST, V. and Robert, G. Using Experience-based Co-design (EBCD) to improve the quality of healthcare: mapping where we are now and establishing future directions National Nursing Research Unit;2014. Pattyn VaB, M. Outsource versus In-house? An Identification of Organizational Conditions Influencing the Choice for Internal or External Evaluators. Canadian Journal of Program Evaluation 2013;Vol 28 No 2. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Minor revision 21 Mar, 2021 Review # 3 received at journal 17 Mar, 2021 Review # 2 received at journal 16 Mar, 2021 Reviewer # 4 agreed at journal 04 Mar, 2021 Reviewer # 3 agreed at journal 02 Mar, 2021 Reviews received at journal 27 Feb, 2021 Reviewer # 2 agreed at journal 27 Feb, 2021 Review # 1 received at journal 20 Feb, 2021 Reviewer # 1 agreed at journal 18 Feb, 2021 Reviewers invited by journal 16 Feb, 2021 Editor assigned by journal 15 Feb, 2021 Submission checks completed at journal 15 Feb, 2021 Editor invited by journal 15 Feb, 2021 First submitted to journal 09 Feb, 2021 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-226630","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":12316369,"identity":"1a8002b9-8201-4341-83fa-0dd6e1f9690b","order_by":0,"name":"Rebeccah Bartlett","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHElEQVRIiWNgGAWjYFACHjCZwCABogqAmL0BKsLAYEBYywEDBgkengMka5FIwK/FvL334IMfNXZ5DNLNzz5/MLCps5d8nfi4oKYuj4G9eZsEFi0yZ84lG/YcSy5mkDlmPOOAQZoEj3TuZuMZxw4XM/AcK8OmRUIix0yasYE5sUEiwRjosMMgLdukedgOAEVyzLBqkX9j/puxoR6oIP0zUMt/CR7Js0At/+oSG+TfYNciwWPGzNhwGGQmyJYDQO/zbpPmbQPZy4NdC09esmTPseOJbTJnihnOGAA5Z4B+4e07nNjGk1ZsgU0L+9mDH37UVCf2S7dvZqiosONnbz+78THPt7rEfvbDG29gC2UYYCNCZBSMglEwCkYBsQAAZeteU4dDfLMAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0001-7047-1295","institution":"Monash University","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Rebeccah","middleName":"","lastName":"Bartlett","suffix":""},{"id":12316370,"identity":"2fb5e3ce-bb9e-4ee9-9847-036de7a2fb29","order_by":1,"name":"Jacqueline Boyle","email":"","orcid":"","institution":"Monash University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Jacqueline","middleName":"","lastName":"Boyle","suffix":""},{"id":12316371,"identity":"a5e6f09f-27ca-4a56-be93-af6830fcaa55","order_by":2,"name":"Jessica Simons Smith","email":"","orcid":"","institution":"University of North Carolina at Chapel Hill Gillings School of Global Public Health","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Jessica","middleName":"Simons","lastName":"Smith","suffix":""},{"id":12316372,"identity":"ada4206f-1dbf-4770-bdce-e45384f4fb9e","order_by":3,"name":"Nadia Khan","email":"","orcid":"","institution":"Monash University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Nadia","middleName":"","lastName":"Khan","suffix":""},{"id":12316373,"identity":"580977cc-a966-4401-bc7a-f9cb49fcf099","order_by":4,"name":"Tracy Robinson","email":"","orcid":"","institution":"Charles Sturt University - Orange Campus","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Tracy","middleName":"","lastName":"Robinson","suffix":""},{"id":12316374,"identity":"19afa64b-fc79-414e-a2ae-f883fdd3fa8e","order_by":5,"name":"Rohit Ramaswamy","email":"","orcid":"","institution":"University of North Carolina at Chapel Hill Gillings School of Global Public Health","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Rohit","middleName":"","lastName":"Ramaswamy","suffix":""}],"badges":[],"createdAt":"2021-02-09 11:32:20","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-226630/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-226630/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":6274233,"identity":"104a1930-a171-457a-9108-53576cf01320","added_by":"auto","created_at":"2021-02-23 20:49:49","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":21640,"visible":true,"origin":"","legend":"Design Thinking Process Adaptation for \u003cBLINDED\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-226630/v1/246abf06d3fa62b0dbf624c8.jpg"},{"id":13669458,"identity":"9fd3e000-8653-4926-8a31-e5b677fa3bed","added_by":"auto","created_at":"2021-09-17 11:01:11","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":570995,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-226630/v1/3f19f840-3d23-48e7-8b02-5f38351d0157.pdf"}],"financialInterests":"","formattedTitle":"Evaluating Human-Centred Design for Public Health A Case Study on Developing a Healthcare App with Refugee Communities.","fulltext":[{"header":"Introduction","content":"\u003cp\u003eDuring 2017, Australia became home to 16,757 refugees.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e Women and children, who comprised 87% \u003csup\u003e1,2\u003c/sup\u003e of these new arrivals, face significant health challenges, including limited access to quality sexual and reproductive health (SRH) services.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e3\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e Poor SRH care has intergenerational consequences, affecting health and psychosocial outcomes for both mothers and their children. \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e Conversely, access to quality SRH services improves a number of health outcomes in women including prevention and management of high-risk pregnancies, reduction in unplanned pregnancies and abortions, reduction in obstetric complications, decreased anaemia and improved nutrition for both mother and baby.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e Despite these benefits, within Australia, women from migrant and refugee communities report less SRH awareness and experience reduced access to SRH-specific care as well as culturally-relevant support that could assist them to make evidence-based decisions about their own health and service utilisation. \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e A new approach to improve access to healthcare for underserved communities, particularly for women from refugee and migrant backgrounds, is needed, one that centres these women in both the process of finding, developing and disseminating the solutions themselves.\u003c/p\u003e\n\u003cp\u003eAddressing public health problems through human-centred design (HCD) is an ethical and effective approach to developing solutions with underserved populations that are more likely to experience significant disadvantage or social marginalisation.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e8\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e HCD utilises multidisciplinary teams to approach the problem-solving process through three distinct phases: Inspiration, Ideation and Implementation.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e HCD utilises \u0026ldquo;techniques which communicate, interact, empathise, and stimulate the people involved, obtaining an understanding of their needs, desires, and experiences, which often transcends that which the people themselves actually realised\u0026rdquo;.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e Human-centred design, design thinking, co-design, co-production and co-creation are all terms that are often used interchangeably despite having nuanced differences in application and outcome. Each of these approaches focus on addressing complex problems and designing solutions with the end user communities (i.e. beneficiaries).\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e12\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e Design thinking is a specific set of stages within the HCD approach which help to guide problem solving teams through the whole experience as it diverges and converges the Inspiration, Ideation and Implementation phases in an iterative manner. Co-design, short for collaborative design is the process of design thinking steps that includes generative research (i.e. learning from end users) and development design (i.e. creating solutions with end users).\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e This second, development design stage, is often termed co-production and together with co-design, these two stages form co-creation.\u003c/p\u003e\n\u003cp\u003eStudies have demonstrated promise that using HCD when developing health interventions can improve health outcomes for diverse populations \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e15\u003c/span\u003e\u0026ndash;\u003cspan class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e; that solutions developed using this approach result in increased uptake of services \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e; produce higher quality products and interventions; and that these products and interventions increased beneficiary satisfaction.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e However, a scoping review analysing 21 different studies for use of HCD in global health across various geographies and populations was unable to draw definitive conclusions about the effectiveness, because of the heterogeneity of implementation, application areas and contexts.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e There remains a lack of understanding regarding how best to achieve and evaluate a successful HCD-driven solution.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e Thus, there is a growing need to evaluate the implementation of programs that apply HCD principles. Implementation evaluations not only assess a program\u0026rsquo;s deliverables against intended goals but also identify the strengths and weaknesses of a the implementation process, informing replication and efforts to scale.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e To date, there have been limited studies or reports detailing an implementation evaluation of an entire HCD project. Most studies address only one aspect of the design process such as planning, \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e18\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e prototyping \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e or assessing stakeholder engagement \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e but do not evaluate the entire approach. This study aimed to evaluate the HCD approach that \u0026lt;\u0026thinsp;BLINDED\u0026gt;, a small\u0026thinsp;\u0026lt;\u0026thinsp;BLINDED\u0026gt;-based not-for-profit focused on improving access to healthcare for refugees and new migrants, undertook in developing a web-based application to deliver local, evidence-based and culturally relevant SRH information to its users.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eContext\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe \u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;web-app (herein simply referred to as an \u0026ldquo;app\u0026rdquo;), provides high quality, rights-based information on family planning, pregnancy and newborn health, sexuality and sexual health, as well as mental health, family violence and adolescent health. Written and video resources provide information on accessing health services in Australia and cover topics such as healthcare rights and responsibilities, accessing translating and interpreting services, public and private insurance, as well as clinic locations.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e The app was originally designed for English and Arabic-speaking communities living in \u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;and is in the process of being translated into other languages. As \u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;is committed to working in partnership with refugee and migrant communities to create products that are both ethical and sustainable\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e, the team chose to apply a HCD approach, using design thinking methods to prototype and eventually develop its digital health intervention. Co-designers for the app included refugee end users, subject matter experts (SMEs) from different, partner organisations that focus on health for multicultural communities, user experience (UX) students and computer programmers.\u003c/p\u003e\n\u003cp\u003eLocal partnerships were integral to completing many of the design steps. In 2017, \u0026lt;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;collaborated with an undergraduate UX class from the \u0026lt;\u0026thinsp;BLINDED\u0026gt;. A semester-long process to design a digital health solution to bridge the gap in refugees\u0026rsquo; access to SRH services resulted in five prototypes. \u0026lt;BLINDED\u0026gt;\u0026rsquo;s founder (\u0026lt;\u0026thinsp;BLINDED\u0026gt;) then selected two designs to combine and develop further in conjunction with the co-designers to incorporate end users\u0026rsquo; values and cultural beliefs.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e The \u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;team used a combination of design thinking approaches developed by IDEO, Stanford\u0026rsquo;s d-School and Mummah et al (2016). \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e10\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eIDEO\u0026rsquo;s Field Guide to Human Centred Design \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e and the Stanford d-School\u0026rsquo;s Process Guide influenced the development of the Empathise, Define, Ideate, Prototype and Test steps \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e however the external assessor renamed the Test step \u0026ldquo;Launch and Share\u0026rdquo;, to ensure appropriate dissemination of any product or early research findings as per Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e The details of activities in each design thinking step used in developing the \u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;app are described in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e, below. Given the importance of evaluating public health interventions yet the lack of robust methodology surrounding those that are co-designed, the \u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;team planned for process evaluation to be undertaken regularly and as objectively as possible.\u003c/p\u003e\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:10.0pt;margin-left:0in;font-size:12px;font-family:\"Calibri\",sans-serif;color:#1F497D;font-style:italic;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px; line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eTable 1 \u0026lt;BLINDED\u0026gt;\u0026apos;s Human-Centred Design Process\u003c/span\u003e\u003c/p\u003e\n\u003ctable style=\"width:524.0pt;margin-left:-15.9pt;border-collapse:collapse;border:none;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 42.6pt;border: 1pt solid windowtext;background: rgb(219, 229, 241);padding: 0in 5.4pt;height: 8.95pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eStage\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 481.4pt;border-top: 1pt solid windowtext;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-image: initial;border-left: none;background: rgb(219, 229, 241);padding: 0in 5.4pt;height: 8.95pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eActivity\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 42.6pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 2.7pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cem\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEmpathise \u0026amp; Define\u003c/span\u003e\u003c/em\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 481.4pt;border-top: none;border-left: none;border-bottom: 1pt solid windowtext;border-right: 1pt solid windowtext;padding: 0in 5.4pt;height: 2.7pt;vertical-align: top;\"\u003e\n \u003cdiv style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\n \u003cul style=\"margin-bottom:0in;list-style-type: disc;\"\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eEmpathy sessions between Arabic-speaking refugees, refugee advocates and healthcare workers took place with final year UX students from \u0026lt;BLINDED\u0026gt; University throughout the first half of 2017.\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eResearchers undertook CBPR activities and group surveys to explore barriers and enablers to accessing healthcare for women from these refugee backgrounds.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003ePartner organisation, \u0026lt;BLINDED\u0026gt;, referred four refugee end users and three others were recruited using snowball-sampling techniques.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026lt;BLINDED\u0026gt; also assisted in connecting the CBPR researchers to women interested in helping the \u0026lt;BLINDED\u0026gt; team their understanding of the healthcare journey of different women from within these communities in \u0026lt;BLINDED\u0026gt;. \u0026nbsp; \u0026nbsp; \u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/div\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 42.6pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 35.35pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cem\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eIdeate \u0026amp; Design\u003c/span\u003e\u003c/em\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 481.4pt;border-top: none;border-left: none;border-bottom: 1pt solid windowtext;border-right: 1pt solid windowtext;padding: 0in 5.4pt;height: 35.35pt;vertical-align: top;\"\u003e\n \u003cdiv style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\n \u003cul style=\"margin-bottom:0in;list-style-type: disc;\"\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eUX students designed a solution based upon the insights gained during these empathy sessions.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThese designs were ideated then iterated with end users and other key stakeholders over several sessions throughout \u0026lt;BLINDED\u0026gt; University\u0026rsquo;s first semester in 2017. \u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/div\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 42.6pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 20.8pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePrototype\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 481.4pt;border-top: none;border-left: none;border-bottom: 1pt solid windowtext;border-right: 1pt solid windowtext;padding: 0in 5.4pt;height: 20.8pt;vertical-align: top;\"\u003e\n \u003cdiv style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\n \u003cul style=\"margin-bottom:0in;list-style-type: disc;\"\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eAfter selecting the winning design, \u0026lt;BLINDED\u0026gt;\u0026rsquo;s founder participated in a hackathon hosted \u0026lt;BLINDED\u0026gt;, a not-for-profit company that connects business analysts, programmers and UX designers with social impact organisations for weekend long prototyping meetups.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eSeveral computer programmers worked on \u0026lt;BLINDED\u0026gt; at the event and continued to develop the technological component of the app until the next hackathon five months later.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThe beta version of this app was developed, tested and iterated with Arabic-speaking refugees over the next 5 months. SMEs vetted health information and simplified content into plain language for accuracy and accessibility.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eAfter development of the initial prototype, a more advanced version was user tested with two different groups of co-designers. First, Arabic-speaking women tested the beta version of the app through a partnership with a local adult education program and neighbourhood house, located in \u0026lt;BLINDED\u0026gt;\u0026rsquo;s inner-city suburbs where one in four people are from migrant or refugee backgrounds.\u003c/span\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u003csup\u003e28\u003c/sup\u003e\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eAgain, the app\u0026rsquo;s content was user tested with SMEs for accuracy and accessibility.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eOnline SMEs from around the world completed functionality and basic content testing via a Qualtrics online survey software survey and a group of local SMEs then met in person to review and edit the health content, ensuring it was evidence-based before simplifying the information further into plain English prior to Arabic translation.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThe seven women who attended user testing sessions and the all SMEs were recruited using snowball-sampling techniques.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/div\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 42.6pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 20.5pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eLaunch \u0026amp; Share\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 481.4pt;border-top: none;border-left: none;border-bottom: 1pt solid windowtext;border-right: 1pt solid windowtext;padding: 0in 5.4pt;height: 20.5pt;vertical-align: top;\"\u003e\n \u003cdiv style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\n \u003cul style=\"margin-bottom:0in;list-style-type: disc;\"\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026lt;BLINDED\u0026gt; launched the beta version of the app in August 2017 with an event attended by co-designers, supporters, and funders.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/div\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size: 14px; line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e\u003cstrong\u003eEvaluation Questions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe evaluation was designed and conducted by an external assessor (\u0026lt;\u0026thinsp;BLINDED\u0026gt;) to reduce bias focused on the following three questions:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003eTo what extent did\u0026thinsp;\u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;complete all the steps of the design thinking process shown in Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e?\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eTo what extent did the final\u0026thinsp;\u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;app incorporate the contributions of all co-designers?\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eTo what extent were the co-designers satisfied with the process?\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThe first question assisted\u0026thinsp;\u0026lt;\u0026thinsp;BLINDED\u0026gt;\u0026rsquo;s team to understand how faithful to design thinking principles the initial co-design sessions were. The answer to this question could help the team improve future co-design endeavours. The second question was intended to determine the extent of co-designer involvement, a hallmark of the HCD approach. Finally, third question would not only assess co-designer satisfaction but also inform decision-making about how to structure future design sessions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthics Approval\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u0026lt;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;University provided ethics approval prior to data collection (Project ID number 13811: Evaluating the process and product of \u0026lt;\u0026thinsp;BLINDED\u0026gt;\u0026rsquo;s mHealth intervention). As the external evaluator was from The University of \u0026lt;\u0026thinsp;BLINDED\u0026gt;, the Institutional Review Board from \u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;also reviewed the evaluation plan and determined no additional ethics approval requirements for this project (\u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;Study #18-1449).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Collection Approach\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe primary data for the first question (completion of designing thinking steps) involved a thorough review of all of \u0026lt;\u0026thinsp;BLINDED\u0026gt;\u0026rsquo;s organisational documents including student design reports, community based participatory research (CBPR) results, meeting notes from prototyping events, Qualtrics data from user testing, emails between \u0026lt;\u0026thinsp;BLINDED\u0026thinsp;\u0026gt;\u0026thinsp;staff and the computer programmers regarding app updates and requests, as well as launch event information. Since there is a notable lack of validated tools evaluate HCD projects a maturity rubric was designed to synthesize the findings from the document review. This rubric was developed through consultation with two experts in the field of implementation science (\u0026lt;\u0026thinsp;BLINDED\u0026gt;), two reproductive and indigenous health experts (\u0026lt;\u0026thinsp;BLINDED\u0026gt;), one HCD expert (\u0026lt;\u0026thinsp;BLINDED\u0026gt;) and one participatory research experts (\u0026lt;\u0026thinsp;BLINDED\u0026gt;). Several iterations of feedback from the expert panel were used to improve the usability, completeness and level of detail of the rubric. While the rubric and design steps used (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e) appears linear, the design process is fluid and the steps listed did not necessarily occur in a stepwise fashion. The rubric assesses the level of completion of each step of the design thinking process on a scale from 0 (non-existent) to 3 (full completion) with a maximum score possible of 12 (Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:10.0pt;margin-left:0in;font-size:12px;font-family:\"Calibri\",sans-serif;color:#1F497D;font-style:italic;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px; line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eTable 2 Human-Centred Design Scoring Rubric\u003c/span\u003e\u003c/p\u003e\n\u003ctable style=\"width: 4.6e+2pt;border-collapse:collapse;border:none;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border: 1pt solid black;background: rgb(219, 229, 241);padding: 0in 5.4pt;height: 5.55pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEmpathise and Define\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 19.7pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eNo Attempt: (0 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEnd users were not engaged, nor were attempts made to understand their lived experience.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 19.7pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eMinimally Attempted: (1 point)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEnd users were engaged through interviews and felt respected.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 41.35pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eModerately Attempted: (2 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEnd users were engaged, felt respected, and were compensated for their participation in co-design sessions. (Compensation may be financially via cash or gift cards, transportation costs to get to sessions, provision of childcare during sessions, or other means.)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEmpathy exercises were undertaken to understand lived experience of end users.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 27.7pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSatisfied: (3 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEnd users in co-design sessions were engaged, felt respected, were compensated and were representative of the whole target population.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEmpathy exercises were undertaken to understand the lived experience end users.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 9.65pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eIdeate and Design\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 20pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eNot Attempted: (0 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSolution was decided on ahead of time and did not emerge in response to what was learned from end users.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 30.6pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eMinimally Attempted: (1 point)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eLearnings from empathy exercises were compiled.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePre-determined solution was minimally modified in response to what was learned from end users.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 36.9pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eModerately Attempted: (2 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eLearnings from empathy exercises were compiled, brainstorming solutions was done in teams, and end user insights guided design of solution.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eGroup consensus was obtained on the problem to be addressed/solved.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 51pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSatisfied: (3 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eLearnings from empathy exercises were compiled, brainstorming solutions sessions were done in teams, end user insights guided the creation of the solution, additional information was gathered from end users if necessary.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eGroup consensus was obtained on the problem to be addressed/solved. Group consensus was obtained regarding which solutions to prototype solve the identified problem.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 6.95pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePrototype\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 17.4pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eNot Attempted: (0 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eNo attempt was made to obtain feedback about prototypes or minimum viable product (MVP) from end users.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 30pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eMinimally Attempted: (1 point)\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSingle prototype made.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEnd user feedback solicited.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eFeedback not incorporated into final version of solution.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 5.2pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eModerately Attempted: (2 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eMultiple iterations of prototypes and/or MVPs created.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePrototypes or MVPs tested with end users.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEnd user feedback incorporated into subsequent iterations.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 34.9pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSatisfied: (3 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eMultiple iterations of prototypes and/or MVPs created and tested with end user population.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEnd user feedback incorporated into subsequent iterations of solution.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSolution validated with subject matter experts and/or existing literature.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 3.5pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eLaunch and Share\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 15.15pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eNot Attempted: (0 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eProduct was launched. No user feedback obtained. No dissemination completed.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 30pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eMinimally Attempted: (1 point)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eProduct launched.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eUser testing completed to understand users\u0026rsquo; experience and satisfaction. \u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 30pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eModerately Attempted: (2 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eProduct launched.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eUser testing completed to understand users\u0026rsquo; experience and satisfaction. \u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eUser feedback incorporated into plans for future iterations.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eProcess or product results shared with program staff.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 30pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSatisfied: (3 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eProduct launched.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eUser testing completed to understand users\u0026rsquo; experience and satisfaction. \u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eUser feedback incorporated into plans for future iterations.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eProcess or product results shared with program staff, co-designers, and wider community.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 455.65pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 10.45pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eTotal Score (max score of 12):\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px; line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\u003cp\u003eA survey (Table 3) and a semi-structured interview guide (Table 4) modelled on the IDEO Field Guide to Human Centred Design and mHealth evaluation guidelines \u003csup\u003e10,29,30\u003c/sup\u003e were created to answer the second and third evaluation questions as well as to obtain clarification and confirmation of the data obtained through the document review.\u003c/p\u003e\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:10.0pt;margin-left:0in;font-size:12px;font-family:\"Calibri\",sans-serif;color:#1F497D;font-style:italic;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eTable\u0026nbsp;\u003c/span\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e3\u003c/span\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;Participant Survey Feedback Form \u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n\u003ctable style=\"width: 4.9e+2pt;border: none;margin-left:-.25pt;border-collapse:collapse;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"2\" style=\"width:169.05pt;border:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:19.65pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cstrong\u003e\u003cspan style='font-size:15px;line-height:200%;font-family: \"Helvetica\",sans-serif;color:black;'\u003eAt the co-design session \u0026hellip;\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border:solid black 1.0pt;border-left:none;background:#D9D9D9;padding:0in 5.4pt 0in 5.4pt;height:19.65pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cstrong\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eStrongly Disagree\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border:solid black 1.0pt;border-left:none;background:#D9D9D9;padding:0in 5.4pt 0in 5.4pt;height:19.65pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cstrong\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eDisagree\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border:solid black 1.0pt;border-left:none;background:#D9D9D9;padding:0in 5.4pt 0in 5.4pt;height:19.65pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cstrong\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eNeither disagree nor agree\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border:solid black 1.0pt;border-left:none;background:#D9D9D9;padding:0in 5.4pt 0in 5.4pt;height:19.65pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cstrong\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eAgree\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border:solid black 1.0pt;border-left:none;background:#D9D9D9;padding:0in 5.4pt 0in 5.4pt;height:19.65pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cstrong\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eStrongly Agree\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border:solid black 1.0pt;border-left:none;background:#D9D9D9;padding:0in 5.4pt 0in 5.4pt;height:19.65pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cstrong\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eN/A\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e1\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThe group work felt collaborative\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e2\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eMy contribution was valued\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e3\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003ePeople attended who do not usually have representation (i.e. people with refugee and migrant backgrounds, people with diverse sexual orientation, people experiencing health accessibility issues)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e4\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThe voices of refugee and/or migrant end users were heard\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e5\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThe roles and responsibilities of my participation were clearly defined\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e3\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e3\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eI understood both the processes and the language used \u0026nbsp; \u0026nbsp;\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e7\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;background:white;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eMy time participating was compensated appropriately\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e8\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;background:white;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThere was a commitment by \u0026lt;BLINDED\u0026gt; staff to develop consensus on what the end product should include\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e3\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e2\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e1\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e9\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eAll co-designers were kept informed of any changes \u0026nbsp;\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e10\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eI felt respected by \u0026lt;BLINDED\u0026gt; staff \u0026nbsp;\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e11\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eI felt respected by all co-design partners \u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e12\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026lt;BLINDED\u0026gt; staff offered me an opportunity for skill development and capability building\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e13\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026lt;BLINDED\u0026gt; staff offered me co-design training and resources\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.6pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e14\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThere were strategies to involve people with different communication needs \u0026nbsp; \u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e15\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThere was enough time to allow relationship building \u0026nbsp; \u0026nbsp;\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;1\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e2\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e3\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e2\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e4\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e16\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026lt;BLINDED\u0026gt; staff made attempts to reduce any power imbalance (e.g. between health professionals and refugees)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:14.05pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 10.1pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e17\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.1pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eI felt safe sharing my opinions \u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.1pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 10.1pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 10.1pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.1pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.1pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.1pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 15.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e18\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eRefugee and/or migrant end users helped shape the common agenda\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 15.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 12.55pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e19\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.55pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eI would participate in another co-design session with \u0026lt;BLINDED\u0026gt; staff\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.55pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 12.55pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 12.55pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.55pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.55pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.55pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e20\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eI would encourage others to participate in a co-design session with \u0026lt;BLINDED\u0026gt; staff\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:25.65pt;border:solid black 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height: 11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e21\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:143.4pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eI see my ideas and contributions reflected in the final \u0026lt;BLINDED\u0026gt; website/app\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026nbsp;0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:56.7pt;border-top:none;border-left:none;border-bottom: solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height: 11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e2\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:42.55pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e4\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:65.05pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e5\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:31.95pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:11.75pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style='font-size:15px;line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e1\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\u003cbr\u003e\u003cp\u003e\u003cstrong\u003eComments/Feedback:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThese questions explored end user representation, co-designers\u0026rsquo; understanding of the co-design sessions and design thinking methods used, issues around communication (i.e. language barriers, role clarification and understanding HCD goals), and co-designers\u0026rsquo; levels of satisfaction, using a Likert scale (Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e) regarding their involvement in the process.\u003c/p\u003e\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:10.0pt;margin-left:0in;font-size:12px;font-family:\"Calibri\",sans-serif;color:#1F497D;font-style:italic;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px; line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eTable\u0026nbsp;\u003c/span\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e4\u0026nbsp;Semi-Structured Interview Guide\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n\u003ctable style=\"border: none;width:472.2pt;border-collapse:collapse;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 25.05pt;border: 1pt solid windowtext;padding: 0in 5.4pt;height: 18.35pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e1\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:447.15pt;border:solid windowtext 1.0pt;border-left:none;padding: 0in 5.4pt 0in 5.4pt;height:18.35pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eCould you elaborate on how you felt your contribution to the co-design sessions was (or was not) recognised and valued?\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 25.05pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 18.35pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e2\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:447.15pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:18.35pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eWere there stakeholders who were not represented at the co-design sessions? If so, who else should have been invited?\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 25.05pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 32.15pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e3\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:447.15pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:32.15pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eWas there any confusion regarding your role, the processes or any language barriers at the co-design sessions that could have been made simpler/easier to understand? If so, what were these?\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 25.05pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 32.15pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e4\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:447.15pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:32.15pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eWere there ways that the \u0026lt;BLINDED\u0026gt; team or other participants made you feel respected or disrespected/unwelcome during this co-design experience?\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 25.05pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 32.15pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e5\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:447.15pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:32.15pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eDo you feel that any power imbalances between co-design participants were addressed e.g. do you feel that your opinion mattered and that it was safe for you to communicate your thoughts and experiences if you wanted to?\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 25.05pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 18.35pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e6\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:447.15pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:18.35pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eHow could this co-design experience be improved in the future?\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 25.05pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 3.65pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e7\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:447.15pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:3.65pt;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eDo you have any other feedback you wish to share?\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\n\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\n\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSurvey and Interview procedures\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe external evaluator (\u0026lt;BLINDED\u0026gt;) engaged three groups of co-designers to complete surveys and semi-structured interviews. This included four refugee end users, three UX designers or computer programmers (UX/programmers) and six SMEs (including one funding representative) totalling 13 respondents. All SMEs and UX/programmers were fluent in English even if it was not their native language. End users considered themselves \u0026ldquo;conversational-level\u0026rdquo; English speakers. 10 of the respondents identified as women. All co-designers had at least a bachelor\u0026rsquo;s-level education. \u0026lt;BLINDED\u0026gt; compensated end users for their time participating in the evaluation through store-bought gift cards.\u003c/p\u003e\n\u003cp\u003eThe co-designers who participated in the evaluation represented the larger co-design groups involved in creating the \u0026lt;BLINDED\u0026gt; app in relation to gender identity, education level, and English proficiency. Interviews with each participating co-designer were consented to in advance and then again in person when they were completed at a location of the co-designer\u0026rsquo;s choosing. All Arabic-speaking end users declined the use of an interpreter. The survey was administered before the interview, with three exceptions: one SME declined to answer the survey, and two phone interviewees completed the surveys after the interview. The surveys were analysed using Qualtrics online survey software. Interviews were recorded and transcribed. Transcripts were de-identified, coded inductively using a hierarchical framework using NVivo 12 software by the external evaluator and a research assistant (\u0026lt;BLINDED\u0026gt;), herself a daughter of immigrants, who requested to work on this project because of its migrant women\u0026rsquo;s health focus.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003e\u003cstrong\u003eSuccesses\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u0026lt;BLINDED\u0026gt; scored 9 out of 12 for fidelity to the design thinking process with a notable need for improvement around the ideation stage (Table 5).\u003c/p\u003e\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:10.0pt;margin-left:0in;font-size:12px;font-family:\"Calibri\",sans-serif;color:#1F497D;font-style:italic;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px; line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eTable 5: \u0026lt;BLINDED\u0026gt; HCD Evaluation Score\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n\u003ctable style=\"width: 4.7e+2pt;border-collapse:collapse;border:none;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 111.75pt;border: 1pt solid black;background: rgb(219, 229, 241);padding: 0in 5.4pt;height: 4.05pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEmpathise and Define\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 319.9pt;border-top: 1pt solid black;border-right: 1pt solid black;border-bottom: 1pt solid black;border-image: initial;border-left: none;background: rgb(219, 229, 241);padding: 0in 5.4pt;height: 4.05pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eComments\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.6pt;border-top: 1pt solid black;border-right: 1pt solid black;border-bottom: 1pt solid black;border-image: initial;border-left: none;background: rgb(219, 229, 241);padding: 0in 5.4pt;height: 4.05pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePoints\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 111.75pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 37.9pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eModerately Attempted: (2 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEnd users were engaged, felt respected, and were compensated for their participation in co-design sessions. (Compensation may be financially via cash or gift cards, transportation costs to get to sessions, provision of childcare during sessions, or other means.)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEmpathy exercises were undertaken to understand lived experience of end users.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 319.9pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;padding: 0in 5.4pt;height: 37.9pt;vertical-align: top;\"\u003e\n \u003cdiv style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\n \u003cul style=\"margin-bottom:0in;list-style-type: undefined;\"\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height:200%;font-family:Symbol;\"\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u003c/span\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThe organisational documents provided evidence of compensation, end user interviews and ethnographic work, CBPR and empathy exercises but there was no documentation regarding whether co-designers felt respected or recognised for their contribution. Instead, this information came from the surveys.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height:200%;font-family:Symbol;\"\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u003c/span\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eAll co-designers selected \u0026lsquo;agree\u0026rsquo; or \u0026lsquo;strongly agree\u0026rsquo; to feeling respected and if they would encourage others to participate in a co-design session with the \u0026lt;BLINDED\u0026gt; team.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThe representativeness of the end user co-designers was also not apparent in any of the organisational documents but emerged as a theme during the interviews.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height:200%;font-family:Symbol;\"\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u003c/span\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eCo-designers\u0026rsquo; views on how representative the end users were varied by the type of co-designer group the participant came from.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height:200%;font-family:Symbol;\"\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u003c/span\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eSMEs and UX/programmers generally agreed that there was enough end user representation at the co-design sessions. As one computer programmer said: \u003cem\u003e\u0026ldquo;We focused bringing on more and more people from the refugee and migrant community which is really good \u0026hellip; I don\u0026rsquo;t think we had a shortage of that diversity \u0026hellip; in terms of cultural background it was quite well represented.\u0026rdquo;\u0026nbsp;\u003c/em\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThe end users themselves felt that there were groups within the Arabic-speaking population that were unrepresented. Some of the suggested groups include individuals who did not attend university, Arabic speakers with no or low English proficiency, middle and late middle age individuals, people with different levels of proficiency with mobile technology, and refugees who had just arrived to Australia compared to refugees who have been living in Australia for some time.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eA more representative end user population would have accrued a higher score.\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/div\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.6pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;padding: 0in 5.4pt;height: 37.9pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e2\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 111.75pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 7pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eIdeate and Design\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 319.9pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 7pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.6pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 7pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePoints\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 111.75pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 22.1pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eMinimally Attempted: (1 point)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eLearnings from empathy exercises were compiled.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePre-determined solution was minimally modified in response to what was learned from end users.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 319.9pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;padding: 0in 5.4pt;height: 22.1pt;vertical-align: top;\"\u003e\n \u003cdiv style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\n \u003cul style=\"margin-bottom:0in;list-style-type: disc;\"\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eAn extensive document review verified that the \u0026lt;BLINDED\u0026gt; team did not have a pre-conceived idea of what the prototype would be, and that the app\u0026rsquo;s features and structures arose in response to insights gained from the end user co-designers.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eThe ideation stage took place within the design student teams and was tested at intervals with end users.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eSurveys and interviews with the co-designers revealed that one quarter of all survey participants marked \u0026lsquo;neither agree nor disagree\u0026rsquo; to whether \u0026lt;BLINDED\u0026gt; confirmed group consensus for either the problem statement or the solutions to be prototyped. The remaining three quarters indicated they \u0026lsquo;agree\u0026rsquo; or \u0026lsquo;strongly agree\u0026rsquo; that \u0026lt;BLINDED\u0026gt; did in fact, achieve this.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eWhilst the issue of consensus did not emerge as a theme across all interviews, one computer programmer articulated the problem this way: \u0026ldquo;I feel like sometimes there was a bit of disconnect between what [\u0026lt;BLINDED\u0026gt; staff] wanted and maybe necessarily what the refugees wanted \u0026hellip; they wanted a resource where they could find health information, locations specifically of hospitals, GPs, pharmacies \u0026hellip; the actual health information, they would rather go directly to the source, rather than \u0026hellip; reading it online.\u0026rdquo; \u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eGiven the conflicting information received from the surveys and the interviews and the lack of documentation, the evaluator reported finding difficulty scoring \u0026lt;BLINDED\u0026gt; in this area.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eA lower score of 1 out of 3 was given to draw attention to this issue in the future.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eMore thorough record keeping during this design stage will shed light on this process and the methods used in future HCD projects.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/div\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.6pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;padding: 0in 5.4pt;height: 22.1pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e1\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 111.75pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 5.05pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePrototype\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 319.9pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 5.05pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.6pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 5.05pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePoints\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 111.75pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 42.05pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSatisfied: (3 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eMultiple iterations of prototypes and/or MVPs created and tested with end user population.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eEnd user feedback incorporated into subsequent iterations of solution.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSolution validated with subject matter experts and/or existing literature.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 319.9pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;padding: 0in 5.4pt;height: 42.05pt;vertical-align: top;\"\u003e\n \u003cdiv style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\n \u003cul style=\"margin-bottom:0in;list-style-type: disc;\"\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eEnd users from different backgrounds tested and contributed to iterations of the app on multiple occasions and continue to do so to this day.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eSME co-designers and their feedback lead to tangible changes in the app appearance, language accessibility and functionality.\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003eOrganisational documents alone verified these requirements\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026lt;BLINDED\u0026gt; received 3 out of 3 in this stage. \u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/div\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.6pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;padding: 0in 5.4pt;height: 42.05pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e3\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 111.75pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 2.55pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eLaunch and Share\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 319.9pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 2.55pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.6pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 2.55pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003ePoints\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 111.75pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 21.7pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eSatisfied: (3 points)\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eProduct launched.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eUser testing completed to understand users\u0026rsquo; experience and satisfaction. \u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eUser feedback incorporated into plans for future iterations.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eProcess or product results shared with program staff, co-designers, and wider community.\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 319.9pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;padding: 0in 5.4pt;height: 21.7pt;vertical-align: top;\"\u003e\n \u003cdiv style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\n \u003cul style=\"margin-bottom:0in;list-style-type: disc;\"\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026lt;BLINDED\u0026gt; held a product release in August 2017 which included co-designers, partners, and funders and presented preliminary findings from the CBPR projects.\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style='line-height:200%;font-family:\"Helvetica\",sans-serif;color:black;'\u003e\u0026lt;BLINDED\u0026gt; received 3 out of 3 in this stage. \u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/div\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.6pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;padding: 0in 5.4pt;height: 21.7pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;border:none;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e3\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 111.75pt;border-right: 1pt solid black;border-bottom: 1pt solid black;border-left: 1pt solid black;border-image: initial;border-top: none;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 7.6pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;margin-right:16.0pt;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003eTotal Score (max score of 12):\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 319.9pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 7.6pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cstrong\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.6pt;border-top: none;border-left: none;border-bottom: 1pt solid black;border-right: 1pt solid black;background: rgb(217, 217, 217);padding: 0in 5.4pt;height: 7.6pt;vertical-align: top;\"\u003e\n \u003cp style='margin:0in;font-size:16px;font-family:\"Calibri\",sans-serif;text-align:center;line-height:200%;'\u003e\u003cspan style=\"font-size: 14px;\"\u003e\u003cspan style=\"line-height: 200%; font-family: Helvetica, sans-serif; color: black;\"\u003e9\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\u003cbr\u003e\u003cp\u003eFeedback obtained through both the interviews and surveys verified that the \u0026lt;BLINDED\u0026gt; team did complete all the steps of the design thinking approach.\u0026nbsp; All survey respondents selected \u0026lsquo;agree\u0026rsquo; or \u0026lsquo;strongly agree\u0026rsquo; to statements assessing the collaborative nature of the group work and that they felt safe sharing their opinions (Table 3). Over 90% reported that they would participate in another \u0026lt;BLINDED\u0026gt; co-design session and that they would recommend participation to a friend or family member. In the interviews, a feeling of enjoyment from participating in the co-design sessions clearly emerged. All 13 evaluation participants reported feeling valued, appreciated, and/or respected during the co-design sessions.\u003c/p\u003e\n\u003cp\u003eWhen we went to \u0026hellip; the students, they were very enthusiastic. They take our notes and they try to discuss with us \u0026hellip; And as users \u0026hellip; they take our notes and they try to improve. And we share, really. We share as a big group, as a teamwork. And we share our ideas together.\u0026nbsp;\u003cem\u003e\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThe surveys and interviews revealed important learnings for the \u0026lt;BLINDED\u0026gt; team when using HCD within this particular context.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCommunication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFirst was the issue of communication. Three of the four end users mentioned that language was a barrier, despite assistance from other community members who acted as interpreters during the co-design sessions and their own self-assessment as being proficient in English. This likely exacerbated the second area for improvement: role clarification. This was a need reported by co-designers from all groups. An end user expressed how she came to understand her role in the co-design session:\u003c/p\u003e\n\u003cp\u003eAfter some time, I could realise what\u0026rsquo;s going on, and understand what I had to do. It wasn\u0026rsquo;t clear in the beginning. Like when I went there, I didn\u0026rsquo;t know why I\u0026rsquo;m going there. I just know that I want to be part of this, this is what I really wanted to do, and yeah, after some time I could understand what\u0026rsquo;s going on, but nobody explained me how.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFragmentation of involvement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThere was fragmentation of co-designers\u0026rsquo; experiences during the sessions. Many spoke about not understanding the project\u0026rsquo;s entire process, wanting to be more involved but not receiving further invitations, or the need to build on previous sessions with end users. While end users were involved at every step of the co-design, different individuals participated at different points and in different ways. Very few end user co-designers were a part of the process from beginning to end. This led to feelings of disconnection and confusion, as one end user put it:\u003c/p\u003e\n\u003cp\u003eFirst, I was really interested, but after some time when I found like nobody\u0026rsquo;s calling you back, so I said no I don\u0026rsquo;t want to waste my time on this. But it\u0026rsquo;s something really helpful and I really like the idea of helping new arrivals from refugee and migrant backgrounds.\u003c/p\u003e\n\u003cp\u003eSMEs who facilitated meetings between \u0026lt;BLINDED\u0026gt; staff and end users also felt the desire to be more involved throughout the entire HCD project. One SME commented:\u003c/p\u003e\n\u003cp\u003eI think it would be good to have a follow up consultation on working on actual usage \u0026hellip; because we haven\u0026rsquo;t sort of touched base again with those women to say, have you used it? \u0026hellip; it was almost still in the design stage, and things hadn\u0026rsquo;t quite been finished. So, I think we could have a follow up that says, this is the latest version of the product, let\u0026rsquo;s have a play around with it, what do you think now? I think that would be really timely.\u003c/p\u003e\n\u003cp\u003eA third area for consideration in future co-design session is the importance of diversity and representation within end-user groups. The SMEs and UX/programmer generally agreed that there was sufficient end user representation at the co-design sessions:\u003c/p\u003e\n\u003cp\u003eWe focused bringing on more and more people from the refugee and migrant community which is really good\u0026hellip;I don\u0026rsquo;t think we had a shortage of that diversity \u0026hellip; in terms of cultural background it was quite well represented.\u003c/p\u003e\n\u003cp\u003eThe end users themselves, however, felt that there were groups within the Arabic-speaking population that were unrepresented. Some of the suggested groups include individuals who did not attend university, Arabic speakers with no or low English proficiency, middle and late middle age individuals, people with different levels of proficiency with mobile technology, and refugees who had just arrived to Australia compared to refugees who have been living in Australia for some time. The risks of only collaborating with end users who are university-educated, recently resettled and English-proficient were summarised by one end user:\u003c/p\u003e\n\u003cp\u003eMaybe we will use this program [\u0026lt;BLINDED\u0026gt;] and maybe never we will use this program because we can search \u0026hellip; what we need by Google \u0026hellip; but that program [\u0026lt;BLINDED\u0026gt;], it\u0026rsquo;s good for different level of the people.\u003c/p\u003e\n\u003cp\u003eIncreased representation from within the end-user group should be considered for future HCD endeavours.\u003c/p\u003e\n\u003cp\u003eFinally, a survey question assessed whether co-designers felt that there was enough time for relationship building during the co-design sessions. End users responded most negatively to the statement, with half stating that they \u0026lsquo;strongly disagree\u0026rsquo; or \u0026lsquo;disagree\u0026rsquo; that there was enough time for this.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThis evaluation found that a co-design process was successfully applied to the development of a web-based app for refugee and migrant women in reproductive health. This evaluation also yielded several important recommendations for improving \u0026lt;BLINDED\u0026gt;\u0026rsquo;s HCD approach moving forward, findings that can be applied to other projects seeking to undertake an authentic community co-design process. First, with so many people of diverse backgrounds contributing to the project, clear communication about roles and expectations is critical. More attention to facilitator training and identification of session goals and following up with consistent communication and seeking end user and SME feedback would help to reduce future confusion.\u003csup\u003e10\u003c/sup\u003e Second, it is important to set realistic expectations and role clarifications with co-designers. Evaluation participants frequently mentioned their role and the purpose of the project as a whole. \u003csup\u003e27\u003c/sup\u003e \u0026nbsp;Design is a non-linear and creative process, which can inadvertently contribute to confusion about the co-designer\u0026rsquo;s purpose and the project\u0026rsquo;s goals.\u003csup\u003e31\u003c/sup\u003e Care should be taken in advance to explain this and answer questions from participants not familiar with the concept.\u003c/p\u003e\n\u003cp\u003eThird, it is important not to view all end users as interchangeable.\u003csup\u003e10\u003c/sup\u003e UX/programmers and some SMEs saw refugees at all the meetings and viewed that as enough end user participation. End users however, felt there were other voices from their community that needed engagement. For example, the fact that all the end users spoke some English meant that co-design sessions could proceed without certified interpreters, but it also meant that the voices and experiences of refugees with low English proficiency were missing. There will naturally be trade-offs in any public health project since limited resources are an unchanging reality, but several process adjustments could address this issue. This evaluation demonstrates that the practice of engaging end users across all empathy, design, and prototyping stages and into product development is possible. 9\u003csup\u003e31\u003c/sup\u003e Collaborating with multiple migrant and refugee advocate organisations to use diverse sampling techniques will help to engage a more representative sample in the future co-design sessions. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFinally, by setting aside adequate time to develop collaborative relationships amongst all co-design groups the HCD process is an opportunity to give power and control back to the end user population for whom one is designing the health intervention. \u003csup\u003e32\u003c/sup\u003e Placing greater effort into building relationships as a part of the co-design session is especially important with \u0026lt;BLINDED\u0026gt;\u0026rsquo;s target refugee population.\u003csup\u003e25\u003c/sup\u003e When properly implemented, the intent of HCD is to provide public health organisations a pathway to sharing (and where applicable, handing over) power in order to achieve true citizen participation and control.\u003csup\u003e24\u003c/sup\u003e Failure to apply HCD principles in an authentic or purposeful way usually results in tokenism, and development of solutions that are unsustainable.\u003csup\u003e32\u003c/sup\u003e It can also \u0026ldquo;exacerbate social exclusion and destroy trust systems\u0026rdquo; when done poorly.\u003csup\u003e33\u003c/sup\u003e A project cannot utilise HCD without a power dynamic shift that ensures the end user, not the UX designer, computer programmer or community organisation, is in the primary decision-making role.\u003csup\u003e11\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFuture Considerations\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAfter reviewing all three data sources, there are several important considerations when planning for evaluation of any HCD-driven projects. First, while this version of the rubric was helpful in gaining a deeper understanding of \u0026lt;BLINDED\u0026gt;\u0026rsquo;s HCD approach, several iterations will increase the tool\u0026rsquo;s usefulness moving forward. Including operational definitions of each of the three stages would clarify the expectations, especially for anyone who is trying to use the rubric to guide future HCD-driven projects. Additionally, some of the requirements were impossible to assess using existing documentation alone. For example, the evaluator was unable to ascertain whether end users felt respected from organisational documents alone and all three data sources were required to complete the rubric.\u003c/p\u003e\n\u003cp\u003eCo-designers\u0026rsquo; perceptions of whether they felt respected and valued during co-design sessions should be included all future feedback forms, surveys and interview guides. Despite this, using multiple data sources (document review, surveys, and interviews) should have helped to verify answers. Instead, each data source provided unique and isolated findings and there proved inconsistency between interviews and the survey responses concerning communication. For example, while the majority (75%) of co-designers responded \u0026lsquo;agree\u0026rsquo; or \u0026lsquo;strongly agree\u0026rsquo; to the survey statement about whether roles and responsibilities had been clearly explained, many anecdotes arose during the interviews around the fact that co-designers did not actually understand their role in the overall project or at specific co-design sessions. Similarly, one of the main themes of the semi-structured interviews was how fragmented the co-designers felt their involvement was though there was no way to verify this finding in the document review or with the surveys. One way to address this problem is by considering evaluation methods alongside HCD planning meetings. Collecting co-designer surveys throughout the project can help understand their experiences \u0026lsquo;in the moment\u0026rsquo; and offers an opportunity for a more agile response if needed, whilst also offering an opportunity to compare experiences and feedback later following end of project co-designer evaluations. \u0026lt;BLINDED\u0026gt; did not have any documentation regarding training materials on design thinking facilitation or how facilitators had introduced the co-designers to various methodologies at different points in the app\u0026rsquo;s development, making difficult to verify these findings. Future documentation on roles, design thinking goals and checking in with co-designers frequently would benefit all involved.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStrengths and Limitations\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis paper adds to the existing literature regarding the rigorous use of HCD in public health. This implementation evaluation provides an important guide to future knowledge about purposefully working with end user communities to design better health interventions. Engaging in evaluation work increases the transparency of organisations and helps them demonstrate their commitment to the HCD ethos. This paper and the rubric are helpful tools for organisations attempted to evaluate their use of design thinking methods, and can assist them to plan, prepare for and execute co-design successful sessions. Additionally, the evaluation included multiple sources of data including surveys, interviews, and records. While \u0026lt;BLINDED\u0026gt;\u0026rsquo;s data collection methods will be improved as a result of this evaluation, multiple sources of data give a more full, rich, and accurate picture of co-designers\u0026rsquo; experiences and the methods themselves.\u003csup\u003e34\u003c/sup\u003e The richness of the data obtained is especially important since HCD is nebulous by nature and there is a lack of demonstrated, rigorous evaluation.\u003csup\u003e20,35\u003c/sup\u003e Finally, the utilisation of an external assessment increased the objectivity of this evaluation.\u003csup\u003e36\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eThere were several limitations to this evaluation study. First, the convenience sampling of co-designers introduced selection bias into the results. It is possible that the co-designers who did not respond to a request to participate in the evaluation would have provided different answers and perspectives. Second, the evaluation took place 9-16 months after most of the co-design sessions, making results vulnerable to recall bias. Several interviewees mentioned at different points that the sessions had happened so long ago it took effort to remember and answer the questions. Third, a yet to be validated rubric guided part of the evaluation. Despite expert feedback and iteration on this rubric before its application, there needs to be more use of the checklist to ensure that it is both reliable and generalisable. Fourth, within the research team, only the research assistant (\u0026lt;BLINDED\u0026gt;) had any personal connection to the migrant or refugee experience. Future research on migrant or refugee communities must include funding to support people from migrant and refugee backgrounds to play an active and ideally, lead, role in designing, conducting, analysing and reporting on said research. Finally, evaluating HCD, design thinking and co-design efforts in public health is challenging due to its abstract, creative, and iterative nature and because results are specific to the local context. There are no definitive guidelines providing specific parameters for assessing a HCD-driven project and these terminologies are frequently interchanged despite their differences in application and outcome. The methods identified in this paper are a first attempt to benchmark this innovative approach and will need to be refined in the future.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eImproving refugees\u0026rsquo; access to sexual and reproductive health is complex and multidimensional and requires innovative and thoughtful problem solving. HCD is one way to address complex problems in an ethical and effective way and it is how \u0026lt;BLINDED\u0026gt; chose to approach the development of its solution to this problem among Arabic-speaking refugees in \u0026lt;BLINDED\u0026gt;, Australia. The surveys and interviews revealed that end user, SME, and UX/programmer co-designers enjoyed participating in the co-design sessions, felt respected and welcomed, and saw their contributions reflected in the final product. Opportunities for growth include engaging a more diverse end user population and communicating expectations and results more clearly during and after the co-design sessions. A comprehensive process evaluation will benefit the field by providing an example of how to assess an organisation\u0026rsquo;s ability to follow all the HCD steps \u003cem\u003eand \u003c/em\u003ewill advance knowledge on the effectiveness of HCD in developing solutions that are aligned with the needs of the target audience. This evaluation of \u0026lt;BLINDED\u0026gt;\u0026rsquo;s HCD approach provides a helpful and rigorous guide in reporting that may encourage other organisations undertaking HCD work to evaluate their own implementation. Such organisations should explore in advance, how they plan to evaluate not only the design steps but also co-designers\u0026rsquo; perceptions around their role and the contributions they made to the end product. Determining how to assess satisfaction with both process and product needs thoughtful consideration to ensure co-designers and evaluators are reflecting and measuring the same outcome respectively. Finally, utilising mixed methodologies has the potential to reveal inconsistent answers across the different sources of data being examined so care needs to be taken to ensure evaluative and probing questions in the semi-structured interview process add clarity and reduce confusion.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cem\u003eEthics approval and consent to participate\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u0026lt;BLIDNED\u0026gt; Project ID number 13811: Evaluating the process and product of Shifra\u0026rsquo;s mHealth intervention\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eConsent for publication\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eAll authors consent to publication\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAvailability of data and materials\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eUpon request\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eCompeting interests\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eConflict of Interest: First author (RB) founded the Shifra app, however an external evaluator (JSS) was used to limit bias in evaluating Shifra\u0026rsquo;s HCD process otherwise no authors report competing interests\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eFunding\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAuthors' contributions\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eJSS undertook the initial research\u003c/p\u003e\n\u003cp\u003eNK assisted with some data assistance\u003c/p\u003e\n\u003cp\u003eRB draft this paper based on the research and managed all edits and version control\u003c/p\u003e\n\u003cp\u003eJB and TR reviewed the paper\u003c/p\u003e\n\u003cp\u003eRR provided supervisory guidance\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAcknowledgements\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThe authors recognise \u0026lt;BLINDED\u0026gt;, \u0026lt;BLINDED\u0026gt; and \u0026lt;BLINDED\u0026gt; for their contribution to the evaluation plan and rubric design of this paper.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAuthors' information (optional)\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eNot Applicable\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eHistorical Settlement Reports. (2018) In: Australian Government; 2018.\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eSexual and Reproductive Health Data Report.\u003c/em\u003e Multicultural Centre for Women's Health;2016.\u003c/li\u003e\n\u003cli\u003eAustin J, Guy S, Lee-Jones L, McGinn T, Schlecht J. Reproductive Health: A Right for Refugees and Internally Displaced Persons. \u003cem\u003eReproductive Health Matters. \u003c/em\u003e2008;16(31):10-21.\u003c/li\u003e\n\u003cli\u003eCasey SE, Chynoweth SK, Cornier N, Gallagher MC, Wheeler EE. Progress and gaps in reproductive health services in three humanitarian settings: mixed-methods case studies. \u003cem\u003eConfl Health. \u003c/em\u003e2015;9(Suppl 1 Taking Stock of Reproductive Health in Humanitarian):S3.\u003c/li\u003e\n\u003cli\u003eHabibov N, Fan L. The Effect of Maternal Healthcare on the Probability of Child Survival in Azerbaijan. \u003cem\u003eBioMed Research International. \u003c/em\u003e2014;2014:1-6.\u003c/li\u003e\n\u003cli\u003eCohen S. \u003cem\u003eThe Broad Benefits of Investing in Sexual and Reproductive Health. .\u003c/em\u003e Guttmacher Institute;2004.\u003c/li\u003e\n\u003cli\u003eBarot S. \u003cem\u003eSexual and Reproductive Health and Rights Are Key to Global Development: The Case for Ramping Up Investment.\u003c/em\u003e Guttmacher Institute;2015.\u003c/li\u003e\n\u003cli\u003eRoy MJ, Donaldson C, Baker R, Kerr S. The potential of social enterprise to enhance health and well-being: a model and systematic review. \u003cem\u003eSoc Sci Med. \u003c/em\u003e2014;123:182-193.\u003c/li\u003e\n\u003cli\u003eIipito Mendonca HNKaVZ, I. . Youth empowerment: the role of service design and mobile technology in accessing reproductive health information. Paper presented at: Proceedings of the 13th Participatory Design Conference: Short Papers, Industry Cases, Workshop Descriptions, Doctoral Consortium papers, and Keynote abstracts2014.\u003c/li\u003e\n\u003cli\u003eIDEO.org. The Field Guide to Human Centered Design. In:2015: \u003ca href=\"http://www.designkit.org/resources/1\"\u003ehttp://www.designkit.org/resources/1\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eGiacomin J. What Is Human Centred Design? \u003cem\u003eThe Design Journal. \u003c/em\u003e2015;17(4):606-623.\u003c/li\u003e\n\u003cli\u003eKindig S. Solving problems with human-centred design. In: Gates Cambridge; 2015.\u003c/li\u003e\n\u003cli\u003eBrown TaW, J. Design Thinking for Social Innovation. In\u003cem\u003e.\u003c/em\u003e Vol 8(1)31-35: Standford Social Innovation Review; 2010.\u003c/li\u003e\n\u003cli\u003eSanders EBN, Stappers PJ. Co-creation and the new landscapes of design. \u003cem\u003eCoDesign. \u003c/em\u003e2008;4(1):5-18.\u003c/li\u003e\n\u003cli\u003eKoehly LM, Morris BA, Skapinsky K, Goergen A, Ludden A. Evaluation of the Families SHARE workbook: an educational tool outlining disease risk and healthy guidelines to reduce risk of heart disease, diabetes, breast cancer and colorectal cancer. \u003cem\u003eBMC Public Health. \u003c/em\u003e2015;15:1120.\u003c/li\u003e\n\u003cli\u003eParmar V. Disseminating maternal health information to rural women: a user centered design framework. Paper presented at: AMIA Symposium2010.\u003c/li\u003e\n\u003cli\u003eCatalani C, Green E, Owiti P, et al. A clinical decision support system for integrating tuberculosis and HIV care in Kenya: a human-centered design approach. \u003cem\u003ePLoS One. \u003c/em\u003e2014;9(8):e103205.\u003c/li\u003e\n\u003cli\u003eItad. \u003cem\u003eEvaluation of The Hewlett Foundation\u0026rsquo;s Strategy to Apply Human-Centered Design to Improve Family Planning and ReproductiveHealth Services in Sub-Saharan Africa.\u003c/em\u003e Hewlett Foundation 2018.\u003c/li\u003e\n\u003cli\u003eSteen M, Manschot, M, and De Koning, N. Benefits of co-design in service design projects. \u003cem\u003eInternational Journal of Design. \u003c/em\u003e2011;5(2):53-60.\u003c/li\u003e\n\u003cli\u003eBazzano AN, Martin J, Hicks E, Faughnan M, Murphy L. Human-centred design in global health: A scoping review of applications and contexts. \u003cem\u003ePLoS ONE. \u003c/em\u003e2017;12(11):e0186744.\u003c/li\u003e\n\u003cli\u003eSaunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. \u003cem\u003eHealth Promot Pract. \u003c/em\u003e2005;6(2):134-147.\u003c/li\u003e\n\u003cli\u003ePalmer VJ, Piper D, Richard L, et al. Balancing Opposing Forces\u0026mdash;A Nested Process Evaluation Study Protocol for a Stepped Wedge Designed Cluster Randomized Controlled Trial of an Experience Based Codesign Intervention. \u003cem\u003eInternational Journal of Qualitative Methods. \u003c/em\u003e2016;15(1).\u003c/li\u003e\n\u003cli\u003eMummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (Integrate, Design, Assess, and Share): A Framework and Toolkit of Strategies for the Development of More Effective Digital Interventions to Change Health Behavior. \u003cem\u003eJ Med Internet Res. \u003c/em\u003e2016;18(12):e317.\u003c/li\u003e\n\u003cli\u003eHendricks S, Conrad N, Douglas TS, Mutsvangwa T. A modified stakeholder participation assessment framework for design thinking in health innovation. \u003cem\u003eHealthc (Amst). \u003c/em\u003e2018;6(3):191-196.\u003c/li\u003e\n\u003cli\u003e\u0026lt;BLINDED\u0026gt;. About Us. 2018; \u003ca href=\"www.shifra.app\"\u003ewww.\u0026lt;BLINDED\u0026gt;.app\u003c/a\u003e, 2018.\u003c/li\u003e\n\u003cli\u003eHussain S. Toes that look like toes: Cambodian children's perspectives on prosthetic legs. \u003cem\u003eQual Health Res. \u003c/em\u003e2011;21(10):1427-1440.\u003c/li\u003e\n\u003cli\u003ed.School S. An Introduction to Design Thinking Process Guide. . \u003ca href=\"https://dschoolold.stanford.edu/sandbox/groups/designresources/wiki/36873/attachments/74b3d/ModeGuideBOOTCAMP2010L.pdf\"\u003ehttps://dschoolold.stanford.edu/sandbox/groups/designresources/wiki/36873/attachments/74b3d/ModeGuideBOOTCAMP2010L.pdf\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eDiversity and Inclusion. 2020; \u003ca href=\"https://www.yarracity.vic.gov.au/the-area/diversity-and-inclusion\"\u003ehttps://www.yarracity.vic.gov.au/the-area/diversity-and-inclusion\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eMarcolino MS, Oliveira JAQ, D'Agostino M, Ribeiro AL, Alkmim MBM, Novillo-Ortiz D. The Impact of mHealth Interventions: Systematic Review of Systematic Reviews. \u003cem\u003eJMIR Mhealth Uhealth. \u003c/em\u003e2018;6(1):e23.\u003c/li\u003e\n\u003cli\u003eBloomfield GS, Vedanthan R, Vasudevan L, Kithei A, Were M, Velazquez EJ. Mobile health for non-communicable diseases in Sub-Saharan Africa: a systematic review of the literature and strategic framework for research. \u003cem\u003eGlobal Health. \u003c/em\u003e2014;10:49.\u003c/li\u003e\n\u003cli\u003eBrown TaK, B. . Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation In: Harper Collins eBooks; 2009.\u003c/li\u003e\n\u003cli\u003eArnstein SR. A Ladder Of Citizen Participation. \u003cem\u003eJournal of the American Institute of Planners. \u003c/em\u003e1969;35(4):216-224.\u003c/li\u003e\n\u003cli\u003eBurkett I. An Introduction to Co-Design. In: Centre for Social Impact,.\u003c/li\u003e\n\u003cli\u003eBrownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. \u003cem\u003eAnnu Rev Public Health. \u003c/em\u003e2009;30:175-201.\u003c/li\u003e\n\u003cli\u003eDonetto ST, V. and Robert, G. \u003cem\u003eUsing Experience-based Co-design (EBCD) to improve the quality of healthcare: mapping where we are now and establishing future directions \u003c/em\u003eNational Nursing Research Unit;2014.\u003c/li\u003e\n\u003cli\u003ePattyn VaB, M. Outsource versus In-house? An Identification of Organizational Conditions Influencing the Choice for Internal or External Evaluators. \u003cem\u003eCanadian Journal of Program Evaluation \u003c/em\u003e2013;Vol 28 No 2.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"research-involvement-and-engagement","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"riae","sideBox":"Learn more about [Research Involvement and Engagement](http://researchinvolvement.biomedcentral.com/)","snPcode":"40900","submissionUrl":"https://submission.nature.com/new-submission/40900/3","title":"Research Involvement and Engagement","twitterHandle":"@MedicalEvidence","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"sexual and reproductive healthcare, healthcare access, refugee communities","lastPublishedDoi":"10.21203/rs.3.rs-226630/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-226630/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground: \u003c/strong\u003eAustralian women from migrant and refugee communities experience reduced access to sexual and reproductive healthcare. Human-centred design is an ethical and effective approach to developing health solutions with underserved populations that are more likely to experience significant disadvantage or social marginalisation. This study aimed to evaluate how well Shifra, a small Australian-based not-for-profit, applied human-centred design when developing a web-based application that delivers local, evidence-based and culturally relevant health information to its non-English speaking users.\u0026nbsp;\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eMethods: \u003c/strong\u003eThis study undertook a document review, survey and semi-structured interviews to evaluate how well Shifra was able to achieve its objectives using a human-centred design approach.\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eResults:\u003c/strong\u003e A co-design process successfully led to the development of a web-based health app for refugee and migrant women. This evaluation also yielded several important recommendations for improving Shifra’s human-centred design approach moving forward.\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eConclusions: \u003c/strong\u003eImproving refugees’ access to sexual and reproductive health is complex and requires innovative and thoughtful problem solving. This evaluation of Shifra’s human-centred design approach provides a helpful and rigorous guide in reporting that may encourage other organisations undertaking human-centred design work to evaluate their own implementation.\u0026nbsp;\u0026nbsp;\u0026nbsp;Keywords: human-centred design; design thinking; refugee health; evaluation\u0026nbsp;Plain language summary: Australian women from non-English speaking migrant and refugee communities face reduced access to sexual and reproductive healthcare and many then go on to experience poor health outcomes as a result. There is an urgent need for new approach to improve access to healthcare for underserved communities, one that centres these women in the process of finding, developing and disseminating the solutions themselves. Human-centred design is an ethical and effective methodology to working with communities to develop these health solutions.\u0026nbsp;This study aimed to evaluate how well Shifra, a small Australian-based not-for-profit focused on improving access to healthcare for refugees and new migrants, undertook human-centred design approach when developing a Smartphone app that deliver local, safe and culturally relevant health information to non-English speaking Australians. The authors interviewed refugees, health and social sector experts and computer programmers involved in creating Shifra to evaluate how well they used human-centred design to achieve its goals. This evaluation found that Shifra’s approach was successful whilst also highlighting several important recommendations for improving collaborative efforts with refugee communities. These findings could help other projects also seeking to undertake an authentic community co-design process.\u003c/p\u003e","manuscriptTitle":"Evaluating Human-Centred Design for Public Health A Case Study on Developing a Healthcare App with Refugee Communities.","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2021-02-23 20:49:48","doi":"10.21203/rs.3.rs-226630/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Minor revision","date":"2021-03-22T00:00:00+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2021-03-18T00:00:00+00:00","index":3,"fulltext":"Recommendation: Minor Revision\nForm responses:\n---\n\nComments to Author:\n---\nRIAE-D-21-00009_CJose\nReview\n\nEvaluating Human-Centred Design for Public Health A Case Study on Developing a Healthcare App with Refugee Communities.\n\nComments:\nThanks to the editor for sending this manuscript my way for review. Overall, this is the kind of study that is lacking in the literature around patient and public involvement. Detailing the process of Human centred design approaches with minority populations is excessively relevant and timely in an era where people are getting an increase awareness and acceptation of social, cultural, gender and identity diversity. I would highly recommend its publication although there is still room for improvement.\nAbstract:\nWell written summaries.\n\nMain text:\nIntroduction\nBackground: Limited studies or reports detailing an implementation evaluation of an entire HCD project.\nThis study aimed to evaluate the HCD approach that Shifra, a small Melbourne-based not-for-profit focused on improving access to healthcare for refugees and new migrants, undertook in developing a web-based application to deliver local, evidence-based and culturally relevant SRH information to its users.\nIntroduction is clear. It points to the poor outcomes of refugee and migrant women regarding sexual health and the need to co-design solutions that meet their needs in a culturally safe manner. However, after the introduction, there is almost no mention on the outcome on health of the app co-designed. So the paper is really on the co-design process per se and not on the sexual health outcomes of those women, which I found a bit disappointing. The process can be good with a poor outcome. A bit more emphasis on the contributions the end users and SME provided and how that shaped the final product would be appreciated.\nThe Context section is very helpful.\nMethods\nA table describing the members of the app-developing team and when they were engaged would help better understand the extent of this co-design approach and the level of engagement. For example, this table could show in rows the number of co-designers in each of the groups (refugees, migrants, computer professionals, students, ect), and in columns the steps of the process in which they were mainly involved. I find it hard to evaluate an engagement process when we don't know exactly who and when they were involved.\nPage 13 - Table 1 - Participant Survey Feedback Form: There are some numbers in the table , items 5, 8, 16 and 21, that needs explanation. Are they results? If so, why only for those items, and why in the Methods section? Also, this should be Table 3 (as mentioned in the text), not Table 1.\nPage 16 - Table 2 should be Table 4, as mentioned in the text.\nPage 17 - The co-designers who participated in the evaluation represented the larger co-design groups involved in creating the Shifra app in relation to gender identity, education level, and English proficiency. Who did choose the participants for the evaluation? What percentage of the team did this group represent? How do we know if this group is a good representation of the whole team? 4 end users versus 9 professionals seems unbalanced.\n\nResults\nPage 23 - Line 33: A mention on which co-designer group the quote was extracted from would be useful.\nPage 23 - This likely exacerbated the second area for improvement: role clarification. Role clarification is almost always a challenging area, whereas all people involved are proficient in the same language or not. I wouldn't put too much emphasis in the language barrier in this section.\nPage 23 - line 20 - All survey respondents selected 'agree' or 'strongly agree' to statements assessing the collaborative nature of the group work and that they felt safe sharing their opinions (Table 3). This table should be in the Results section then, and with data of all the items (see previous comment on Page 13 - Table 1).\nOverall, this section is quite succinct. It is not clear to me how they assess the first question with this maturity rubric. More explanation on the material reviewed and the process of extracting the data and score it would clarify this.\nDiscussion\nPage 26 - Line 16 : This evaluation found that a co-design process was successfully applied to the… How was the success measured? Is 9 on a max of 12, a good and validated result of a successful process? Examples of other studies using the same kind of tool?\nGenerally, examples of co-designers contributions to the process would have been appreciated. How did the co-designers assessed if their contribution were reflected in the final app and why did they feel their contribution to the co-design sessions were recognised and valued? Any examples, verbatims?\nWhat about the end-product? Is the app meeting its objectives? How did the HCD approach meet the needs of end users (migrant and refugee women in Australia) outside the group of end-user co-designers? Although evaluating the process itself is very important, the evaluation of the outcome of the product on the target population would have been critical. pIt is stated in Page 4 : Implementation evaluations not only assess a program's deliverables against intended goals but also identify the strengths and weaknesses of the implementation process, informing replication and efforts to scale.21 To date, there have been limited studies or reports detailing an implementation evaluation of an entire HCD project. Most studies address only one aspect of the design process such as planning,18,22 prototyping23 or assessing stakeholder engagement24 but do not evaluate the entire approach. This study does not evaluate the entire approach either. Some data, even preliminary, on the assessment of the program's deliverables against intended gaols (last step of Figure 1) would be very relevant and would complement nicely this paper. Ie; the HCD-driven solution. What is the value of this app now for the end users?\nGiven the sensitive subject matter of sexual health, some questions assessing the comfort level of co-designers in talking about this issue would be beneficial. How did the team provide sufficient support to manage taboos and sensitive narratives around sexual health in this minority population? How did the team successfully supported the end users in terms of cultural sensitivity and safety?\n* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons after the final decision on the manuscript has been made. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **Yes**\n* Level of interest: **An article of importance in its field**\n* Quality of written English: **Acceptable**\n* Declaration of competing interests: **I declare that I have no competing interests.**\n* I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.: **\nI agree to the open peer review policy of the journal**\n"},{"type":"editorInvitedReview","content":"","date":"2021-03-17T00:00:00+00:00","index":2,"fulltext":"Recommendation: Minor Revision\nForm responses:\n---\n\nComments to Author:\n---\nThank you for conducting and writing this interesting paper that provides useful insight to others who want to apply human-centred design or who want to hear experiences of working with refugees and migrant populations. I think the piece is strong and some minor corrections would strengthen the piece.\n\nFirstly, I question whether an approach is ever ethical rather than can be used or applied ethically or not, whereas you position HCD as ethical in itself. You may stand by this but I think this can be challenged.\n\nThe article is quite table heavy and this gets a bit frustrating and becomes inaccessible. Consider putting the results especially into text and losing the tables.\n\nSimilarly, it would be easier if the results had an intro paragraph which introduced the issues, and the areas for improvement, and then the subsequent text kept these two apart - I struggle to understand why/how you connect these.\n\nAdditionally, I think it is important to make more explicit in the discussion that the quality of collaboration was not naturally better because of the use of HCD, thereby challenging this approach as a natural way of enabling end users to effectively influence.\n\nI hope you finalise this work and consider how you can disseminate the results and work with the people you engaged in the process. In future, it may be interesting to invite them to test your findings in such papers.\n* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons after the final decision on the manuscript has been made. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **No**\n* Level of interest: **An article whose findings are important to those with closely related research interests**\n* Quality of written English: **Acceptable**\n* Declaration of competing interests: **I declare that I have no competing interests.**\n* I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.: **\nI agree to the open peer review policy of the journal**\n"},{"type":"reviewerAgreed","content":"","date":"2021-03-05T00:00:00+00:00","index":4,"fulltext":""},{"type":"reviewerAgreed","content":"","date":"2021-03-03T00:00:00+00:00","index":3,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2021-02-28T00:00:00+00:00","index":0,"fulltext":""},{"type":"reviewerAgreed","content":"","date":"2021-02-28T00:00:00+00:00","index":2,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2021-02-21T00:00:00+00:00","index":1,"fulltext":"Recommendation: Accept\nForm responses:\n---\n\nComments to Author:\n---\nAn interesting research that provides novel insights into the development process and evaluation of digital solutions for public health using Human Centred Design theory. All steps of the research are well explained. Negative points: representativeness could have been better, very limited amount of end users involved. Having to ask for research data is a barrier to access from patient communities. It's a valuable research for citizen and patient communities, especially with regards to introducing sound methodologies and rigorous evaluation of the co-creation process.\n* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons after the final decision on the manuscript has been made. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **Yes**\n* Level of interest: **An article of importance in its field**\n* Quality of written English: **Acceptable**\n* Declaration of competing interests: **I declare that I have no competing interests**\n* I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.: **\nI agree to the open peer review policy of the journal**\n"},{"type":"reviewerAgreed","content":"","date":"2021-02-19T00:00:00+00:00","index":1,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2021-02-17T00:00:00+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2021-02-16T00:00:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2021-02-15T23:00:00+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2021-02-15T23:00:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"Research Involvement and Engagement","date":"2021-02-09T06:31:48+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"research-involvement-and-engagement","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"riae","sideBox":"Learn more about [Research Involvement and Engagement](http://researchinvolvement.biomedcentral.com/)","snPcode":"40900","submissionUrl":"https://submission.nature.com/new-submission/40900/3","title":"Research Involvement and Engagement","twitterHandle":"@MedicalEvidence","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"913d14f4-7dce-46e3-aaa5-f7157a671275","owner":[],"postedDate":"February 23rd, 2021","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":2585670,"name":"Health Economics \u0026 Outcomes Research"},{"id":2585671,"name":"Health Policy"}],"tags":[],"updatedAt":"2021-06-09T16:43:20+00:00","versionOfRecord":[],"versionCreatedAt":"2021-02-23 20:49:48","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-226630","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-226630","identity":"rs-226630","version":["v1"]},"buildId":"omnImTCwR2MFx8CMYfrG7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}