Human Factors Validation Study of an Artificial Neural Network‑based Preoperative Decision‑support Tool for Noninvasive Lymph Node Staging (NILS) in Women with Primary Breast Cancer (ISRCTN99301435)

Human Factors Validation Study of an Artificial Neural Network‑based Preoperative Decision‑support Tool for Noninvasive Lymph Node Staging (NILS) in Women with Primary Breast Cancer (ISRCTN99301435) | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Human Factors Validation Study of an Artificial Neural Network‑based Preoperative Decision‑support Tool for Noninvasive Lymph Node Staging (NILS) in Women with Primary Breast Cancer (ISRCTN99301435) Anna Allfelt, Pär Ola Bendahl, Looket Dihge, Mattias Ohlsson, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6335418/v2 This work is licensed under a CC BY 4.0 License Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Abstract Background The integration of clinical decision support tools in medical practice is challenging and must be carefully undertaken, especially in cancer management. Noninvasive Lymph Node Status (NILS) is a web-based tool designed to estimate the probability of healthy axillary lymph nodes in female patients with breast cancer scheduled for primary surgery. The aim was to identify barriers to NILS adoption in a clinical setting and assess whether intended users can operate the tool without significant errors or difficulties. Additionally, the study aimed to evaluate the appropriateness of result interpretation for decisions to abstain from sentinel lymph node biopsy (SNLB) and measure overall user satisfaction with the tool. Methods This mixed-methods multicenter on-site qualitative human factor validation study was conducted in a simulated clinical environment, replicating both real-world physical and digital conditions. Based on the identified target user population for the NILS model, twenty physicians comprised the cohort. The study used simulated clinical cases, the System Usability Scale (SUS), and the After-Scenario Questionnaire (ASQ) to evaluate usability and satisfaction. An oral interview was conducted after the evaluations. The study followed a structured protocol, with distinct roles assigned to the test participants, leader, and observer. Results Twenty physicians from four hospitals, with a median of 9.5 years of specialist practice, participated. Most participants were surgeons (75%, N = 15), while the remaining 25% (N = 5) were oncologists. Usability scores were high, with a mean SUS score of 89.5 (“excellent”) and a mean ASQ score of 6.3 (Likert scale 1–7). Several interface challenges were identified, including difficulty in locating the reset and information buttons, the ability to enter values outside valid ranges (age and tumor size), and the risk of unintentionally modifying the entered values while scrolling. Conclusions This study identified the key usability factors, barriers, and facilitators affecting the NILS model implementation. Physicians found the NILS interface easy to use and valued the presented risk estimates of healthy axillary lymph node status in their decisions to perform or abstain from SLNB. Redesign initiatives are ongoing. An iterative process of addressing usability in a clinical setting is crucial for successful implementation. Trial Registration ISRCTN99301435 General Surgery Breast neoplasm Staging Axillary lymph nodes Decision aid Sentinel lymph node biopsy Human factors validation study Figures Figure 1 Introduction The introduction of decision support tools in clinical medicine is challenging and must be carefully undertaken, especially in critical situations such as cancer management. The consideration of human factors, such as usability, in a clinical setting is crucial for successful implementation. Breast cancer is the most common cancer affecting women, with 2.3 million new cases worldwide annually.[ 1 ] The majority of patients are diagnosed with early-stage breast cancer, and only 20–30% present with nodal metastases in the axilla—a rate that has been declining during the last decades.[ 2 , 3 ] As a tool to support adjuvant treatment recommendations, the sentinel lymph node biopsy (SLNB) procedure is the gold standard for nodal staging in patients with clinically node-negative breast cancer (cN0).[ 4 ] However, recent de-escalation approaches have questioned the necessity of SLNB for all patients, as suggested by the American Society of Clinical Oncology (ASCO) guidelines presented in 2021.[ 5 ] Supporting this notion, the randomized non-inferiority SOUND trial demonstrated equal 5-year distant disease-free survival in patients with early cN0 breast cancer with small tumors (≤ 2 cm) randomized to either SLNB or its omission.[ 6 ] The randomized non-inferiority INSEMA trial demonstrated that omitting SLNB is noninferior to performing SLNB, even in cases involving larger tumors (< 5 cm). The trial showed equivalent 6-year invasive disease-free survival rates in the study arms.[ 7 ] Additionally, the ASCO guidelines support a case-by-case evaluation of omitting SLNB, as initially proposed by the authors of the Choosing Wisely guidelines.[ 5 ] The Noninvasive Lymph Node Status (NILS) model is a web-based tool designed to estimate the probability of healthy axillary lymph nodes in women with primary T1-2 invasive breast cancer with cN0 scheduled for primary surgery.[ 8 ] By using preoperative patient data and tumor characteristics, the machine learning-based NILS-algorithm provides a noninvasive prediction distinguishing node negative (N0) from node positive (N+) with a cut-off acknowledging a false-negative rate of 10% which is clinically accepted for the standard SLNB-technique.[ 9 , 10 ] The NILS model, therefore, provides support to attending physicians in deciding whether to perform or abstain from SLNB during primary breast cancer surgery. Introduced in 2019, the NILS model has been validated across geographical and temporal cohorts, in addition to when utilizing preoperatively-available data only.[ 11 , 12 ] A health-economic decision-analytic model demonstrated that implementing the NILS model could lead to significant cost reductions and potential overall health benefits, particularly for patients undergoing breast-conserving surgery.[ 13 ] The NILS model is not yet CE-marked and is not available on the market. This study aimed to identify barriers to the NILS model use in a simulated clinical setting, assess whether intended users could operate the NILS model without significant errors or difficulties, evaluate the appropriateness of result interpretation for decisions to abstain from SLNB, and measure overall user satisfaction with the tool. Methods The NILS calculator was evaluated in this premarket mixed-methods human factor validation test under simulated conditions. Only the calculator interface was included in the scope of this study, while access and information pages were not. Before data collection, the study was registered in the ISRCTN registry (ISRCTN99301435, registration date 15th Nov, 2024). The Consolidated Criteria for Reporting Qualitative Research checklist was used in this qualitative study.[ 14 ] The NILS model - user interface The technical details of the NILS algorithm and the validation processes have been published elsewhere.[ 8 , 11 , 12 ] Herein, the usability aspects are described. Graphical representation of device and user interface The device is a webpage ( https://nils.cec.lu.se/ ) that is accessible via a browser on a computer, tablet, or smartphone. Upon logging in, users are greeted by an information page regarding the NILS model, which includes disclaimer information. The calculator can be accessed by clicking the “To calculator” button or by selecting “Calculator” from the menu. The menu also includes options for additional information about the NILS model, details about the research and the team, disclaimers, and contact details. The calculator interface is divided into two sections: the left side for entering patient and tumor characteristics and the right side for displaying the results, which include the probability of healthy lymph nodes and a cut-off point (Fig. 1 A and B). After performing the calculation, the results can be copied to the clipboard and pasted onto other documents such as the patient’s medical records. The vendor of the interface logs all inputs and outputs. Study population and selection The study´s target user population comprised licensed and board-certified medical physicians specializing in general surgery or oncology, representing the intended users. Despite the variations in medical specialties, the participants were considered a unified user group for the study. Given that the NILS model is not yet available on the market, the participants had no prior experience with it, which aligns with the study’s design objectives. Number of test participants According to the U.S. Food and Drug Administration guidance on Human Factors Validation,[ 15 , 16 ] 20 test participants can identify at least 95% of usability problems, with an average detection rate of 98.5%. Therefore, we considered the number of test participants sufficient. Accessible but diverse study sites were invited to participate in the usability study through oral and e-mail invitations that included a brief explanation of the objectives of the study. Test participants were offered a small symbolic compensation for their participation. Study design and setting We conducted a mixed-method experimental observational study involving user scenarios in a simulated on-site real-world environment at the respective hospitals of the test participants. This approach allowed the test leader and observer to closely monitor and analyze user interactions and behaviors under controlled, yet realistic conditions (Fig. 1 C). A pilot session was conducted before the start of the study. The test participants received a brief introduction to the test session from the test leader to explain how the test would be conducted. A printed copy of the instructions for use (Supplement 1) was provided and they were encouraged to read them before they started the test. They had access to the instructions for use throughout the test, mimicking real-world use of the device. Additionally, the test participants received the test protocol, which included a demographic questionnaire, five simulated cases (Supplement 1), and a response form. They were required to record the calculated probability, interpret the smoothed histogram, and select the appropriate clinical pathway based on the NILS model’s results and other available information. The options included: “Consider omitting SLNB,” “Consider performing SLNB,” “Definitely perform SLNB,” and “NILS cannot assist in making the clinical decision.” Suggested treatment decisions were not evaluated in this study. Lastly, in each session, the overall usability of the device was evaluated through the validated System Usability Scale (SUS), which is one of the most commonly used usability assessment questionnaires,[ 17 , 18 ] and satisfaction was assessed using the After-Scenario Questionnaire (ASQ), in which test participants responded using Likert Scales from 1–5 and 1–7, respectively.[ 19 ] Written feedback was obtained. Oral interview questions were administered after the interface use if the test leader or observer noticed that the participant had completed a task incorrectly, nearly made an error, or showed use difficulties. Each test session was scheduled for one hour, including the validated questionnaires and oral interview. While the test participants performed each scenario, both the test leader and observer recorded performance on each task in two separate predefined protocols, as: “correct use,” ”use error,” “close call,” or “use difficulty” (Supplement 2). Subsequently, the tasks were evaluated as either a “pass“ or “fail.” Discrepancies between the test leader’s and observers’ recordings were resolved through discussion and documented as uncertainty in the report. Summary of previous usability evaluations A formative evaluation was conducted with six participants who responded to questions after using the calculator, none of whom reported any serious usability issues. However, during testing, concerns were raised regarding the potential for inaccurate data entry or misinterpretation of the results. These concerns prompted enhancements to the user interface, such as clearly marking the position of the tumor in the breast and using a different font color to highlight missing data. Briefly, the most hazardous potential harm to be avoided by a safe device design is a false-estimated indication of benign sentinel lymph nodes. A list of identified potential use errors is provided in Supplement 3. Identification and description of critical tasks Critical tasks were identified after considering previously identified risks in the risk analysis: reset the calculator, enter clinical data, enter mammography data, enter core biopsy data, and identify the appropriate clinical pathway (considering the results of the NILS calculator in combination with all other available information of the case).[ 20 ] The simulated cases were created to ensure that all critical tasks were performed during the test and that the test conditions were sufficiently realistic to represent actual conditions of use.[ 21 ] Predefined acceptance criteria for evaluation Since the NILS model has not yet been released into the market and has limited exposure to intended users, the team anticipated potential usability issues. To assess performance, an acceptance criterion was set at ≥ 90% successful task completion, evaluated on a pass/fail basis for each task. Data analysis Descriptive statistics were calculated for test participants’ characteristics. SPSS Statistics version 28 (IBM Corp., Armonk, NY, USA) was used for all statistical analyses. The SUS score was calculated[ 22 ] (mean and median) and interpreted according to the Adjective Rating Scale.[ 23 ] In terms of qualitative data, notes taken during test sessions by the test leader and observer were rigorously reviewed during data analysis and systematically categorized to gather comments and suggestions of redesign of the interface. Results In this multisite simulated test session-based usability study with intended users, 20 physicians from four hospitals participated, including 9 (45%) within University Hospitals and 11 (55%) within Regional Hospitals covering three health care regions. Of the twenty participants, 13 (65%) were women and 11 (55%) were < 50 years. The majority were surgeons, comprising 75% (N = 15) of the participants, while the remaining 25% (N = 5) were oncologists (Table 1 ). Table 1 Self-reported characteristics of usability testing participants N (%) Physician characteristics Usability testing participants (N = 20) Specialty Surgeon 15 (75%) Oncologist 5 (25%) Completed years at specialist (years), median (IQR) 9.5 (7.0-–17.3) Use of any tool/system for prediction in clinical work Yes 14 (70%) No 6 (30%) - If, yes – which? N = 14, Predict (100%) Workplace (Hospital) University Hospital 9 (45%) Regional Hospital 1 2 (10%) Regional Hospital 2 5 (25%) Regional Hospital 3 4 (20%) Highest academic degree MD 12 (60%) MD + PhD 6 (30%) Associate Professor 1 (5%) Professor 1 (5%) Gender Male 7 (35%) Female 13 (65%) Other - Do not want to say - Age (years), median (IQR) 48.5 (43.3–52.0) Abbreviations: IQR, interquartile range; MD, Doctor of Medicine; PhD, Doctor of Philosophy Task completion The three critical tasks (1–6 data entering points/task) of entering clinical, mammography, and core needle biopsy data, respectively, were all above the predefined acceptance criteria of ≥ 90% and thus defined as passed (Table 2 and Supplement 4). The critical task of finding the reset button was just below the predefined acceptance criteria and was defined as a failure. The task of performing the calculation (i.e., conditional on the accuracy of entered data) was just below the predefined acceptance criteria for the first scenario only. Table 2 Frequency of observed outcome and result, analysis per task Description Observed outcome, N (%) Result, N (%) Scenario Correct use Use error Use difficulty Close call Pass Fail 1 Reset the calculator* N/A N/A N/A N/A N/A N/A Enter clinical data 20 (100) 0 (0) 0 (0) 0 (0) 20 (100) 0 (0) Enter mammography data 18 (90) 1 (5) 1 (5) 0 (0) 19 (95) 1 (5) Enter core biopsy data 18 (90) 2 (10) 0 (0) 0 (0) 18 (90) 2 (10) Perform calculation 17 (85) 3 (15) 0 (0) 0 (0) 17 (85) 3 (15) Select the appropriate clinical pathway 16 (80) 2 (10) 1 (5) 1 (5) 18 (90) 2 (10) 2 Reset the calculator 17 (85) 3 (15) 0 (0) 0 (0) 17 (85) 3 (15) Enter clinical data 20 (100) 0 (0) 0 (0) 0 (0) 20 (100) 0 (0) Enter mammography data 19 (95) 0 (0) 1 (5) 0 (0) 20 (100) 0 (0) Enter core biopsy data 18 (90) 2 (10) 0 (0) 0 (0) 18 (90) 2 (10) Perform calculation 18 (90) 2 (10) 0 (0) 0 (0) 18 (90) 2 (10) Select the appropriate clinical pathway 19 (95) 1 (5) 0 (0) 0 (0) 19 (95) 1 (5) 3 Reset the calculator 17 (85) 3 (15) 0 (0) 0 (0) 17 (85) 3 (15) Enter clinical data 4 (20) 0 (0) 16 (80) 0 (0) 20 (100) 0 (0) Select the appropriate clinical pathway 20 (100) 0 (0) 0 (0) 0 (0) 20 (100) 0 (0) 4 Reset the calculator 17 (85) 3 (15) 0 (0) 0 (0) 17 (85) 3 (15) Enter clinical data 19 (95) 1 (5) 0 (0) 0 (0) 19 (95) 1 (5) Enter mammography data 20 (100) 0 (0) 0 (0) 0 (0) 20 (100) 0 (0) Enter core biopsy data 19 (95) 1 (5) 0 (0) 0 (0) 19 (95) 1 (5) Perform calculation 18 (90) 2 (10) 0 (0) 0 (0) 18 (90) 2 (10) Select the appropriate clinical pathway 19 (95) 1 (5) 0 (0) 0 (0) 19 (95) 1 (5) 5 Reset the calculator 17 (85) 3 (15) 0 (0) 0 (0) 17 (85) 3 (15) Enter clinical data 20 (100) 0 (0) 0 (0) 0 (0) 20 (100) 0 (0) Enter mammography data 20 (100) 0 (0) 0 (0) 0 (0) 20 (100) 0 (0) Enter core biopsy data 18 (90) 2 (10) 0 (0) 0 (0) 18 (90) 2 (10) Perform calculation 18 (90) 2 (10) 0 (0) 0 (0) 18 (90) 2 (10) Select the appropriate clinical pathway 20 (100) 0 (0) 0 (0) 0 (0) 20 (100) 0 (0) *N/A: Resetting the calculator was not applicable in scenario 1. Results presented in plain font indicates a task result categorized as "pass" (meeting or exceeding the predefined acceptance criteria of ≥ 90%), whereas result presented in bold font indicates a task result categorized as "fail" (falling below the predefined acceptance criteria of ≥ 90%). The critical task of selecting the appropriate clinical pathway (based on the calculation results and case information) met or exceeded the predefined acceptance criteria, and was successfully completed for all five scenarios. Usability and satisfaction Test participants found the NILS tool to be highly user-friendly; the SUS score averaged 89.5 and 89.4 points (mean) on a question- and test participant-based analysis, respectively (Supplements 5 and 6). Moreover, the NILS model yielded high overall satisfaction with an averaged mean ASQ score of 6.3 (maximum 7) (Supplement 7). Usability domains The test participants explained their interpretation of the NILS histogram following each case and declared a good understanding of the presented results (Table 3 ). However, they requested more detailed information regarding what the histogram represented, and how to interpret it in a clinical setting. Table 3 Illustrative comments about the NILS model Theme Comment General impression “Easy to use with an appealing layout.” “Pleasant number of variables.” “The results are aligned with my clinical intuition.” Barriers to the NILS model use “I am not completely comfortable with the presented histogram and the suggested threshold. I would prefer a recommendation from the calculator regarding omission or not of SLNB.” “I'd like a simple explanation of what the imputed values represent and how they will impact the risk estimation.” Clinical content “Curious to why tumor grade and HER2-status are not part of the calculator. The inclusion of these parameters would, in my opinion, increase the credibility of the model.” “For multifocal tumors, what if the smaller ones have a more worrisome profile? Then I would decide on SLNB depending on these characteristics.” “For grade 3 tumors I would always opt for SLNB. That is what my clinical experience tells me.” “It seems more intuitive to be presented the risk of metastatic lymph nodes.” “I'd prefer an overview of contraindications for the NILS calculator, such as neoadjuvant treatment.” Interpretation of the histogram “The risk is too high for metastatic lymph node to safely omit SNLB, the probability for benign lymph nodes is below the threshold value, can´t omit SLNB.” “The calculated value is above the threshold value – probably benign lymph nodes. There is a low risk for metastatic lymph nodes, the consideration to omit SNLB can be discussed with the patient.” “The probability of benign lymph nodes is just above the threshold to consider omitting SLNB. Since the calculated value is very close to the threshold, I would have done SLNB.” “The calculated probability is clearly below threshold value; it is not appropriate to omit SLNB.” Comments are paraphrased to preserve the original posters' anonymity. Translation from the original language spoken at the sessions (Swedish) to English by the researchers. Several redesign elements were identified, addressing both technical components and educational/informational challenges (Table 4 ). A recurring concern was the exclusion of the histological tumor grade and human epidermal growth factor 2 (HER2) status, which are clinically important for recommending neoadjuvant or adjuvant treatments. The term “multifocality” caused some hesitation, as participants were concerned that smaller tumors, excluded from calculations, might have less favorable characteristics than the largest tumor, requiring additional consideration. The reset button was difficult to locate because of its inconspicuous black color and placement at the bottom of the calculator, which became hidden when users viewed the resulting histogram at the top. In addition, users can accidentally scroll within the last cell of an entered value, often altering the Ki67 value. This led one participant to enter an incorrect value, resulting in a miscalculated probability of benign lymph nodes. Furthermore, the information buttons were often missed by the participants and hence were rarely used. Table 4. Major design requirements derived from the usability study with intended users Source Issue Requirements It is possible to enter patients’ age outside of the accepted range. When the age is outside the accepted range the cell becomes yellow, however, still it is possible to enter a value. If notice is not taken, unnecessary work is done inserting all other variables. It will, however, not be possible to make a NILS calculation. A clear notice that inserted value is out of range/not possible to insert value out of range. The NILS calculator is applicable to T1-2 breast cancer. However, in the calculator it is possible to enter a tumor size of up to 90 mm. If notice is not taken that tumor size is out of range, unnecessary work is done inserting all other variables. It will, however, not be possible to make a NILS calculation. A clear notice that inserted value is out of range/not possible to insert value out of range. The placement of the reset button is counterintuitive in the web interface, making it difficult to use correctly. The reset button is placed at the bottom of the calculator. When making the calculation the user is directed automatically to the histogram, which is displayed in the upper part of the calculator and the reset button then falls out of sight. It would enable correct use to place the reset button at the top of the calculator, in the eyeline of the user when entering data for a subsequent case. Further, the black colored reset button does not draw attention, so another color is preferable. When scrolling down the webpage to reach the calculate button, it is possible to accidentally scroll within the last cell, most commonly the Ki67 value for the largest invasive tumor. The scrolling can alter the inserted value and if not identified and corrected cause the calculated value to be wrong. Ensure that scrolling cannot alter an entered value. Uncertainty about how to handle multifocality and central tumor, the information provided by the calculator was often overlooked. When struggling with the decision on which tumor variables to enter in case of multifocality and how a central tumor is defined, the information button was seldom identified. To enhance the ability to find the information button, it needs to be more clearly marked. In the histogram, the cut-off is marked with a stretched line and the cut-off value is presented in text below. The calculated percentages are marked as a filled line with calculated value presented at the top within the histogram figure. The cut-off value can be misinterpreted as the calculated value if the y-axis and the text below only is viewed. The cut-off value should also be visualized with a number in addition to the line in the histogram. Moreover, in the text below the histogram, the resulting percentage and not the cut-off value should be emphasized. The histogram is presented without declaring what the x-axis and y-axis represent. For correct interpretation of the presented histogram/result, proper labelling is warranted. Add the following descriptions for the x- and y-axes, respectively: “Estimated probability of healthy lymph nodes in the axilla” and “Fraction of patients in the cohort used for model development”. The information on applicable use population, e.g. exclusion of neoadjuvant-treated patients, is not easily found when working with the NILS calculator. If disclaimers are not noted, the NILS model can be used by mistake on the wrong patient population. In the current version, “Disclaimers” can be found in the ribbon at the top of the calculator interface. It is accessible only by navigating to an adjacent webpage. To improve usability, consider including a heading within the calculator page stating “The NILS calculator is not intended for use in every scenario.”, followed by a link to “Instructions for Use.” Discussion Using an experimental mixed-methods observational study design with user scenarios in a simulated on-site environment, we found broad support among licensed medical physicians for the NILS model. This tool provides risk estimates for healthy sentinel lymph node status, aiding attending physicians in deciding whether or not SLNB can be abstained. The test participants identified several usability concerns, as well as barriers and facilitators to the implementation and widespread use of the NILS model. These insights have prompted future iterative design modifications. Principal findings Many new insights were gained during this usability study. Several aspects of the existing interface posed challenges for the test participants. These included difficulties in locating the reset and information buttons, as well as technical issues, such as the potential to enter values outside the applicable ranges (e.g., for age and tumor size). In addition, the risk of unintentionally altering the entered value during scrolling was detected. Furthermore, a learning curve was evident. Data entry and calculations were the most challenging in scenario 1 and became progressively easier in subsequent scenarios. This is reflected in the only failure of the “perform calculation” task, which occurred during the first scenario. The accomplished SUS score, which is recommended to be regarded as a percentile and not as a percentage, was categorized as “excellent” (score > 85). The slight discrepancy between the per-question and per-participant analyses was due to one slightly skeptical participant who was dissatisfied with the model's role as a clinical support tool. This participant specifically sought a definitive clinical decision regarding whether to perform SLNB. A high ASQ score indicated that the test participants found it easy to complete the tasks in the NILS calculator, were satisfied with the time taken to complete the tasks and found the support information adequate. Comparison to similar devices Benchmark devices for the NILS model have not been identified. A tool known as Predict, which estimates the survival effect of selected adjuvant therapies for breast cancer, has been widely utilized by both patients and healthcare professionals.[ 24 ] A completed usability study on Predict has provided six key recommendations that have been considered in the planning of this NILS model usability study. As for the NILS model, which is designed to facilitate decision-making, Predict serves as an adjunct to recommended adjuvant therapies according to established guidelines, rather than as a stand-alone decision tool. However, a significant distinction between the NILS model and Predict lies in their functionalities. Unlike Predict, which compares oncological outcomes and the advantages and disadvantages of different treatment options for patients with specific characteristics, the NILS model aids in the decision to abstain from or perform SLNB. Clinical contextualization Comprehending the underlying rationale of the model predictions is essential for building trust in the results calculated using the NILS model. Therefore, we suggest, in accordance with Farmer et al.,[ 24 ] that the provision of contextual information making the calculator´s results useful in their clinical setting is important for a positive attitude of the intended users, that is, health care professionals. In this study, we identified the need for enhanced assistance in interpreting histograms, including explaining the rationale behind the cutoff value. There was also a request for a comprehensive explanation of the inclusion and exclusion criteria for the relevant patient population to deepen knowledge of why certain variables, such as HER2 status and histologic tumor grade, were not included in the NILS model. Technological barriers Mistrust and unfamiliarity with machine learning models are frequently identified as significant barriers to their integration into clinical practice.[ 25 ] Although interpreting the resulting histogram was challenging, none of the test participants showed technological mistrust towards the machine learning model behind the NILS calculator. This trust in the model is anticipated to be generalizable to intended users, given the careful consideration of the test participants' representativeness of future intended users. Strengths and limitations The study was rigorously planned and executed through thorough data control. All endpoints and acceptance criteria were registered before study initiation. Conducted with the intended users, including clinical surgeons and oncologists with representative post-specialist clinical experience, the study was conducted in a simulated on-site real-world physical and digital environment within three healthcare regions. This approach enhanced the generalizability of the results to actual clinical practice. Having both a test leader and observer ensured strict adherence to the study protocol, guaranteeing objectivity throughout the process. There were no requirements for user training before the device was used. However, in a real-world scenario, users might take varying amounts of time to read the “Instruction for Users”; some might read it thoroughly from front to back, while others might only glance at it before starting to use the device, depending on their personality type and the time available. Moreover, we hypothesized that the NILS model could be effectively utilized in a multidisciplinary conference setting, where physicians from various specialties discuss and formulate treatment plans for specific patients, considering the available data not included in the NILS model. Although this particular clinical scenario was not replicated in this usability study, we anticipate that similar criticisms and evaluations will emerge. Future perspective Several key redesign elements were identified, encompassing technical aspects and educational and informational issues. Collectively, these changes will enhance the usability and workflow, thereby preparing the NILS model for clinical use as a decision-support tool. The proposed improvements will be implemented and evaluated with a selected group of test participants, with a specific focus on the identified and redesigned issues. In a post-market setting, the user interface will be iteratively updated and evaluated as needed using information gained from the post-market process. Conclusion This qualitative premarket usability study of the NILS model, conducted through test sessions with simulated clinical cases, involved intended users, that is, surgeons and oncologists, in a real-world clinical setting. This study identified the key usability factors, barriers, and facilitators influencing implementation. Physicians found the NILS model intuitive and valued its presented risk estimates for healthy axillary lymph node status in their decision to perform or abstain from SLNB. These findings will directly inform the refinement of the NILS model, ensuring improved usability and clinical integration. Abbreviations ASQ After-Scenario Questionnaire BCS breast-conserving surgery HER2 human epidermal growth factor receptor 2 NILS noninvasive lymph node staging SLNB sentinel lymph node biopsy SUS System Usability Scale Declarations Ethics approval, consent to participate and consent of publication This usability study was not a clinical study, and did not require ethical approval from the Swedish Ethical Review Authority. The usability study was approved by the Head of Research in Region Skåne (2 nd Apr, 2024) regarding the intended conduct of the study, and Region Skåne is thus the legal authority for the study. All participants in the study provided written informed consent. Competing interests The authors declare that they have no competing interests. Availability of data and materials The raw datasets are available from the corresponding author on reasonable request. Declaration of generative AI and AI-assisted technologies in the writing process During the preparation of this manuscript, the authors used Copilot for language editing and generated an initial draft of Figure 1C. Following the use of this tool, the authors thoroughly reviewed and revised the content as necessary and take full responsibility for the final content of this publication. Acknowledgments The authors express their gratitude to the test participants at Skåne University Hospital, Kristianstad Regional Hospital, Karlskrona Regional Hospital, and Växjö Regional Hospital. Special thanks go to research nurse Helena Erixon for her exceptional contributions. Furthermore, we thank MEDOS for their valuable cooperation in the planning of this study and development of the test protocols. Funding This work was supported by grants from the Lund University (Sweden), South Swedish Health Care Region (Sweden), Governmental Funding of Clinical Research within the National Health Service Sweden (ALF young researcher Ida Skarping), Erling Persson Foundation (Sweden), Skåne University Hospital Funds, Swedish Breast Cancer Funding, Sjöberg Foundation, and Vetenskapsrådet [Swedish Research Council] (Sweden) (external review). In this academic study, the funding resources had no role in the study design, data collection, analyses, data interpretation, writing of the manuscript or the decision to submit the manuscript for publication. Authors' contributions Mrs. A. Allfelt and Dr. I. Skarping had full access to all data in the study and take responsibility for the integrity of the data and accuracy of the data analysis. Dr. I Skarping is the senior author responsible for study oversight. Concept and design: Rydén, Skarping Acquisition, analysis, or interpretation of data: Allfelt, Bendahl, Dihge, Ohlsson, Rydén, and Skarping Drafting of the manuscript : Allfelt, Skarping Critical review of the manuscript for important intellectual content: Allfelt, Bendahl, Dihge, Ohlsson, Rydén, and Skarping Statistical analysis: Allfelt, Skarping Obtained funding: Rydén, Skarping Administrative, technical, or material support: Allfelt, Rydén, Skarping Supervision : Rydén, Skarping References Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F (2021) Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin 71(3):209–249 Cady B, Stone MD, Schuler JG, Thakur R, Wanner MA, Lavin PT (1996) The new era in breast cancer. Invasion, size, and nodal involvement dramatically decreasing as a result of mammographic screening. Arch Surg 131(3):301–308 Mansel RE, Fallowfield L, Kissin M, Goyal A, Newcombe RG, Dixon JM, Yiangou C, Horgan K, Bundred N, Monypenny I et al (2006) Randomized multicenter trial of sentinel node biopsy versus standard axillary treatment in operable breast cancer: the ALMANAC Trial. J Natl Cancer Inst 98(9):599–609 Veronesi U, Paganelli G, Viale G, Luini A, Zurrida S, Galimberti V, Intra M, Veronesi P, Robertson C, Maisonneuve P et al (2003) A randomized comparison of sentinel-node biopsy with routine axillary dissection in breast cancer. N Engl J Med 349(6):546–553 Brackstone M, Baldassarre FG, Perera FE, Cil T, Chavez Mac Gregor M, Dayes IS, Engel J, Horton JK, King TA, Kornecki A et al (2021) Management of the Axilla in Early-Stage Breast Cancer: Ontario Health (Cancer Care Ontario) and ASCO Guideline. J Clin Oncol 39(27):3056–3082 Gentilini OD, Botteri E, Sangalli C, Galimberti V, Porpiglia M, Agresti R, Luini A, Viale G, Cassano E, Peradze N et al (2023) Sentinel Lymph Node Biopsy vs No Axillary Surgery in Patients With Small Breast Cancer and Negative Results on Ultrasonography of Axillary Lymph Nodes: The SOUND Randomized Clinical Trial. JAMA Oncol 9(11):1557–1564 Reimer T, Stachs A, Veselinovic K, Kühn T, Heil J, Polata S, Marmé F, Müller T, Hildebrandt G, Krug D et al (2024) Axillary Surgery in Breast Cancer - Primary Results of the INSEMA Trial. N Engl J Med Dihge L, Bendahl PO, Skarping I, Hjärtström M, Ohlsson M, Rydén L (2023) The implementation of NILS: A web-based artificial neural network decision support tool for noninvasive lymph node staging in breast cancer. Front Oncol 13:1102254 Krag DN, Anderson SJ, Julian TB, Brown AM, Harlow SP, Ashikaga T, Weaver DL, Miller BJ, Jalovec LM, Frazier TG et al (2007) Technical outcomes of sentinel-lymph-node resection and conventional axillary-lymph-node dissection in patients with clinically node-negative breast cancer: results from the NSABP B-32 randomised phase III trial. Lancet Oncol 8(10):881–888 Pesek S, Ashikaga T, Krag LE, Krag D (2012) The false-negative rate of sentinel node biopsy in patients with breast cancer: a meta-analysis. World J Surg 36(9):2239–2251 Hjärtström M, Dihge L, Bendahl PO, Skarping I, Ellbrant J, Ohlsson M, Rydén L (2023) Noninvasive Staging of Lymph Node Status in Breast Cancer Using Machine Learning: External Validation and Further Model Development. JMIR Cancer 9:e46474 Skarping I, Ellbrant J, Dihge L, Ohlsson M, Huss L, Bendahl PO, Rydén L (2024) Retrospective validation study of an artificial neural network-based preoperative decision-support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer (ISRCTN14341750). BMC Cancer 24(1):86 Skarping I, Nilsson K, Dihge L, Fridhammar A, Ohlsson M, Huss L, Bendahl PO, Steen Carlsson K, Rydén L (2022) The implementation of a noninvasive lymph node staging (NILS) preoperative prediction model is cost effective in primary breast cancer. Breast Cancer Res Treat 194(3):577–586 Tong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 19(6):349–357 U.S. Food and Drug Administration: Applying Human Factors and Usability Engineering to Medical Devices. Guidance for Industry and Food and Drug Administration Staff; (2016) https://www.fda.gov/media/80481/download Faulkner L (2003) Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 35(3):379–383 Lewis JR (2018) The System Usability Scale: Past, Present, and Future. Int J Human–Computer Interact 34(7):577–590 Hyzy M, Bond R, Mulvenna M, Bai L, Dix A, Leigh S, Hunt S (2022) System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis. JMIR Mhealth Uhealth 10(8):e37290 Lewis JR (1995) IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. Int J Human–Computer Interact 7(1):57–78 International Electrotechnical Commission: IEC TR 62366-2:2016 Medical devices - Part 2: Guidance on the application of usability engeneering to medical devices (Edition 1.0) (2016) ; https://www.iso.org/standard/69126.html Russ AL, Saleem JJ (2018) Ten factors to consider when developing usability scenarios and tasks for health information technology. J Biomed Inf 78:123–133 Brooke J (1996) SUS-A quick and dirty usability scale. Usability evaluation Ind 189(194):4–7 Bangor A, Kortum P, Miller J (2009) Determining what individual SUS scores mean: Adding an adjective rating scale. J usability Stud 4(3):114–123 Farmer GD, Pearson M, Skylark WJ, Freeman ALJ, Spiegelhalter DJ (2021) Redevelopment of the Predict: Breast Cancer website and recommendations for developing interfaces to support decision-making. Cancer Med 10(15):5141–5153 Asan O, Bayrak AE, Choudhury A (2020) Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians. J Med Internet Res 22(6):e15154 Additional Declarations The authors declare no competing interests. Supplementary Files SupplementaryMaterial1AD250329.pdf SupplementaryMaterial27250329.docx Cite Share Download PDF Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6335418","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":437900016,"identity":"5ce6f3a2-5cd0-4a3c-b904-13490cb28c60","order_by":0,"name":"Anna Allfelt","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8klEQVRIiWNgGAWjYPACCxn2BuYGhg8MDIwN7MRpkeDhOcDY2DgDpIWZFC3NPMRo4Z/de/ADQw1QC3tj+2PbHBvZBmYeA8YfFXiMv3MuWYLhGFALz8HG5txtacYgLcw8Z/BYcyPHQIKBTYLHXiIRpOVwIlgLYxtuHfI3cox/MPwD2iL/sLHZctt/sBbGn/9wazG4kWMmwdgG1CIB9D7jtgNgLQy8Dbi1GAK1WCT2gfyS2Dizd1uycRszW8FhnmO4tcgBHXbjwzcbOR72wwc+/NxmJ9vP3rzx4Y8aPN4HgQRkDhsQHyCgYRSMglEwCkYBAQAAUtJJd695K5gAAAAASUVORK5CYII=","orcid":"https://orcid.org/0009-0002-7889-9821","institution":"Lunds University","correspondingAuthor":true,"prefix":"","firstName":"Anna","middleName":"","lastName":"Allfelt","suffix":""},{"id":437900017,"identity":"3350aecd-eabc-4645-b765-8a6ab06e7352","order_by":1,"name":"Pär Ola Bendahl","email":"","orcid":"https://orcid.org/0000-0001-8862-1845","institution":"Lunds University","correspondingAuthor":false,"prefix":"","firstName":"Pär","middleName":"Ola","lastName":"Bendahl","suffix":""},{"id":437900018,"identity":"0ea1d510-a679-490f-9eaa-5268d3e5ce61","order_by":2,"name":"Looket Dihge","email":"","orcid":"https://orcid.org/0000-0002-7932-3982","institution":"Lunds University","correspondingAuthor":false,"prefix":"","firstName":"Looket","middleName":"","lastName":"Dihge","suffix":""},{"id":437900019,"identity":"1a9543e8-27b3-4a9d-bee9-1c34c3800bcc","order_by":3,"name":"Mattias Ohlsson","email":"","orcid":"https://orcid.org/0000-0003-1145-4297","institution":"Lunds University","correspondingAuthor":false,"prefix":"","firstName":"Mattias","middleName":"","lastName":"Ohlsson","suffix":""},{"id":437900020,"identity":"e32ec943-7a1d-4e76-b1d1-a3feb3f9dfcd","order_by":4,"name":"Lisa Rydén","email":"","orcid":"https://orcid.org/0000-0001-7515-3130","institution":"Lunds University","correspondingAuthor":false,"prefix":"","firstName":"Lisa","middleName":"","lastName":"Rydén","suffix":""},{"id":437900021,"identity":"7d800b13-1b96-4bf6-8c0d-d37e745b7d6a","order_by":5,"name":"Ida Skarping","email":"","orcid":"https://orcid.org/0000-0002-1265-8649","institution":"Lunds University","correspondingAuthor":false,"prefix":"","firstName":"Ida","middleName":"","lastName":"Skarping","suffix":""}],"badges":[],"createdAt":"2025-03-29 17:50:58","currentVersionCode":2,"declarations":{"humanSubjects":true,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":true,"humanSubjectConsent":true,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-6335418/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-6335418/v2","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":83849721,"identity":"f0322903-24e4-40b4-ba70-ea12189a281c","added_by":"auto","created_at":"2025-06-03 15:49:24","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":1313497,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eThe NILS model webpage intercept as seen in the usability test\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ea)\u003c/strong\u003e \u003cstrong\u003eThe calculator\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eb)\u003c/strong\u003e \u003cstrong\u003eThe output of the calculator: histogram and result\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ec)\u003c/strong\u003e \u003cstrong\u003eSchematic image of the simulated on-site real-world test sessions.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLegend 1A:\u003c/strong\u003e Users can adjust the values for “Age at diagnosis,” “Position in the breast,” and “Proliferation index, Ki67” using up or down arrows. Other fields are completed by clicking on the appropriate options corresponding to the patient details. Each field includes an information button that provides additional information regarding the parameters to be entered. The tumor characteristics “Vascular invasion,” “ER,” “PR,” and “Ki67” can be marked as “Unknown,” and the calculator will still generate a probability of benign axillary lymph nodes. Once all data has been entered, including any “Unknown” values, the user presses the “Calculate” button. The user can reset the calculator to input new values from another patient.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLegend 1B\u003c/strong\u003e: Output includes the probability of healthy lymph nodes (%) and a preset cutoff based on the false-negative rate for sentinel lymph node biopsy. The text results (including a list of inserted data points) can be copied and pasted onto other documents. The interface logs all the inputs and outputs with individual calculation id and timestamps for identification and traceability.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLegend 1C:\u003c/strong\u003e Test sessions were performed with the test leader (first author, a breast surgeon working clinically at the University Hospital test site), a test participant, and an observer in the same room. A computer or laptop already logged into the NILS webpage was provided to the participants to complete the test. The computer was the property of each hospital while the laptop was provided by the test team (Dell Latitude 5430). The observer transcribed all spoken comments made by the test participant. Discrepancies between the test leader’s and observer’s recordings were resolved through discussion and documented as uncertainty in the report. Although the NILS webpage can be accessed via a browser on computers, tablets, or smartphones, only PCs were evaluated, because this is the current standard setup in clinical settings. The image is generated with AI (Microsoft Copilot) followed by substantial modification by the researchers.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-6335418/v2/8f06d66470b51f9095b33b1b.png"},{"id":83850585,"identity":"20195786-c961-40b5-8400-1dbe25aee1be","added_by":"auto","created_at":"2025-06-03 16:05:26","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2637263,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6335418/v2/02b22db9-e5cf-477d-80a8-3e2f2c707c1f.pdf"},{"id":83849723,"identity":"5eb99b6c-5327-494a-9369-cb1354c30a51","added_by":"auto","created_at":"2025-06-03 15:49:24","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":6973228,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterial1AD250329.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6335418/v2/fb555790639ba64da8b0f25d.pdf"},{"id":83849720,"identity":"6f844103-1a3f-40b1-83f8-152912d4fe45","added_by":"auto","created_at":"2025-06-03 15:49:24","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":69675,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterial27250329.docx","url":"https://assets-eu.researchsquare.com/files/rs-6335418/v2/247848da6ed863c2e536edc7.docx"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"Human Factors Validation Study of an Artificial Neural Network‑based Preoperative Decision‑support Tool for Noninvasive Lymph Node Staging (NILS) in Women with Primary Breast Cancer (ISRCTN99301435)","fulltext":[{"header":"Introduction","content":"\u003cp\u003eThe introduction of decision support tools in clinical medicine is challenging and must be carefully undertaken, especially in critical situations such as cancer management. The consideration of human factors, such as usability, in a clinical setting is crucial for successful implementation.\u003c/p\u003e \u003cp\u003eBreast cancer is the most common cancer affecting women, with 2.3\u0026nbsp;million new cases worldwide annually.[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e] The majority of patients are diagnosed with early-stage breast cancer, and only 20\u0026ndash;30% present with nodal metastases in the axilla\u0026mdash;a rate that has been declining during the last decades.[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] As a tool to support adjuvant treatment recommendations, the sentinel lymph node biopsy (SLNB) procedure is the gold standard for nodal staging in patients with clinically node-negative breast cancer (cN0).[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] However, recent de-escalation approaches have questioned the necessity of SLNB for all patients, as suggested by the American Society of Clinical Oncology (ASCO) guidelines presented in 2021.[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] Supporting this notion, the randomized non-inferiority SOUND trial demonstrated equal 5-year distant disease-free survival in patients with early cN0 breast cancer with small tumors (\u0026le;\u0026thinsp;2 cm) randomized to either SLNB or its omission.[\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] The randomized non-inferiority INSEMA trial demonstrated that omitting SLNB is noninferior to performing SLNB, even in cases involving larger tumors (\u0026lt;\u0026thinsp;5 cm). The trial showed equivalent 6-year invasive disease-free survival rates in the study arms.[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] Additionally, the ASCO guidelines support a case-by-case evaluation of omitting SLNB, as initially proposed by the authors of the Choosing Wisely guidelines.[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eThe Noninvasive Lymph Node Status (NILS) model is a web-based tool designed to estimate the probability of healthy axillary lymph nodes in women with primary T1-2 invasive breast cancer with cN0 scheduled for primary surgery.[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] By using preoperative patient data and tumor characteristics, the machine learning-based NILS-algorithm provides a noninvasive prediction distinguishing node negative (N0) from node positive (N+) with a cut-off acknowledging a false-negative rate of 10% which is clinically accepted for the standard SLNB-technique.[\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] The NILS model, therefore, provides support to attending physicians in deciding whether to perform or abstain from SLNB during primary breast cancer surgery. Introduced in 2019, the NILS model has been validated across geographical and temporal cohorts, in addition to when utilizing preoperatively-available data only.[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e] A health-economic decision-analytic model demonstrated that implementing the NILS model could lead to significant cost reductions and potential overall health benefits, particularly for patients undergoing breast-conserving surgery.[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e] The NILS model is not yet CE-marked and is not available on the market.\u003c/p\u003e \u003cp\u003eThis study aimed to identify barriers to the NILS model use in a simulated clinical setting, assess whether intended users could operate the NILS model without significant errors or difficulties, evaluate the appropriateness of result interpretation for decisions to abstain from SLNB, and measure overall user satisfaction with the tool.\u003c/p\u003e \u003cp\u003eMethods\u003c/p\u003e \u003cp\u003eThe NILS calculator was evaluated in this premarket mixed-methods human factor validation test under simulated conditions. Only the calculator interface was included in the scope of this study, while access and information pages were not.\u003c/p\u003e \u003cp\u003eBefore data collection, the study was registered in the ISRCTN registry (ISRCTN99301435, registration date 15th Nov, 2024). The Consolidated Criteria for Reporting Qualitative Research checklist was used in this qualitative study.[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]\u003c/p\u003e"},{"header":"The NILS model - user interface","content":"\u003cp\u003eThe technical details of the NILS algorithm and the validation processes have been published elsewhere.[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e] Herein, the usability aspects are described.\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eGraphical representation of device and user interface\u003c/h2\u003e \u003cp\u003eThe device is a webpage (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://nils.cec.lu.se/\u003c/span\u003e\u003cspan address=\"https://nils.cec.lu.se/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e) that is accessible via a browser on a computer, tablet, or smartphone. Upon logging in, users are greeted by an information page regarding the NILS model, which includes disclaimer information. The calculator can be accessed by clicking the \u0026ldquo;To calculator\u0026rdquo; button or by selecting \u0026ldquo;Calculator\u0026rdquo; from the menu. The menu also includes options for additional information about the NILS model, details about the research and the team, disclaimers, and contact details.\u003c/p\u003e \u003cp\u003eThe calculator interface is divided into two sections: the left side for entering patient and tumor characteristics and the right side for displaying the results, which include the probability of healthy lymph nodes and a cut-off point (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eA and B). After performing the calculation, the results can be copied to the clipboard and pasted onto other documents such as the patient\u0026rsquo;s medical records. The vendor of the interface logs all inputs and outputs.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Study population and selection","content":"\u003cp\u003eThe study\u0026acute;s target user population comprised licensed and board-certified medical physicians specializing in general surgery or oncology, representing the intended users. Despite the variations in medical specialties, the participants were considered a unified user group for the study. Given that the NILS model is not yet available on the market, the participants had no prior experience with it, which aligns with the study\u0026rsquo;s design objectives.\u003c/p\u003e"},{"header":"Number of test participants","content":"\u003cp\u003eAccording to the U.S. Food and Drug Administration guidance on Human Factors Validation,[\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] 20 test participants can identify at least 95% of usability problems, with an average detection rate of 98.5%. Therefore, we considered the number of test participants sufficient. Accessible but diverse study sites were invited to participate in the usability study through oral and e-mail invitations that included a brief explanation of the objectives of the study. Test participants were offered a small symbolic compensation for their participation.\u003c/p\u003e"},{"header":"Study design and setting","content":"\u003cp\u003eWe conducted a mixed-method experimental observational study involving user scenarios in a simulated on-site real-world environment at the respective hospitals of the test participants. This approach allowed the test leader and observer to closely monitor and analyze user interactions and behaviors under controlled, yet realistic conditions (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eC). A pilot session was conducted before the start of the study.\u003c/p\u003e \u003cp\u003eThe test participants received a brief introduction to the test session from the test leader to explain how the test would be conducted. A printed copy of the instructions for use (Supplement 1) was provided and they were encouraged to read them before they started the test. They had access to the instructions for use throughout the test, mimicking real-world use of the device. Additionally, the test participants received the test protocol, which included a demographic questionnaire, five simulated cases (Supplement 1), and a response form. They were required to record the calculated probability, interpret the smoothed histogram, and select the appropriate clinical pathway based on the NILS model\u0026rsquo;s results and other available information. The options included: \u0026ldquo;Consider omitting SLNB,\u0026rdquo; \u0026ldquo;Consider performing SLNB,\u0026rdquo; \u0026ldquo;Definitely perform SLNB,\u0026rdquo; and \u0026ldquo;NILS cannot assist in making the clinical decision.\u0026rdquo; Suggested treatment decisions were not evaluated in this study. Lastly, in each session, the overall usability of the device was evaluated through the validated System Usability Scale (SUS), which is one of the most commonly used usability assessment questionnaires,[\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] and satisfaction was assessed using the After-Scenario Questionnaire (ASQ), in which test participants responded using Likert Scales from 1\u0026ndash;5 and 1\u0026ndash;7, respectively.[\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] Written feedback was obtained. Oral interview questions were administered after the interface use if the test leader or observer noticed that the participant had completed a task incorrectly, nearly made an error, or showed use difficulties. Each test session was scheduled for one hour, including the validated questionnaires and oral interview. While the test participants performed each scenario, both the test leader and observer recorded performance on each task in two separate predefined protocols, as: \u0026ldquo;correct use,\u0026rdquo; \u0026rdquo;use error,\u0026rdquo; \u0026ldquo;close call,\u0026rdquo; or \u0026ldquo;use difficulty\u0026rdquo; (Supplement 2). Subsequently, the tasks were evaluated as either a \u0026ldquo;pass\u0026ldquo; or \u0026ldquo;fail.\u0026rdquo; Discrepancies between the test leader\u0026rsquo;s and observers\u0026rsquo; recordings were resolved through discussion and documented as uncertainty in the report.\u003c/p\u003e"},{"header":"Summary of previous usability evaluations","content":"\u003cp\u003eA formative evaluation was conducted with six participants who responded to questions after using the calculator, none of whom reported any serious usability issues. However, during testing, concerns were raised regarding the potential for inaccurate data entry or misinterpretation of the results. These concerns prompted enhancements to the user interface, such as clearly marking the position of the tumor in the breast and using a different font color to highlight missing data. Briefly, the most hazardous potential harm to be avoided by a safe device design is a false-estimated indication of benign sentinel lymph nodes. A list of identified potential use errors is provided in Supplement 3.\u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eIdentification and description of critical tasks\u003c/h2\u003e \u003cp\u003eCritical tasks were identified after considering previously identified risks in the risk analysis: reset the calculator, enter clinical data, enter mammography data, enter core biopsy data, and identify the appropriate clinical pathway (considering the results of the NILS calculator in combination with all other available information of the case).[\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] The simulated cases were created to ensure that all critical tasks were performed during the test and that the test conditions were sufficiently realistic to represent actual conditions of use.[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]\u003c/p\u003e \u003c/div\u003e"},{"header":"Predefined acceptance criteria for evaluation","content":"\u003cp\u003eSince the NILS model has not yet been released into the market and has limited exposure to intended users, the team anticipated potential usability issues. To assess performance, an acceptance criterion was set at \u0026ge;\u0026thinsp;90% successful task completion, evaluated on a pass/fail basis for each task.\u003c/p\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eData analysis\u003c/h2\u003e \u003cp\u003eDescriptive statistics were calculated for test participants\u0026rsquo; characteristics. SPSS Statistics version 28 (IBM Corp., Armonk, NY, USA) was used for all statistical analyses. The SUS score was calculated[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] (mean and median) and interpreted according to the Adjective Rating Scale.[\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] In terms of qualitative data, notes taken during test sessions by the test leader and observer were rigorously reviewed during data analysis and systematically categorized to gather comments and suggestions of redesign of the interface.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003e In this multisite simulated test session-based usability study with intended users, 20 physicians from four hospitals participated, including 9 (45%) within University Hospitals and 11 (55%) within Regional Hospitals covering three health care regions. Of the twenty participants, 13 (65%) were women and 11 (55%) were \u0026lt;\u0026thinsp;50 years. The majority were surgeons, comprising 75% (N\u0026thinsp;=\u0026thinsp;15) of the participants, while the remaining 25% (N\u0026thinsp;=\u0026thinsp;5) were oncologists (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSelf-reported characteristics of usability testing participants\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eN (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003ePhysician characteristics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUsability testing participants (N\u0026thinsp;=\u0026thinsp;20)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpecialty\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSurgeon\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e15 (75%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOncologist\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5 (25%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCompleted years at specialist (years), median (IQR)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9.5 (7.0-\u0026ndash;17.3)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUse of any tool/system for prediction in clinical work\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eYes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e14 (70%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6 (30%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e- If, yes \u0026ndash; which?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eN\u0026thinsp;=\u0026thinsp;14, Predict (100%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWorkplace (Hospital)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eUniversity Hospital\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9 (45%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRegional Hospital 1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2 (10%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRegional Hospital 2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5 (25%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRegional Hospital 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e4 (20%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHighest academic degree\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMD\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e12 (60%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMD\u0026thinsp;+\u0026thinsp;PhD\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6 (30%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAssociate Professor\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1 (5%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eProfessor\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1 (5%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGender\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7 (35%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFemale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e13 (65%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOther\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDo not want to say\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAge (years), median (IQR)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e48.5 (43.3\u0026ndash;52.0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"3\"\u003eAbbreviations: IQR, interquartile range; MD, Doctor of Medicine; PhD, Doctor of Philosophy\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eTask completion\u003c/h2\u003e \u003cp\u003eThe three critical tasks (1\u0026ndash;6 data entering points/task) of entering clinical, mammography, and core needle biopsy data, respectively, were all above the predefined acceptance criteria of \u0026ge;\u0026thinsp;90% and thus defined as passed (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e and Supplement 4). The critical task of finding the reset button was just below the predefined acceptance criteria and was defined as a failure. The task of performing the calculation (i.e., conditional on the accuracy of entered data) was just below the predefined acceptance criteria for the first scenario only.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eFrequency of observed outcome and result, analysis per task\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"8\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDescription\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"4\" nameend=\"c6\" namest=\"c3\"\u003e \u003cp\u003eObserved outcome, N (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c8\" namest=\"c7\"\u003e \u003cp\u003eResult, N (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eScenario\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCorrect use\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eUse error\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eUse difficulty\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eClose call\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ePass\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eFail\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"5\" rowspan=\"6\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eReset the calculator*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eN/A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eN/A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eN/A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eN/A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eN/A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eN/A\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter clinical data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter mammography data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter core biopsy data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePerform calculation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e17 (85)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e3 (15)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSelect the appropriate clinical pathway\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e16 (80)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"5\" rowspan=\"6\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eReset the calculator\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e17 (85)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e3 (15)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter clinical data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter mammography data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter core biopsy data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePerform calculation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSelect the appropriate clinical pathway\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eReset the calculator\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e17 (85)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e3 (15)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter clinical data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e4 (20)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e16 (80)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSelect the appropriate clinical pathway\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"5\" rowspan=\"6\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eReset the calculator\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e17 (85)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e3 (15)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter clinical data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter mammography data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter core biopsy data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePerform calculation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSelect the appropriate clinical pathway\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"5\" rowspan=\"6\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eReset the calculator\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e17 (85)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e3 (15)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter clinical data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter mammography data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEnter core biopsy data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePerform calculation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSelect the appropriate clinical pathway\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e20 (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0 (0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"8\"\u003e*N/A: Resetting the calculator was not applicable in scenario 1.\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"8\"\u003eResults presented in plain font indicates a task result categorized as \"pass\" (meeting or exceeding the predefined acceptance criteria of \u0026ge;\u0026thinsp;90%), whereas result presented in bold font indicates a task result categorized as \"fail\" (falling below the predefined acceptance criteria of \u0026ge;\u0026thinsp;90%).\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe critical task of selecting the appropriate clinical pathway (based on the calculation results and case information) met or exceeded the predefined acceptance criteria, and was successfully completed for all five scenarios.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eUsability and satisfaction\u003c/h2\u003e \u003cp\u003eTest participants found the NILS tool to be highly user-friendly; the SUS score averaged 89.5 and 89.4 points (mean) on a question- and test participant-based analysis, respectively (Supplements 5 and 6). Moreover, the NILS model yielded high overall satisfaction with an averaged mean ASQ score of 6.3 (maximum 7) (Supplement 7).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eUsability domains\u003c/h2\u003e \u003cp\u003eThe test participants explained their interpretation of the NILS histogram following each case and declared a good understanding of the presented results (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). However, they requested more detailed information regarding what the histogram represented, and how to interpret it in a clinical setting.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eIllustrative comments about the NILS model\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTheme\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eComment\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGeneral impression\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;Easy to use with an appealing layout.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;Pleasant number of variables.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;The results are aligned with my clinical intuition.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBarriers to the NILS model use\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;I am not completely comfortable with the presented histogram and the suggested threshold. I would prefer a recommendation from the calculator regarding omission or not of SLNB.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;I'd like a simple explanation of what the imputed values represent and how they will impact the risk estimation.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eClinical content\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;Curious to why tumor grade and HER2-status are not part of the calculator. The inclusion of these parameters would, in my opinion, increase the credibility of the model.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;For multifocal tumors, what if the smaller ones have a more worrisome profile? Then I would decide on SLNB depending on these characteristics.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;For grade 3 tumors I would always opt for SLNB. That is what my clinical experience tells me.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;It seems more intuitive to be presented the risk of metastatic lymph nodes.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;I'd prefer an overview of contraindications for the NILS calculator, such as neoadjuvant treatment.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInterpretation of the histogram\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;The risk is too high for metastatic lymph node to safely omit SNLB, the probability for benign lymph nodes is below the threshold value, can\u0026acute;t omit SLNB.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;The calculated value is above the threshold value \u0026ndash; probably benign lymph nodes. There is a low risk for metastatic lymph nodes, the consideration to omit SNLB can be discussed with the patient.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;The probability of benign lymph nodes is just above the threshold to consider omitting SLNB. Since the calculated value is very close to the threshold, I would have done SLNB.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ldquo;The calculated probability is clearly below threshold value; it is not appropriate to omit SLNB.\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"2\"\u003eComments are paraphrased to preserve the original posters' anonymity. Translation from the original language spoken at the sessions (Swedish) to English by the researchers.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eSeveral redesign elements were identified, addressing both technical components and educational/informational challenges (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). A recurring concern was the exclusion of the histological tumor grade and human epidermal growth factor 2 (HER2) status, which are clinically important for recommending neoadjuvant or adjuvant treatments. The term \u0026ldquo;multifocality\u0026rdquo; caused some hesitation, as participants were concerned that smaller tumors, excluded from calculations, might have less favorable characteristics than the largest tumor, requiring additional consideration. The reset button was difficult to locate because of its inconspicuous black color and placement at the bottom of the calculator, which became hidden when users viewed the resulting histogram at the top. In addition, users can accidentally scroll within the last cell of an entered value, often altering the Ki67 value. This led one participant to enter an incorrect value, resulting in a miscalculated probability of benign lymph nodes. Furthermore, the information buttons were often missed by the participants and hence were rarely used.\u003c/p\u003e \n\u003cp\u003e\u003cstrong\u003eTable 4. Major design requirements derived from the usability study with intended users\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 25.5776%;\"\u003e\n \u003cp\u003eSource\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 41.0891%;\"\u003e\n \u003cp\u003eIssue\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 33.3333%;\"\u003e\n \u003cp\u003eRequirements\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 25.5776%;\"\u003e\n \u003cp\u003eIt is possible to enter patients\u0026rsquo; age outside of the accepted range.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 41.0891%;\"\u003e\n \u003cp\u003eWhen the age is outside the accepted range the cell becomes yellow, however, still it is possible to enter a value. If notice is not taken, unnecessary work is done inserting all other variables. It will, however, not be possible to make a NILS calculation.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 33.3333%;\"\u003e\n \u003cp\u003eA clear notice that inserted value is out of range/not possible to insert value out of range.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 25.5776%;\"\u003e\n \u003cp\u003eThe NILS calculator is applicable to T1-2 breast cancer. However, in the calculator it is possible to enter a tumor size of up to 90 mm.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 41.0891%;\"\u003e\n \u003cp\u003eIf notice is not taken that tumor size is out of range, unnecessary work is done inserting all other variables. It will, however, not be possible to make a NILS calculation.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 33.3333%;\"\u003e\n \u003cp\u003eA clear notice that inserted value is out of range/not possible to insert value out of range.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 25.5776%;\"\u003e\n \u003cp\u003eThe placement of the reset button is counterintuitive in the web interface, making it difficult to use correctly.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 41.0891%;\"\u003e\n \u003cp\u003eThe reset button is placed at the bottom of the calculator. When making the calculation the user is directed automatically to the histogram, which is displayed in the upper part of the calculator and the reset button then falls out of sight.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 33.3333%;\"\u003e\n \u003cp\u003eIt would enable correct use to place the reset button at the top of the calculator, in the eyeline of the user when entering data for a subsequent case. Further, the black colored reset button does not draw attention, so another color is preferable.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 25.5776%;\"\u003e\n \u003cp\u003eWhen scrolling down the webpage to reach the calculate button, it is possible to accidentally scroll within the last cell, most commonly the Ki67 value for the largest invasive tumor.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 41.0891%;\"\u003e\n \u003cp\u003eThe scrolling can alter the inserted value and if not identified and corrected cause the calculated value to be wrong.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 33.3333%;\"\u003e\n \u003cp\u003eEnsure that scrolling cannot alter an entered value.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 25.5776%;\"\u003e\n \u003cp\u003eUncertainty about how to handle multifocality and central tumor, the information provided by the calculator was often overlooked.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 41.0891%;\"\u003e\n \u003cp\u003eWhen struggling with the decision on which tumor variables to enter in case of multifocality and how a central tumor is defined, the information button was seldom identified.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 33.3333%;\"\u003e\n \u003cp\u003eTo enhance the ability to find the information button, it needs to be more clearly marked.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 25.5776%;\"\u003e\n \u003cp\u003eIn the histogram, the cut-off is marked with a stretched line and the cut-off value is presented in text below. The calculated percentages are marked as a filled line with calculated value presented at the top within the histogram figure.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 41.0891%;\"\u003e\n \u003cp\u003eThe cut-off value can be misinterpreted as the calculated value if the y-axis and the text below only is viewed.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 33.3333%;\"\u003e\n \u003cp\u003eThe cut-off value should also be visualized with a number in addition to the line in the histogram. Moreover, in the text below the histogram, the resulting percentage and not the cut-off value should be emphasized.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 25.5776%;\"\u003e\n \u003cp\u003eThe histogram is presented without declaring what the x-axis and y-axis represent.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 41.0891%;\"\u003e\n \u003cp\u003eFor correct interpretation of the presented histogram/result, proper labelling is warranted.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 33.3333%;\"\u003e\n \u003cp\u003eAdd the following descriptions for the x- and y-axes, respectively: \u0026ldquo;Estimated probability of healthy lymph nodes in the axilla\u0026rdquo; and \u0026ldquo;Fraction of patients in the cohort used for model development\u0026rdquo;.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 25.5776%;\"\u003e\n \u003cp\u003eThe information on applicable use population, e.g. exclusion of neoadjuvant-treated patients, is not easily found when working with the NILS calculator.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 41.0891%;\"\u003e\n \u003cp\u003eIf disclaimers are not noted, the NILS model can be used by mistake on the wrong patient population.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 33.3333%;\"\u003e\n \u003cp\u003eIn the current version, \u0026ldquo;Disclaimers\u0026rdquo; can be found in the ribbon at the top of the calculator interface. It is accessible only by navigating to an adjacent webpage. To improve usability, consider including a heading within the calculator page stating \u0026ldquo;The NILS calculator is not intended for use in every scenario.\u0026rdquo;, followed by a link to \u0026ldquo;Instructions for Use.\u0026rdquo;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eUsing an experimental mixed-methods observational study design with user scenarios in a simulated on-site environment, we found broad support among licensed medical physicians for the NILS model. This tool provides risk estimates for healthy sentinel lymph node status, aiding attending physicians in deciding whether or not SLNB can be abstained. The test participants identified several usability concerns, as well as barriers and facilitators to the implementation and widespread use of the NILS model. These insights have prompted future iterative design modifications.\u003c/p\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003ePrincipal findings\u003c/h2\u003e \u003cp\u003eMany new insights were gained during this usability study. Several aspects of the existing interface posed challenges for the test participants. These included difficulties in locating the reset and information buttons, as well as technical issues, such as the potential to enter values outside the applicable ranges (e.g., for age and tumor size). In addition, the risk of unintentionally altering the entered value during scrolling was detected. Furthermore, a learning curve was evident. Data entry and calculations were the most challenging in scenario 1 and became progressively easier in subsequent scenarios. This is reflected in the only failure of the \u0026ldquo;perform calculation\u0026rdquo; task, which occurred during the first scenario.\u003c/p\u003e \u003cp\u003eThe accomplished SUS score, which is recommended to be regarded as a percentile and not as a percentage, was categorized as \u0026ldquo;excellent\u0026rdquo; (score\u0026thinsp;\u0026gt;\u0026thinsp;85). The slight discrepancy between the per-question and per-participant analyses was due to one slightly skeptical participant who was dissatisfied with the model's role as a clinical support tool. This participant specifically sought a definitive clinical decision regarding whether to perform SLNB. A high ASQ score indicated that the test participants found it easy to complete the tasks in the NILS calculator, were satisfied with the time taken to complete the tasks and found the support information adequate.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003eComparison to similar devices\u003c/h2\u003e \u003cp\u003eBenchmark devices for the NILS model have not been identified. A tool known as Predict, which estimates the survival effect of selected adjuvant therapies for breast cancer, has been widely utilized by both patients and healthcare professionals.[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] A completed usability study on Predict has provided six key recommendations that have been considered in the planning of this NILS model usability study.\u003c/p\u003e \u003cp\u003e As for the NILS model, which is designed to facilitate decision-making, Predict serves as an adjunct to recommended adjuvant therapies according to established guidelines, rather than as a stand-alone decision tool. However, a significant distinction between the NILS model and Predict lies in their functionalities. Unlike Predict, which compares oncological outcomes and the advantages and disadvantages of different treatment options for patients with specific characteristics, the NILS model aids in the decision to abstain from or perform SLNB.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003eClinical contextualization\u003c/h2\u003e \u003cp\u003eComprehending the underlying rationale of the model predictions is essential for building trust in the results calculated using the NILS model. Therefore, we suggest, in accordance with Farmer et al.,[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] that the provision of contextual information making the calculator\u0026acute;s results useful in their clinical setting is important for a positive attitude of the intended users, that is, health care professionals. In this study, we identified the need for enhanced assistance in interpreting histograms, including explaining the rationale behind the cutoff value. There was also a request for a comprehensive explanation of the inclusion and exclusion criteria for the relevant patient population to deepen knowledge of why certain variables, such as HER2 status and histologic tumor grade, were not included in the NILS model.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003eTechnological barriers\u003c/h2\u003e \u003cp\u003eMistrust and unfamiliarity with machine learning models are frequently identified as significant barriers to their integration into clinical practice.[\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e] Although interpreting the resulting histogram was challenging, none of the test participants showed technological mistrust towards the machine learning model behind the NILS calculator. This trust in the model is anticipated to be generalizable to intended users, given the careful consideration of the test participants' representativeness of future intended users.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec20\" class=\"Section2\"\u003e \u003ch2\u003eStrengths and limitations\u003c/h2\u003e \u003cp\u003eThe study was rigorously planned and executed through thorough data control. All endpoints and acceptance criteria were registered before study initiation. Conducted with the intended users, including clinical surgeons and oncologists with representative post-specialist clinical experience, the study was conducted in a simulated on-site real-world physical and digital environment within three healthcare regions. This approach enhanced the generalizability of the results to actual clinical practice. Having both a test leader and observer ensured strict adherence to the study protocol, guaranteeing objectivity throughout the process.\u003c/p\u003e \u003cp\u003eThere were no requirements for user training before the device was used. However, in a real-world scenario, users might take varying amounts of time to read the \u0026ldquo;Instruction for Users\u0026rdquo;; some might read it thoroughly from front to back, while others might only glance at it before starting to use the device, depending on their personality type and the time available. Moreover, we hypothesized that the NILS model could be effectively utilized in a multidisciplinary conference setting, where physicians from various specialties discuss and formulate treatment plans for specific patients, considering the available data not included in the NILS model. Although this particular clinical scenario was not replicated in this usability study, we anticipate that similar criticisms and evaluations will emerge.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003eFuture perspective\u003c/h2\u003e \u003cp\u003eSeveral key redesign elements were identified, encompassing technical aspects and educational and informational issues. Collectively, these changes will enhance the usability and workflow, thereby preparing the NILS model for clinical use as a decision-support tool. The proposed improvements will be implemented and evaluated with a selected group of test participants, with a specific focus on the identified and redesigned issues. In a post-market setting, the user interface will be iteratively updated and evaluated as needed using information gained from the post-market process.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThis qualitative premarket usability study of the NILS model, conducted through test sessions with simulated clinical cases, involved intended users, that is, surgeons and oncologists, in a real-world clinical setting. This study identified the key usability factors, barriers, and facilitators influencing implementation. Physicians found the NILS model intuitive and valued its presented risk estimates for healthy axillary lymph node status in their decision to perform or abstain from SLNB. These findings will directly inform the refinement of the NILS model, ensuring improved usability and clinical integration.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eASQ\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;After-Scenario Questionnaire\u003c/p\u003e\n\u003cp\u003eBCS\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;breast-conserving surgery\u003c/p\u003e\n\u003cp\u003eHER2\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;human epidermal growth factor receptor 2\u003c/p\u003e\n\u003cp\u003eNILS\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;noninvasive lymph node staging\u003c/p\u003e\n\u003cp\u003eSLNB\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;sentinel lymph node biopsy\u003c/p\u003e\n\u003cp\u003eSUS\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;System Usability Scale \u003c/p\u003e"},{"header":"Declarations","content":"\u003ch3\u003eEthics approval, consent to participate and consent of publication\u003c/h3\u003e\n\u003cp\u003eThis usability study was not a clinical study, and did not require ethical approval from the Swedish Ethical Review Authority. The usability study was approved by the Head of Research in Region Sk\u0026aring;ne (2\u003csup\u003end\u003c/sup\u003e Apr, 2024) regarding the intended conduct of the study, and Region Sk\u0026aring;ne is thus the legal authority for the study. All participants in the study provided written informed consent.\u003c/p\u003e\n\u003ch3\u003eCompeting interests\u003c/h3\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003ch3\u003eAvailability of data and materials\u003c/h3\u003e\n\u003cp\u003eThe raw datasets are available from the corresponding author on reasonable request.\u0026nbsp;\u003c/p\u003e\n\u003ch3\u003eDeclaration of generative AI and AI-assisted technologies in the writing process\u003c/h3\u003e\n\u003cp\u003eDuring the preparation of this manuscript, the authors used Copilot for language editing and generated an initial draft of Figure 1C. Following the use of this tool, the authors thoroughly reviewed and revised the content as necessary and take full responsibility for the final content of this publication.\u003c/p\u003e\n\u003ch3\u003eAcknowledgments\u0026nbsp;\u003c/h3\u003e\n\u003cp\u003eThe authors express their gratitude to the test participants at Sk\u0026aring;ne University Hospital, Kristianstad Regional Hospital, Karlskrona Regional Hospital, and V\u0026auml;xj\u0026ouml; Regional Hospital. Special thanks go to research nurse Helena Erixon for her exceptional contributions. Furthermore, we thank MEDOS for their valuable cooperation in the planning of this study and development of the test protocols.\u003c/p\u003e\n\u003ch3\u003eFunding\u003c/h3\u003e\n\u003cp\u003eThis work was supported by grants from the Lund University (Sweden), South Swedish Health Care Region (Sweden), Governmental Funding of Clinical Research within the National Health Service Sweden (ALF young researcher Ida Skarping), Erling Persson Foundation (Sweden), Sk\u0026aring;ne University Hospital Funds, Swedish Breast Cancer Funding, Sj\u0026ouml;berg Foundation, and Vetenskapsr\u0026aring;det [Swedish Research Council] (Sweden) (external review). In this academic study, the funding resources had no role in the study design, data collection, analyses, data interpretation, writing of the manuscript or the decision to submit the manuscript for publication.\u003c/p\u003e\n\u003ch3\u003eAuthors\u0026apos; contributions\u003c/h3\u003e\n\u003cp\u003eMrs. A. Allfelt and Dr. I. Skarping had full access to all data in the study and take responsibility for the integrity of the data and accuracy of the data analysis. Dr. I Skarping is the senior author responsible for study oversight.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConcept and design:\u003c/strong\u003e Ryd\u0026eacute;n, Skarping\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcquisition, analysis, or interpretation of data:\u003c/strong\u003e Allfelt, Bendahl, Dihge, Ohlsson, Ryd\u0026eacute;n, and Skarping\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDrafting of the manuscript\u003c/strong\u003e: Allfelt, Skarping\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCritical review of the manuscript for important intellectual content:\u0026nbsp;\u003c/strong\u003eAllfelt, Bendahl, Dihge, Ohlsson, Ryd\u0026eacute;n, and Skarping\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStatistical analysis:\u003c/strong\u003e Allfelt, Skarping\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eObtained funding:\u0026nbsp;\u003c/strong\u003eRyd\u0026eacute;n, Skarping\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAdministrative, technical, or material support:\u003c/strong\u003e Allfelt, Ryd\u0026eacute;n, Skarping\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSupervision\u003c/strong\u003e: Ryd\u0026eacute;n, Skarping\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eSung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F (2021) Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin 71(3):209\u0026ndash;249\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCady B, Stone MD, Schuler JG, Thakur R, Wanner MA, Lavin PT (1996) The new era in breast cancer. Invasion, size, and nodal involvement dramatically decreasing as a result of mammographic screening. Arch Surg 131(3):301\u0026ndash;308\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMansel RE, Fallowfield L, Kissin M, Goyal A, Newcombe RG, Dixon JM, Yiangou C, Horgan K, Bundred N, Monypenny I et al (2006) Randomized multicenter trial of sentinel node biopsy versus standard axillary treatment in operable breast cancer: the ALMANAC Trial. J Natl Cancer Inst 98(9):599\u0026ndash;609\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVeronesi U, Paganelli G, Viale G, Luini A, Zurrida S, Galimberti V, Intra M, Veronesi P, Robertson C, Maisonneuve P et al (2003) A randomized comparison of sentinel-node biopsy with routine axillary dissection in breast cancer. N Engl J Med 349(6):546\u0026ndash;553\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrackstone M, Baldassarre FG, Perera FE, Cil T, Chavez Mac Gregor M, Dayes IS, Engel J, Horton JK, King TA, Kornecki A et al (2021) Management of the Axilla in Early-Stage Breast Cancer: Ontario Health (Cancer Care Ontario) and ASCO Guideline. J Clin Oncol 39(27):3056\u0026ndash;3082\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGentilini OD, Botteri E, Sangalli C, Galimberti V, Porpiglia M, Agresti R, Luini A, Viale G, Cassano E, Peradze N et al (2023) Sentinel Lymph Node Biopsy vs No Axillary Surgery in Patients With Small Breast Cancer and Negative Results on Ultrasonography of Axillary Lymph Nodes: The SOUND Randomized Clinical Trial. JAMA Oncol 9(11):1557\u0026ndash;1564\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eReimer T, Stachs A, Veselinovic K, K\u0026uuml;hn T, Heil J, Polata S, Marm\u0026eacute; F, M\u0026uuml;ller T, Hildebrandt G, Krug D et al (2024) Axillary Surgery in Breast Cancer - Primary Results of the INSEMA Trial. N Engl J Med\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDihge L, Bendahl PO, Skarping I, Hj\u0026auml;rtstr\u0026ouml;m M, Ohlsson M, Ryd\u0026eacute;n L (2023) The implementation of NILS: A web-based artificial neural network decision support tool for noninvasive lymph node staging in breast cancer. Front Oncol 13:1102254\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKrag DN, Anderson SJ, Julian TB, Brown AM, Harlow SP, Ashikaga T, Weaver DL, Miller BJ, Jalovec LM, Frazier TG et al (2007) Technical outcomes of sentinel-lymph-node resection and conventional axillary-lymph-node dissection in patients with clinically node-negative breast cancer: results from the NSABP B-32 randomised phase III trial. Lancet Oncol 8(10):881\u0026ndash;888\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePesek S, Ashikaga T, Krag LE, Krag D (2012) The false-negative rate of sentinel node biopsy in patients with breast cancer: a meta-analysis. World J Surg 36(9):2239\u0026ndash;2251\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHj\u0026auml;rtstr\u0026ouml;m M, Dihge L, Bendahl PO, Skarping I, Ellbrant J, Ohlsson M, Ryd\u0026eacute;n L (2023) Noninvasive Staging of Lymph Node Status in Breast Cancer Using Machine Learning: External Validation and Further Model Development. JMIR Cancer 9:e46474\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSkarping I, Ellbrant J, Dihge L, Ohlsson M, Huss L, Bendahl PO, Ryd\u0026eacute;n L (2024) Retrospective validation study of an artificial neural network-based preoperative decision-support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer (ISRCTN14341750). BMC Cancer 24(1):86\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSkarping I, Nilsson K, Dihge L, Fridhammar A, Ohlsson M, Huss L, Bendahl PO, Steen Carlsson K, Ryd\u0026eacute;n L (2022) The implementation of a noninvasive lymph node staging (NILS) preoperative prediction model is cost effective in primary breast cancer. Breast Cancer Res Treat 194(3):577\u0026ndash;586\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 19(6):349\u0026ndash;357\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eU.S. Food and Drug Administration: Applying Human Factors and Usability Engineering to Medical Devices. Guidance for Industry and Food and Drug Administration Staff; (2016) \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.fda.gov/media/80481/download\u003c/span\u003e\u003cspan address=\"https://www.fda.gov/media/80481/download\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFaulkner L (2003) Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 35(3):379\u0026ndash;383\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLewis JR (2018) The System Usability Scale: Past, Present, and Future. Int J Human\u0026ndash;Computer Interact 34(7):577\u0026ndash;590\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHyzy M, Bond R, Mulvenna M, Bai L, Dix A, Leigh S, Hunt S (2022) System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis. JMIR Mhealth Uhealth 10(8):e37290\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLewis JR (1995) IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. Int J Human\u0026ndash;Computer Interact 7(1):57\u0026ndash;78\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eInternational Electrotechnical Commission: IEC TR 62366-2:2016 Medical devices - Part 2: Guidance on the application of usability engeneering to medical devices (Edition 1.0) (2016) ; \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.iso.org/standard/69126.html\u003c/span\u003e\u003cspan address=\"https://www.iso.org/standard/69126.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRuss AL, Saleem JJ (2018) Ten factors to consider when developing usability scenarios and tasks for health information technology. J Biomed Inf 78:123\u0026ndash;133\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrooke J (1996) SUS-A quick and dirty usability scale. Usability evaluation Ind 189(194):4\u0026ndash;7\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBangor A, Kortum P, Miller J (2009) Determining what individual SUS scores mean: Adding an adjective rating scale. J usability Stud 4(3):114\u0026ndash;123\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFarmer GD, Pearson M, Skylark WJ, Freeman ALJ, Spiegelhalter DJ (2021) Redevelopment of the Predict: Breast Cancer website and recommendations for developing interfaces to support decision-making. Cancer Med 10(15):5141\u0026ndash;5153\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAsan O, Bayrak AE, Choudhury A (2020) Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians. J Med Internet Res 22(6):e15154\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Lund University","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Breast neoplasm, Staging, Axillary lymph nodes, Decision aid, Sentinel lymph node biopsy, Human factors validation study","lastPublishedDoi":"10.21203/rs.3.rs-6335418/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6335418/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eThe integration of clinical decision support tools in medical practice is challenging and must be carefully undertaken, especially in cancer management. Noninvasive Lymph Node Status (NILS) is a web-based tool designed to estimate the probability of healthy axillary lymph nodes in female patients with breast cancer scheduled for primary surgery. The aim was to identify barriers to NILS adoption in a clinical setting and assess whether intended users can operate the tool without significant errors or difficulties. Additionally, the study aimed to evaluate the appropriateness of result interpretation for decisions to abstain from sentinel lymph node biopsy (SNLB) and measure overall user satisfaction with the tool.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eThis mixed-methods multicenter on-site qualitative human factor validation study was conducted in a simulated clinical environment, replicating both real-world physical and digital conditions. Based on the identified target user population for the NILS model, twenty physicians comprised the cohort. The study used simulated clinical cases, the System Usability Scale (SUS), and the After-Scenario Questionnaire (ASQ) to evaluate usability and satisfaction. An oral interview was conducted after the evaluations. The study followed a structured protocol, with distinct roles assigned to the test participants, leader, and observer.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003e Twenty physicians from four hospitals, with a median of 9.5 years of specialist practice, participated. Most participants were surgeons (75%, N\u0026thinsp;=\u0026thinsp;15), while the remaining 25% (N\u0026thinsp;=\u0026thinsp;5) were oncologists. Usability scores were high, with a mean SUS score of 89.5 (\u0026ldquo;excellent\u0026rdquo;) and a mean ASQ score of 6.3 (Likert scale 1\u0026ndash;7). Several interface challenges were identified, including difficulty in locating the reset and information buttons, the ability to enter values outside valid ranges (age and tumor size), and the risk of unintentionally modifying the entered values while scrolling.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eThis study identified the key usability factors, barriers, and facilitators affecting the NILS model implementation. Physicians found the NILS interface easy to use and valued the presented risk estimates of healthy axillary lymph node status in their decisions to perform or abstain from SLNB. Redesign initiatives are ongoing. An iterative process of addressing usability in a clinical setting is crucial for successful implementation.\u003c/p\u003e\u003ch2\u003eTrial Registration\u003c/h2\u003e \u003cp\u003eISRCTN99301435\u003c/p\u003e","manuscriptTitle":"Human Factors Validation Study of an Artificial Neural Network‑based Preoperative Decision‑support Tool for Noninvasive Lymph Node Staging (NILS) in Women with Primary Breast Cancer (ISRCTN99301435)","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2025-06-03 15:49:19","doi":"10.21203/rs.3.rs-6335418/v2","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2025-04-03 10:29:53","doi":"10.21203/rs.3.rs-6335418/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"38bb8590-35c4-46d6-8002-630e01dddf9f","owner":[],"postedDate":"June 3rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":46629924,"name":"General Surgery"}],"tags":[],"updatedAt":"2025-04-03T10:29:53+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-03 15:49:19","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v2","identity":"rs-6335418","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6335418","identity":"rs-6335418","version":["v2"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}