I-Viewer: An Online Digital Pathology Analysis Platform with Agentic-RAG AI Copilot

I-Viewer: An Online Digital Pathology Analysis Platform with Agentic-RAG AI Copilot | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article I-Viewer: An Online Digital Pathology Analysis Platform with Agentic-RAG AI Copilot Ruichen Rong, Danni Luo, Zifan Gu, Peiran Quan, Ismael Villanueva-Miranda, and 8 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5404747/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Digital pathology has seen significant advancements in artificial intelligence (AI) applications. However, challenges persist in integrating these solutions into digital pathology platforms for human and AI collaborations. We introduce I-Viewer, an online AI Copilot framework designed to facilitate real-time human-AI and human-human collaboration for digital pathology analysis. The I-Viewer platform enables precise annotations and descriptions from tissue to the nuclei level through an Agentic-Retrieval Augmented Generation (RAG) system. By leveraging agents' outputs as reference points, aggregating information through the RAG system, and incorporating Large Language Models (LLM) for human feedback and refinement, I-Viewer sets a new standard for collaborative and accurate digital pathology analysis. We demonstrate I-Viewer's effectiveness on different pathology tasks using three datasets across different types of cancers, including non-small cell lung cancer, breast cancer, and colorectal cancer. The results show that I-Viewer achieves significant improvements in annotation speed and accuracy for pathology tasks, such as detecting cell morphology, cellular structures, and tumor growth patterns, outperforming current individual foundation models. Through its advanced AI agents, collaborative features, and LLM integrations, I-Viewer optimizes diagnostic workflows in clinical care and biomedical research. Biological sciences/Computational biology and bioinformatics/Machine learning Biological sciences/Computational biology and bioinformatics/Software Biological sciences/Computational biology and bioinformatics/Image processing Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. Introduction Recent advancements in medical imaging technology have transformed the field of digital pathology, enabling the production of high-resolution whole-slide images (WSI) that capture intricate tissue structures and morphological patterns at a large scale 1 , 2 . The conventional manual assessment of WSI is subjective, time-consuming and prone to errors 3 , 4 . In response, various artificial intelligence (AI) algorithms have been developed to improve accuracy, reproducibility, and efficiency in digital pathology analysis 5 – 7 . These AI algorithms have demonstrated success across a spectrum of tasks, including nuclei and tissues segmentation 8 – 11 , subtype classification 12 , 13 , diagnosis 5 , 14 , 15 , and patient survival prognosis 6 , 16 , 17 . The recent emergence of foundation models and Large Language Models (LLMs) has further expanded the potential for integrating knowledge bases and extracting information in pathology 18 , 19 . These advanced AI algorithms have the potentials to streamline the human-AI collaboration analysis lifecycle of digital pathology images. Studies demonstrate that AI assistance can significantly reduce the time required for report writing and decision-making 20 . Additionally, AI can enhance diagnostic accuracy, particularly for pathologists with varying levels of expertise 21 – 23 . However, significant challenges remain in developing AI tools that match human-expert-level visual assessment and histological analyses 24 . One major challenge is mimicking the pathologist's routine of examining tissues under various magnifications, as current AI algorithms struggle to provide real-time translation of histological findings from individual nuclei to the broader tissue level 25 . This limitation is particularly problematic under high magnification, where regions of interest (ROI) may encompass thousands of nuclei and key tissue structure components, requiring highly efficient algorithms for detection and extraction of morphological features. Another challenging is to develop efficient pipelines that can understand positional relationships and features within the tumor microenvironment (TME) across various AI tools and different magnification levels. For instance, identifying tumor nuclei within lymph nodes requires the algorithms to “understand” sophisticated spatial context. Such capabilities are crucial for accurately interpreting the intricate relationships between different tissue components and their surrounding environment. Given these challenges, the development of comprehensive AI solutions for digital pathology remains a complex task. Existing platforms lack the infrastructure to integrate diverse AI functionalities, such as object identification, TME feature extraction, and medical question answering, into a unified system that enables collaborative user-AI interactions and result refinement. This again deviates from the pathologists’ routine of performing and sharing most analysis work in a centralized software environment. While some initiatives have explored integrated solutions, they have notable limitations. For example, Digital Slide Archive (HistomicsUI) enables multiple users to access and annotate the same slides but lacks efficient integration with its AI tools (HistomicsTK), hindering real-time annotation capabilities 26 . On the other hand, standalone software like QuPath offers flexible analysis and visualization tools for digital pathology but falls short in providing robust collaboration features essential for pathologists’ workflows 27 . Moreover, the recent developments of LLMs and foundation models, which require significant computational resources, further challenges the analysis capability and efficiency of these standalone frameworks 28 . Additionally, the growing number of pre-trained AI assistants makes it crucial to aggregate and infer results from different analysis algorithms and AI tools 5 . Existing platforms lack the capability to trigger appropriate assistants (AI modules or functionalities) for user-specific questions and resolve conflicts among different assistants while accurately summarizing results. This highlights the need for an advanced AI system capable of responding to user inquiries, integrating them into ongoing analysis, and providing iterative refinement based on expert feedback. Such a system is essential for not only enhancing current diagnostic processes but also developing a comprehensive data and annotation collection solution, a key component in advancing future foundation models in digital pathology. In response to these challenges, we introduce I-Viewer: a comprehensive online framework designed to facilitate collaborative pathology analysis empowered by an AI Copilot. I-Viewer deploys advanced real-time AI agents for nuclei classification and segmentation, tissue structure and spatial pattern analysis, and LLM-enabled question answering. At its core, I-Viewer utilizes a Retrieval Augmented Generation (RAG) based copilot system, which combines retrieval of relevant information with generation capabilities to organize these agents, improving the efficiency, steerability, and transparency of complex question answering. The RAG integrates the power of retrieval mechanisms with generative models enhancing the quality and relevance of generated outputs by augmenting LLMs with retrieval-based techniques. This hybrid approach facilitates better contextual understanding and reduces the risk of generating irrelevant or incorrect information. The I-Viewer platform also provides flexible frontend and backend interfaces, allowing the enhancement and customization of AI solutions. This adaptability enables the seamless integration of new algorithms and methodologies as the field evolves. By integrating human experts' collective expertise with AI's analytical capabilities, I-Viewer fosters collaboration among pathologists, trainees, and researchers. It serves as a unified platform that simplifies scientific research and diagnostic processes, ultimately improving patient care and advancing our understanding of complex diseases. Through this integrated approach, I-Viewer has the potential to bridge the gap between advanced AI technologies and practical clinical applications in digital pathology. 2. Results 2.1 I-Viewer Overview I-Viewer provides real-time assistance through deep learning models, LLMs, and specialized analysis pipelines, in order to enrich histological insights for user and optimize pathologists' workflows. This systems address key challenges using advanced techniques: 1) real-time nuclei segmentation and classification with HD-Yolo; 2) identification of architectural pattern, tissue interface, and spatial relationships across various magnification levels via multimodal large language models (MLLMs) and TME feature extraction pipeline 29 ; and 3) aggregation of agent level results with Agentic-RAG system to target user queries, infer potential discoveries, informed by medical domain knowledge. These innovations (summarized in Supplementary Texts S1 ) enable I-Viewer to provide comprehensive, context-aware assistance in digital pathology analysis. Table 1 presents a comprehensive comparison of I-Viewer with current pathological AI platforms, including MONAI 30 , HistomicsUI 26 , QuPath 27 , Nucleo.AI 5 , and PathChat 31 . The comparison spans several different aspects: multi-user collaboration, AI assistant at various levels, AI copilot functionality, online/offline agent capabilities, support for customized pipelines, and Agentic-RAG integration. This analysis highlights I-Viewer's advantages in support for multi-user online collaboration, flexible AI agent capabilities, and an advanced AI copilot system. The results demonstrate I-Viewer as a uniquely comprehensive solution in the current landscape of pathological AI platforms. Table 1 Functionality comparison of I-Viewer with existing software platforms. I-Viewer MONAI Nucleo.AI QuPath HistomicsUI PathChat Multi-user collaboration X X - - X - Nuclei level AI assistant X X X X - - ROI level AI assistant X X - X - X MLLM Chatbot /AI copilot X - - - - X Online + offline agents X - - - - - Customized pipelines X - - X - - Agentic-RAG AI copilot X - - - - - 2.2 Real-time Nuclei Classification and Segmentation I-Viewer incorporates the state-of-the-art HD-Yolo algorithm 29 for real-time nuclei segmentation, classification, and morphological feature extraction. Table 2 presents a comprehensive comparative analysis of various algorithms across two benchmark datasets for nuclei segmentation: the non-small cell lung cancer (NSCLC) HD-Staining dataset 32 and breast cancer NuCLS dataset 10 . In both benchmark datasets, HD-Yolo demonstrates superior performance over existing algorithms (Hover-Net 8 , Yolov8-seg, and Stardist 33 – 35 ) in both segmentation and classification tasks, as evidenced by metrics including precision, recall, F1, mean intersection over union (mIoU), and inference time (summarized in Table 2 ). HD-Yolo's superior performance can be attributed to its unique lightweight mask header and attention mechanism, which significantly enhance segmentation accuracy. Notably, HD-Yolo surpasses Yolov8-seg in accuracy while maintaining comparable inference speeds, achieving a balance between performance and computational efficiency. This combination of high accuracy and speed makes HD-Yolo particularly well-suited for the real-time analysis demands of digital pathology workflows within the I-Viewer framework. Table 2 Nuclei detection, classification, and segmentation results. Dataset Model Precision Recall F1 mIoU Inference time (s) NSCLC (Non-small cell lung cancer) HD-Yolo 0.831 0.674 0.741 0.842 0.004 Yolov8-seg 0.811 0.648 0.720 0.748 0.004 Stardist* 0.732 0.583 0.649 0.795 0.009 Hover-Net 0.660 0.395 0.581 0.772 0.101 NuCLS (Breast Cancer) HD-Yolo 0.924 0.866 0.894 0.819 0.004 Yolov8-seg 0.901 0.802 0.848 0.773 0.004 Stardist* 0.880 0.750 0.810 0.798 0.010 Hover-Net 0.820 0.751 0.784 0.820 0.083 In the benchmarking tasks, we used HD-Yolo, Yolov8-seg, Stardist, and Hover-Net on NSCLC and NuCLS datasets. Inference time is measured in seconds per image. *For Stardist, only detection performance is reported due to label mismatches between model outputs and dataset labels. Boldface indicates best performance. Figure 1 presents one example comparing the segmentation results of various models: Hover-Net and Stardist fail to segment occluded nuclei and often split large nuclei into smaller pieces, while Yolov8-seg separates occluded nuclei with distinct boundaries. In contrast, HD-Yolo generates smoothed masks and accurately segments occluded nuclei, demonstrating its advanced capabilities in handling complex nuclear arrangements typically encountered in histopathological images. I-Viewer leverages the efficiency and accuracy of the HD-Yolo algorithm to provide real-time segmentation results in response to user viewport changes. This approach eliminates the need for offline pre-calculation or the division of WSIs into smaller ROIs, a requirement of most existing tools 5 . Instead, I-Viewer dynamically tracks the user's viewport during panning or zooming operations, starting at 10x magnification. It streams image patches within the ROI to a background job queue and Redis cache at their original resolutions (refer to Section 4.2 , I-Viewer Infrastructure ). HD-Yolo agents analyze these patches concurrently, ensuring scalability and optimizing processing efficiency. This design ensures that the service can automatically adapt to varying workloads, thereby optimizing processing efficiency. The result is a responsive system that delivers high-quality, real-time nuclei segmentation and classification, enhancing both user experience and workflow efficiency in digital pathology analysis. 2.3 Architectural Pattern Analysis and Tissue Interface I-Viewer aggregates task-specific agents to recognize general histopathological patterns, including nuclear morphology, secondary architecture, and growth pattern interpretation. These agents leverage MLLMs, and the TME analysis pipeline to extract relevant information from ROIs (refer to Section 4.2 , I-Viewer Infrastructure ). Table 3 presents the performance of different approaches on the identification of nuclear morphology, secondary structures and architectures, and tumor growth patterns. The evaluation methods differ based on the model type: Contrastive Language-Image Pre-Training (CLIP) 36 -based models PLIP 19 and Conch 28 are evaluated by the cosine similarity between the ROI and predefined keywords lists of potential patterns and features, whereas MLLMs, GPT-4V, HD-LLaVA, and LLaVA-Med 37 , are evaluated by their accuracy in answering multiple-choice questions. We observed that GPT-4V outperformed other open-source MLLMs, likely due to its advanced image encoder, robust language model, and extensive high-quality image captioning dataset 38 , 39 . However, GPT-4V has a limitation: it struggles to generate high-confidence identifications when presented with large ROIs containing numerous details. This is primarily because morphology and structures related to nuclei-tissue interactions are challenging to identify at low magnifications. Table 3 Performance comparison of different foundation models on architecture pattern and tissue interface recognition. Models Nuclear/Cell Morphology Structures and Architectures Tumor Growth Pattern and Interface with Normal Tissue Precision I-Viewer Agents 0.803 0.775 0.667 GPT-4V 0.840 0.750 0.733 HD-LLaVA 0.531 0.704 0.271 LLaVA-Med 0.521 0.435 0.250 Conch 0.183 0.203 0.212 PLIP 0.204 0.188 0.232 Recall I-Viewer Agents 0.885 0.912 0.783 GPT-4V 0.844 0.706 0.342 HD-LLaVA 0.301 0.685 0.224 LLaVA-Med 0.334 0.294 0.231 Conch 0.243 0.311 0.187 PLIP 0.234 0.285 0.130 To overcome the limitations of single-scale analysis, I-Viewer incorporates a comprehensive TME analysis pipeline that assesses histological characteristics across multiple magnifications. This pipeline consolidates data on nuclei density distribution, shape morphology statistics, and intricate spatial relationships between nuclei and secondary structures. By integrating information across different magnifications, I-Viewer agents achieve high confidence in identifying critical features, including nuclear pleomorphism, tumor-infiltrating lymphocytes and tumor metastasis. As demonstrated in Table 3 , the I-Viewer system significantly enhances recall rates across all three aspects of analysis (nuclear morphology, secondary structures and architectures, and tumor growth patterns) by synergistically combining information from GPT-4V, HD-LLaVA, and the TME feature extraction pipeline. This multi-model, multi-scale approach enables I-Viewer to overcome the limitations of individual models and provide more comprehensive and accurate histopathological assessments. 2.4 LLM with Medical Domain Knowledge for AI-Human Collaboration Ensuring the effectiveness and consistency of LLM-based AI assistant is crucial in medical applications. This involves accurately identifying and mitigating hallucinations, addressing specific inquiries, and providing insights through a human-in-the-loop approach. At its core, I-Viewer features a sophisticated chatbot engine based on LLaMA3 and GPT-4. The engine enables the provision of comprehensive summaries informed by medical knowledge, synthesizing results from various AI agents. In addition, I-Viewer employs an Agentic-RAG caching system (refer to Section 4.2 , I-Viewer Infrastructure ) to efficiently retrieve analysis results from multiple specialized agents, ensuring that the LLM's responses are guided by concrete, relevant data. Thereby, this approach significantly reduces inconsistency, potential errors, and hallucinations in the AI's output. By leveraging advancements in both LLMs and RAG technologies, I-Viewer can integrate information from various sources, produce detailed histology descriptions, and engage with user queries responsively. These features enable users to conduct multi-round question-answering sessions efficiently with I-Viewer. Users can perform common tasks such as summarizing histology reports and inferring potential risks, and incorporating their analyses in the I-Viewer annotation tool. This workflow enhances the accuracy and reliability of pathological assessments while maintaining the critical role of human expertise in the decision-making process. Figure 2 presents a comparative analysis of histology insights synthesized by I-Viewer, GPT-4V, and LLaVA-Med (prompts are shown in Supplementary Fig. 1 ). In the lung adenocarcinoma study (Fig. 2A), the I-Viewer copilot demonstrates high accuracy in identifying key features, including papillary structures, nuclei differentiation, and infiltrative growth patterns. Moreover, I-Viewer agents enhance their analysis by providing additional observations on nuclei composition and distribution. In the colon cancer ROI (Fig. 2B), I-Viewer and GPT-4V offer consistent interpretations of growth patterns and architectural characteristics, demonstrating their concordance in analyzing different types of cancer images. To maintain response consistency and caution, I-Viewer uses the Agentic-RAG system and LLM safeguards. When users pose questions that extend beyond basic histological observations, such as requests for diagnosis or prognosis, the system deliberately limits its responses to avoid potential hallucinations or overreach. For instance, in Fig. 2A , I-Viewer refrains from providing definitive diagnostic or prognostic information based solely on the ROI. Instead, it indicates the need for additional user inputs to ensure more informed and responsible insights. This approach maintains the system's reliability while respecting the complexities and nuances of medical interpretation. 2.5 I-Viewer is an Extensible Framework for Researchers I-Viewer is designed to be scalable and adaptable, allowing for the improvement of current models and the easy addition of new AI models. This design ensures that I-Viewer can evolve alongside advancements in digital pathology and AI technologies. The extensible architecture of I-Viewer is built upon several key features. Its loosely coupled system ensures user-friendliness for both biologists and machine learning engineers. The frontend incorporates customized high-performance OpenSeadragon (OSD) 40 plugins, enabling flexible rendering of annotations across various services, users, and projects. The robust backend architecture utilizes message queues, microservices, and model registries, simplifying the deployment of existing pipelines and AI models. Additionally, I-Viewer offers RESTful APIs for adaptable data curation, supporting model development and fine-tuning. This comprehensive infrastructure allows custom analysis pipelines to be integrated into I-Viewer through a straightforward four-step process. (refer to Supplementary Texts S2 and Supplementary Fig. 1 ). To demonstrate I-Viewer's extensibility, we integrate the onion peeling algorithm into I-Viewer for analyzing epithelium layers in pathology images for oral premalignant disorder (OPMD) 41 . Accurate quantification of epithelial layers is essential for evaluating the severity of epithelial dysplasia, a precursor to oral cancer, which aids in early detection, clinical grading, and patient prognosis 42 . The onion peeling algorithm relies on pre-calculated nuclei detection and region segmentation, making real-time counting of epithelial layers challenging for existing frameworks. To address this limitation, I-Viewer leverages its model registries and the Agentic-RAG system to integrate the onion peeling pipeline as an agent to automatically retrieve nuclei and ROI information from interconnected agents, facilitating real-time delivery of algorithm results. As illustrated in Fig. 3 , I-Viewer efficiently annotates epithelium layers produced by the onion peeling algorithm, demonstrating capabilities that surpass those of LLMs and other existing frameworks. This successful integration demonstrates I-Viewer's flexibility, allowing users to deploy new AI algorithms and leverage the copilot system for information retrieval and integration. 2.6 I-Viewer is a Secure and User-friendly Online Platform for Interactive Pathology Image Sharing and Annotation I-Viewer is a cloud-based platform that provides a secure and collaborative environment for digital pathology through robust user authorization and authentication mechanisms. It supports simultaneous user interactions with AI, enabling joint annotation and communication across shared slides and projects. I-Viewer's collaborative capabilities include real-time collaboration where multiple users can work on the same slide simultaneously, enhancing knowledge sharing. The platform offers AI-assisted annotations, allowing users to leverage AI algorithms for automated annotations while retaining the ability to modify and refine results. The access control ensures appropriate levels of access and functionality for different team members, from trainees to senior pathologists. To accommodate common pathologist needs, I-Viewer has an intuitive layer-based visualization system. This frontend framework organizes original WSIs, AI-generated results, and user annotations into a structured hierarchy. Users can selectively display individual layers and manipulate objects and annotations across different layers at various zoom levels. I-Viewer also features dual-view panels, facilitating cross-examination between different layers to compare differences and enhance the interpretation of AI outputs. To demonstrate I-Viewer's functionality, we provide two supplementary videos: Supplementary Video 1 showcases how users can modify AI-predicted segmentation shapes, descriptions, and types. These adjustments are automatically tagged with the user's ID, distinguishing them from AI-generated annotations. This feature ensures clarity between manual edits and AI outputs, enhancing traceability and accountability. Supplementary Video 2 illustrates users interacting with the copilot system to identify secondary structures and growth patterns, generate ROI descriptions, and infer potential discoveries. Notably, I-Viewer's Agentic-RAG system can initiate nuclei counting and TME density analysis pipelines in response to user queries about nuclei summaries and growth patterns. I-Viewer enhances transparency by clearly referencing sources and providing rationale for generated answers, fostering trust and understanding between clinicians and AI. This improved interpretability allows users to grasp the reasoning behind complex solutions, facilitating informed decision-making. The Agentic-RAG system in I-Viewer represents a sophisticated approach to effectively utilize the capabilities of multiple AI agents, offering a powerful tool for comprehensive pathological analysis. 3. Discussions and Conclusion In this study, we present a novel digital pathology analysis platform power by the Agent-RAG AI copilot system. I-Viewer assembles robust model collections and provides a highly efficient infrastructure that promotes collaborations among pathologists, researchers, and AI. This platform represents the first-of-its-kind open-source ecosystem to effectively integrate task-specific deep learning algorithms, LLMs, and user inputs to enhance clinical care and biomedical research. I-Viewer serves as an AI copilot to improve the annotation process. To demonstrate its clinical relevance and broad compatibility, we applied I-Viewer to non-small cell lung cancer and breast cancer tissues. This showcases how pathologists can seamlessly interact with deep learning algorithms during annotation. A typical workflow might involve using pre-trained segmentation models to generate masks and labels for each cell, which pathologists can then refine and verify. This approach significantly reduces the resource-intensive process of manually labeling each cell 43 . Additionally, pathologists can generate histological descriptions using specialized LLMs trained on pathological images. The copilot framework enables efficient and accurate annotation of digital pathology images, significantly improving existing diagnostic applications and integrated platforms 27 , 44 , 45 . I-Viewer offers a highly efficient data collection solution for foundation models. Training foundation models requires extensive datasets with high-quality labels to accurately recognize and interpret various tissue features 46 , 47 . However, diverse and high-quality annotation datasets at various magnifications are scarce 46 , 48 , 49 . This is partially because creating large-scale annotations necessitates collaboration between pathologists across institutions, which requires careful coordination and standardization to ensure consistency and reliability. Existing datasets, such as PathVQA and OpenPath, have limitations, including low-resolution annotations, non-pathological slides, and ambiguous question-answer pairs 19 , 50 . I-Viewer addresses these challenges by enabling verified users to upload high-resolution slides and create and modify annotations on a uniform platform. Importantly, by uploading these slides to a uniform platform, I-Viewer alleviates the need to share pathology images on social media - a practice endorsed by the pathology community 51 – 54 - as demonstrated by implementations at the United States and Canadian Academy for Pathology (USCAP) conferences 52 and preliminary application in building foundation models 19 . I-Viewer mediates information exchange between pathologists, medical trainees, and researchers. It enables real-time collaboration allowing multiple users to work on the same slide simultaneously. For instance, trainees can independently annotate ROIs and add comments, which can then be reviewed, modified, or verified by experienced pathologists. Besides benefitting medical trainees, the I-Viewer framework also supports researchers by reducing the model training and validation lifecycle. It provides flexible interfaces for uploading custom algorithms and pipelines, allowing researchers to receive pathologist feedback on model limitations and iteratively update models for improved performance. Additionally, all annotations, models, and expert feedback are stored on a single server, eliminating the need for complicated data transfers and enabling efficient collaboration among research groups. In all, I-Viewer enables cross-team collaboration feasibility and robust workflow, accelerating data collection and model evaluation between pathologists and model developers. With the human feedback strategy, this ecosystem enables efficient active learning to enhance the performance and development of reliable AI models. 4. Data and Methods 4.1 Datasets We evaluated the performance of our selected AI agents using diverse datasets across different tasks. For the nuclei segmentation task, we conducted two benchmarks that covered various cancer types, including the HD-Staining benchmark for NSCLC and the NuCLS benchmark for breast cancer 10 , 32 . For architectural pattern analysis and tissue interface, we collaborated with pathologists to curate an internal image captioning dataset. To demonstrate I-Viewer’s extensibility, we included the OPMD dataset to develop the onion pealing algorithm for analyzing epithelium layers 41 , 42 . HD-Staining benchmark. The HD-Staining benchmark is derived from the National Lung Screening Trial (NLST) dataset 55 , encompassing 127 patches (500 × 500 pixels) extracted from 39 pathological ROIs of Lung Adenocarcinoma patients. These image patches were divided into training, validation, and testing sets based on slide IDs. Specifically, 105 patches from 29 slides were designated for training, 12 patches from 5 slides for validation, and the remaining 10 patches from 5 slides for testing. A board-certified pathologist (L.J.) manually labeled and segmented the nuclei in these patches into six distinct categories: tumor nuclei, stromal nuclei, lymphocyte nuclei, macrophage nuclei, red blood cells, and karyorrhexis. The training set included over 12,000 cell nuclei, distributed as follows: 24.1% tumor nuclei, 23.9% stromal nuclei, 29.5% lymphocytes, 5.8% red blood cells, and 16.7% other categories. The validation and testing sets comprised 1227 and 1086 nuclei, respectively. NuCLS benchmark. The NuCLS benchmark consists of numerous image patches and annotations curated from breast cancer images within the TCGA dataset 56 . These pathological patches were divided into single-rater and multi-rater datasets (the latter being used as an inferred P-truth testing dataset). Nuclei in the patches were annotated collaboratively by pathologists, pathology residents, and medical students. The single-rater annotations used for model training and validation were provided by a team of 25 non-pathologists and subsequently corrected and validated by one of seven experienced pathologists. While the multi-rater dataset, validated by multiple pathologists, serves as an independent testing dataset. The corrected single-rater dataset includes 1744 image patches, 54,916 annotated nuclei with class labels, and 27,976 unlabeled annotations. The multi-rater evaluation dataset comprises 53 image patches, 1203 labeled annotations, and 150 unlabeled annotations. Nuclei were categorized into 20 subcategories and 4 super-classes: tumor, stromal, stromal tumor-infiltrating lymphocytes (sTILs), and others. All models in this study were trained and validated on the corrected single-rater dataset and subsequently evaluated on the multi-rater dataset. Performance results are reported for the super-classes to facilitate comparison with other models. Image captioning dataset. The internal image captioning dataset is collected from 233 image patches from 40 ROIs in lung adenocarcinoma and colorectal cancer (CRC). Each patch includes detailed descriptions from board-certified pathologist (Z.C.) regarding nuclei morphologies, secondary structure architectures, and tumor growth patterns. Specifically, for nuclei morphologies, we focused on nuclear pleomorphism, nuclear-to-cytoplasmic ratio, and prominent nucleoli; for architectural patterns, we reported the presence of alveolar, cribriform, glandular, hobnailed, lepidic, papillary, and trabecular; regarding growth patterns, we classified them into circumscribed, encapsulated, infiltrative, and solid growth. Definitions of each pathological terms are based on the Chap. 2 of the book "The practice of surgical pathology: a beginner's guide to the diagnostic process" 57 . Oral premalignant disorder (OPMD) dataset. The OPMD dataset contains 14,425 image patches for epithelium (n = 7,812), connective tissue (n = 4,022), and background (n = 2,591). The image patches are collected from hematoxylin and eosin (H&E)-stained sections of oral mucosal biopsies from 173 subjects. These subjects include 135 patients with OPMD cases retrieved from the EPOC trial at The University of Texas MD Anderson Cancer Center, 13 samples of clinically normal oral epithelium of oral mucosal biopsies, and 25 samples of oral squamous cell carcinomas (OSCC) from oral cavity locations and human papillomavirus (HPV) negative cases, which were randomly selected from TCGA Head and Neck Squamous Cell Carcinoma (HNSCC) dataset. All slides were scanned at 40× magnification and evaluated by an expert oral pathologist to annotate regions of the epithelium (tumor or nondysplastic), connective tissue, and background. 4.2 I-Viewer Infrastructure The I-Viewer system consists of three components: user management system (4.2.1), frontend interface (4.2.2), and backend services (4.2.3). The user management system controls permissions to access slides and create/modify/delete annotations. The frontend interface displays images and annotations, providing tools for annotating slides, interacting with other users, and interfacing with AI. The backend services organize AI agents and manage databases. The overall infrastructure and connections between these modules are illustrated in Supplementary Fig. 3 . From a functional perspective, all components work together to enable users to examine and annotate pathology slides with AI assistance (Fig. 4). Once granted permission, users can browse pathology slides and existing annotations in their web browser, manually annotate images, use AI agents for annotation, or collaborate with other users on existing annotations. Users can also communicate with the AI copilot for additional information and pathological insights. The I-Viewer Agentic-RAG router automatically triggers the necessary AI agents and delivers results based on user inputs. Finalized interpretations and annotations are saved in databases for histology description writing and diagnostic decision support. These database entries can also be used for model development and agent refinement. 4.2.1 User Management System I-Viewer offers a web-based user management system utilizing the PHP Laravel framework, ensuring a secure platform where access to images and annotations is strictly limited to authorized users. The system supports two user roles: administrators and regular users. Administrators can import images in batches, manage slide visibility for specific users or groups, and adjust API settings. Regular users can create and modify annotations, as well as pull annotations from the database through APIs. The user management system ensures a proper annotation sequence for human-in-the-loop analysis, allowing multiple users to collaborate on the same slides simultaneously using both manual annotation tools and AI annotators without interference. All annotations can be updated in real-time, with their sources clearly marked to distinguish between user-generated and AI-generated annotations. 4.2.2 Frontend Interface I-Viewer employed OpenSeadragon (OSD) 58 to display whole slide images. We developed a synchronized dual viewer that presents the same image region in two distinct window frames, allowing users to compare original images, annotated images, and annotations from various users and models together. A high-performance OSD plugin using konva.js and a priority queue ensures smooth rendering of extensive annotation volumes, enabling users to view and interact with thousands of annotations across different users and projects without experiencing delays. To optimize display quality while conserving memory resources, our priority queue prioritizes displaying newly added and modified annotations, gradually adjusting the presentation of smaller annotations when users zoom in or out. When browsing slides, users can add, modify, and delete annotations, assign keyword tags, and leave comments for other users in a pop-out window. I-Viewer utilized Gradio 59 to render the copilot frontend, which includes a comment frame, an image view with download buttons, and a chatbot dialogue window. These components allow users to modify existing annotations, use the AI copilot to generate new comments, and conduct multi-round vision question answering (VQA) about the contents in the ROI. 4.2.3 Backend Services The I-Viewer system employs a combination of message queue, model registry, microservices, and Agentic-RAG to deploy AI models for assisting in automatic annotation. It integrates three cutting-edge AI agents to tackle common challenges in histological annotation. Nuclei segmentation agents. Pre-trained HD-Yolo algorithms are employed for nuclei detection, classification, and segmentation. The HD-Yolo algorithm utilizes the Yolo framework for detection and classification, coupled with a lightweight fully convolutional network (FCN) header for enhanced segmentation results. Trained on extensive nuclei segmentation datasets across different cancer types, HD-Yolo outperforms existing algorithms in both accuracy and speed, particularly in densely stacked nuclei regions. With a single 32GB V100 GPU, HD-Yolo can annotate an ROI region of 2560 x 2560 instantly and an ROI region of 100k x 100k in less than 5 minutes. HD-Yolo functions as an auto-annotation tool, instantly detecting and displaying all nuclei within the current viewport when users zoom in to 10x magnification level. The algorithm is deployable on both CPU and GPU platforms, serialized to the Open Neural Network Exchange (ONNX) format for CPU servers and the TensorRT format for GPU servers. MLLM ROI caption agents. Pre-trained MLLMs were utilized to generate ROI-level captions 60 . MLLMs are versatile image captioning agents capable of describing intrinsic image contents such as objects, structures, and content relationships. For example, LLaVA-Med adapted the LLaVA 61 , 62 framework to the biomedical domain using internal multimodal instruction-following data, achieving state-of-the-art performance among open-source foundation models on certain biomedical metrics. To further enhance the model's understanding of histological features and domain-specific content, we developed the HD-LLaVA model, which leveraged the pretrained Conch 28 foundation image encoder, the LLaMA3-13b LLM, and with LLaVA-1.6 framework. The projection layer was fine-tuned using a combination of the publicly available OpenPath and PathVQA dataset reevaluated by GPT-4V. For advanced performance, commercial GPT-4V/GPT-4o models were provided as second opinions if users had permissions to upload the de-identified and encoded image ROIs to the OpenAI server (data security was outlined in section 4.2.4 ). The I-Viewer system integrates LLaVA-Med 37 , HD-LLaVA, and GPT-4V/GPT-4o to identify various tissue components, architectural patterns and growth patterns, such as glands, papillary, and infiltrative growth. The LLaVA-Med and HD-LLaVA models were deployed under the Ollama framework, and the GPT models were accessed via the OpenAI API. Tumor-Microenvironment Feature Analysis Agents. We utilized the TME feature extraction pipeline previously published with HD-Yolo. The pipeline analyzed nuclei morphologies, distributions, and interactions with the surrounding environment, providing statistics on nuclei densities, morphology differentiation, nuclei-nuclei interactions, and potentially identifying interactions with tissues such as tumor infiltration and metastasis. Upon user selection of an ROI, the TME analysis agent provided additional statistical details about the region by automatically retrieving all existing annotations and summarizing these interaction features, thereby eliminating the need for users to repeatedly zoom in and out. Agentic-RAG router. The Agentic-RAG system offered significant advantages in organizing multiple AI agents, thereby enhancing efficiency, transparency, and solving conflicts between agents while providing consistent response for complex user queries. The overall workflow of the I-Viewer Agentic-RAG system is illustrated in Fig. 5 . The system utilized a vector database and index search engine to map user prompt embedding to the existing prompt embeddings. Then the system extracted all agents from the top 3 hits to form a candidate agent list. The candidate agent list was then triggered in topological order to generate initial responses for a given ROI. The aggregation component then summarized results from different agents, utilizing confidence scores and voting strategy to resolve conflicts if any. The refined initial results were parsed into LLM with medical domain knowledge, specifically GPT-4 and LLaMA3, to summarize the results and deliver the responses to the user. Several optimizations were implemented in the Agentic-RAG system to improve information retrieval efficiency and accuracy. Firstly, intermediate results from each agent were cached for reuse to avoid redundant computations during multi-round QA tasks initiated by users. Secondly, to address the cold-start problem, predefined prompts and corresponding candidate agent lists were stored in the vector database. Thirdly, I-Viewer relied on human feedback to refine its vector database. Users could confirm the correctness of the current answer or select agents if results did not meet their requirements. I-Viewer will update the vector database based on user feedback, thereby enhancing robustness and providing personalized QA for users. By integrating various specialized AI tools, the RAG system dynamically retrieved and generated relevant information tailored to specific questions, ensuring more accurate and contextually appropriate responses. This approach streamlined the process of identifying the best-suited agent for a given task and mitigated conflicts between agents by aggregating diverse results into coherent summaries. The Agentic-RAG system in I-Viewer was implemented using LlamaIndex. To filter out non-pathological questions, unsafe queries, and sensitive information, the RAG pipeline was safeguarded with Llama Guard 63 for both user prompts and AI responses. 4.2.4 Secured Data Transfer and Communication Frontend and backend communication . Proxy server and token authentication was employed to secure communications between users, the frontend, and the backend. Proxy servers enhanced privacy, improved network performance through caching and load balancing, while providing security benefits like filtering malicious content and monitoring internet usage. The token authentication system verified user identities for incoming requests and managed user permissions to access images, modify annotations, and trigger certain backend services. Additionally, distributed locks were utilized in job message queues to resolve conflicting database transactions among users and AIs with varying permission levels. This approach prevented unauthorized users from accidentally modifying existing annotations and ensured that content remained consistently updated across multiple user accesses. De-identification and encryption . Each image underwent a de-identification process to ensure that none of the thumbnail image, tag image, or headers containing patient-sensitive information could be accessed by incoming requests from the frontend or backend. When a query was sent from the proxy server, the system validated the transaction by checking the user UUID, image UUID, and authentication code. Subsequently, the proxy server sent requests to access file fragments restricted to specified coordinates. The file binary was then encoded into a base64 byte string during the transaction, ensuring that no raw information could be accessed through injection. The input byte string was decoded for frontend rendering and agent analysis on the backend server. 4.3 Evaluation Methods We compared I-Viewer HD-Yolo agents with existing models Yolo-seg, Hover-Net, and Stardist regarding both accuracy and speed. To evaluate the performance of different algorithms, we reported detection precision, recall, F-1 score, segmentation mIoU, and inference time as performance indicators. We further compared I-Viewer ROI agents with existing foundation models: GPT-4V, LLaVA-Med, PLIP, and Conch. We selected 18 nuclei morphologies, architecture patterns, and tissue interface histology terms covered by the descriptions to form our dictionary. Specifically, for nuclei morphologies, we focused on nuclear pleomorphism, nuclear-to-cytoplasmic ratio, and prominent nucleoli; for architectural patterns, we reported the presence of alveolar, cribriform, glandular, hobnailed, lepidic, papillary, and trabecular; regarding growth patterns, we classified them into circumscribed, encapsulated, infiltrative, and solid growth. For image-text matching foundation models Conch and PLIP, trained under CLIP framework, we calculated cosine similarities between the embeddings of image contents and the embeddings of keywords/short phrases from our dictionary. Keywords with a cosine similarity greater than 0.1 are considered potential outcomes of the model. For MLLMs, we used ChatGPT to summarize image captions and QA results into dictionaries, then performed 1-gram matching between generated keywords and ground truth keywords. We reported precision and recall rates for all models. It is important to note that we used existing LLMs to aggregate and summarize information from different agents instead of fine-tuning the LLMs for specific tasks. Therefore, the evaluation of general language understanding (GLUE), multi-task understanding (MMLU), and language professionalism (ROUGE, diversity) fell beyond the scope of this research. 5. Data availability The NuCLS dataset used to train HD-Yolo can be downloaded from: https://sites.google.com/view/nucls/home . The OpenPath dataset used to finetune MLLM can be downloaded from: https://drive.google.com/drive/folders/1b5UT8BzUphkHZavRG-fmiyY9JWYIWZER , we extract the image URLs in the CSV file and downloaded the original image from Twitter website. The PathVQA dataset is downloaded through HuggingFace dataset: https://huggingface.co/datasets/flaviagiammarino/path-vqa . The sample slide from TCGA used in video demo can be downloaded from the following link: https://drive.google.com/file/d/1KFV4r_hXpBjvE5BbDoethwYjktGriyS_/view?usp=sharing . 6. Code availability The I-Viewer is distributed as a docker-composed system and its source codes are hosted at GitHub: https://github.com/QBRC/iviewer_copilot . We listed the main backend components as follows: 1) the HD-Yolo component that implements the cell segmentation and classification task ( https://github.com/QBRC/iviewer_copilot/tree/master/nuclei ); 2) the AI copilot backend that integrates Agentic-RAG, LLM, and pathological copilot system: https://github.com/QBRC/iviewer_copilot/tree/master/copilot ; 3) the image retrieval backend: https://github.com/QBRC/iviewer_copilot/tree/master/deepzoom ; 4) the annotation database backend that stores annotations from users and AI models: https://github.com/QBRC/iviewer_copilot/tree/master/annotation . Declarations Author Contribution R.R, I.G., X.Z. and G.X. wrote the main manuscript text.D.L., J. W. S. Y. developed the software.P.Q., I.V. provided computational support.Z.C. and P.L provided insights on project development.D.M., Y.X., X.Z. and G.X provide project supervision.R.R, I.G. prepared figures 1-5 and table 1-3. All authors reviewed the manuscript. References Shafi, S. & Parwani, A. V. Artificial intelligence in diagnostic pathology. Diagn Pathol 18, 109 (2023). https://doi.org/10.1186/s13000-023-01375-z Amgad, M. et al. A population-level digital histologic biomarker for enhanced prognosis of invasive breast cancer. Nature medicine 30, 85–97 (2024). https://doi.org/10.1038/s41591-023-02643-7 Manjunath, B. et al. in 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro , 2006. 255–258 (IEEE). Boor, P. Deep learning applications in digital pathology. Nat Rev Nephrol (2024). https://doi.org/10.1038/s41581-024-00870-w Huang, Z. et al. A pathologist-AI collaboration framework for enhancing diagnostic accuracies and efficiencies. Nat Biomed Eng (2024). https://doi.org/10.1038/s41551-024-01223-5 Wang, S. et al. Deep learning of cell spatial organizations identifies clinically relevant insights in tissue images. Nat Commun 14, 7872 (2023). https://doi.org/10.1038/s41467-023-43172-8 Shmatko, A., Ghaffari Laleh, N., Gerstung, M. & Kather, J. N. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nat Cancer 3, 1026–1038 (2022). https://doi.org/10.1038/s43018-022-00436-4 Graham, S. et al. Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med Image Anal 58, 101563 (2019). https://doi.org/10.1016/j.media.2019.101563 Wang, S., Yang, D. M., Rong, R., Zhan, X. & Xiao, G. Pathology Image Analysis Using Segmentation Deep Learning Algorithms. The American journal of pathology 189, 1686–1698 (2019). https://doi.org/10.1016/j.ajpath.2019.05.007 Amgad, M. et al. NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer. Gigascience 11 (2022). https://doi.org/10.1093/gigascience/giac037 Villanueva-Miranda, I. et al. Enhancing Medical Imaging Segmentation with GB-SAM: A Novel Approach to Tissue Segmentation Using Granular Box Prompts. Cancers (Basel) 16 (2024). https://doi.org/10.3390/cancers16132391 Vu, Q. D. et al. Methods for Segmentation and Classification of Digital Microscopy Tissue Images. Front Bioeng Biotechnol 7, 53 (2019). https://doi.org/10.3389/fbioe.2019.00053 Xu, H. et al. A whole-slide foundation model for digital pathology from real-world data. Nature 630, 181–188 (2024). https://doi.org/10.1038/s41586-024-07441-w Jiang, Y., Yang, M., Wang, S., Li, X. & Sun, Y. Emerging role of deep learning-based artificial intelligence in tumor pathology. Cancer Commun (Lond) 40, 154–166 (2020). https://doi.org/10.1002/cac2.12012 Jiang, R. et al. A transformer-based weakly supervised computational pathology method for clinical-grade diagnosis and molecular marker discovery of gliomas. Nature Machine Intelligence (2024). https://doi.org/10.1038/s42256-024-00868-w Kather, J. N. et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Med 16, e1002730 (2019). https://doi.org/10.1371/journal.pmed.1002730 Cheng, J. et al. Identification of topological features in renal tumor microenvironment associated with patient survival. Bioinformatics 34, 1024–1030 (2018). https://doi.org/10.1093/bioinformatics/btx723 Huang, J. et al. A critical assessment of using ChatGPT for extracting structured data from clinical notes. NPJ Digit Med 7, 106 (2024). https://doi.org/10.1038/s41746-024-01079-8 Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J. & Zou, J. A visual-language foundation model for pathology image analysis using medical Twitter. Nature medicine 29, 2307–2316 (2023). https://doi.org/10.1038/s41591-023-02504-3 Khosravi, M., Zare, Z., Mojtabaeian, S. M. & Izadi, R. Artificial Intelligence and Decision-Making in Healthcare: A Thematic Analysis of a Systematic Review of Reviews. Health Services Research and Managerial Epidemiology 11, 23333928241234863 (2024). https://doi.org/10.1177/23333928241234863 Challa, B. et al. Artificial Intelligence-Aided Diagnosis of Breast Cancer Lymph Node Metastasis on Histologic Slides in a Digital Workflow. Mod Pathol 36, 100216 (2023). https://doi.org/10.1016/j.modpat.2023.100216 Acs, B., Rantalainen, M. & Hartman, J. Artificial intelligence as the next step towards precision pathology. J Intern Med 288, 62–81 (2020). https://doi.org/10.1111/joim.13030 Ilhan, B., Guneri, P. & Wilder-Smith, P. The contribution of artificial intelligence to reducing the diagnostic delay in oral cancer. Oral Oncol 116, 105254 (2021). https://doi.org/10.1016/j.oraloncology.2021.105254 Panayides, A. S. et al. AI in Medical Imaging Informatics: Current Challenges and Future Directions. IEEE J Biomed Health Inform 24, 1837–1857 (2020). https://doi.org/10.1109/JBHI.2020.2991043 Rasoolijaberi, M. et al. Multi-Magnification Image Search in Digital Pathology. IEEE J Biomed Health Inform 26, 4611–4622 (2022). https://doi.org/10.1109/JBHI.2022.3181531 Gutman, D. A. et al. The Digital Slide Archive: A Software Platform for Management, Integration, and Analysis of Histology for Cancer Research. Cancer Res 77, e75-e78 (2017). https://doi.org/10.1158/0008-5472.CAN-17-0629 Bankhead, P. et al. QuPath: Open source software for digital pathology image analysis. Sci Rep 7, 16878 (2017). https://doi.org/10.1038/s41598-017-17204-5 Lu, M. Y. et al. A visual-language foundation model for computational pathology. Nature medicine 30, 863–874 (2024). https://doi.org/10.1038/s41591-024-02856-4 Rong, R. et al. A Deep Learning Approach for Histology-Based Nucleus Segmentation and Tumor Microenvironment Characterization. Mod Pathol 36, 100196 (2023). https://doi.org/10.1016/j.modpat.2023.100196 Cardoso, M. J. et al. Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701 (2022). Lu, M. Y. et al. A Multimodal Generative AI Copilot for Human Pathology. Nature (2024). https://doi.org/10.1038/s41586-024-07618-3 Wang, S., Yang, D. M., Rong, R., Zhan, X. & Xiao, G. Pathology image analysis using segmentation deep learning algorithms. The American journal of pathology 189, 1686–1698 (2019). Weigert, M. & Schmidt, U. in 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC). 1–4 (IEEE). Weigert, M., Schmidt, U., Haase, R., Sugawara, K. & Myers, G. in Proceedings of the IEEE/CVF winter conference on applications of computer vision. 3666–3673. Schmidt, U., Weigert, M., Broaddus, C. & Myers, G. in Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16–20 , 2018, Proceedings, Part II 11. 265–273 (Springer). Radford, A. et al. in Proceedings of the 38th International Conference on Machine Learning Vol. 139 (eds Meila Marina & Zhang Tong) 8748–8763 (PMLR, Proceedings of Machine Learning Research, 2021). Li, C. et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (2024). Achiam, J. et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). Yang, Z. et al. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 9, 1 (2023). Gilman, I. et al. OpenSeadragon. (2024). Zhang, X. et al. A Deep Learning Onion Peeling Approach to Measure Oral Epithelium Layer Number. Cancers 15 (2023). Zhang, X. et al. Deep learning-based pathology image analysis predicts cancer progression risk in patients with oral leukoplakia. Cancer Medicine 12, 7508–7518 (2023). https://doi.org/https://doi.org/10.1002/cam4.5478 Wen, Z. et al. Deep Learning-Based H-Score Quantification of Immunohistochemistry-Stained Images. Mod Pathol 37, 100398 (2024). https://doi.org/10.1016/j.modpat.2023.100398 Ali, H. R. et al. Computational pathology of pre-treatment biopsies identifies lymphocyte density as a predictor of response to neoadjuvant chemotherapy in breast cancer. Breast Cancer Res 18, 21 (2016). https://doi.org/10.1186/s13058-016-0682-8 da Silva, L. M. et al. Independent real-world application of a clinical-grade automated prostate cancer detection system. J Pathol 254, 147–158 (2021). https://doi.org/10.1002/path.5662 Thirunavukarasu, A. J. et al. Large language models in medicine. Nat Med 29, 1930–1940 (2023). https://doi.org/10.1038/s41591-023-02448-8 Bhinder, B., Gilvary, C., Madhukar, N. S. & Elemento, O. Artificial Intelligence in Cancer Research and Precision Medicine. Cancer Discov 11, 900–915 (2021). https://doi.org/10.1158/2159-8290.CD-21-0090 Kaplan, J. et al. Scaling laws for neural language models. arXiv preprint arXiv :2001. 08361 (2020). Wang, Y. et al. Enhancing semantic segmentation in chest X-ray images through image preprocessing: ps-KDE for pixel-wise substitution by kernel density estimation. PLoS One 19, e0299623 (2024). https://doi.org/10.1371/journal.pone.0299623 He, X., Zhang, Y., Mou, L., Xing, E. & Xie, P. Pathvqa: 30000 + questions for medical visual question answering. arXiv preprint arXiv :2003. 10286 (2020). Crane, G. M. & Gardner, J. M. Pathology Image-Sharing on Social Media: Recommendations for Protecting Privacy While Motivating Education. AMA J Ethics 18, 817–825 (2016). https://doi.org/10.1001/journalofethics.2016.18.8.stas1-1608 Cohen, D. et al. #InSituPathologists: how the #USCAP2015 meeting went viral on Twitter and founded the social media movement for the United States and Canadian Academy of Pathology. Mod Pathol 30, 160–168 (2017). https://doi.org/10.1038/modpathol.2016.223 Allen, T. C. Social media: pathologists' force multiplier. Arch Pathol Lab Med 138, 1000–1001 (2014). https://doi.org/10.5858/arpa.2014-0071-ED Misialek, M. J. & Allen, T. C. You're On Social Media! So Now What? Arch Pathol Lab Med 140, 393 (2016). https://doi.org/10.5858/arpa.2015-0493-ED Kramer, B. S., Berg, C. D., Aberle, D. R. & Prorok, P. C. Vol. 18 109–111 (SAGE Publications Sage UK: London, England, 2011). Tomczak, K., Czerwińska, P. & Wiznerowicz, M. Review The Cancer Genome Atlas (TCGA): an immeasurable source of knowledge. Contemporary Oncology/Współczesna Onkologia 2015, 68–77 (2015). Molavi, D. W. The practice of surgical pathology: a beginner's guide to the diagnostic process . (Springer, 2017). Foundation, C. An open-source, web-based viewer for high-resolution zoomable images, implemented in pure JavaScript, for desktop and mobile , (2009). Abid, A. et al. Gradio: Hassle-free sharing and testing of ml models in the wild. arXiv preprint arXiv :1906. 02569 (2019). Caffagni, D. et al. The (R) Evolution of Multimodal Large Language Models: A Survey. arXiv preprint arXiv:2402.12451 (2024). Liu, H., Li, C., Wu, Q. & Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems 36 (2024). Liu, H., Li, C., Li, Y. & Lee, Y. J. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26296–26306. Inan, H. et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 (2023). Additional Declarations No competing interests reported. Supplementary Files SupplementaryInformation.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5404747","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":376134418,"identity":"46ee3e3d-485d-4beb-a7c9-719b4a0ed09e","order_by":0,"name":"Ruichen Rong","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Ruichen","middleName":"","lastName":"Rong","suffix":""},{"id":376134419,"identity":"ac76ecb3-e1ef-44f7-b62b-a5af76112919","order_by":1,"name":"Danni Luo","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Danni","middleName":"","lastName":"Luo","suffix":""},{"id":376134420,"identity":"c60d9233-b97d-4bc7-9804-5b1fc3896d3c","order_by":2,"name":"Zifan Gu","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Zifan","middleName":"","lastName":"Gu","suffix":""},{"id":376134421,"identity":"34fe8286-d3b5-4b8e-b882-6fbf0301dc4a","order_by":3,"name":"Peiran Quan","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Peiran","middleName":"","lastName":"Quan","suffix":""},{"id":376134422,"identity":"4d0e1a8d-89cd-4fdd-bc66-1c7cf5f16411","order_by":4,"name":"Ismael Villanueva-Miranda","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Ismael","middleName":"","lastName":"Villanueva-Miranda","suffix":""},{"id":376134423,"identity":"9093f977-d41e-47e3-84fd-67a0f22deadb","order_by":5,"name":"Jonathan Wang","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Jonathan","middleName":"","lastName":"Wang","suffix":""},{"id":376134424,"identity":"fd599907-2c36-4abe-bc92-38ccbeafef85","order_by":6,"name":"Shengjie Yang","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Shengjie","middleName":"","lastName":"Yang","suffix":""},{"id":376134425,"identity":"f0904bb5-40dc-4df7-8a42-7258cbd63eb2","order_by":7,"name":"Zhikai Chi","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Zhikai","middleName":"","lastName":"Chi","suffix":""},{"id":376134426,"identity":"176c16e7-d7d6-44cc-8865-452fd6f36a38","order_by":8,"name":"Patrick Leavey","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Patrick","middleName":"","lastName":"Leavey","suffix":""},{"id":376134427,"identity":"1400b29b-3ef4-4450-9295-11b7c36cd6c9","order_by":9,"name":"Donghan M. Yang","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Donghan","middleName":"M.","lastName":"Yang","suffix":""},{"id":376134429,"identity":"2b2587b5-25a6-450c-9e83-eb27d321aec5","order_by":10,"name":"Yang Xie","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Yang","middleName":"","lastName":"Xie","suffix":""},{"id":376134431,"identity":"8654555e-6d74-48ff-bdc6-9f166f9ae217","order_by":11,"name":"Xiaowei Zhan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABLUlEQVRIie2QMUvDQBiGLxxclpNbv+PE3xAJREUkfyWhQxarQ6F0kBAoZArO13+hi3MkEJdYV0dLoLMupUOtfq2liGl0dbhn+T5e3ufuOEIMhn+IU6wHEF/jeEs/fBzsW4G1KxYq1iiFMCGUfUUtikw2y0qheykE5C9F2HRS88ExoWo4rc9QEXBZYhLHQj3m5LVf/FQYZa7LKyBsvzxyuyn4UndsTAqQ1xeBpcc7FOIpbBIOwXqxbp47DJccnIo7+NSGIqg9U90lEIBopk62yjIGf6W8NxU55Hh4AsSBc09ZWyWh4HBUrKbiFLznLkrgqPRlVkE4ympPLcpCarzlPhtHDeXp4Xair+IDX0d3MB+AL+xwKjERIuOHL/P+aeOXN/Ddcd7WNxgMBsOvfAIpBGI4ExycJwAAAABJRU5ErkJggg==","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":true,"prefix":"","firstName":"Xiaowei","middleName":"","lastName":"Zhan","suffix":""},{"id":376134434,"identity":"60164885-cdfb-49d9-8c0a-650d5fa96bf3","order_by":12,"name":"Guanghua Xiao","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Guanghua","middleName":"","lastName":"Xiao","suffix":""}],"badges":[],"createdAt":"2024-11-06 17:38:36","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5404747/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5404747/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":69894260,"identity":"86bb7bad-8ed1-4bb7-9e36-d13e7195cb0e","added_by":"auto","created_at":"2024-11-26 10:59:31","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":2484366,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"Figure118.png","url":"https://assets-eu.researchsquare.com/files/rs-5404747/v1/aeddd2cde360e7226c35baba.png"},{"id":69894262,"identity":"f425aacd-5f2e-4d4f-9d8c-97706b9eacbe","added_by":"auto","created_at":"2024-11-26 10:59:32","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":4668805,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"Figure215.png","url":"https://assets-eu.researchsquare.com/files/rs-5404747/v1/ae740015640d5bf941493604.png"},{"id":69894258,"identity":"4cd37194-03fe-44bf-ad89-0cc61a78a435","added_by":"auto","created_at":"2024-11-26 10:59:31","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":2866512,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"Figure39.png","url":"https://assets-eu.researchsquare.com/files/rs-5404747/v1/98b310a00f493207326b7bf1.png"},{"id":69895024,"identity":"73f55231-b65a-4002-adb9-09736c77f7a8","added_by":"auto","created_at":"2024-11-26 11:07:31","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":356878,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"Figure45.png","url":"https://assets-eu.researchsquare.com/files/rs-5404747/v1/8ff25fc60589b6a8827cf246.png"},{"id":69895025,"identity":"1c530f18-c81d-4885-a3eb-6bc2a9150037","added_by":"auto","created_at":"2024-11-26 11:07:31","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":361770,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"Figure54.png","url":"https://assets-eu.researchsquare.com/files/rs-5404747/v1/58244a4fde24150aa0ae24e6.png"},{"id":76627918,"identity":"ea636637-c2d5-4bfc-8523-35fac3afe7a4","added_by":"auto","created_at":"2025-02-19 06:03:40","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":12465216,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5404747/v1/488e1e90-cdb4-4a91-abf0-3dd2fe4d4aff.pdf"},{"id":69894264,"identity":"e7fbd1d9-6316-47e1-9aeb-bda827c50349","added_by":"auto","created_at":"2024-11-26 10:59:32","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":497051,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryInformation.docx","url":"https://assets-eu.researchsquare.com/files/rs-5404747/v1/071c6f60d72d8d77ab9f0659.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"I-Viewer: An Online Digital Pathology Analysis Platform with Agentic-RAG AI Copilot","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eRecent advancements in medical imaging technology have transformed the field of digital pathology, enabling the production of high-resolution whole-slide images (WSI) that capture intricate tissue structures and morphological patterns at a large scale\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. The conventional manual assessment of WSI is subjective, time-consuming and prone to errors\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e,\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e. In response, various artificial intelligence (AI) algorithms have been developed to improve accuracy, reproducibility, and efficiency in digital pathology analysis\u003csup\u003e\u003cspan additionalcitationids=\"CR6\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e. These AI algorithms have demonstrated success across a spectrum of tasks, including nuclei and tissues segmentation\u003csup\u003e\u003cspan additionalcitationids=\"CR9 CR10\" citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e, subtype classification\u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e,\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e, diagnosis\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e,\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e,\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e, and patient survival prognosis\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e,\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e. The recent emergence of foundation models and Large Language Models (LLMs) has further expanded the potential for integrating knowledge bases and extracting information in pathology\u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e,\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e. These advanced AI algorithms have the potentials to streamline the human-AI collaboration analysis lifecycle of digital pathology images. Studies demonstrate that AI assistance can significantly reduce the time required for report writing and decision-making\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e. Additionally, AI can enhance diagnostic accuracy, particularly for pathologists with varying levels of expertise\u003csup\u003e\u003cspan additionalcitationids=\"CR22\" citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eHowever, significant challenges remain in developing AI tools that match human-expert-level visual assessment and histological analyses\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e. One major challenge is mimicking the pathologist's routine of examining tissues under various magnifications, as current AI algorithms struggle to provide real-time translation of histological findings from individual nuclei to the broader tissue level\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e. This limitation is particularly problematic under high magnification, where regions of interest (ROI) may encompass thousands of nuclei and key tissue structure components, requiring highly efficient algorithms for detection and extraction of morphological features. Another challenging is to develop efficient pipelines that can understand positional relationships and features within the tumor microenvironment (TME) across various AI tools and different magnification levels. For instance, identifying tumor nuclei within lymph nodes requires the algorithms to \u0026ldquo;understand\u0026rdquo; sophisticated spatial context. Such capabilities are crucial for accurately interpreting the intricate relationships between different tissue components and their surrounding environment.\u003c/p\u003e \u003cp\u003eGiven these challenges, the development of comprehensive AI solutions for digital pathology remains a complex task. Existing platforms lack the infrastructure to integrate diverse AI functionalities, such as object identification, TME feature extraction, and medical question answering, into a unified system that enables collaborative user-AI interactions and result refinement. This again deviates from the pathologists\u0026rsquo; routine of performing and sharing most analysis work in a centralized software environment. While some initiatives have explored integrated solutions, they have notable limitations. For example, Digital Slide Archive (HistomicsUI) enables multiple users to access and annotate the same slides but lacks efficient integration with its AI tools (HistomicsTK), hindering real-time annotation capabilities\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e. On the other hand, standalone software like QuPath offers flexible analysis and visualization tools for digital pathology but falls short in providing robust collaboration features essential for pathologists\u0026rsquo; workflows\u003csup\u003e\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e. Moreover, the recent developments of LLMs and foundation models, which require significant computational resources, further challenges the analysis capability and efficiency of these standalone frameworks\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e. Additionally, the growing number of pre-trained AI assistants makes it crucial to aggregate and infer results from different analysis algorithms and AI tools\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. Existing platforms lack the capability to trigger appropriate assistants (AI modules or functionalities) for user-specific questions and resolve conflicts among different assistants while accurately summarizing results. This highlights the need for an advanced AI system capable of responding to user inquiries, integrating them into ongoing analysis, and providing iterative refinement based on expert feedback. Such a system is essential for not only enhancing current diagnostic processes but also developing a comprehensive data and annotation collection solution, a key component in advancing future foundation models in digital pathology.\u003c/p\u003e \u003cp\u003eIn response to these challenges, we introduce I-Viewer: a comprehensive online framework designed to facilitate collaborative pathology analysis empowered by an AI Copilot. I-Viewer deploys advanced real-time AI agents for nuclei classification and segmentation, tissue structure and spatial pattern analysis, and LLM-enabled question answering. At its core, I-Viewer utilizes a Retrieval Augmented Generation (RAG) based copilot system, which combines retrieval of relevant information with generation capabilities to organize these agents, improving the efficiency, steerability, and transparency of complex question answering. The RAG integrates the power of retrieval mechanisms with generative models enhancing the quality and relevance of generated outputs by augmenting LLMs with retrieval-based techniques. This hybrid approach facilitates better contextual understanding and reduces the risk of generating irrelevant or incorrect information.\u003c/p\u003e \u003cp\u003eThe I-Viewer platform also provides flexible frontend and backend interfaces, allowing the enhancement and customization of AI solutions. This adaptability enables the seamless integration of new algorithms and methodologies as the field evolves. By integrating human experts' collective expertise with AI's analytical capabilities, I-Viewer fosters collaboration among pathologists, trainees, and researchers. It serves as a unified platform that simplifies scientific research and diagnostic processes, ultimately improving patient care and advancing our understanding of complex diseases. Through this integrated approach, I-Viewer has the potential to bridge the gap between advanced AI technologies and practical clinical applications in digital pathology.\u003c/p\u003e"},{"header":"2. Results","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 I-Viewer Overview\u003c/h2\u003e \u003cp\u003eI-Viewer provides real-time assistance through deep learning models, LLMs, and specialized analysis pipelines, in order to enrich histological insights for user and optimize pathologists' workflows. This systems address key challenges using advanced techniques: 1) real-time nuclei segmentation and classification with HD-Yolo; 2) identification of architectural pattern, tissue interface, and spatial relationships across various magnification levels via multimodal large language models (MLLMs) and TME feature extraction pipeline\u003csup\u003e\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e; and 3) aggregation of agent level results with Agentic-RAG system to target user queries, infer potential discoveries, informed by medical domain knowledge. These innovations (summarized in \u003cb\u003eSupplementary Texts S1\u003c/b\u003e) enable I-Viewer to provide comprehensive, context-aware assistance in digital pathology analysis.\u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e presents a comprehensive comparison of I-Viewer with current pathological AI platforms, including MONAI\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e, HistomicsUI\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e, QuPath\u003csup\u003e\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e, Nucleo.AI\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e, and PathChat\u003csup\u003e\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e. The comparison spans several different aspects: multi-user collaboration, AI assistant at various levels, AI copilot functionality, online/offline agent capabilities, support for customized pipelines, and Agentic-RAG integration. This analysis highlights I-Viewer's advantages in support for multi-user online collaboration, flexible AI agent capabilities, and an advanced AI copilot system. The results demonstrate I-Viewer as a uniquely comprehensive solution in the current landscape of pathological AI platforms.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eFunctionality comparison of I-Viewer with existing software platforms.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eI-Viewer\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMONAI\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNucleo.AI\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eQuPath\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eHistomicsUI\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003ePathChat\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMulti-user collaboration\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eNuclei level AI assistant\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eROI level AI assistant\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMLLM Chatbot /AI copilot\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eOnline\u0026thinsp;+\u0026thinsp;offline agents\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eCustomized pipelines\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAgentic-RAG AI copilot\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Real-time Nuclei Classification and Segmentation\u003c/h2\u003e \u003cp\u003eI-Viewer incorporates the state-of-the-art HD-Yolo algorithm\u003csup\u003e\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e for real-time nuclei segmentation, classification, and morphological feature extraction. Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents a comprehensive comparative analysis of various algorithms across two benchmark datasets for nuclei segmentation: the non-small cell lung cancer (NSCLC) HD-Staining dataset\u003csup\u003e\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e and breast cancer NuCLS dataset\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e. In both benchmark datasets, HD-Yolo demonstrates superior performance over existing algorithms (Hover-Net\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e, Yolov8-seg, and Stardist\u003csup\u003e\u003cspan additionalcitationids=\"CR34\" citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e\u003c/sup\u003e) in both segmentation and classification tasks, as evidenced by metrics including precision, recall, F1, mean intersection over union (mIoU), and inference time (summarized in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). HD-Yolo's superior performance can be attributed to its unique lightweight mask header and attention mechanism, which significantly enhance segmentation accuracy. Notably, HD-Yolo surpasses Yolov8-seg in accuracy while maintaining comparable inference speeds, achieving a balance between performance and computational efficiency. This combination of high accuracy and speed makes HD-Yolo particularly well-suited for the real-time analysis demands of digital pathology workflows within the I-Viewer framework.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eNuclei detection, classification, and segmentation results.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDataset\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eF1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003emIoU\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eInference time (s)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003e\u003cb\u003eNSCLC\u003c/b\u003e\u003c/p\u003e \u003cp\u003e\u003cb\u003e(Non-small cell lung cancer)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHD-Yolo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e0.831\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.674\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e0.741\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e0.842\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.004\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eYolov8-seg\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.811\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.648\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.720\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.748\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.004\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStardist*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.732\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.583\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.649\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.795\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.009\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHover-Net\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.660\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.395\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.581\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.772\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.101\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003e\u003cb\u003eNuCLS\u003c/b\u003e\u003c/p\u003e \u003cp\u003e\u003cb\u003e(Breast Cancer)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHD-Yolo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e0.924\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.866\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e0.894\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.819\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.004\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eYolov8-seg\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.901\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.802\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.848\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.773\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.004\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStardist*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.880\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.750\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.810\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.798\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.010\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHover-Net\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.820\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.751\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.784\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e0.820\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.083\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"7\"\u003e\u003cem\u003eIn the benchmarking tasks, we used HD-Yolo, Yolov8-seg, Stardist, and Hover-Net on NSCLC and NuCLS datasets. Inference time is measured in seconds per image.\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"7\"\u003e\u003cem\u003e*For Stardist, only detection performance is reported due to label mismatches between model outputs and dataset labels. Boldface indicates best performance.\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eFigure 1\u003c/b\u003e presents one example comparing the segmentation results of various models: Hover-Net and Stardist fail to segment occluded nuclei and often split large nuclei into smaller pieces, while Yolov8-seg separates occluded nuclei with distinct boundaries. In contrast, HD-Yolo generates smoothed masks and accurately segments occluded nuclei, demonstrating its advanced capabilities in handling complex nuclear arrangements typically encountered in histopathological images.\u003c/p\u003e \u003cp\u003eI-Viewer leverages the efficiency and accuracy of the HD-Yolo algorithm to provide real-time segmentation results in response to user viewport changes. This approach eliminates the need for offline pre-calculation or the division of WSIs into smaller ROIs, a requirement of most existing tools\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. Instead, I-Viewer dynamically tracks the user's viewport during panning or zooming operations, starting at 10x magnification. It streams image patches within the ROI to a background job queue and Redis cache at their original resolutions (refer to Section \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e, \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003eI-Viewer Infrastructure\u003c/span\u003e). HD-Yolo agents analyze these patches concurrently, ensuring scalability and optimizing processing efficiency. This design ensures that the service can automatically adapt to varying workloads, thereby optimizing processing efficiency. The result is a responsive system that delivers high-quality, real-time nuclei segmentation and classification, enhancing both user experience and workflow efficiency in digital pathology analysis.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Architectural Pattern Analysis and Tissue Interface\u003c/h2\u003e \u003cp\u003eI-Viewer aggregates task-specific agents to recognize general histopathological patterns, including nuclear morphology, secondary architecture, and growth pattern interpretation. These agents leverage MLLMs, and the TME analysis pipeline to extract relevant information from ROIs (refer to Section \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e, \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003eI-Viewer Infrastructure\u003c/span\u003e). Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e presents the performance of different approaches on the identification of nuclear morphology, secondary structures and architectures, and tumor growth patterns. The evaluation methods differ based on the model type: Contrastive Language-Image Pre-Training (CLIP)\u003csup\u003e\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e\u003c/sup\u003e-based models PLIP\u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e and Conch\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e are evaluated by the cosine similarity between the ROI and predefined keywords lists of potential patterns and features, whereas MLLMs, GPT-4V, HD-LLaVA, and LLaVA-Med\u003csup\u003e\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e\u003c/sup\u003e, are evaluated by their accuracy in answering multiple-choice questions. We observed that GPT-4V outperformed other open-source MLLMs, likely due to its advanced image encoder, robust language model, and extensive high-quality image captioning dataset\u003csup\u003e\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e,\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e\u003c/sup\u003e. However, GPT-4V has a limitation: it struggles to generate high-confidence identifications when presented with large ROIs containing numerous details. This is primarily because morphology and structures related to nuclei-tissue interactions are challenging to identify at low magnifications.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance comparison of different foundation models on architecture pattern and tissue interface recognition.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eModels\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNuclear/Cell Morphology\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eStructures and Architectures\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTumor Growth Pattern and Interface with Normal Tissue\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePrecision\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eI-Viewer Agents\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.803\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.775\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.667\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGPT-4V\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e0.840\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cem\u003e0.750\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e0.733\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHD-LLaVA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.531\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.704\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.271\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLLaVA-Med\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.521\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.435\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.250\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eConch\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.183\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.203\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.212\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePLIP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.204\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.188\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.232\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRecall\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eI-Viewer Agents\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e0.885\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.912\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e0.783\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGPT-4V\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003e0.844\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cem\u003e0.706\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003e0.342\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHD-LLaVA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.301\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.685\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.224\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLLaVA-Med\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.334\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.294\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.231\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eConch\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.243\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.311\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.187\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePLIP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.234\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.285\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.130\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTo overcome the limitations of single-scale analysis, I-Viewer incorporates a comprehensive TME analysis pipeline that assesses histological characteristics across multiple magnifications. This pipeline consolidates data on nuclei density distribution, shape morphology statistics, and intricate spatial relationships between nuclei and secondary structures. By integrating information across different magnifications, I-Viewer agents achieve high confidence in identifying critical features, including nuclear pleomorphism, tumor-infiltrating lymphocytes and tumor metastasis. As demonstrated in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, the I-Viewer system significantly enhances recall rates across all three aspects of analysis (nuclear morphology, secondary structures and architectures, and tumor growth patterns) by synergistically combining information from GPT-4V, HD-LLaVA, and the TME feature extraction pipeline. This multi-model, multi-scale approach enables I-Viewer to overcome the limitations of individual models and provide more comprehensive and accurate histopathological assessments.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4 LLM with Medical Domain Knowledge for AI-Human Collaboration\u003c/h2\u003e \u003cp\u003eEnsuring the effectiveness and consistency of LLM-based AI assistant is crucial in medical applications. This involves accurately identifying and mitigating hallucinations, addressing specific inquiries, and providing insights through a human-in-the-loop approach. At its core, I-Viewer features a sophisticated chatbot engine based on LLaMA3 and GPT-4. The engine enables the provision of comprehensive summaries informed by medical knowledge, synthesizing results from various AI agents. In addition, I-Viewer employs an Agentic-RAG caching system (refer to Section \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e, \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003eI-Viewer Infrastructure\u003c/span\u003e) to efficiently retrieve analysis results from multiple specialized agents, ensuring that the LLM's responses are guided by concrete, relevant data. Thereby, this approach significantly reduces inconsistency, potential errors, and hallucinations in the AI's output.\u003c/p\u003e \u003cp\u003eBy leveraging advancements in both LLMs and RAG technologies, I-Viewer can integrate information from various sources, produce detailed histology descriptions, and engage with user queries responsively. These features enable users to conduct multi-round question-answering sessions efficiently with I-Viewer. Users can perform common tasks such as summarizing histology reports and inferring potential risks, and incorporating their analyses in the I-Viewer annotation tool. This workflow enhances the accuracy and reliability of pathological assessments while maintaining the critical role of human expertise in the decision-making process.\u003c/p\u003e \u003cp\u003e \u003cb\u003eFigure 2\u003c/b\u003e presents a comparative analysis of histology insights synthesized by I-Viewer, GPT-4V, and LLaVA-Med (prompts are shown in \u003cb\u003eSupplementary Fig.\u0026nbsp;1\u003c/b\u003e). In the lung adenocarcinoma study (Fig.\u0026nbsp;2A), the I-Viewer copilot demonstrates high accuracy in identifying key features, including papillary structures, nuclei differentiation, and infiltrative growth patterns. Moreover, I-Viewer agents enhance their analysis by providing additional observations on nuclei composition and distribution. In the colon cancer ROI (Fig.\u0026nbsp;2B), I-Viewer and GPT-4V offer consistent interpretations of growth patterns and architectural characteristics, demonstrating their concordance in analyzing different types of cancer images.\u003c/p\u003e \u003cp\u003eTo maintain response consistency and caution, I-Viewer uses the Agentic-RAG system and LLM safeguards. When users pose questions that extend beyond basic histological observations, such as requests for diagnosis or prognosis, the system deliberately limits its responses to avoid potential hallucinations or overreach. For instance, in \u003cb\u003eFig.\u0026nbsp;2A\u003c/b\u003e, I-Viewer refrains from providing definitive diagnostic or prognostic information based solely on the ROI. Instead, it indicates the need for additional user inputs to ensure more informed and responsible insights. This approach maintains the system's reliability while respecting the complexities and nuances of medical interpretation.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e2.5 I-Viewer is an Extensible Framework for Researchers\u003c/h2\u003e \u003cp\u003eI-Viewer is designed to be scalable and adaptable, allowing for the improvement of current models and the easy addition of new AI models. This design ensures that I-Viewer can evolve alongside advancements in digital pathology and AI technologies. The extensible architecture of I-Viewer is built upon several key features. Its loosely coupled system ensures user-friendliness for both biologists and machine learning engineers. The frontend incorporates customized high-performance OpenSeadragon (OSD)\u003csup\u003e\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e\u003c/sup\u003e plugins, enabling flexible rendering of annotations across various services, users, and projects. The robust backend architecture utilizes message queues, microservices, and model registries, simplifying the deployment of existing pipelines and AI models. Additionally, I-Viewer offers RESTful APIs for adaptable data curation, supporting model development and fine-tuning. This comprehensive infrastructure allows custom analysis pipelines to be integrated into I-Viewer through a straightforward four-step process. (refer to \u003cb\u003eSupplementary Texts S2\u003c/b\u003e and \u003cb\u003eSupplementary Fig.\u0026nbsp;1\u003c/b\u003e).\u003c/p\u003e \u003cp\u003eTo demonstrate I-Viewer's extensibility, we integrate the onion peeling algorithm into I-Viewer for analyzing epithelium layers in pathology images for oral premalignant disorder (OPMD)\u003csup\u003e\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e\u003c/sup\u003e. Accurate quantification of epithelial layers is essential for evaluating the severity of epithelial dysplasia, a precursor to oral cancer, which aids in early detection, clinical grading, and patient prognosis\u003csup\u003e\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e\u003c/sup\u003e. The onion peeling algorithm relies on pre-calculated nuclei detection and region segmentation, making real-time counting of epithelial layers challenging for existing frameworks. To address this limitation, I-Viewer leverages its model registries and the Agentic-RAG system to integrate the onion peeling pipeline as an agent to automatically retrieve nuclei and ROI information from interconnected agents, facilitating real-time delivery of algorithm results. As illustrated in \u003cb\u003eFig.\u0026nbsp;3\u003c/b\u003e, I-Viewer efficiently annotates epithelium layers produced by the onion peeling algorithm, demonstrating capabilities that surpass those of LLMs and other existing frameworks. This successful integration demonstrates I-Viewer's flexibility, allowing users to deploy new AI algorithms and leverage the copilot system for information retrieval and integration.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e2.6 I-Viewer is a Secure and User-friendly Online Platform for Interactive Pathology Image Sharing and Annotation\u003c/h2\u003e \u003cp\u003eI-Viewer is a cloud-based platform that provides a secure and collaborative environment for digital pathology through robust user authorization and authentication mechanisms. It supports simultaneous user interactions with AI, enabling joint annotation and communication across shared slides and projects. I-Viewer's collaborative capabilities include real-time collaboration where multiple users can work on the same slide simultaneously, enhancing knowledge sharing. The platform offers AI-assisted annotations, allowing users to leverage AI algorithms for automated annotations while retaining the ability to modify and refine results. The access control ensures appropriate levels of access and functionality for different team members, from trainees to senior pathologists.\u003c/p\u003e \u003cp\u003eTo accommodate common pathologist needs, I-Viewer has an intuitive layer-based visualization system. This frontend framework organizes original WSIs, AI-generated results, and user annotations into a structured hierarchy. Users can selectively display individual layers and manipulate objects and annotations across different layers at various zoom levels. I-Viewer also features dual-view panels, facilitating cross-examination between different layers to compare differences and enhance the interpretation of AI outputs. To demonstrate I-Viewer's functionality, we provide two supplementary videos: \u003cb\u003eSupplementary Video 1\u003c/b\u003e showcases how users can modify AI-predicted segmentation shapes, descriptions, and types. These adjustments are automatically tagged with the user's ID, distinguishing them from AI-generated annotations. This feature ensures clarity between manual edits and AI outputs, enhancing traceability and accountability. \u003cb\u003eSupplementary Video 2\u003c/b\u003e illustrates users interacting with the copilot system to identify secondary structures and growth patterns, generate ROI descriptions, and infer potential discoveries. Notably, I-Viewer's Agentic-RAG system can initiate nuclei counting and TME density analysis pipelines in response to user queries about nuclei summaries and growth patterns.\u003c/p\u003e \u003cp\u003eI-Viewer enhances transparency by clearly referencing sources and providing rationale for generated answers, fostering trust and understanding between clinicians and AI. This improved interpretability allows users to grasp the reasoning behind complex solutions, facilitating informed decision-making. The Agentic-RAG system in I-Viewer represents a sophisticated approach to effectively utilize the capabilities of multiple AI agents, offering a powerful tool for comprehensive pathological analysis.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Discussions and Conclusion","content":"\u003cp\u003eIn this study, we present a novel digital pathology analysis platform power by the Agent-RAG AI copilot system. I-Viewer assembles robust model collections and provides a highly efficient infrastructure that promotes collaborations among pathologists, researchers, and AI. This platform represents the first-of-its-kind open-source ecosystem to effectively integrate task-specific deep learning algorithms, LLMs, and user inputs to enhance clinical care and biomedical research.\u003c/p\u003e \u003cp\u003e \u003cb\u003eI-Viewer serves as an AI copilot to improve the annotation process.\u003c/b\u003e To demonstrate its clinical relevance and broad compatibility, we applied I-Viewer to non-small cell lung cancer and breast cancer tissues. This showcases how pathologists can seamlessly interact with deep learning algorithms during annotation. A typical workflow might involve using pre-trained segmentation models to generate masks and labels for each cell, which pathologists can then refine and verify. This approach significantly reduces the resource-intensive process of manually labeling each cell\u003csup\u003e\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e\u003c/sup\u003e. Additionally, pathologists can generate histological descriptions using specialized LLMs trained on pathological images. The copilot framework enables efficient and accurate annotation of digital pathology images, significantly improving existing diagnostic applications and integrated platforms\u003csup\u003e\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e,\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e,\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e \u003cb\u003eI-Viewer offers a highly efficient data collection solution for foundation models.\u003c/b\u003e Training foundation models requires extensive datasets with high-quality labels to accurately recognize and interpret various tissue features\u003csup\u003e\u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e,\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e\u003c/sup\u003e. However, diverse and high-quality annotation datasets at various magnifications are scarce\u003csup\u003e\u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e,\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e48\u003c/span\u003e,\u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e49\u003c/span\u003e\u003c/sup\u003e. This is partially because creating large-scale annotations necessitates collaboration between pathologists across institutions, which requires careful coordination and standardization to ensure consistency and reliability. Existing datasets, such as PathVQA and OpenPath, have limitations, including low-resolution annotations, non-pathological slides, and ambiguous question-answer pairs\u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e,\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e\u003c/sup\u003e. I-Viewer addresses these challenges by enabling verified users to upload high-resolution slides and create and modify annotations on a uniform platform. Importantly, by uploading these slides to a uniform platform, I-Viewer alleviates the need to share pathology images on social media - a practice endorsed by the pathology community\u003csup\u003e\u003cspan additionalcitationids=\"CR52 CR53\" citationid=\"CR51\" class=\"CitationRef\"\u003e51\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR54\" class=\"CitationRef\"\u003e54\u003c/span\u003e\u003c/sup\u003e - as demonstrated by implementations at the United States and Canadian Academy for Pathology (USCAP) conferences\u003csup\u003e\u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e52\u003c/span\u003e\u003c/sup\u003e and preliminary application in building foundation models\u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e \u003cb\u003eI-Viewer mediates information exchange between pathologists, medical trainees, and researchers.\u003c/b\u003e It enables real-time collaboration allowing multiple users to work on the same slide simultaneously. For instance, trainees can independently annotate ROIs and add comments, which can then be reviewed, modified, or verified by experienced pathologists. Besides benefitting medical trainees, the I-Viewer framework also supports researchers by reducing the model training and validation lifecycle. It provides flexible interfaces for uploading custom algorithms and pipelines, allowing researchers to receive pathologist feedback on model limitations and iteratively update models for improved performance. Additionally, all annotations, models, and expert feedback are stored on a single server, eliminating the need for complicated data transfers and enabling efficient collaboration among research groups.\u003c/p\u003e \u003cp\u003eIn all, I-Viewer enables cross-team collaboration feasibility and robust workflow, accelerating data collection and model evaluation between pathologists and model developers. With the human feedback strategy, this ecosystem enables efficient active learning to enhance the performance and development of reliable AI models.\u003c/p\u003e"},{"header":"4. Data and Methods","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Datasets\u003c/h2\u003e \u003cp\u003eWe evaluated the performance of our selected AI agents using diverse datasets across different tasks. For the nuclei segmentation task, we conducted two benchmarks that covered various cancer types, including the HD-Staining benchmark for NSCLC and the NuCLS benchmark for breast cancer\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e,\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e. For architectural pattern analysis and tissue interface, we collaborated with pathologists to curate an internal image captioning dataset. To demonstrate I-Viewer\u0026rsquo;s extensibility, we included the OPMD dataset to develop the onion pealing algorithm for analyzing epithelium layers\u003csup\u003e\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e,\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e \u003cb\u003eHD-Staining benchmark.\u003c/b\u003e The HD-Staining benchmark is derived from the National Lung Screening Trial (NLST) dataset\u003csup\u003e\u003cspan citationid=\"CR55\" class=\"CitationRef\"\u003e55\u003c/span\u003e\u003c/sup\u003e, encompassing 127 patches (500 \u0026times; 500 pixels) extracted from 39 pathological ROIs of Lung Adenocarcinoma patients. These image patches were divided into training, validation, and testing sets based on slide IDs. Specifically, 105 patches from 29 slides were designated for training, 12 patches from 5 slides for validation, and the remaining 10 patches from 5 slides for testing. A board-certified pathologist (L.J.) manually labeled and segmented the nuclei in these patches into six distinct categories: tumor nuclei, stromal nuclei, lymphocyte nuclei, macrophage nuclei, red blood cells, and karyorrhexis. The training set included over 12,000 cell nuclei, distributed as follows: 24.1% tumor nuclei, 23.9% stromal nuclei, 29.5% lymphocytes, 5.8% red blood cells, and 16.7% other categories. The validation and testing sets comprised 1227 and 1086 nuclei, respectively.\u003c/p\u003e \u003cp\u003e \u003cb\u003eNuCLS benchmark.\u003c/b\u003e The NuCLS benchmark consists of numerous image patches and annotations curated from breast cancer images within the TCGA dataset\u003csup\u003e\u003cspan citationid=\"CR56\" class=\"CitationRef\"\u003e56\u003c/span\u003e\u003c/sup\u003e. These pathological patches were divided into single-rater and multi-rater datasets (the latter being used as an inferred P-truth testing dataset). Nuclei in the patches were annotated collaboratively by pathologists, pathology residents, and medical students. The single-rater annotations used for model training and validation were provided by a team of 25 non-pathologists and subsequently corrected and validated by one of seven experienced pathologists. While the multi-rater dataset, validated by multiple pathologists, serves as an independent testing dataset. The corrected single-rater dataset includes 1744 image patches, 54,916 annotated nuclei with class labels, and 27,976 unlabeled annotations. The multi-rater evaluation dataset comprises 53 image patches, 1203 labeled annotations, and 150 unlabeled annotations. Nuclei were categorized into 20 subcategories and 4 super-classes: tumor, stromal, stromal tumor-infiltrating lymphocytes (sTILs), and others. All models in this study were trained and validated on the corrected single-rater dataset and subsequently evaluated on the multi-rater dataset. Performance results are reported for the super-classes to facilitate comparison with other models.\u003c/p\u003e \u003cp\u003e \u003cb\u003eImage captioning dataset.\u003c/b\u003e The internal image captioning dataset is collected from 233 image patches from 40 ROIs in lung adenocarcinoma and colorectal cancer (CRC). Each patch includes detailed descriptions from board-certified pathologist (Z.C.) regarding nuclei morphologies, secondary structure architectures, and tumor growth patterns. Specifically, for nuclei morphologies, we focused on nuclear pleomorphism, nuclear-to-cytoplasmic ratio, and prominent nucleoli; for architectural patterns, we reported the presence of alveolar, cribriform, glandular, hobnailed, lepidic, papillary, and trabecular; regarding growth patterns, we classified them into circumscribed, encapsulated, infiltrative, and solid growth. Definitions of each pathological terms are based on the Chap.\u0026nbsp;2 of the book \"The practice of surgical pathology: a beginner's guide to the diagnostic process\"\u003csup\u003e\u003cspan citationid=\"CR57\" class=\"CitationRef\"\u003e57\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e \u003cb\u003eOral premalignant disorder (OPMD) dataset.\u003c/b\u003e The OPMD dataset contains 14,425 image patches for epithelium (n\u0026thinsp;=\u0026thinsp;7,812), connective tissue (n\u0026thinsp;=\u0026thinsp;4,022), and background (n\u0026thinsp;=\u0026thinsp;2,591). The image patches are collected from hematoxylin and eosin (H\u0026amp;E)-stained sections of oral mucosal biopsies from 173 subjects. These subjects include 135 patients with OPMD cases retrieved from the EPOC trial at The University of Texas MD Anderson Cancer Center, 13 samples of clinically normal oral epithelium of oral mucosal biopsies, and 25 samples of oral squamous cell carcinomas (OSCC) from oral cavity locations and human papillomavirus (HPV) negative cases, which were randomly selected from TCGA Head and Neck Squamous Cell Carcinoma (HNSCC) dataset. All slides were scanned at 40\u0026times; magnification and evaluated by an expert oral pathologist to annotate regions of the epithelium (tumor or nondysplastic), connective tissue, and background.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e4.2 I-Viewer Infrastructure\u003c/h2\u003e \u003cp\u003eThe I-Viewer system consists of three components: user management system (4.2.1), frontend interface (4.2.2), and backend services (4.2.3). The user management system controls permissions to access slides and create/modify/delete annotations. The frontend interface displays images and annotations, providing tools for annotating slides, interacting with other users, and interfacing with AI. The backend services organize AI agents and manage databases. The overall infrastructure and connections between these modules are illustrated in \u003cb\u003eSupplementary Fig.\u0026nbsp;3\u003c/b\u003e. From a functional perspective, all components work together to enable users to examine and annotate pathology slides with AI assistance (Fig.\u0026nbsp;4). Once granted permission, users can browse pathology slides and existing annotations in their web browser, manually annotate images, use AI agents for annotation, or collaborate with other users on existing annotations. Users can also communicate with the AI copilot for additional information and pathological insights. The I-Viewer Agentic-RAG router automatically triggers the necessary AI agents and delivers results based on user inputs. Finalized interpretations and annotations are saved in databases for histology description writing and diagnostic decision support. These database entries can also be used for model development and agent refinement.\u003c/p\u003e \u003cdiv id=\"Sec13\" class=\"Section3\"\u003e \u003ch2\u003e4.2.1 User Management System\u003c/h2\u003e \u003cp\u003eI-Viewer offers a web-based user management system utilizing the PHP Laravel framework, ensuring a secure platform where access to images and annotations is strictly limited to authorized users. The system supports two user roles: administrators and regular users. Administrators can import images in batches, manage slide visibility for specific users or groups, and adjust API settings. Regular users can create and modify annotations, as well as pull annotations from the database through APIs. The user management system ensures a proper annotation sequence for human-in-the-loop analysis, allowing multiple users to collaborate on the same slides simultaneously using both manual annotation tools and AI annotators without interference. All annotations can be updated in real-time, with their sources clearly marked to distinguish between user-generated and AI-generated annotations.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section3\"\u003e \u003ch2\u003e4.2.2 Frontend Interface\u003c/h2\u003e \u003cp\u003eI-Viewer employed OpenSeadragon (OSD)\u003csup\u003e\u003cspan citationid=\"CR58\" class=\"CitationRef\"\u003e58\u003c/span\u003e\u003c/sup\u003e to display whole slide images. We developed a synchronized dual viewer that presents the same image region in two distinct window frames, allowing users to compare original images, annotated images, and annotations from various users and models together. A high-performance OSD plugin using konva.js and a priority queue ensures smooth rendering of extensive annotation volumes, enabling users to view and interact with thousands of annotations across different users and projects without experiencing delays. To optimize display quality while conserving memory resources, our priority queue prioritizes displaying newly added and modified annotations, gradually adjusting the presentation of smaller annotations when users zoom in or out. When browsing slides, users can add, modify, and delete annotations, assign keyword tags, and leave comments for other users in a pop-out window. I-Viewer utilized Gradio\u003csup\u003e\u003cspan citationid=\"CR59\" class=\"CitationRef\"\u003e59\u003c/span\u003e\u003c/sup\u003e to render the copilot frontend, which includes a comment frame, an image view with download buttons, and a chatbot dialogue window. These components allow users to modify existing annotations, use the AI copilot to generate new comments, and conduct multi-round vision question answering (VQA) about the contents in the ROI.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section3\"\u003e \u003ch2\u003e4.2.3 Backend Services\u003c/h2\u003e \u003cp\u003eThe I-Viewer system employs a combination of message queue, model registry, microservices, and Agentic-RAG to deploy AI models for assisting in automatic annotation. It integrates three cutting-edge AI agents to tackle common challenges in histological annotation.\u003c/p\u003e \u003cp\u003e \u003cb\u003eNuclei segmentation agents.\u003c/b\u003e Pre-trained HD-Yolo algorithms are employed for nuclei detection, classification, and segmentation. The HD-Yolo algorithm utilizes the Yolo framework for detection and classification, coupled with a lightweight fully convolutional network (FCN) header for enhanced segmentation results. Trained on extensive nuclei segmentation datasets across different cancer types, HD-Yolo outperforms existing algorithms in both accuracy and speed, particularly in densely stacked nuclei regions. With a single 32GB V100 GPU, HD-Yolo can annotate an ROI region of 2560 x 2560 instantly and an ROI region of 100k x 100k in less than 5 minutes. HD-Yolo functions as an auto-annotation tool, instantly detecting and displaying all nuclei within the current viewport when users zoom in to 10x magnification level. The algorithm is deployable on both CPU and GPU platforms, serialized to the Open Neural Network Exchange (ONNX) format for CPU servers and the TensorRT format for GPU servers.\u003c/p\u003e \u003cp\u003e \u003cb\u003eMLLM ROI caption agents.\u003c/b\u003e Pre-trained MLLMs were utilized to generate ROI-level captions\u003csup\u003e\u003cspan citationid=\"CR60\" class=\"CitationRef\"\u003e60\u003c/span\u003e\u003c/sup\u003e. MLLMs are versatile image captioning agents capable of describing intrinsic image contents such as objects, structures, and content relationships. For example, LLaVA-Med adapted the LLaVA\u003csup\u003e\u003cspan citationid=\"CR61\" class=\"CitationRef\"\u003e61\u003c/span\u003e,\u003cspan citationid=\"CR62\" class=\"CitationRef\"\u003e62\u003c/span\u003e\u003c/sup\u003e framework to the biomedical domain using internal multimodal instruction-following data, achieving state-of-the-art performance among open-source foundation models on certain biomedical metrics. To further enhance the model's understanding of histological features and domain-specific content, we developed the HD-LLaVA model, which leveraged the pretrained Conch\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e foundation image encoder, the LLaMA3-13b LLM, and with LLaVA-1.6 framework. The projection layer was fine-tuned using a combination of the publicly available OpenPath and PathVQA dataset reevaluated by GPT-4V. For advanced performance, commercial GPT-4V/GPT-4o models were provided as second opinions if users had permissions to upload the de-identified and encoded image ROIs to the OpenAI server (data security was outlined in section \u003cspan refid=\"Sec16\" class=\"InternalRef\"\u003e4.2.4\u003c/span\u003e). The I-Viewer system integrates LLaVA-Med\u003csup\u003e\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e\u003c/sup\u003e, HD-LLaVA, and GPT-4V/GPT-4o to identify various tissue components, architectural patterns and growth patterns, such as glands, papillary, and infiltrative growth. The LLaVA-Med and HD-LLaVA models were deployed under the Ollama framework, and the GPT models were accessed via the OpenAI API.\u003c/p\u003e \u003cp\u003e \u003cb\u003eTumor-Microenvironment Feature Analysis Agents.\u003c/b\u003e We utilized the TME feature extraction pipeline previously published with HD-Yolo. The pipeline analyzed nuclei morphologies, distributions, and interactions with the surrounding environment, providing statistics on nuclei densities, morphology differentiation, nuclei-nuclei interactions, and potentially identifying interactions with tissues such as tumor infiltration and metastasis. Upon user selection of an ROI, the TME analysis agent provided additional statistical details about the region by automatically retrieving all existing annotations and summarizing these interaction features, thereby eliminating the need for users to repeatedly zoom in and out.\u003c/p\u003e \u003cp\u003e \u003cb\u003eAgentic-RAG router.\u003c/b\u003e The Agentic-RAG system offered significant advantages in organizing multiple AI agents, thereby enhancing efficiency, transparency, and solving conflicts between agents while providing consistent response for complex user queries. The overall workflow of the I-Viewer Agentic-RAG system is illustrated in \u003cb\u003eFig.\u0026nbsp;5\u003c/b\u003e. The system utilized a vector database and index search engine to map user prompt embedding to the existing prompt embeddings. Then the system extracted all agents from the top 3 hits to form a candidate agent list. The candidate agent list was then triggered in topological order to generate initial responses for a given ROI. The aggregation component then summarized results from different agents, utilizing confidence scores and voting strategy to resolve conflicts if any. The refined initial results were parsed into LLM with medical domain knowledge, specifically GPT-4 and LLaMA3, to summarize the results and deliver the responses to the user.\u003c/p\u003e \u003cp\u003eSeveral optimizations were implemented in the Agentic-RAG system to improve information retrieval efficiency and accuracy. Firstly, intermediate results from each agent were cached for reuse to avoid redundant computations during multi-round QA tasks initiated by users. Secondly, to address the cold-start problem, predefined prompts and corresponding candidate agent lists were stored in the vector database. Thirdly, I-Viewer relied on human feedback to refine its vector database. Users could confirm the correctness of the current answer or select agents if results did not meet their requirements. I-Viewer will update the vector database based on user feedback, thereby enhancing robustness and providing personalized QA for users.\u003c/p\u003e \u003cp\u003eBy integrating various specialized AI tools, the RAG system dynamically retrieved and generated relevant information tailored to specific questions, ensuring more accurate and contextually appropriate responses. This approach streamlined the process of identifying the best-suited agent for a given task and mitigated conflicts between agents by aggregating diverse results into coherent summaries. The Agentic-RAG system in I-Viewer was implemented using LlamaIndex. To filter out non-pathological questions, unsafe queries, and sensitive information, the RAG pipeline was safeguarded with Llama Guard\u003csup\u003e\u003cspan citationid=\"CR63\" class=\"CitationRef\"\u003e63\u003c/span\u003e\u003c/sup\u003e for both user prompts and AI responses.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section3\"\u003e \u003ch2\u003e4.2.4 Secured Data Transfer and Communication\u003c/h2\u003e \u003cp\u003e \u003cb\u003eFrontend and backend communication\u003c/b\u003e. Proxy server and token authentication was employed to secure communications between users, the frontend, and the backend. Proxy servers enhanced privacy, improved network performance through caching and load balancing, while providing security benefits like filtering malicious content and monitoring internet usage. The token authentication system verified user identities for incoming requests and managed user permissions to access images, modify annotations, and trigger certain backend services. Additionally, distributed locks were utilized in job message queues to resolve conflicting database transactions among users and AIs with varying permission levels. This approach prevented unauthorized users from accidentally modifying existing annotations and ensured that content remained consistently updated across multiple user accesses.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDe-identification and encryption\u003c/b\u003e. Each image underwent a de-identification process to ensure that none of the thumbnail image, tag image, or headers containing patient-sensitive information could be accessed by incoming requests from the frontend or backend. When a query was sent from the proxy server, the system validated the transaction by checking the user UUID, image UUID, and authentication code. Subsequently, the proxy server sent requests to access file fragments restricted to specified coordinates. The file binary was then encoded into a base64 byte string during the transaction, ensuring that no raw information could be accessed through injection. The input byte string was decoded for frontend rendering and agent analysis on the backend server.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Evaluation Methods\u003c/h2\u003e \u003cp\u003eWe compared I-Viewer HD-Yolo agents with existing models Yolo-seg, Hover-Net, and Stardist regarding both accuracy and speed. To evaluate the performance of different algorithms, we reported detection precision, recall, F-1 score, segmentation mIoU, and inference time as performance indicators. We further compared I-Viewer ROI agents with existing foundation models: GPT-4V, LLaVA-Med, PLIP, and Conch. We selected 18 nuclei morphologies, architecture patterns, and tissue interface histology terms covered by the descriptions to form our dictionary. Specifically, for nuclei morphologies, we focused on nuclear pleomorphism, nuclear-to-cytoplasmic ratio, and prominent nucleoli; for architectural patterns, we reported the presence of alveolar, cribriform, glandular, hobnailed, lepidic, papillary, and trabecular; regarding growth patterns, we classified them into circumscribed, encapsulated, infiltrative, and solid growth.\u003c/p\u003e \u003cp\u003eFor image-text matching foundation models Conch and PLIP, trained under CLIP framework, we calculated cosine similarities between the embeddings of image contents and the embeddings of keywords/short phrases from our dictionary. Keywords with a cosine similarity greater than 0.1 are considered potential outcomes of the model. For MLLMs, we used ChatGPT to summarize image captions and QA results into dictionaries, then performed 1-gram matching between generated keywords and ground truth keywords. We reported precision and recall rates for all models. It is important to note that we used existing LLMs to aggregate and summarize information from different agents instead of fine-tuning the LLMs for specific tasks. Therefore, the evaluation of general language understanding (GLUE), multi-task understanding (MMLU), and language professionalism (ROUGE, diversity) fell beyond the scope of this research.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Data availability","content":"\u003cp\u003eThe NuCLS dataset used to train HD-Yolo can be downloaded from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://sites.google.com/view/nucls/home\u003c/span\u003e\u003cspan address=\"https://sites.google.com/view/nucls/home\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. The OpenPath dataset used to finetune MLLM can be downloaded from:\u003c/p\u003e \u003cp\u003e \u003cspan class=\"ExternalRef\"\u003e \u003cspan class=\"RefSource\"\u003ehttps://drive.google.com/drive/folders/1b5UT8BzUphkHZavRG-fmiyY9JWYIWZER\u003c/span\u003e \u003cspan address=\"https://drive.google.com/drive/folders/1b5UT8BzUphkHZavRG-fmiyY9JWYIWZER\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e \u003c/span\u003e, we extract the image URLs in the CSV file and downloaded the original image from Twitter website. The PathVQA dataset is downloaded through HuggingFace dataset: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://huggingface.co/datasets/flaviagiammarino/path-vqa\u003c/span\u003e\u003cspan address=\"https://huggingface.co/datasets/flaviagiammarino/path-vqa\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. The sample slide from TCGA used in video demo can be downloaded from the following link: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://drive.google.com/file/d/1KFV4r_hXpBjvE5BbDoethwYjktGriyS_/view?usp=sharing\u003c/span\u003e\u003cspan address=\"https://drive.google.com/file/d/1KFV4r_hXpBjvE5BbDoethwYjktGriyS_/view?usp=sharing\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e"},{"header":"6. Code availability","content":"\u003cp\u003eThe I-Viewer is distributed as a docker-composed system and its source codes are hosted at GitHub: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/QBRC/iviewer_copilot\u003c/span\u003e\u003cspan address=\"https://github.com/QBRC/iviewer_copilot\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. We listed the main backend components as follows: 1) the HD-Yolo component that implements the cell segmentation and classification task (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/QBRC/iviewer_copilot/tree/master/nuclei\u003c/span\u003e\u003cspan address=\"https://github.com/QBRC/iviewer_copilot/tree/master/nuclei\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e); 2) the AI copilot backend that integrates Agentic-RAG, LLM, and pathological copilot system: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/QBRC/iviewer_copilot/tree/master/copilot\u003c/span\u003e\u003cspan address=\"https://github.com/QBRC/iviewer_copilot/tree/master/copilot\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e; 3) the image retrieval backend: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/QBRC/iviewer_copilot/tree/master/deepzoom\u003c/span\u003e\u003cspan address=\"https://github.com/QBRC/iviewer_copilot/tree/master/deepzoom\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e; 4) the annotation database backend that stores annotations from users and AI models: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/QBRC/iviewer_copilot/tree/master/annotation\u003c/span\u003e\u003cspan address=\"https://github.com/QBRC/iviewer_copilot/tree/master/annotation\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eR.R, I.G., X.Z. and G.X. wrote the main manuscript text.D.L., J. W. S. Y. developed the software.P.Q., I.V. provided computational support.Z.C. and P.L provided insights on project development.D.M., Y.X., X.Z. and G.X provide project supervision.R.R, I.G. prepared figures 1-5 and table 1-3. All authors reviewed the manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eShafi, S. \u0026amp; Parwani, A. V. Artificial intelligence in diagnostic pathology. Diagn Pathol 18, 109 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13000-023-01375-z\u003c/span\u003e\u003cspan address=\"10.1186/s13000-023-01375-z\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmgad, M. \u003cem\u003eet al.\u003c/em\u003e A population-level digital histologic biomarker for enhanced prognosis of invasive breast cancer. Nature medicine 30, 85\u0026ndash;97 (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-023-02643-7\u003c/span\u003e\u003cspan address=\"10.1038/s41591-023-02643-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eManjunath, B. \u003cem\u003eet al.\u003c/em\u003e in \u003cem\u003e3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro\u003c/em\u003e, 2006. 255\u0026ndash;258 (IEEE).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBoor, P. Deep learning applications in digital pathology. Nat Rev Nephrol (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41581-024-00870-w\u003c/span\u003e\u003cspan address=\"10.1038/s41581-024-00870-w\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang, Z. \u003cem\u003eet al.\u003c/em\u003e A pathologist-AI collaboration framework for enhancing diagnostic accuracies and efficiencies. Nat Biomed Eng (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41551-024-01223-5\u003c/span\u003e\u003cspan address=\"10.1038/s41551-024-01223-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, S. \u003cem\u003eet al.\u003c/em\u003e Deep learning of cell spatial organizations identifies clinically relevant insights in tissue images. Nat Commun 14, 7872 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41467-023-43172-8\u003c/span\u003e\u003cspan address=\"10.1038/s41467-023-43172-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShmatko, A., Ghaffari Laleh, N., Gerstung, M. \u0026amp; Kather, J. N. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nat Cancer 3, 1026\u0026ndash;1038 (2022). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s43018-022-00436-4\u003c/span\u003e\u003cspan address=\"10.1038/s43018-022-00436-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGraham, S. \u003cem\u003eet al.\u003c/em\u003e Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med Image Anal 58, 101563 (2019). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.media.2019.101563\u003c/span\u003e\u003cspan address=\"10.1016/j.media.2019.101563\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, S., Yang, D. M., Rong, R., Zhan, X. \u0026amp; Xiao, G. Pathology Image Analysis Using Segmentation Deep Learning Algorithms. The American journal of pathology 189, 1686\u0026ndash;1698 (2019). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.ajpath.2019.05.007\u003c/span\u003e\u003cspan address=\"10.1016/j.ajpath.2019.05.007\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmgad, M. \u003cem\u003eet al.\u003c/em\u003e NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer. Gigascience 11 (2022). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/gigascience/giac037\u003c/span\u003e\u003cspan address=\"10.1093/gigascience/giac037\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVillanueva-Miranda, I. \u003cem\u003eet al.\u003c/em\u003e Enhancing Medical Imaging Segmentation with GB-SAM: A Novel Approach to Tissue Segmentation Using Granular Box Prompts. Cancers (Basel) 16 (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/cancers16132391\u003c/span\u003e\u003cspan address=\"10.3390/cancers16132391\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVu, Q. D. \u003cem\u003eet al.\u003c/em\u003e Methods for Segmentation and Classification of Digital Microscopy Tissue Images. Front Bioeng Biotechnol 7, 53 (2019). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fbioe.2019.00053\u003c/span\u003e\u003cspan address=\"10.3389/fbioe.2019.00053\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXu, H. \u003cem\u003eet al.\u003c/em\u003e A whole-slide foundation model for digital pathology from real-world data. Nature 630, 181\u0026ndash;188 (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41586-024-07441-w\u003c/span\u003e\u003cspan address=\"10.1038/s41586-024-07441-w\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJiang, Y., Yang, M., Wang, S., Li, X. \u0026amp; Sun, Y. Emerging role of deep learning-based artificial intelligence in tumor pathology. Cancer Commun (Lond) 40, 154\u0026ndash;166 (2020). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/cac2.12012\u003c/span\u003e\u003cspan address=\"10.1002/cac2.12012\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJiang, R. \u003cem\u003eet al.\u003c/em\u003e A transformer-based weakly supervised computational pathology method for clinical-grade diagnosis and molecular marker discovery of gliomas. Nature Machine Intelligence (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s42256-024-00868-w\u003c/span\u003e\u003cspan address=\"10.1038/s42256-024-00868-w\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKather, J. N. \u003cem\u003eet al.\u003c/em\u003e Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Med 16, e1002730 (2019). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1371/journal.pmed.1002730\u003c/span\u003e\u003cspan address=\"10.1371/journal.pmed.1002730\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheng, J. \u003cem\u003eet al.\u003c/em\u003e Identification of topological features in renal tumor microenvironment associated with patient survival. Bioinformatics 34, 1024\u0026ndash;1030 (2018). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/bioinformatics/btx723\u003c/span\u003e\u003cspan address=\"10.1093/bioinformatics/btx723\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang, J. \u003cem\u003eet al.\u003c/em\u003e A critical assessment of using ChatGPT for extracting structured data from clinical notes. NPJ Digit Med 7, 106 (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41746-024-01079-8\u003c/span\u003e\u003cspan address=\"10.1038/s41746-024-01079-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J. \u0026amp; Zou, J. A visual-language foundation model for pathology image analysis using medical Twitter. Nature medicine 29, 2307\u0026ndash;2316 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-023-02504-3\u003c/span\u003e\u003cspan address=\"10.1038/s41591-023-02504-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKhosravi, M., Zare, Z., Mojtabaeian, S. M. \u0026amp; Izadi, R. Artificial Intelligence and Decision-Making in Healthcare: A Thematic Analysis of a Systematic Review of Reviews. Health Services Research and Managerial Epidemiology 11, 23333928241234863 (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/23333928241234863\u003c/span\u003e\u003cspan address=\"10.1177/23333928241234863\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChalla, B. \u003cem\u003eet al.\u003c/em\u003e Artificial Intelligence-Aided Diagnosis of Breast Cancer Lymph Node Metastasis on Histologic Slides in a Digital Workflow. Mod Pathol 36, 100216 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.modpat.2023.100216\u003c/span\u003e\u003cspan address=\"10.1016/j.modpat.2023.100216\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAcs, B., Rantalainen, M. \u0026amp; Hartman, J. Artificial intelligence as the next step towards precision pathology. J Intern Med 288, 62\u0026ndash;81 (2020). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/joim.13030\u003c/span\u003e\u003cspan address=\"10.1111/joim.13030\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIlhan, B., Guneri, P. \u0026amp; Wilder-Smith, P. The contribution of artificial intelligence to reducing the diagnostic delay in oral cancer. Oral Oncol 116, 105254 (2021). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.oraloncology.2021.105254\u003c/span\u003e\u003cspan address=\"10.1016/j.oraloncology.2021.105254\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePanayides, A. S. \u003cem\u003eet al.\u003c/em\u003e AI in Medical Imaging Informatics: Current Challenges and Future Directions. IEEE J Biomed Health Inform 24, 1837\u0026ndash;1857 (2020). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/JBHI.2020.2991043\u003c/span\u003e\u003cspan address=\"10.1109/JBHI.2020.2991043\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRasoolijaberi, M. \u003cem\u003eet al.\u003c/em\u003e Multi-Magnification Image Search in Digital Pathology. IEEE J Biomed Health Inform 26, 4611\u0026ndash;4622 (2022). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/JBHI.2022.3181531\u003c/span\u003e\u003cspan address=\"10.1109/JBHI.2022.3181531\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGutman, D. A. \u003cem\u003eet al.\u003c/em\u003e The Digital Slide Archive: A Software Platform for Management, Integration, and Analysis of Histology for Cancer Research. Cancer Res 77, e75-e78 (2017). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1158/0008-5472.CAN-17-0629\u003c/span\u003e\u003cspan address=\"10.1158/0008-5472.CAN-17-0629\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBankhead, P. \u003cem\u003eet al.\u003c/em\u003e QuPath: Open source software for digital pathology image analysis. Sci Rep 7, 16878 (2017). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-017-17204-5\u003c/span\u003e\u003cspan address=\"10.1038/s41598-017-17204-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLu, M. Y. \u003cem\u003eet al.\u003c/em\u003e A visual-language foundation model for computational pathology. Nature medicine 30, 863\u0026ndash;874 (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-024-02856-4\u003c/span\u003e\u003cspan address=\"10.1038/s41591-024-02856-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRong, R. \u003cem\u003eet al.\u003c/em\u003e A Deep Learning Approach for Histology-Based Nucleus Segmentation and Tumor Microenvironment Characterization. Mod Pathol 36, 100196 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.modpat.2023.100196\u003c/span\u003e\u003cspan address=\"10.1016/j.modpat.2023.100196\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCardoso, M. J. \u003cem\u003eet al.\u003c/em\u003e Monai: An open-source framework for deep learning in healthcare. \u003cem\u003earXiv preprint arXiv:2211.02701\u003c/em\u003e (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLu, M. Y. \u003cem\u003eet al.\u003c/em\u003e A Multimodal Generative AI Copilot for Human Pathology. Nature (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41586-024-07618-3\u003c/span\u003e\u003cspan address=\"10.1038/s41586-024-07618-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, S., Yang, D. M., Rong, R., Zhan, X. \u0026amp; Xiao, G. Pathology image analysis using segmentation deep learning algorithms. The American journal of pathology 189, 1686\u0026ndash;1698 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWeigert, M. \u0026amp; Schmidt, U. in \u003cem\u003e2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC).\u003c/em\u003e 1\u0026ndash;4 (IEEE).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWeigert, M., Schmidt, U., Haase, R., Sugawara, K. \u0026amp; Myers, G. in \u003cem\u003eProceedings of the IEEE/CVF winter conference on applications of computer vision.\u003c/em\u003e 3666\u0026ndash;3673.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchmidt, U., Weigert, M., Broaddus, C. \u0026amp; Myers, G. in \u003cem\u003eMedical Image Computing and Computer Assisted Intervention\u0026ndash;MICCAI 2018: 21st International Conference, Granada, Spain, September 16\u0026ndash;20\u003c/em\u003e, 2018, \u003cem\u003eProceedings, Part II 11.\u003c/em\u003e 265\u0026ndash;273 (Springer).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRadford, A. \u003cem\u003eet al.\u003c/em\u003e in \u003cem\u003eProceedings of the 38th International Conference on Machine Learning\u003c/em\u003e Vol. 139 (eds Meila Marina \u0026amp; Zhang Tong) 8748\u0026ndash;8763 (PMLR, Proceedings of Machine Learning Research, 2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi, C. \u003cem\u003eet al.\u003c/em\u003e Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAchiam, J. \u003cem\u003eet al.\u003c/em\u003e Gpt-4 technical report. \u003cem\u003earXiv preprint arXiv:2303.08774\u003c/em\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang, Z. \u003cem\u003eet al.\u003c/em\u003e The dawn of lmms: Preliminary explorations with gpt-4v (ision). \u003cem\u003earXiv preprint arXiv:2309.17421\u003c/em\u003e 9, 1 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGilman, I. \u003cem\u003eet al.\u003c/em\u003e OpenSeadragon. (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang, X. \u003cem\u003eet al.\u003c/em\u003e A Deep Learning Onion Peeling Approach to Measure Oral Epithelium Layer Number. Cancers 15 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang, X. \u003cem\u003eet al.\u003c/em\u003e Deep learning-based pathology image analysis predicts cancer progression risk in patients with oral leukoplakia. Cancer Medicine 12, 7508\u0026ndash;7518 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/https://doi.org/10.1002/cam4.5478\u003c/span\u003e\u003cspan address=\"10.1002/cam4.5478\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWen, Z. \u003cem\u003eet al.\u003c/em\u003e Deep Learning-Based H-Score Quantification of Immunohistochemistry-Stained Images. Mod Pathol 37, 100398 (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.modpat.2023.100398\u003c/span\u003e\u003cspan address=\"10.1016/j.modpat.2023.100398\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAli, H. R. \u003cem\u003eet al.\u003c/em\u003e Computational pathology of pre-treatment biopsies identifies lymphocyte density as a predictor of response to neoadjuvant chemotherapy in breast cancer. Breast Cancer Res 18, 21 (2016). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13058-016-0682-8\u003c/span\u003e\u003cspan address=\"10.1186/s13058-016-0682-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eda Silva, L. M. \u003cem\u003eet al.\u003c/em\u003e Independent real-world application of a clinical-grade automated prostate cancer detection system. J Pathol 254, 147\u0026ndash;158 (2021). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/path.5662\u003c/span\u003e\u003cspan address=\"10.1002/path.5662\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThirunavukarasu, A. J. \u003cem\u003eet al.\u003c/em\u003e Large language models in medicine. Nat Med 29, 1930\u0026ndash;1940 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-023-02448-8\u003c/span\u003e\u003cspan address=\"10.1038/s41591-023-02448-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBhinder, B., Gilvary, C., Madhukar, N. S. \u0026amp; Elemento, O. Artificial Intelligence in Cancer Research and Precision Medicine. Cancer Discov 11, 900\u0026ndash;915 (2021). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1158/2159-8290.CD-21-0090\u003c/span\u003e\u003cspan address=\"10.1158/2159-8290.CD-21-0090\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKaplan, J. \u003cem\u003eet al.\u003c/em\u003e Scaling laws for neural language models. \u003cem\u003earXiv preprint arXiv\u003c/em\u003e:2001.\u003cem\u003e08361\u003c/em\u003e (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, Y. \u003cem\u003eet al.\u003c/em\u003e Enhancing semantic segmentation in chest X-ray images through image preprocessing: ps-KDE for pixel-wise substitution by kernel density estimation. PLoS One 19, e0299623 (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1371/journal.pone.0299623\u003c/span\u003e\u003cspan address=\"10.1371/journal.pone.0299623\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHe, X., Zhang, Y., Mou, L., Xing, E. \u0026amp; Xie, P. Pathvqa: 30000\u0026thinsp;+\u0026thinsp;questions for medical visual question answering. \u003cem\u003earXiv preprint arXiv\u003c/em\u003e:2003.\u003cem\u003e10286\u003c/em\u003e (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCrane, G. M. \u0026amp; Gardner, J. M. Pathology Image-Sharing on Social Media: Recommendations for Protecting Privacy While Motivating Education. AMA J Ethics 18, 817\u0026ndash;825 (2016). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1001/journalofethics.2016.18.8.stas1-1608\u003c/span\u003e\u003cspan address=\"10.1001/journalofethics.2016.18.8.stas1-1608\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCohen, D. \u003cem\u003eet al.\u003c/em\u003e #InSituPathologists: how the #USCAP2015 meeting went viral on Twitter and founded the social media movement for the United States and Canadian Academy of Pathology. Mod Pathol 30, 160\u0026ndash;168 (2017). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/modpathol.2016.223\u003c/span\u003e\u003cspan address=\"10.1038/modpathol.2016.223\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAllen, T. C. Social media: pathologists' force multiplier. Arch Pathol Lab Med 138, 1000\u0026ndash;1001 (2014). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.5858/arpa.2014-0071-ED\u003c/span\u003e\u003cspan address=\"10.5858/arpa.2014-0071-ED\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMisialek, M. J. \u0026amp; Allen, T. C. You're On Social Media! So Now What? Arch Pathol Lab Med 140, 393 (2016). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.5858/arpa.2015-0493-ED\u003c/span\u003e\u003cspan address=\"10.5858/arpa.2015-0493-ED\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKramer, B. S., Berg, C. D., Aberle, D. R. \u0026amp; Prorok, P. C. Vol. 18 109\u0026ndash;111 (SAGE Publications Sage UK: London, England, 2011).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTomczak, K., Czerwińska, P. \u0026amp; Wiznerowicz, M. Review The Cancer Genome Atlas (TCGA): an immeasurable source of knowledge. \u003cem\u003eContemporary Oncology/Wsp\u0026oacute;łczesna Onkologia\u003c/em\u003e 2015, 68\u0026ndash;77 (2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMolavi, D. W. \u003cem\u003eThe practice of surgical pathology: a beginner's guide to the diagnostic process\u003c/em\u003e. (Springer, 2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFoundation, C. \u003cem\u003eAn open-source, web-based viewer for high-resolution zoomable images, implemented in pure JavaScript, for desktop and mobile\u003c/em\u003e, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e\u0026lt;https://openseadragon.github.io/\u0026gt;\u003c/span\u003e\u003cspan address=\"http://%3Chttps://openseadragon.github.io/%3E\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2009).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbid, A. \u003cem\u003eet al.\u003c/em\u003e Gradio: Hassle-free sharing and testing of ml models in the wild. \u003cem\u003earXiv preprint arXiv\u003c/em\u003e:1906.\u003cem\u003e02569\u003c/em\u003e (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCaffagni, D. \u003cem\u003eet al.\u003c/em\u003e The (R) Evolution of Multimodal Large Language Models: A Survey. \u003cem\u003earXiv preprint arXiv:2402.12451\u003c/em\u003e (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu, H., Li, C., Wu, Q. \u0026amp; Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems 36 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu, H., Li, C., Li, Y. \u0026amp; Lee, Y. J. in \u003cem\u003eProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.\u003c/em\u003e 26296\u0026ndash;26306.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eInan, H. \u003cem\u003eet al.\u003c/em\u003e Llama guard: Llm-based input-output safeguard for human-ai conversations. \u003cem\u003earXiv preprint arXiv:2312.06674\u003c/em\u003e (2023).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-5404747/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5404747/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eDigital pathology has seen significant advancements in artificial intelligence (AI) applications. However, challenges persist in integrating these solutions into digital pathology platforms for human and AI collaborations. We introduce I-Viewer, an online AI Copilot framework designed to facilitate real-time human-AI and human-human collaboration for digital pathology analysis. The I-Viewer platform enables precise annotations and descriptions from tissue to the nuclei level through an Agentic-Retrieval Augmented Generation (RAG) system. By leveraging agents' outputs as reference points, aggregating information through the RAG system, and incorporating Large Language Models (LLM) for human feedback and refinement, I-Viewer sets a new standard for collaborative and accurate digital pathology analysis.\u003c/p\u003e \u003cp\u003eWe demonstrate I-Viewer's effectiveness on different pathology tasks using three datasets across different types of cancers, including non-small cell lung cancer, breast cancer, and colorectal cancer. The results show that I-Viewer achieves significant improvements in annotation speed and accuracy for pathology tasks, such as detecting cell morphology, cellular structures, and tumor growth patterns, outperforming current individual foundation models. Through its advanced AI agents, collaborative features, and LLM integrations, I-Viewer optimizes diagnostic workflows in clinical care and biomedical research.\u003c/p\u003e","manuscriptTitle":"I-Viewer: An Online Digital Pathology Analysis Platform with Agentic-RAG AI Copilot","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-11-26 10:59:26","doi":"10.21203/rs.3.rs-5404747/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"12f04e64-ce3c-40fe-9d33-6b20291bd47e","owner":[],"postedDate":"November 26th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":40042783,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"},{"id":40042784,"name":"Biological sciences/Computational biology and bioinformatics/Software"},{"id":40042785,"name":"Biological sciences/Computational biology and bioinformatics/Image processing"}],"tags":[],"updatedAt":"2025-02-19T05:38:15+00:00","versionOfRecord":[],"versionCreatedAt":"2024-11-26 10:59:26","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5404747","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5404747","identity":"rs-5404747","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}