Deep Learning Segmentation of Non-perfusion Area from Color Fundus Images and AI-generated Fluorescein Angiography

Deep Learning Segmentation of Non-perfusion Area from Color Fundus Images and AI-generated Fluorescein Angiography | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Deep Learning Segmentation of Non-perfusion Area from Color Fundus Images and AI-generated Fluorescein Angiography Kanato Masayoshi, Yusaku Katada, Nobuhiro Ozawa, Mari Ibuki, Kazuno Negishi, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3871406/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 11 May, 2024 Read the published version in Scientific Reports → Version 1 posted 5 You are reading this latest preprint version Abstract The non-perfusion area (NPA) of the retina is an important indicator in the visual prognosis of patients with retinal vein occlusion (RVO). However, the current evaluation method of NPA, fluorescein angiography (FA), is invasive and burdensome. In this study, we examined the use of deep learning models for detecting NPA in color fundus images, bypassing the need for FA, and we also investigated the utility of synthetic FA generated from color fundus images. The models were evaluated using the Dice score and Monte Carlo dropout uncertainty. We retrospectively collected 403 sets of color fundus and FA images from 319 RVO patients. We trained three deep learning models on FA, color fundus images, and synthetic FA. As a result, though the FA model achieved the highest score, the other two models also performed comparably. We found no statistical significance in median Dice scores between the models. However, the color fundus model showed significantly higher uncertainty than the other models (p < 0.05). In conclusion, deep learning models can detect NPAs from color fundus images with reasonable accuracy, though with somewhat less prediction stability. Synthetic FA stabilizes the prediction and reduces misleading uncertainty estimates by enhancing image quality. Health sciences/Diseases/Eye diseases/Retinal diseases Health sciences/Medical research/Translational research Biological sciences/Computational biology and bioinformatics/Image processing Biological sciences/Computational biology and bioinformatics/Machine learning Deep Learning Generative Artificial Intelligence Retinal Vein Occlusion Fluorescein Angiography Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. Introduction Retinal vein occlusion (RVO) is a vision-threatening disease caused by blocked retinal veins. In the assessment of RVO, non-perfusion areas (NPA) on the retina are a key indicator for prognosis and treatment. To evaluate NPAs, a normal color fundus image is insufficient. Instead, ophthalmologists primarily rely on fluorescein angiography (FA), which requires a contrast agent (fluorescein) 1 . However, the intravenous infusion of fluorescein is invasive and requires significant time and human resources 2 . While optical coherence tomography angiography (OCTA), a novel imaging method for the retina, can be a noninvasive alternative to FA, it requires an expensive device and therefore has limited accessibility. 3–5 . To offer a safer and more affordable diagnostic method of RVO, two AI approaches have been previously proposed: ( 1 ) segmentation AI that can predict NPA from only color fundus 6–9 and ( 2 ) generative adversarial network (GAN) models that can translate color fundus images into synthetic FA-like images 10–14 . These approaches have the potential to allow RVO patients to avoid costly or invasive examinations. Nonetheless, there were knowledge gaps that needed to be filled. First, there was an insufficient comparison between AI models using the gold standard method (FA) and ones using color fundus images. Although it is reported that AI could predict NPAs using only color fundus images, that is insufficient to determine whether color fundus images can replace FA in RVO diagnosis because FA models might perform better enough to accept the cost and potential adverse effects of FA. Second, the diagnostic utility of synthetic FA was unclear. Though the GAN model researchers have shown potential benefits such as enhancing the retinal vessels that are hardly visible in color fundus images, the clinical benefits of synthetic FA should be clearly demonstrated. To address these knowledge gaps, we quantitatively compared three deep learning models, each trained on different types of images (Fig. 1 ). FA model was trained on FA images, which is expected to perform the best as it uses the gold standard modality. The color fundus model was trained on color fundus images. The color fundus + synthetic FA model was trained on color fundus images and synthetic FA images generated from color fundus images. The last two models do not require real FA images hence less invasive and costly, but the performance might deteriorate compared to the FA model. Through these experiments, the present study aimed to address the following questions: ( 1 ) Can deep learning models reliably detect NPAs using only color fundus images with the same accuracy as models using FA? ( 2 ) Can synthetic FA provide additional value over color fundus images in NPA prediction? 2. Results Dataset Table 1 Demographic characteristics of the dataset (mean ± 95% confidence interval (CI)) The NPA ratio is the ratio of NPA pixels to all fundus pixels in the image. Demographics (n = 403) Age 65.7 ± 0.6 20–50 44 (10.9%) 50–80 313 (77.7%) 80–100 46 (11.4%) Systolic blood pressure (mmHg) 130.1 ± 1.0 Diastolic blood pressure (mmHg) 76.6 ± 0.7 Sex Male 213 (52.7%) Female 191 (49.1%) NPA ratio (%) 34 (No NPA) 80 (19.9%) 0–30% 124 (30.8%) 30–80% 186 (46.2%) 80–100% 13 (3.2%) Ethnicity Japanese 403 (100%) Table 2 Inter-annotator agreement measured by Dice score (%) Median Dice scores are shown with the interquartile ranges (IQR) in the brackets. Dice score (%) A B C Ground truth A 100 87.7 [52.6, 97.4] 94.9 [77.5, 99.7] 99.5 [97.8, 100] B - 100 96.4 [82.8, 99.9] 88.9 [56.7, 97.9] C - - 100 97.9 [87.3, 100] We retrospectively collected 403 pairs of color fundus and FA images from 319 RVO patients at Keio University Hospital, Tokyo, Japan. Table 1 shows the demographic characteristics of the dataset. Three ophthalmologists created NPA annotation (Fig. 2 ) and the inter-annotator agreement is shown in Table 2 . Synthetic FA Generation Table 3 Similarity metrics of the synthetic FA and color fundus to the FA Structural Similarity Index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) between the true FA images and synthetic FA or grayscale color fundus images are shown. The higher SSIM and lower LPIPS indicate more similarity. SSIM LPIPS Synthetic FA 0.507 0.091 Grayscale 0.526 0.087 The similarity metrics of the synthetic FA and color fundus are shown in Table 3 . The similarity between synthetic FA and real FA was nearly identical to that between grayscale color fundus images and real FA. This is unsurprising as most structures in FA are visible and similar in color fundus. The difference between the two modalities (FA and color fundus) will be important in NPA assessment, but such minor differences do not affect the image similarity metrics. Segmentation The FA model achieved the best accuracy with a median Dice score of 0.820; however, the color fundus model also demonstrated comparable performance (Fig. 3 A, 3 B). The color fundus + synthetic FA model performed slightly better than the color fundus model but did not outperform the FA model. While not statistically significant, the confidence intervals in Fig. 3 C suggest that the FA model likely performed the best, followed by the color fundus + synthetic FA model, and lastly the color fundus model. The FA model yielded more stable predictions than other models. Although the median Dice score and sensitivity were similar to other models, their interquartile ranges (IQR) were narrower. Additionally, the Dice score was greater than 0.6 for all samples except for two, and the sensitivity was greater than 0.5 for all samples. In contrast, the other models, namely color fundus and color fundus + synthetic FA models, exhibited wider IQRs, lower minimum Dice scores, and lower sensitivities. Uncertainty Estimation The Monte Carlo dropout uncertainty was significantly higher in the color fundus model than in the other two models (Fig. 4 ). The difference was statistically significant for the median standard deviation (SD) and the proportion of the area with SD > 0.1 in non-NPA pixels. Although the FA model demonstrated lower uncertainty compared to the color fundus + synthetic FA model, the difference was not statistically significant. Analysis of Individual Samples Examination of individual samples revealed that the influence of GAN-generated FA on segmentation accuracy varied across the dataset. Specifically, the addition of synthetic FA either improved, worsened, or had no impact on the Dice score, depending on the samples. Figure 5 showcases three representative cases. Synthetic FA lightened the shadowing and enhanced image clarity (highlighted in the yellow circle). As a result, the model using synthetic FA displayed fewer uncertain areas in non-NPA regions (orange circle). This observation aligns with the earlier noted reduction in areas with SD > 0.1 in non-NPA regions by the color fundus + synthetic FA model. However, synthetic FA had the downside of obscuring abnormalities such as hemorrhages, leading to inaccurate predictions (blue circle). 3. Discussion In this research, we compared the FA model and two non-FA models in terms of accuracy and uncertainty. We also examined the clinical utility of synthetic FA generated from color fundus images. As a result, the FA model achieved the best accuracy, while the other two models also attained comparable accuracy. As for uncertainty, the FA model yielded the most stable prediction, while the color fundus model showed the highest Monte Carlo uncertainty. Despite the comparable accuracy, the prediction from the non-FA models was unstable compared to the FA model. This is likely because some color fundus images lacked visible abnormalities such as hemorrhages. In other words, the accuracy of NPA prediction using color fundus was inconsistent; when lesions are visible in both color and FA images, the color fundus model can perform comparably to the FA model; however, when lesions are only visible in FA, the color fundus model performs worse than FA. Furthermore, the volatile imaging quality of the color fundus images might also have destabilized the color fundus model. Compared to FA, color fundus images are more vulnerable to image artifacts such as the angle of the camera and the direction of the lighting (e.g., shadows). These artifacts can lower the Dice score by deteriorating the image quality. The impact of synthetic FA on accuracy was mixed; it led to both increases and decreases in Dice scores, depending on the samples. Meanwhile, synthetic FA consistently reduced Monte Carlo uncertainty, likely due to GAN’s capability of image enhancement. As previously mentioned, some color fundus images suffer from quality issues, leading to greater uncertainty. However, GANs can improve image quality. In fact, GANs are commonly used in image enhancement tasks such as noise reduction and super-resolution 15–19 . In this study, our GAN model presumably acquired image enhancement capability because the target images (FA) had better image quality than the source images (color fundus). By using synthetic FA, we could reduce the misleading uncertainty estimates. This improvement is clinically beneficial. Uncertainty can help clinicians identify areas requiring further examination for abnormalities. However, "false alarms" of uncertainty estimates can mislead clinicians into investigating completely normal areas, thereby wasting their time. By using synthetic FA for NPA prediction, such false alarms can be decreased, and therefore the uncertainty estimates will be more reliable and helpful. However, inappropriate “improvement” in image quality could also be harmful to segmentation accuracy. Most generative AI has a problem called hallucination, in which AI generates false information with certainty 20,21 . In this study, although the GAN model was trained on images that contain NPAs, most parts were normal and therefore the GAN model might have tended to generate normal images. That would have obscured abnormalities, negatively affecting Dice scores and sensitivity. Although our results suggest the limited effect of synthetic data on accuracy, some studies have reported that GAN-generated images could improve prediction performance in medical image tasks such as contrast-enhanced CT synthesis and other medical fields such as pathology 22,23 . Collectively, it is suggested that the effectiveness of GANs is task-dependent. In general, GANs cannot acquire additional information from patients; they can only refine existing features within the existing images. There should be a substantial limitation in that it cannot detect what is not there. Therefore, theoretically, using GAN-generated images for downstream tasks would only be beneficial when downstream models fail to extract features effectively. Our analysis was limited due to the small size of the dataset, which could have affected the accuracy of the segmentation model and GAN model. Therefore, more extensive research is needed to determine the utility of synthetic data in medical image AI. Future work should aim to develop a more stable NPA segmentation model that performs well even when abnormalities are subtle or not readily apparent on color fundus images. In conclusion, the deep learning models can predict NPAs solely from color fundus images with acceptable accuracy. This result is prospective towards the aim of providing RVO patients with safe and accessible examinations. However, at this point, NPA prediction relying solely on color fundus images can lead to missed lesions, given its instability. Further research is needed to overcome this challenge. The unstable performance can be attributed to two factors. First, the color fundus model performs comparably only when there are visible lesions of NPA in the color fundus images. When an input image completely lacks indicative features, the model performance deteriorates. Second, the quality of color fundus images is more likely to be impaired than FA due to image artifacts such as shadowing. The primary contribution of the GAN-generated FA is its image enhancement effect such as noise reduction and brightness adjustment. Although the improvement in accuracy is subtle, GAN-generated FA lowers “false alarm” in Monte Carlo dropout uncertainty estimates and thereby enhances their clinical utility as an indicator for requiring doctors’ further inspection of a specific part in a fundus image. 4. Materials and Methods Ethical statement The study was conducted in accordance with relevant guidelines and regulations including the Declaration of Helsinki and was approved by the institutional review board of the Keio University School of Medicine (approval no. 20170049). Due to the retrospective observational nature of this study, the informed consent was obtained through an opt-out approach from all participants. Identifying information was anonymized prior to analysis. The study involved no interventions in humans or animals. Dataset We retrospectively collected 403 sets of color fundus and FA images from 319 RVO patients at the Keio University School of Medicine. The annotations for the NPAs were performed by three licensed ophthalmologists, using both color fundus and FA images for reference. They also aligned the color and FA images by an affine transformation. The low-quality samples, on which doctors could not make a diagnosis, were excluded from the dataset. The regions blocked with bleeding were labeled positive when NPA was supposed to exist underneath. After the annotations were completed by three independent ophthalmologists, we generated ground truth using the union set of the NPA maps by the three annotators. The dataset was then divided into training (330 images), validation (38 images), and test (35 images) subsets. Each subset had specific roles in segmentation and synthetic FA generation. In the segmentation task, the roles of each dataset were straightforward: the training set for model training, the validation set for monitoring generalization performance, and the test set for final evaluation. For the training of the synthetic FA generation model, we used the validation set to minimize potential data leakage between the segmentation and generation models, ensuring an unbiased evaluation. Using the same dataset for both segmentation and generation models could lead to data leakage, and the segmentation model would be able to exploit this leakage. The segmentation model using synthetic FA would have indirect access to FA during training, which however would not occur in test or real-world use. FA Synthesis To generate synthetic FA from the color fundus, we utilized generative adversarial networks (GANs) 24,25 . This technique is widely used in image generation across various fields, including medicine 26–29 . Specifically, we used Fundus2Angio architecture, which was designed for color-to-FA translation 10 . The quality of the generated synthetic FA images was measured by two metrics: Structural Similarity Index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) 30,31 . Both measures are generally used to quantify the similarity between two images. SSIM focuses on the luminance, contrast, and structure of two images, while LPIPS leverages deep learning to capture complex visual similarities, better aligning with human perception. We also calculated these metrics for the grayscale images of the color fundus for comparison. Segmentation For the segmentation of NPAs, we used U-Net with Monte Carlo dropout 32,33 . U-Net is a widely used medical image segmentation model, and Monte Carlo dropout is an uncertainty estimation method for deep learning models. By combining these methods, we can analyze both the prediction accuracy and uncertainty estimates. We opted for U-Net due to its straightforward architecture. Our previous research indicated that a deep learning model with simple architecture produces more informative uncertainty estimation 34 . The segmentation model was trained with different input types: (1) FA model, (2) color fundus model, and (3) color + synthesized FA model. The models were trained with a batch size of 4, the Adam optimizer, an initial learning rate of 0.0004, and the cross-entropy loss function. Evaluation Using the test subset, we evaluated the accuracy of the segmentation models by the Dice coefficient score and sensitivity. Sensitivity is the ratio of true-positive pixels to the total NPA pixels in an image. Since we cannot assume the Gaussian distribution on the Dice score, the Wilcoxon signed-rank test was used to test the difference in prediction performance. The confidence intervals for differences in Dice scores were calculated using the bootstrap method. Additionally, we calculated the standard deviation (SD) from 100 Monte Carlo dropout predictions for uncertainty estimation. To compare the nature of each model in terms of uncertainty, the median SD and the area fraction with SD > 0.1 were quantified and compared using the Wilcoxon signed-rank test. Furthermore, we performed individual-level comparisons to assess the impact of the absence of FA or the inclusion of synthesized FA on a case-by-case basis. For multiple testing correction, the Bonferroni method was applied. Abbreviations artificial intelligence (AI) retinal vein occlusion (RVO) non perfusion area (NPA) fluorescein angiography (FA) generative adversarial network (GAN) Monte Carlo (MC) structural similarity index (SSIM) learned perceptual image patch similarity (LPIPS). Declarations Acknowledgments The authors thank the members of the Laboratory of Photobiology, Keio University School of Medicine for their technical and administrative support. Especially, we would like to express our gratitude to Kae Otsuka for their assistance with data collection. Author Contributions K.M., Y.K., and T.K. conceptualized the research and wrote the main manuscript. Y.K., N.O., and M.I. curated the dataset. K.M. wrote the code and analyzed the data. K.N. and T.K. supervised the research. All authors reviewed the manuscript. Competing interests The authors TK, YK, and KM are inventors on patents and patent applications related to this work (JP7143862, EP19741888, CN201980018367, US16/930510, and JP2022147056). Data Availability The dataset is not publicly available due to their containing information that could compromise the privacy of research participants but may be available from the corresponding author upon reasonable request. References The Royal College of Ophthalmologists. Clinical Guidelines Retinal Vein Occlusion (RVO) . (The Royal College of Ophthalmologists, 2022). Kornblau, I. S. & El-Annan, J. F. Adverse reactions to fluorescein angiography: A comprehensive review of the literature. Surv. Ophthalmol. 64, 679–693 (2019). Nobre Cardoso, J. et al. Systematic Evaluation of Optical Coherence Tomography Angiography in Retinal Vein Occlusion. Am. J. Ophthalmol. 163, 93–107.e6 (2016). Nagasato, D. et al. Automated detection of a nonperfusion area caused by retinal vein occlusion in optical coherence tomography angiography images using deep learning. PLoS One 14, (2019). Hirano, Y. et al. Multimodal Imaging of Microvascular Abnormalities in Retinal Vein Occlusion. J. Clin. Med. Res. 10, (2021). Inoda, S. et al. Deep-learning-based AI for evaluating estimated nonperfusion areas requiring further examination in ultra-widefield fundus images. Sci. Rep. 12, 21826 (2022). Ren, X. et al. Artificial intelligence to distinguish retinal vein occlusion patients using color fundus photographs. Eye 37, 2026–2032 (2023). Miao, J. et al. Deep Learning Models for Segmenting Non-perfusion Area of Color Fundus Photographs in Patients With Branch Retinal Vein Occlusion. Front. Med. 9, 794045 (2022). Nunez do Rio, J. M. et al. Deep Learning-Based Segmentation and Quantification of Retinal Capillary Non-Perfusion on Ultra-Wide-Field Retinal Fluorescein Angiography. J. Clin. Med. Res. (2020) doi: 10.3390/jcm9082537 . Kamran, S. A. et al. Fundus2Angio: A Conditional GAN Architecture for Generating Fluorescein Angiography Images from Retinal Fundus Photography. in Advances in Visual Computing 125–138 (Springer International Publishing, 2020). Huang, K. et al. Lesion-aware generative adversarial networks for color fundus image to fundus fluorescein angiography translation. Comput. Methods Programs Biomed. 229, 107306 (2023). Pham, Q. T. M., Ahn, S., Shin, J. & Song, S. J. Generating future fundus images for early age-related macular degeneration based on generative adversarial networks. Comput. Methods Programs Biomed. 216, 106648 (2022). Kamran, S. A. et al. RV-GAN: Segmenting Retinal Vascular Structure in Fundus Photographs Using a Novel Multi-scale Generative Adversarial Network. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 34–44 (Springer International Publishing, 2021). Tavakkoli, A., Kamran, S. A., Hossain, K. F. & Zuckerbrod, S. L. A novel deep learning conditional generative adversarial network for producing angiography images from retinal fundus photographs. Sci. Rep. 10, 21580 (2020). Gupta, R., Sharma, A. & Kumar, A. Super-Resolution using GANs for Medical Imaging. Procedia Comput. Sci. 173, 28–35 (2020). Zhang, L., Dai, H. & Sang, Y. Med-SRNet: GAN-Based Medical Image Super-Resolution via High-Resolution Representation Learning. Comput. Intell. Neurosci. 2022, 1744969 (2022). Ahmad, W., Ali, H., Shah, Z. & Azmat, S. A new generative adversarial network for medical images super resolution. Sci. Rep. 12, 9533 (2022). Yang, Q. et al. Low-Dose CT Image Denoising Using a Generative Adversarial Network With Wasserstein Distance and Perceptual Loss. IEEE Trans. Med. Imaging 37, 1348–1357 (2018). Deng, Z. et al. RFormer: Transformer-Based Generative Adversarial Network for Real Fundus Image Restoration on a New Clinical Benchmark. IEEE J Biomed Health Inform 26, 4645–4655 (2022). Denck, J., Guehring, J., Maier, A. & Rothgang, E. MR-contrast-aware image-to-image translations with generative adversarial networks. Int. J. Comput. Assist. Radiol. Surg. 16, 2069–2078 (2021). Cohen, J. P., Luck, M. & Honari, S. Distribution Matching Losses Can Hallucinate Features in Medical Image Translation. arXiv [cs.CV] (2018). Teramoto, A. et al. Deep learning approach to classification of lung cytological images: Two-step training using actual and synthesized images by progressive growing of generative adversarial networks. PLoS One 15, e0229951 (2020). Levine, A. B. et al. Synthesis of diagnostic quality cancer pathology images by generative adversarial networks. J. Pathol. 252, 178–188 (2020). Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-Image Translation with Conditional Adversarial Networks. arXiv [cs.CV] (2016). Goodfellow, I. et al. Generative Adversarial Nets. in Advances in Neural Information Processing Systems (eds. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. & Weinberger, K. Q.) vol. 27 (Curran Associates, Inc., 2014). Fujioka, T. et al. Breast Ultrasound Image Synthesis using Deep Convolutional Generative Adversarial Networks. Diagnostics (Basel) 9, (2019). Koshino, K. et al. Narrative review of generative adversarial networks in medical and molecular imaging. Ann Transl Med 9, 821 (2021). Jeong, J. J. et al. Systematic Review of Generative Adversarial Networks (GANs) for Medical Image Classification and Segmentation. J. Digit. Imaging 35, 137–152 (2022). Skandarani, Y., Jodoin, P.-M. & Lalande, A. GANs for Medical Image Synthesis: An Empirical Study. J. Imaging Sci. Technol. 9, (2023). Zhang, R., Isola, P., Efros, A. A., Shechtman, E. & Wang, O. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. arXiv [cs.CV] (2018). Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004). Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 vol. 9351 234–241 (Springer Verlag, 2015). Gal, Y. & Ghahramani, Z. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. in Proceedings of The 33rd International Conference on Machine Learning vol. 48 1050–1059 (International Machine Learning Society (IMLS), 2016). Masayoshi, K. et al. Automatic segmentation of non-perfusion area from fluorescein angiography using deep learning with uncertainty estimation. Informatics in Medicine Unlocked 32, 101060 (2022). Additional Declarations Competing interest reported. The authors T.K., Y.K., and K.M. are inventors on patents and patent applications related to this work (JP7143862, EP19741888, CN201980018367, US16/930510, and JP2022147056). Cite Share Download PDF Status: Published Journal Publication published 11 May, 2024 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 28 Feb, 2024 Editor assigned by journal 29 Jan, 2024 Editor invited by journal 25 Jan, 2024 Submission checks completed at journal 25 Jan, 2024 First submitted to journal 16 Jan, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3871406","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":269829007,"identity":"bae81058-35b4-41d9-af9f-b120b4983ec2","order_by":0,"name":"Kanato Masayoshi","email":"","orcid":"","institution":"Laboratory of Photobiology, Keio University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Kanato","middleName":"","lastName":"Masayoshi","suffix":""},{"id":269829008,"identity":"3784c98a-32fb-49ef-a92d-311ee6737e37","order_by":1,"name":"Yusaku Katada","email":"","orcid":"","institution":"Laboratory of Photobiology, Keio University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Yusaku","middleName":"","lastName":"Katada","suffix":""},{"id":269829010,"identity":"04244806-d929-4f12-b97e-16303415ec68","order_by":2,"name":"Nobuhiro Ozawa","email":"","orcid":"","institution":"Laboratory of Photobiology, Keio University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Nobuhiro","middleName":"","lastName":"Ozawa","suffix":""},{"id":269829012,"identity":"355f17bb-a6f7-4eb6-8fe7-3b7b68e55900","order_by":3,"name":"Mari Ibuki","email":"","orcid":"","institution":"Laboratory of Photobiology, Keio University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Mari","middleName":"","lastName":"Ibuki","suffix":""},{"id":269829014,"identity":"b3f810ca-33ab-4c8c-a080-45064e4a53d9","order_by":4,"name":"Kazuno Negishi","email":"","orcid":"","institution":"Department of Ophthalmology, Keio University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Kazuno","middleName":"","lastName":"Negishi","suffix":""},{"id":269829016,"identity":"983f74f7-d9d1-4ffa-8a62-aa6b9449ddf3","order_by":5,"name":"Toshihide Kurihara","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA20lEQVRIiWNgGAWjYBACCSA+DGLwMzAfYGwA8w8QqUWygS2BeC3MIIbBAR4DoBYigOSM3IeHC2ruyRufP5bAOKPCQo6B8Sx+a6Ql0g0OzzhWbLjtwOEDjBvOSBgzMJxLwKtFTiKN4TAP0BvbDrYlMD5sk0hsYDhjQISWfwn2m5uBfnn4jwgt0iAtvG0JiRvYgFo2NhChRbLnGVBLX0LyjDNsCQdnHJMwZiPkF4njacyfeb4l2Pb3Hz74sKemTo5fgkCIoQCwUjaJM8TrgAL+HpK1jIJRMApGwfAGAAkVR/qcREc+AAAAAElFTkSuQmCC","orcid":"","institution":"Laboratory of Photobiology, Keio University School of Medicine","correspondingAuthor":true,"prefix":"","firstName":"Toshihide","middleName":"","lastName":"Kurihara","suffix":""}],"badges":[],"createdAt":"2024-01-17 01:14:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3871406/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3871406/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-024-61561-x","type":"published","date":"2024-05-11T21:18:08+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":50383913,"identity":"f6882eda-2a5d-40a2-84b6-10d3a7703508","added_by":"auto","created_at":"2024-01-30 17:27:09","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":190329,"visible":true,"origin":"","legend":"\u003cp\u003eModel training and research questions\u003c/p\u003e\n\u003cp\u003eThe figure shows the abstract of the present research. We trained three deep learning models on different input sources. By comparing these models, we answered two research questions regarding the utility of color fundus and synthetic FA images.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-3871406/v1/37d23e0f6b9b5b6954c55e1c.png"},{"id":50382412,"identity":"87ee8063-d761-4e85-8be9-971fb937e5e8","added_by":"auto","created_at":"2024-01-30 17:19:09","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":95105,"visible":true,"origin":"","legend":"\u003cp\u003eExample of preprocessed and annotated images\u003c/p\u003e\n\u003cp\u003eThree licensed ophthalmologists aligned the color fundus and FA images, and then they independently annotated the NPA. We defined the ground truth as the union set of the three annotations. Generated FA images are not shown here since they were generated from the color fundus images and were not raw data.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-3871406/v1/930767469d5df88baf38ef76.png"},{"id":50382417,"identity":"c1512609-378d-4ba6-a0a6-b9e6b2ac295f","added_by":"auto","created_at":"2024-01-30 17:19:09","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":147194,"visible":true,"origin":"","legend":"\u003cp\u003eAccuracy of NPA prediction with different input sources\u003c/p\u003e\n\u003cp\u003eThe red line indicates median, and the whiskers show 1.5 times IQR. \u003cstrong\u003e(A, B)\u003c/strong\u003e Dice score and sensitivity of models with different input. \u003cstrong\u003e(C)\u003c/strong\u003e The bootstrap 95% confidence intervals of the sample-wise gap in Dice score between the two models. No statistical significance was observed in any pairs.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-3871406/v1/cb1fb9ecdb82363138cd534e.png"},{"id":50382413,"identity":"0ba7ebf5-ec62-472d-95e2-fca3c806d095","added_by":"auto","created_at":"2024-01-30 17:19:09","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":63866,"visible":true,"origin":"","legend":"\u003cp\u003eMonte Carlo dropout uncertainty\u003c/p\u003e\n\u003cp\u003eStandard deviation (SD) was acquired from 100 predictions using Monte Carlo dropout. \u003cstrong\u003e(A)\u003c/strong\u003e Median SD in each image, measuring the degree of uncertainty in general. \u0026nbsp;\u003cstrong\u003e(B)\u003c/strong\u003e Ratio of pixels with SD \u0026gt; 0.1 to non-NPA pixels, representing the uncertainty outside true NPAs. For both measures, the uncertainty in the color fundus model was larger than the others with statistical significance.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-3871406/v1/b7080b4b21a328fac1e87e8d.png"},{"id":50382416,"identity":"e2bffae8-2fa2-4970-9af3-5e947d986db6","added_by":"auto","created_at":"2024-01-30 17:19:09","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":1300691,"visible":true,"origin":"","legend":"\u003cp\u003eRepresentative samples of prediction and uncertainty\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(A)\u003c/strong\u003e Dice scores of three representative samples on different models. \u003cstrong\u003e(B)\u003c/strong\u003eInput and output of each sample. Samples (a)-(c) are correspondent in (A) and (B). \u003cstrong\u003e(a)\u003c/strong\u003e All models yielded accurate predictions. \u003cstrong\u003e(b)\u003c/strong\u003e The color fundus model lagged in accuracy compared to the FA model due to shadowing, which was, however, mitigated by synthetic FA. \u003cstrong\u003e(c)\u003c/strong\u003e The use of GAN reduced accuracy by obfuscating key details in the color fundus images. Meanwhile, the color fundus model had high-uncertainty area in non-NPA regions.\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-3871406/v1/a7e020c173c5a000ac8695b0.png"},{"id":56488222,"identity":"a42325fc-f7a3-4eba-84d7-f8d42aac2b67","added_by":"auto","created_at":"2024-05-14 21:30:01","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1791653,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3871406/v1/3a8f6342-4ed0-4ed7-9cca-b505a692e168.pdf"}],"financialInterests":"Competing interest reported. The authors T.K., Y.K., and K.M. are inventors on patents and patent applications related to this work (JP7143862, EP19741888, CN201980018367, US16/930510, and JP2022147056).","formattedTitle":"Deep Learning Segmentation of Non-perfusion Area from Color Fundus Images and AI-generated Fluorescein Angiography","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eRetinal vein occlusion (RVO) is a vision-threatening disease caused by blocked retinal veins. In the assessment of RVO, non-perfusion areas (NPA) on the retina are a key indicator for prognosis and treatment. To evaluate NPAs, a normal color fundus image is insufficient. Instead, ophthalmologists primarily rely on fluorescein angiography (FA), which requires a contrast agent (fluorescein) \u003csup\u003e1\u003c/sup\u003e. However, the intravenous infusion of fluorescein is invasive and requires significant time and human resources \u003csup\u003e2\u003c/sup\u003e. While optical coherence tomography angiography (OCTA), a novel imaging method for the retina, can be a noninvasive alternative to FA, it requires an expensive device and therefore has limited accessibility. \u003csup\u003e3\u0026ndash;5\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eTo offer a safer and more affordable diagnostic method of RVO, two AI approaches have been previously proposed: (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e) segmentation AI that can predict NPA from only color fundus \u003csup\u003e6\u0026ndash;9\u003c/sup\u003e and (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e) generative adversarial network (GAN) models that can translate color fundus images into synthetic FA-like images \u003csup\u003e10\u0026ndash;14\u003c/sup\u003e. These approaches have the potential to allow RVO patients to avoid costly or invasive examinations.\u003c/p\u003e \u003cp\u003eNonetheless, there were knowledge gaps that needed to be filled. First, there was an insufficient comparison between AI models using the gold standard method (FA) and ones using color fundus images. Although it is reported that AI could predict NPAs using only color fundus images, that is insufficient to determine whether color fundus images can replace FA in RVO diagnosis because FA models might perform better enough to accept the cost and potential adverse effects of FA. Second, the diagnostic utility of synthetic FA was unclear. Though the GAN model researchers have shown potential benefits such as enhancing the retinal vessels that are hardly visible in color fundus images, the clinical benefits of synthetic FA should be clearly demonstrated.\u003c/p\u003e \u003cp\u003eTo address these knowledge gaps, we quantitatively compared three deep learning models, each trained on different types of images (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). FA model was trained on FA images, which is expected to perform the best as it uses the gold standard modality. The color fundus model was trained on color fundus images. The color fundus\u0026thinsp;+\u0026thinsp;synthetic FA model was trained on color fundus images and synthetic FA images generated from color fundus images. The last two models do not require real FA images hence less invasive and costly, but the performance might deteriorate compared to the FA model. Through these experiments, the present study aimed to address the following questions:\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e(\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e) Can deep learning models reliably detect NPAs using only color fundus images with the same accuracy as models using FA?\u003c/p\u003e \u003cp\u003e(\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e) Can synthetic FA provide additional value over color fundus images in NPA prediction?\u003c/p\u003e"},{"header":"2. Results","content":"\u003cp\u003e\u003cstrong\u003eDataset\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n \u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\n \u003ctable id=\"Tab1\" border=\"1\"\u003e\n \u003ccaption\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003e\u003cstrong\u003eDemographic characteristics of the dataset (mean\u0026thinsp;\u0026plusmn;\u0026thinsp;95% confidence interval (CI))\u003c/strong\u003e The NPA ratio is the ratio of NPA pixels to all fundus pixels in the image.\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eDemographics (n\u0026thinsp;=\u0026thinsp;403)\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eAge\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e65.7\u0026thinsp;\u0026plusmn;\u0026thinsp;0.6\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e20\u0026ndash;50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e44 (10.9%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e50\u0026ndash;80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e313 (77.7%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e80\u0026ndash;100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e46 (11.4%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eSystolic blood pressure (mmHg)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e130.1\u0026thinsp;\u0026plusmn;\u0026thinsp;1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eDiastolic blood pressure (mmHg)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e76.6\u0026thinsp;\u0026plusmn;\u0026thinsp;0.7\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eSex\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eMale\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e213 (52.7%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFemale\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e191 (49.1%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eNPA ratio (%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e34\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e(No NPA)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e80 (19.9%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u0026ndash;30%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e124 (30.8%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e30\u0026ndash;80%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e186 (46.2%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e80\u0026ndash;100%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e13 (3.2%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eEthnicity\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eJapanese\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e403 (100%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cdiv class=\"gridtable\"\u003e\n \u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\n \u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\n \u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\n \u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\n \u003cdiv class=\"colspec\" align=\"char\"\u003e\u0026nbsp;\u003c/div\u003e\n \u003ctable id=\"Tab2\" border=\"1\"\u003e\n \u003ccaption\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003e\u003cstrong\u003eInter-annotator agreement measured by Dice score (%)\u003c/strong\u003e Median Dice scores are shown with the interquartile ranges (IQR) in the brackets.\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eDice score (%)\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eA\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eB\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eC\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eGround truth\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eA\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e87.7 [52.6, 97.4]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e94.9 [77.5, 99.7]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e99.5 [97.8, 100]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eB\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e96.4 [82.8, 99.9]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e88.9 [56.7, 97.9]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e97.9 [87.3, 100]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eWe retrospectively collected 403 pairs of color fundus and FA images from 319 RVO patients at Keio University Hospital, Tokyo, Japan. Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e shows the demographic characteristics of the dataset. Three ophthalmologists created NPA annotation (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e) and the inter-annotator agreement is shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSynthetic FA Generation\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n \u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\n \u003cdiv class=\"colspec\" align=\"char\"\u003e\u0026nbsp;\u003c/div\u003e\n \u003cdiv class=\"colspec\" align=\"char\"\u003e\u0026nbsp;\u003c/div\u003e\n \u003ctable id=\"Tab3\" border=\"1\"\u003e\n \u003ccaption\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003e\u003cstrong\u003eSimilarity metrics of the synthetic FA and color fundus to the FA\u003c/strong\u003e Structural Similarity Index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) between the true FA images and synthetic FA or grayscale color fundus images are shown. The higher SSIM and lower LPIPS indicate more similarity.\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSSIM\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eLPIPS\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eSynthetic FA\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.507\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.091\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eGrayscale\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.526\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.087\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eThe similarity metrics of the synthetic FA and color fundus are shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e. The similarity between synthetic FA and real FA was nearly identical to that between grayscale color fundus images and real FA. This is unsurprising as most structures in FA are visible and similar in color fundus. The difference between the two modalities (FA and color fundus) will be important in NPA assessment, but such minor differences do not affect the image similarity metrics.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSegmentation\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe FA model achieved the best accuracy with a median Dice score of 0.820; however, the color fundus model also demonstrated comparable performance (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003eA, \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003eB). The color fundus\u0026thinsp;+\u0026thinsp;synthetic FA model performed slightly better than the color fundus model but did not outperform the FA model. While not statistically significant, the confidence intervals in Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003eC suggest that the FA model likely performed the best, followed by the color fundus\u0026thinsp;+\u0026thinsp;synthetic FA model, and lastly the color fundus model.\u003c/p\u003e\n\u003cp\u003eThe FA model yielded more stable predictions than other models. Although the median Dice score and sensitivity were similar to other models, their interquartile ranges (IQR) were narrower. Additionally, the Dice score was greater than 0.6 for all samples except for two, and the sensitivity was greater than 0.5 for all samples. In contrast, the other models, namely color fundus and color fundus\u0026thinsp;+\u0026thinsp;synthetic FA models, exhibited wider IQRs, lower minimum Dice scores, and lower sensitivities.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUncertainty Estimation\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe Monte Carlo dropout uncertainty was significantly higher in the color fundus model than in the other two models (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e). The difference was statistically significant for the median standard deviation (SD) and the proportion of the area with SD\u0026thinsp;\u0026gt;\u0026thinsp;0.1 in non-NPA pixels. Although the FA model demonstrated lower uncertainty compared to the color fundus\u0026thinsp;+\u0026thinsp;synthetic FA model, the difference was not statistically significant.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAnalysis of Individual Samples\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eExamination of individual samples revealed that the influence of GAN-generated FA on segmentation accuracy varied across the dataset. Specifically, the addition of synthetic FA either improved, worsened, or had no impact on the Dice score, depending on the samples.\u003c/p\u003e\n\u003cp\u003eFigure \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e showcases three representative cases. Synthetic FA lightened the shadowing and enhanced image clarity (highlighted in the yellow circle). As a result, the model using synthetic FA displayed fewer uncertain areas in non-NPA regions (orange circle). This observation aligns with the earlier noted reduction in areas with SD\u0026thinsp;\u0026gt;\u0026thinsp;0.1 in non-NPA regions by the color fundus\u0026thinsp;+\u0026thinsp;synthetic FA model. However, synthetic FA had the downside of obscuring abnormalities such as hemorrhages, leading to inaccurate predictions (blue circle).\u003c/p\u003e"},{"header":"3. Discussion","content":"\u003cp\u003eIn this research, we compared the FA model and two non-FA models in terms of accuracy and uncertainty. We also examined the clinical utility of synthetic FA generated from color fundus images. As a result, the FA model achieved the best accuracy, while the other two models also attained comparable accuracy. As for uncertainty, the FA model yielded the most stable prediction, while the color fundus model showed the highest Monte Carlo uncertainty.\u003c/p\u003e \u003cp\u003eDespite the comparable accuracy, the prediction from the non-FA models was unstable compared to the FA model. This is likely because some color fundus images lacked visible abnormalities such as hemorrhages. In other words, the accuracy of NPA prediction using color fundus was inconsistent; when lesions are visible in both color and FA images, the color fundus model can perform comparably to the FA model; however, when lesions are only visible in FA, the color fundus model performs worse than FA. Furthermore, the volatile imaging quality of the color fundus images might also have destabilized the color fundus model. Compared to FA, color fundus images are more vulnerable to image artifacts such as the angle of the camera and the direction of the lighting (e.g., shadows). These artifacts can lower the Dice score by deteriorating the image quality.\u003c/p\u003e \u003cp\u003eThe impact of synthetic FA on accuracy was mixed; it led to both increases and decreases in Dice scores, depending on the samples. Meanwhile, synthetic FA consistently reduced Monte Carlo uncertainty, likely due to GAN\u0026rsquo;s capability of image enhancement. As previously mentioned, some color fundus images suffer from quality issues, leading to greater uncertainty. However, GANs can improve image quality. In fact, GANs are commonly used in image enhancement tasks such as noise reduction and super-resolution \u003csup\u003e15\u0026ndash;19\u003c/sup\u003e. In this study, our GAN model presumably acquired image enhancement capability because the target images (FA) had better image quality than the source images (color fundus).\u003c/p\u003e \u003cp\u003eBy using synthetic FA, we could reduce the misleading uncertainty estimates. This improvement is clinically beneficial. Uncertainty can help clinicians identify areas requiring further examination for abnormalities. However, \"false alarms\" of uncertainty estimates can mislead clinicians into investigating completely normal areas, thereby wasting their time. By using synthetic FA for NPA prediction, such false alarms can be decreased, and therefore the uncertainty estimates will be more reliable and helpful.\u003c/p\u003e \u003cp\u003eHowever, inappropriate \u0026ldquo;improvement\u0026rdquo; in image quality could also be harmful to segmentation accuracy. Most generative AI has a problem called hallucination, in which AI generates false information with certainty \u003csup\u003e20,21\u003c/sup\u003e. In this study, although the GAN model was trained on images that contain NPAs, most parts were normal and therefore the GAN model might have tended to generate normal images. That would have obscured abnormalities, negatively affecting Dice scores and sensitivity.\u003c/p\u003e \u003cp\u003eAlthough our results suggest the limited effect of synthetic data on accuracy, some studies have reported that GAN-generated images could improve prediction performance in medical image tasks such as contrast-enhanced CT synthesis and other medical fields such as pathology \u003csup\u003e22,23\u003c/sup\u003e. Collectively, it is suggested that the effectiveness of GANs is task-dependent. In general, GANs cannot acquire additional information from patients; they can only refine existing features within the existing images. There should be a substantial limitation in that it cannot detect what is not there. Therefore, theoretically, using GAN-generated images for downstream tasks would only be beneficial when downstream models fail to extract features effectively.\u003c/p\u003e \u003cp\u003eOur analysis was limited due to the small size of the dataset, which could have affected the accuracy of the segmentation model and GAN model. Therefore, more extensive research is needed to determine the utility of synthetic data in medical image AI. Future work should aim to develop a more stable NPA segmentation model that performs well even when abnormalities are subtle or not readily apparent on color fundus images.\u003c/p\u003e \u003cp\u003eIn conclusion, the deep learning models can predict NPAs solely from color fundus images with acceptable accuracy. This result is prospective towards the aim of providing RVO patients with safe and accessible examinations. However, at this point, NPA prediction relying solely on color fundus images can lead to missed lesions, given its instability. Further research is needed to overcome this challenge. The unstable performance can be attributed to two factors. First, the color fundus model performs comparably only when there are visible lesions of NPA in the color fundus images. When an input image completely lacks indicative features, the model performance deteriorates. Second, the quality of color fundus images is more likely to be impaired than FA due to image artifacts such as shadowing. The primary contribution of the GAN-generated FA is its image enhancement effect such as noise reduction and brightness adjustment. Although the improvement in accuracy is subtle, GAN-generated FA lowers \u0026ldquo;false alarm\u0026rdquo; in Monte Carlo dropout uncertainty estimates and thereby enhances their clinical utility as an indicator for requiring doctors\u0026rsquo; further inspection of a specific part in a fundus image.\u003c/p\u003e"},{"header":"4. Materials and Methods","content":"\u003ch2\u003eEthical statement\u003c/h2\u003e\n\u003cp\u003eThe study was conducted in accordance with relevant guidelines and regulations including the Declaration of Helsinki and was approved by the institutional review board of the Keio University School of Medicine (approval no. 20170049). Due to the retrospective observational nature of this study, the informed consent was obtained through an opt-out approach from all participants. Identifying information was anonymized prior to analysis. The study involved no interventions in humans or animals.\u003c/p\u003e\n\u003ch2\u003eDataset\u003c/h2\u003e\n\u003cp\u003eWe retrospectively collected 403 sets of color fundus and FA images from 319 RVO patients at the Keio University School of Medicine. The annotations for the NPAs were performed by three licensed ophthalmologists, using both color fundus and FA images for reference. They also aligned the color and FA images by an affine transformation. The low-quality samples, on which doctors could not make a diagnosis, were excluded from the dataset. The regions blocked with bleeding were labeled positive when NPA was supposed to exist underneath. After the annotations were completed by three independent ophthalmologists, we generated ground truth using the union set of the NPA maps by the three annotators.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe dataset was then divided into training (330 images), validation (38 images), and test (35 images) subsets. Each subset had specific roles in segmentation and synthetic FA generation. In the segmentation task, the roles of each dataset were straightforward: the training set for model training, the validation set for monitoring generalization performance, and the test set for final evaluation. For the training of the synthetic FA generation model, we used the validation set to minimize potential data leakage between the segmentation and generation models, ensuring an unbiased evaluation. Using the same dataset for both segmentation and generation models could lead to data leakage, and the segmentation model would be able to exploit this leakage. The segmentation model using synthetic FA would have indirect access to FA during training, which however would not occur in test or real-world use.\u003c/p\u003e\n\u003ch2\u003eFA Synthesis\u003c/h2\u003e\n\u003cp\u003eTo generate synthetic FA from the color fundus, we utilized generative adversarial networks (GANs)\u0026nbsp;\u003csup\u003e24,25\u003c/sup\u003e. This technique is widely used in image generation across various fields, including medicine\u0026nbsp;\u003csup\u003e26\u0026ndash;29\u003c/sup\u003e. Specifically, we used Fundus2Angio architecture, which was designed for color-to-FA translation\u0026nbsp;\u003csup\u003e10\u003c/sup\u003e.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe quality of the generated synthetic FA images was measured by two metrics: Structural Similarity Index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS)\u0026nbsp;\u003csup\u003e30,31\u003c/sup\u003e. Both measures are generally used to quantify the similarity between two images. SSIM focuses on the luminance, contrast, and structure of two images, while LPIPS leverages deep learning to capture complex visual similarities, better aligning with human perception. We also calculated these metrics for the grayscale images of the color fundus for comparison.\u003c/p\u003e\n\u003ch2\u003eSegmentation\u003c/h2\u003e\n\u003cp\u003eFor the segmentation of NPAs, we used U-Net with Monte Carlo dropout\u0026nbsp;\u003csup\u003e32,33\u003c/sup\u003e. U-Net is a widely used medical image segmentation model, and Monte Carlo dropout is an uncertainty estimation method for deep learning models. By combining these methods, we can analyze both the prediction accuracy and uncertainty estimates. We opted for U-Net due to its straightforward architecture. Our previous research indicated that a deep learning model with simple architecture produces more informative uncertainty estimation\u0026nbsp;\u003csup\u003e34\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eThe segmentation model was trained with different input types: (1) FA model, (2) color fundus model, and (3) color + synthesized FA model. The models were trained with a batch size of 4, the Adam optimizer, an initial learning rate of 0.0004, and the cross-entropy loss function.\u003c/p\u003e\n\u003ch2\u003eEvaluation\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eUsing the test subset, we evaluated the accuracy of the segmentation models by the Dice coefficient score and sensitivity. Sensitivity is the ratio of true-positive pixels to the total NPA pixels in an image. Since we cannot assume the Gaussian distribution on the Dice score, the Wilcoxon signed-rank test was used to test the difference in prediction performance. The confidence intervals for differences in Dice scores were calculated using the bootstrap method.\u003c/p\u003e\n\u003cp\u003eAdditionally, we calculated the standard deviation (SD) from 100 Monte Carlo dropout predictions for uncertainty estimation. To compare the nature of each model in terms of uncertainty, the median SD and the area fraction with SD \u0026gt; 0.1 were quantified and compared using the Wilcoxon signed-rank test. Furthermore, we performed individual-level comparisons to assess the impact of the absence of FA or the inclusion of synthesized FA on a case-by-case basis. For multiple testing correction, the Bonferroni method was applied.\u0026nbsp;\u003cbr\u003e\u0026nbsp;\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cdiv class=\"DefinitionList\"\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eartificial intelligence (AI)\u003c/div\u003e \u003cdiv class=\"Description\"\u003e\u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eretinal vein occlusion (RVO)\u003c/div\u003e \u003cdiv class=\"Description\"\u003e\u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003enon\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eperfusion area (NPA)\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003efluorescein angiography (FA)\u003c/div\u003e \u003cdiv class=\"Description\"\u003e\u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003egenerative adversarial network (GAN)\u003c/div\u003e \u003cdiv class=\"Description\"\u003e\u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eMonte Carlo (MC)\u003c/div\u003e \u003cdiv class=\"Description\"\u003e\u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003estructural similarity index (SSIM)\u003c/div\u003e \u003cdiv class=\"Description\"\u003e\u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003elearned perceptual image patch similarity (LPIPS).\u003c/div\u003e \u003cdiv class=\"Description\"\u003e\u003c/div\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgments\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors thank the members of the Laboratory of Photobiology, Keio University School of Medicine for their technical and administrative support. Especially, we would like to express our gratitude to Kae Otsuka for their assistance with data collection.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor Contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eK.M., Y.K., and T.K. conceptualized the research and wrote the main manuscript. Y.K., N.O., and M.I. curated the dataset. K.M. wrote the code and analyzed the data. K.N. and T.K. supervised the research. All authors reviewed the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors TK, YK, and KM are inventors on patents and patent applications related to this work (JP7143862, EP19741888, CN201980018367, US16/930510, and JP2022147056).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe dataset is not publicly available due to their containing information that could compromise the privacy of research participants but may be available from the corresponding author upon reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eThe Royal College of Ophthalmologists. \u003cem\u003eClinical Guidelines Retinal Vein Occlusion (RVO)\u003c/em\u003e. (The Royal College of Ophthalmologists, 2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKornblau, I. S. \u0026amp; El-Annan, J. F. Adverse reactions to fluorescein angiography: A comprehensive review of the literature. Surv. Ophthalmol. 64, 679\u0026ndash;693 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNobre Cardoso, J. \u003cem\u003eet al.\u003c/em\u003e Systematic Evaluation of Optical Coherence Tomography Angiography in Retinal Vein Occlusion. Am. J. Ophthalmol. 163, 93\u0026ndash;107.e6 (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNagasato, D. \u003cem\u003eet al.\u003c/em\u003e Automated detection of a nonperfusion area caused by retinal vein occlusion in optical coherence tomography angiography images using deep learning. PLoS One 14, (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHirano, Y. \u003cem\u003eet al.\u003c/em\u003e Multimodal Imaging of Microvascular Abnormalities in Retinal Vein Occlusion. J. Clin. Med. Res. 10, (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eInoda, S. \u003cem\u003eet al.\u003c/em\u003e Deep-learning-based AI for evaluating estimated nonperfusion areas requiring further examination in ultra-widefield fundus images. Sci. Rep. 12, 21826 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRen, X. \u003cem\u003eet al.\u003c/em\u003e Artificial intelligence to distinguish retinal vein occlusion patients using color fundus photographs. Eye 37, 2026\u0026ndash;2032 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMiao, J. \u003cem\u003eet al.\u003c/em\u003e Deep Learning Models for Segmenting Non-perfusion Area of Color Fundus Photographs in Patients With Branch Retinal Vein Occlusion. Front. Med. 9, 794045 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNunez do Rio, J. M. \u003cem\u003eet al.\u003c/em\u003e Deep Learning-Based Segmentation and Quantification of Retinal Capillary Non-Perfusion on Ultra-Wide-Field Retinal Fluorescein Angiography. J. Clin. Med. Res. (2020) doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/jcm9082537\u003c/span\u003e\u003cspan address=\"10.3390/jcm9082537\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKamran, S. A. \u003cem\u003eet al.\u003c/em\u003e Fundus2Angio: A Conditional GAN Architecture for Generating Fluorescein Angiography Images from Retinal Fundus Photography. in \u003cem\u003eAdvances in Visual Computing\u003c/em\u003e 125\u0026ndash;138 (Springer International Publishing, 2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang, K. \u003cem\u003eet al.\u003c/em\u003e Lesion-aware generative adversarial networks for color fundus image to fundus fluorescein angiography translation. Comput. Methods Programs Biomed. 229, 107306 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePham, Q. T. M., Ahn, S., Shin, J. \u0026amp; Song, S. J. Generating future fundus images for early age-related macular degeneration based on generative adversarial networks. Comput. Methods Programs Biomed. 216, 106648 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKamran, S. A. \u003cem\u003eet al.\u003c/em\u003e RV-GAN: Segmenting Retinal Vascular Structure in Fundus Photographs Using a Novel Multi-scale Generative Adversarial Network. in \u003cem\u003eMedical Image Computing and Computer Assisted Intervention \u0026ndash; MICCAI 2021\u003c/em\u003e 34\u0026ndash;44 (Springer International Publishing, 2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTavakkoli, A., Kamran, S. A., Hossain, K. F. \u0026amp; Zuckerbrod, S. L. A novel deep learning conditional generative adversarial network for producing angiography images from retinal fundus photographs. Sci. Rep. 10, 21580 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGupta, R., Sharma, A. \u0026amp; Kumar, A. Super-Resolution using GANs for Medical Imaging. Procedia Comput. Sci. 173, 28\u0026ndash;35 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang, L., Dai, H. \u0026amp; Sang, Y. Med-SRNet: GAN-Based Medical Image Super-Resolution via High-Resolution Representation Learning. \u003cem\u003eComput. Intell. Neurosci.\u003c/em\u003e 2022, 1744969 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAhmad, W., Ali, H., Shah, Z. \u0026amp; Azmat, S. A new generative adversarial network for medical images super resolution. Sci. Rep. 12, 9533 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang, Q. \u003cem\u003eet al.\u003c/em\u003e Low-Dose CT Image Denoising Using a Generative Adversarial Network With Wasserstein Distance and Perceptual Loss. IEEE Trans. Med. Imaging 37, 1348\u0026ndash;1357 (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDeng, Z. \u003cem\u003eet al.\u003c/em\u003e RFormer: Transformer-Based Generative Adversarial Network for Real Fundus Image Restoration on a New Clinical Benchmark. IEEE J Biomed Health Inform 26, 4645\u0026ndash;4655 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDenck, J., Guehring, J., Maier, A. \u0026amp; Rothgang, E. MR-contrast-aware image-to-image translations with generative adversarial networks. Int. J. Comput. Assist. Radiol. Surg. 16, 2069\u0026ndash;2078 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCohen, J. P., Luck, M. \u0026amp; Honari, S. Distribution Matching Losses Can Hallucinate Features in Medical Image Translation. \u003cem\u003earXiv [cs.CV]\u003c/em\u003e (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTeramoto, A. \u003cem\u003eet al.\u003c/em\u003e Deep learning approach to classification of lung cytological images: Two-step training using actual and synthesized images by progressive growing of generative adversarial networks. PLoS One 15, e0229951 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLevine, A. B. \u003cem\u003eet al.\u003c/em\u003e Synthesis of diagnostic quality cancer pathology images by generative adversarial networks. J. Pathol. 252, 178\u0026ndash;188 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIsola, P., Zhu, J.-Y., Zhou, T. \u0026amp; Efros, A. A. Image-to-Image Translation with Conditional Adversarial Networks. \u003cem\u003earXiv [cs.CV]\u003c/em\u003e (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGoodfellow, I. \u003cem\u003eet al.\u003c/em\u003e Generative Adversarial Nets. in \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e (eds. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. \u0026amp; Weinberger, K. Q.) vol. 27 (Curran Associates, Inc., 2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFujioka, T. \u003cem\u003eet al.\u003c/em\u003e Breast Ultrasound Image Synthesis using Deep Convolutional Generative Adversarial Networks. Diagnostics (Basel) 9, (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKoshino, K. \u003cem\u003eet al.\u003c/em\u003e Narrative review of generative adversarial networks in medical and molecular imaging. Ann Transl Med 9, 821 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJeong, J. J. \u003cem\u003eet al.\u003c/em\u003e Systematic Review of Generative Adversarial Networks (GANs) for Medical Image Classification and Segmentation. J. Digit. Imaging 35, 137\u0026ndash;152 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSkandarani, Y., Jodoin, P.-M. \u0026amp; Lalande, A. GANs for Medical Image Synthesis: An Empirical Study. J. Imaging Sci. Technol. 9, (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang, R., Isola, P., Efros, A. A., Shechtman, E. \u0026amp; Wang, O. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. \u003cem\u003earXiv [cs.CV]\u003c/em\u003e (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, Z., Bovik, A. C., Sheikh, H. R. \u0026amp; Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600\u0026ndash;612 (2004).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRonneberger, O., Fischer, P. \u0026amp; Brox, T. U-net: Convolutional networks for biomedical image segmentation. in \u003cem\u003eMedical Image Computing and Computer-Assisted Intervention \u0026ndash; MICCAI 2015\u003c/em\u003e vol. 9351 234\u0026ndash;241 (Springer Verlag, 2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGal, Y. \u0026amp; Ghahramani, Z. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. in \u003cem\u003eProceedings of The 33rd International Conference on Machine Learning\u003c/em\u003e vol. 48 1050\u0026ndash;1059 (International Machine Learning Society (IMLS), 2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMasayoshi, K. \u003cem\u003eet al.\u003c/em\u003e Automatic segmentation of non-perfusion area from fluorescein angiography using deep learning with uncertainty estimation. Informatics in Medicine Unlocked 32, 101060 (2022).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Deep Learning, Generative Artificial Intelligence, Retinal Vein Occlusion, Fluorescein Angiography","lastPublishedDoi":"10.21203/rs.3.rs-3871406/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3871406/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe non-perfusion area (NPA) of the retina is an important indicator in the visual prognosis of patients with retinal vein occlusion (RVO). However, the current evaluation method of NPA, fluorescein angiography (FA), is invasive and burdensome. In this study, we examined the use of deep learning models for detecting NPA in color fundus images, bypassing the need for FA, and we also investigated the utility of synthetic FA generated from color fundus images. The models were evaluated using the Dice score and Monte Carlo dropout uncertainty. We retrospectively collected 403 sets of color fundus and FA images from 319 RVO patients. We trained three deep learning models on FA, color fundus images, and synthetic FA. As a result, though the FA model achieved the highest score, the other two models also performed comparably. We found no statistical significance in median Dice scores between the models. However, the color fundus model showed significantly higher uncertainty than the other models (p\u0026thinsp;\u0026lt;\u0026thinsp;0.05). In conclusion, deep learning models can detect NPAs from color fundus images with reasonable accuracy, though with somewhat less prediction stability. Synthetic FA stabilizes the prediction and reduces misleading uncertainty estimates by enhancing image quality.\u003c/p\u003e","manuscriptTitle":"Deep Learning Segmentation of Non-perfusion Area from Color Fundus Images and AI-generated Fluorescein Angiography","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-30 17:19:04","doi":"10.21203/rs.3.rs-3871406/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-02-28T14:57:22+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-01-29T08:22:45+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-01-25T11:29:30+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-01-25T11:22:45+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2024-01-17T01:05:42+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f762dc01-99d2-4d72-8fee-96c6a82208d7","owner":[],"postedDate":"January 30th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":28420583,"name":"Health sciences/Diseases/Eye diseases/Retinal diseases"},{"id":28420584,"name":"Health sciences/Medical research/Translational research"},{"id":28420585,"name":"Biological sciences/Computational biology and bioinformatics/Image processing"},{"id":28420586,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"}],"tags":[],"updatedAt":"2025-01-17T00:05:59+00:00","versionOfRecord":{"articleIdentity":"rs-3871406","link":"https://doi.org/10.1038/s41598-024-61561-x","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2024-05-11 21:18:08","publishedOnDateReadable":"May 11th, 2024"},"versionCreatedAt":"2024-01-30 17:19:04","video":"","vorDoi":"10.1038/s41598-024-61561-x","vorDoiUrl":"https://doi.org/10.1038/s41598-024-61561-x","workflowStages":[]},"version":"v1","identity":"rs-3871406","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3871406","identity":"rs-3871406","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}