An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia

cutaneous leishmaniasis ALL Metrics - Views Downloads How to cite this article Mohammed AB, Mohammed FS, Zewdu FT et al. An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia [version 1; peer review: 1 approved with reservations, 1 not approved]. NIHR Open Res 2025, 5:12 (https://doi.org/10.3310/nihropenres.13869.1) NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article. Export Citation Sciwheel EndNote Ref. Manager Bibtex ProCite Sente Select a format first ▬ ✚ Research Article [version 1; peer review: 1 approved with reservations, 1 not approved] Amel Beshir Mohammed1,2, Fewzia Shikur Mohammed1,2, Feleke Tilahun Zewdu3,4, [...] Shimelis Doni Nigusse https://orcid.org/0000-0003-0455-2518 1,2, Saba Lambert https://orcid.org/0000-0003-4376-5495 1,5, Michael Marks https://orcid.org/0000-0002-7585-4743 5-7, Stephen L. Walker5,7, Endalamaw Gadisa https://orcid.org/0000-0002-5179-6278 8, SHARP CollaborationAmel Beshir Mohammed1,2, Fewzia Shikur Mohammed1,2, [...] Feleke Tilahun Zewdu3,4, Shimelis Doni Nigusse https://orcid.org/0000-0003-0455-2518 1,2, Saba Lambert https://orcid.org/0000-0003-4376-5495 1,5, Michael Marks https://orcid.org/0000-0002-7585-4743 5-7, Stephen L. Walker5,7, Endalamaw Gadisa https://orcid.org/0000-0002-5179-6278 8, SHARP Collaboration PUBLISHED 18 Feb 2025 Author details Author details 1 ALERT Hospital, Addis Ababa, Ethiopia 2 Department of Dermatovenereology, College of Health Sciences, Addis Ababa University, Addis Ababa, Addis Ababa, Ethiopia 3 Boru Meda Hospital, Boru Meda, Ethiopia 4 Department of Epidemiology and Biostatistics, Institute of Public Health, College of Medicine and Health Sciences, University of Gondar, Gondar, Ethiopia 5 Department of Clinical Research, Faculty of Infectious and Tropical Diseases, London School of Hygiene & Tropical Medicine, London, UK 6 Division of Infection and Immunity, University College London, London, UK 7 Hospital for Tropical Diseases, University College London Hospitals NHS Foundation Trust, London, UK 8 Department of Malaria and Neglected Tropical Disease Research, Armauer Hansen Research Institute, Addis Ababa, Ethiopia 2 Department of Dermatovenereology, College of Health Sciences, Addis Ababa University, Addis Ababa, Addis Ababa, Ethiopia 3 Boru Meda Hospital, Boru Meda, Ethiopia 4 Department of Epidemiology and Biostatistics, Institute of Public Health, College of Medicine and Health Sciences, University of Gondar, Gondar, Ethiopia 5 Department of Clinical Research, Faculty of Infectious and Tropical Diseases, London School of Hygiene & Tropical Medicine, London, UK 6 Division of Infection and Immunity, University College London, London, UK 7 Hospital for Tropical Diseases, University College London Hospitals NHS Foundation Trust, London, UK 8 Department of Malaria and Neglected Tropical Disease Research, Armauer Hansen Research Institute, Addis Ababa, Ethiopia Amel Beshir Mohammed Roles: Methodology, Writing – Review & Editing Roles: Methodology, Writing – Review & Editing Fewzia Shikur Mohammed Roles: Investigation, Methodology, Writing – Review & Editing Roles: Investigation, Methodology, Writing – Review & Editing Feleke Tilahun Zewdu Roles: Investigation, Methodology, Writing – Review & Editing Roles: Investigation, Methodology, Writing – Review & Editing Shimelis Doni Nigusse Roles: Investigation, Methodology, Writing – Review & Editing Roles: Investigation, Methodology, Writing – Review & Editing Saba Lambert Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Writing – Review & Editing Michael Marks Roles: Conceptualization, Data Curation, Formal Analysis, Funding Acquisition, Investigation, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing Roles: Conceptualization, Data Curation, Formal Analysis, Funding Acquisition, Investigation, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing Stephen L. Walker Roles: Conceptualization, Formal Analysis, Funding Acquisition, Investigation, Methodology, Writing – Original Draft Preparation Roles: Conceptualization, Formal Analysis, Funding Acquisition, Investigation, Methodology, Writing – Original Draft Preparation Endalamaw Gadisa Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Writing – Review & Editing OPEN PEER REVIEW REVIEWER STATUS Cutaneous leishmaniasis (CL) remains a major public health challenge, especially in endemic regions like Ethiopia, where an estimated 40,000 new cases occur annually. Effective treatment evaluation for CL relies on consistent clinical assessments, yet variability in lesion descriptions can complicate reliable outcome measures. We conducted an inter-reliability study of clinicians’ evaluations of CL lesion morphology and size at ALERT Hospital, Addis Ababa. Twelve clinicians independently examined 12 patients with parasitologically confirmed CL, each clinician assessing lesion morphology, size, and severity. We found high consistency in reporting major morphological categories (e.g., plaques) but significant variability in secondary features like dyspigmentation and scale, as well as mucosal involvement. Lesion size measurements showed limited variability, suggesting its reliability as a potential measure for future clinical trials. Disparities in severity assessments highlight the need for a standardized scoring system in CL. Our findings underscore the importance of training for consistent, high-quality clinical evaluations of CL and suggests that lesion size could be a reproducible outcome measure in treatment efficacy trials. Cutaneous leishmaniasis is a significant public health issue in Ethiopia, with around 40,000 new cases each year. The disease can cause a wide variety of different types of skin lesions which can complicate diagnosis and assessment. Conducting reliable treatment evaluation requires consistent assessment of cutaneous leishmaniasis lesions, especially in clinical trials. We conducted a prospective study at ALERT Hospital in Addis Ababa to assess how consistently 12 clinicians, who were all experienced in managing patients with cutaneous leishmaniasis, evaluated the appearance, size and severity of cutaneous leishmaniasis lesions in 12 patients. We found that whilst clinicians were broadly consistent in identifying the major features such as the type of lesions they showed more significant variability in describing secondary features and severity. Lesion size measurements were more consistent, making them a promising tool for future clinical trials. The findings highlight the need for better training and standardized scoring systems to improve the reliability of clinical evaluations in cutaneous leishmaniasis. cutaneous leishmaniasis Corresponding Author(s) Michael Marks ([email protected]) Grant information: This project is funded by the National Institute for Health and Care Research (NIHR) under its [‘Research and Innovation for Global Health Transformation programme (Grant Reference Number NIHR200125)]. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Copyright: © 2025 Mohammed AB et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. How to cite: Mohammed AB, Mohammed FS, Zewdu FT et al. An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia [version 1; peer review: 1 approved with reservations, 1 not approved]. NIHR Open Res 2025, 5:12 (https://doi.org/10.3310/nihropenres.13869.1) First published: 18 Feb 2025, 5:12 (https://doi.org/10.3310/nihropenres.13869.1) Latest published: 14 May 2025, 5:12 (https://doi.org/10.3310/nihropenres.13869.2) The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. There is a newer version of this article available. of this article available. Cutaneous leishmaniasis (CL) is the most common form of leishmaniasis. The disease causes skin lesions on exposed parts of the body at the site of an infected sandfly bite. With or without treatment CL lesions can lead to life-long scars and result in serious disability or stigma. It is estimated that 600 000 to 1 million new cases occur worldwide annually with 40,000 new cases of CL in Ethiopia1. Three Leishmania species have been reported in Ethiopia, but the vast majority of cases are thought to be due to L aethiopica. L. aethiopica is associated with a variety of clinical phenotypes. The most frequent presentation is localized CL (LCL), but infection is also associated with mucocutaneous leishmaniasis (MCL), and diffuse CL (DCL). LCL usually starts as a papule that gradually enlarges forming nodules or plaques. Ulceration may occur with L. aethiopica. MCL can occur either via the bite of the sand fly at the border of mucosal surfaces or through extension from an adjacent cutaneous lesion. It most commonly involves the nasal mucosa but the lips and oral mucosa can also be involved. DCL is a chronic and progressive disorder which starts with papular or nodular lesions which become larger and more numerous2. Despite the large number of affected individuals, studies of the clinical manifestations of L. aethiopica, including the morphology of the lesions, progression of the disease and response to therapy, are limited. A study in North-west Ethiopia reported the most common features to be induration, erythema, ulceration, and crusting. The WHO strategic framework for integrated control and management of “skin-related” NTDs highlighted anti-microbial therapy for CL as a research gap and this is particularly the case for treatment of L. aethiopica infection3,4. Robust outcome measures are required for treatment efficacy studies. Outcome measures in CL in studies are based on clinical evaluation of individual lesions including re-epithelization in ulcerated lesions and flattening in non-ulcerated ones and lesion measurement. In 2018, a proposal for harmonized clinical trial methodologies for cutaneous leishmaniasis was published to develop a consensus and standardize clinical evaluation as a measure of disease response and cure in CL5. Given the reliance on clinical observations of lesion morphology and severity it is therefore essential to evaluate whether the clinical evaluation of individual physicians is comparable. We therefore conducted a study to assess the inter-rater reliability of clinicians’ assessment of lesion morphology and size in individuals affected by CL in Ethiopia. Neither patients nor the public were directly involved in the design or analysis of the study. The study was conducted nested within a larger cohort study of CL in Ethiopia. As part of this larger study, we conducted an inter-observer exercise at ALERT comprehensive specialized Hospital, Addis Ababa6. Examiners were selected from the clinical staff of ALERT Hospital and members of the Skin Health Africa Research Programme (SHARP) team. We collected information on their previous training in skin disease and experience of managing CL. Clinicians at ALERT Hospital identified 12 individuals with a parasitologically confirmed diagnosis of CL who were invited to participate in the study. A standardized proforma was used to capture routine clinical data about each affected individual. For individuals with more than one clinical lesion of CL, the treating clinician indicated the “index” lesion to be assessed in the inter-observer assessment exercise. This “index” lesion was the lesion identified by the affected individual as the one of greatest concern prior to the start of the exercise. A computer generated latin-square was used to randomize an order of evaluation for each observer and affected individual. Each observer was assigned a unique letter and each affected individual a unique number. To measure inter-observer agreement each observer independently examined all 12 individuals with CL once for a period of 5 minutes. Examiners recorded on a standard data collection sheet their assessment of each “index” lesion. Standardized definitions of CL developed for our cohort study (Box 1) were used6. For each lesion examiners were asked to record morphology, size and the presence or absence of clinical features including induration, ulceration, crust, hyper/hypopigmentation, scarring and mucosal involvement. Examiners measured the largest diameter of the lesion using a disposable tape measure. Examiners were asked to provide an assessment of the clinical phenotype of CL and to classify the disease severity as mild or moderate or severe. All data were double entered into an electronic study database for analysis. CL is diagnosed in a person with skin and/or mucosal lesions with evidence of Leishmania infection in the affected tissues characterised by the presence of Leishmania amastigotes on smear microscopy or growth of Leishmania promastigotes in culture or the detection of Leishmania DNA by polymerase chain reaction. Localized CL: A confirmed case of leishmaniasis, with no mucosal involvement, characterized by ten or fewer cutaneous papules and/or nodules and/or plaques with or without ulceration involving one body site Multi-regional localized CL: A confirmed case of leishmaniasis, with no mucosal involvement, characterized by ten or fewer cutaneous papules and/or nodules and/or plaques with or without ulceration involving two or more body sites Mucocutaneous leishmaniasis: A confirmed case of leishmaniasis characterized by ten or fewer papules and/or nodules and/or plaques with or without ulceration involving skin and an adjacent mucosal surface Mucosal leishmaniasis: A confirmed case of leishmaniasis characterized by papules and/or nodules and/or plaques with or without ulceration involving exclusively a mucosal surface Diffuse CL: A confirmed case of leishmaniasis characterized by eleven or more papules and/or nodules and/or plaques with or without mucosal involvement. Body sites are classified as: 1. Head and neck 2. Torso - anterior (including genitalia) 3. Torso - posterior (including buttocks) 4. Right upper limb 5. Left upper limb 6. Right lower limb 7. Left lower limb As there is no published data on the degree of agreement in assessing cases of Cutaneous Leishmaniasis we used a convenience sample size based on ensuring we had a range of both examiners and patients with clinical differing clinical phenotypes. The primary planned analysis was descriptive. We report the proportion of affected individuals judged to have each feature by the examiners. For lesion size we describe variability by reporting the median and IQR lesion size reported for each index lesion. Findings are reported in line with the STROBE guidelines. Analysis was conducted in R version 4.2.1. Twelve affected individuals agreed to participate in the study. Seven were male and five female. The median age was 25 years. Based on the treating clinician’s assessment six had been diagnosed with LCL, four with MCL and two with mucosal CL. All twelve individuals were currently receiving inpatient treatment with intramuscular sodium stibogluconate. Twelve examiners took part in the study. Seven of the clinicians were male and 5 were female. The median age of examiners was 39. The majority (n = 10, 83%) were clinicians with specialist postgraduate training in dermatology and six (50%) were accredited dermatologists. (Table 1). | Median Age (years) | 39 | | Male | 7 | | Years experience | 8.3 (SD 6.8) | | Clinical specialty | | | Dermatologist | 6 | | Dermatology Trainee | 3 | | Infectious Diseases | 1 | | MSc in Dermatology | 1 | | GP with Dermatology postgraduate diploma | 1 | Some clinical features appeared to be more consistently reported by examiners than others (Table 2). For lesion morphology all examiners agreed on the presence of plaques in one individual, whilst 10 or more examiners agreed on the presence of plaques in five. In the remaining individuals with CL either eight or nine examiners assessed the lesion to be a plaque. All examiners agreed there were no nodular lesions in eight individuals. For the remaining four affected individuals one to four examiners felt there were nodular lesions. There was greater variability in reporting of the presence or absence of papules. For five individuals all examiners agreed papules were absent whereas for seven individuals between one to five examiners assessed they were present. Involvement of the mucosa was reported by all examiners for three individuals and by 11 examiners in two others. For three individuals with CL all examiners agreed there was no mucosal involvement and for a further two individuals only one or two examiners assessed the mucosa to be affected. For two individuals there was more marked disagreement with seven and nine examiners reporting involvement and five and three examiners reporting no mucosal involvement. For lesion features all examiners agreed on the presence of erythema in three individuals and at least 10 examiners agreed it was present in a further four. For the remaining individuals with CL there was marked variability with between three to eight examiners reporting erythema. For five individuals all examiners reported the presence of induration and for a further four at least 11 examiners felt induration was present. For the remaining three affected individuals nine examiners felt induration was present and three felt it was absent. For six individuals with CL all examiners agreed that there was no evidence of ulceration. For the remaining six patients the number of examiners reporting ulceration varied from one to nine. For two individuals all 12 examiners agreed that there was evidence of crust and for two individuals all examiners agreed crust was absent. For the remaining eight individuals with CL the number of examiners reporting crust varied from one to eleven. There were no individuals with CL whom examiners were in agreement on the presence or absence of scale but there were six where at least 10 examiners agreed scale was present and three where at least 10 examiners agreed it was absent. Finally, all examiners agreed there was no evidence of scarring in four individuals. For the remaining eight between one and eleven examiners deemed scarring to be present. Estimates of the reported maximum lesion diameter were broadly consistent across examiners but graphical visualization with a modified Bland-Altman plot suggests variation increased with the size of the lesion (Figure 1)7. The majority of observers assessed that two third of patients had moderate to severe disease with the remaining affected individuals assessed as having mild-moderate disease. There were only two individuals with CL in whom there was a 2-grade discrepancy in the assessment of severity but there were no individuals in whom all examiners assigned the same severity score and only one in whom there was more than 80% agreement. Disagreement about mucosal involvement also meant that there were only four (33%) individuals with CL where all 12 examiners and the treating clinicians agreed on the clinical phenotype (Table 3). In the first ever study evaluating the inter-observer reliability of clinical examination to assess lesion morphology in CL we demonstrated considerable variation in the reporting of several key clinical features. Reporting of overall lesion morphology as plaque, nodule or papule was generally highly consistent but the presence or absence of secondary clinical characteristics such as dyspigmentation or the presence of scale were more variable. Lesion size is proposed as a key characteristic for assessing treatment outcomes in future trials5. Reassuringly, we showed that there was limited variation in the assessed size of lesions when measured by multiple independent assessors, although there seemed to be evidence that variability was highest for larger CL lesions. Overall, these data suggest that lesion size assessment is likely to be a reliable measure in both observational and interventional prospective studies. We found assessment of mucosal involvement to be variable. As this has important implications for treatment and characterization for clinical trials it may be advisable in some cases to obtain an assessment by an ear, nose and throat specialist to allow accurate characterization. There is not yet a standardized definition of severity for CL6. In the absence of such a system we found little consensus amongst clinicians when assessing individuals with CL which poses challenges in reliable reporting of their characteristics in both observational and interventional studies. Development of a validated severity score should be considered to help standardize practice. Inter-observer variability is well recognized in clinical assessment particularly when assessing subjective features of disease. Similar to our findings, previous studies in other NTDs have shown that the degree of reliability may vary for different clinical features within a given disease. For example, diagnosis of scabies by healthcare workers has been shown to vary markedly depending on the severity of the infestation8,9. Similarly for leprosy studies have shown marked differences in inter-observer reliability in determining the presence or absence of nerve thickening or in the reliability of different methods of assessing sensory loss within skin lesions10. The new WHO roadmap for NTDs focuses on strengthening responses to skin-NTDs3. Achieving this will require considerable investment in training of healthcare workers in the diagnosis and management of a range of skin diseases including CL. Given the degree of variability between expert examiners in this study our data suggest such efforts will require considerable effort to ensure high-quality, reliable and reproducible clinical diagnosis and assessment of clinical healing parameters. Our study has limitations. Firstly, all affected individuals were inpatients receiving care for CL at a referral centre. This is likely to have resulted in selection of individuals with more severe disease than might be seen elsewhere. Similarly, all individuals were receiving treatment and therefore their lesions might differ from untreated ones. Whilst this might affect lesion morphology, we believe it is unlikely to have affected our assessment of inter-examiner reliability. The generalizability of our findings to regions of the world where other Leishmania species are endemic is unclear, particularly in settings where ulcerated lesions may be more common. Assessment of inter-examiner reliability for multiple examiners is challenging. Conventionally kappa-scores are calculated to compare the performance of an examiner against a reference-standard. Multi-examiner weighted kappa scores have been developed to extend this approach to situations broadly analogous to our study design. However, both conventional and multi-examiner kappa scores have weaknesses11. Disagreements limited to a small number of individuals being examined can have an outsize impact on kappa scores even when there is near universal consensus on the majority individuals. We therefore opted to only present descriptive statistics. Finally, our study focuses on the assessment of individuals with known CL by experts. Further studies are needed to assess the performance of other cadres of healthcare workers in the initial evaluation of individuals with suspected CL. Trials of novel and improved therapeutics for CL are urgently needed. To provide comprehensive data on efficacy such trials will need to combine reliable and reproducible parasitological, clinical and patient reported outcome measures. Measurement of lesion size is likely to be a critical component of such measures, and our data are reassuring that this can be reliably measured by multiple independent assessors. More work is needed to identify and improve assessment of other clinical features in order to develop robust trial outcome measures for CL. Ethical approval was obtained from AHRI/ALERT Ethics Review Committee (Ref: PO/23/21), the National Research Ethics Review Committee (Ref: 7/2-506/m259/35), and the London School of Hygiene and Tropical Medicine (Ref: 26421) in October 2021. Written individual informed consent was obtained from all participants (examiners and affected individuals) in the study. The study complied with the Declaration of Helsinki. LSHTM Data Compass: An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia. https://doi.org/10.17037/DATA.0000449112 The project contains the following underlying data: LSHTM Data Compass - An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia. https://doi.org/10.17037/DATA.0000449112 The project contains the following supplementary data: CL_Interobserver_CRF – An ODK data collection form as used in the study CL_interobserver Codebook – A codebook describing the data coding Inter-observer adult consent – a Copy of the patient information sheet and consent form Data is available under CC by 4.0 license. We wish to thank the individuals and communities for their participation in the work of the Skin Health Africa Research Programme. Faculty Opinions recommendedReferences - 1. Alvar J, Vélez ID, Bern C, et al.: Leishmaniasis worldwide and global estimates of its incidence. PLoS One. 2012; 7(5): 35671. PubMed Abstract | Publisher Full Text | Free Full Text - 2. Bryceson AD: Diffuse Cutaneous Leishmaniasis in Ethiopia. I. the clinical and histological features of the disease. Trans R Soc Trop Med Hyg. 1969; 63(6): 708–37. PubMed Abstract | Publisher Full Text - 3. Ending the neglect to attain the sustainable development goals: a strategic framework for integrated control and management of skin-related neglected tropical diseases. (accessed June 25, 2024). Reference Source - 4. van Griensven J, Gadisa E, Aseffa A, et al.: Treatment of Cutaneous Leishmaniasis caused by Leishmania aethiopica: a systematic review. PLoS Negl Trop Dis. 2016; 10(3): e0004495. PubMed Abstract | Publisher Full Text | Free Full Text - 5. Olliaro P, Grogl M, Boni M, et al.: Harmonized clinical trial methodologies for localized Cutaneous Leishmaniasis and potential for extensive network with capacities for clinical evaluation. PLoS Negl Trop Dis. 2018; 12(1): e0006141. PubMed Abstract | Publisher Full Text | Free Full Text - 6. Mohammed AB, Mohammed FS, Zewdu FT, et al.: Protocol for a prospective observational cohort study of Cutaneous Leishmaniasis in Ethiopia [version 1; peer review: 2 approved with reservations]. NIHR Open Res. 2023; 3: 49. PubMed Abstract | Publisher Full Text | Free Full Text - 7. Christensen HS, Borgbjerg J, Børty L, et al.: On Jones et al.’s method for extending Bland-Altman plots to limits of agreement with the mean for multiple observers. BMC Med Res Methodol. 2020; 20(1): 304.Diffuse cutaneous leishmaniasis PubMed Abstract | Publisher Full Text | Free Full Text - 8. Walker SL, Collinson S, Timothy J, et al.: A community-based validation of the International Alliance for the Control of Scabies Consensus Criteria by expert and non-expert examiners in Liberia. PLoS Negl Trop Dis. 2020; 14(10): e0008717. PubMed Abstract | Publisher Full Text | Free Full Text - 9. Osti MH, Sokana O, Lake S, et al.: The body distribution of scabies skin lesions. JEADV Clinical Practice. 2022; 1(2): 111–121. Publisher Full Text - 10. Ponnighaus JM, Fine PE, Bliss L: Certainty levels in the diagnosis of leprosy. Int J Lepr Other Mycobact Dis. 1987; 55(3): 454–62. PubMed Abstract - 11. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1): 159–74. PubMed Abstract | Publisher Full Text - 12. Marks M: Data for: "An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in Cutaneous Leishmaniasis caused by Leishmania aethiopica in Ethiopia". [Data Collection]. London School of Hygiene & Tropical Medicine, London, United Kingdom, 2025. http://www.doi.org/10.17037/DATA.00004491 Author details Author details 1 ALERT Hospital, Addis Ababa, Ethiopia 2 Department of Dermatovenereology, College of Health Sciences, Addis Ababa University, Addis Ababa, Addis Ababa, Ethiopia 3 Boru Meda Hospital, Boru Meda, Ethiopia 4 Department of Epidemiology and Biostatistics, Institute of Public Health, College of Medicine and Health Sciences, University of Gondar, Gondar, Ethiopia 5 Department of Clinical Research, Faculty of Infectious and Tropical Diseases, London School of Hygiene & Tropical Medicine, London, UK 6 Division of Infection and Immunity, University College London, London, UK 7 Hospital for Tropical Diseases, University College London Hospitals NHS Foundation Trust, London, UK 8 Department of Malaria and Neglected Tropical Disease Research, Armauer Hansen Research Institute, Addis Ababa, Ethiopia 2 Department of Dermatovenereology, College of Health Sciences, Addis Ababa University, Addis Ababa, Addis Ababa, Ethiopia 3 Boru Meda Hospital, Boru Meda, Ethiopia 4 Department of Epidemiology and Biostatistics, Institute of Public Health, College of Medicine and Health Sciences, University of Gondar, Gondar, Ethiopia 5 Department of Clinical Research, Faculty of Infectious and Tropical Diseases, London School of Hygiene & Tropical Medicine, London, UK 6 Division of Infection and Immunity, University College London, London, UK 7 Hospital for Tropical Diseases, University College London Hospitals NHS Foundation Trust, London, UK 8 Department of Malaria and Neglected Tropical Disease Research, Armauer Hansen Research Institute, Addis Ababa, Ethiopia Amel Beshir Mohammed Roles: Methodology, Writing – Review & Editing Roles: Methodology, Writing – Review & Editing Fewzia Shikur Mohammed Roles: Investigation, Methodology, Writing – Review & Editing Roles: Investigation, Methodology, Writing – Review & Editing Feleke Tilahun Zewdu Roles: Investigation, Methodology, Writing – Review & Editing Roles: Investigation, Methodology, Writing – Review & Editing Shimelis Doni Nigusse Roles: Investigation, Methodology, Writing – Review & Editing Roles: Investigation, Methodology, Writing – Review & Editing Saba Lambert Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Writing – Review & Editing Michael Marks Roles: Conceptualization, Data Curation, Formal Analysis, Funding Acquisition, Investigation, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing Roles: Conceptualization, Data Curation, Formal Analysis, Funding Acquisition, Investigation, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing Stephen L. Walker Roles: Conceptualization, Formal Analysis, Funding Acquisition, Investigation, Methodology, Writing – Original Draft Preparation Roles: Conceptualization, Formal Analysis, Funding Acquisition, Investigation, Methodology, Writing – Original Draft Preparation Endalamaw Gadisa Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Writing – Review & Editing Competing interests No competing interests were disclosed. Grant information This project is funded by the National Institute for Health and Care Research (NIHR) under its [‘Research and Innovation for Global Health Transformation programme (Grant Reference Number NIHR200125)]. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Article Versions (2) Copyright © 2025 Mohammed AB et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. metrics VIEWS $counts.viewCount downloads Citations CITE how to cite this article Mohammed AB, Mohammed FS, Zewdu FT et al. An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia [version 1; peer review: 1 approved with reservations, 1 not approved]. NIHR Open Res 2025, 5:12 (https://doi.org/10.3310/nihropenres.13869.1) NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article. track receive updates on this article Track an article to receive email alerts on any updates to this article. Current Reviewer Status: ? Key to Reviewer Statuses VIEW HIDE ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions Version 1 VERSION 1 PUBLISHED 18 Feb 2025 Views 0 How to cite this report: Gómez MA. Reviewer Report For: An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia [version 1; peer review: 1 approved with reservations, 1 not approved]. NIHR Open Res 2025, 5:12 (https://doi.org/10.3310/nihropenres.15071.r35115) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/5-12/v1#referee-response-35115 https://openresearch.nihr.ac.uk/articles/5-12/v1#referee-response-35115 NOTE: it is important to ensure the information in square brackets after the title is included in this citation. Reviewer Report 12 Apr 2025 Maria Adelaida Gómez, Centro Internacional de Entrenamiento e Investigaciones Medicas, Cali, Colombia Not Approved VIEWS 0 Major comments: I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above. Close - It is not clear at what time over the course of disease and treatment were the patients evaluated. In the discussion section one could infer patients were undergoing treatment, but it was not stated Major comments: Minor comments - It is not clear at what time over the course of disease and treatment were the patients evaluated. In the discussion section one could infer patients were undergoing treatment, but it was not stated at what moment over the course of treatment were the patients evaluated. This leads me to question the relationship between the need (assessment of clinical cure) and the evaluation time point. Why was the study conducted on a single time point, instead of evaluating the disease progression over the course of treatment, for example? - For a study like this, images of lesions illustrating the characteristics that were more consistently identified vs. those that were not are necessary. This could also serve as a tool for reference to other clinicians experiencing similar difficulties in clinical assessment of CL (especially non-ulcerated lesions). - In the discussion the authors mention “There is not yet a standardized definition of severity for CL”. WHO and PAHO have generated guidelines that allow characterization of complicated vs. uncomplicated CL, based on number of lesions, lesion size, anatomical site of lesions, among others. This classification is the basis for selection of local vs. systemic treatments. This can be used as a baseline for definition of severity. Minor comments - I suggest caution with the statement of lesions developing at the site of sandfly bite, as this is not always the case (“The disease causes skin lesions on exposed parts of the body at the site of an infected sandfly bite”) - First paragraph of results section “mucosal CL”. In the box the classifications are either mucocutaneous or mucosal leishmaniasis. In this case, what are the authors referring to? MCL or CL? Also table 2 has “Pure mucosal”, please keep consistency in the use of terms. - What instrument was used for measurement of lesion size? This is relevant as the instrument used may introduce bias in the variability, or lack of, in this measurement among observers. - Is the work clearly and accurately presented and does it cite the current literature? Yes - Is the study design appropriate and is the work technically sound? Partly - Are sufficient details of methods and analysis provided to allow replication by others? Partly - If applicable, is the statistical analysis and its interpretation appropriate? Partly - Are all the source data underlying the results available to ensure full reproducibility? No - Are the conclusions drawn adequately supported by the results? Yes

1. Mohammed A, Mohammed F, Zewdu F, Nigusse S, et al.: An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia. NIHR Open Research. 2025; 5. Publisher Full TextCompeting Interests: No competing interests were disclosed. Reviewer Expertise: Human Cutaneous Leishmaniasis CITE HOW TO CITE THIS REPORT Gómez MA. Reviewer Report For: An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia [version 1; peer review: 1 approved with reservations, 1 not approved]. NIHR Open Res 2025, 5:12 (https://doi.org/10.3310/nihropenres.15071.r35115) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/5-12/v1#referee-response-35115 https://openresearch.nihr.ac.uk/articles/5-12/v1#referee-response-35115 NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article. Views 0 How to cite this report: Fuller LC and Wilder-Smith a. Reviewer Report For: An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia [version 1; peer review: 1 approved with reservations, 1 not approved]. NIHR Open Res 2025, 5:12 (https://doi.org/10.3310/nihropenres.15071.r34785) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/5-12/v1#referee-response-34785 https://openresearch.nihr.ac.uk/articles/5-12/v1#referee-response-34785 NOTE: it is important to ensure the information in square brackets after the title is included in this citation. Reviewer Report 26 Feb 2025 annika Wilder-Smith, Faculty of Medicine, University of Zurich, Zurich, Switzerland Approved with Reservations VIEWS 0 This manuscript reports a study nested within a larger piece of work that is referenced. It describes the variability in clinical findings between 12 clinicians in assessing 12 known cutaneous leishmaniasis cases. The setting is a specialist hospital in Ethiopia ... Continue reading We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above. Close This manuscript reports a study nested within a larger piece of work that is referenced. It describes the variability in clinical findings between 12 clinicians in assessing 12 known cutaneous leishmaniasis cases. The setting is a specialist hospital in Ethiopia with acknowledged experience in managing leishmaniasis and hosting the country's flagship dermatology training program. The overarching aim of the report is to highlight the need for 1. standardised reporting structure for cutaneous leishmaniasis to support future studies developing or assessing new therapeutic approaches. 2. Development of a standardised severity score and 3. Development of standardised training for clinical assessors involved in future studies to ensure consistency. Whilst the training and exposure of the majority of the clinicians will be "relevant" it is not standardised and it might be worth emphasising in the discussion this fact and including a recommendation for ensuring any clinical assessors in cutaneous leishmaniasis studies have undertaken specific training on lesion analysis. It might be useful to state if any such training exists (eg on the Open WHO platform NTD channel - obviously under migration to WHO academy at the moment), is this fit for purpose ie generalisable across the globe, or is there an educational gap? I assume there is no scope for inclusion of clinical photographs to illustrate the cases and phenotype characteristics. Specifically small comments:

para 3 you mention a study in North west Ethiopia reporting common features - can you reference this?

you mention a standardised proforma - is it worth showing this?

Is it worth stating that although the clinicians included were experienced you are not able to specifically comment on cutaneous leishmaniasis training or else report on how this is covered in the local Ethiopian Dermatology Curriculum?

Whilst noting the phenotypes and clinical features being measured, might you consider highlighting those most likely to be of clinical relevance in measuring treatment impact? Mucosal involvement: whilst you might want to involve ENT in assisting the development of training tools to guide clinical assessors as to whether mucosal involvement is present or absent, it should be fairly straightforward to clarify with appropriate training. The comments are minor and I hope helpful. 1. standardised reporting structure for cutaneous leishmaniasis to support future studies developing or assessing new therapeutic approaches. 2. Development of a standardised severity score and 3. Development of standardised training for clinical assessors involved in future studies to ensure consistency. Whilst the training and exposure of the majority of the clinicians will be "relevant" it is not standardised and it might be worth emphasising in the discussion this fact and including a recommendation for ensuring any clinical assessors in cutaneous leishmaniasis studies have undertaken specific training on lesion analysis. It might be useful to state if any such training exists (eg on the Open WHO platform NTD channel - obviously under migration to WHO academy at the moment), is this fit for purpose ie generalisable across the globe, or is there an educational gap? I assume there is no scope for inclusion of clinical photographs to illustrate the cases and phenotype characteristics. Specifically small comments:

para 3 you mention a study in North west Ethiopia reporting common features - can you reference this?

you mention a standardised proforma - is it worth showing this?

Is it worth stating that although the clinicians included were experienced you are not able to specifically comment on cutaneous leishmaniasis training or else report on how this is covered in the local Ethiopian Dermatology Curriculum?

Whilst noting the phenotypes and clinical features being measured, might you consider highlighting those most likely to be of clinical relevance in measuring treatment impact? Mucosal involvement: whilst you might want to involve ENT in assisting the development of training tools to guide clinical assessors as to whether mucosal involvement is present or absent, it should be fairly straightforward to clarify with appropriate training. The comments are minor and I hope helpful. - Is the work clearly and accurately presented and does it cite the current literature? Yes - Is the study design appropriate and is the work technically sound? Yes - Are sufficient details of methods and analysis provided to allow replication by others? Partly - If applicable, is the statistical analysis and its interpretation appropriate? Yes - Are all the source data underlying the results available to ensure full reproducibility? Partly - Are the conclusions drawn adequately supported by the results? Yes Competing Interests: No competing interests were disclosed. Reviewer Expertise: I am a dermatologist with research experience in skin neglected tropical diseases. CITE HOW TO CITE THIS REPORT Fuller LC and Wilder-Smith a. Reviewer Report For: An assessment of interobserver agreement on lesion size, morphology and clinical phenotype in cutaneous leishmaniasis caused by Leishmania aethiopica in Ethiopia [version 1; peer review: 1 approved with reservations, 1 not approved]. NIHR Open Res 2025, 5:12 (https://doi.org/10.3310/nihropenres.15071.r34785) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/5-12/v1#referee-response-34785 https://openresearch.nihr.ac.uk/articles/5-12/v1#referee-response-34785 NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article. Alongside their report, reviewers assign a status to the article: - Approved - Approved with reservations - Not approved | Invited Reviewers | || |---|---|---| | 1 | 2 | | | Version 2 (revision) 14 May 25 | read | read | | Version 1 18 Feb 25 | read | read | Sign up for content alerts You are now signed up to receive this alert Alongside their report, reviewers assign a status to the article: Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list: Examples of 'Non-Financial Competing Interests' - Within the past 4 years, you have held joint grants, published or collaborated with any of the authors of the selected paper. - You have a close personal relationship (e.g. parent, spouse, sibling, or domestic partner) with any of the authors. - You are a close professional associate of any of the authors (e.g. scientific mentor, recent student). - You work at the same institute as any of the authors. - You hope/expect to benefit (e.g. favour or employment) as a result of your submission. - You are an Editor for the journal in which the article is published. Examples of 'Financial Competing Interests' - You expect to receive, or in the past 4 years have received, any of the following from any commercial organisation that may gain financially from your submission: a salary, fees, funding, reimbursements. - You expect to receive, or in the past 4 years have received, shared grant support or other funding with any of the authors. - You hold, or are currently applying for, any patents or significant stocks/shares relating to the subject matter of the paper you are commenting on. Sign up for content alerts and receive a weekly or monthly email with all newly published articles Register with NIHR Open Research Already registered? Sign in close Error If you are a previous or current NIHR award holder, sign up for information about developments, publishing and publications from NIHR Open Research. We'll keep you updated on any major new updates to NIHR Open Research Sign In If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password. Email us for further assistance. The email address should be the one you originally registered with F1000. Email address not valid, please try again You registered with F1000 via Google, so we cannot reset your password. To sign in, please click here. If you still need help with your Google account password, please click here. You registered with F1000 via Facebook, so we cannot reset your password. To sign in, please click here. If you still need help with your Facebook account password, please click here. Code not correct, please try again Server error, please try again. If your email address is registered with us, we will email you instructions to reset your password. If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance. Please wait...