Inter-rater agreement and variability in visual reading and of [18F]Flutemetamol PET

Inter-rater agreement and variability in visual reading and of [18F]Flutemetamol PET | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Inter-rater agreement and variability in visual reading and of [18F]Flutemetamol PET Akinori Takenaka, Takashi Nihashi, Keita Sakurai, Keiji Notomi, and 11 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4490178/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 24 Sep, 2024 Read the published version in Annals of Nuclear Medicine → Version 1 posted 4 You are reading this latest preprint version Abstract Objective: The purpose of this study was to validate the concordance of visual ratings of [18F]flutemetamol amyloid positron emission tomography (PET) images and to investigate the correlation between the agreement of each rater and the Centiloid (CL) scale. Methods: A total of 192 participants, clinically classified as cognitively normal (CN) (n=59), mild cognitive impairment (MCI) (n=65), Alzheimer's disease (AD) (n=55), or non-AD dementia (n=13), participated in this study. Three experts conducted visual ratings of the amyloid PET images for all 192 patients, assigning a confidence level to each rating on a three-point scale (certain, probable, or neither). The positive or negative determination of amyloid PET results was made by majority vote. The CL value was calculated using the CapAIBL pipeline. Results: Overall, 101 images were determined to be positive, while 91 images were negative. Of the 101 positive results, the three raters were in complete agreement for 92 results and in disagreement for 9 results. Of the 91 negative results, the three raters were in complete agreement for 75 results and in disagreement for 16 results. Interrater reliability among the three experts was particularly high, with both Fleiss’ kappa and Conger’s kappa measuring 0.83 (0.76-0.89). The CL value was significantly greater for positive cases with complete agreement and significantly lower for negative cases with complete agreement. Cases with rater disagreement had intermediate CLs. In cases with a high confidence level, the visual ratings judged as positive or negative were in almost complete agreement. However, as confidence levels decreased, experts’ visual ratings became more variable. The lower the confidence level was, the greater the number of cases with disagreement in visual ratings. Conclusions: Three experts independently rated 192 amyloid PET images, achieving a high level of interrater agreement. However, in patients with intermediate amyloid accumulation, visual ratings varied. Therefore, caution should be taken in determining positive and negative decisions in these patients. Amyloid positron emission tomography visual rating Centiloid scale agreement Figures Figure 1 Figure 2 Figure 3 Full Text Additional Declarations Tables 1-4 are available in the Supplementary Files section. Supplementary Files Table1.docx Table2.docx Table3.docx Table4.docx Cite Share Download PDF Status: Published Journal Publication published 24 Sep, 2024 Read the published version in Annals of Nuclear Medicine → Version 1 posted Reviewers agreed at journal 29 May, 2024 Reviewers invited by journal 29 May, 2024 Editor assigned by journal 28 May, 2024 First submitted to journal 28 May, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4490178","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":308167708,"identity":"d404bf09-f501-40dd-9fb5-df4c264782a0","order_by":0,"name":"Akinori Takenaka","email":"","orcid":"","institution":"Fujita Health University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Akinori","middleName":"","lastName":"Takenaka","suffix":""},{"id":308167709,"identity":"ae7b24c3-94c7-403a-9eb2-441838414cbe","order_by":1,"name":"Takashi Nihashi","email":"","orcid":"","institution":"National Center for Geriatrics and Gerontology","correspondingAuthor":false,"prefix":"","firstName":"Takashi","middleName":"","lastName":"Nihashi","suffix":""},{"id":308167710,"identity":"1a1e30a5-7d21-42f0-ad2d-0bac59377fd5","order_by":2,"name":"Keita Sakurai","email":"","orcid":"","institution":"National Center for Geriatrics and Gerontology","correspondingAuthor":false,"prefix":"","firstName":"Keita","middleName":"","lastName":"Sakurai","suffix":""},{"id":308167711,"identity":"08b661b2-d640-49f5-85e3-1f72c3afea39","order_by":3,"name":"Keiji Notomi","email":"","orcid":"","institution":"Micron Inc. Imaging Service Dept.","correspondingAuthor":false,"prefix":"","firstName":"Keiji","middleName":"","lastName":"Notomi","suffix":""},{"id":308167712,"identity":"3ef53b2b-b39f-42e1-86ce-2d517cc1e1c1","order_by":4,"name":"Hokuto Ono","email":"","orcid":"","institution":"Micron Inc. Imaging Service Dept.","correspondingAuthor":false,"prefix":"","firstName":"Hokuto","middleName":"","lastName":"Ono","suffix":""},{"id":308167713,"identity":"5d2794e3-ba8c-4283-a3f5-da243e87f748","order_by":5,"name":"Yoshitaka Inui","email":"","orcid":"","institution":"Fujita Health University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Yoshitaka","middleName":"","lastName":"Inui","suffix":""},{"id":308167714,"identity":"2c0638a2-b9fd-4e75-802f-83c477ef6917","order_by":6,"name":"Shinji Ito","email":"","orcid":"","institution":"Anjo Kosei Hospital","correspondingAuthor":false,"prefix":"","firstName":"Shinji","middleName":"","lastName":"Ito","suffix":""},{"id":308167715,"identity":"43d527fb-a36c-4efa-8bb8-9bc9384615eb","order_by":7,"name":"Yutaka Arahata","email":"","orcid":"","institution":"National Center for Geriatrics and Gerontology","correspondingAuthor":false,"prefix":"","firstName":"Yutaka","middleName":"","lastName":"Arahata","suffix":""},{"id":308167716,"identity":"dfa0bd08-43ff-4826-a0bb-a1f84a4f781d","order_by":8,"name":"Akinori Takeda","email":"","orcid":"","institution":"National Center for Geriatrics and Gerontology","correspondingAuthor":false,"prefix":"","firstName":"Akinori","middleName":"","lastName":"Takeda","suffix":""},{"id":308167717,"identity":"bf38ccd6-4a69-48fa-a6d3-1df37ce24010","order_by":9,"name":"Kazunari Ishii","email":"","orcid":"","institution":"Kindai University Faculty of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Kazunari","middleName":"","lastName":"Ishii","suffix":""},{"id":308167718,"identity":"8f41550a-412d-49ce-89b3-7586ae0107da","order_by":10,"name":"Kenji Ishii","email":"","orcid":"","institution":"Tokyo Metropolitan Institute of Gerontology","correspondingAuthor":false,"prefix":"","firstName":"Kenji","middleName":"","lastName":"Ishii","suffix":""},{"id":308167719,"identity":"8ec6a647-db00-4811-8473-9f7c1502f2b2","order_by":11,"name":"Kengo Ito","email":"","orcid":"","institution":"National Center for Geriatrics and Gerontology","correspondingAuthor":false,"prefix":"","firstName":"Kengo","middleName":"","lastName":"Ito","suffix":""},{"id":308167720,"identity":"d80ae720-10f5-4131-98bf-41f39e09ea5d","order_by":12,"name":"Hiroshi Toyama","email":"","orcid":"","institution":"Fujita Health University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Hiroshi","middleName":"","lastName":"Toyama","suffix":""},{"id":308167721,"identity":"9c392e49-cd71-4600-bfdd-9807c0949541","order_by":13,"name":"Akinori Nakamura","email":"","orcid":"","institution":"National Center for Geriatrics and Gerontology","correspondingAuthor":false,"prefix":"","firstName":"Akinori","middleName":"","lastName":"Nakamura","suffix":""},{"id":308167722,"identity":"7b4a5a73-24a2-4a00-af23-6b6673b3e8f5","order_by":14,"name":"Takashi Kato","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAwklEQVRIiWNgGAWjYDACHhBRAeeyEaGDh4GxgeEMyVoY20hxlz3P4ecPPs67Iy/vv4D5xQcGvjzCtvC2GTbO3PbMcOONB2yWMxjYiglr4WcwbObddjjBcMYBNmMeBrbEBsJa2D82884hSQtvD9CWhsMJ8vwNzI+J03LmTOHMGccOG26QYGxjnGFAhF/Ye9I3fPhQc1hevv/w4Q8fKo4RDjE4MLiR2CbBYHAsgXgt8v0HmD8wMNSQoGUUjIJRMApGCgAAM509LvdyXH4AAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0002-4293-4333","institution":"National Center for Geriatrics and Gerontology","correspondingAuthor":true,"prefix":"","firstName":"Takashi","middleName":"","lastName":"Kato","suffix":""}],"badges":[],"createdAt":"2024-05-28 10:31:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4490178/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4490178/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s12149-024-01977-7","type":"published","date":"2024-09-24T15:57:30+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":58245831,"identity":"204e3946-8f79-4ddc-b753-1f273aedcf02","added_by":"auto","created_at":"2024-06-13 02:23:52","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":34479,"visible":true,"origin":"","legend":"\u003cp\u003eScatter plot of Centiloid values across four groups.\u003c/p\u003e\n\u003cp\u003eCentiloid levels were plotted for each group. (1) All rated negative (nnn). (2) One rated positive and two rated negative (nnp). (3) Two rated positive and one rated negative (npp). (4) All rated positive (ppp). One-way analysis of variance followed by the Bonferroni post hoc correction.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4490178/v1/a7a994efa8eea181511925f0.png"},{"id":58245826,"identity":"3fae474b-70d8-489a-b275-793fb107a2fe","added_by":"auto","created_at":"2024-06-13 02:23:52","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":54338,"visible":true,"origin":"","legend":"\u003cp\u003eROC analyses corresponding to inter-rater variabelity\u003c/p\u003e\n\u003cp\u003e(a) ROC analysis of the discriminative ability to distinguish between the all-negative group and others (one-positive, two-positive, all-positive groups). ROC analysis revealed an AUC of 0.97, a cutoff value of 18.26, and a sensitivity/specificity of 0.87/0.99. (b) ROC analysis of the discriminative ability to distinguish between the all-positive group and others (all negative, one-positive, two-positive groups). ROC analysis revealed an AUC of 0.99, a cutoff value of 29.87, and a sensitivity/specificity of 0.97/0.95.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4490178/v1/ef18c7b3204b08c34ce13578.png"},{"id":58246956,"identity":"31d16643-0ab0-46e2-b967-c469878fd68f","added_by":"auto","created_at":"2024-06-13 02:31:52","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":48799,"visible":true,"origin":"","legend":"\u003cp\u003eThe relationship between the sum of confidence scores, Centiloid values, and expert visual ratings.\u003c/p\u003e\n\u003cp\u003eThe confidence levels were discretized as follows: Neither: -1, Probable: 0, and Certain: +1. The sum of the scores determined by three interpreters was considered the confidence level for each case. For example, if all three raters gave a confidence level of +1 in a case, the total confidence level was +3 for that case. The vertical axis is the sum of confidence levels, the horizontal axis is the Centiloid value, and the relationship between confidence and the Centiloid scale is plotted. Circles indicate agreement cases, crosses indicate disagreement cases, red indicates amyloid PET positive, and blue indicates amyloid PET negative.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4490178/v1/469456afa076a9e7757ab795.png"},{"id":65627202,"identity":"07a59211-3ace-473d-a7b0-227b699b7faa","added_by":"auto","created_at":"2024-09-30 16:13:17","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":367659,"visible":true,"origin":"","legend":"","description":"","filename":"VisualAmyloidPETtakenaka0528v02.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4490178/v1_covered_ea330239-567c-455c-aeff-0b6c7c4c95e8.pdf"},{"id":58245827,"identity":"5014a4cb-0002-4274-8914-882c082aefbe","added_by":"auto","created_at":"2024-06-13 02:23:52","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":18528,"visible":true,"origin":"","legend":"","description":"","filename":"Table1.docx","url":"https://assets-eu.researchsquare.com/files/rs-4490178/v1/bcb564185708265cd739c34a.docx"},{"id":58245829,"identity":"d82e9724-626b-458a-80a7-ba5d0112a0f4","added_by":"auto","created_at":"2024-06-13 02:23:52","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":17842,"visible":true,"origin":"","legend":"","description":"","filename":"Table2.docx","url":"https://assets-eu.researchsquare.com/files/rs-4490178/v1/300cb3740d56e35971c13675.docx"},{"id":58245832,"identity":"ac70d372-cb73-4e29-8809-3fafbc74c408","added_by":"auto","created_at":"2024-06-13 02:23:52","extension":"docx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":18402,"visible":true,"origin":"","legend":"","description":"","filename":"Table3.docx","url":"https://assets-eu.researchsquare.com/files/rs-4490178/v1/de5586df62297f7f8a7ae782.docx"},{"id":58245830,"identity":"39090c13-0024-4f21-95ed-cf7edb5cffb7","added_by":"auto","created_at":"2024-06-13 02:23:52","extension":"docx","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":18891,"visible":true,"origin":"","legend":"","description":"","filename":"Table4.docx","url":"https://assets-eu.researchsquare.com/files/rs-4490178/v1/bd35e0939cd0502a0467fa3c.docx"}],"financialInterests":"\u003cp\u003eTables 1-4 are available in the Supplementary Files section.\u003c/p\u003e","formattedTitle":"Inter-rater agreement and variability in visual reading and of [18F]Flutemetamol PET","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"annals-of-nuclear-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"anme","sideBox":"Learn more about [Annals of Nuclear Medicine](http://link.springer.com/journal/12149)","snPcode":"12149","submissionUrl":"https://www.editorialmanager.com/anme/default2.aspx","title":"Annals of Nuclear Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Amyloid, positron emission tomography, visual rating, Centiloid scale, agreement","lastPublishedDoi":"10.21203/rs.3.rs-4490178/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4490178/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eObjective: The purpose of this study was to validate the concordance of visual ratings of [18F]flutemetamol amyloid positron emission tomography (PET) images and to investigate the correlation between the agreement of each rater and the Centiloid (CL) scale.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eMethods: A total of 192 participants, clinically classified as cognitively normal (CN) (n=59), mild cognitive impairment (MCI) (n=65), Alzheimer's disease (AD) (n=55), or non-AD dementia (n=13), participated in this study. Three experts conducted visual ratings of the amyloid PET images for all 192 patients, assigning a confidence level to each rating on a three-point scale (certain, probable, or neither). The positive or negative determination of amyloid PET results was made by majority vote. The CL value was calculated using the CapAIBL pipeline.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eResults: Overall, 101 images were determined to be positive, while 91 images were negative. Of the 101 positive results, the three raters were in complete agreement for 92 results and in disagreement for 9 results. Of the 91 negative results, the three raters were in complete agreement for 75 results and in disagreement for 16 results. Interrater reliability among the three experts was particularly high, with both Fleiss’ kappa and Conger’s kappa measuring 0.83 (0.76-0.89). The CL value was significantly greater for positive cases with complete agreement and significantly lower for negative cases with complete agreement. Cases with rater disagreement had intermediate CLs. In cases with a high confidence level, the visual ratings judged as positive or negative were in almost complete agreement. However, as confidence levels decreased, experts’ visual ratings became more variable. The lower the confidence level was, the greater the number of cases with disagreement in visual ratings.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConclusions: Three experts independently rated 192 amyloid PET images, achieving a high level of interrater agreement. However, in patients with intermediate amyloid accumulation, visual ratings varied. Therefore, caution should be taken in determining positive and negative decisions in these patients.\u003c/p\u003e","manuscriptTitle":"Inter-rater agreement and variability in visual reading and of [18F]Flutemetamol PET","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-13 02:23:47","doi":"10.21203/rs.3.rs-4490178/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"","date":"2024-05-30T03:03:31+00:00","index":0,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-05-29T10:57:54+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-05-29T00:02:05+00:00","index":"","fulltext":""},{"type":"submitted","content":"Annals of Nuclear Medicine","date":"2024-05-28T06:30:53+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"annals-of-nuclear-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"anme","sideBox":"Learn more about [Annals of Nuclear Medicine](http://link.springer.com/journal/12149)","snPcode":"12149","submissionUrl":"https://www.editorialmanager.com/anme/default2.aspx","title":"Annals of Nuclear Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"80e1537b-6312-4937-80ac-4eeee192df84","owner":[],"postedDate":"June 13th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-09-30T16:02:49+00:00","versionOfRecord":{"articleIdentity":"rs-4490178","link":"https://doi.org/10.1007/s12149-024-01977-7","journal":{"identity":"annals-of-nuclear-medicine","isVorOnly":false,"title":"Annals of Nuclear Medicine"},"publishedOn":"2024-09-24 15:57:30","publishedOnDateReadable":"September 24th, 2024"},"versionCreatedAt":"2024-06-13 02:23:47","video":"","vorDoi":"10.1007/s12149-024-01977-7","vorDoiUrl":"https://doi.org/10.1007/s12149-024-01977-7","workflowStages":[]},"version":"v1","identity":"rs-4490178","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4490178","identity":"rs-4490178","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}