IRIS: Iterative Improvement of Distantly Supervised Named Entity Annotations

IRIS: Iterative Improvement of Distantly Supervised Named Entity Annotations | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article IRIS: Iterative Improvement of Distantly Supervised Named Entity Annotations Ken Yano, Makoto Miwa, Sophia Ananiadou This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4818136/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background : Named entity recognition (NER) aims to detect entity mentions from text and classify them into predefined types. It is a fundamental task in information extraction and many other downstream tasks. However, the necessity of extensive human efforts to annotate a large amount of training data imposes restrictions on the state-of-the-art supervised deep learning methods, especially in specific domains. To address this problem, various distantly supervised methods (DS-NER), which aim to train NER models using automatically annotated training data based on external knowledge, such as dictionaries and knowledge bases, have been proposed. Although DS-NER methods are effective, the annotation quality tends to be noisy, which requires further manual checking and correction. Methods : We propose a named entity annotation framework called IRIS ( I teRative Improvement dS-ner).The IRIS framework is a client-server web application that aims to curate noisy named entity annotations made by DS-NERs initially annotated using only domain dictionaries. The tool provides a fast and efficient corpus search capability to correct annotation manually. Besides, a back-end offline process periodically launched to re-train the DS-NER by the updated annotations generates new annotations. The web tool provides a UI to compare current and new annotations, enabling curators to decide whether to replace old annotations with new ones. This curation process continues until the corpus quality converges. Results : We adopt a simulation-based method using the BC5CDR dataset to evaluate the framework. The results show the effectiveness of our proposed annotation framework in reducing manual annotation or curation workload compared with manual annotation from scratch to obtain annotation results of similar quality. Conclusions : Our proposed IRIS annotation framework introduces a novel approach, iteratively and semiautomatically correcting noisy entity annotations made by DS-NER. The results of the simulation-based experiments demonstrate that the IRIS framework significantly reduces human annotation efforts compared with naive manual annotation from scratch. distantly supervised NER named entity annotation automatic annotation Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4818136","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":338431024,"identity":"2973ded8-6f68-4bf5-b6c6-40b77ae2ea6f","order_by":0,"name":"Ken Yano","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA3UlEQVRIiWNgGAWjYNCCAwwM/AyMDQgeUVokG4BaDpCkxeAAsYoNjp89+IDhjE2e8fHDzZ8/MNTKMTCexa/T4ExesgHDjbRiszOJbRIHGI4bMzCcS8Cv5UCOmQTDh8OJ2w4ktgEddiyxgeGMAX4t59+AtPxP3Nz/sPkDcVpugGy5cSBxg0RiA9BhNYS1SN54Y2yQcCY5ccaNh20SZwwOGLMR8gvf+RzDBx+O2SX296c//lBRUSfHL0EgxBRA0ghTDQ4zsEmcwauDQb4BlV8HTDo9+LWMglEwCkbBiAMAD29TjBGDnk4AAAAASUVORK5CYII=","orcid":"","institution":"National Institute of Advanced Industrial Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Ken","middleName":"","lastName":"Yano","suffix":""},{"id":338431025,"identity":"78ccc62d-0fc3-4f10-8460-6e5a955b7fc3","order_by":1,"name":"Makoto Miwa","email":"","orcid":"","institution":"Toyota Technological Institute","correspondingAuthor":false,"prefix":"","firstName":"Makoto","middleName":"","lastName":"Miwa","suffix":""},{"id":338431026,"identity":"69031ddc-d16b-4856-ac75-763bf8983f61","order_by":2,"name":"Sophia Ananiadou","email":"","orcid":"","institution":"University of Manchester","correspondingAuthor":false,"prefix":"","firstName":"Sophia","middleName":"","lastName":"Ananiadou","suffix":""}],"badges":[],"createdAt":"2024-07-28 21:53:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4818136/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4818136/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":65358601,"identity":"fffee88b-259f-4b13-b6c4-d0bcf734dcde","added_by":"auto","created_at":"2024-09-26 12:47:24","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":4081245,"visible":true,"origin":"","legend":"","description":"","filename":"submit.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4818136/v1_covered_92ef6842-086a-414c-b424-481775d57dc8.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"IRIS: Iterative Improvement of Distantly Supervised Named Entity Annotations","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"distantly supervised NER, named entity annotation, automatic annotation","lastPublishedDoi":"10.21203/rs.3.rs-4818136/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4818136/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e: Named entity recognition (NER) aims to detect entity mentions from text and classify them into predefined types. It is a fundamental task in information extraction and many other downstream tasks. However, the necessity of extensive human efforts to annotate a large amount of training data imposes restrictions on the state-of-the-art supervised deep learning methods, especially in specific domains. To address this problem, various distantly supervised methods (DS-NER), which aim to train NER models using automatically annotated training data based on external knowledge, such as dictionaries and knowledge bases, have been proposed. Although DS-NER methods are effective, the annotation quality tends to be noisy, which requires further manual checking and correction.\u0026nbsp;\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e: We propose a named entity annotation framework called \u003cstrong\u003eIRIS \u003c/strong\u003e(\u003cstrong\u003eI\u003c/strong\u003eteRative Improvement dS-ner).The IRIS framework is a client-server web application that aims to curate noisy named entity annotations made by DS-NERs initially annotated using only domain dictionaries. The tool provides a fast and efficient corpus search capability to correct annotation manually. Besides, a back-end offline process periodically launched to re-train the DS-NER by the updated annotations generates new annotations. The web tool provides a UI to compare current and new annotations, enabling curators to decide whether to replace old annotations with new ones. This curation process continues until the corpus quality converges.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: We adopt a simulation-based method using the BC5CDR dataset to evaluate the framework. The results show the effectiveness of our proposed annotation framework in reducing manual annotation or curation workload compared with manual annotation from scratch to obtain annotation results of similar quality.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusions\u003c/strong\u003e: Our proposed IRIS annotation framework introduces a novel approach, iteratively and semiautomatically correcting noisy entity annotations made by DS-NER. The results of the simulation-based experiments demonstrate that the IRIS framework significantly reduces human annotation efforts compared with naive manual annotation from scratch.\u003c/p\u003e","manuscriptTitle":"IRIS: Iterative Improvement of Distantly Supervised Named Entity Annotations","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-23 02:15:51","doi":"10.21203/rs.3.rs-4818136/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"dd883a63-e92d-4d03-adb7-998747e4ce28","owner":[],"postedDate":"August 23rd, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-09-26T12:39:13+00:00","versionOfRecord":[],"versionCreatedAt":"2024-08-23 02:15:51","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4818136","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4818136","identity":"rs-4818136","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}