Automated Detection of Core Comments in Online UAV Discussion Forums

Automated Detection of Core Comments in Online UAV Discussion Forums | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Automated Detection of Core Comments in Online UAV Discussion Forums Doheon Han, Nuno Moniz, Jane Cleland-Huang, Nitesh Chawla This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4530754/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Unmanned aerial vehicles (UAVs) have become popular, and accordingly, diagnosing failure is essential but not easy due to its difficulty and complexity. Online UAV forums can help users diagnose the failures since they contain abundant information from diverse users. However, matching responses to user needs is not easy because forum comments are often unclear or contradictory, and it can be even harder when forums contain difficult content. Large Language Models have shown good achievement for this task, but they have limitations such as their inexplicableness, or inconsistent results. Therefore, we try a traditional machine-learning approach, text classification in a supervised manner, with human-annotated labels. This approach shows competitive performance with analyzable and consistent results, however, some issues such as imbalance and user subjectivity are identified. The subjectivity issue is more serious in difficult forums like UAV forums, meaning each user defines correctness or helpfulness with their standards. In this paper, we tackle determining core comments, i.e., answers or helpful comments, in online UAV discussion forums while dealing with the imbalance and subjectivity issues. With a thorough experimental evaluation, we confirm the effect of our strategy. Empirically, we discover how individual subjectivity may harm the training and evaluation process. In addition, we analyze the results in quantitative and qualitative ways to define helpfulness in the context of online forums. UAV Text Classification Answers Helpfulness Label Disagreement Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4530754","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":314380903,"identity":"da8631f6-f5bf-4d07-b4b2-8955a603cdab","order_by":0,"name":"Doheon Han","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAn0lEQVRIiWNgGAWjYDCCG0D8oYKBsYGBwYB4LYwzzpCqhZmzjRQtfLebnz1mnHdHtoG9eZsEUVok7xwzNy7c9sy4gedYGXFaDG4kmEnP3HY4sUEix4xYLenfpHnnALXIvyFaS46ZNG8DyBYeIrVI3sgpk5xx7LBxG09asQVRWvhupG+T+FBzWLaf/fDGG0RpgQM20pSPglEwCkbBKMALAD20MiNJ1puKAAAAAElFTkSuQmCC","orcid":"","institution":"University of Notre Dame","correspondingAuthor":true,"prefix":"","firstName":"Doheon","middleName":"","lastName":"Han","suffix":""},{"id":314380904,"identity":"db6c3de9-f9d7-4e38-afb7-fe5a8b713b58","order_by":1,"name":"Nuno Moniz","email":"","orcid":"","institution":"University of Notre Dame","correspondingAuthor":false,"prefix":"","firstName":"Nuno","middleName":"","lastName":"Moniz","suffix":""},{"id":314380906,"identity":"8685aa70-b2e4-4efd-b3da-e659232a7383","order_by":2,"name":"Jane Cleland-Huang","email":"","orcid":"","institution":"University of Notre Dame","correspondingAuthor":false,"prefix":"","firstName":"Jane","middleName":"","lastName":"Cleland-Huang","suffix":""},{"id":314380908,"identity":"d3d7826b-7728-4339-b7fd-6a74bb98d49c","order_by":3,"name":"Nitesh Chawla","email":"","orcid":"","institution":"University of Notre Dame","correspondingAuthor":false,"prefix":"","firstName":"Nitesh","middleName":"","lastName":"Chawla","suffix":""}],"badges":[],"createdAt":"2024-06-05 02:08:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4530754/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4530754/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":66266789,"identity":"a9c95966-eddf-49d4-ac82-fb5167dea1af","added_by":"auto","created_at":"2024-10-09 11:53:56","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1614717,"visible":true,"origin":"","legend":"","description":"","filename":"AutomatedDetectionofCoreCommentsinOnlineUAVDiscussionForumsDiscoverData.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4530754/v1_covered_0d52f933-eeae-4ab0-ad0e-239984beae7b.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Automated Detection of Core Comments in Online UAV Discussion Forums","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"UAV, Text Classification, Answers, Helpfulness, Label Disagreement","lastPublishedDoi":"10.21203/rs.3.rs-4530754/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4530754/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Unmanned aerial vehicles (UAVs) have become popular, and accordingly, diagnosing failure is essential but not easy due to its difficulty and complexity.\nOnline UAV forums can help users diagnose the failures since they contain abundant information from diverse users.\nHowever, matching responses to user needs is not easy because forum comments are often unclear or contradictory, and it can be even harder when forums contain difficult content.\nLarge Language Models have shown good achievement for this task, but they have limitations such as their inexplicableness, or inconsistent results.\nTherefore, we try a traditional machine-learning approach, text classification in a supervised manner, with human-annotated labels.\nThis approach shows competitive performance with analyzable and consistent results, however, some issues such as imbalance and user subjectivity are identified.\nThe subjectivity issue is more serious in difficult forums like UAV forums, meaning each user defines correctness or helpfulness with their standards.\nIn this paper, we tackle determining core comments, i.e., answers or helpful comments, in online UAV discussion forums while dealing with the imbalance and subjectivity issues.\nWith a thorough experimental evaluation, we confirm the effect of our strategy. \nEmpirically, we discover how individual subjectivity may harm the training and evaluation process.\nIn addition, we analyze the results in quantitative and qualitative ways to define helpfulness in the context of online forums.","manuscriptTitle":"Automated Detection of Core Comments in Online UAV Discussion Forums","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-20 03:28:45","doi":"10.21203/rs.3.rs-4530754/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"589dce96-9751-48a7-99a8-10c21c2a2ab8","owner":[],"postedDate":"June 20th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-10-09T11:53:20+00:00","versionOfRecord":[],"versionCreatedAt":"2024-06-20 03:28:45","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4530754","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4530754","identity":"rs-4530754","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}