LLM Aspect Prediction: Reviewing Academic Papers from Different Aspects with Large Language Model

LLM Aspect Prediction: Reviewing Academic Papers from Different Aspects with Large Language Model | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article LLM Aspect Prediction: Reviewing Academic Papers from Different Aspects with Large Language Model Zihao Hu, Fumiyo Fukumoto, Dongjin Yu This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8120128/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Peer review is a vital process in scholarly publishing, where reviewers assess and score various aspects of a manuscript—such as novelty, clarity, and significance—based on defined evaluation criteria. This process demands substantial cognitive and time effort and remains prone to human bias and inconsistency. To address these challenges, we present LLMAspectPrediction, a framework that predicts fine-grained aspect scores of academic papers, assisting reviewers through consistent, guideline-informed assessments and offering authors actionable feedback aligned with peer review standards. The method comprises three stages. First, raw texts are organized to fit the input format. Meanwhile, a vector database enables retrieval of content-similar papers based on topic distribution using an additional corpus. Then, prompt templates grounded in peer review rubrics guide LLM-based evaluations of specific aspects. Finally, LLM-generated evaluations serve as weak supervision signals to fine-tune a pre-trained model for robust score prediction. Experiments demonstrate that our approach achieves state-of-the-art performance and that its components are essential for enhancing overall model effectiveness. Large Language Model Score prediction Peer review Pretrained Language Model Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8120128","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":557128322,"identity":"1d2c45dd-3b6d-4f02-bfda-51ad2792e725","order_by":0,"name":"Zihao Hu","email":"","orcid":"","institution":"Hangzhou Dianzi University","correspondingAuthor":false,"prefix":"","firstName":"Zihao","middleName":"","lastName":"Hu","suffix":""},{"id":557128325,"identity":"0e7ab7f6-8f44-4490-907d-945f231a698d","order_by":1,"name":"Fumiyo Fukumoto","email":"","orcid":"","institution":"University of Yamanashi","correspondingAuthor":false,"prefix":"","firstName":"Fumiyo","middleName":"","lastName":"Fukumoto","suffix":""},{"id":557128327,"identity":"2b023078-a55f-4777-a387-22d11ab9cad6","order_by":2,"name":"Dongjin Yu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAq0lEQVRIiWNgGAWjYHACA4OPDcxgBvFaDGeSrIWZlyQtBjeSNxTb7rBObGBv3ibBUHOHGC1pBca5Z9ITG3iOlUkwHHtGjJYcA+PctsOJDRI5ZhKMDYeJ1GIJ0iL/hhQtjGBbeIjUInnmWYFhb1u6cRtPWrFFwjEitPAdT95m8LPNWraf/fDGGx9qiNCicICBDRwfbCAigbAGBgb5BgbmB8QoHAWjYBSMghEMADyFOk4qjrXAAAAAAElFTkSuQmCC","orcid":"","institution":"Hangzhou Dianzi University","correspondingAuthor":true,"prefix":"","firstName":"Dongjin","middleName":"","lastName":"Yu","suffix":""}],"badges":[],"createdAt":"2025-11-15 07:23:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8120128/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8120128/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":97988914,"identity":"eb756df5-1ca9-408c-828a-137ead566a32","added_by":"auto","created_at":"2025-12-11 14:18:03","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4934,"visible":true,"origin":"","legend":"","description":"","filename":"8469d39f0e484041b345e7ecfb060518.json","url":"https://assets-eu.researchsquare.com/files/rs-8120128/v1/f2030bde3778d8ee5f8179da.json"},{"id":98423583,"identity":"3cb9596d-4aeb-4958-8855-4ca38af07210","added_by":"auto","created_at":"2025-12-17 16:32:23","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":508454,"visible":true,"origin":"","legend":"","description":"","filename":"ArticleTitle.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8120128/v1_covered_b221f9a1-3977-4dec-b22b-14582e3c2d16.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"LLM Aspect Prediction: Reviewing Academic Papers from Different Aspects with Large Language Model","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Large Language Model, Score prediction, Peer review, Pretrained Language Model","lastPublishedDoi":"10.21203/rs.3.rs-8120128/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8120128/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Peer review is a vital process in scholarly publishing, where reviewers assess and score various aspects of a manuscript—such as novelty, clarity, and significance—based on defined evaluation criteria. This process demands substantial cognitive and time effort and remains prone to human bias and inconsistency. To address these challenges, we present LLMAspectPrediction, a framework that predicts fine-grained aspect scores of academic papers, assisting reviewers through consistent, guideline-informed assessments and offering authors actionable feedback aligned with peer review standards. The method comprises three stages. First, raw texts are organized to fit the input format. Meanwhile, a vector database enables retrieval of content-similar papers based on topic distribution using an additional corpus. Then, prompt templates grounded in peer review rubrics guide LLM-based evaluations of specific aspects. Finally, LLM-generated evaluations serve as weak supervision signals to fine-tune a pre-trained model for robust score prediction. Experiments demonstrate that our approach achieves state-of-the-art performance and that its components are essential for enhancing overall model effectiveness.","manuscriptTitle":"LLM Aspect Prediction: Reviewing Academic Papers from Different Aspects with Large Language Model","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-11 14:17:58","doi":"10.21203/rs.3.rs-8120128/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"68e09cc2-e50e-43d4-a900-9d91985518b2","owner":[],"postedDate":"December 11th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-15T14:24:03+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-11 14:17:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8120128","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8120128","identity":"rs-8120128","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}