Reliability of LLM-based Citation Sentiment Analysis under Structural Pressure: A Holistic Pressure Index Approach for Scientometric Inference

Reliability of LLM-based Citation Sentiment Analysis under Structural Pressure: A Holistic Pressure Index Approach for Scientometric Inference | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Reliability of LLM-based Citation Sentiment Analysis under Structural Pressure: A Holistic Pressure Index Approach for Scientometric Inference Aristotelis Kampatzis, Antonis Sidiropoulos This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9462261/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The evaluation of Large Language Models (LLMs) in sentiment analysis tasks primarily relies on standard accuracy metrics, which do not account for how demanding the conditions are under which a prediction is generated. This study introduces the Holistic Pressure Index (HPI) as a measure for quantifying the structural and semantic pressure exerted on models during Citation Sentiment Analysis (CSA). Different families of architectures (encoder, encoder–decoder, and decoder-only) were employed, in combination with Logistic Regression for coefficient estimation and SHapley Additive exPlanations (SHAP) for interpretability at the token level. In addition, appropriate statistical tests, such as the Chi-square test and Fisher’s exact test, were applied to validate the significance of the observed differences. The results indicate that models exhibit variations in the range of pressure within which they maintain correct predictions, while decision/prediction stability does not necessarily correspond to deep semantic understanding. Consequently, HPI is proposed as a complementary diagnostic framework that characterizes a model’s robustness profile, without constituting evidence of its general capability. At the same time, it highlights that the reliability of LLM-based CSA is not uniformly distributed across citation contexts, thereby directly affecting the validity of scientometric conclusions. citation sentiment analysis holistic pressure index structural complexity large language models SHAP scientometrics Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9462261","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":632291592,"identity":"279bd338-d5fa-431b-b773-d7af5bea9322","order_by":0,"name":"Aristotelis Kampatzis","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAv0lEQVRIiWNgGAWjYFACxgbGBiDFz8xDqhbJZuK1gDQBCYMDxGrhn324+eOMmm15xsd5j0kw/LpHWIvEucQ2yQ3HbhebHeZLk2DsKybCmjOMbYwP2G4nbjvMYybB2JNAWIf8Gcbmjw/+3U7c3EysFoMzjA2SG9tuJ25gBmph+EGEFkOgwyRn9t0uljjMY2yR2ECEFrkz7I8/9ny7ncfff8bwxoc/RGiBAYjSxDbidUC1MPwhQcsoGAWjYBSMGAAAh8U8gW+qjG0AAAAASUVORK5CYII=","orcid":"","institution":"International Hellenic University (IHU)","correspondingAuthor":true,"prefix":"","firstName":"Aristotelis","middleName":"","lastName":"Kampatzis","suffix":""},{"id":632291593,"identity":"6bbfc869-6caf-478f-84e9-f81d84d4eb8e","order_by":1,"name":"Antonis Sidiropoulos","email":"","orcid":"","institution":"International Hellenic University (IHU)","correspondingAuthor":false,"prefix":"","firstName":"Antonis","middleName":"","lastName":"Sidiropoulos","suffix":""}],"badges":[],"createdAt":"2026-04-19 13:08:33","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9462261/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9462261/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108493793,"identity":"3a167807-8576-422c-a6ec-dfbba396e158","added_by":"auto","created_at":"2026-05-05 10:01:43","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2520434,"visible":true,"origin":"","legend":"","description":"","filename":"ReliabilityofLLMbasedCitationSentimentAnalysisunderStructuralPressure.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9462261/v1_covered_905eb7d6-8af6-4d16-8eda-0e87267093ee.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Reliability of LLM-based Citation Sentiment Analysis under Structural Pressure: A Holistic Pressure Index Approach for Scientometric Inference","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"citation sentiment analysis, holistic pressure index, structural complexity, large language models, SHAP, scientometrics","lastPublishedDoi":"10.21203/rs.3.rs-9462261/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9462261/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe evaluation of Large Language Models (LLMs) in sentiment analysis tasks primarily relies on standard accuracy metrics, which do not account for how demanding the conditions are under which a prediction is generated. This study introduces the Holistic Pressure Index (HPI) as a measure for quantifying the structural and semantic pressure exerted on models during Citation Sentiment Analysis (CSA). Different families of architectures (encoder, encoder\u0026ndash;decoder, and decoder-only) were employed, in combination with Logistic Regression for coefficient estimation and SHapley Additive exPlanations (SHAP) for interpretability at the token level. In addition, appropriate statistical tests, such as the Chi-square test and Fisher\u0026rsquo;s exact test, were applied to validate the significance of the observed differences. The results indicate that models exhibit variations in the range of pressure within which they maintain correct predictions, while decision/prediction stability does not necessarily correspond to deep semantic understanding. Consequently, HPI is proposed as a complementary diagnostic framework that characterizes a model\u0026rsquo;s robustness profile, without constituting evidence of its general capability. At the same time, it highlights that the reliability of LLM-based CSA is not uniformly distributed across citation contexts, thereby directly affecting the validity of scientometric conclusions.\u003c/p\u003e","manuscriptTitle":"Reliability of LLM-based Citation Sentiment Analysis under Structural Pressure: A Holistic Pressure Index Approach for Scientometric Inference","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-05 07:12:44","doi":"10.21203/rs.3.rs-9462261/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"7563315e-118d-40c2-9375-0e1e54b78bf8","owner":[],"postedDate":"May 5th, 2026","published":true,"recentEditorialEvents":[{"type":"editorInvitedReview","content":"","date":"2026-04-30T08:40:55+00:00","index":14,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-05T07:12:44+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-05 07:12:44","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9462261","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9462261","identity":"rs-9462261","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}