Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models

Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models Shunshun Liu, Talon R. Booth, Yangfeng Ji, Wesley Reinhart, Prasanna V. Balachandran This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8289815/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Large language models (LLMs) have shown promise for scientific data extraction from publications, but rely on manual prompt refinement. We present an expert-grounded automatic prompt optimization framework that enhances LLM entity extraction reliability. Using high-entropy alloy lattice constant extraction as a testbed, we optimized prompts for Claude 3.5 Sonnet through feedback cycles on seven expert-annotated publications. Despite a modest optimization budget, recall improved from 0.27 to > 0.9, demonstrating that a small, expert-curated dataset can yield significant improvements. The approach was applied to extract lattice constants from 2,267 publications, yielding data for 1,861 compositions. The optimized prompt transferred effectively to newer models: Claude 4.5 Sonnet , GPT-5, and Gemini 2.5 Flash. Analysis revealed three categories of LLM mistakes: contextual hallucination, semantic misinterpretation, and unit conversion errors, emphasizing the need for validation protocols. These results establish feedback-guided prompt optimization as a low-cost, transferable methodology 1 for reliable scientific data extraction, providing a scalable pathway for complex LLM-assisted research tasks. Physical sciences/Engineering Physical sciences/Materials science Physical sciences/Mathematics and computing Large language models prompt optimization high-entropy alloys lattice constants scientific data extraction Full Text Additional Declarations No competing interests reported. Supplementary Files SupportingInformation.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8289815","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":560596486,"identity":"5ca20e31-e6d2-4048-94da-08fe1e5b20f8","order_by":0,"name":"Shunshun Liu","email":"","orcid":"","institution":"University of Virginia","correspondingAuthor":false,"prefix":"","firstName":"Shunshun","middleName":"","lastName":"Liu","suffix":""},{"id":560596487,"identity":"2709725a-5584-4a8a-934f-e90020de7abe","order_by":1,"name":"Talon R. Booth","email":"","orcid":"","institution":"University of Virginia","correspondingAuthor":false,"prefix":"","firstName":"Talon","middleName":"R.","lastName":"Booth","suffix":""},{"id":560596488,"identity":"db3dea19-9da1-46ff-97ef-d689569ee9ed","order_by":2,"name":"Yangfeng Ji","email":"","orcid":"","institution":"University of Virginia","correspondingAuthor":false,"prefix":"","firstName":"Yangfeng","middleName":"","lastName":"Ji","suffix":""},{"id":560596489,"identity":"bd8c045a-404a-4d2f-b77c-7138f3b0b52a","order_by":3,"name":"Wesley Reinhart","email":"","orcid":"","institution":"Pennsylvania State University, State College","correspondingAuthor":false,"prefix":"","firstName":"Wesley","middleName":"","lastName":"Reinhart","suffix":""},{"id":560596490,"identity":"d101b1e2-8030-44fc-9145-1eb21930f8c5","order_by":4,"name":"Prasanna V. Balachandran","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1UlEQVRIiWNgGAWjYFACHvYPDBUMPAwMjA0MCSCBA4S1sDEwnCFZC2MbsgAhLeYSuccefJx3WMa8f3Hbgwc1DHJ8NxLwa7GckZduOHPbYR6ZGw/bDRKOMRhLEtJicOaMgTTvtjQeCYmDbRIJbAyJG4jS8ncOTMs/hnrCWo73mEkzNtjwSPA3tkkktjEkGBD0S3tfsmHPMaAWCUaglj4Jw5lnHuDXYs7Me/DBjxoJewn+488kf3yzkec7TshhcJYEWKUEfuWoWvgPEFY9CkbBKBgFIxMAANJcRMZmdxeVAAAAAElFTkSuQmCC","orcid":"","institution":"University of Virginia","correspondingAuthor":true,"prefix":"","firstName":"Prasanna","middleName":"V.","lastName":"Balachandran","suffix":""}],"badges":[],"createdAt":"2025-12-05 17:08:22","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8289815/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8289815/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":98275618,"identity":"3c847321-ee41-45f4-93d3-721146c1fa96","added_by":"auto","created_at":"2025-12-16 03:36:24","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7428,"visible":true,"origin":"","legend":"","description":"","filename":"1fb357cb23d5473b836941b58412d9c0.json","url":"https://assets-eu.researchsquare.com/files/rs-8289815/v1/347f72c86dce91ad4f2cb854.json"},{"id":99789394,"identity":"f47ac87d-c3a9-4bcb-80b0-e0d0527a4161","added_by":"auto","created_at":"2026-01-08 12:49:34","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5326433,"visible":true,"origin":"","legend":"","description":"","filename":"MainManuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8289815/v1_covered_ba4265b4-852f-48c7-a72a-c94ea3aef9dc.pdf"},{"id":98275619,"identity":"95ce40c9-7476-40f4-a032-22180a78d5af","added_by":"auto","created_at":"2025-12-16 03:36:24","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":990035,"visible":true,"origin":"","legend":"","description":"","filename":"SupportingInformation.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8289815/v1/1549474b8df7bc25e4bbffdf.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Large language models, prompt optimization, high-entropy alloys, lattice constants, scientific data extraction","lastPublishedDoi":"10.21203/rs.3.rs-8289815/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8289815/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Large language models (LLMs) have shown promise for scientific data extraction from publications, but rely on manual prompt refinement. We present an expert-grounded automatic prompt optimization framework that enhances LLM entity extraction reliability. Using high-entropy alloy lattice constant extraction as a testbed, we optimized prompts for Claude 3.5 Sonnet through feedback cycles on seven expert-annotated publications. Despite a modest optimization budget, recall improved from 0.27 to \u003e 0.9, demonstrating that a small, expert-curated dataset can yield significant improvements. The approach was applied to extract lattice constants from 2,267 publications, yielding data for 1,861 compositions. The optimized prompt transferred effectively to newer models: Claude 4.5 Sonnet , GPT-5, and Gemini 2.5 Flash. Analysis revealed three categories of LLM mistakes: contextual hallucination, semantic misinterpretation, and unit conversion errors, emphasizing the need for validation protocols. These results establish feedback-guided prompt optimization as a low-cost, transferable methodology 1 for reliable scientific data extraction, providing a scalable pathway for complex LLM-assisted research tasks.","manuscriptTitle":"Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-16 03:36:19","doi":"10.21203/rs.3.rs-8289815/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6bdc4825-7a68-40ac-8ef4-f58762014271","owner":[],"postedDate":"December 16th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":59679343,"name":"Physical sciences/Engineering"},{"id":59679344,"name":"Physical sciences/Materials science"},{"id":59679345,"name":"Physical sciences/Mathematics and computing"}],"tags":[],"updatedAt":"2026-01-02T04:53:40+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-16 03:36:19","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8289815","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8289815","identity":"rs-8289815","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}