How LLMs Assess Public Speaking? Methodology of Explaining LLM Judgments through Linguistic Patterns and Rhetorical Criteria

How LLMs Assess Public Speaking? Methodology of Explaining LLM Judgments through Linguistic Patterns and Rhetorical Criteria | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article How LLMs Assess Public Speaking? Methodology of Explaining LLM Judgments through Linguistic Patterns and Rhetorical Criteria Alisa Barkar, Mathieu Chollet, Matthieu Labeau, Beatrice Biancardi, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7139734/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 13 You are reading this latest preprint version Abstract This study examines how Large Language Models, specifically GPT-4o-mini, evaluate public speaking performances based on textual transcripts. We collect a new annotation of speeches from the 3MT_French dataset and compare GPT-4o-mini annotations to those of an expert across both concrete rhetorical criteria and abstract subjective dimensions. In contrast to the expert, GPT-4o-mini exhibits limited cross-criterion integration and struggles with subjective judgments such as persuasiveness and creativity. Further, we utilise linguistic features in order to interpret provided annotations and demonstrated that GPT-4o-mini relies heavily on surface-level linguistic features and tends to prioritise structural and stylistic markers, while expert annotations reflect broader discourse-level understanding and persuasive intent. These findings highlight the limitations of using LLMs in high-level rhetorical evaluation and suggest the need for hybrid systems that combine model capabilities with theory-driven evaluation criteria. The annotated dataset and code are released to support future work in this direction: https://github.com/abarkar/ How-LLMs-Assess-Public-Speaking-/tree/main Large Language Models (LLMs) Public Speaking Assessment Explainability in NLP Textual Evaluation Criteria Persuasive Communication Lexical Feature Analysis Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 07 Oct, 2025 Reviews received at journal 07 Oct, 2025 Reviews received at journal 30 Sep, 2025 Reviews received at journal 18 Sep, 2025 Reviews received at journal 16 Sep, 2025 Reviewers agreed at journal 15 Sep, 2025 Reviewers agreed at journal 13 Sep, 2025 Reviewers agreed at journal 09 Sep, 2025 Reviewers agreed at journal 09 Sep, 2025 Reviewers invited by journal 08 Sep, 2025 Editor assigned by journal 20 Aug, 2025 Submission checks completed at journal 19 Jul, 2025 First submitted to journal 16 Jul, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7139734","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":503138955,"identity":"bad0d1fa-e118-443d-928a-4cdeebefd849","order_by":0,"name":"Alisa Barkar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABOUlEQVRIie2Rv0vDQBTH3+MgWU6zprQ0/8KVgD+gP/6VK4G4FBxcCg4mBNIl4Hqgf4Tg0jFyaJe4B+wiQiaHli4ZFD3SUoqJoJtIPsM7eHcf3vdxADU1fxEdPVVjAKqOBUBre6OtG2XIjoKCF+ZWQVGpFHWtELqrwDeKEZBgkcPcOoxkturlXWro8v4Gp/3TfSPGYFxWTIlhI4Ksc/0Y2lcj7tJG5LopJs5xaHL0k4oxSlFhJAoDbDLikrKUHjy9h4RpJujPXtmwJAbLN5ADYegrcsQ/6EApKYYXTFPB/AqFSfSaaspQ7EU2AR5TZhaKZBrwSqWjgjVbLHMETc4wch1qJsUuMxVs6Fcp7dnkZfk6nvcEPbmFvNtvGxP5kOL0nFmX8q5K2cT72ij+d1N/yK8e19TU1Px3PgH/JmMV80FF0AAAAABJRU5ErkJggg==","orcid":"","institution":"LTCI, Institut Polytechnique de Paris","correspondingAuthor":true,"prefix":"","firstName":"Alisa","middleName":"","lastName":"Barkar","suffix":""},{"id":503138957,"identity":"b40113d0-a575-4b83-ac92-f6e96d20ac8a","order_by":1,"name":"Mathieu Chollet","email":"","orcid":"","institution":"University of Glasgow","correspondingAuthor":false,"prefix":"","firstName":"Mathieu","middleName":"","lastName":"Chollet","suffix":""},{"id":503138958,"identity":"f49d05b4-850a-467d-ba26-e852caf97196","order_by":2,"name":"Matthieu Labeau","email":"","orcid":"","institution":"LTCI, Institut Polytechnique de Paris","correspondingAuthor":false,"prefix":"","firstName":"Matthieu","middleName":"","lastName":"Labeau","suffix":""},{"id":503138960,"identity":"83e323b3-94dc-4438-9266-cf97f24f999f","order_by":3,"name":"Beatrice Biancardi","email":"","orcid":"","institution":"CESI LINEACT, CESI","correspondingAuthor":false,"prefix":"","firstName":"Beatrice","middleName":"","lastName":"Biancardi","suffix":""},{"id":503138962,"identity":"026cbd35-72c0-4391-a008-7909d034ffad","order_by":4,"name":"Chloé Clavel","email":"","orcid":"","institution":"ALMAnaCH, INRIA","correspondingAuthor":false,"prefix":"","firstName":"Chloé","middleName":"","lastName":"Clavel","suffix":""}],"badges":[],"createdAt":"2025-07-16 11:53:25","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7139734/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7139734/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":90812453,"identity":"79fa75c8-65e7-402a-a22c-fb2590f2c494","added_by":"auto","created_at":"2025-09-08 12:21:04","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1314841,"visible":true,"origin":"","legend":"","description":"","filename":"LREwdeclarations.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7139734/v1_covered_f5443bb8-ffbd-466f-953b-a73868945e0d.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"\u003cp\u003eHow LLMs Assess Public Speaking? Methodology of Explaining LLM Judgments through Linguistic Patterns and Rhetorical Criteria\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"language-resources-and-evaluation","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"lrev","sideBox":"Learn more about [Language Resources and Evaluation](http://link.springer.com/journal/10579)","snPcode":"10579","submissionUrl":"https://submission.nature.com/new-submission/10579/3","title":"Language Resources and Evaluation","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Large Language Models (LLMs), Public Speaking Assessment, Explainability in NLP, Textual Evaluation Criteria, Persuasive Communication, Lexical Feature Analysis","lastPublishedDoi":"10.21203/rs.3.rs-7139734/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7139734/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"This study examines how Large Language Models, specifically GPT-4o-mini, evaluate public speaking performances based on textual transcripts. We collect a new annotation of speeches from the 3MT_French dataset and compare GPT-4o-mini annotations to those of an expert across both concrete rhetorical criteria and abstract subjective dimensions. In contrast to the expert, GPT-4o-mini exhibits limited cross-criterion integration and struggles with subjective judgments such as persuasiveness and creativity. Further, we utilise linguistic features in order to interpret provided annotations and demonstrated that GPT-4o-mini relies heavily on surface-level linguistic features and tends to prioritise structural and stylistic markers, while expert annotations reflect broader discourse-level understanding and persuasive intent. These findings highlight the limitations of using LLMs in high-level rhetorical evaluation and suggest the need for hybrid systems that combine model capabilities with theory-driven evaluation criteria. The annotated dataset and code are released to support future work in this direction: https://github.com/abarkar/ How-LLMs-Assess-Public-Speaking-/tree/main","manuscriptTitle":"How LLMs Assess Public Speaking? Methodology of Explaining LLM Judgments through Linguistic Patterns and Rhetorical Criteria","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-08 12:12:57","doi":"10.21203/rs.3.rs-7139734/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-10-07T09:01:28+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-07T08:15:46+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-09-30T15:43:05+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-09-18T16:25:24+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-09-16T12:31:47+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"15681541005541600570385204855509709761","date":"2025-09-15T20:01:55+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"186155549089298414054410243019010110994","date":"2025-09-13T16:49:30+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"127953483790096340165569252329906523594","date":"2025-09-09T19:03:11+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"287774028396075689721368444652363185896","date":"2025-09-09T06:17:00+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-09-08T15:53:31+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-08-20T13:01:59+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-07-19T12:29:15+00:00","index":"","fulltext":""},{"type":"submitted","content":"Language Resources and Evaluation","date":"2025-07-16T11:50:26+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"language-resources-and-evaluation","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"lrev","sideBox":"Learn more about [Language Resources and Evaluation](http://link.springer.com/journal/10579)","snPcode":"10579","submissionUrl":"https://submission.nature.com/new-submission/10579/3","title":"Language Resources and Evaluation","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"bfda5974-c493-4ab0-ad50-2bbff34f0dc7","owner":[],"postedDate":"September 8th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[],"tags":[],"updatedAt":"2026-01-27T11:11:06+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-08 12:12:57","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7139734","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7139734","identity":"rs-7139734","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}