GPT vs Human Legal Texts Annotations: A Comparative Study with Privacy Policies

GPT vs Human Legal Texts Annotations: A Comparative Study with Privacy Policies | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article GPT vs Human Legal Texts Annotations: A Comparative Study with Privacy Policies David Cevallos-Salas, José Estrada-Jiménez, Danny S. Guamán, David Rodríguez, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5799153/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract High-quality corpora of annotated privacy policies are scarce, yet essential for training, testing, and evaluating accurate machine learning models. However, elaborating new corpora remains an error-prone and resource-intensive task, heavily reliant on highly specialized and hard-to-find human annotators. Recent advancements in Generative Pre-trained Transformers (GPTs) open the possibility of using them to annotate privacy policies with performance comparable to that of human annotators, thereby streamlining the process while reducing human resource demands. This paper presents a novel method for annotating privacy policies based on a codebook, a well-designed prompt, and the analysis of logarithmic probabilities (logprobs) of a GPT's output tokens during the annotation process. We validated our method using the GPT-4o model and the well-known, open, multi-class, and multi-label OPP-115 corpus, achieving performance comparable to 80% of human annotators in segment-level annotation and matching 90% of human annotators in a full-text level annotation. Furthermore, incorporating logprobs analysis allowed the method to match the performance of all human annotators in full-text level annotation, suggesting that context enhances the task. These findings demonstrate the potential of our method to automate annotations with performance similar to human annotators while significantly reducing resource demands. corpus annotation GPT privacy policy LLM Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5799153","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":400163203,"identity":"7cd39a56-0d05-4544-9b06-d50caf52e7de","order_by":0,"name":"David Cevallos-Salas","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABJUlEQVRIie2RsUrEMBjHUwq9JbZrOui9Qo5Clzv6LA2BdinONwhGDuoknQsHvoLgC6QE2iV6a8ceglOHPkAR0zsHxYg3CuZHIF9CfuT7JwAYDH8RrgY8VPaNqlHkHfftUxSLqXpFfXaCAj4pifXAf1HcRi54D6IL3FSMD6Owg+au7sB6Sdhs0+kUX2a42gIaYElYVebCDeVzioFMCYM11imYJ7GAgJOSESbOmLDDNguRlQvCUKZtDO9eD8p1WeyZgKOwHu97pbwpZd7rlZbySYk9pG6BjoqPoFLYdAvUKn77wqstposC7acsK4rkZYLiOg1ymGizuDuyGfp1NHc8WnXDqL7y9qlGw9XyvJgJ7Yt9tPdlpfqJ1eT8fP4b+ggGg8Hwf3kHMKdrngyG9JQAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0002-3098-3090","institution":"Escuela Politécnica Nacional","correspondingAuthor":true,"prefix":"","firstName":"David","middleName":"","lastName":"Cevallos-Salas","suffix":""},{"id":400163204,"identity":"7f464db7-6df8-45d2-90f1-e3f3c5f14c84","order_by":1,"name":"José Estrada-Jiménez","email":"","orcid":"https://orcid.org/0009-0001-3931-9532","institution":"Escuela Politécnica Nacional","correspondingAuthor":false,"prefix":"","firstName":"José","middleName":"","lastName":"Estrada-Jiménez","suffix":""},{"id":400163205,"identity":"57d3691b-0903-483f-a9ac-0445802ab873","order_by":2,"name":"Danny S. Guamán","email":"","orcid":"https://orcid.org/0000-0003-2794-3079","institution":"Escuela Politécnica Nacional","correspondingAuthor":false,"prefix":"","firstName":"Danny","middleName":"S.","lastName":"Guamán","suffix":""},{"id":400163206,"identity":"14fed063-1ef5-4987-8611-87c80a8db012","order_by":3,"name":"David Rodríguez","email":"","orcid":"https://orcid.org/0000-0002-0911-4608","institution":"Universidad Politécnica de Madrid","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Rodríguez","suffix":""},{"id":400163207,"identity":"f7282876-0134-4bc9-9edf-de05223f7e72","order_by":4,"name":"José M. del Álamo","email":"","orcid":"https://orcid.org/0000-0002-6513-0303","institution":"Universidad Politécnica de Madrid","correspondingAuthor":false,"prefix":"","firstName":"José","middleName":"M. del","lastName":"Álamo","suffix":""}],"badges":[],"createdAt":"2025-01-09 21:12:54","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-5799153/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5799153/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":73444329,"identity":"0b752ddc-7399-40b5-bc15-d4bdca565ff2","added_by":"auto","created_at":"2025-01-10 04:36:27","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1134412,"visible":true,"origin":"","legend":"","description":"","filename":"GPTvsHumanLegalTextsAnnotationsAComparativeStudywithPrivacyPolicies.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5799153/v1_covered_ef85ba2e-cc8a-4d8b-aca6-d09025b463d3.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eGPT vs Human Legal Texts Annotations: A Comparative Study with Privacy Policies\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[{"identity":"ad4428c9-b809-4d88-9180-4e4183b0492f","identifier":"10.13039/501100012511","name":"Escuela Politécnica Nacional","awardNumber":"Ph.D. programme","order_by":0}],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Escuela Politécnica Nacional","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"corpus, annotation, GPT, privacy policy, LLM","lastPublishedDoi":"10.21203/rs.3.rs-5799153/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5799153/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eHigh-quality corpora of annotated privacy policies are scarce, yet essential for training, testing, and evaluating accurate machine learning models. However, elaborating new corpora remains an error-prone and resource-intensive task, heavily reliant on highly specialized and hard-to-find human annotators. Recent advancements in Generative Pre-trained Transformers (GPTs) open the possibility of using them to annotate privacy policies with performance comparable to that of human annotators, thereby streamlining the process while reducing human resource demands. This paper presents a novel method for annotating privacy policies based on a codebook, a well-designed prompt, and the analysis of logarithmic probabilities (logprobs) of a GPT's output tokens during the annotation process. We validated our method using the GPT-4o model and the well-known, open, multi-class, and multi-label OPP-115 corpus, achieving performance comparable to 80% of human annotators in segment-level annotation and matching 90% of human annotators in a full-text level annotation. Furthermore, incorporating logprobs analysis allowed the method to match the performance of all human annotators in full-text level annotation, suggesting that context enhances the task. These findings demonstrate the potential of our method to automate annotations with performance similar to human annotators while significantly reducing resource demands.\u003c/p\u003e","manuscriptTitle":"GPT vs Human Legal Texts Annotations: A Comparative Study with Privacy Policies","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-10 04:28:20","doi":"10.21203/rs.3.rs-5799153/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"adca3468-b213-45fd-a99b-5273221c1fae","owner":[],"postedDate":"January 10th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-01-10T04:28:20+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-10 04:28:20","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5799153","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5799153","identity":"rs-5799153","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}