Can Large Language Models Be Used to Code Text for Thematic Analysis? An Explorative Study

Can Large Language Models Be Used to Code Text for Thematic Analysis? An Explorative Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Can Large Language Models Be Used to Code Text for Thematic Analysis? An Explorative Study Zhiyong Han, Aaron Tavasi, JuYoung Lee, Joshua Luzuriaga, Kevin Suresh, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5937488/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 24 Jul, 2025 Read the published version in Discover Artificial Intelligence → Version 1 posted 9 You are reading this latest preprint version Abstract In practice, thematic analysis of text involves six stages, among which text coding is particularly cognitively demanding, labor-intensive, and time-consuming. This study investigates and compares the potential of two large language models (LLMs), namely ChatGPT-4 and OpenAI o1-preview, to perform text coding, with the goal of reducing the time and effort required by human researchers. Our results indicate that both models exhibit decreased coding comprehensiveness as document length increases, and both demonstrate low coding accuracy, primarily due to limitations in textual comprehension and reasoning. These findings highlight significant challenges in using LLMs to support thematic analysis, emphasizing the need for human oversight and rigorous validation to ensure analytic accuracy and validity. Large language model ChatGPT Text coding Thematic analysis OpenAI o1-preview Comprehension Reasoning Introduction Thematic analysis is a widely used qualitative method that enables researchers to identify and interpret patterns and themes within rich, textual data. In practice, the process is typically divided into six stages: familiarization with the data, text coding, development of a coding framework, theme identification, theme refinement, and reporting [ 1 – 5 ]. Throughout this process, researchers must pay meticulous attention to detail and engage in critical interpretation of thematic meaning—examining how words and phrases convey meaning, how that meaning may shift across contexts, and how meanings relate to one another within the structure of the text. This analytic rigor ensures that findings remain grounded in the data. When conducted carefully, thematic analysis can uncover underlying patterns and narratives shaped by human experiences, offering deep insights into participants’ perspectives and meanings [ 6 , 7 ]. Importantly, researchers must approach the analysis without preconceptions, as preconceived notions can lead to biased theme identification [ 8 , 9 ]. To preserve the integrity of the analysis, themes should emerge inductively from the data, guided by unbiased interpretation rather than predetermined expectations. Thematic analysis is cognitively demanding, time-consuming, and labor-intensive, particularly during the text familiarization and coding stages, because both are iterative in nature. Text familiarization requires investigators to immerse themselves deeply in the data, while coding involves systematically identifying and labeling segments of text with tags that summarize their core content. During the coding stage, researchers (also known as coders at this stage) must continually refine and revise their codes to fully reflect the meanings of text segments. To ensure reliability, multiple coders are required to minimize personal bias and misinterpretation of textual meanings, and following independent coding, the coders discuss and reconcile their codes [ 10 , 11 ]. Therefore, iterative coding demands constant comparison across the text to ensure consistent and accurate code application, and this is why it is cognitively challenging, time-consuming, and labor-intensive especially when the volume of the textual data is large and the content is complex. After code reconciliation, investigators cluster codes to facilitate theme extraction [ 7 , 10 , 11 , 12 ] This process, while still time-consuming and cognitively demanding, is typically less intensive than the initial text familiarization and coding phases. To enhance theme extraction, investigators often use techniques such as identifying patterns and relationships among codes; analyzing code frequencies and co-occurrences; examining deviant or negative cases that challenge emerging themes; and considering the broader contextual factors influencing the data [ 1 , 12 ]. By employing these strategies, researchers can systematically extract themes that accurately represent the underlying significance and perspectives within the textual data [ 13 ]. LLMs are powerful computing systems, based on natural language processing techniques and the transformer architecture, that have been trained on vast amounts of human-written text on almost any topics, enabling them to recognize patterns and structures in human writing through iterative training processes [ 14 , 15 ]. Unlike how humans learn and read languages, LLMs use a technique called tokenization during training to break text into small units called tokens. These tokens may represent entire words, sub-words, or even individual characters [ 16 – 19 ]. Over millions/billions of iterations, LLMs refine their ability to anticipate which tokens are statistically most likely to follow others in a context relevant fashion. This iterative process results in a rich statistical representation of language in terms of relationships between and among tokens in huamn written texts. However, this mathematical token-to-token processing of language is totally different from humans’ understanding of language, which is based on grammatical rules and meanings of words [ 20 , 21 ] The mathematical processing of language associated with tokenization of text also means that the trained LLMs do not understand the meaning of words, phrases, and sentences as humans do. Nevertheless, LLM’s extensive exposure to token patterns in human-written text covering almost any topics enables them to generate human-like coherent, contextually relevant, and fluent responses to user queries on any specific topic [ 22 , 23 ]. The impressive human-like language skills of LLMs have sparked intense interest and investigations about how LLMs can assist, if not replace, humans to complete tasks more quickly and effectively in various fields, such as computer coding, education, disease diagnosis, journalism, and law. One potential application of LLMs like ChatGPT is thematic analysis. Some recent research suggests that LLMs possess several advantages that make them suited for this task. Specifically, ChatGPT have an “exceptional ability to understand and generate human-like text” [ 24 ], can standardize and streamline the coding process, increasing efficiency and consistency [ 25 ], and can enhance code diversity [ 26 ]. Furthermore, ChatGPT offers additional benefits, including rapid data processing, improved work efficiency, concise summaries, and preliminary insights [ 26 ]. In one study [ 27 ], researchers compared the ability of ChatGPT to that of humans in generating themes from interviews, and they found similarities and differences between human coders and ChatGPT, and found that human-centered coding was superior because human coders brought a depth of analysis, sensitivity to nuances, and interpretive flexibility that ChatGPT lacked [ 27 ]. Therefore, they concluded that ChatGPT is a powerful tool to supplement complex human-centered tasks. However, it is important to note that this study only prompted ChatGPT to analyze statements and then generate themes without performing other steps of thematic analysis. Regardless, it is suggested that incorporating generative-AI such as ChatGPT in qualitative analysis allows analyzing large volumes of textual data quickly and comprehensively by offering efficient and speedy data processing, and discerning insights and themes in the data that may not be immediately apparent to human coders [ 27 ]. In practice, it is found that the outcome of using ChatGPT to conduct qualitative analysis depends on well-designed prompts [ 24 ], which is a critical issue that limits effective use of ChatGPT by investigators in qualitative analysis tasks, especially among those who lack expertise and experience with ChatGPT [ 24 , 28 , 29 ]. Nevertheless, findings from several studies suggest that ChatGPT has the promise as a tool for enhancing the efficiency and consistency of data coding [ 25 , 30 ]. Therefore, ChatGPT may have the potential to be a coding assistant for human coders, and the involvement of ChatGPT is likely to reduce coding time and increase coding accuracy as well as uncover information overlooked by human coders. However, it should be acknowledged that although LLMs can process and analyze textual data and generate impressive outputs that appear to grasp the literal meanings of words and sentences, which, while crucial, is insufficient on its own when it comes to text coding because to fully understand what a writer or speaker intends to convey coders must go beyond semantics and engage with pragmatic dimensions such as implicature, where meaning is implied rather than explicitly stated, and presupposition, all of which involves recognizing background assumptions embedded in language. It is also important to note that deciphering meanings in text also involves identifying speech acts - whether a sentence functions as a question, a request, or a warning - especially when expressed indirectly. In addition, coders must track discourse coherence, resolve referential ambiguity, and apply cultural knowledge to interpret idioms, humor, and context-specific expressions. These layers of interpretation highlight the complex interplay between language, context, and human intention. Consequently, LLMs’ “understanding” of language via tokenization of text means that they may struggle to accurately interpret or code texts that rely on context-bound features of language use, such as pragmatic meaning [ 31 ]. Additionally, it remains to be investigated is LLMs can recognize context-sensitive directives [ 32 ], especially when cultural nuance is central to interpretation [ 33 ]. When considering the use of LLMs to support thematic analysis, it is therefore essential to evaluate their capacity to interpret linguistic functions that extend beyond semantic understanding. Another important issue is that the question of whether LLMs can genuinely comprehend and reason remains a subject of active debate [ 20 , 21 ]. Investigating this question is complicated by the fact that the inner workings of LLMs are effectively a “black box.” Due to the enormous complexity and sheer number of parameters involved, there are currently no well-established methods for users—or even developers—to systematically dissect and understand how these models process input and generate responses. As a result, it remains difficult to assess what kind of representations or reasoning processes, if any, underlie the outputs of these large-scale statistical models. Despite these challenges, numerous studies have explored the reasoning capacities of LLMs. Webb and colleagues suggest that the ability of models like GPT-3 to solve a wide range of problems may be attributed to their analogical reasoning capabilities [ 34 ]. Conversely, others contend that these models lack the robustness and general characteristics of true analogical reasoning [ 35 ]. Alternative approaches have been used to test the reasoning abilities of LLMs. For example, Mirzadeh and colleagues [ 36 ] examining the limitations of mathematical reasoning in LLMs have observed that their performance tends to worsen as the complexity of a question increases, particularly when more clauses are involved. According to their analysis, the models are not genuinely capable of reasoning; instead, they reproduce the reasoning sequences they encountered during training. As a result, what may appear to be logical reasoning in LLMs can be more accurately described as a pattern-based simulation rather than a true deductive process. In another study, Han et al tested whether LLMs use reason to solve word ladder puzzles, and they found no evidence of reasoning [ 37 ]. In a recent review of the studies related to reasoning capabilities of LLMs [ 38 ], Sullivan and Elsayed reviewed the literature to assess whether LLMs can perform symbolic reasoning, and they reached a conclusion that “the probabilistic architecture of current LLMs, is not able to inherently perform symbolic reasoning, requiring external impetus to move in this direction.” Therefore, LLMs at their current states have serious limitations in using reasoning in different contexts. Given that thematic analysis, particularly text coding and theme identification, requires deep comprehension, including the ability to isolate relevant segments based on semantic and pragmatic meanings, presupposition, and the use of reasoning to iteratively develop appropriate codes, the apparent disparity between LLMs’ limitations in comprehension and their application to cognitively demanding tasks raises important questions. Even for humans, interpreting pragmatic meaning in text can be challenging; for LLMs, which lack true contextual understanding, this difficulty is likely magnified [ 31 ]. Nevertheless, studies [ 25 , 30 , 39 ] have suggested that LLMs hold potential in performing thematic analysis. This promise may stem from their advanced pattern recognition and ability to replicate statistical regularities learned from vast training data, enabling them to produce outputs that functionally mimic certain aspects of human coding. This paradox highlights the complex relationship between comprehension, reasoning, and statistical modeling in LLMs. While they may not genuinely “understand” text, their probabilistic mechanisms can at times yield analyses that appear meaningful. However, caution is warranted: without true comprehension, LLMs risk generating outputs that are coherent yet shallow, or that overlook subtle pragmatic nuances essential to robust qualitative interpretation. Nevertheless, by systematically prompting LLMs to code diverse types of text, especially text that was not included in the training data, and rigorously evaluating their performance - particularly in terms of coding accuracy - researchers can gain insights into the strengths and weaknesses of these models in thematic analysis. Such an approach not only assesses their current capabilities but also identifies areas for improvement, ultimately contributing to the enhancement of LLM performance and reliability in qualitative research methodologies. This study investigates the potential of two large language models (LLMs), ChatGPT-4 and OpenAI o1-preview, to perform a specific task of thematic analysis - namely, text coding. We selected these two models because both are based on OpenAI’s GPT architecture, with o1-preview representing a more advanced version that incorporates an additional chain-of-thought reasoning mechanism [ 40 ]. Our aim was to evaluate each model’s suitability for text coding and to assess whether the inclusion of chain-of-thought reasoning would make o1-preview the superior model. To ensure novelty and minimize the likelihood that the models had encountered the data during training, we used documents unlikely to have appeared in their training sets. Specifically, we compared the comprehensiveness and accuracy of the codes each model generated. Our findings show that both ChatGPT-4 and o1-preview struggled with coding - particularly for longer texts - and that many of their generated codes were inaccurate or required substantial revision. These results suggest that, in their current forms, both models lack the comprehension and reasoning capabilities necessary to reliably and accurately code texts for thematic analysis. Significant improvements will be required before such models can be used independently in qualitative research without compromising analytical quality. Materials and Methods Large Language Models: The LLMs used in this study were ChatGPT4 and OpenAI 01-preview by OpenAI. Both models were available via subscription. ChatGPT4 was released on March 14, 2023, and OpenAI o1-preview on September 12, 2024. Unlike ChatGPT4 and other ChatGPT models, OpenAI o1-preview is "designed to spend more time thinking before they respond. They can reason through complex tasks and solve harder problems than previous models in science, coding, and math" as stated by OpenAI ( https://openai.com/index/introducing-openai-o1-preview/ ). Documents preparation: To mitigate the risk that any of our documents might already exist in the training data of the LLMs, thereby rendering the outcomes of our analysis potentially predetermined and unreliable, we used one original document created by our team and several published documents that became available only after September 12, 2024, the release date of OpenAI o1-preview model. Document 1. Between 2020 and 2023, we offered a student elective titled COVID-19 and Lessons for Pandemic Drug Research . We received a total of 13 student comments about the course. These comments were de-identified and initially used for course improvement purposes. For this study, the de-identified comments were compiled into a single Word document totaling 1,149 words, with each comment presented as a separate paragraph. The use of these de-identified comments in this study was determined to be exempt from IRB requirements. Document 2. This document was derived from an opinion article titled Rare Disease Innovation at the FDA — Opportunities for Implementation by Shore et al., published online in JAMA on October 2, 2024 [ 41 ]. The article was converted into a Word document, with all citation numbers removed. The final document contained 1,185 words. Document 3. This document was derived from a research article titled Rates of Sudden Unexpected Infant Death Before and During the COVID-19 Pandemic by Guare et al., published in JAMA Network Open on September 26, 2024 [ 42 ]. The article was converted into a Word document, with citation numbers and references to figures and tables removed. The final word count was 3,188 words. Document 4. This document was a shortened version of Document 3, created by removing the introduction and part of the methods section. The resulting document was 2,254 words long. Text coding of students’ comments by LLMs: LLMs were prompted to perform unbiased inductive thematic coding of the students’ comments in Document 1 using the following prompts. I am conducting an open-ended thematic analysis of a document and require your assistance. Your task is to analyze the content of the document to identify recurring topics, concerns, insights, or other relevant patterns. The approach should be inductive and unbiased, focusing on coding across various dimensions. Please focus on text coding and do not generate overarching themes at this stage. If a comment contains ambiguous or unclear content, please flag it for further review. Please ensure that each code is supported with textual evidence from the document to maintain transparency and verifiability in the coding process. Coding Density After text coding had been completed by LLMs, coding density was estimated as the percentage of coded words (i.e., the number of words in the coded text segments) over the total number of words in the entire document. This was computed using the formula: Coding Density (%) = (Number of coded words / Total number of words of the text) x 100. Coding accuracy assessment by human coders All investigators in this study served as human coders. Each investigator critically reviewed and familiarized themselves with the student comments. Independently, they assessed the accuracy of the LLM-generated codes by comparing them to the original text. Following individual assessments, the investigators met to discuss their findings and reach consensus through collaborative review. Results Comparative Analysis of Text Coding Abilities: ChatGPT4 vs. OpenAI o1-mini In our first investigation, we investigated the text coding capabilities of ChatGPT4 and OpenAI o1-preview using a 1149-word long document we compiled with 13 students' comments about an elective course. In the document, the comments were presented as separate paragraphs. We prompted the models to code the document in three independent tests and evaluated their performance using two metrics: a) total number of text segments identified and coded, and b) coding density (percentage of coded words over total document words) as defined in the methods. The results (Table 1 ) revealed significant differences in coding capabilities between the two models. ChatGPT4 consistently identified and coded fewer text segments compared to OpenAI o1-preview. Additionally, ChatGPT4's coding density was substantially lower (19.58–25.50%) than that of OpenAI o1-preview (51.74–80.85%). These findings indicate that ChatGPT4 significantly under-coded the document. Table 1 Coding Capabilities Comparison: ChatGPT4 versus OpenAI o1-preview. ChatGPT4 OpenAI o1-preview Test 1 Test 2 Test 3 Test 1 Test 2 Test 3 No. of Text Segments Coded 12 13 9 28 30 58 Coding Density 25.50% 24.63% 19.58% 66.06% 51.74% 80.85% Mean Coding Density 23.23% (SD: 2.61%) 66.22% (SD:11.90%) A single document containing thirteen students' comments (totaling 1149 words) was fed to ChatGPT4 and OpenAI o1-preiew for coding. Coding capacity was determined by (a) number of text segments identified and coded and (b) coding density estimated as the percentage of the number of coded words over the total number of words in the entire document. SD: standard deviation Document Lengths and Coding Density: ChatGPT4 versus OpenAI o1-preview To understand whether document length might have caused under-coding by ChatGPT4, we altered the length of the document byaltering the total number of student comments between two shorter documents. We then prompted ChatGPT4 and OpenAI o1-preview to code the full-length document and the two shorter documents. The results in Table 2 show that as the document became shorter, the coding density became larger for both ChatGPT4 and OpenAI o1-preview, suggesting that coding density was inversely correlated with document length. Although it was shown that ChatGPT-4 could not code a large dataset at once [ 43 ], to the best of our knowledge, no previous study has analyzed how LLM coding density varies with document length, making our findings a novel contribution to the literature on the capabilities and limitations of LLMs in qualitative research. Table 2 Coding density is inversely related to document length. Document length by word number ChatGPT4 (coding density%) OpenAI o1-preview (coding density %) Test 1 Test 2 Test 3 Mean Test 1 Test 2 Test 3 Mean 1149 (comments 1–13) 14.52 17.11 30.84 20.82 (SD: 7.16) 66.06 51.74 80.85 66.22 (SD: 11.88) 488 (comments 1–5) 33.19 35.24 40.98 36.47 (SD: 3.30) 84.56 85.60 91.15 87.10 (SD: 1.04) 141 (comment 1) 93.83 94.52 84.24 90.86 (SD: 4.69) 104.11* 104.80* 95.89 101.60 (SD: 4.05) Three documents of different lengths were produced by using different numbers of student's comments. Then, ChatGPT4 and OpenAI o1-preview were prompted to code each document in three independent tests. Coding density was estimated as the percentage of the number of coded words over the total number of words of the entire document. *The coding densities exceeded 100% due to overlapping coded text segments. This overlap caused the total word count of the coded segments to surpass the overall word count of the paragraph. SD: Standard deviation. To investigate this issue further, we used three additional documents ranging in lengths from 1185-word long to 3188-word long and prompted ChatGPT4 and OpenAI o1-preview to code them. The results in Table 3 show that as the document length increased, the coding density of both models decreased. Table 3 An inverse relationship between document length and coding density. Document length by word number ChatGPT4 (coding density %) OpenAI o1-preview (coding density%) Test 1 Test 2 Test 3 Mean Test 1 Test 2 Test 3 Mean 1185 Words 10.88 18.39 15.61 14.96 (SD: 3.10) 45.57 67.51 79.07 64.05 (SD: 13.90) 2251 Words 5.64 6.17 13.19 8.33 (SD: 3.44) 32.27 44.38 50.19 42.28 (SD: 7.46) 3188 Words 5.23 3.98 2.69 3.97 (SD: 1.04) 29.29 31.65 29.51 30.15 (SD: 1.06) ChatGPT4 and OpenAI o1-preview were prompted to code two published articles of different lengths: one of 1185-words long and the other 3188-words long. Coding density was estimated as the percentage of the number of coded words over the total number of words in the entire document. SD: Standard deviation Interestingly, regardless of document length, the coding density of each document by ChatGPT4 was substantially smaller than that by OpenAI o1-preview (Table 3 ). These results and the results in Table 2 indicate that coding density of both models was inversely related to document length (Table 3 ). Coding Consistency as Measured by Coding Density Across the Text: ChatGPT4 versus OpenAI o1-preview One explanation for low coding density is that coding is not consistent throughout the text - that is, some sections receive more attention and therefore more codes, and other sections receive less or no coding. In such a case, determination of coding density of a document in a paragraph-by-paragraph way should be accurately captured if coding is applied unevenly across different sections of the document. To investigate this possibility, we reanalyzed the coding of the student comment document, in which each comment was presented as a separate paragraph and determined the coding density in each paragraph. The results in Table 4 showed that ChatGPT4 mainly focused its coding on the first and second paragraphs and skipped numerous paragraphs in three independent tests (Table 4 ). OpenAI o1-preview did not skip any paragraphs, but its coding density varied from paragraph to paragraph in tests 1 and 2 whereas it was consistently high in most paragraphs in test 3 (Table 4 ). These results indicated inconsistency in coding density across the text and section skipping that varied from test to test. Table 4 Coding density in different sections of the text by ChatGPT4 and OpenAI o1-preview. Document Section Coding Density (%) of ChatGPT4 Coding Density (%) of OpenAI o1-preview Test 1 Test 2 Test 3 Test 1 Test 2 Test 3 Paragraph 1 61.22 89.11 39.45 85.03 82.31 102.71* Paragraph 2 25.34 44.52 15.75 93.15 75.34 71.23 Paragraph 3 40.00 52.00 0.00 102.67* 49.33 74.67 Paragraph 4 0.00 30.67 0.00 60.00 24.00 101.33* Paragraph 5 0.00 0.00 0.00 83.33 85.41 79.16 Paragraph 6 27.63 30.26 30.26 57.89 53.95 88.16 Paragraph 7 23.33 0.00 0.00 19.17 20.00 75.00 Paragraph 8 0.00 0.00 46.42 80.36 35.71 51.79 Paragraph 9 23.71 0.00 23.71 65.98 48.45 60.82 Paragraph 10 0.00 0.00 44.07 101.69 57.63 91.53 Paragraph 11 17.21 0.00 0.00 64.75 50.00 99.18 Paragraph 12 37.97 0.00 25.31 25.31 40.50 96.20 Paragraph 13 40.32 24.19 40.32 101.61* 58.06 103.22* A document with thirteen paragraphs totaling 1149 words was fed to ChatGPT4 and OpenAI o1-preview for coding in three independent tests. The table lists the coding density of each of the paragraphs. *The coding densities exceeded 100% due to overlapping coded text segments. This overlap caused the total word count of the coded segments to surpass the overall word count of the paragraph. Coding Accuracy: A Comparative Analysis of ChatGPT4 and OpenAI o1-preview We defined coding accuracy as the degree to which codes accurately capture and represent the text data's meaning. To evaluate the coding accuracy of ChatGPT4 and OpenAI o1-preview, ChatGPT4 and OpenAI o1-preview were each prompted to code a document in three separate tests. We then reviewed each code and the correspondent text segment to determine whether the code was acceptable or inaccurate enough to require revision according to the true meaning of the actual text segment. Table 5 summarizes the results of our review of the codes generated by ChatGPT4 and OpenAI o1-preview. Both models exhibited consistent low coding accuracy across three tests even though the coding accuracy of OpenAI o1-preview was higher than that of ChatGPT4 (Table 5 ). Table 5 Coding Accuracy. ChatGPT4 OpenAI o1-preview Test 1 Test 2 Test 3 Test 1 Test 2 Test 3 Total No. of Codes Generated 12 13 9 34 36 47 No. of Codes Needed Revision 10 10 8 21 25 26 No. of Acceptable Codes 2 3 1 13 11 21 Coding Accuracy 16.67% 23.07% 11.11% 38.23% 30.55% 44.68% This table presents the results of coding reliability assessment, where codes assigned by ChatGPT1 and OpenAI o1-preview in three independent tests (see Table 1 ) were reviewed by human evaluators to determine the accuracy of each code in capturing the essence of the corresponding text segment. Table 6 presents illustrative examples of coding inaccuracies by ChatGPT-4 and OpenAI o1-preview model. For instance, the text segment “ there should be more training which teaches important figures influencing healthcare decisions how to critically evaluate research articles ” emphasizes the need for additional training. However, o1-preview coded it as “ Need for training in critical evaluation ,” overlooking the significance of the word “more” and thereby implying the absence of prior training. In another example, ChatGPT-4 coded the sentence “ Political motivations may also have played a factor and thus, as a result, much harm was produced ” as “ Political influence on health decisions .” This code is inaccurate because it shows that ChatGPT4 failed to distinguish the different meanings between "influence" and "motivation." Similarly, the statement “ underscore the critical importance of maintaining rigorous standards for drug approval, irrespective of external pressures ” was coded by OpenAI o1-preview as “ Need for rigorous standards in drug approval .” This simplification omits the emphasis on maintaining existing standards and disregards the contextual qualifier “ irrespective of external pressures .” Additionally, the phrase “ Later studies were conducted which disproved the efficacy of HCQ in treating COVID and even showed alarming side effects such as cardiac arrest and ventricular arrhythmia ” was coded by OpenAI o1-preview as “ Discovery of HCQ’s Inefficacy and Side Effects .” This code fails to reflect the critical role of later studies in overturning earlier claims, thereby missing an important temporal and evidentiary nuance. Collectively, these examples demonstrate that both ChatGPT-4 and OpenAI o1-preview frequently overlook contextual cues (e.g., “more”, “later studies”), miss semantic nuances (e.g., the importance of “maintaining” standards or differentiating “motivation” from “influence”), and tend to rely on superficial word pattern recognition rather than genuine comprehension of meaning. Table 6 Examples of codes by ChatGPT4 and OpenAI o1-preview and code evaluation and revision by human coders. [Model] [Assigned Code] /Corresponding text segment: [ChatGPT4] [Mismanagement of Research & Approval Process] / "due to the low cost of the drug [hydroxychloroquine], cases reports, and poorly designed pilot studies, the promise of the drug with respect to its success in treating COVID was blown out of proportion..." Evaluation: The essence of the text segment highlights the issue with the claim about hydroxychloroquine's effectiveness in treating COVID-19 with poor evidence. Therefore, "Mismanagement of Research & Approval Process" is an inaccurate code. New code: Exaggerated claim of drug efficacy based on Questionable Evidence [Model] [Assigned Code] /Corresponding text segment: [ChatGPT4] [Premature Implementation] / "the above scenario resulted unfortunately due to an overzealousness to arrive at a solution to the COVID pandemic before conducting comprehensive studies to properly determine the benefit of the drug" Evaluation: "Premature Implementation" doesn't fully capture the essence of the original text, because the text refers to rushed decision to use a drug for COVID-19 without thorough research and certainty about the drug's benefits. New code: Drug Use Without Comprehensive Studies [Model] [Assigned Code] /Corresponding text segment: [ChatGPT4] [Course Content Overview] \ “The elective highlighted the fundamental differences and values of observational studies, retrospective studies, and RTCs.” Evaluation: "Course Content Overview" suggests a broad summary of all topics covered in the course. But this is not what the text says. Rather, the text emphasizes that the course highlighted the fundamental differences and values of various research methods. It specifically mentions observational studies, retrospective studies, and RCTs. Therefore, the course aimed to educate students about different research designs and their respective roles and importance in scientific research. New code: Education in Research Methodologies [Model] [Assigned Code] /Corresponding text segment : [ChatGPT4] [Researcher Challenges] \ “and how researchers are not spared from becoming reactionary in an exceedingly challenging time.” Evaluation : The text suggests that researchers are not immune to reacting impulsively or emotionally, rather than critically thinking in a time that is extremely challenging. The text comments on the behavioral tendencies of researchers during crises and how even experts can fall victim to emotional or impulsive reactions during difficult times, rather than approaching issues with a critical and nuanced perspective. In contrast, the code "Researcher Challenges" misses the nuance of the text and is inaccurate. New code : Researchers Are Not Immune from Behaving Reactionarily During Crises [Model] [Assigned Code] /Corresponding text segment : [ChatGPT4] [Importance of Quality Research] \ "It is very important to understand what goes into quality research." Evaluation : When a person says, "It is very important to understand what goes into quality research," he/she means that comprehending the essential elements, principles and processes that constitute rigorous and reliable research is crucial. This includes clear objectives, methodological rigor, data quality, objective and unbiased interpretation of data, validity and reliability (findings are generalizable and replicable), and adhering to ethical standards. Therefore, "Importance of Quality Research" doesn't accurately convey the message of the text. New code : Understanding how to Evaluate Research Quality [Model] [Assigned Code] /Corresponding text segment : [ChatGPT4] [Political influence on health decisions] \ "Political motivations may also have played a factor and thus, as a result, much harm was produced" Evaluation : Evaluation : The code says, "political influence" instead of "political motivation," which is what the text states. While ChatGPT4’s coding alludes to the indirect impact of political motivation and its resulting influence on decision making, it is noteworthy that "influence" and "motivation" have distinct meanings, and the text specifically says political motivation was responsible for the harm. Therefore, "Political Influence Causing Harm" is an inaccurate code. New code : Political Motivation Caused Harm [Model] [Assigned Code] /Corresponding text segment : [ChatGPT4] [Scientific Discovery and Innovation] \ "Regarding ACE2 decoy, the ingeniousness of this method captured my imagination, and I feel that this overall is the most promising approach to date." Evaluation : The text expresses admiration for the innovative approaches in scientific research, particularly highlighting the potential of ACE2 decoy as the most promising new therapeutic strategies. The statement has an emotional undertone and appreciation for creativity. Therefore, "Scientific Discovery and Innovation" as a code is too generic and fails to capture the essence of the text. New code : Inspired by Ingenious Scientific Approaches [Model] [Assigned Code] /Corresponding text segment : [ChatGPT4] [Political vs. Scientific Decisions in Drug Approval] \ "Such decisions, appearing more political than scientific, underscore the critical importance of maintaining rigorous standards for drug approval, irrespective of external pressures or the urgency of health crises." Evaluation : "Political vs. Scientific Decisions in Drug Approval" is not the core message of the text. Rather, the text highlights the importance of continuing to uphold rigorous standards, which is a different emphasis than just "need for rigorous standards." The text also says that an approval decision should not be influenced by external pressure. New code : Imperative to Disregard External Pressure and Adhere to Rigorous Standards for Drug Approval [Model] [Assigned Code] /Corresponding text segment : [ChatGPT4] [Rapid Findings] \ "...their findings must be interpreted with an understanding of their limitations." Evaluation : "Rapid Findings" is a nonsensical code because it has nothing to do with what the text conveys. In essence, the text says that findings are not absolute and should be interpreted with caution and avoid over-interpretation. New code : Must Carefully Evaluate Findings [Model] [Assigned Code] /Corresponding text segment : [ChatGPT4] [Importance of Integrity and Transparency] \ "...it is imperative for researchers, publishers, and all involved parties to adhere to the highest standards of integrity, transparency, and diligence..." Evaluation : "Importance of Integrity and Transparency" is a good start, but it can be refined to capture the full essence of the statement. The word "imperative" in the statement indicates the importance for all to adhere to the highest standards of integrity, transparency, and diligence. Evaluation: New code : Necessity of Accountability for All Stakeholders [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Discovery of HCQ's Inefficacy and Side Effects] \ "Later studies were conducted which disproved the efficacy of HCQ in treating COVID and even showed alarming side effects such as cardiac arrest and ventricular arrhythmia." Evaluation : "Discovery of HCQ's Inefficacy and Side Effects" is too generic to reflect the meaning of the text. The text emphasizes that it is the "later studies", which were well-designed, disapproved the efficacy of HCQ in treating COVID as claimed by earlier studies, and showed toxic side effects of HCQ. New code : Claims of HCQ Efficacy for COVID Were Disapproved by New Studies [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Need for training in critical evaluation] \ "...there should be more training which teaches important figures influencing healthcare decisions how to critically evaluate research articles." Evaluation : The code "Need for training in critical evaluation" fails to pay attention to the significance of the words "more training" in the text. The main point of the text is that there is a need for additional or increased training in critical evaluation skills. Therefore, this code is inaccurate. New code : Call For More Training in Critically Evaluating Skills [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Use of multiple statistical tools] \ "The use of multiple statistical tools in these studies is important because it allows the investigators to get a clearer picture of the treatment's actual effects or lack thereof." Evaluation : “Use of multiple statistical tools” does not capture the essence of the text because it implies that statistical tools were either not used at al or only one was used. Rather, the text underscores the importance of using multiple statistical tools to obtain a more comprehensive and a clearer picture of the treatment's effects. New code : Need to Assess Treatment Efficacy Through the Use of Multiple Statistical Methods [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Need for rigorous standards in drug approval] \ "underscore the critical importance of maintaining rigorous standards for drug approval, irrespective of external pressures" Evaluation : “Need for rigorous standards in drug approval” does not capture the essence of the text because it implies that there were no rigorous standards used. Rather, the text highlights the importance of continuing to uphold rigorous standards, which is a different emphasis than just "need for rigorous standards." The text also says that an approval decision should not be influenced by external pressure. New code : Imperative To Disregard External Pressure and Adhere to Rigorous Standards for Drug Approval [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Need for Skills in Identifying Poor Research] \ "Understanding and identifying this is a skill that is important to future physicians who will be looking at medical research for the entirety of their careers." Evaluation : It is important to note that "this" in the text refers to published studies of poor quality with unreliable data. Also, the text says that having the skill to identify such studies is important to future physicians. Therefore, "Need for Skills in Identifying Poor Research" does not capture the full essence of the text. New code : Future physicians Need the Skill to Identifying Poor Research [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Political Research] \ "How good research can be contorted into a political tool." Evaluation : "Political Research" is a wrong code because the text is not about political research but about using research findings for political purposes. New code : Use Good Research as Political Tool [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Peer Interpretations and Discourse] \ "I really enjoyed hearing my peers’ interpretations of research throughout this elective, and it led to some remarkably interesting discourse." Evaluation : "Peer Interpretations and Discourse" does not capture the full essence of the text because the text expresses appreciation of listening to peers' unique interpretations of research, which enriches one's own understanding. Additionally, these varying perspectives led to engaging, insightful, and thought-provoking conversations. In short, listening to and discussing with peers about issues is enjoyable and intellectually stimulating. New code : High Value of Learning from Peers [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Better Understanding of Clinical Decisions] \ "This course also helped me better understand the treatment decisions that the doctors I have worked with made while I was on clerkships a year ago." Evaluation : "Better Understanding of Clinical Decisions" is a generic code because the person now realizes that the knowledge he has learned from this course helps him to better understand the treatment decisions made by doctors with whom he worked previously. In other words, his newly acquired knowledge from the course enables him to comprehend the reasoning behind decisions that seemed unclear or puzzling a year ago. New code : Learning in the Course Enhanced Understanding of Past Clinical Decisions [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Necessity for High Standards in Health Crises] \ "It is imperative for researchers, publishers, and all involved parties to adhere to the highest standards of integrity, transparency, and diligence, given the direct consequences their findings can have on public health measures and global strategies." Evaluation : "Necessity for High Standards in Health Crises" does not fully reflect the meaning of the text because it failed to capture the meaning of the word "adhere" in the text. Therefore, it is about the imperative of adhering to the highest standards to prevent negative consequences on public health measures and global strategies. New code : Imperative to adhere to the highest standards [Model] [Assigned Code] /Corresponding text segment : [OpenAI o1-preview] [Emphasis on Randomized Controlled Trials (RCTs)] \ "Due to the stakes involved, as much as possible, it should only be the results of randomized controlled trials which are allowed to influence healthcare decisions" Evaluation: " Emphasis on Randomized Controlled Trials (RCTs)" does not fully capture the meaning in the text. The text emphasizes the importance of making health decisions based on findings from RCTs. New code : Health Decision Should Rely on Randomized Controlled Trials Codes generated by ChatGPT4 and OpenAI o1-preview are evaluated and revised by human coders against the original corresponding text segments. Discussion In this study, we aimed to evaluate the potential of ChatGPT-4 and OpenAI’s o1-preview model in automating the text coding process for thematic analysis. We selected these two models because both are based on the GPT architecture developed by OpenAI, but o1-preview is more advanced due to its integration of a chain-of-thought reasoning mechanism that enhances problem-solving and analytical capabilities in large language models [ 40 ]. The chain-of-thought mechanism allows the model to solve complex problems by breaking them down into simpler, intermediate steps, which are then used iteratively to generate subsequent reasoning. This process enables the model to refine its output, correct earlier missteps, and improve overall accuracy. Accordingly, we sought to assess the suitability of both models for text coding and to determine whether the chain-of-thought mechanism would confer a performance advantage to the o1-preview model. Specifically, we evaluated the models using three metrics: coding density, coding consistency across the text, and coding accuracy. Both coding density and coding consistency measures the degree to which codes are applied uniformly and reliably throughout the data, and they are crucial in thematic analysis to ensure reliable, valid, and trustworthy findings [ 8 – 11 ]. As the data in Table 1 shows, in the 3 independent tests we performed, ChatGPT-4 consistently identified fewer text segments and exhibited a lower coding density (19.58–25.50%) compared to the coding density of OpenAI o1-preview (51.74–80.85%). However, as document length increased, coding density of both models, especially ChatGPT4, decreased (Tables 2 and 3 ). Furthermore, regardless of document length, ChatGPT-4 showed a consistently lower coding density than OpenAI o1-preview (Tables 2 and 3 ). This is like the finding of other investigators showing that ChatGPT-4 could not code a large dataset at once [ 43 ]. In this study, the low coding density of ChatGPT4 apparently can be explained by ChatGPT-4's tendency to concentrate its coding efforts on the initial paragraphs while skipping numerous subsequent sections in a document (Table 4 ). There is no apparent way to speculate why section skipping occurred because skipping was random as the sections skipped varied from test to test ( Table 4 ). Nevertheless, these results indicate an issue of uneven coding attention, and it undermines the reliability of ChatGPT4 for thematic analysis as it will lead to biased or incomplete theme identification. While not skipping any paragraphs in a document, OpenAI o1-preview exhibited variability in coding density from test to test, indicating inter-test variability in coding density. The implication here is that OpenAI o1-preview requires calibration or iterative prompting to achieve stable performance across all sections of the entire text consistently. These findings indicate that both ChatGPT4 and OpenAI o1-preview are limited in their scalability and utility in extensive qualitative studies concerning large datasets. Given that both models performed extensive coding of short text, it suggests that coding of longer documents by these models requires additional text segmentation to ensure comprehensive coding. The examples of coding inaccuracies in Table 6 reveal significant limitations in their coding accuracy. These models struggle to capture nuanced contextual cues and distinguish meanings between terms, and moreover, they focus on surface-level patterns rather than deeper text meanings as discussed below and are prone to oversimplification and misinterpretation. In this study, we use the term deep text meanings to refer to underlying themes, nuances, and contextual implications that are not immediately evident through superficial semantic interpretation. These include, but are not limited to, pragmatic elements such as speaker intention, tone, and implicature, as well as discourse-level features such as power dynamics, social context, and cultural references. Because current large language models (LLMs) are unable to adequately process these dimensions, their outputs require rigorous human review and are of limited utility in domains that demand nuanced interpretive understanding. To enhance their suitability for qualitative analysis, LLMs must be further refined to capture not only semantic content but also the pragmatic and discourse-level features embedded in text. We use the term meaning to refer to the complex and nuanced understanding of language and context that characterizes human cognition. While large language models (LLMs) can produce outputs that appear coherent—and at times even insightful—this coherence often creates an illusion of understanding. In reality, LLMs do not possess genuine comprehension; rather, they generate responses based on probabilistic associations between tokens. This distinction is critical, as it highlights the limitations of current LLMs in grasping the depth and complexity of human language. Their inability to code text accurately offers further evidence that LLMs do not truly understand the meanings of words, phrases, or sentences—they merely simulate understanding through statistical prediction. The coding inaccuracies observed in this study suggest that when applied to texts containing complex ideas or sensitive topics, these models are likely to produce incomplete or misleading representations of the data. These limitations underscore the broader challenge of using LLMs for tasks that require text comprehension and interpretation of nuanced meanings. However, our findings do not entirely preclude the use of LLMs in thematic analysis. As suggested by others [ 43 ], LLMs may “serve as an additional member of the analysis team, contributing to researcher triangulation through knowledge building and sensemaking.” It is said that most tokens in a reasoning chain in current LLMs are generated solely for language fluency and have little to do with reasoning [ 44 ]. Therefore, for LLMs to be able to reason they need to be significantly improved to “have the freedom to reason without any language constraints, and then translate their findings into language only when necessary.” [ 44 ]. To achieve this kind of improvement, it is suggested that to pretrain LLMs with continuous thoughts to enable them to generalize more effectively across a wider range of reasoning scenarios [ 44 ]. In the future, perhaps, text coding accuracy can be used, among other methods, to measure reasoning capability of improved LLMs. This study has several limitations that should be addressed in future research. First, our analysis focused on two specific large language models (ChatGPT-4 and OpenAI o1-preview) and the results may not generalize to other models or future iterations. Second, we analyzed a limited set of documents and topics; future studies should explore a broader range of qualitative data types and subject matter to better assess the generalizability of our findings.Third, our analysis relied primarily on Document 1, which consisted of student comments that were generally brief, composed of short sentences, and limited in both content richness and linguistic complexity. Future research should incorporate texts that are intentionally constructed to exhibit greater linguistic and conceptual depth. Such texts typically include multi-clause sentences with embedded structures, abstract or context-dependent language, pragmatic cues, semantic ambiguity, subtle presuppositions, and culturally informed references. Linguistic complexity should be understood to include syntactic variation, lexical and semantic richness, and pragmatic and discourse-level features. Fourth, this study focused exclusively on a single phase of thematic analysis – that is,text coding. Future research should prompt LLMs to perform all six phases of thematic analysis to evaluate their broader applicability in qualitative inquiry. Conclusion This study highlights the promising yet currently limited role of large language models in thematic analysis. OpenAI o1-preview demonstrated superior performance in coding density and consistency compared to ChatGPT-4, yet both models fell short in coding accuracy. Furthermore, our findings provide new evidence that even the most advanced OpenAI o1-preview lacks the capability of comprehending nuanced semantic meanings in text. While LLMs can significantly reduce the time and effort required for initial text coding, their integration into qualitative research necessitates careful consideration of their limitations. By advancing model training, enhancing contextual understanding, and adopting hybrid analytical frameworks, LLMs may become valuable tools that augment the capabilities of researchers, ultimately contributing to more efficient and comprehensive thematic analyses. Declarations Acknowledgements Not applicable Funding: The authors declare that no funds, grants, or other support were received during the preparation of this manuscript. Availability of data and materials All data generated or analyzed during this study are included in this published article. Ethics, Consent to Participate, and Consent to Publish declarations: not applicable. References Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006:3(2);77–101. https://doi.org/10.1191/1478088706qp063oa Liebenberg L, Jamal A, Ikeda J. Extending youth voices in a participatory thematic analysis approach. Int J Qual Methods. 2020:19;1609406920934614. https://doi.org/10.1177/1609406920934614 Maguire M, Delahunt B. Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. AISHE J. 2017:3;3351-33514. Xu W, Zammit K. Applying thematic analysis to education: a hybrid approach to interpreting data in practitioner research. Int J Qual Methods. 2020:19. https://doi.org/10.1177/1609406920918810 Naeem M, et al. A Step-by-Step Process of Thematic Analysis to Develop a Conceptual Model in Qualitative Research. Int J Qual Methods. 2023:22. https://doi.org/10.1177/16094069231205789 Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval. 2006:27(2);237-246. https://doi.org/10.1177/1098214005283748 Elliott V. Thinking about the coding process in qualitative data analysis. Qual Rep. 2018:23(11);2850–2861. https://doi.org/10.46743/2160-3715/2018.3560 Morse JM, Mitcham C. Exploring qualitatively-derived concepts: inductive-deductive pitfalls. Int J Qual Methods. 2002:1(4);28-35. https://doi.org/10.1177/160940690200100404 Patton MQ. Qualitative research and evaluation methods (4th ed.). Thousand Oaks, CA: Sage. Olson JD, et al. Applying Constant Comparative Method with Multiple Investigators and Inter-Coder Reliability. Qual Rep. 2016:21(1);26-42. https://doi.org/10.46743/2160-3715/2016.2447 Richards KAR, Hemphill MA. A Practical guide to collaborative qualitative data analysis. J Teach Phys Educ. 2017:37(2);225-231. https://doi.org/10.1123/jtpe.2017-0084 DeSantis L, Ugarriza D. The concept of theme as used in qualitative nursing research. West J Nurs Res. 2000:22(3);351–372. https://doi.org/10.1177/019394590002200308 Scriven M. Pros and cons about goal-free evaluation. Eval Pract. 1991:12(1);55–63. https://doi.org/10.1016/0886-1633(91)90024-r Peters ME, et al. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018, pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. https://aclanthology.org/N18-1202/ Vaswani A, et al. Attention is all you need. Advances in neural information processing systems, 2017:30. https://doi.org/10.48550/arXiv.1706.03762 Sennrich R, Haddow B, Birch A. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pages 1715–1725, Berlin, Germany. Association for Computational Linguistics. https://aclanthology.org/P16-1162/ Kudo T, Richardson J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXivpreprintarXiv:1808.06226. 2018. https://doi.org/10.48550/arXiv.1808.06226 Haslett A. Cai ZG. How much semantic information is available in large language model tokens? PsyArXiv Preprints. 2025. https://doi.org/10.31234/osf.io/nhwdz Rajaraman N, Jiao J, Ramchandran K. Toward a theory of tokenization in llms. arXivpreprintarXiv:2404.08335. https://doi.org/10.48550/arXiv.2404.08335 Mitchell M, Krakauer DC. The debate over understanding in AI’s large language models, Proc. Natl. Acad. Sci. U.S.A. 120:(13):e2215907120, https://doi.org/10.1073/pnas.2215907120 (2023. Mitchell M. How do we know how smart AI systems are? Science. 2023:381(6654);adj5957. https://doi.org/10.1126/science.adj5957 Devlin J, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) 2019 Jun (pp. 4171-4186). https://aclanthology.org/N19-1423.pdf Radford A, et al. Improving Language Understanding by Generative Pre-Training. OpenAI blog, 2018, page 12. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf Zhang H, et al. Redefining qualitative analysis in the AI era: Utilizing ChatGPT for efficient thematic analysis. arXivpreprintarXiv:2309.10771. 2023. https://doi.org/10.48550/arXiv.2309.10771 Katz A, et al. Exploring the efficacy of ChatGPT in analyzing Student Teamwork Feedback with an existing taxonomy. arXivpreprintarXiv:2305.11882. 2023. https://doi.org/10.48550/arXiv.2305.11882 Gao J, et al. CoAIcoder: Examining the effectiveness of AI-assisted human-to-human collaboration in qualitative analysis. ACM Trans Comput Hum Interact. 2023:31(1);1-38. https://doi.org/10.48550/arXiv.2304.05560 Hamilton L, et al. Exploring the Use of AI in Qualitative Analysis: A Comparative Study of Guaranteed Income Data. Int J Qual Methods. 2023:22. https://doi.org/10.1177/16094069231201504 Fiannaca AJ, et al. Programming without a programming language: challenges and opportunities for designing developer tools for prompt programming. Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg Germany, 2023-04-19). Article235. https://doi.org/10.1145/3544549.3585737 Zamfrescu-Pereira J., et al. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 437. 2023. https://doi.org/10.1145/3544548.3581388 Morgan DL. Exploring the use of artificial intelligence for qualitative data analysis: the case of ChatGPT. Int J Qual Methods. 2023:22. https://doi.org/10.1177/16094069231211248 Andersson M, McIntyre D. Can ChatGPT recognize impoliteness? An exploratory study of the pragmatic awareness of a large language model. J Pragmat. 2025:239:16-36. https://doi.org/10.1016/j.pragma.2025.02.001 Frantz CK, et al. Subjective Interpretation of Directive Terms: Methodology and Preliminary Results," 2015 International Conference on Culture and Computing (Culture Computing), Kyoto, Japan, 2015, pp. 137-144. doi: 10.1109/Culture.and.Computing.2015.40. Kazemian R, et al. Beyond sentence grammar: Persian directives in interaction. J Pragmat. 2025:236:40-59. https://doi.org/10.1016/j.pragma.2024.12.010 Webb T, Holyoak KJ, Lu H. Emergent analogical reasoning in large language models. Nat Hum Behav. 2023:7;1526–1541. https://doi.org/10.1038/s41562-023-01659-w Lewis M, Mitchell M. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXivpreprintarXiv:2402.08955. 2024. https://doi.org/10.48550/arXiv.2402.08955 Mirzadeh I, et al. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXivpreprintarXiv:2410.05229. 2024. https://doi.org/10.48550/arXiv.2410.05229 Han Z, et al. Beyond text generation: assessing large language models’ ability to reason logically and follow strict rules. AI. 2025:6(1);12. https://doi.org/10.3390/ai6010012 Sullivan R, Elsayed N. Can large language models act as symbolic reasoners? arXivpreprintarXiv:2410.21490. https://doi.org/10.48550/arXiv.2410.21490 Turobov A, Coyle D, Harding V. Using ChatGPT for thematic analysis. arXivpreprintarXiv:2405.08828. 2024. https://doi.org/10.48550/arXiv.2405.08828 Wei J, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst. 2022:35;24824-24837. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf. Shore CK, Worku TL, Kahn J. Rare disease innovation at the FDA-opportunities for implementation. JAMA. 2024:332(23);1973–1974. https://doi.org/10.1001/jama.2024.20358 Guare EG, et al. Rates of sudden unexpected infant death before and during the COVID-19 pandemic. JAMA Netw Open. 2024:7(9);e2435722. https://doi.org/10.1001/jamanetworkopen.2024.35722 Lee VV, et al. Harnessing ChatGPT for thematic analysis: are we ready? J Med Internet Res. 2024:26;e54974. https://doi.org/10.2196/54974 Hao S, et al. Training large language models to reason in a continuous latent space. arXivpreprint arXiv:2412.06769. 2024. https://doi.org/10.48550/arXiv.2412.06769 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 24 Jul, 2025 Read the published version in Discover Artificial Intelligence → Version 1 posted Editorial decision: Revision requested 07 May, 2025 Reviews received at journal 03 May, 2025 Reviewers agreed at journal 03 May, 2025 Reviews received at journal 28 Apr, 2025 Reviewers agreed at journal 28 Apr, 2025 Reviewers invited by journal 28 Apr, 2025 Editor assigned by journal 28 Apr, 2025 Submission checks completed at journal 25 Apr, 2025 First submitted to journal 18 Apr, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5937488","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":449179991,"identity":"059b2f51-be1a-4d34-9de4-e214719915ab","order_by":0,"name":"Zhiyong Han","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7UlEQVRIiWNgGAWjYDACCcYGMG3AwMD4GC5IrBZmYyK1QGmgFjZporTIz25uk/i4o5bBnP3sserCnDo53Qbmg7d58GhhnHOwTXLmmeMMlj15abdnbjtsbHaALdkanxZmicQ2ad62YwwGB3LMbvNuO5C47QCPmTQ+LWxwLeffmBXzbqsDauH/hlcLD0RLDYPBjRwzZt5tzCBb2PBqkZBIbLac2XaAx+DGG2NpXpBfDrMZW87Bo0V+RvrDGx/b6uQMzucYfgY6TM7sePPDG2/waAECFmAsHEZyCTN+5WAlHxgY6ggrGwWjYBSMgpELAH4tSUkZoQ0PAAAAAElFTkSuQmCC","orcid":"","institution":"Hackensack Meridian School of Medicine","correspondingAuthor":true,"prefix":"","firstName":"Zhiyong","middleName":"","lastName":"Han","suffix":""},{"id":449179993,"identity":"346d2fed-ec8b-4d38-9760-f6c7f6d0b425","order_by":1,"name":"Aaron Tavasi","email":"","orcid":"","institution":"Hackensack Meridian School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Aaron","middleName":"","lastName":"Tavasi","suffix":""},{"id":449179994,"identity":"10d5f536-dc94-4ae4-a9e7-a7ad31ec6ca3","order_by":2,"name":"JuYoung Lee","email":"","orcid":"","institution":"Hackensack Meridian School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"JuYoung","middleName":"","lastName":"Lee","suffix":""},{"id":449179995,"identity":"ef48e405-f417-4870-9008-24a094541d73","order_by":3,"name":"Joshua Luzuriaga","email":"","orcid":"","institution":"Hackensack Meridian School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Joshua","middleName":"","lastName":"Luzuriaga","suffix":""},{"id":449179996,"identity":"19f2fa9d-f543-43fa-99dd-030b63b2b7c1","order_by":4,"name":"Kevin Suresh","email":"","orcid":"","institution":"Hackensack Meridian School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Kevin","middleName":"","lastName":"Suresh","suffix":""},{"id":449179997,"identity":"ee17c9c6-4923-4162-8d85-f840bdcbace5","order_by":5,"name":"Michael Oppenheim","email":"","orcid":"","institution":"Hackensack Meridian School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Michael","middleName":"","lastName":"Oppenheim","suffix":""},{"id":449179998,"identity":"f64b9ce0-8553-4aa6-a8ee-ed370095addc","order_by":6,"name":"Fortunato Battaglia","email":"","orcid":"","institution":"Hackensack Meridian School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Fortunato","middleName":"","lastName":"Battaglia","suffix":""},{"id":449179999,"identity":"611e1242-46c8-4759-b796-bcbaa311cc28","order_by":7,"name":"Stanley R. Terlecky","email":"","orcid":"","institution":"Hackensack Meridian School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Stanley","middleName":"R.","lastName":"Terlecky","suffix":""}],"badges":[],"createdAt":"2025-01-31 15:53:06","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5937488/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5937488/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s44163-025-00441-3","type":"published","date":"2025-07-24T15:57:27+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":87756689,"identity":"6726eea9-9b98-4550-8842-c24c1525a2e5","added_by":"auto","created_at":"2025-07-28 16:07:44","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2388074,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5937488/v1/6540f0b0-242f-430a-b9f8-c7c123b8a9ce.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Can Large Language Models Be Used to Code Text for Thematic Analysis? An Explorative Study","fulltext":[{"header":"Introduction","content":"\u003cp\u003eThematic analysis is a widely used qualitative method that enables researchers to identify and interpret patterns and themes within rich, textual data. In practice, the process is typically divided into six stages: familiarization with the data, text coding, development of a coding framework, theme identification, theme refinement, and reporting [\u003cspan additionalcitationids=\"CR2 CR3 CR4\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Throughout this process, researchers must pay meticulous attention to detail and engage in critical interpretation of thematic meaning\u0026mdash;examining how words and phrases convey meaning, how that meaning may shift across contexts, and how meanings relate to one another within the structure of the text. This analytic rigor ensures that findings remain grounded in the data. When conducted carefully, thematic analysis can uncover underlying patterns and narratives shaped by human experiences, offering deep insights into participants\u0026rsquo; perspectives and meanings [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eImportantly, researchers must approach the analysis without preconceptions, as preconceived notions can lead to biased theme identification [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. To preserve the integrity of the analysis, themes should emerge inductively from the data, guided by unbiased interpretation rather than predetermined expectations.\u003c/p\u003e \u003cp\u003eThematic analysis is cognitively demanding, time-consuming, and labor-intensive, particularly during the text familiarization and coding stages, because both are iterative in nature. Text familiarization requires investigators to immerse themselves deeply in the data, while coding involves systematically identifying and labeling segments of text with tags that summarize their core content. During the coding stage, researchers (also known as coders at this stage) must continually refine and revise their codes to fully reflect the meanings of text segments. To ensure reliability, multiple coders are required to minimize personal bias and misinterpretation of textual meanings, and following independent coding, the coders discuss and reconcile their codes [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Therefore, iterative coding demands constant comparison across the text to ensure consistent and accurate code application, and this is why it is cognitively challenging, time-consuming, and labor-intensive especially when the volume of the textual data is large and the content is complex.\u003c/p\u003e \u003cp\u003eAfter code reconciliation, investigators cluster codes to facilitate theme extraction [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e] This process, while still time-consuming and cognitively demanding, is typically less intensive than the initial text familiarization and coding phases. To enhance theme extraction, investigators often use techniques such as identifying patterns and relationships among codes; analyzing code frequencies and co-occurrences; examining deviant or negative cases that challenge emerging themes; and considering the broader contextual factors influencing the data [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. By employing these strategies, researchers can systematically extract themes that accurately represent the underlying significance and perspectives within the textual data [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eLLMs are powerful computing systems, based on natural language processing techniques and the transformer architecture, that have been trained on vast amounts of human-written text on almost any topics, enabling them to recognize patterns and structures in human writing through iterative training processes [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Unlike how humans learn and read languages, LLMs use a technique called tokenization during training to break text into small units called tokens. These tokens may represent entire words, sub-words, or even individual characters [\u003cspan additionalcitationids=\"CR17 CR18\" citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. Over millions/billions of iterations, LLMs refine their ability to anticipate which tokens are statistically most likely to follow others in a context relevant fashion. This iterative process results in a rich statistical representation of language in terms of relationships between and among tokens in huamn written texts. However, this mathematical token-to-token processing of language is totally different from humans\u0026rsquo; understanding of language, which is based on grammatical rules and meanings of words [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] The mathematical processing of language associated with tokenization of text also means that the trained LLMs do not understand the meaning of words, phrases, and sentences as humans do. Nevertheless, LLM\u0026rsquo;s extensive exposure to token patterns in human-written text covering almost any topics enables them to generate human-like coherent, contextually relevant, and fluent responses to user queries on any specific topic [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe impressive human-like language skills of LLMs have sparked intense interest and investigations about how LLMs can assist, if not replace, humans to complete tasks more quickly and effectively in various fields, such as computer coding, education, disease diagnosis, journalism, and law. One potential application of LLMs like ChatGPT is thematic analysis. Some recent research suggests that LLMs possess several advantages that make them suited for this task. Specifically, ChatGPT have an \u0026ldquo;exceptional ability to understand and generate human-like text\u0026rdquo; [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], can standardize and streamline the coding process, increasing efficiency and consistency [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], and can enhance code diversity [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Furthermore, ChatGPT offers additional benefits, including rapid data processing, improved work efficiency, concise summaries, and preliminary insights [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. In one study [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e], researchers compared the ability of ChatGPT to that of humans in generating themes from interviews, and they found similarities and differences between human coders and ChatGPT, and found that human-centered coding was superior because human coders brought a depth of analysis, sensitivity to nuances, and interpretive flexibility that ChatGPT lacked [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. Therefore, they concluded that ChatGPT is a powerful tool to supplement complex human-centered tasks. However, it is important to note that this study only prompted ChatGPT to analyze statements and then generate themes without performing other steps of thematic analysis. Regardless, it is suggested that incorporating generative-AI such as ChatGPT in qualitative analysis allows analyzing large volumes of textual data quickly and comprehensively by offering efficient and speedy data processing, and discerning insights and themes in the data that may not be immediately apparent to human coders [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn practice, it is found that the outcome of using ChatGPT to conduct qualitative analysis depends on well-designed prompts [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], which is a critical issue that limits effective use of ChatGPT by investigators in qualitative analysis tasks, especially among those who lack expertise and experience with ChatGPT [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e, \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e]. Nevertheless, findings from several studies suggest that ChatGPT has the promise as a tool for enhancing the efficiency and consistency of data coding [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. Therefore, ChatGPT may have the potential to be a coding assistant for human coders, and the involvement of ChatGPT is likely to reduce coding time and increase coding accuracy as well as uncover information overlooked by human coders. However, it should be acknowledged that although LLMs can process and analyze textual data and generate impressive outputs that appear to grasp the literal meanings of words and sentences, which, while crucial, is insufficient on its own when it comes to text coding because to fully understand what a writer or speaker intends to convey coders must go beyond semantics and engage with pragmatic dimensions such as implicature, where meaning is implied rather than explicitly stated, and presupposition, all of which involves recognizing background assumptions embedded in language. It is also important to note that deciphering meanings in text also involves identifying speech acts - whether a sentence functions as a question, a request, or a warning - especially when expressed indirectly. In addition, coders must track discourse coherence, resolve referential ambiguity, and apply cultural knowledge to interpret idioms, humor, and context-specific expressions. These layers of interpretation highlight the complex interplay between language, context, and human intention. Consequently, LLMs\u0026rsquo; \u0026ldquo;understanding\u0026rdquo; of language via tokenization of text means that they may struggle to accurately interpret or code texts that rely on context-bound features of language use, such as pragmatic meaning [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. Additionally, it remains to be investigated is LLMs can recognize context-sensitive directives [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e], especially when cultural nuance is central to interpretation [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. When considering the use of LLMs to support thematic analysis, it is therefore essential to evaluate their capacity to interpret linguistic functions that extend beyond semantic understanding.\u003c/p\u003e \u003cp\u003eAnother important issue is that the question of whether LLMs can genuinely comprehend and reason remains a subject of active debate [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. Investigating this question is complicated by the fact that the inner workings of LLMs are effectively a \u0026ldquo;black box.\u0026rdquo; Due to the enormous complexity and sheer number of parameters involved, there are currently no well-established methods for users\u0026mdash;or even developers\u0026mdash;to systematically dissect and understand how these models process input and generate responses. As a result, it remains difficult to assess what kind of representations or reasoning processes, if any, underlie the outputs of these large-scale statistical models. Despite these challenges, numerous studies have explored the reasoning capacities of LLMs. Webb and colleagues suggest that the ability of models like GPT-3 to solve a wide range of problems may be attributed to their analogical reasoning capabilities [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]. Conversely, others contend that these models lack the robustness and general characteristics of true analogical reasoning [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]. Alternative approaches have been used to test the reasoning abilities of LLMs. For example, Mirzadeh and colleagues [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e] examining the limitations of mathematical reasoning in LLMs have observed that their performance tends to worsen as the complexity of a question increases, particularly when more clauses are involved. According to their analysis, the models are not genuinely capable of reasoning; instead, they reproduce the reasoning sequences they encountered during training. As a result, what may appear to be logical reasoning in LLMs can be more accurately described as a pattern-based simulation rather than a true deductive process. In another study, Han et al tested whether LLMs use reason to solve word ladder puzzles, and they found no evidence of reasoning [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e]. In a recent review of the studies related to reasoning capabilities of LLMs [\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e], Sullivan and Elsayed reviewed the literature to assess whether LLMs can perform symbolic reasoning, and they reached a conclusion that \u0026ldquo;the probabilistic architecture of current LLMs, is not able to inherently perform symbolic reasoning, requiring external impetus to move in this direction.\u0026rdquo; Therefore, LLMs at their current states have serious limitations in using reasoning in different contexts.\u003c/p\u003e \u003cp\u003eGiven that thematic analysis, particularly text coding and theme identification, requires deep comprehension, including the ability to isolate relevant segments based on semantic and pragmatic meanings, presupposition, and the use of reasoning to iteratively develop appropriate codes, the apparent disparity between LLMs\u0026rsquo; limitations in comprehension and their application to cognitively demanding tasks raises important questions. Even for humans, interpreting pragmatic meaning in text can be challenging; for LLMs, which lack true contextual understanding, this difficulty is likely magnified [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. Nevertheless, studies [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e] have suggested that LLMs hold potential in performing thematic analysis. This promise may stem from their advanced pattern recognition and ability to replicate statistical regularities learned from vast training data, enabling them to produce outputs that functionally mimic certain aspects of human coding. This paradox highlights the complex relationship between comprehension, reasoning, and statistical modeling in LLMs. While they may not genuinely \u0026ldquo;understand\u0026rdquo; text, their probabilistic mechanisms can at times yield analyses that appear meaningful. However, caution is warranted: without true comprehension, LLMs risk generating outputs that are coherent yet shallow, or that overlook subtle pragmatic nuances essential to robust qualitative interpretation. Nevertheless, by systematically prompting LLMs to code diverse types of text, especially text that was not included in the training data, and rigorously evaluating their performance - particularly in terms of coding accuracy - researchers can gain insights into the strengths and weaknesses of these models in thematic analysis. Such an approach not only assesses their current capabilities but also identifies areas for improvement, ultimately contributing to the enhancement of LLM performance and reliability in qualitative research methodologies.\u003c/p\u003e \u003cp\u003eThis study investigates the potential of two large language models (LLMs), ChatGPT-4 and OpenAI o1-preview, to perform a specific task of thematic analysis - namely, text coding. We selected these two models because both are based on OpenAI\u0026rsquo;s GPT architecture, with o1-preview representing a more advanced version that incorporates an additional chain-of-thought reasoning mechanism [\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e]. Our aim was to evaluate each model\u0026rsquo;s suitability for text coding and to assess whether the inclusion of chain-of-thought reasoning would make o1-preview the superior model. To ensure novelty and minimize the likelihood that the models had encountered the data during training, we used documents unlikely to have appeared in their training sets. Specifically, we compared the comprehensiveness and accuracy of the codes each model generated. Our findings show that both ChatGPT-4 and o1-preview struggled with coding - particularly for longer texts - and that many of their generated codes were inaccurate or required substantial revision. These results suggest that, in their current forms, both models lack the comprehension and reasoning capabilities necessary to reliably and accurately code texts for thematic analysis. Significant improvements will be required before such models can be used independently in qualitative research without compromising analytical quality.\u003c/p\u003e"},{"header":"Materials and Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eLarge Language Models:\u003c/h2\u003e \u003cp\u003eThe LLMs used in this study were ChatGPT4 and OpenAI 01-preview by OpenAI. Both models were available via subscription. ChatGPT4 was released on March 14, 2023, and OpenAI o1-preview on September 12, 2024. Unlike ChatGPT4 and other ChatGPT models, OpenAI o1-preview is \"designed to spend more time thinking before they respond. They can reason through complex tasks and solve harder problems than previous models in science, coding, and math\" as stated by OpenAI (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://openai.com/index/introducing-openai-o1-preview/\u003c/span\u003e\u003cspan address=\"https://openai.com/index/introducing-openai-o1-preview/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eDocuments preparation:\u003c/h3\u003e\n\u003cp\u003eTo mitigate the risk that any of our documents might already exist in the training data of the LLMs, thereby rendering the outcomes of our analysis potentially predetermined and unreliable, we used one original document created by our team and several published documents that became available only after September 12, 2024, the release date of OpenAI o1-preview model.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDocument 1.\u003c/b\u003e Between 2020 and 2023, we offered a student elective titled \u003cem\u003eCOVID-19 and Lessons for Pandemic Drug Research\u003c/em\u003e. We received a total of 13 student comments about the course. These comments were de-identified and initially used for course improvement purposes. For this study, the de-identified comments were compiled into a single Word document totaling 1,149 words, with each comment presented as a separate paragraph. The use of these de-identified comments in this study was determined to be exempt from IRB requirements.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDocument 2.\u003c/b\u003e This document was derived from an opinion article titled \u003cem\u003eRare Disease Innovation at the FDA \u0026mdash; Opportunities for Implementation\u003c/em\u003e by Shore et al., published online in \u003cem\u003eJAMA\u003c/em\u003e on October 2, 2024 [\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e]. The article was converted into a Word document, with all citation numbers removed. The final document contained 1,185 words.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDocument 3.\u003c/b\u003e This document was derived from a research article titled \u003cem\u003eRates of Sudden Unexpected Infant Death Before and During the COVID-19 Pandemic\u003c/em\u003e by Guare et al., published in \u003cem\u003eJAMA Network Open\u003c/em\u003e on September 26, 2024 [\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e]. The article was converted into a Word document, with citation numbers and references to figures and tables removed. The final word count was 3,188 words.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDocument 4.\u003c/b\u003e This document was a shortened version of Document 3, created by removing the introduction and part of the methods section. The resulting document was 2,254 words long.\u003c/p\u003e\n\u003ch3\u003eText coding of students’ comments by LLMs:\u003c/h3\u003e\n\u003cp\u003eLLMs were prompted to perform unbiased inductive thematic coding of the students\u0026rsquo; comments in Document 1 using the following prompts.\u003c/p\u003e \u003cp\u003e \u003cem\u003eI am conducting an open-ended thematic analysis of a document and require your assistance. Your task is to analyze the content of the document to identify recurring topics, concerns, insights, or other relevant patterns. The approach should be inductive and unbiased, focusing on coding across various dimensions. Please focus on text coding and do not generate overarching themes at this stage. If a comment contains ambiguous or unclear content, please flag it for further review. Please ensure that each code is supported with textual evidence from the document to maintain transparency and verifiability in the coding process.\u003c/em\u003e \u003c/p\u003e\n\u003ch3\u003eCoding Density\u003c/h3\u003e\n\u003cp\u003eAfter text coding had been completed by LLMs, coding density was estimated as the percentage of coded words (i.e., the number of words in the coded text segments) over the total number of words in the entire document. This was computed using the formula: Coding Density (%) = (Number of coded words / Total number of words of the text) x 100.\u003c/p\u003e\n\u003ch3\u003eCoding accuracy assessment by human coders\u003c/h3\u003e\n\u003cp\u003eAll investigators in this study served as human coders. Each investigator critically reviewed and familiarized themselves with the student comments. Independently, they assessed the accuracy of the LLM-generated codes by comparing them to the original text. Following individual assessments, the investigators met to discuss their findings and reach consensus through collaborative review.\u003c/p\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eComparative Analysis of Text Coding Abilities: ChatGPT4 vs. OpenAI o1-mini\u003c/h2\u003e \u003cp\u003eIn our first investigation, we investigated the text coding capabilities of ChatGPT4 and OpenAI o1-preview using a 1149-word long document we compiled with 13 students' comments about an elective course. In the document, the comments were presented as separate paragraphs. We prompted the models to code the document in three independent tests and evaluated their performance using two metrics: a) total number of text segments identified and coded, and b) coding density (percentage of coded words over total document words) as defined in the methods.\u003c/p\u003e \u003cp\u003eThe results (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) revealed significant differences in coding capabilities between the two models. ChatGPT4 consistently identified and coded fewer text segments compared to OpenAI o1-preview. Additionally, ChatGPT4's coding density was substantially lower (19.58\u0026ndash;25.50%) than that of OpenAI o1-preview (51.74\u0026ndash;80.85%). These findings indicate that ChatGPT4 significantly under-coded the document.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCoding Capabilities Comparison: ChatGPT4 versus OpenAI o1-preview.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eChatGPT4\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eOpenAI o1-preview\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNo. of Text Segments Coded\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e28\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e30\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e58\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCoding Density\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e25.50%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e24.63%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e19.58%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e66.06%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e51.74%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e80.85%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMean Coding Density\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003e23.23% (SD: 2.61%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003e66.22% (SD:11.90%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eA single document containing thirteen students' comments (totaling 1149 words) was fed to ChatGPT4 and OpenAI o1-preiew for coding. Coding capacity was determined by (a) number of text segments identified and coded and (b) coding density estimated as the percentage of the number of coded words over the total number of words in the entire document. SD: standard deviation\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eDocument Lengths and Coding Density: ChatGPT4 versus OpenAI o1-preview\u003c/h3\u003e\n\u003cp\u003eTo understand whether document length might have caused under-coding by ChatGPT4, we altered the length of the document byaltering the total number of student comments between two shorter documents. We then prompted ChatGPT4 and OpenAI o1-preview to code the full-length document and the two shorter documents. The results in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e show that as the document became shorter, the coding density became larger for both ChatGPT4 and OpenAI o1-preview, suggesting that coding density was inversely correlated with document length. Although it was shown that ChatGPT-4 could not code a large dataset at once [\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e], to the best of our knowledge, no previous study has analyzed how LLM coding density varies with document length, making our findings a novel contribution to the literature on the capabilities and limitations of LLMs in qualitative research.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCoding density is inversely related to document length.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"9\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eDocument length by word number\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"4\" nameend=\"c5\" namest=\"c2\"\u003e \u003cp\u003eChatGPT4 (coding density%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"4\" nameend=\"c9\" namest=\"c6\"\u003e \u003cp\u003eOpenAI o1-preview (coding density %)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMean\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003eMean\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1149\u003c/p\u003e \u003cp\u003e(comments 1\u0026ndash;13)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e14.52\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17.11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e30.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e20.82\u003c/p\u003e \u003cp\u003e(SD: 7.16)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e66.06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e51.74\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e80.85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e66.22\u003c/p\u003e \u003cp\u003e(SD: 11.88)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e488\u003c/p\u003e \u003cp\u003e(comments 1\u0026ndash;5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e33.19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e35.24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e40.98\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e36.47\u003c/p\u003e \u003cp\u003e(SD: 3.30)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e84.56\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e85.60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e91.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e87.10\u003c/p\u003e \u003cp\u003e(SD: 1.04)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e141\u003c/p\u003e \u003cp\u003e(comment 1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e93.83\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e94.52\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e84.24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e90.86\u003c/p\u003e \u003cp\u003e(SD: 4.69)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e104.11*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e104.80*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e95.89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e101.60\u003c/p\u003e \u003cp\u003e(SD: 4.05)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThree documents of different lengths were produced by using different numbers of student's comments. Then, ChatGPT4 and OpenAI o1-preview were prompted to code each document in three independent tests. Coding density was estimated as the percentage of the number of coded words over the total number of words of the entire document. *The coding densities exceeded 100% due to overlapping coded text segments. This overlap caused the total word count of the coded segments to surpass the overall word count of the paragraph. SD: Standard deviation.\u003c/p\u003e \u003cp\u003eTo investigate this issue further, we used three additional documents ranging in lengths from 1185-word long to 3188-word long and prompted ChatGPT4 and OpenAI o1-preview to code them. The results in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e show that as the document length increased, the coding density of both models decreased.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eAn inverse relationship between document length and coding density.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"9\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eDocument length by word number\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"4\" nameend=\"c5\" namest=\"c2\"\u003e \u003cp\u003eChatGPT4 (coding density %)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"4\" nameend=\"c9\" namest=\"c6\"\u003e \u003cp\u003eOpenAI o1-preview (coding density%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMean\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003eMean\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1185 Words\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e10.88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18.39\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e15.61\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e14.96\u003c/p\u003e \u003cp\u003e(SD: 3.10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e45.57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e67.51\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e79.07\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e64.05\u003c/p\u003e \u003cp\u003e(SD: 13.90)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2251 Words\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5.64\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6.17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e13.19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e8.33\u003c/p\u003e \u003cp\u003e(SD: 3.44)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e32.27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e44.38\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e50.19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e42.28\u003c/p\u003e \u003cp\u003e(SD: 7.46)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3188 Words\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5.23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e3.98\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2.69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e3.97\u003c/p\u003e \u003cp\u003e(SD: 1.04)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e29.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e31.65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e29.51\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e30.15\u003c/p\u003e \u003cp\u003e(SD: 1.06)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eChatGPT4 and OpenAI o1-preview were prompted to code two published articles of different lengths: one of 1185-words long and the other 3188-words long. Coding density was estimated as the percentage of the number of coded words over the total number of words in the entire document. SD: Standard deviation\u003c/p\u003e \u003cp\u003eInterestingly, regardless of document length, the coding density of each document by ChatGPT4 was substantially smaller than that by OpenAI o1-preview (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). These results and the results in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e indicate that coding density of both models was inversely related to document length (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eCoding Consistency as Measured by Coding Density Across the Text: ChatGPT4 versus OpenAI o1-preview\u003c/h2\u003e \u003cp\u003eOne explanation for low coding density is that coding is not consistent throughout the text - that is, some sections receive more attention and therefore more codes, and other sections receive less or no coding. In such a case, determination of coding density of a document in a paragraph-by-paragraph way should be accurately captured if coding is applied unevenly across different sections of the document. To investigate this possibility, we reanalyzed the coding of the student comment document, in which each comment was presented as a separate paragraph and determined the coding density in each paragraph. The results in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e showed that ChatGPT4 mainly focused its coding on the first and second paragraphs and skipped numerous paragraphs in three independent tests (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). OpenAI o1-preview did not skip any paragraphs, but its coding density varied from paragraph to paragraph in tests 1 and 2 whereas it was consistently high in most paragraphs in test 3 (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). These results indicated inconsistency in coding density across the text and section skipping that varied from test to test.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cb\u003eCoding density in different sections of the text by ChatGPT4 and OpenAI o1-preview.\u003c/b\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eDocument Section\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eCoding Density (%) of ChatGPT4\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eCoding Density (%) of OpenAI o1-preview\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e61.22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e89.11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e39.45\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e85.03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e82.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e102.71*\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e25.34\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e44.52\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e15.75\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e93.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e75.34\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e71.23\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e40.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e52.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e102.67*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e49.33\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e74.67\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e30.67\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e60.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e24.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e101.33*\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e83.33\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e85.41\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e79.16\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e27.63\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e30.26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e30.26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e57.89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e53.95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e88.16\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e23.33\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e19.17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e20.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e75.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e46.42\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e80.36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e35.71\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e51.79\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e23.71\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e23.71\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e65.98\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e48.45\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e60.82\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e44.07\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e101.69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e57.63\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e91.53\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e17.21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e64.75\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e50.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e99.18\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e37.97\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e25.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e25.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e40.50\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e96.20\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParagraph 13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e40.32\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e24.19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e40.32\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e101.61*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e58.06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e103.22*\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eA document with thirteen paragraphs totaling 1149 words was fed to ChatGPT4 and OpenAI o1-preview for coding in three independent tests. The table lists the coding density of each of the paragraphs. *The coding densities exceeded 100% due to overlapping coded text segments. This overlap caused the total word count of the coded segments to surpass the overall word count of the paragraph.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eCoding Accuracy: A Comparative Analysis of ChatGPT4 and OpenAI o1-preview\u003c/h2\u003e \u003cp\u003eWe defined coding accuracy as the degree to which codes accurately capture and represent the text data's meaning. To evaluate the coding accuracy of ChatGPT4 and OpenAI o1-preview, ChatGPT4 and OpenAI o1-preview were each prompted to code a document in three separate tests. We then reviewed each code and the correspondent text segment to determine whether the code was acceptable or inaccurate enough to require revision according to the true meaning of the actual text segment.\u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e summarizes the results of our review of the codes generated by ChatGPT4 and OpenAI o1-preview. Both models exhibited consistent low coding accuracy across three tests even though the coding accuracy of OpenAI o1-preview was higher than that of ChatGPT4 (Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCoding Accuracy.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eChatGPT4\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eOpenAI o1-preview\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTest 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eTest 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eTest 3\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTotal No. of Codes Generated\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e34\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e47\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNo. of Codes Needed Revision\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e26\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNo. of Acceptable Codes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCoding Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e16.67%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e23.07%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e11.11%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e38.23%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e30.55%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e44.68%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThis table presents the results of coding reliability assessment, where codes assigned by ChatGPT1 and OpenAI o1-preview in three independent tests (see Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) were reviewed by human evaluators to determine the accuracy of each code in capturing the essence of the corresponding text segment.\u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e6\u003c/span\u003e presents illustrative examples of coding inaccuracies by ChatGPT-4 and OpenAI o1-preview model. For instance, the text segment \u0026ldquo;\u003cem\u003ethere should be more training which teaches important figures influencing healthcare decisions how to critically evaluate research articles\u003c/em\u003e\u0026rdquo; emphasizes the need for additional training. However, o1-preview coded it as \u0026ldquo;\u003cem\u003eNeed for training in critical evaluation\u003c/em\u003e,\u0026rdquo; overlooking the significance of the word \u0026ldquo;more\u0026rdquo; and thereby implying the absence of prior training.\u003c/p\u003e \u003cp\u003eIn another example, ChatGPT-4 coded the sentence \u0026ldquo;\u003cem\u003ePolitical motivations may also have played a factor and thus, as a result, much harm was produced\u003c/em\u003e\u0026rdquo; as \u0026ldquo;\u003cem\u003ePolitical influence on health decisions\u003c/em\u003e.\u0026rdquo; This code is inaccurate because it shows that ChatGPT4 failed to distinguish the different meanings between \"influence\" and \"motivation.\"\u003c/p\u003e \u003cp\u003eSimilarly, the statement \u0026ldquo;\u003cem\u003eunderscore the critical importance of maintaining rigorous standards for drug approval, irrespective of external pressures\u003c/em\u003e\u0026rdquo; was coded by OpenAI o1-preview as \u0026ldquo;\u003cem\u003eNeed for rigorous standards in drug approval\u003c/em\u003e.\u0026rdquo; This simplification omits the emphasis on maintaining existing standards and disregards the contextual qualifier \u0026ldquo;\u003cem\u003eirrespective of external pressures\u003c/em\u003e.\u0026rdquo;\u003c/p\u003e \u003cp\u003eAdditionally, the phrase \u0026ldquo;\u003cem\u003eLater studies were conducted which disproved the efficacy of HCQ in treating COVID and even showed alarming side effects such as cardiac arrest and ventricular arrhythmia\u003c/em\u003e\u0026rdquo; was coded by OpenAI o1-preview as \u0026ldquo;\u003cem\u003eDiscovery of HCQ\u0026rsquo;s Inefficacy and Side Effects\u003c/em\u003e.\u0026rdquo; This code fails to reflect the critical role of later studies in overturning earlier claims, thereby missing an important temporal and evidentiary nuance.\u003c/p\u003e \u003cp\u003eCollectively, these examples demonstrate that both ChatGPT-4 and OpenAI o1-preview frequently overlook contextual cues (e.g., \u0026ldquo;more\u0026rdquo;, \u0026ldquo;later studies\u0026rdquo;), miss semantic nuances (e.g., the importance of \u0026ldquo;maintaining\u0026rdquo; standards or differentiating \u0026ldquo;motivation\u0026rdquo; from \u0026ldquo;influence\u0026rdquo;), and tend to rely on superficial word pattern recognition rather than genuine comprehension of meaning.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab6\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 6\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExamples of codes by ChatGPT4 and OpenAI o1-preview and code evaluation and revision by human coders.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"1\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[Model] [Assigned Code] /Corresponding text segment:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Mismanagement of Research \u0026amp; Approval Process] / \"due to the low cost of the drug [hydroxychloroquine], cases reports, and poorly designed pilot studies, the promise of the drug with respect to its success in treating COVID was blown out of proportion...\"\u003c/p\u003e \u003cp\u003eEvaluation:\u003c/p\u003e \u003cp\u003eThe essence of the text segment highlights the issue with the claim about hydroxychloroquine's effectiveness in treating COVID-19 with poor evidence. Therefore, \"Mismanagement of Research \u0026amp; Approval Process\" is an inaccurate code.\u003c/p\u003e \u003cp\u003eNew code:\u003c/p\u003e \u003cp\u003eExaggerated claim of drug efficacy based on Questionable Evidence\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[Model] [Assigned Code] /Corresponding text segment:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Premature Implementation] / \"the above scenario resulted unfortunately due to an overzealousness to arrive at a solution to the COVID pandemic before conducting comprehensive studies to properly determine the benefit of the drug\"\u003c/p\u003e \u003cp\u003eEvaluation:\u003c/p\u003e \u003cp\u003e\"Premature Implementation\" doesn't fully capture the essence of the original text, because the text refers to rushed decision to use a drug for COVID-19 without thorough research and certainty about the drug's benefits.\u003c/p\u003e \u003cp\u003eNew code:\u003c/p\u003e \u003cp\u003eDrug Use Without Comprehensive Studies\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[Model] [Assigned Code] /Corresponding text segment:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Course Content Overview] \\ \u0026ldquo;The elective highlighted the fundamental differences and values of observational studies, retrospective studies, and RTCs.\u0026rdquo;\u003c/p\u003e \u003cp\u003eEvaluation:\u003c/p\u003e \u003cp\u003e\"Course Content Overview\" suggests a broad summary of all topics covered in the course. But this is not what the text says. Rather, the text emphasizes that the course highlighted the fundamental differences and values of various research methods. It specifically mentions observational studies, retrospective studies, and RCTs. Therefore, the course aimed to educate students about different research designs and their respective roles and importance in scientific research.\u003c/p\u003e \u003cp\u003eNew code:\u003c/p\u003e \u003cp\u003eEducation in Research Methodologies\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Researcher Challenges] \\ \u0026ldquo;and how researchers are not spared from becoming reactionary in an exceedingly challenging time.\u0026rdquo;\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eThe text suggests that researchers are not immune to reacting impulsively or emotionally, rather than critically thinking in a time that is extremely challenging. The text comments on the behavioral tendencies of researchers during crises and how even experts can fall victim to emotional or impulsive reactions during difficult times, rather than approaching issues with a critical and nuanced perspective. In contrast, the code \"Researcher Challenges\" misses the nuance of the text and is inaccurate.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eResearchers Are Not Immune from Behaving Reactionarily During Crises\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Importance of Quality Research] \\ \"It is very important to understand what goes into quality research.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eWhen a person says, \"It is very important to understand what goes into quality research,\" he/she means that comprehending the essential elements, principles and processes that constitute rigorous and reliable research is crucial. This includes clear objectives, methodological rigor, data quality, objective and unbiased interpretation of data, validity and reliability (findings are generalizable and replicable), and adhering to ethical standards. Therefore, \"Importance of Quality Research\" doesn't accurately convey the message of the text.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eUnderstanding how to Evaluate Research Quality\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Political influence on health decisions] \\ \"Political motivations may also have played a factor and thus, as a result, much harm was produced\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e: The code says, \"political influence\" instead of \"political motivation,\" which is what the text states. While ChatGPT4\u0026rsquo;s coding alludes to the indirect impact of political motivation and its resulting influence on decision making, it is noteworthy that \"influence\" and \"motivation\" have distinct meanings, and the text specifically says political motivation was responsible for the harm. Therefore, \"Political Influence Causing Harm\" is an inaccurate code.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003ePolitical Motivation Caused Harm\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Scientific Discovery and Innovation] \\ \"Regarding ACE2 decoy, the ingeniousness of this method captured my imagination, and I feel that this overall is the most promising approach to date.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eThe text expresses admiration for the innovative approaches in scientific research, particularly highlighting the potential of ACE2 decoy as the most promising new therapeutic strategies. The statement has an emotional undertone and appreciation for creativity. Therefore, \"Scientific Discovery and Innovation\" as a code is too generic and fails to capture the essence of the text.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eInspired by Ingenious Scientific Approaches\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Political vs. Scientific Decisions in Drug Approval] \\ \"Such decisions, appearing more political than scientific, underscore the critical importance of maintaining rigorous standards for drug approval, irrespective of external pressures or the urgency of health crises.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e\"Political vs. Scientific Decisions in Drug Approval\" is not the core message of the text. Rather, the text highlights the importance of continuing to uphold rigorous standards, which is a different emphasis than just \"need for rigorous standards.\" The text also says that an approval decision should not be influenced by external pressure.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eImperative to Disregard External Pressure and Adhere to Rigorous Standards for Drug Approval\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Rapid Findings] \\ \"...their findings must be interpreted with an understanding of their limitations.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e\"Rapid Findings\" is a nonsensical code because it has nothing to do with what the text conveys. In essence, the text says that findings are not absolute and should be interpreted with caution and avoid over-interpretation.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eMust Carefully Evaluate Findings\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[ChatGPT4] [Importance of Integrity and Transparency] \\ \"...it is imperative for researchers, publishers, and all involved parties to adhere to the highest standards of integrity, transparency, and diligence...\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e\"Importance of Integrity and Transparency\" is a good start, but it can be refined to capture the full essence of the statement. The word \"imperative\" in the statement indicates the importance for all to adhere to the highest standards of integrity, transparency, and diligence.\u003c/p\u003e \u003cp\u003eEvaluation:\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eNecessity of Accountability for All Stakeholders\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Discovery of HCQ's Inefficacy and Side Effects] \\ \"Later studies were conducted which disproved the efficacy of HCQ in treating COVID and even showed alarming side effects such as cardiac arrest and ventricular arrhythmia.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e\"Discovery of HCQ's Inefficacy and Side Effects\" is too generic to reflect the meaning of the text. The text emphasizes that it is the \"later studies\", which were well-designed, disapproved the efficacy of HCQ in treating COVID as claimed by earlier studies, and showed toxic side effects of HCQ.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eClaims of HCQ Efficacy for COVID Were Disapproved by New Studies\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Need for training in critical evaluation] \\ \"...there should be more training which teaches important figures influencing healthcare decisions how to critically evaluate research articles.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eThe code \"Need for training in critical evaluation\" fails to pay attention to the significance of the words \"more training\" in the text. The main point of the text is that there is a need for additional or increased training in critical evaluation skills. Therefore, this code is inaccurate.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eCall For More Training in Critically Evaluating Skills\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Use of multiple statistical tools] \\ \"The use of multiple statistical tools in these studies is important because it allows the investigators to get a clearer picture of the treatment's actual effects or lack thereof.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e\u0026ldquo;Use of multiple statistical tools\u0026rdquo; does not capture the essence of the text because it implies that statistical tools were either not used at al or only one was used. Rather, the text underscores the importance of using multiple statistical tools to obtain a more comprehensive and a clearer picture of the treatment's effects.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eNeed to Assess Treatment Efficacy Through the Use of Multiple Statistical Methods\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Need for rigorous standards in drug approval] \\ \"underscore the critical importance of maintaining rigorous standards for drug approval, irrespective of external pressures\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e: \u0026ldquo;Need for rigorous standards in drug approval\u0026rdquo; does not capture the essence of the text because it implies that there were no rigorous standards used. Rather, the text highlights the importance of continuing to uphold rigorous standards, which is a different emphasis than just \"need for rigorous standards.\" The text also says that an approval decision should not be influenced by external pressure.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eImperative To Disregard External Pressure and Adhere to Rigorous Standards for Drug Approval\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Need for Skills in Identifying Poor Research] \\ \"Understanding and identifying this is a skill that is important to future physicians who will be looking at medical research for the entirety of their careers.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e: It is important to note that \"this\" in the text refers to published studies of poor quality with unreliable data. Also, the text says that having the skill to identify such studies is important to future physicians. Therefore, \"Need for Skills in Identifying Poor Research\" does not capture the full essence of the text.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eFuture physicians Need the Skill to Identifying Poor Research\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Political Research] \\ \"How good research can be contorted into a political tool.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e: \"Political Research\" is a wrong code because the text is not about political research but about using research findings for political purposes.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eUse Good Research as Political Tool\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Peer Interpretations and Discourse] \\ \"I really enjoyed hearing my peers\u0026rsquo; interpretations of research throughout this elective, and it led to some remarkably interesting discourse.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e: \"Peer Interpretations and Discourse\" does not capture the full essence of the text because the text expresses appreciation of listening to peers' unique interpretations of research, which enriches one's own understanding. Additionally, these varying perspectives led to engaging, insightful, and thought-provoking conversations. In short, listening to and discussing with peers about issues is enjoyable and intellectually stimulating.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eHigh Value of Learning from Peers\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Better Understanding of Clinical Decisions] \\ \"This course also helped me better understand the treatment decisions that the doctors I have worked with made while I was on clerkships a year ago.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e: \"Better Understanding of Clinical Decisions\" is a generic code because the person now realizes that the knowledge he has learned from this course helps him to better understand the treatment decisions made by doctors with whom he worked previously. In other words, his newly acquired knowledge from the course enables him to comprehend the reasoning behind decisions that seemed unclear or puzzling a year ago.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eLearning in the Course Enhanced Understanding of Past Clinical Decisions\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Necessity for High Standards in Health Crises] \\ \"It is imperative for researchers, publishers, and all involved parties to adhere to the highest standards of integrity, transparency, and diligence, given the direct consequences their findings can have on public health measures and global strategies.\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation\u003c/b\u003e: \"Necessity for High Standards in Health Crises\" does not fully reflect the meaning of the text because it failed to capture the meaning of the word \"adhere\" in the text. Therefore, it is about the imperative of adhering to the highest standards to prevent negative consequences on public health measures and global strategies.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eImperative to adhere to the highest standards\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e[Model] [Assigned Code] /Corresponding text segment\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e[OpenAI o1-preview] [Emphasis on Randomized Controlled Trials (RCTs)] \\ \"Due to the stakes involved, as much as possible, it should only be the results of randomized controlled trials which are allowed to influence healthcare decisions\"\u003c/p\u003e \u003cp\u003e\u003cb\u003eEvaluation: \"\u003c/b\u003eEmphasis on Randomized Controlled Trials (RCTs)\" does not fully capture the meaning in the text. The text emphasizes the importance of making health decisions based on findings from RCTs.\u003c/p\u003e \u003cp\u003e\u003cb\u003eNew code\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eHealth Decision Should Rely on Randomized Controlled Trials\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eCodes generated by ChatGPT4 and OpenAI o1-preview are evaluated and revised by human coders against the original corresponding text segments.\u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eIn this study, we aimed to evaluate the potential of ChatGPT-4 and OpenAI\u0026rsquo;s o1-preview model in automating the text coding process for thematic analysis. We selected these two models because both are based on the GPT architecture developed by OpenAI, but o1-preview is more advanced due to its integration of a chain-of-thought reasoning mechanism that enhances problem-solving and analytical capabilities in large language models [\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe chain-of-thought mechanism allows the model to solve complex problems by breaking them down into simpler, intermediate steps, which are then used iteratively to generate subsequent reasoning. This process enables the model to refine its output, correct earlier missteps, and improve overall accuracy.\u003c/p\u003e \u003cp\u003eAccordingly, we sought to assess the suitability of both models for text coding and to determine whether the chain-of-thought mechanism would confer a performance advantage to the o1-preview model. Specifically, we evaluated the models using three metrics: coding density, coding consistency across the text, and coding accuracy.\u003c/p\u003e \u003cp\u003eBoth coding density and coding consistency measures the degree to which codes are applied uniformly and reliably throughout the data, and they are crucial in thematic analysis to ensure reliable, valid, and trustworthy findings [\u003cspan additionalcitationids=\"CR9 CR10\" citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. As the data in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e shows, in the 3 independent tests we performed, ChatGPT-4 consistently identified fewer text segments and exhibited a lower coding density (19.58\u0026ndash;25.50%) compared to the coding density of OpenAI o1-preview (51.74\u0026ndash;80.85%). However, as document length increased, coding density of both models, especially ChatGPT4, decreased (Tables\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e and \u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Furthermore, regardless of document length, ChatGPT-4 showed a consistently lower coding density than OpenAI o1-preview (Tables\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e and \u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). This is like the finding of other investigators showing that ChatGPT-4 could not code a large dataset at once [\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e]. In this study, the low coding density of ChatGPT4 apparently can be explained by ChatGPT-4's tendency to concentrate its coding efforts on the initial paragraphs while skipping numerous subsequent sections in a document (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). There is no apparent way to speculate why section skipping occurred because skipping was random as the sections skipped varied from test to test \u003cb\u003e(\u003c/b\u003eTable\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). Nevertheless, these results indicate an issue of uneven coding attention, and it undermines the reliability of ChatGPT4 for thematic analysis as it will lead to biased or incomplete theme identification.\u003c/p\u003e \u003cp\u003eWhile not skipping any paragraphs in a document, OpenAI o1-preview exhibited variability in coding density from test to test, indicating inter-test variability in coding density. The implication here is that OpenAI o1-preview requires calibration or iterative prompting to achieve stable performance across all sections of the entire text consistently.\u003c/p\u003e \u003cp\u003eThese findings indicate that both ChatGPT4 and OpenAI o1-preview are limited in their scalability and utility in extensive qualitative studies concerning large datasets. Given that both models performed extensive coding of short text, it suggests that coding of longer documents by these models requires additional text segmentation to ensure comprehensive coding.\u003c/p\u003e \u003cp\u003eThe examples of coding inaccuracies in Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e6\u003c/span\u003e reveal significant limitations in their coding accuracy. These models struggle to capture nuanced contextual cues and distinguish meanings between terms, and moreover, they focus on surface-level patterns rather than deeper text meanings as discussed below and are prone to oversimplification and misinterpretation.\u003c/p\u003e \u003cp\u003eIn this study, we use the term deep text meanings to refer to underlying themes, nuances, and contextual implications that are not immediately evident through superficial semantic interpretation. These include, but are not limited to, pragmatic elements such as speaker intention, tone, and implicature, as well as discourse-level features such as power dynamics, social context, and cultural references. Because current large language models (LLMs) are unable to adequately process these dimensions, their outputs require rigorous human review and are of limited utility in domains that demand nuanced interpretive understanding. To enhance their suitability for qualitative analysis, LLMs must be further refined to capture not only semantic content but also the pragmatic and discourse-level features embedded in text.\u003c/p\u003e \u003cp\u003eWe use the term meaning to refer to the complex and nuanced understanding of language and context that characterizes human cognition. While large language models (LLMs) can produce outputs that appear coherent\u0026mdash;and at times even insightful\u0026mdash;this coherence often creates an illusion of understanding. In reality, LLMs do not possess genuine comprehension; rather, they generate responses based on probabilistic associations between tokens.\u003c/p\u003e \u003cp\u003eThis distinction is critical, as it highlights the limitations of current LLMs in grasping the depth and complexity of human language. Their inability to code text accurately offers further evidence that LLMs do not truly understand the meanings of words, phrases, or sentences\u0026mdash;they merely simulate understanding through statistical prediction. The coding inaccuracies observed in this study suggest that when applied to texts containing complex ideas or sensitive topics, these models are likely to produce incomplete or misleading representations of the data.\u003c/p\u003e \u003cp\u003eThese limitations underscore the broader challenge of using LLMs for tasks that require text comprehension and interpretation of nuanced meanings. However, our findings do not entirely preclude the use of LLMs in thematic analysis. As suggested by others [\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e], LLMs may \u0026ldquo;serve as an additional member of the analysis team, contributing to researcher triangulation through knowledge building and sensemaking.\u0026rdquo;\u003c/p\u003e \u003cp\u003eIt is said that most tokens in a reasoning chain in current LLMs are generated solely for language fluency and have little to do with reasoning [\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e]. Therefore, for LLMs to be able to reason they need to be significantly improved to \u0026ldquo;have the freedom to reason without any language constraints, and then translate their findings into language only when necessary.\u0026rdquo; [\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e]. To achieve this kind of improvement, it is suggested that to pretrain LLMs with continuous thoughts to enable them to generalize more effectively across a wider range of reasoning scenarios [\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e]. In the future, perhaps, text coding accuracy can be used, among other methods, to measure reasoning capability of improved LLMs.\u003c/p\u003e \u003cp\u003eThis study has several limitations that should be addressed in future research. First, our analysis focused on two specific large language models (ChatGPT-4 and OpenAI o1-preview) and the results may not generalize to other models or future iterations. Second, we analyzed a limited set of documents and topics; future studies should explore a broader range of qualitative data types and subject matter to better assess the generalizability of our findings.Third, our analysis relied primarily on Document 1, which consisted of student comments that were generally brief, composed of short sentences, and limited in both content richness and linguistic complexity. Future research should incorporate texts that are intentionally constructed to exhibit greater linguistic and conceptual depth. Such texts typically include multi-clause sentences with embedded structures, abstract or context-dependent language, pragmatic cues, semantic ambiguity, subtle presuppositions, and culturally informed references. Linguistic complexity should be understood to include syntactic variation, lexical and semantic richness, and pragmatic and discourse-level features. Fourth, this study focused exclusively on a single phase of thematic analysis \u0026ndash; that is,text coding. Future research should prompt LLMs to perform all six phases of thematic analysis to evaluate their broader applicability in qualitative inquiry.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThis study highlights the promising yet currently limited role of large language models in thematic analysis. OpenAI o1-preview demonstrated superior performance in coding density and consistency compared to ChatGPT-4, yet both models fell short in coding accuracy. Furthermore, our findings provide new evidence that even the most advanced OpenAI o1-preview lacks the capability of comprehending nuanced semantic meanings in text. While LLMs can significantly reduce the time and effort required for initial text coding, their integration into qualitative research necessitates careful consideration of their limitations. By advancing model training, enhancing contextual understanding, and adopting hybrid analytical frameworks, LLMs may become valuable tools that augment the capabilities of researchers, ultimately contributing to more efficient and comprehensive thematic analyses.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding:\u003c/strong\u003e The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll data generated or analyzed during this study are included in this published article.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthics, Consent to Participate, and Consent to Publish declarations:\u003c/strong\u003e not applicable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eBraun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006:3(2);77\u0026ndash;101. https://doi.org/10.1191/1478088706qp063oa\u003c/li\u003e\n\u003cli\u003eLiebenberg L, Jamal A, Ikeda J. Extending youth voices in a participatory thematic analysis approach. Int J Qual Methods. 2020:19;1609406920934614. https://doi.org/10.1177/1609406920934614\u003c/li\u003e\n\u003cli\u003eMaguire M, Delahunt B. Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. AISHE J. 2017:3;3351-33514.\u003c/li\u003e\n\u003cli\u003eXu W, Zammit K. Applying thematic analysis to education: a hybrid approach to interpreting data in practitioner research. Int J Qual Methods. 2020:19. https://doi.org/10.1177/1609406920918810\u003c/li\u003e\n\u003cli\u003eNaeem M, et al. A Step-by-Step Process of Thematic Analysis to Develop a Conceptual Model in Qualitative Research. Int J Qual Methods. 2023:22. https://doi.org/10.1177/16094069231205789\u003c/li\u003e\n\u003cli\u003eThomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval. 2006:27(2);237-246. https://doi.org/10.1177/1098214005283748\u003c/li\u003e\n\u003cli\u003eElliott V. Thinking about the coding process in qualitative data analysis. Qual Rep. 2018:23(11);2850\u0026ndash;2861. https://doi.org/10.46743/2160-3715/2018.3560\u003c/li\u003e\n\u003cli\u003eMorse JM, Mitcham C. Exploring qualitatively-derived concepts: inductive-deductive pitfalls. Int J Qual Methods. 2002:1(4);28-35. https://doi.org/10.1177/160940690200100404\u003c/li\u003e\n\u003cli\u003ePatton MQ. Qualitative research and evaluation methods (4th ed.). Thousand Oaks, CA: Sage.\u003c/li\u003e\n\u003cli\u003eOlson JD, et al. Applying Constant Comparative Method with Multiple Investigators and Inter-Coder Reliability. Qual Rep. 2016:21(1);26-42. https://doi.org/10.46743/2160-3715/2016.2447\u003c/li\u003e\n\u003cli\u003eRichards KAR, Hemphill MA. A Practical guide to collaborative qualitative data analysis. J Teach Phys Educ. 2017:37(2);225-231. https://doi.org/10.1123/jtpe.2017-0084\u003c/li\u003e\n\u003cli\u003eDeSantis L, Ugarriza D. The concept of theme as used in qualitative nursing research. West J Nurs Res. 2000:22(3);351\u0026ndash;372. https://doi.org/10.1177/019394590002200308\u003c/li\u003e\n\u003cli\u003eScriven M. Pros and cons about goal-free evaluation. Eval Pract. 1991:12(1);55\u0026ndash;63. https://doi.org/10.1016/0886-1633(91)90024-r\u003c/li\u003e\n\u003cli\u003ePeters ME, et al. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018, pages 2227\u0026ndash;2237, New Orleans, Louisiana. Association for Computational Linguistics. https://aclanthology.org/N18-1202/\u003c/li\u003e\n\u003cli\u003eVaswani A, et al. Attention is all you need. Advances in neural information processing systems, 2017:30. https://doi.org/10.48550/arXiv.1706.03762\u003c/li\u003e\n\u003cli\u003eSennrich R, Haddow B, Birch A. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pages 1715\u0026ndash;1725, Berlin, Germany. Association for Computational Linguistics. https://aclanthology.org/P16-1162/\u003c/li\u003e\n\u003cli\u003eKudo T, Richardson J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXivpreprintarXiv:1808.06226. 2018. https://doi.org/10.48550/arXiv.1808.06226\u003c/li\u003e\n\u003cli\u003eHaslett A. Cai ZG. How much semantic information is available in large language model tokens? PsyArXiv Preprints. 2025. https://doi.org/10.31234/osf.io/nhwdz\u003c/li\u003e\n\u003cli\u003eRajaraman N, Jiao J, Ramchandran K. Toward a theory of tokenization in llms. arXivpreprintarXiv:2404.08335. https://doi.org/10.48550/arXiv.2404.08335\u003c/li\u003e\n\u003cli\u003eMitchell M, Krakauer DC. The debate over understanding in AI\u0026rsquo;s large language models, Proc. Natl. Acad. Sci. U.S.A. 120:(13):e2215907120, https://doi.org/10.1073/pnas.2215907120 (2023.\u003c/li\u003e\n\u003cli\u003eMitchell M. How do we know how smart AI systems are? Science. 2023:381(6654);adj5957. https://doi.org/10.1126/science.adj5957\u003c/li\u003e\n\u003cli\u003eDevlin J, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) 2019 Jun (pp. 4171-4186). https://aclanthology.org/N19-1423.pdf\u003c/li\u003e\n\u003cli\u003eRadford A, et al. Improving Language Understanding by Generative Pre-Training. OpenAI blog, 2018, page 12. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf\u003c/li\u003e\n\u003cli\u003eZhang H, et al. Redefining qualitative analysis in the AI era: Utilizing ChatGPT for efficient thematic analysis. arXivpreprintarXiv:2309.10771. 2023. https://doi.org/10.48550/arXiv.2309.10771\u003c/li\u003e\n\u003cli\u003eKatz A, et al. Exploring the efficacy of ChatGPT in analyzing Student Teamwork Feedback with an existing taxonomy. arXivpreprintarXiv:2305.11882. 2023. https://doi.org/10.48550/arXiv.2305.11882\u003c/li\u003e\n\u003cli\u003eGao J, et al. CoAIcoder: Examining the effectiveness of AI-assisted human-to-human collaboration in qualitative analysis. ACM Trans Comput Hum Interact. 2023:31(1);1-38. https://doi.org/10.48550/arXiv.2304.05560\u003c/li\u003e\n\u003cli\u003eHamilton L, et al. Exploring the Use of AI in Qualitative Analysis: A Comparative Study of Guaranteed Income Data. Int J Qual Methods. 2023:22. https://doi.org/10.1177/16094069231201504\u003c/li\u003e\n\u003cli\u003eFiannaca AJ, et al. Programming without a programming language: challenges and opportunities for designing developer tools for prompt programming. Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg Germany, 2023-04-19). Article235. https://doi.org/10.1145/3544549.3585737\u003c/li\u003e\n\u003cli\u003eZamfrescu-Pereira J., et al. Why Johnny can\u0026rsquo;t prompt: how non-AI experts try (and fail) to design LLM prompts. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI \u0026apos;23). Association for Computing Machinery, New York, NY, USA, Article 437. 2023. https://doi.org/10.1145/3544548.3581388\u003c/li\u003e\n\u003cli\u003eMorgan DL. Exploring the use of artificial intelligence for qualitative data analysis: the case of ChatGPT. Int J Qual Methods. 2023:22. https://doi.org/10.1177/16094069231211248\u003c/li\u003e\n\u003cli\u003eAndersson M, McIntyre D. Can ChatGPT recognize impoliteness? An exploratory study of the pragmatic awareness of a large language model. J Pragmat. 2025:239:16-36. https://doi.org/10.1016/j.pragma.2025.02.001\u003c/li\u003e\n\u003cli\u003eFrantz CK, et al. Subjective Interpretation of Directive Terms: Methodology and Preliminary Results,\u0026quot; 2015 International Conference on Culture and Computing (Culture Computing), Kyoto, Japan, 2015, pp. 137-144. doi: 10.1109/Culture.and.Computing.2015.40. \u003c/li\u003e\n\u003cli\u003eKazemian R, et al. Beyond sentence grammar: Persian directives in interaction. J Pragmat. 2025:236:40-59. https://doi.org/10.1016/j.pragma.2024.12.010\u003c/li\u003e\n\u003cli\u003eWebb T, Holyoak KJ, Lu H. Emergent analogical reasoning in large language models. Nat Hum Behav. 2023:7;1526\u0026ndash;1541. https://doi.org/10.1038/s41562-023-01659-w\u003c/li\u003e\n\u003cli\u003eLewis M, Mitchell M. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXivpreprintarXiv:2402.08955. 2024. https://doi.org/10.48550/arXiv.2402.08955\u003c/li\u003e\n\u003cli\u003eMirzadeh I, et al. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXivpreprintarXiv:2410.05229. 2024. https://doi.org/10.48550/arXiv.2410.05229\u003c/li\u003e\n\u003cli\u003eHan Z, et al. Beyond text generation: assessing large language models\u0026rsquo; ability to reason logically and follow strict rules. AI. 2025:6(1);12. https://doi.org/10.3390/ai6010012\u003c/li\u003e\n\u003cli\u003eSullivan R, Elsayed N. Can large language models act as symbolic reasoners? arXivpreprintarXiv:2410.21490. https://doi.org/10.48550/arXiv.2410.21490\u003c/li\u003e\n\u003cli\u003eTurobov A, Coyle D, Harding V. Using ChatGPT for thematic analysis. arXivpreprintarXiv:2405.08828. 2024. https://doi.org/10.48550/arXiv.2405.08828\u003c/li\u003e\n\u003cli\u003eWei J, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst. 2022:35;24824-24837. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf.\u003c/li\u003e\n\u003cli\u003eShore CK, Worku TL, Kahn J. Rare disease innovation at the FDA-opportunities for implementation. JAMA. 2024:332(23);1973\u0026ndash;1974. https://doi.org/10.1001/jama.2024.20358\u003c/li\u003e\n\u003cli\u003eGuare EG, et al. Rates of sudden unexpected infant death before and during the COVID-19 pandemic. JAMA Netw Open. 2024:7(9);e2435722. https://doi.org/10.1001/jamanetworkopen.2024.35722\u003c/li\u003e\n\u003cli\u003eLee VV, et al. Harnessing ChatGPT for thematic analysis: are we ready? J Med Internet Res. 2024:26;e54974. https://doi.org/10.2196/54974\u003c/li\u003e\n\u003cli\u003eHao S, et al. Training large language models to reason in a continuous latent space. arXivpreprint arXiv:2412.06769. 2024. https://doi.org/10.48550/arXiv.2412.06769\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Large language model, ChatGPT, Text coding, Thematic analysis, OpenAI o1-preview, Comprehension, Reasoning","lastPublishedDoi":"10.21203/rs.3.rs-5937488/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5937488/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eIn practice, thematic analysis of text involves six stages, among which text coding is particularly cognitively demanding, labor-intensive, and time-consuming. This study investigates and compares the potential of two large language models (LLMs), namely ChatGPT-4 and OpenAI o1-preview, to perform text coding, with the goal of reducing the time and effort required by human researchers. Our results indicate that both models exhibit decreased coding comprehensiveness as document length increases, and both demonstrate low coding accuracy, primarily due to limitations in textual comprehension and reasoning. These findings highlight significant challenges in using LLMs to support thematic analysis, emphasizing the need for human oversight and rigorous validation to ensure analytic accuracy and validity.\u003c/p\u003e","manuscriptTitle":"Can Large Language Models Be Used to Code Text for Thematic Analysis? An Explorative Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-30 11:02:32","doi":"10.21203/rs.3.rs-5937488/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-05-07T13:11:04+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-05-03T14:42:32+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"122142116719834256247377150842970817752","date":"2025-05-03T14:36:32+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-04-28T14:29:33+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"220303302480956917024102949968572771916","date":"2025-04-28T12:48:38+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-04-28T10:28:16+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-04-28T10:27:54+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-04-25T11:29:10+00:00","index":"","fulltext":""},{"type":"submitted","content":"Discover Artificial Intelligence","date":"2025-04-19T00:27:09+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"08039746-32c6-4449-9ffb-1347c35abed4","owner":[],"postedDate":"April 30th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-07-28T16:00:52+00:00","versionOfRecord":{"articleIdentity":"rs-5937488","link":"https://doi.org/10.1007/s44163-025-00441-3","journal":{"identity":"discover-artificial-intelligence","isVorOnly":false,"title":"Discover Artificial Intelligence"},"publishedOn":"2025-07-24 15:57:27","publishedOnDateReadable":"July 24th, 2025"},"versionCreatedAt":"2025-04-30 11:02:32","video":"","vorDoi":"10.1007/s44163-025-00441-3","vorDoiUrl":"https://doi.org/10.1007/s44163-025-00441-3","workflowStages":[]},"version":"v1","identity":"rs-5937488","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5937488","identity":"rs-5937488","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}