AI vs. Humans: A Comparative Analysis of Time, Cost, and Performance on a Clinical Code Conversion Task

AI vs. Humans: A Comparative Analysis of Time, Cost, and Performance on a Clinical Code Conversion Task | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article AI vs. Humans: A Comparative Analysis of Time, Cost, and Performance on a Clinical Code Conversion Task Carly Hudson, Marcus Randall, Candice Bowman, Anu Joy, Adrian Goldsworthy This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5143761/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background: Healthcare services generate and store large quantities of data which historically required significant resources to manually analyse and gain meaningful insights to assist in informing healthcare policy and system design. Recent advancements in automation tools, such as generative artificial intelligence (GAI), provides new opportunities to disencumber human labour and thinking for difficult tasks. This study explores the potential utilisation of GAI for a healthcare data analysis task, specifically, the conversion of clinical data from one diagnostic classification system to another (i.e., SNOMED-CT-AU to the ICD-10-CM). Additionally, it examines the time and cost benefit of performing this using GAI when compared to a human rater. Methods: Conversions were completed on SNOMED-CT-AU codes to the ICD-10-CM using three methods: 1) manual conversion using the National Library of Medicine’s I-MAGIC tool, 2) conversion using ChatGPT 4o, and 3) conversion using Claude Sonnet 3.5. The accuracy of the GAI tools was mapped against the manually extracted codes, and examined in terms of a perfect, partial, or incorrect match. Researchers’ time was recorded and extrapolated to calculate and compare the cost associated with each method. Results: When compared to the manually extracted codes, Claude Sonnet 3.5 yielded the highest level of agreement over ChatGPT 4o, whilst also being the most time and cost effective. Conclusion: GAI tools have greater utility than they have currently been given credit for. The automation of big data healthcare analytics, whilst still the domain of humans is increasingly capable of being undertaken using automation tools with low barriers to entry. The further development of the capability of GAI alongside the capability of the healthcare system to use it appropriately has the potential to result in significant resource savings. Medical Informatics Generative Artificial Intelligence SNOMED ICD-10 diagnostic coding data analytics Figures Figure 1 Figure 2 Figure 3 Figure 4 1 Introduction The volume of data hospital and health services generate each year far exceeds the analytical capacity of humans [1]. Murphy [1] estimated hospitals produce 50 petabytes (equivalent of 50,000,000 gigabytes) of data per year, 97% of which remains unanalysed or unused. Electronic health records, contain data such as patient demographics, images, clinical notes, and pathology results, providing opportunities for retrospective analysis enabling data-informed decisions and predictions on service utilisation [1, 2]. Increasingly financially restricted and resource limited health care systems are unable to manually process large datasets restricting the potential to improve healthcare system efficiency [1, 3]. GAI has potential to support the analysis of large-scale datasets within healthcare settings. The nature of healthcare has traditionally been a very human focused activity, and as such, it has often resisted large-scale efforts for effective automation, particularly in the form of clinical and administrative decision making. A recent literature review by Li et al. [4] has identified some of the key areas in which GAI are starting to make an impact within healthcare, including generating discharge summaries [5], determining appropriate screening procedures for a patient [6], and answering clinical questions and providing medical education [7-10]. The increasing complexity of healthcare challenges around the world necessitates new data analysis processes which expeditiously and efficiently makes use of the large datasets available to healthcare systems. The recent advent of automation tools, such as generative artificial intelligence (GAI) provides new opportunities to efficiently complete large healthcare data analytics tasks. The widespread implementation and usage of GAI is one of the most rapid advancements in technology we have seen in recent years. OpenAI’s ChatGPT [11] is currently one of the most popular GAI tools available with over 100 million online users per week [12]. ChatGPT enables a user to enter in a prompt, command, or question, and ChatGPT will provide a response. Its interface is driven by a large language model (LLM), a form of natural language processing able to learn and improve its conversational abilities using a combination of self and semi-structured training [13]. Processing of data is achieved by the application of large-scale neural networks, using feed-forward and convolutional architectures [13]. Following the widespread success of ChatGPT, competitors have since launched other GAI tools available to the general public, including Google Gemini [14], Microsoft Copilot [15], and Claude [16]. Despite GAI tools’ benefits, they currently are limited by their lack of context and human understanding. Accuracy and completeness of outputs are limited by the data available to the GAI model (i.e., what it has been trained on, its access to real time searching), which may be biased or inaccurate. GAI tools also have limited knowledge of more specialised topics, resulting in a tendency to “hallucinate”, a phenomenon whereby a GAI tool generates information to fill knowledge gaps, decreasing the accuracy of outputs. Healthcare professionals require up to date understanding of the currently dynamic limitations of GAI to optimally select tasks which it is likely to excel at and prompt it appropriately. A key challenge in analysing big healthcare data is the consistency of data recording between health services. Standardised diagnostic coding systems aid in keeping clinical information consistent, by providing a universal language by which diagnoses can be coded across health care services. The “Systematized Nomenclature of Medicine – Clinical Terms” (SNOMED-CT) [17] is a diagnostic coding system utilised by 48 countries (as of August 2024) [18] to capture detailed clinical information about procedures, diseases, and clinical findings. SNOMED-CT presents diagnoses in the form of a numeric code (e.g., “230690007”) and a corresponding descriptor (e.g., “Stroke”). SNOMED-CT uses a polyhierarchical structure, wherein any given code may belong to one or more ‘parent’ codes (e.g., “asthma” may be categorised under both “respiratory diseases” and “allergic conditions”). Whilst SNOMED-CT provides a comprehensive framework for diagnostic coding at a patient-level, symptoms, procedures, and findings, the framework is complex and difficult to utilise for those with limited training. The International Statistical Classification of Diseases and Related Health Problems (ICD) [19] is currently the global standard for coding diagnostic information. ICD focuses on the classification of diseases, disorders, and causes of death using alphanumeric codes. These are determined using a hierarchical system, wherein codes are categorised by chapters (e.g., F – “mental and behavioural disorders”), and then into narrower categories as more detail is provided (e.g., “F10 – mental and behavioural disorders due to use of alcohol”; “F10.1 – mental and behavioural disorders due to use of alcohol: harmful use”). Whilst the ICD provides less detail than SNOMED-CT, the broader categories facilitate population health analytics, and provides a standard for international health system comparison. To the authors’ knowledge, it is yet to be explored whether GAI can assist in the conversion of clinical data from one diagnostic coding language to another, such as from SNOMED-CT to the ICD. Such conversions require specialised knowledge of clinical coding and are time consuming to complete manually. The completion of diagnostic code conversion tasks using AI models may allow for less qualified staff to complete the task in less time, thereby reducing the cost of data processing. The primary aim of this research, therefore, is to examine whether publicly accessible GAI tools, namely ChatGPT 4o and Claude Sonnet 3.5, can accurately convert clinical diagnostic codes from SNOMED—CT to ICD-10. In addition, the following sub-aims will be addressed: i. Compare the level of agreement between ChatGPT 4o and a human rater. ii. Compare the level of agreement between Claude Sonnet 3.5 and a human rater. iii. Compare the level of agreement between ChatGPT 4o and Claude Sonnet 3.5. iv. Examine the economic benefit, in terms of time and labour cost, of using GAI to complete this task, in comparison to a human rater. 2 Method The SNOMED codes used in this study originate from a broader emergency department dataset, retrieved as part of a study investigating mental health presentations to hospital emergency departments (ethics approval: HREC/2023/QGC/95219). This dataset consists of 19,764 unique SNOMED-CT-AU numeric codes (e.g., 48694002) and SNOMED-CT-AU names (e.g., “Anxiety reaction”) representing the diagnoses made to the ED over a three-year period (August 2020 to August 2023). The current evaluation utilises a 10% subset of this data ( N = 1976). To convert the SNOMED-CT-AU codes to ICD-10-CM, a three-phase approach was used. Firstly, codes were manually converted by human raters. Secondly, the codes were converted using ChatGPT 4o. Finally, the same set of codes were converted using Claude Sonnet 3.5. At the time of analysis both GAI tools required paid subscriptions. 2.1 Phase 1: Manual Conversion of SNOMED-CT-AU Codes The SNOMED-CT-AU codes were manually converted by a team of three raters (AG = 800; AJ = 644; CH = 532). Conversions were completed using the Interactive Map-Assisted Generation of ICD Codes (I-MAGIC) algorithm ( https://imagic.nlm.nih.gov/imagic/code/map ), an online tool which provides mapping between the two diagnostic coding systems [ 20 ]. Codes were entered into the online tool in the format of “SNOMED-CT-AU Name (SNOMED-CT Code)” (e.g., “Anxiety reaction (48694002)”). The resulting ICD-10-CM equivalent code was then extracted. In cases where multiple ICD-10-CM codes were returned, only the first one was extracted and considered for analysis. In some cases (e.g., “Chest clear (301708006)”), the I-MAGIC algorithm found no equivalent ICD-10-CM code (“This SNOMED CT term cannot be mapped to ICD-10-CM”). There were also instances where SNOMED-CT codes could not be found in the I-MAGIC tool. Given that the dataset was using the Australian extension of SNOMED-CT, and the mapping tool uses the standard SNOMED-CT list, it is likely that this is due to some codes being region-specific [ 21 ]. In these cases, it was noted that an equivalent could not be found. 2.2 Phase 2: Conversion of SNOMED-CT-AU Codes using ChatGPT4o ChatGPT 4o [ 11 ] was used to automatically convert the SNOMED-CT-AU codes and names to the ICD-10-CM codes (completed in August 2024). The Excel file of SNOMED-CT-AU codes and names was uploaded to ChatGPT 4o. A prompt was trialled and adapted utilising an iterative approach to efficiently convert the codes and names, whilst avoiding hallucinations (providing false information) and data processing errors. It was necessary to state that ChatGPT 4o could take as much time as required to complete this task, otherwise the message would time out and cease to produce output. It was also identified there was a limit to the number of codes that could be processed at one time without increasing the number of errors such as incorrect conversions, creating new input data, and skipping codes within the dataset. As a result, 25 codes, split into five batches of five codes, were converted at a time. The following final prompt was used: “Please manually and sequentially convert the SNOMED CT AU codes into ICD-10-CM codes and names which have not been converted yet. If you need to, complete this task in batches of 5. Take as long as you need. Do not hallucinate, and only convert codes which have been provided to you. Do not create new codes to convert. Provide an update after 5 batches have been completed. Provide a .xlsx file at the end.” (Fig. 1 ). Following each conversion of 25 codes, ChatGPT4o needed to be prompted to continue; “ Please continue. ” (Fig. 2 ). Once all the codes had been completed, ChatGPT4o provided the requested .xlsx file containing the original input data with the corresponding final output data. 2.3 Phase 3: Conversion of SNOMED-CT-AU Codes Using Claude Sonnet 3.5 At the time of conducting this study (September 2024), Claude Sonnet 3.5 [ 16 ] was unable to import or export Excel files. As a result, data was copy and pasted into the chat as a means of providing the input data. However, there were limitations on the maximum message length that was permissible; as a result, data was provided in three batches of 500, and a final batch of 476. The prompt was developed in a similar manner to Phase 2 utilising an iterative approach. Claude Sonnet 3.5 required fewer directions in order to accurately undertake the requested task, allowing for the prompt to be easily developed within a relatively short timeframe. However, the limits on the length of the output message that Claude Sonnet 3.5 provided restricted the amount of data it was able to process to 50 codes at a time. When trialling the prompt, it was noted that Claude Sonnet 3.5 needed to be explicitly told not to skip lines, as in reflected in final the prompt: “ Please convert these SNOMED CT AU codes and names into ICD-10-CM codes and names. Convert them in sequential order starting from the top and ensuring you do not skip any. I have provided 500 codes in total and need 500 responses at the end. Start with the first 50. If you are unable to convert a code, please state this. ” (Fig. 3 ). As Claude Sonnet 3.5 was unable to directly export an Excel file, it was asked to produce R code which could be run to generate an output file; “Can you write the code for R to create an Excel file of this data. Write it out in full so it exports all entries 1-100 ” Fig. 4). Due to the limits on the output message length, this was completed in batches of 100. The code was then run using R Studio to produce the final output file. Figure 4 – Claude Sonnet 3.5 producing R Studio code to generate .xlsx file 2.4 Statistical Analysis Six rows of data were removed from the analysis where at least one of the three tools was unable to perform a conversion, leaving 1970 cases included in the final analysis. For the purpose of analysis, the ICD codes were split into three components; 1) letter code (“level one”), 2) major numeric code (before decimal point; “level 2”), and 3) minor numeric code (after decimal point; “level 3”) (Table 1 ). A pattern matching program was developed using the C programming language to look for partial and perfect matches for the 1970 cases between: manual coding and ChatGPT 4o; manual coding and Claude Sonnet 3.5; and ChatGPT 4o and Claude Sonnet 3.5. The program converted the codes from each method into its component parts. Some manual editing was necessary to account for and fill in missing values (using null codes). Each component was assigned a weighted value, which can be adjusted depending on purpose of analysis (Table 1 ). In this analysis, two sets of weights were utilised (Table 1 ). In the first instance, weights 1-2-3 were used, with three points allocated for complete matches, two points for partial matches at level two, and one point for partial matches at level one. In the second instance, to enable detection of only perfect matches, weights 0-0-1 were used, with a point only allocated if a perfect match occurred. In both cases, no points were allocated in the case of total incorrect matches. The total number of points was added for each set of weighted values to produce a final agreement score for each comparison. Table 1 – Match Weights for ICD Codes Level 1 Level 2 Level 3 Example ICD Code F 30 .9 Perfect match ✔ ✔ ✔ Level 2 partial match ✔ ✔ ✘ Level 1 partial match ✔ ✘ ✘ Incorrect match ✘ ✘ ✘ Assigned weights 1 1 2 3 Assigned weights 2 0 0 1 2.5 Time and Cost Analysis The time taken to perform conversions in each phase was recorded to enable comparisons of the time and labour cost of each method. Cost of completing the task was calculated by multiplying the time for each method by the cost of a research assistant, which was set at $ 52.20AUD per hour (in line with the pay rate for a university-employed research assistant; excluding oncosts). Set up costs (i.e., cost of purchasing a ChatGPT 4o or Claude Sonnet 3.5 plan) were also included. 3 Results Table 2 displays the final scores for each set of assigned weights for each of the comparisons. For each set of assigned weights, a higher final score indicated higher level of agreement. Table 2 – Levels of agreement for each assigned weights and comparison Manual coding vs. ChatGPT 4o Manual coding vs. Claude Sonnet 3.5 ChatGPT 4o vs. Claude Sonnet 3.5 Assigned weights 1 (1-2-3) 3471 3525 4027 Assigned weights 2 (0-0-1) 578 599 757 For both sets of assigned weights, Claude Sonnet 3.5 had a higher level of agreement with the manual coding than ChatGPT 4o, indicating greater accuracy. In both cases, ChatGPT 4o and Claude Sonnet 3.5 had a high level of agreement when compared to each other. 3.1 Time and Cost Analysis Table 3 displays the time and associated cost of a research assistant performing the data conversions with each tool. The time and cost have been displayed for the 10% subset ( n = 1,976) converted within this study, as well as an extrapolation of costs for if the wider dataset ( n = 19,764) were to be converted from SNOMED to ICD. Table 3 – Time and Cost for Each Method Time FTEs in weeks* Labour cost (AUD) Cost of GAI Tool (AUD) Total cost (AUD) 10% subset ( n = 1976) Manual coding 24h 31m 0.64 $ 1,279.77 N/A $ 1,279.77 ChatGPT 4o 5h 45m 0.15 $ 300.15 $ 30.00 $ 330.15 Claude Sonnet 3.5 3h 10m 0.08 $ 165.30 $ 30.00 $ 195.30 Extrapolation for full dataset ( n = 19,764) Manual coding 245h 12m 6.45 $ 12,799.44 N/A $ 12,799.44 ChatGPT 4o 57h 30m 1.51 $ 3,001.50 $ 30.00 $ 3,031.50 Claude Sonnet 3.5 31h 40m 0.83 $ 1,653.00 $ 30.00 $ 1,683.00 * Assuming a 38-hour work week Of the three methods used, manual coding was the most time consuming and therefore costly, taking 24 hours and 31 minutes ( $ 1,279.77 AUD) to convert the subset utilised within this study. When extrapolated to the full dataset, this would take an estimated 245 hours and 12 minutes to complete, at a labour cost of $ 12,799.44. Of the two GAI methods, Claude Sonnet 3.5 was the most time and cost-effective, taking 3 hours and 10 minutes ( $ 195.30, including subscription cost). ChatGPT 4o was nearly double the time and cost of Claude Sonnet 3.5, taking 5 hours and 45 minutes ( $ 195.30, including subscription cost). Regardless, ChatGPT4o was still a significant time and cost saving when compared to completing the task manually. 4 Discussion This evaluation provided a case study to investigate the ability for GAI tools to process and analyse large healthcare datasets. To the authors’ knowledge, this paper is the first to challenge GAI tools to complete a clinical diagnostic coding conversion task, and to compare the results against that of a manual rater. Conversions of clinical diagnostic codes to other coding systems, such as the task presented in this study, is a complex and time-consuming task commonly undertaken within healthcare data processing. Therefore, this study highlights an example of a potential use for GAI within health data analytics. The analysis of this study examined levels of agreement between the two GAI tools and the manual rater. Whilst Claude Sonnet 3.5 outperformed ChatGPT 4o for both sets of assigned weights, there are several caveats to consider. For instance, the clinical validity of ICD codes, particularly in cases where these were identified as partial or incorrect matches was not assessed. This may have resulted in several potentially valid codes being incorrectly coded. For example, the SNOMED code “314041007 Abdominal pain in early pregnancy” was manually converted to “R10.9 Unspecified abdominal pain”. As this formed the benchmark for comparison between the GAI tools, conversions made by ChatGPT 4o (“O26.83 Pregnancy related abdominal pain”) and Claude Sonnet 3.5 (“O26.892 Other specified pregnancy related conditions, first trimester”) were considered as incorrect matches, and assigned zero points. During analysis, the GAI tools identified additional or arguably better matches between SNOMED-CT and ICD-10-CM. Additionally, there were several cases where the I-MAGIC tool was unable to produce a match for a SNOMED-CT code (e.g., 102508009 “Well female child”), whereas ChatGPT 4o and Claude Sonnet 3.5 were both able to produce the same alternative ICD-10-CM code (i.e., “Z00.129 Encounter for routine child health examination without abnormal findings”). This suggests that further formal analysis may demonstrate GAI tools outperform human raters. It is likely therefore that the results of this study significantly underestimate the accuracy and clinical validity of the matches produced by the GAI tools. Despite GAI tools demonstrating significant time and cost savings, several challenges were noted throughout the conversion process. With regards to ChatGPT 4o, the process of performing the SNOMED-CT-AU to ICD-10-CM conversion was not fully automated, nor would it be straightforward for someone inexperienced with writing GAI prompts to perform. When piloting the prompt, ChatGPT 4o had the tendency to skip lines, chunks of data, or hallucinate (produce new input data that was not provided in the dataset). It was therefore required to explicitly ask ChatGPT 4o to “ manually and sequentially ” convert the provided codes, and to “… not hallucinate, and only convert codes which have been provided…” and “…not create new codes to convert” . When completing the final batch of conversions, the output needed to be monitored for accuracy. Despite not hallucinating during the task, ChatGPT 4o produced new input data when it ran out of codes it had been provided. When providing additional prompts after the algorithm had performed well, it was beneficial to provide positive reinforcement to inform ChatGPT 4o it performed the task correctly. This avoided ChatGPT 4o changing its original output. There were also instances where ChatGPT 4o would attempt to terminate the task (“ Unfortunately I have run out of time to process additional conversions”) but was able to be prompted to continue without further issue. These nuances required some level of skill and familiarity with ChatGPT 4o and GAI prompts. In terms of the time and labour required, ChatGPT 4o was not simply a ‘set and forget’ solution to a large data task. Due to limitations on the volume of codes it was able to process before sometime hallucinating, a manual ‘nudge’ (i.e., “ Please continue with the next batch” ) was required after every 25 lines had been converted. This required continual monitoring of ChatGPT 4o whilst it was processing to continue and ensure that lines of data had not been skipped. Importantly, this rendered the task impractical to complete in the background whilst undertaking other work. ChatGPT 4o also imposes limits on the number of messages that are permitted within a certain timeframe (40 messages per three hours). Given the number of nudges required to process this data, in addition to further messages to adapt and rectify the prompt if it is not processing correctly, the message limit is quickly reached and requires waiting until the window has lapsed before proceeding with the rest of the task. This dramatically inflates the timeframe in which the task is able to be completed. Claude Sonnet 3.5 provided a more streamlined tool which did not require as much skill or time to produce a prompt. One of the key limitations of Claude Sonnet 3.5 was the process of importing and exporting data. Unlike ChatGPT 4o, Claude Sonnet 3.5 does not yet have the functionality to directly import or export Excel spreadsheet files. As a result, it was necessary to copy and paste lines of data from the spreadsheet into Claude Sonnet 3.5. This led to a further limitation, which was the restrictions on both message length and number of messages permitted. As the amount of data exceeded that which was able to be accepted into the input field, it was necessary to break up the prompt into smaller, more manageable chunks (in this study, 500 lines at a time). Although Claude Sonnet 3.5 did not appear to hallucinate with a greater number of conversions, only 50 were requested to be completed at a time due to limits on the maximum length of the output message which could be provided. This however meant that the message limit (approximately 45 messages every 5 hours, dependent on message length) was quickly consumed. Given that Claude Sonnet 3.5 processed significantly faster than ChatGPT 4o, there was a longer waiting period between exceeding the message limit and the limit being renewed. As Claude Sonnet 3.5 was unable to directly export into Excel at the end of the task, this significantly increased the time burden of the task, as it was requested to produce R Studio code which could be run to produce a final output dataset. In addition to requiring the worker to have some knowledge of how to run code in R Studio, this, in fact, accounted for the majority of the time to complete the task. For instance, it took 1 hour and 15 minutes to complete the code conversion, with the remainder of the time (1 hour and 55 minutes) accounting for writing and running the R Studio code. The ability to produce downloadable Excel file into Claude Sonnet 3.5 would rectify this current obstacle significantly reducing the time and cost taken to complete data analysis. Despite the limitations of GAI, there are clear benefits for its uses in completing large data analysis tasks. When completing this task manually, the human raters found this to be both mentally, emotionally draining and physically fatiguing, with high risk of repetitive strain injury. The human raters found this task to be boring, tedious, and unstimulating, which, over a long period of time, is likely to decrease both staff morale and mental wellbeing. When placing employees at high risk of repetitive strain injuries, there is risk of higher costs and projects delays resulting from researchers taking leave to recover from injury and stress [ 22 , 23 ]. 4.1 Study Limitations Whilst this case study provides valuable insights into the use of GAI to complete a large health data analysis task, there are several limitations which should be noted. Firstly, given that this is an Australian dataset, the SNOMED-CT codes came from the Australian edition (SNOMED-CT-AU), whilst the I-MAGIC tool only caters for the standard version. Therefore, this may account for why some codes were unable to be manually converted using the I-MAGIC tool. Additionally, multiple raters completed the manual coding task, introducing potential issues around inter-rater reliability, particularly when coders are less familiar with the task. Further, to date the I-MAGIC tool uses the ICD-10-CM and has not yet been updated for the new edition of the ICD (11th edition). To date there is not yet a mapping tool which enables SNOMED-CT to be converted to the newer version of the ICD. In addition, this study only considered ICD-10-CM codes to be ‘correct’ if they either perfectly or partially matched the manual code. Given the aim of this study was to examine whether this task could be completed using GAI, it was outside of the scope of the study to manually examine each ‘incorrect’ match to see whether or not it was clinically valid. However, this is likely to significantly undermine the results and underestimate the level of agreement between GAI tools and manual ratings. A further limitation of this study is the speed at which GAI tools are being developed and improved. It is likely that in the time period following this study, newer tools will be developed which may yield different results in terms of accuracy and processing speed. However, these will only improve the efficiency and accuracy of GAI tools. 4.2 Recommendations for Future Research There is significant scope for future research within this field. Firstly, further analysis of the produced data from this study is planned to examine the clinical validity of partial or incorrect matches, which will further strengthen the results of this study by producing more accurate ratings between the GAI and manual coding output. This study used the paid versions of both ChatGPT 4o and Claude Sonnet, which offer additional functionalities and greater processing speed than what is offered in the free version. This study could be replicated using the free versions to compare whether the paid upgrade yields any difference in terms of level of agreement and processing time. It is also yet to be determined whether the time and cost savings on this task would translate to other data conversion tasks. Further studies using GAI tools on are needed to determine if time and cost differences are consistent across tasks. Additionally, as new GAI tools are released with improvements to speed and functionality, it is recommended that this study is repeated to examine how these improvements impact the speed and accuracy by which this task can be completed. The concept of asking GAI to complete other similar data analysis tasks such as these should be considered, to further explore the capabilities of GAI on health care data. 4.3 Conclusions This study provides a case study for using GAI to complete manual data processing tasks which would otherwise be tedious, time consuming, costly, and both mentally and physically fatiguing to complete. The results from this study highlight that manual processing is prohibitive in terms of time, cost, and that alternative methods, such as the use of GAI, should be explored. GAI provides a potential gateway to explore and make use of the significant quantities of unanalysed data to assist in improving outcomes for healthcare staff, researchers, systems, and importantly, patients. Declarations Data was obtained under the Public Health Act with a waiver of consent. Funding: This research was supported by an Australian Government Research Training Program Scholarship. Conflicts of Interest: The authors declare that there are no conflicts of interest. Data Availability: Data is available upon reasonable request to the corresponding author. Author Contributions: CH - study conceptualisation, methodology, data collection, data analysis, project administration, writing (original draft), writing (review and editing); MR - methodology, data analysis, writing (original draft), writing (review and editing), supervision; CB - resources, data analysis, writing (original draft), writing (review and editing), supervision; AJ - data collection, writing (review and editing); AG - methodology, data collection, data analysis, writing (original draft), writing (review and editing) References Kieran Murphy. How Data Will Improve Healthcare Without Adding Staff or Beds. In: Cornell University, Institut Européen d'Administration des Affaires, Organisation WIP, editors. Global Innovation Index 20192019. Dash S, Shakyawar SK, Sharma M, Kaushik S. Big data in healthcare: management, analysis and future prospects. Journal of Big Data. 2019;6(1):54. Australian Medical Association. 2024 Public Hopsital Report Card. 2024. Li J, Dada A, Puladi B, Kleesiek J, Egger J. ChatGPT in healthcare: a taxonomy and systematic review. Computer Methods and Programs in Biomedicine. 2024:108013. Patel SB, Lam K. ChatGPT: the future of discharge summaries? The Lancet Digital Health. 2023;5(3):e107-e8. Rao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD. Evaluating ChatGPT as an adjunct for radiologic decision-making. MedRxiv. 2023:2023.02. 02.23285399. Ahn C. Exploring ChatGPT for information of cardiopulmonary resuscitation. Resuscitation. 2023;185. Boßelmann CM, Leu C, Lal D. Are AI language models such as ChatGPT ready to improve the care of individuals with epilepsy? Epilepsia (Series 4). 2023;64(5). Mogali SR. Initial impressions of ChatGPT for anatomy education. Anatomical sciences education. 2024;17(2):444-7. Sng GGR, Tung JYM, Lim DYZ, Bee YM. Potential and pitfalls of ChatGPT and natural-language artificial intelligence models for diabetes education. Diabetes Care. 2023;46(5):e103-e5. AI O. ChatGPT 2024 [Available from: https://chat.openai.com/. Emma Thorne. ChatGPT hits 100M weekly users. LinkedIn News. 2023. Thakur K, Barker HG, Khan Pathan A-S. Artificial Intelligence and Large Language Models : An Introduction to the Technological Future. Boca Raton, UNITED STATES: CRC Press LLC; 2024. Google AI. Gemini 2024 [Available from: https://gemini.google.com/. Microsoft. Copilot 2024 [Available from: https://copilot.microsoft.com/. Anthropic. Claude.ai 2024 [Available from: https://claude.ai/. SNOMED International. What is SNOMED CT? 2024 [Available from: https://www.snomed.org/what-is-snomed-ct. SNOMED International. Members 2024 [Available from: https://www.snomed.org/members. World Health Organisation. International Statistical Classification of Diseases and Related Health Problems (ICD) 2024 [Available from: https://www.who.int/standards/classifications/classification-of-diseases. National Library of Medicine. SNOMED CT to ICD-10-CM Map 2021 [Available from: https://www.nlm.nih.gov/research/umls/mapping_projects/snomedct_to_icd10cm.html. National Clinical Terminology Service. What is SNOMED CT and the AMT? 2024 [Available from: https://www.healthterminologies.gov.au/understanding-clinical-terminology-landing/what-is-snomed-ct-and-amt/. Basakci Calik B, Yagci N, Oztop M, Caglar D. Effects of risk factors related to computer use on musculoskeletal pain in office workers. International Journal of Occupational Safety and Ergonomics. 2022;28(1):269-74. Greggi C, Visconti VV, Albanese M, Gasperini B, Chiavoghilefu A, Prezioso C, et al. Work-Related Musculoskeletal Disorders: A Systematic Review and Meta-Analysis. Journal of Clinical Medicine. 2024;13(13):3964. Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5143761","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":358374697,"identity":"813290d0-1650-425a-875a-616d44b17600","order_by":0,"name":"Carly Hudson","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAx0lEQVRIiWNgGAWjYDACCTBikDOAsJmJ12JMupbEDURr4Z/dY3jzR8Wd9O3svQdvMFRYJzawnzHAb8mdM8bWPGee5e7sOZdswXAmPbGBJwe/FgOJHDNpxrbDuRtu5JhJABmJDQxEaJH82XY43QCs5R9QC/8bwlokeNsOJ0C0NAC1SBCwReJGWjHQL4cNd/acMbZIOJZu3CbxrACvFv4ZyRuBIXZY3py9x/DGhxpr2X7+5A14taCCBCBmI0H9KBgFo2AUjAIcAAB7nUO9SyhCTwAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0002-8187-2868","institution":"Bond University; University of New England","correspondingAuthor":true,"prefix":"","firstName":"Carly","middleName":"","lastName":"Hudson","suffix":""},{"id":358374698,"identity":"9a0c09b4-4845-48e1-b6c6-f0da59403a74","order_by":1,"name":"Marcus Randall","email":"","orcid":"https://orcid.org/0000-0003-2325-1032","institution":"Bond University","correspondingAuthor":false,"prefix":"","firstName":"Marcus","middleName":"","lastName":"Randall","suffix":""},{"id":358374699,"identity":"a08b8796-6937-4345-9977-c43f8f5219f8","order_by":2,"name":"Candice Bowman","email":"","orcid":"https://orcid.org/0000-0001-8043-9563","institution":"Bond University; Gold Coast Hospital and Health Service","correspondingAuthor":false,"prefix":"","firstName":"Candice","middleName":"","lastName":"Bowman","suffix":""},{"id":358374700,"identity":"b4c1b090-0aa2-4d95-9f5d-bc20ad25a7d1","order_by":3,"name":"Anu Joy","email":"","orcid":"","institution":"Gold Coast Hospital and Health Service; Griffith University","correspondingAuthor":false,"prefix":"","firstName":"Anu","middleName":"","lastName":"Joy","suffix":""},{"id":358374701,"identity":"2c2e7c66-dbf9-4ed7-b383-42a4b39bd61b","order_by":4,"name":"Adrian Goldsworthy","email":"","orcid":"https://orcid.org/0000-0001-7628-5725","institution":"Bond University; Murdoch University","correspondingAuthor":false,"prefix":"","firstName":"Adrian","middleName":"","lastName":"Goldsworthy","suffix":""}],"badges":[],"createdAt":"2024-09-24 09:35:00","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-5143761/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5143761/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":65575722,"identity":"cb753f00-bbbf-4bee-a9cd-5872e0562a80","added_by":"auto","created_at":"2024-09-30 07:32:50","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":167535,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eChatGPT 4o prompt and output\u003c/em\u003e\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-5143761/v1/f4f5fff6deba663cf955cc24.png"},{"id":65575723,"identity":"23bfe0a2-854a-48dd-a6c7-d1d04016bbfa","added_by":"auto","created_at":"2024-09-30 07:32:50","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":112305,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eChatGPT4o being prompted to continue with the next batch of conversions.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-5143761/v1/cf9e8bf5682066d36cc5b4a0.png"},{"id":65575725,"identity":"c2cd87d9-bf21-4e2e-afd2-aec484efab64","added_by":"auto","created_at":"2024-09-30 07:32:50","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":301487,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eClaude Sonnet 3.5 prompt and output\u003c/em\u003e\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-5143761/v1/12d79c32aee84108f9307fb4.png"},{"id":65575724,"identity":"34b0a7a7-6af1-4972-a8ec-0918ffc74a2a","added_by":"auto","created_at":"2024-09-30 07:32:50","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":221112,"visible":true,"origin":"","legend":"\u003cp\u003eClaude Sonnet 3.5 producing R Studio code to generate .xlsx file\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-5143761/v1/580dd6e1dac53c5973aaf2c5.png"},{"id":65576886,"identity":"55ceee4e-8639-416d-8c18-e121b53ebac2","added_by":"auto","created_at":"2024-09-30 07:40:50","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1280491,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5143761/v1/ef069063-5f94-4efe-a651-0f03932ae6e1.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eAI vs. Humans: A Comparative Analysis of Time, Cost, and Performance on a Clinical Code Conversion Task\u003c/p\u003e","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eThe volume of data \u0026nbsp;hospital and health services generate each year far exceeds the analytical capacity of humans\u0026nbsp;[1]. Murphy\u0026nbsp;[1]\u0026nbsp;estimated hospitals produce 50 petabytes (equivalent of 50,000,000 gigabytes) of data per year, 97% of which remains unanalysed or unused. Electronic health records, contain data \u0026nbsp;such as patient demographics, images, clinical notes, and pathology results, providing opportunities for retrospective analysis enabling data-informed decisions and predictions on service utilisation\u0026nbsp;[1, 2]. Increasingly financially restricted and resource limited health care systems are unable to manually process large datasets restricting the potential to improve healthcare system efficiency\u0026nbsp;[1, 3].\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eGAI has potential to support the analysis of large-scale datasets within healthcare settings. The nature of healthcare has traditionally been a very human focused activity, and as such, it has often resisted large-scale efforts for effective automation, particularly in the form of clinical and administrative decision making. A recent literature review by Li et al.\u0026nbsp;[4]\u0026nbsp;has identified some of the key areas in which GAI are starting to make an impact within healthcare, including generating discharge summaries\u0026nbsp;[5], determining appropriate screening procedures for a patient\u0026nbsp;[6], and answering clinical questions and providing medical education\u0026nbsp;[7-10].\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe increasing complexity of healthcare challenges around the world necessitates new data analysis processes which expeditiously and efficiently makes use of the large datasets available to healthcare systems. The recent advent of automation tools, such as generative artificial intelligence (GAI) provides new opportunities to efficiently complete large healthcare data analytics tasks. \u0026nbsp;The widespread implementation and usage of GAI is one of the most rapid advancements in technology we have seen in recent years. OpenAI’s ChatGPT\u0026nbsp;[11]\u0026nbsp;is currently one of the most popular GAI tools available with over 100 million online users per week\u0026nbsp;[12]. ChatGPT enables a user to enter in a prompt, command, or question, and ChatGPT will provide a response. Its interface is driven by a large language model (LLM), a form of natural language processing able to learn and improve its conversational abilities using a combination of self and semi-structured training\u0026nbsp;[13]. Processing of data is achieved by the application of large-scale neural networks, using feed-forward and convolutional architectures\u0026nbsp;[13]. \u0026nbsp;Following the widespread success of ChatGPT, competitors have since launched other GAI tools available to the general public, including Google Gemini\u0026nbsp;[14], Microsoft Copilot\u0026nbsp;[15], and Claude\u0026nbsp;[16]. \u0026nbsp;Despite GAI tools’ benefits, they currently are limited by their lack of context and human understanding. Accuracy and completeness of outputs are limited by the data available to the GAI model (i.e., what it has been trained on, its access to real time searching), which may be biased or inaccurate. GAI tools also have limited knowledge of more specialised topics, resulting in a tendency to “hallucinate”, a phenomenon whereby a GAI tool generates information to fill knowledge gaps, decreasing the accuracy of outputs. Healthcare professionals require up to date understanding of the currently dynamic limitations of GAI to optimally select tasks which it is likely to excel at and prompt it appropriately.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eA key challenge in analysing big healthcare data is the consistency of data recording between health services. Standardised diagnostic coding systems aid in keeping clinical information consistent, by providing a universal language by which diagnoses can be coded across health care services. The “Systematized Nomenclature of Medicine – Clinical Terms” (SNOMED-CT)\u0026nbsp;[17]\u0026nbsp;is a diagnostic coding system utilised by 48 countries (as of August 2024)\u0026nbsp;[18]\u0026nbsp;to capture detailed clinical information about procedures, diseases, and clinical findings. SNOMED-CT presents diagnoses in the form of a numeric code (e.g., “230690007”) and a corresponding descriptor (e.g., “Stroke”). SNOMED-CT uses a polyhierarchical structure, wherein any given code may belong to one or more ‘parent’ codes (e.g., “asthma” may be categorised under both “respiratory diseases” and “allergic conditions”). \u0026nbsp;Whilst SNOMED-CT provides a comprehensive framework for diagnostic coding at a patient-level, symptoms, procedures, and findings, the framework is complex and difficult to utilise for those with limited training.\u003c/p\u003e\n\u003cp\u003eThe International Statistical Classification of Diseases and Related Health Problems (ICD)\u0026nbsp;[19]\u0026nbsp;is currently the global standard for coding diagnostic information. ICD focuses on the classification of diseases, disorders, and causes of death using alphanumeric codes. These are determined using a hierarchical system, wherein codes are categorised by chapters (e.g., F – “mental and behavioural disorders”), and then into narrower categories as more detail is provided (e.g., “F10 – mental and behavioural disorders due to use of alcohol”; “F10.1 – mental and behavioural disorders due to use of alcohol: harmful use”). Whilst the ICD provides less detail than SNOMED-CT, the broader categories facilitate population health analytics, and provides a standard for international health system comparison.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTo the authors’ knowledge, it is yet to be explored whether GAI can assist in the conversion of clinical data from one diagnostic coding language to another, such as from SNOMED-CT to the ICD. Such conversions require specialised knowledge of clinical coding and are time consuming to complete manually. \u0026nbsp;The completion of diagnostic code conversion tasks using AI models may allow for less qualified staff to complete the task in less time, thereby reducing the cost of data processing.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe primary aim of this research, therefore, is to examine whether publicly accessible GAI tools, namely ChatGPT 4o and Claude Sonnet 3.5, can accurately convert clinical diagnostic codes from SNOMED—CT to ICD-10. In addition, the following sub-aims will be addressed:\u003c/p\u003e\n\u003cp\u003ei.\u0026nbsp; \u0026nbsp; \u0026nbsp;Compare the level of agreement between ChatGPT 4o and a human rater.\u003c/p\u003e\n\u003cp\u003eii.\u0026nbsp; \u0026nbsp;\u0026nbsp;Compare the level of agreement between Claude Sonnet 3.5 and a human rater.\u003c/p\u003e\n\u003cp\u003eiii.\u0026nbsp;\u0026nbsp;Compare the level of agreement between ChatGPT 4o and Claude Sonnet 3.5.\u003c/p\u003e\n\u003cp\u003eiv. \u0026nbsp; Examine the economic benefit, in terms of time and labour cost, of using GAI to complete this task, in comparison to a human rater.\u003c/p\u003e"},{"header":"2 Method","content":"\u003cp\u003eThe SNOMED codes used in this study originate from a broader emergency department dataset, retrieved as part of a study investigating mental health presentations to hospital emergency departments (ethics approval: HREC/2023/QGC/95219). This dataset consists of 19,764 unique SNOMED-CT-AU numeric codes (e.g., 48694002) and SNOMED-CT-AU names (e.g., \u0026ldquo;Anxiety reaction\u0026rdquo;) representing the diagnoses made to the ED over a three-year period (August 2020 to August 2023). The current evaluation utilises a 10% subset of this data (\u003cem\u003eN\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1976).\u003c/p\u003e \u003cp\u003eTo convert the SNOMED-CT-AU codes to ICD-10-CM, a three-phase approach was used. Firstly, codes were manually converted by human raters. Secondly, the codes were converted using ChatGPT 4o. Finally, the same set of codes were converted using Claude Sonnet 3.5. At the time of analysis both GAI tools required paid subscriptions.\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Phase 1: Manual Conversion of SNOMED-CT-AU Codes\u003c/h2\u003e \u003cp\u003eThe SNOMED-CT-AU codes were manually converted by a team of three raters (AG\u0026thinsp;=\u0026thinsp;800; AJ\u0026thinsp;=\u0026thinsp;644; CH\u0026thinsp;=\u0026thinsp;532). Conversions were completed using the Interactive Map-Assisted Generation of ICD Codes (I-MAGIC) algorithm (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://imagic.nlm.nih.gov/imagic/code/map\u003c/span\u003e\u003cspan address=\"https://imagic.nlm.nih.gov/imagic/code/map\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), an online tool which provides mapping between the two diagnostic coding systems [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. Codes were entered into the online tool in the format of \u0026ldquo;SNOMED-CT-AU Name (SNOMED-CT Code)\u0026rdquo; (e.g., \u0026ldquo;Anxiety reaction (48694002)\u0026rdquo;). The resulting ICD-10-CM equivalent code was then extracted.\u003c/p\u003e \u003cp\u003eIn cases where multiple ICD-10-CM codes were returned, only the first one was extracted and considered for analysis. In some cases (e.g., \u0026ldquo;Chest clear (301708006)\u0026rdquo;), the I-MAGIC algorithm found no equivalent ICD-10-CM code (\u0026ldquo;This SNOMED CT term cannot be mapped to ICD-10-CM\u0026rdquo;). There were also instances where SNOMED-CT codes could not be found in the I-MAGIC tool. Given that the dataset was using the Australian extension of SNOMED-CT, and the mapping tool uses the standard SNOMED-CT list, it is likely that this is due to some codes being region-specific [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. In these cases, it was noted that an equivalent could not be found.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Phase 2: Conversion of SNOMED-CT-AU Codes using ChatGPT4o\u003c/h2\u003e \u003cp\u003eChatGPT 4o [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] was used to automatically convert the SNOMED-CT-AU codes and names to the ICD-10-CM codes (completed in August 2024). The Excel file of SNOMED-CT-AU codes and names was uploaded to ChatGPT 4o. A prompt was trialled and adapted utilising an iterative approach to efficiently convert the codes and names, whilst avoiding hallucinations (providing false information) and data processing errors. It was necessary to state that ChatGPT 4o could take as much time as required to complete this task, otherwise the message would time out and cease to produce output. It was also identified there was a limit to the number of codes that could be processed at one time without increasing the number of errors such as incorrect conversions, creating new input data, and skipping codes within the dataset. As a result, 25 codes, split into five batches of five codes, were converted at a time.\u003c/p\u003e \u003cp\u003eThe following final prompt was used: \u003cem\u003e\u0026ldquo;Please manually and sequentially convert the SNOMED CT AU codes into ICD-10-CM codes and names which have not been converted yet. If you need to, complete this task in batches of 5. Take as long as you need. Do not hallucinate, and only convert codes which have been provided to you. Do not create new codes to convert. Provide an update after 5 batches have been completed. Provide a .xlsx file at the end.\u0026rdquo;\u003c/em\u003e (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eFollowing each conversion of 25 codes, ChatGPT4o needed to be prompted to continue; \u0026ldquo;\u003cem\u003ePlease continue.\u003c/em\u003e\u0026rdquo; (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eOnce all the codes had been completed, ChatGPT4o provided the requested .xlsx file containing the original input data with the corresponding final output data.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Phase 3: Conversion of SNOMED-CT-AU Codes Using Claude Sonnet 3.5\u003c/h2\u003e \u003cp\u003eAt the time of conducting this study (September 2024), Claude Sonnet 3.5 [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] was unable to import or export Excel files. As a result, data was copy and pasted into the chat as a means of providing the input data. However, there were limitations on the maximum message length that was permissible; as a result, data was provided in three batches of 500, and a final batch of 476. The prompt was developed in a similar manner to Phase 2 utilising an iterative approach. Claude Sonnet 3.5 required fewer directions in order to accurately undertake the requested task, allowing for the prompt to be easily developed within a relatively short timeframe. However, the limits on the length of the output message that Claude Sonnet 3.5 provided restricted the amount of data it was able to process to 50 codes at a time. When trialling the prompt, it was noted that Claude Sonnet 3.5 needed to be explicitly told not to skip lines, as in reflected in final the prompt: \u0026ldquo;\u003cem\u003ePlease convert these SNOMED CT AU codes and names into ICD-10-CM codes and names. Convert them in sequential order starting from the top and ensuring you do not skip any. I have provided 500 codes in total and need 500 responses at the end. Start with the first 50. If you are unable to convert a code, please state this.\u003c/em\u003e\u0026rdquo; (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eAs Claude Sonnet 3.5 was unable to directly export an Excel file, it was asked to produce R code which could be run to generate an output file; \u003cem\u003e\u0026ldquo;Can you write the code for R to create an Excel file of this data. Write it out in full so it exports all entries 1-100\u003c/em\u003e\u0026rdquo; Fig.\u0026nbsp;4). Due to the limits on the output message length, this was completed in batches of 100. The code was then run using R Studio to produce the final output file.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eFigure 4\u003c/em\u003e \u0026ndash; Claude Sonnet 3.5 producing R Studio code to generate .xlsx file\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4 Statistical Analysis\u003c/h2\u003e \u003cp\u003eSix rows of data were removed from the analysis where at least one of the three tools was unable to perform a conversion, leaving 1970 cases included in the final analysis. For the purpose of analysis, the ICD codes were split into three components; 1) letter code (\u0026ldquo;level one\u0026rdquo;), 2) major numeric code (before decimal point; \u0026ldquo;level 2\u0026rdquo;), and 3) minor numeric code (after decimal point; \u0026ldquo;level 3\u0026rdquo;) (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). A pattern matching program was developed using the C programming language to look for partial and perfect matches for the 1970 cases between: manual coding and ChatGPT 4o; manual coding and Claude Sonnet 3.5; and ChatGPT 4o and Claude Sonnet 3.5. The program converted the codes from each method into its component parts. Some manual editing was necessary to account for and fill in missing values (using null codes).\u003c/p\u003e \u003cp\u003eEach component was assigned a weighted value, which can be adjusted depending on purpose of analysis (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). In this analysis, two sets of weights were utilised (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). In the first instance, weights 1-2-3 were used, with three points allocated for complete matches, two points for partial matches at level two, and one point for partial matches at level one. In the second instance, to enable detection of only perfect matches, weights 0-0-1 were used, with a point only allocated if a perfect match occurred. In both cases, no points were allocated in the case of total incorrect matches. The total number of points was added for each set of weighted values to produce a final agreement score for each comparison.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u0026ndash; \u003cem\u003eMatch Weights for ICD Codes\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLevel 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLevel 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLevel 3\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExample ICD Code\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eF\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e30\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003e.9\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePerfect match\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e✔\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e✔\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e✔\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eLevel 2 partial match\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e✔\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e✔\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e✘\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eLevel 1 partial match\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e✔\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e✘\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e✘\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eIncorrect match\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e✘\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e✘\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e✘\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAssigned weights 1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAssigned weights 2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e0\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e2.5 Time and Cost Analysis\u003c/h2\u003e \u003cp\u003eThe time taken to perform conversions in each phase was recorded to enable comparisons of the time and labour cost of each method. Cost of completing the task was calculated by multiplying the time for each method by the cost of a research assistant, which was set at \u003cspan\u003e$\u003c/span\u003e52.20AUD per hour (in line with the pay rate for a university-employed research assistant; excluding oncosts). Set up costs (i.e., cost of purchasing a ChatGPT 4o or Claude Sonnet 3.5 plan) were also included.\u003c/p\u003e \u003c/div\u003e"},{"header":"3 Results","content":"\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e displays the final scores for each set of assigned weights for each of the comparisons. For each set of assigned weights, a higher final score indicated higher level of agreement.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u0026ndash; Levels of agreement for each assigned weights and comparison\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eManual coding vs. ChatGPT 4o\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eManual coding vs. Claude Sonnet 3.5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eChatGPT 4o vs. Claude Sonnet 3.5\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAssigned weights 1 (1-2-3)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3471\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3525\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e4027\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAssigned weights 2 (0-0-1)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e578\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e599\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e757\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eFor both sets of assigned weights, Claude Sonnet 3.5 had a higher level of agreement with the manual coding than ChatGPT 4o, indicating greater accuracy. In both cases, ChatGPT 4o and Claude Sonnet 3.5 had a high level of agreement when compared to each other.\u003c/p\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Time and Cost Analysis\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e displays the time and associated cost of a research assistant performing the data conversions with each tool. The time and cost have been displayed for the 10% subset (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1,976) converted within this study, as well as an extrapolation of costs for if the wider dataset (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;19,764) were to be converted from SNOMED to ICD.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u0026ndash; Time and Cost for Each Method\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTime\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eFTEs in weeks*\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eLabour cost (AUD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eCost of GAI Tool (AUD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eTotal cost (AUD)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003e10% subset (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1976)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eManual coding\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e24h 31m\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.64\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e1,279.77\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eN/A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e1,279.77\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eChatGPT 4o\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5h 45m\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e300.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e30.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e330.15\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eClaude Sonnet 3.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e3h 10m\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.08\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e165.30\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e30.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e195.30\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eExtrapolation for full dataset (\u003c/b\u003e\u003cb\u003en\u003c/b\u003e\u0026thinsp;\u003cb\u003e=\u0026thinsp;19,764)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eManual coding\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e245h 12m\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e6.45\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e12,799.44\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eN/A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e12,799.44\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eChatGPT 4o\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e57h 30m\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1.51\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e3,001.50\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e30.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e3,031.50\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eClaude Sonnet 3.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e31h 40m\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.83\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e1,653.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e30.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e1,683.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"7\"\u003e* Assuming a 38-hour work week\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eOf the three methods used, manual coding was the most time consuming and therefore costly, taking 24 hours and 31 minutes (\u003cspan\u003e$\u003c/span\u003e1,279.77 AUD) to convert the subset utilised within this study. When extrapolated to the full dataset, this would take an estimated 245 hours and 12 minutes to complete, at a labour cost of \u003cspan\u003e$\u003c/span\u003e12,799.44.\u003c/p\u003e \u003cp\u003eOf the two GAI methods, Claude Sonnet 3.5 was the most time and cost-effective, taking 3 hours and 10 minutes (\u003cspan\u003e$\u003c/span\u003e195.30, including subscription cost). ChatGPT 4o was nearly double the time and cost of Claude Sonnet 3.5, taking 5 hours and 45 minutes (\u003cspan\u003e$\u003c/span\u003e195.30, including subscription cost). Regardless, ChatGPT4o was still a significant time and cost saving when compared to completing the task manually.\u003c/p\u003e \u003c/div\u003e"},{"header":"4 Discussion","content":"\u003cp\u003eThis evaluation provided a case study to investigate the ability for GAI tools to process and analyse large healthcare datasets. To the authors\u0026rsquo; knowledge, this paper is the first to challenge GAI tools to complete a clinical diagnostic coding conversion task, and to compare the results against that of a manual rater. Conversions of clinical diagnostic codes to other coding systems, such as the task presented in this study, is a complex and time-consuming task commonly undertaken within healthcare data processing. Therefore, this study highlights an example of a potential use for GAI within health data analytics.\u003c/p\u003e \u003cp\u003eThe analysis of this study examined levels of agreement between the two GAI tools and the manual rater. Whilst Claude Sonnet 3.5 outperformed ChatGPT 4o for both sets of assigned weights, there are several caveats to consider. For instance, the clinical validity of ICD codes, particularly in cases where these were identified as partial or incorrect matches was not assessed. This may have resulted in several potentially valid codes being incorrectly coded. For example, the SNOMED code \u0026ldquo;314041007 Abdominal pain in early pregnancy\u0026rdquo; was manually converted to \u0026ldquo;R10.9 Unspecified abdominal pain\u0026rdquo;. As this formed the benchmark for comparison between the GAI tools, conversions made by ChatGPT 4o (\u0026ldquo;O26.83 Pregnancy related abdominal pain\u0026rdquo;) and Claude Sonnet 3.5 (\u0026ldquo;O26.892 Other specified pregnancy related conditions, first trimester\u0026rdquo;) were considered as incorrect matches, and assigned zero points. During analysis, the GAI tools identified additional or arguably better matches between SNOMED-CT and ICD-10-CM. Additionally, there were several cases where the I-MAGIC tool was unable to produce a match for a SNOMED-CT code (e.g., 102508009 \u0026ldquo;Well female child\u0026rdquo;), whereas ChatGPT 4o and Claude Sonnet 3.5 were both able to produce the same alternative ICD-10-CM code (i.e., \u0026ldquo;Z00.129 Encounter for routine child health examination without abnormal findings\u0026rdquo;). This suggests that further formal analysis may demonstrate GAI tools outperform human raters. It is likely therefore that the results of this study significantly underestimate the accuracy and clinical validity of the matches produced by the GAI tools.\u003c/p\u003e \u003cp\u003eDespite GAI tools demonstrating significant time and cost savings, several challenges were noted throughout the conversion process. With regards to ChatGPT 4o, the process of performing the SNOMED-CT-AU to ICD-10-CM conversion was not fully automated, nor would it be straightforward for someone inexperienced with writing GAI prompts to perform. When piloting the prompt, ChatGPT 4o had the tendency to skip lines, chunks of data, or hallucinate (produce new input data that was not provided in the dataset). It was therefore required to explicitly ask ChatGPT 4o to \u0026ldquo;\u003cem\u003emanually and sequentially\u003c/em\u003e\u0026rdquo; convert the provided codes, and to \u0026ldquo;\u0026hellip;\u003cem\u003enot hallucinate, and only convert codes which have been provided\u0026hellip;\u0026rdquo;\u003c/em\u003e and \u003cem\u003e\u0026ldquo;\u0026hellip;not create new codes to convert\u0026rdquo;\u003c/em\u003e. When completing the final batch of conversions, the output needed to be monitored for accuracy. Despite not hallucinating during the task, ChatGPT 4o produced new input data when it ran out of codes it had been provided.\u003c/p\u003e \u003cp\u003eWhen providing additional prompts after the algorithm had performed well, it was beneficial to provide positive reinforcement to inform ChatGPT 4o it performed the task correctly. This avoided ChatGPT 4o changing its original output. There were also instances where ChatGPT 4o would attempt to terminate the task (\u0026ldquo;\u003cem\u003eUnfortunately I have run out of time to process additional\u003c/em\u003e conversions\u0026rdquo;) but was able to be prompted to continue without further issue. These nuances required some level of skill and familiarity with ChatGPT 4o and GAI prompts.\u003c/p\u003e \u003cp\u003eIn terms of the time and labour required, ChatGPT 4o was not simply a \u0026lsquo;set and forget\u0026rsquo; solution to a large data task. Due to limitations on the volume of codes it was able to process before sometime hallucinating, a manual \u0026lsquo;nudge\u0026rsquo; (i.e., \u0026ldquo;\u003cem\u003ePlease continue with the next batch\u0026rdquo;\u003c/em\u003e) was required after every 25 lines had been converted. This required continual monitoring of ChatGPT 4o whilst it was processing to continue and ensure that lines of data had not been skipped. Importantly, this rendered the task impractical to complete in the background whilst undertaking other work. ChatGPT 4o also imposes limits on the number of messages that are permitted within a certain timeframe (40 messages per three hours). Given the number of nudges required to process this data, in addition to further messages to adapt and rectify the prompt if it is not processing correctly, the message limit is quickly reached and requires waiting until the window has lapsed before proceeding with the rest of the task. This dramatically inflates the timeframe in which the task is able to be completed.\u003c/p\u003e \u003cp\u003eClaude Sonnet 3.5 provided a more streamlined tool which did not require as much skill or time to produce a prompt. One of the key limitations of Claude Sonnet 3.5 was the process of importing and exporting data. Unlike ChatGPT 4o, Claude Sonnet 3.5 does not yet have the functionality to directly import or export Excel spreadsheet files. As a result, it was necessary to copy and paste lines of data from the spreadsheet into Claude Sonnet 3.5. This led to a further limitation, which was the restrictions on both message length and number of messages permitted. As the amount of data exceeded that which was able to be accepted into the input field, it was necessary to break up the prompt into smaller, more manageable chunks (in this study, 500 lines at a time). Although Claude Sonnet 3.5 did not appear to hallucinate with a greater number of conversions, only 50 were requested to be completed at a time due to limits on the maximum length of the output message which could be provided. This however meant that the message limit (approximately 45 messages every 5 hours, dependent on message length) was quickly consumed. Given that Claude Sonnet 3.5 processed significantly faster than ChatGPT 4o, there was a longer waiting period between exceeding the message limit and the limit being renewed. As Claude Sonnet 3.5 was unable to directly export into Excel at the end of the task, this significantly increased the time burden of the task, as it was requested to produce R Studio code which could be run to produce a final output dataset. In addition to requiring the worker to have some knowledge of how to run code in R Studio, this, in fact, accounted for the majority of the time to complete the task. For instance, it took 1 hour and 15 minutes to complete the code conversion, with the remainder of the time (1 hour and 55 minutes) accounting for writing and running the R Studio code. The ability to produce downloadable Excel file into Claude Sonnet 3.5 would rectify this current obstacle significantly reducing the time and cost taken to complete data analysis.\u003c/p\u003e \u003cp\u003eDespite the limitations of GAI, there are clear benefits for its uses in completing large data analysis tasks. When completing this task manually, the human raters found this to be both mentally, emotionally draining and physically fatiguing, with high risk of repetitive strain injury. The human raters found this task to be boring, tedious, and unstimulating, which, over a long period of time, is likely to decrease both staff morale and mental wellbeing. When placing employees at high risk of repetitive strain injuries, there is risk of higher costs and projects delays resulting from researchers taking leave to recover from injury and stress [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e].\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Study Limitations\u003c/h2\u003e \u003cp\u003eWhilst this case study provides valuable insights into the use of GAI to complete a large health data analysis task, there are several limitations which should be noted. Firstly, given that this is an Australian dataset, the SNOMED-CT codes came from the Australian edition (SNOMED-CT-AU), whilst the I-MAGIC tool only caters for the standard version. Therefore, this may account for why some codes were unable to be manually converted using the I-MAGIC tool. Additionally, multiple raters completed the manual coding task, introducing potential issues around inter-rater reliability, particularly when coders are less familiar with the task. Further, to date the I-MAGIC tool uses the ICD-10-CM and has not yet been updated for the new edition of the ICD (11th edition). To date there is not yet a mapping tool which enables SNOMED-CT to be converted to the newer version of the ICD. In addition, this study only considered ICD-10-CM codes to be \u0026lsquo;correct\u0026rsquo; if they either perfectly or partially matched the manual code. Given the aim of this study was to examine whether this task could be completed using GAI, it was outside of the scope of the study to manually examine each \u0026lsquo;incorrect\u0026rsquo; match to see whether or not it was clinically valid. However, this is likely to significantly undermine the results and underestimate the level of agreement between GAI tools and manual ratings.\u003c/p\u003e \u003cp\u003eA further limitation of this study is the speed at which GAI tools are being developed and improved. It is likely that in the time period following this study, newer tools will be developed which may yield different results in terms of accuracy and processing speed. However, these will only improve the efficiency and accuracy of GAI tools.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Recommendations for Future Research\u003c/h2\u003e \u003cp\u003eThere is significant scope for future research within this field. Firstly, further analysis of the produced data from this study is planned to examine the clinical validity of partial or incorrect matches, which will further strengthen the results of this study by producing more accurate ratings between the GAI and manual coding output.\u003c/p\u003e \u003cp\u003eThis study used the paid versions of both ChatGPT 4o and Claude Sonnet, which offer additional functionalities and greater processing speed than what is offered in the free version. This study could be replicated using the free versions to compare whether the paid upgrade yields any difference in terms of level of agreement and processing time.\u003c/p\u003e \u003cp\u003eIt is also yet to be determined whether the time and cost savings on this task would translate to other data conversion tasks. Further studies using GAI tools on are needed to determine if time and cost differences are consistent across tasks.\u003c/p\u003e \u003cp\u003eAdditionally, as new GAI tools are released with improvements to speed and functionality, it is recommended that this study is repeated to examine how these improvements impact the speed and accuracy by which this task can be completed. The concept of asking GAI to complete other similar data analysis tasks such as these should be considered, to further explore the capabilities of GAI on health care data.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Conclusions\u003c/h2\u003e \u003cp\u003eThis study provides a case study for using GAI to complete manual data processing tasks which would otherwise be tedious, time consuming, costly, and both mentally and physically fatiguing to complete. The results from this study highlight that manual processing is prohibitive in terms of time, cost, and that alternative methods, such as the use of GAI, should be explored. GAI provides a potential gateway to explore and make use of the significant quantities of unanalysed data to assist in improving outcomes for healthcare staff, researchers, systems, and importantly, patients.\u003c/p\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003cp\u003eData was obtained under the Public Health Act with a waiver of consent.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding:\u0026nbsp;\u003c/strong\u003eThis research was supported by an Australian Government Research Training Program Scholarship.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConflicts of Interest:\u0026nbsp;\u003c/strong\u003eThe authors declare that there are no conflicts of interest.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Availability:\u0026nbsp;\u003c/strong\u003eData is available upon reasonable request to the corresponding author.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor Contributions:\u0026nbsp;\u003c/strong\u003eCH - study conceptualisation, methodology, data collection, data analysis, project administration, writing (original draft), writing (review and editing); MR - methodology, data analysis, writing (original draft), writing (review and editing), supervision; CB - resources, data analysis, writing (original draft), writing (review and editing), supervision; AJ - data collection, writing (review and editing); AG - methodology, data collection, data analysis, writing (original draft), writing (review and editing)\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eKieran Murphy. How Data Will Improve Healthcare Without Adding Staff or Beds. In: Cornell University, Institut Europ\u0026eacute;en d\u0026apos;Administration des Affaires, Organisation WIP, editors. Global Innovation Index 20192019.\u003c/li\u003e\n\u003cli\u003eDash S, Shakyawar SK, Sharma M, Kaushik S. Big data in healthcare: management, analysis and future prospects. Journal of Big Data. 2019;6(1):54.\u003c/li\u003e\n\u003cli\u003eAustralian Medical Association. 2024 Public Hopsital Report Card. 2024.\u003c/li\u003e\n\u003cli\u003eLi J, Dada A, Puladi B, Kleesiek J, Egger J. ChatGPT in healthcare: a taxonomy and systematic review. Computer Methods and Programs in Biomedicine. 2024:108013.\u003c/li\u003e\n\u003cli\u003ePatel SB, Lam K. ChatGPT: the future of discharge summaries? The Lancet Digital Health. 2023;5(3):e107-e8.\u003c/li\u003e\n\u003cli\u003eRao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD. Evaluating ChatGPT as an adjunct for radiologic decision-making. MedRxiv. 2023:2023.02. 02.23285399.\u003c/li\u003e\n\u003cli\u003eAhn C. Exploring ChatGPT for information of cardiopulmonary resuscitation. Resuscitation. 2023;185.\u003c/li\u003e\n\u003cli\u003eBo\u0026szlig;elmann CM, Leu C, Lal D. Are AI language models such as ChatGPT ready to improve the care of individuals with epilepsy? Epilepsia (Series 4). 2023;64(5).\u003c/li\u003e\n\u003cli\u003eMogali SR. Initial impressions of ChatGPT for anatomy education. Anatomical sciences education. 2024;17(2):444-7.\u003c/li\u003e\n\u003cli\u003eSng GGR, Tung JYM, Lim DYZ, Bee YM. Potential and pitfalls of ChatGPT and natural-language artificial intelligence models for diabetes education. Diabetes Care. 2023;46(5):e103-e5.\u003c/li\u003e\n\u003cli\u003eAI O. ChatGPT 2024 [Available from: https://chat.openai.com/.\u003c/li\u003e\n\u003cli\u003eEmma Thorne. ChatGPT hits 100M weekly users. LinkedIn News. 2023.\u003c/li\u003e\n\u003cli\u003eThakur K, Barker HG, Khan Pathan A-S. Artificial Intelligence and Large Language Models : An Introduction to the Technological Future. Boca Raton, UNITED STATES: CRC Press LLC; 2024.\u003c/li\u003e\n\u003cli\u003eGoogle AI. Gemini 2024 [Available from: https://gemini.google.com/.\u003c/li\u003e\n\u003cli\u003eMicrosoft. Copilot 2024 [Available from: https://copilot.microsoft.com/.\u003c/li\u003e\n\u003cli\u003eAnthropic. Claude.ai 2024 [Available from: https://claude.ai/.\u003c/li\u003e\n\u003cli\u003eSNOMED International. What is SNOMED CT? 2024 [Available from: https://www.snomed.org/what-is-snomed-ct.\u003c/li\u003e\n\u003cli\u003eSNOMED International. Members 2024 [Available from: https://www.snomed.org/members.\u003c/li\u003e\n\u003cli\u003eWorld Health Organisation. International Statistical Classification of Diseases and Related Health Problems (ICD) 2024 [Available from: https://www.who.int/standards/classifications/classification-of-diseases.\u003c/li\u003e\n\u003cli\u003eNational Library of Medicine. SNOMED CT to ICD-10-CM Map 2021 [Available from: https://www.nlm.nih.gov/research/umls/mapping_projects/snomedct_to_icd10cm.html.\u003c/li\u003e\n\u003cli\u003eNational Clinical Terminology Service. What is SNOMED CT and the AMT? 2024 [Available from: https://www.healthterminologies.gov.au/understanding-clinical-terminology-landing/what-is-snomed-ct-and-amt/.\u003c/li\u003e\n\u003cli\u003eBasakci Calik B, Yagci N, Oztop M, Caglar D. Effects of risk factors related to computer use on musculoskeletal pain in office workers. International Journal of Occupational Safety and Ergonomics. 2022;28(1):269-74.\u003c/li\u003e\n\u003cli\u003eGreggi C, Visconti VV, Albanese M, Gasperini B, Chiavoghilefu A, Prezioso C, et al. Work-Related Musculoskeletal Disorders: A Systematic Review and Meta-Analysis. Journal of Clinical Medicine. 2024;13(13):3964.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Bond University","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Generative Artificial Intelligence, SNOMED, ICD-10, diagnostic coding, data analytics","lastPublishedDoi":"10.21203/rs.3.rs-5143761/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5143761/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground:\u003c/strong\u003e Healthcare services generate and store large quantities of data which historically required significant resources to manually analyse and gain meaningful insights to assist in informing healthcare policy and system design. \u0026nbsp;Recent advancements in automation tools, such as generative artificial intelligence (GAI), provides new opportunities to disencumber human labour and thinking for difficult tasks. This study explores the potential utilisation of GAI for a healthcare data analysis task, specifically, the conversion of clinical data from one diagnostic classification system to another (i.e., SNOMED-CT-AU to the ICD-10-CM). Additionally, it examines the time and cost benefit of performing this using GAI when compared to a human rater.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods: \u003c/strong\u003eConversions were completed on SNOMED-CT-AU codes to the ICD-10-CM using three methods: 1) manual conversion using the National Library of Medicine’s I-MAGIC tool, 2) conversion using ChatGPT 4o, and 3) conversion using Claude Sonnet 3.5. The accuracy of the GAI tools was mapped against the manually extracted codes, and examined in terms of a perfect, partial, or incorrect match. Researchers’ time was recorded and extrapolated to calculate and compare the cost associated with each method.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults: \u003c/strong\u003eWhen compared to the manually extracted codes, Claude Sonnet 3.5 yielded the highest level of agreement over ChatGPT 4o, whilst also being the most time and cost effective.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion:\u003c/strong\u003e GAI tools have greater utility than they have currently been given credit for. The automation of big data healthcare analytics, whilst still the domain of humans is increasingly capable of being undertaken using automation tools with low barriers to entry. The further development of the capability of GAI alongside the capability of the healthcare system to use it appropriately has the potential to result in significant resource savings.\u003c/p\u003e","manuscriptTitle":"AI vs. Humans: A Comparative Analysis of Time, Cost, and Performance on a Clinical Code Conversion Task","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-09-30 07:32:45","doi":"10.21203/rs.3.rs-5143761/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"1a2ab9fd-4c09-4676-9f95-00bf90ff5c0d","owner":[],"postedDate":"September 30th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":38126557,"name":"Medical Informatics"}],"tags":[],"updatedAt":"2024-09-30T07:32:45+00:00","versionOfRecord":[],"versionCreatedAt":"2024-09-30 07:32:45","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5143761","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5143761","identity":"rs-5143761","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}