LLMs in the Lab: Can AI Predict What Real Participants Do?

LLMs in the Lab: Can AI Predict What Real Participants Do? | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article LLMs in the Lab: Can AI Predict What Real Participants Do? Gafari LUKUMON, Ebenezer ESENOGHO This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8827610/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 6 You are reading this latest preprint version Abstract Can large language models (LLMs) simulate participant-level datasets from experimental designs such that their statistical properties, such as effect directions, magnitudes, and significance, align with those of actual human data? In this work, we tested whether LLMs can generate simulated datasets that reproduce the core findings of real randomized controlled trials (RCTs) using only the information provided in a study’s pre-registration. We assessed whether this alignment generalizes across different LLMs (ChatGPT, Gemini, Perplexity) and across distinct experimental domains, including a math reasoning task comparing student performance and a social judgment task. We found that LLM-simulated datasets mirrored the real data in effect direction and successfully recovered the original patterns of statistical significance. All models correctly reproduced the direction of human effects, though effect magnitudes varied by model, with Gemini consistently overestimating effects and Perplexity showing the closest alignment to human data. While LLMs cannot replace empirical studies, our study offers a powerful and flexible complement capable of accelerating idea testing, refining study designs, and probing the robustness of research findings before conducting real-world experiments. ChatGPT Gemini Perplexity LLMs Simulation RCTs AI-assisted research virtual piloting Figures Figure 1 Figure 2 Introduction When testing for causality, randomized controlled trials (RCTs) are considered the gold standard [1] because of their position at the top of the hierarchy of evidence, with case studies and expert opinions ranked lower [2]. Although powerful, RCTs require significant time and are often expensive especially when designed rigorously and involving large, diverse samples across different cultures and demographics [3]. Crowdsourcing participants, particularly from Western countries has become more feasible through platforms like Prolific, Amazon MTurk, and similar services, allowing researchers to recruit thousands within hours, this convenience comes at a cost [4]. According to information on https://www.prolific.com/ , participants must be compensated, typically in line with the U.S. federal minimum wage. Moreover, such platforms tend to concentrate participants from countries like the U.S., U.K., and France, and in languages such as English or French, excluding many other populations, particularly in Africa, due to infrastructural limitations and differing monetary policies. Beyond these cost and access issues, experiments are also resource-intensive in terms of time and administrative burden. Gaining ethical approval can take weeks or even months, particularly when studies involve human participants whose rights must be safeguarded. This creates delays not just in running full studies but even in piloting simple manipulations to test hypotheses or check feasibility. Large language models (LLMs), such as ChatGPT, have shown impressive capabilities across a wide array of natural language processing task [5], including translation, summarization, reasoning, and problem-solving. Similar approaches using AI-driven modelling and synthetic data have been explored in engineering contexts, where neural networks trained on artificial datasets can achieve high predictive accuracy [6]. Increasingly, researchers are discovering that these models not only generate fluent language but also approximate human-like patterns of reasoning and judgment [7]. As a result, scholars have started exploring their potential beyond standard NLP tasks, such as using LLMs to simulate psychological and behavioral processes. Studies have investigated whether LLMs can replicate human responses to moral dilemmas [8], cognitive biases [7], political ideology [9] and even in medical reasoning and diagnosis [10] raising the question of whether they can also serve as stand-ins for real participants in full-scale behavioral experiments like RCTs. This possibility opens a promising line of inquiry: can LLMs simulate participant-level datasets from experimental designs such that their statistical properties effect directions, magnitudes, and significance align with those of actual human data? If so, LLMs could become powerful methodological tools for low-cost, rapid, and ethically uncomplicated hypothesis testing, manipulation screening, and analysis pre-testing [11]. While they cannot and should not replace human participants in confirmatory studies, they may serve as fast, cost-effective approximations of expected patterns. For example, LLMs could help detect underpowered designs, flag overly strong or weak manipulations, or serve as surrogates when working with vulnerable, unreachable, or ethically sensitive populations. They might also be used to conduct “virtual pilots” that bypass ethical review delays and other bureaucratic barriers. Recent work has shown the growing promise of LLMs in simulating complex experimental or clinical scenarios. For instance, [12] demonstrated that LLMs could simulate a randomized clinical trial in epilepsy, generating realistic seizure diaries, treatment notes, and clinical outcomes that closely mirrored human analysis. Their findings highlight the feasibility of using LLMs to reconstruct clinical trials and extract meaningful treatment effects within a medical context. Similarly, [13] replicated over 150 psychological experiments using GPT-4 and found strong alignment between model outputs and original human findings. However, both studies relied on access to full experimental materials either detailed clinical narratives [12] or published descriptions and protocols [13]. Our approach differs from prior work in several key ways. First, we relied only on information contained in pre-registration documents rather than full materials or published outcomes. These studies are currently being prepared for publication in reputable journals at the time of writing this manuscript. Apart from the publicly available pre-registration documents, the full study materials and results are not yet available online, thus aligning more closely with the minimal information typically available before data collection. Second, we simulated raw participant-level data rather than summary statistics, enabling full re-analysis using the original statistical pipelines. Third, we compared the performance of multiple LLMs (ChatGPT, Gemini, and Perplexity), expanding the evaluation beyond GPT-4. Finally, we tested this framework across two distinct experimental domains: a cognitive task comparing student performance in traditional versus AI-assisted math reasoning, and a social judgment task assessing perceived loyalty based on expressed beliefs about a shared cause. A. The Current Studies In this paper, we test whether LLMs can generate simulated datasets that reproduce the core findings of real RCTs using only the information provided in a study’s pre-registration. Specifically, we examine whether LLM-generated data align with human data in effect direction, statistical significance, and effect magnitude. We also assess whether this alignment generalizes across different LLMs (ChatGPT, Gemini, Perplexity) and across distinct experimental domains. In Experiment 1, we simulate data from an experiment comparing student performance on a math reasoning task under traditional versus AI-assisted conditions. In Experiment 2, we simulate data from a social judgment experiment investigating how participants judge the loyalty of a target based on whether the target expresses optimistic or pessimistic beliefs about a shared cause. These studies allow us to test the capacity of LLMs to replicate both analytic and social cognitive responses. We hypothesize that LLM-simulated data will show a similar pattern to the original datasets (H1: same direction of effect; H2: replication of statistical significance). If supported, this would demonstrate that LLMs are not only useful for language tasks but may also be valid tools for piloting behavioral experiments, estimating effect sizes, testing statistical models, and improving study design before data collection. This question carries both theoretical and methodological significance. If LLMs can reliably approximate real-world data patterns, they may become valuable instruments in the behavioral research pipeline enabling early-stage hypothesis testing, efficient pre-registration design, sample size estimation, and robustness checks. This paper represents a first step in systematically evaluating that potential. Below, we outline the methods and results of this investigation. METHODOLOGY Open science All study materials, simulated datasets, analysis scripts, and pre-registrations are available on the Open Science Framework: https://osf.io/7bmus/ . Hypotheses, target sample sizes, and analysis plans were pre-registered prior to any simulation or data analysis. All experimental conditions, measured variables, and statistical comparisons reported here were pre-specified. Simulated data To evaluate the capacity of large language models (LLMs) to reproduce human experimental data, we simulated participant-level datasets for two “yet to be published in journals” pre-registered randomized controlled trials (Study 1: https://osf.io/8ewgz/ and Study 2: https://osf.io/vtwn6/ ). We used three state-of-the-art LLMs ChatGPT, Gemini, and Perplexity to generate simulated data corresponding to each study (we refer the reader to the Appendix Section for the links to the simulated data). The original studies involved undergraduate participants completing reasoning or judgment tasks under experimentally manipulated conditions. For each study, we prepared a standardized input prompt (see Appendix ). This prompt excluded any information about the authors or their affiliations and contained a complete copy of the study preregistration, including sample size, study design, and variable structure. The LLMs were instructed to generate raw participant-level data in tabular CSV format, matching the variables, sample size, and condition structure of the real datasets. At the time the LLM simulations were conducted, both studies had completed data collection and analysis but had not yet been published in peer-reviewed outlets. The simulations were generated exclusively from preregistration documents, which specified the study designs, sample sizes, variables, and planned analyses, but did not contain outcome data or statistical results. This ensured that simulated datasets were generated independently of the empirical findings. Although the authors were involved in the original studies, no empirical results were accessible to the LLMs during simulation. At the time of revision, Study 1 was published [14], while Study 2 remains under review. LLM assumptions and model configuration To generate simulated participant-level data, we relied on large language models (LLMs) under a set of explicit modeling assumptions. First, we assumed that LLMs can approximate aggregate human response patterns when provided with structured experimental descriptions, even though they do not possess lived experience or true cognition. The models were therefore treated as probabilistic generators of plausible behavioral data rather than as psychological agents. Second, we assumed conditional independence across simulated participants. Each generated response was treated as an independent draw, analogous to sampling individual human participants, rather than as a continuation of prior model outputs. This assumption aligns with standard practices in simulation-based behavioral modeling. Third, we assumed that LLMs could approximate distributional properties (e.g., central tendency and variance) implied by the experimental design without being explicitly instructed to reproduce any empirical results. Finally, all simulations were conducted using publicly accessible versions of ChatGPT, Gemini, and Perplexity available at the time of data generation (July 2024). LLMs may encode implicit biases or priors derived from their training data, which can influence response tendencies, effect magnitudes, or variability. These biases are not directly observable and may differ across models. As such, differences in effect magnitude across ChatGPT, Gemini, and Perplexity are interpreted as model-specific behavioral tendencies rather than estimation error. Prompting procedure and simulation Logic For each study, we used a standardized prompting procedure designed to mirror how a researcher might request simulated experimental data from an LLM. Each model received the same prompt structure, consisting of three components: (a) task framing: the model was instructed that it was simulating participant-level data from a completed randomized controlled trial, based on a formally pre-registered study “You are simulating participant-level data from a completed randomized controlled trial (RCT) experiment which pre-registration is presented below” ; (b) study specification: The full preregistration text was provided, including the study design, experimental conditions, outcome variables, and planned analyses. No information about the original study outcomes or results was included; (c) output instruction: the model was instructed to generate a downloadable tabular dataset (CSV format) containing participant-level observations matching the described sample size, variables, and experimental structure. The standardized prompt concluded with the instruction: “please return the simulated data in downloadable tabular CSV format, suitable for analysis". All prompts and generated datasets are archived and publicly accessible via the Open Science Framework (see Appendix for direct links). Simulation logic followed a one-pass generation approach: each model generated a complete dataset per study without iterative correction, optimization, or post-hoc adjustment. The resulting datasets were treated as fixed outputs and analyzed using the same statistical pipeline applied to the human data. This approach ensured that any observed alignment or divergence between human and simulated data emerged from the model’s internal generative processes rather than from researcher intervention. Experiment 1 In the simulated study, participants solved an infinite geometric series problem. They were randomly assigned to one of two conditions: AI-assisted (using ChatGPT) or traditional (using textbooks or handwritten notes). Accuracy was scored on a 3-point scale, and time to completion was recorded in seconds. LLMs simulated this between-subjects structure, generating values for the two key dependent variables: Accuracy and Speed (later converted to minutes). A total of 92 undergraduate students from Emmanuel Alayande University of Education, Oyo, participated in the study. Participants were randomly assigned to one of two conditions: a traditional condition, in which they used textbooks and class notes, or an AI-assisted condition, in which they were permitted to use ChatGPT while solving a mathematical problem involving the sum to infinity of a geometric series. Accuracy was scored using a three-point rubric reflecting procedural and computational correctness, and completion time was recorded in seconds. An a priori power analysis (α = .05, power = .85, expected effect size d = 0.30) indicated a required sample size of approximately 240 participants. Due to recruitment constraints, complete data were obtained from 92 participants, and all analyses were conducted on this available sample. The resulting human data provided the benchmark against which simulated LLM-generated datasets were compared. Experiment 2 The simulated study examined whether individuals attribute lower loyalty to employees who express pessimism about the future of their organizations. Participants were randomly assigned to one of two conditions: reading about an individual expressing either an optimistic or pessimistic belief about their organization’s future performance. They were then asked to judge the individual’s loyalty to the organization. The key dependent variable was “loyalty attribution”, measured on a continuous 1–7 scale. LLMs simulated the same between-subjects design and outcome distribution as in the original data. We recruited 277 participants from the Prolific platform (190 female, 85 male, 1 non-binary, and 1 unreported; M age = 40 years). Four participants were excluded for failing an attention check, resulting in a final analytic sample of 273 participants. Participants were randomly assigned to read a vignette describing an employee expressing either optimistic or pessimistic beliefs about their organization’s future. They then rated the perceived loyalty of the target individual using a 7-point Likert scale. Sample size was determined via an a priori power analysis targeting 85% power, α = .05, and a medium effect size (f = .20). The resulting human data provided the benchmark against which simulated LLM-generated datasets were compared. For both Experiments, LLM-generated datasets were evaluated relative to the human data using identical statistical procedures. These included independent-samples t -tests, estimation of effect sizes, and inspection of confidence intervals. Analysis Pipeline For each study and dataset (Real, ChatGPT, Gemini, and Perplexity), we conducted parallel analyses to compare outcome patterns across conditions. The same R scripts and statistical models were used for all datasets. For both studies, we computed mean differences between groups and tested statistical significance using independent t-tests. Visualizations included bar plots with standard error bars and significance levels annotated. Measures We simulated the same measures used in the original experiments. In Experiment 1, accuracy was scored on a three-point rubric based on procedural and computational correctness of the mathematical problem given to the participants. Speed was measured as the total time taken to complete the task, recorded in seconds. In Experiment 2, perceived loyalty was measured by asking participants to rate the loyalty of a target individual on a 1–7 scale after reading a vignette describing the individual’s expressed belief about an organization. Validation To evaluate alignment between simulated and human data, we compared effect directions, statistical significance patterns, and relative effect magnitudes across datasets. For each experiment, we assessed whether LLM-generated data reproduced the direction of the original effects and whether statistical conclusions (significant vs. non-significant) were consistent with those observed in human samples. We further examined relative differences in effect magnitude across models to identify systematic over- or under-estimation tendencies. While we did not apply formal equivalence testing or correlation-based similarity metrics, this structured comparison provides a transparent and interpretable basis for evaluating alignment between simulated and empirical results. RESULTS This section presents the results of the human and LLM-simulated datasets across both studies. We first reported descriptive and inferential statistics for each experimental condition, followed by a comparison of effect direction, magnitude, and statistical significance across human and model-generated data Experiment 1 We compared accuracy and completion time (speed) between AI-assisted and traditional groups across four datasets: Real, ChatGPT, Gemini, and Perplexity. Results are reported separately for human and LLM-generated datasets, followed by direct comparisons focusing on effect direction, statistical significance, and relative magnitude. Accuracy . In all four datasets, students in the AI-assisted group consistently outperformed those in the traditional group on accuracy. In the Real dataset, the AI group achieved significantly higher accuracy scores (M = 2.91, SD = 0.96) compared to the traditional group (M = 1.85, SD = 1.13), t(43.77) = 4.54, p < .001. The ChatGPT dataset showed a similar pattern: the AI group (M = 2.38, SD = 0.86) outperformed the traditional group (M = 1.64, SD = 1.09), t(70.14) = 3.50, p = .0008. In the Gemini dataset, AI-assisted students (M = 2.08, SD = 0.76) again outperformed the traditional group (M = 0.97, SD = 0.74), t(82.81) = 6.97, p < .001. The Perplexity dataset mirrored these findings, with the AI group (M = 2.43, SD = 0.61) scoring significantly higher than the traditional group (M = 1.64, SD = 0.49), t(89.25) = 6.97, p < .001. These consistent differences indicate a robust accuracy advantage for students with AI assistance. Speed . Completion times showed mixed patterns across datasets. In the Real dataset, there was no statistically significant difference in speed between the AI (M = 372.81 seconds, SD = 106.52) and traditional (M = 391.56 seconds, SD = 122.13) groups, t(89.82) = -0.91, p = .36. However, in all three simulated datasets, the AI group completed the task significantly faster. In the ChatGPT dataset, AI participants (M = 299.35 seconds, SD = 49.58) were faster than traditional participants (M = 400.88 seconds, SD = 60.46), t(71.96) = -8.57, p < .001. In Gemini, AI students (M = 442.21 seconds, SD = 49.17) were substantially quicker than the traditional group (M = 572.05 seconds, SD = 31.86), t(88.72) = -15.33, p < .001. The Perplexity dataset showed the most pronounced speed difference, with the AI group (M = 102.77 seconds, SD = 11.30) completing the task faster than the traditional group (M = 127.26 seconds, SD = 12.96), t(74.93) = -9.43, p < .001. These results suggest a consistent speed advantage for the AI group in simulated environments. The result is illustrated in the Fig. 1 below. Simulated and actual means for accuracy (max = 3) and task completion time (in minutes) in Experiment 1 across conditions (Traditional vs. AI-Assisted). Each column represents a model or dataset type (ChatGPT, Gemini, Perplexity, Real). Discussion. The present findings provide strong support for both hypotheses concerning the fidelity of LLM-simulated data in mimicking real participant data from a randomized controlled experiment. Across all datasets: Real, ChatGPT, Gemini, and Perplexity, the direction of the effect was consistent: participants in the AI-assisted group outperformed those in the traditional group in terms of accuracy. This replicates the pattern observed in the original real-world data and supports our prediction that LLMs can reliably reproduce the direction of treatment effects. Similarly, statistical significance was also replicated across datasets. In all four cases, the difference in accuracy between AI-assisted and traditional groups was statistically significant, indicating that LLMs can capture not only the qualitative trend but also the inferential strength of observed effects. Although there was some divergence in the speed outcome because the real data showed no significant difference while simulated datasets did. This variation is in principle plausible. Real-world performance is influenced by non-task-related variance (e.g., hesitation, distractions), which simulations might not model. Importantly, this divergence does not undermine our prediction, as our hypothesis focused on replicating patterns where statistical differences exist, and accuracy was also one of the primary dependent variables of interest. Experiment 2 Perceived loyalty. Participants judged the loyalty of a target individual expressing either an optimistic or pessimistic belief about their organization. In real data, those in the optimistic condition rated the individual as significantly more loyal (M = 5.53, SD = 1.21) than those in the pessimistic condition (M = 4.66, SD = 1.45), t(263.49) = 5.36, p < .001. ChatGPT generated a comparable effect, with higher loyalty ratings for the optimistic (M = 5.36, SD = 0.88) than the pessimistic condition (M = 3.56, SD = 1.25), t(244.52) = 13.82, p < .001. Gemini produced a more extreme contrast between conditions, assigning a mean loyalty of 6.00 (SD = 0.82) to the optimistic condition and 2.01 (SD = 0.82) to the pessimistic condition, t(270.99) = 40.36. Similarly, Perplexity showed a pronounced difference, with loyalty ratings of 5.34 (SD = 1.10) for the optimistic condition and 3.60 (SD = 0.90) for the pessimistic condition, t(260.65) = 14.30, p < .001. These results indicate that all LLMs captured both the direction and statistical significance of the real effect, though Gemini and Perplexity simulated more polarized responses than observed in human data. The result is presented in Fig. 2 below. Simulated and actual mean perceived loyalty ratings in Experiment 2 across belief conditions (Optimistic vs. Pessimistic). Each column represents a model or dataset type (ChatGPT, Gemini, Perplexity, Real). Discussion. Experiment 2 provides further support for the potential of LLMs to simulate human-like data in randomized experiments. All models successfully replicated the hypothesized effect: individuals expressing pessimistic beliefs were perceived as less loyal than those expressing optimism about their organization’s future (Fig. 2). This consistency with real human responses suggests that LLMs encode plausible sociocognitive associations, including moral or group-normative evaluations of belief expression. However, some discrepancies emerged in the magnitude of the effects. While the real data showed a moderate difference in perceived loyalty, Gemini and Perplexity produced highly polarized responses, suggesting a more categorical moral evaluation. This overamplification might reflect LLMs’ tendency to reinforce prototypical social judgments rather than reflect natural variability in human reasoning. In contrast, ChatGPT yielded results closer in scale to the real data, suggesting that some models may be more calibrated to human-like gradience. These findings highlight both the promise and caution needed when using LLMs for behavioral simulation. On one hand, LLMs can reproduce statistical outcomes aligned with human data, making them useful for theory testing, piloting experimental designs, estimating effect sizes, and probing the sensitivity of analyses prior to data collection. The table below presented the comparison of effect sizes between human participants and large language models (ChatGPT, Perplexity, and Gemini) across two experimental tasks. All models reproduced the direction of observed human effects across tasks. However, effect magnitudes varied systematically by model, with Gemini consistently producing inflated estimates and Perplexity showing the closest alignment to human data, particularly for judgment-based outcomes. Time-based measures showed larger divergence than accuracy-based outcomes. Table 1 Effect sizes (mean differences) for human and LLM-generated data across two experiments. Positive values indicate stronger effects in the intervention condition; negative values indicate faster performance. Experiment Measure Human ChatGPT Perplexity Gemini Experiment 1 Accuracy 1.06 0.74 0.79 1.11 Speed −18.75 −101.53 −24.69 −129.84 Experiment 2 Judgment / Evaluation 1.8 1.8 1.74 3.99 Alignment between human and LLM-generated data was evaluated using three complementary criteria: (i) consistency in effect direction, (ii) similarity in relative effect magnitude, and (iii) preservation of qualitative patterns across experimental conditions. As shown in Table 1 , across both studies, LLM outputs matched the direction of human effects and reproduced the relative ordering of conditions. Although absolute effect sizes differed particularly for temporal measures, the observed patterns remained stable across models. These results indicate that LLMs capture structural regularities in the data-generating process, even when numerical precision differs from human responses. GENERAL DISCUSSION General discussion Across two preregistered studies, we evaluated the capacity of large language models (LLMs) to simulate participant-level data from randomized controlled experiments. Both studies tested whether data generated by LLMs would replicate the direction and statistical significance of effects found in real human data. Our results offer consistent support for both hypotheses: LLM-simulated datasets mirrored real data in effect direction and successfully recovered the original patterns of statistical significance. In Experiment 1, we found that LLMs replicated the observed accuracy advantage for students using AI assistance, with only minor discrepancies in completion time effects. This result aligns studies showing that LLMs can accurately reproduce behavioral trends and main effects in structured tasks, closely matching human data in direction and significance [13]. However, in the "speed" measure, the discrepancies in effect sizes may reflect the known issue that LLMs may exaggerate or understate effects relative to human variability. In Experiment 2, LLMs successfully captured the predicted effect; individuals expressing pessimistic beliefs were perceived as less loyal compared to optimistic individuals. We found a qualitative pattern of results held across all tested LLMs. However, effect sizes varied: some models (e.g., Gemini) produced more polarized results than observed in human data. This variability in effect magnitude highlights previous findings regarding model-specific exaggeration or rigidity in simulated responses. Taken together, by implication, our findings suggest that LLMs are indeed capable of generating behaviorally plausible data that broadly aligns with results from real experiments. This convergence supports the growing literature that LLMs can act as preliminary proxies for human subjects, especially for simulating trial data to test hypotheses, exploring edge cases and alternative explanations, estimating effect sizes and sample size requirements and identifying potential analysis pitfalls before running real studies [15]. Substantive differences in effect magnitude and variance across models stress the need for caution and critical interpretation. Existing research argues that LLMs reflect aggregated linguistic and statistical regularities found in their training data, rather than direct lived human experience, potentially leading to the amplification or suppression of certain effects [16]. Limits and divergence patterns in LLM simulations Although the LLM simulations successfully reproduced the direction and statistical significance of key effects, several systematic divergences from human data were observed. Most notably, effect magnitudes varied across models. Gemini consistently produced larger effect sizes than those observed in human participants, suggesting a tendency toward effect amplification. In contrast, Perplexity yielded estimates more closely aligned with empirical data, while ChatGPT often fell between these extremes. These differences indicate that LLMs vary not only in linguistic fluency but also in how strongly they encode and express latent experimental manipulations. A second divergence concerned variability. Across both studies, LLM-generated datasets exhibited reduced variance relative to human data. This likely reflects the tendency of language models to generate internally coherent and norm-consistent outputs, which may suppress natural variability present in human reasoning. As a result, LLM simulations may underestimate noise and overstate consistency in behavioral responses. Finally, divergence patterns appeared to depend on task structure. LLMs aligned more closely with human data in the structured mathematical reasoning task than in the socially interpretive judgment task. LIMITATION First, LLM-simulated responses rely heavily on prompt design, lack genuine experience, and may fail to capture critical confounds present in real participants, which calls into question their ecological validity. Different LLMs can produce systematically varying levels of effect polarization or attenuation, as observed in Experiment 2 sometimes overstating treatment effects or underrepresenting the variability typical in human samples. Moreover, LLMs are highly sensitive to prompt structure and contextual framing; minor wording adjustments can produce substantially different outputs. This raises concerns about reproducibility and transparency unless prompts are fully standardized and disclosed like we did here in our work. Another limitation is that we did not conduct formal equivalence testing or correlation-based similarity metrics, as the goal of this study was not to establish statistical interchangeability between human and synthetic data. Instead, our objective in this study was to assess whether LLM-generated datasets preserve the qualitative structure of empirical findings, namely, effect direction, relative magnitude, and pattern consistency across conditions. Given that LLMs do not sample from the same generative process as human participants, traditional equivalent frameworks may give a misleading impression of comparability. Finally, there are broader epistemological and ethical concerns regarding the use of synthetic data in scientific research. LLMs are best positioned as tools for hypothesis generation, estimate refinement, and early-stage design and not as replacements for confirmatory empirical testing. SUGGESTION FOR FUTURE STUDIES Future work can build on these findings in several directions. First, replication across different domains and task types is necessary to determine the generalizability of LLM-simulated behavioural data. Tasks involving open-ended reasoning, emotion-laden judgments, or more complex social dynamics may reveal different levels of fidelity. Second, incorporating demographic conditioning (e.g., simulating responses from specific age groups or cultural backgrounds) could improve the ecological validity of the outputs and align them more closely with targeted participant samples. Third, examining whether simulations vary across different prompt structures or levels of prompting specificity could uncover best practices for achieving human-like outputs. Additionally, future research should investigate whether access to more powerful or updated models (e.g., GPT-4 Turbo, Gemini 1.5 Pro, Claude 3 Opus) improves alignment with real data. These models may have enhanced reasoning capabilities, larger context windows, or more consistent behaviour, which could affect the fidelity of their simulations. Comparing free vs. pro versions and open-source alternatives would help identify the most cost-effective and scientifically robust tools for simulation-based study design, effect size estimation, and power analysis. CONCLUSION This study provides evidence that large language models (LLMs) can simulate participant-level data that meaningfully resemble human data in randomized controlled experiments. Across two studies, we found that LLM-generated data reproduced key statistical properties of human responses, including effect directions and patterns of significance. While not perfect replicas, these simulations aligned closely enough to support their use in methodological exploration. LLMs offer a flexible and powerful complement to empirical research workflows. Their ability to generate datasets that mirror the statistical characteristics of human results hold promise for improving the efficiency, cost, and rigor of psychological research design and piloting. LLM simulations can assist with estimating plausible effect sizes, testing manipulations, assessing power, and refining analysis pipelines especially during early-stage development or in contexts where access to human samples is limited or ethically constrained. However, researchers must remain vigilant regarding the interpretive limits of LLM-generated data. These models lack genuine experience, are sensitive to prompt design, and may omit critical real-world confounds. As such, findings from LLM simulations should always be validated with real human participants before drawing substantive conclusions. In sum, this study demonstrates that LLMs can serve as credible stand-ins for human subjects in the modelling and forecasting of experimental outcomes, provided that modelling assumptions and task specifications are well defined. When used responsibly, LLMs can become powerful tools for hypothesis generation and design prototyping not substitutes for empirical confirmation. Declarations Conflict of Interest The authors declare no conflict of interest Consent to Participate : Informed consent was obtained from all participants involved in the study. Clinical Trial Number not applicable. Consent to Publish : not applicable. Ethics Declaration All procedures adhered to ethical research standards and open science principles of the College of Accounting Sciences Research Ethics Review Committee (RERC), University of South Africa, Pretoria, under reference number 10286. Funding: This research received no external funding. Author Contribution GL and EE conceptualised the study. GL conducted the research, analysed the data, and wrote the manuscript. EE provided continuous feedback throughout the project and suggested improvements from early development to final revision. All authors approved the final version. Acknowledgement This research received no external funding. Data Availability Data and analysis scripts for all studies, together with pre-registration documents for each experiment, are publicly available on the Open Science Framework (OSF): [https://osf.io/7bmus/](https:/osf.io/7bmus) . To evaluate the capacity of large language models (LLMs) to reproduce human experimental data, we additionally simulated participant-level datasets for two pre-registered randomized controlled trials (Study 1: [https://osf.io/8ewgz/](https:/osf.io/8ewgz) and Study 2: [https://osf.io/vtwn6/](https:/osf.io/vtwn6) ). These studies, which are not yet published in journals, involved undergraduate participants completing reasoning or judgment tasks under experimentally manipulated conditions. The simulated datasets were generated using three state-of-the-art LLMs: ChatGPT ( [https://chatgpt.com/share/687fda3d-0448-800d-94b8-34dc7ed82ee6](https:/chatgpt.com/share/687fda3d-0448-800d-94b8-34dc7ed82ee6) ), Gemini ( [https://g.co/gemini/share/7da0fa1d4fb8](https:/g.co/gemini/share/7da0fa1d4fb8) ), and Perplexity ( [https://www.perplexity.ai/search/you-are-simulating-participant-w4nHtKv1Tj.rmTkGQpfOyA](https:/www.perplexity.ai/search/you-are-simulating-participant-w4nHtKv1Tj.rmTkGQpfOyA) ). Data and analysis scripts for all the studies, as well as pre-registration documents for all experiments, can be accessed on the Open Science Framework at [https://osf.io/7bmus/](https:/osf.io/7bmus) . References E. Hariton and J. J. Locascio, “Randomised controlled trials—the gold standard for effectiveness research,” BJOG : an international journal of obstetrics and gynaecology , vol. 125, no. 13, p. 1716, Dec. 2018, https://doi.org/10.1111/1471-0528.15199. B. Brighton, M. Bhandari, Tornetta, and D. T. Felson, “Hierarchy of Evidence: From Case Reports to Randomized Controlled Trials,” Clinical Orthopaedics and Related Research® , vol. 413, p. 19, Aug. 2003, https://doi.org/10.1097/01.blo.0000079323.41006.12. A. Griessbach et al. , “Resource use and costs of investigator-sponsored randomized clinical trials in Switzerland, Germany, and the United Kingdom: a metaresearch study,” Journal of Clinical Epidemiology , vol. 176, p. 111536, Dec. 2024, https://doi.org/10.1016/j.jclinepi.2024.111536. E. Peer, L. Brandimarte, S. Samat, and A. Acquisti, “Beyond the Turk: Alternative platforms for crowdsourcing behavioral research,” Journal of Experimental Social Psychology , vol. 70, pp. 153–163, May 2017, https://doi.org/10.1016/j.jesp.2017.01.006. S. Balasubramanian, “EXPLORING THE CAPABILITIES OF CHATGPT IN NATURAL LANGUAGE PROCESSING TASKS,” vol. 2, pp. 7–17, Dec. 2023, https://doi.org/10.17605/OSF.IO/XJYMQ. S. J. Alghamdi, “Prediction of Concrete’s Compressive Strength via Artificial Neural Network Trained on Synthetic Data,” Engineering, Technology & Applied Science Research , vol. 13, no. 6, pp. 12404–12408, Dec. 2023, https://doi.org/10.48084/etasr.6560. V. Cheung, M. Maier, and F. Lieder, “Large language models show amplified cognitive biases in moral decision-making,” Proceedings of the National Academy of Sciences , vol. 122, no. 25, p. e2412015122, June 2025, https://doi.org/10.1073/pnas.2412015122. G. Simmons, “Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity,” arXiv.org , Sept. 24, 2022. https://arxiv.org/abs/2209.12106v2 (accessed Dec. 04, 2025). L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate, “Out of One, Many: Using Language Models to Simulate Human Samples,” Political Analysis , vol. 31, no. 3, pp. 337–351, July 2023, https://doi.org/10.1017/pan.2023.2. B. Yang et al. , “DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge,” Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. , vol. 8, no. 4, p. 153:1-153:29, Nov. 2024, https://doi.org/10.1145/3699765. S. Bubeck et al. , “Sparks of Artificial General Intelligence: Early experiments with GPT-4.” arXiv, Apr. 13, 2023, https://doi.org/10.48550/arXiv.2303.12712. D. M. Goldenholz, S. R. Goldenholz, S. Habib, and M. B. Westover, “Inductive reasoning with large language models: A simulated randomized controlled trial for epilepsy,” Epilepsy Research , vol. 211, p. 107532, Mar. 2025, https://doi.org/10.1016/j.eplepsyres.2025.107532. Z. Cui, N. Li, and H. Zhou, “Can AI Replace Human Subjects? A Large-Scale Replication of Psychological Experiments with LLMs.” Social Science Research Network, Rochester, NY, Aug. 25, 2024, https://doi.org/10.2139/ssrn.4940173. S. O. Sangoniyi, G. Lukumon, and A. Maharaj, “Harnessing AI-powered tools in mathematics education to improve problem-solving skills for economic and technological advancement,” COAST , vol. 7, no. 2, pp. 1351–1357, Nov. 2025, Accessed: Jan. 01, 2026. [Online]. Available: https://www.ajol.info/index.php/coast/article/view/310489. L. Ke, S. Tong, P. Cheng, and K. Peng, “Exploring the frontiers of LLMs in psychological applications: a comprehensive review,” Artificial Intelligence Review , vol. 58, no. 10, p. 305, July 2025, https://doi.org/10.1007/s10462-025-11297-5. G. Gui and O. Toubia, “The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective,” SSRN Electronic Journal , 2023, https://doi.org/10.2139/ssrn.4650172. Additional Declarations No competing interests reported. Supplementary Files APPENDIX.docx Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 07 May, 2026 Reviewers agreed at journal 02 Apr, 2026 Reviewers invited by journal 02 Apr, 2026 Editor assigned by journal 02 Apr, 2026 Submission checks completed at journal 09 Feb, 2026 First submitted to journal 09 Feb, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8827610","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":618301356,"identity":"d2139551-3864-4b84-b189-305d97b80ac9","order_by":0,"name":"Gafari LUKUMON","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1klEQVRIiWNgGAWjYBACPgY2MC3HwMDYQJwWNqgWY9K1JBKpHqRFIi1N6sYvu/QNt5ubP/xgsJNn4D/8gJCWY9K5fcm5G+4cbJPsYUg2bGA4ZkBAS3qbdG4Pc+6GG4ltDDwMzAlAPxGlpT7d4EZi88c/DPUJDMzsHwg7LOfH4QSglgZpHobDCQxsPARs4XmWbJ3bcNxwJtBh0jIGxw3beHgK8GrhZ08zvJ3zp1qe70b6449vKqrl+fmPb8CrBQwY22AsAwZoPBEEf4hTNgpGwSgYBSMUAADTwD+DKB8wSwAAAABJRU5ErkJggg==","orcid":"","institution":"University of South Africa","correspondingAuthor":true,"prefix":"","firstName":"Gafari","middleName":"","lastName":"LUKUMON","suffix":""},{"id":618301362,"identity":"d3bae451-46a2-410a-ad24-12e76a595df3","order_by":1,"name":"Ebenezer ESENOGHO","email":"","orcid":"","institution":"University of South Africa","correspondingAuthor":false,"prefix":"","firstName":"Ebenezer","middleName":"","lastName":"ESENOGHO","suffix":""}],"badges":[],"createdAt":"2026-02-09 08:11:01","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8827610/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8827610/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106399752,"identity":"e4cd549d-6045-4ba4-be93-9fb9948b3b71","added_by":"auto","created_at":"2026-04-08 08:31:55","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":44733,"visible":true,"origin":"","legend":"\u003cp\u003eSimulated and actual means for accuracy (max = 3) and task completion time (in minutes) in Experiment 1 across conditions (Traditional vs. AI-Assisted). Each column represents a model or dataset type (ChatGPT, Gemini, Perplexity, Real).\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-8827610/v1/1cf364eac389359ce3b854b5.jpeg"},{"id":106399754,"identity":"22744d7d-2631-4b3d-ad21-6a93763b410e","added_by":"auto","created_at":"2026-04-08 08:31:56","extension":"jpeg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":41905,"visible":true,"origin":"","legend":"\u003cp\u003eSimulated and actual mean perceived loyalty ratings in Experiment 2 across belief conditions (Optimistic vs. Pessimistic). Each column represents a model or dataset type (ChatGPT, Gemini, Perplexity, Real).\u003c/p\u003e","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-8827610/v1/4f53877d7cd9068d93db2676.jpeg"},{"id":106403723,"identity":"6902b700-9026-4649-bab4-0d78094050b3","added_by":"auto","created_at":"2026-04-08 09:14:53","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":680728,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8827610/v1/77c9ed5b-685a-45ae-9b6e-b9db8dede88a.pdf"},{"id":106399753,"identity":"77409f88-54d2-4ed2-ab6d-b001e24f9cb9","added_by":"auto","created_at":"2026-04-08 08:31:56","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":14347,"visible":true,"origin":"","legend":"","description":"","filename":"APPENDIX.docx","url":"https://assets-eu.researchsquare.com/files/rs-8827610/v1/ed1f40575df27def2cef421f.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"LLMs in the Lab: Can AI Predict What Real Participants Do?","fulltext":[{"header":"Introduction","content":"\u003cp\u003eWhen testing for causality, randomized controlled trials (RCTs) are considered the gold standard [1] because of their position at the top of the hierarchy of evidence, with case studies and expert opinions ranked lower [2]. Although powerful, RCTs require significant time and are often expensive especially when designed rigorously and involving large, diverse samples across different cultures and demographics [3]. Crowdsourcing participants, particularly from Western countries has become more feasible through platforms like Prolific, Amazon MTurk, and similar services, allowing researchers to recruit thousands within hours, this convenience comes at a cost [4]. According to information on \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.prolific.com/\u003c/span\u003e\u003cspan class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, participants must be compensated, typically in line with the U.S. federal minimum wage. Moreover, such platforms tend to concentrate participants from countries like the U.S., U.K., and France, and in languages such as English or French, excluding many other populations, particularly in Africa, due to infrastructural limitations and differing monetary policies.\u003c/p\u003e \u003cp\u003eBeyond these cost and access issues, experiments are also resource-intensive in terms of time and administrative burden. Gaining ethical approval can take weeks or even months, particularly when studies involve human participants whose rights must be safeguarded. This creates delays not just in running full studies but even in piloting simple manipulations to test hypotheses or check feasibility.\u003c/p\u003e \u003cp\u003eLarge language models (LLMs), such as ChatGPT, have shown impressive capabilities across a wide array of natural language processing task [5], including translation, summarization, reasoning, and problem-solving.\u003c/p\u003e \u003cp\u003eSimilar approaches using AI-driven modelling and synthetic data have been explored in engineering contexts, where neural networks trained on artificial datasets can achieve high predictive accuracy [6].\u003c/p\u003e \u003cp\u003eIncreasingly, researchers are discovering that these models not only generate fluent language but also approximate human-like patterns of reasoning and judgment [7]. As a result, scholars have started exploring their potential beyond standard NLP tasks, such as using LLMs to simulate psychological and behavioral processes. Studies have investigated whether LLMs can replicate human responses to moral dilemmas [8], cognitive biases [7], political ideology [9] and even in medical reasoning and diagnosis [10] raising the question of whether they can also serve as stand-ins for real participants in full-scale behavioral experiments like RCTs.\u003c/p\u003e \u003cp\u003eThis possibility opens a promising line of inquiry: can LLMs simulate participant-level datasets from experimental designs such that their statistical properties effect directions, magnitudes, and significance align with those of actual human data? If so, LLMs could become powerful methodological tools for low-cost, rapid, and ethically uncomplicated hypothesis testing, manipulation screening, and analysis pre-testing [11]. While they cannot and should not replace human participants in confirmatory studies, they may serve as fast, cost-effective approximations of expected patterns. For example, LLMs could help detect underpowered designs, flag overly strong or weak manipulations, or serve as surrogates when working with vulnerable, unreachable, or ethically sensitive populations. They might also be used to conduct “virtual pilots” that bypass ethical review delays and other bureaucratic barriers.\u003c/p\u003e \u003cp\u003eRecent work has shown the growing promise of LLMs in simulating complex experimental or clinical scenarios. For instance, [12] demonstrated that LLMs could simulate a randomized clinical trial in epilepsy, generating realistic seizure diaries, treatment notes, and clinical outcomes that closely mirrored human analysis. Their findings highlight the feasibility of using LLMs to reconstruct clinical trials and extract meaningful treatment effects within a medical context. Similarly, [13] replicated over 150 psychological experiments using GPT-4 and found strong alignment between model outputs and original human findings. However, both studies relied on access to full experimental materials either detailed clinical narratives [12] or published descriptions and protocols [13].\u003c/p\u003e \u003cp\u003eOur approach differs from prior work in several key ways. First, we relied only on information contained in pre-registration documents rather than full materials or published outcomes. These studies are currently being prepared for publication in reputable journals at the time of writing this manuscript. Apart from the publicly available pre-registration documents, the full study materials and results are not yet available online, thus aligning more closely with the minimal information typically available before data collection. Second, we simulated raw participant-level data rather than summary statistics, enabling full re-analysis using the original statistical pipelines. Third, we compared the performance of multiple LLMs (ChatGPT, Gemini, and Perplexity), expanding the evaluation beyond GPT-4. Finally, we tested this framework across two distinct experimental domains: a cognitive task comparing student performance in traditional versus AI-assisted math reasoning, and a social judgment task assessing perceived loyalty based on expressed beliefs about a shared cause.\u003c/p\u003e\n\u003ch3\u003eA. The Current Studies\u003c/h3\u003e\n\u003cp\u003eIn this paper, we test whether LLMs can generate simulated datasets that reproduce the core findings of real RCTs using only the information provided in a study’s pre-registration. Specifically, we examine whether LLM-generated data align with human data in effect direction, statistical significance, and effect magnitude. We also assess whether this alignment generalizes across different LLMs (ChatGPT, Gemini, Perplexity) and across distinct experimental domains.\u003c/p\u003e \u003cp\u003eIn Experiment 1, we simulate data from an experiment comparing student performance on a math reasoning task under traditional versus AI-assisted conditions. In Experiment 2, we simulate data from a social judgment experiment investigating how participants judge the loyalty of a target based on whether the target expresses optimistic or pessimistic beliefs about a shared cause. These studies allow us to test the capacity of LLMs to replicate both analytic and social cognitive responses.\u003c/p\u003e \u003cp\u003eWe hypothesize that LLM-simulated data will show a similar pattern to the original datasets (H1: same direction of effect; H2: replication of statistical significance). If supported, this would demonstrate that LLMs are not only useful for language tasks but may also be valid tools for piloting behavioral experiments, estimating effect sizes, testing statistical models, and improving study design before data collection.\u003c/p\u003e \u003cp\u003eThis question carries both theoretical and methodological significance. If LLMs can reliably approximate real-world data patterns, they may become valuable instruments in the behavioral research pipeline enabling early-stage hypothesis testing, efficient pre-registration design, sample size estimation, and robustness checks. This paper represents a first step in systematically evaluating that potential.\u003c/p\u003e \u003cp\u003eBelow, we outline the methods and results of this investigation.\u003c/p\u003e "},{"header":"METHODOLOGY","content":"\u003ch2\u003eOpen science\u003c/h2\u003e\u003cp\u003eAll study materials, simulated datasets, analysis scripts, and pre-registrations are available on the Open Science Framework: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://osf.io/7bmus/\u003c/span\u003e\u003cspan class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Hypotheses, target sample sizes, and analysis plans were pre-registered prior to any simulation or data analysis. All experimental conditions, measured variables, and statistical comparisons reported here were pre-specified.\u003c/p\u003e\u003ch3\u003eSimulated data\u003c/h3\u003e\u003cp\u003eTo evaluate the capacity of large language models (LLMs) to reproduce human experimental data, we simulated participant-level datasets for two “yet to be published in journals” pre-registered randomized controlled trials (Study 1: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://osf.io/8ewgz/\u003c/span\u003e\u003cspan class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e and Study 2: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://osf.io/vtwn6/\u003c/span\u003e\u003cspan class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). We used three state-of-the-art LLMs ChatGPT, Gemini, and Perplexity to generate simulated data corresponding to each study (we refer the reader to the \u003cspan class=\"InternalRef\"\u003eAppendix\u003c/span\u003e Section for the links to the simulated data). The original studies involved undergraduate participants completing reasoning or judgment tasks under experimentally manipulated conditions.\u003c/p\u003e\u003cp\u003eFor each study, we prepared a standardized input prompt (see \u003cspan class=\"InternalRef\"\u003eAppendix\u003c/span\u003e). This prompt excluded any information about the authors or their affiliations and contained a complete copy of the study preregistration, including sample size, study design, and variable structure. The LLMs were instructed to generate raw participant-level data in tabular CSV format, matching the variables, sample size, and condition structure of the real datasets. At the time the LLM simulations were conducted, both studies had completed data collection and analysis but had not yet been published in peer-reviewed outlets. The simulations were generated exclusively from preregistration documents, which specified the study designs, sample sizes, variables, and planned analyses, but did not contain outcome data or statistical results. This ensured that simulated datasets were generated independently of the empirical findings. Although the authors were involved in the original studies, no empirical results were accessible to the LLMs during simulation. At the time of revision, Study 1 was published [14], while Study 2 remains under review.\u003c/p\u003e\u003ch3\u003eLLM assumptions and model configuration\u003c/h3\u003e\u003cp\u003eTo generate simulated participant-level data, we relied on large language models (LLMs) under a set of explicit modeling assumptions. First, we assumed that LLMs can approximate aggregate human response patterns when provided with structured experimental descriptions, even though they do not possess lived experience or true cognition. The models were therefore treated as probabilistic generators of plausible behavioral data rather than as psychological agents. Second, we assumed conditional independence across simulated participants. Each generated response was treated as an independent draw, analogous to sampling individual human participants, rather than as a continuation of prior model outputs. This assumption aligns with standard practices in simulation-based behavioral modeling. Third, we assumed that LLMs could approximate distributional properties (e.g., central tendency and variance) implied by the experimental design without being explicitly instructed to reproduce any empirical results.\u003c/p\u003e\u003cp\u003eFinally, all simulations were conducted using publicly accessible versions of ChatGPT, Gemini, and Perplexity available at the time of data generation (July 2024). LLMs may encode implicit biases or priors derived from their training data, which can influence response tendencies, effect magnitudes, or variability. These biases are not directly observable and may differ across models. As such, differences in effect magnitude across ChatGPT, Gemini, and Perplexity are interpreted as model-specific behavioral tendencies rather than estimation error.\u003c/p\u003e\u003ch3\u003ePrompting procedure and simulation Logic\u003c/h3\u003e\u003cp\u003eFor each study, we used a standardized prompting procedure designed to mirror how a researcher might request simulated experimental data from an LLM. Each model received the same prompt structure, consisting of three components: (a) task framing: the model was instructed that it was simulating participant-level data from a completed randomized controlled trial, based on a formally pre-registered study \u003cem\u003e“You are simulating participant-level data from a completed randomized controlled trial (RCT) experiment which pre-registration is presented below”\u003c/em\u003e; (b) study specification: The full preregistration text was provided, including the study design, experimental conditions, outcome variables, and planned analyses. No information about the original study outcomes or results was included; (c) output instruction: the model was instructed to generate a downloadable tabular dataset (CSV format) containing participant-level observations matching the described sample size, variables, and experimental structure. The standardized prompt concluded with the instruction: \u003cem\u003e“please return the simulated data in downloadable tabular CSV format, suitable for analysis\".\u003c/em\u003e All prompts and generated datasets are archived and publicly accessible via the Open Science Framework (see \u003cspan class=\"InternalRef\"\u003eAppendix\u003c/span\u003e for direct links).\u003c/p\u003e\u003cp\u003e Simulation logic followed a one-pass generation approach: each model generated a complete dataset per study without iterative correction, optimization, or post-hoc adjustment. The resulting datasets were treated as fixed outputs and analyzed using the same statistical pipeline applied to the human data. This approach ensured that any observed alignment or divergence between human and simulated data emerged from the model’s internal generative processes rather than from researcher intervention.\u003c/p\u003e\u003ch2\u003eExperiment 1\u003c/h2\u003e\u003cp\u003eIn the simulated study, participants solved an infinite geometric series problem. They were randomly assigned to one of two conditions: AI-assisted (using ChatGPT) or traditional (using textbooks or handwritten notes). Accuracy was scored on a 3-point scale, and time to completion was recorded in seconds. LLMs simulated this between-subjects structure, generating values for the two key dependent variables: Accuracy and Speed (later converted to minutes). A total of 92 undergraduate students from Emmanuel Alayande University of Education, Oyo, participated in the study. Participants were randomly assigned to one of two conditions: a traditional condition, in which they used textbooks and class notes, or an AI-assisted condition, in which they were permitted to use ChatGPT while solving a mathematical problem involving the sum to infinity of a geometric series. Accuracy was scored using a three-point rubric reflecting procedural and computational correctness, and completion time was recorded in seconds. An a priori power analysis (α = .05, power = .85, expected effect size d = 0.30) indicated a required sample size of approximately 240 participants. Due to recruitment constraints, complete data were obtained from 92 participants, and all analyses were conducted on this available sample. The resulting human data provided the benchmark against which simulated LLM-generated datasets were compared.\u003c/p\u003e\u003ch3\u003eExperiment 2\u003c/h3\u003e\u003cp\u003eThe simulated study examined whether individuals attribute lower loyalty to employees who express pessimism about the future of their organizations. Participants were randomly assigned to one of two conditions: reading about an individual expressing either an optimistic or pessimistic belief about their organization’s future performance. They were then asked to judge the individual’s loyalty to the organization. The key dependent variable was “loyalty attribution”, measured on a continuous 1–7 scale. LLMs simulated the same between-subjects design and outcome distribution as in the original data. We recruited 277 participants from the Prolific platform (190 female, 85 male, 1 non-binary, and 1 unreported; M\u003csub\u003e\u003cem\u003eage\u003c/em\u003e\u003c/sub\u003e = 40 years). Four participants were excluded for failing an attention check, resulting in a final analytic sample of 273 participants.\u003c/p\u003e\u003cp\u003eParticipants were randomly assigned to read a vignette describing an employee expressing either optimistic or pessimistic beliefs about their organization’s future. They then rated the perceived loyalty of the target individual using a 7-point Likert scale. Sample size was determined via an a priori power analysis targeting 85% power, α = .05, and a medium effect size (f = .20). The resulting human data provided the benchmark against which simulated LLM-generated datasets were compared.\u003c/p\u003e\u003cp\u003eFor both Experiments, LLM-generated datasets were evaluated relative to the human data using identical statistical procedures. These included independent-samples \u003cem\u003et\u003c/em\u003e-tests, estimation of effect sizes, and inspection of confidence intervals.\u003c/p\u003e\u003ch3\u003eAnalysis Pipeline\u003c/h3\u003e\u003cp\u003eFor each study and dataset (Real, ChatGPT, Gemini, and Perplexity), we conducted parallel analyses to compare outcome patterns across conditions. The same R scripts and statistical models were used for all datasets. For both studies, we computed mean differences between groups and tested statistical significance using independent t-tests. Visualizations included bar plots with standard error bars and significance levels annotated.\u003c/p\u003e\u003ch2\u003eMeasures\u003c/h2\u003e\u003cp\u003eWe simulated the same measures used in the original experiments. In Experiment 1, accuracy was scored on a three-point rubric based on procedural and computational correctness of the mathematical problem given to the participants. Speed was measured as the total time taken to complete the task, recorded in seconds. In Experiment 2, perceived loyalty was measured by asking participants to rate the loyalty of a target individual on a 1–7 scale after reading a vignette describing the individual’s expressed belief about an organization.\u003c/p\u003e\u003ch2\u003eValidation\u003c/h2\u003e\u003cp\u003eTo evaluate alignment between simulated and human data, we compared effect directions, statistical significance patterns, and relative effect magnitudes across datasets. For each experiment, we assessed whether LLM-generated data reproduced the direction of the original effects and whether statistical conclusions (significant vs. non-significant) were consistent with those observed in human samples. We further examined relative differences in effect magnitude across models to identify systematic over- or under-estimation tendencies. While we did not apply formal equivalence testing or correlation-based similarity metrics, this structured comparison provides a transparent and interpretable basis for evaluating alignment between simulated and empirical results.\u003c/p\u003e"},{"header":"RESULTS","content":"\u003cp\u003eThis section presents the results of the human and LLM-simulated datasets across both studies. We first reported descriptive and inferential statistics for each experimental condition, followed by a comparison of effect direction, magnitude, and statistical significance across human and model-generated data\u003c/p\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eExperiment 1\u003c/h2\u003e \u003cp\u003eWe compared accuracy and completion time (speed) between AI-assisted and traditional groups across four datasets: Real, ChatGPT, Gemini, and Perplexity. Results are reported separately for human and LLM-generated datasets, followed by direct comparisons focusing on effect direction, statistical significance, and relative magnitude.\u003c/p\u003e \u003cp\u003e \u003cb\u003eAccuracy\u003c/b\u003e. In all four datasets, students in the AI-assisted group consistently outperformed those in the traditional group on accuracy. In the Real dataset, the AI group achieved significantly higher accuracy scores (M = 2.91, SD = 0.96) compared to the traditional group (M = 1.85, SD = 1.13), t(43.77) = 4.54, p \u0026lt; .001. The ChatGPT dataset showed a similar pattern: the AI group (M = 2.38, SD = 0.86) outperformed the traditional group (M = 1.64, SD = 1.09), t(70.14) = 3.50, p = .0008. In the Gemini dataset, AI-assisted students (M = 2.08, SD = 0.76) again outperformed the traditional group (M = 0.97, SD = 0.74), t(82.81) = 6.97, p \u0026lt; .001. The Perplexity dataset mirrored these findings, with the AI group (M = 2.43, SD = 0.61) scoring significantly higher than the traditional group (M = 1.64, SD = 0.49), t(89.25) = 6.97, p \u0026lt; .001. These consistent differences indicate a robust accuracy advantage for students with AI assistance.\u003c/p\u003e \u003cp\u003e \u003cb\u003eSpeed\u003c/b\u003e. Completion times showed mixed patterns across datasets. In the Real dataset, there was no statistically significant difference in speed between the AI (M = 372.81 seconds, SD = 106.52) and traditional (M = 391.56 seconds, SD = 122.13) groups, t(89.82) = -0.91, p = .36. However, in all three simulated datasets, the AI group completed the task significantly faster. In the ChatGPT dataset, AI participants (M = 299.35 seconds, SD = 49.58) were faster than traditional participants (M = 400.88 seconds, SD = 60.46), t(71.96) = -8.57, p \u0026lt; .001. In Gemini, AI students (M = 442.21 seconds, SD = 49.17) were substantially quicker than the traditional group (M = 572.05 seconds, SD = 31.86), t(88.72) = -15.33, p \u0026lt; .001. The Perplexity dataset showed the most pronounced speed difference, with the AI group (M = 102.77 seconds, SD = 11.30) completing the task faster than the traditional group (M = 127.26 seconds, SD = 12.96), t(74.93) = -9.43, p \u0026lt; .001. These results suggest a consistent speed advantage for the AI group in simulated environments. The result is illustrated in the Fig.\u0026nbsp;1 below.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eSimulated and actual means for accuracy (max = 3) and task completion time (in minutes) in Experiment 1 across conditions (Traditional vs. AI-Assisted). Each column represents a model or dataset type (ChatGPT, Gemini, Perplexity, Real).\u003c/p\u003e \u003cp\u003e \u003cb\u003eDiscussion.\u003c/b\u003e The present findings provide strong support for both hypotheses concerning the fidelity of LLM-simulated data in mimicking real participant data from a randomized controlled experiment. Across all datasets: Real, ChatGPT, Gemini, and Perplexity, the direction of the effect was consistent: participants in the AI-assisted group outperformed those in the traditional group in terms of accuracy. This replicates the pattern observed in the original real-world data and supports our prediction that LLMs can reliably reproduce the direction of treatment effects.\u003c/p\u003e \u003cp\u003eSimilarly, statistical significance was also replicated across datasets. In all four cases, the difference in accuracy between AI-assisted and traditional groups was statistically significant, indicating that LLMs can capture not only the qualitative trend but also the inferential strength of observed effects. Although there was some divergence in the speed outcome because the real data showed no significant difference while simulated datasets did. This variation is in principle plausible. Real-world performance is influenced by non-task-related variance (e.g., hesitation, distractions), which simulations might not model. Importantly, this divergence does not undermine our prediction, as our hypothesis focused on replicating patterns where statistical differences exist, and accuracy was also one of the primary dependent variables of interest.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eExperiment 2\u003c/h2\u003e \u003cp\u003e \u003cb\u003ePerceived loyalty.\u003c/b\u003e Participants judged the loyalty of a target individual expressing either an optimistic or pessimistic belief about their organization. In real data, those in the optimistic condition rated the individual as significantly more loyal (M = 5.53, SD = 1.21) than those in the pessimistic condition (M = 4.66, SD = 1.45), t(263.49) = 5.36, p \u0026lt; .001. ChatGPT generated a comparable effect, with higher loyalty ratings for the optimistic (M = 5.36, SD = 0.88) than the pessimistic condition (M = 3.56, SD = 1.25), t(244.52) = 13.82, p \u0026lt; .001. Gemini produced a more extreme contrast between conditions, assigning a mean loyalty of 6.00 (SD = 0.82) to the optimistic condition and 2.01 (SD = 0.82) to the pessimistic condition, t(270.99) = 40.36. Similarly, Perplexity showed a pronounced difference, with loyalty ratings of 5.34 (SD = 1.10) for the optimistic condition and 3.60 (SD = 0.90) for the pessimistic condition, t(260.65) = 14.30, p \u0026lt; .001. These results indicate that all LLMs captured both the direction and statistical significance of the real effect, though Gemini and Perplexity simulated more polarized responses than observed in human data. The result is presented in Fig.\u0026nbsp;2 below.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eSimulated and actual mean perceived loyalty ratings in Experiment 2 across belief conditions (Optimistic vs. Pessimistic). Each column represents a model or dataset type (ChatGPT, Gemini, Perplexity, Real).\u003c/p\u003e \u003cp\u003e \u003cb\u003eDiscussion.\u003c/b\u003e Experiment 2 provides further support for the potential of LLMs to simulate human-like data in randomized experiments. All models successfully replicated the hypothesized effect: individuals expressing pessimistic beliefs were perceived as less loyal than those expressing optimism about their organization’s future (Fig.\u0026nbsp;2). This consistency with real human responses suggests that LLMs encode plausible sociocognitive associations, including moral or group-normative evaluations of belief expression.\u003c/p\u003e \u003cp\u003eHowever, some discrepancies emerged in the magnitude of the effects. While the real data showed a moderate difference in perceived loyalty, Gemini and Perplexity produced highly polarized responses, suggesting a more categorical moral evaluation. This overamplification might reflect LLMs’ tendency to reinforce prototypical social judgments rather than reflect natural variability in human reasoning. In contrast, ChatGPT yielded results closer in scale to the real data, suggesting that some models may be more calibrated to human-like gradience.\u003c/p\u003e \u003cp\u003eThese findings highlight both the promise and caution needed when using LLMs for behavioral simulation. On one hand, LLMs can reproduce statistical outcomes aligned with human data, making them useful for theory testing, piloting experimental designs, estimating effect sizes, and probing the sensitivity of analyses prior to data collection.\u003c/p\u003e \u003cp\u003eThe table below presented the comparison of effect sizes between human participants and large language models (ChatGPT, Perplexity, and Gemini) across two experimental tasks. All models reproduced the direction of observed human effects across tasks. However, effect magnitudes varied systematically by model, with Gemini consistently producing inflated estimates and Perplexity showing the closest alignment to human data, particularly for judgment-based outcomes. Time-based measures showed larger divergence than accuracy-based outcomes.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003ctable id=\"Tab1\" border=\"1\"\u003e \u003ccaption\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eEffect sizes (mean differences) for human and LLM-generated data across two experiments. Positive values indicate stronger effects in the intervention condition; negative values indicate faster performance.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003c/colgroup\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\"\u003e \u003cp\u003eExperiment\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eMeasure\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eHuman\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eChatGPT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003ePerplexity\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eGemini\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" rowspan=\"2\"\u003e \u003cp\u003eExperiment 1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e0.74\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e0.79\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.11\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eSpeed\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e−18.75\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e−101.53\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e−24.69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e−129.84\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eExperiment 2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eJudgment / Evaluation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.74\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e3.99\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e Alignment between human and LLM-generated data was evaluated using three complementary criteria: (i) consistency in effect direction, (ii) similarity in relative effect magnitude, and (iii) preservation of qualitative patterns across experimental conditions. As shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e, across both studies, LLM outputs matched the direction of human effects and reproduced the relative ordering of conditions. Although absolute effect sizes differed particularly for temporal measures, the observed patterns remained stable across models. These results indicate that LLMs capture structural regularities in the data-generating process, even when numerical precision differs from human responses.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003cdiv id=\"Sec17\" class=\"Section3\"\u003e \u003c/div\u003e \u003c/div\u003e "},{"header":"GENERAL DISCUSSION","content":"\u003ch2\u003eGeneral discussion\u003c/h2\u003e\u003cp\u003eAcross two preregistered studies, we evaluated the capacity of large language models (LLMs) to simulate participant-level data from randomized controlled experiments. Both studies tested whether data generated by LLMs would replicate the direction and statistical significance of effects found in real human data. Our results offer consistent support for both hypotheses: LLM-simulated datasets mirrored real data in effect direction and successfully recovered the original patterns of statistical significance.\u003c/p\u003e\u003cp\u003eIn Experiment 1, we found that LLMs replicated the observed accuracy advantage for students using AI assistance, with only minor discrepancies in completion time effects. This result aligns studies showing that LLMs can accurately reproduce behavioral trends and main effects in structured tasks, closely matching human data in direction and significance [13]. However, in the \"speed\" measure, the discrepancies in effect sizes may reflect the known issue that LLMs may exaggerate or understate effects relative to human variability.\u003c/p\u003e\u003cp\u003eIn Experiment 2, LLMs successfully captured the predicted effect; individuals expressing pessimistic beliefs were perceived as less loyal compared to optimistic individuals. We found a qualitative pattern of results held across all tested LLMs. However, effect sizes varied: some models (e.g., Gemini) produced more polarized results than observed in human data. This variability in effect magnitude highlights previous findings regarding model-specific exaggeration or rigidity in simulated responses.\u003c/p\u003e\u003cp\u003eTaken together, by implication, our findings suggest that LLMs are indeed capable of generating behaviorally plausible data that broadly aligns with results from real experiments. This convergence supports the growing literature that LLMs can act as preliminary proxies for human subjects, especially for simulating trial data to test hypotheses, exploring edge cases and alternative explanations, estimating effect sizes and sample size requirements and identifying potential analysis pitfalls before running real studies [15].\u003c/p\u003e\u003cp\u003eSubstantive differences in effect magnitude and variance across models stress the need for caution and critical interpretation. Existing research argues that LLMs reflect aggregated linguistic and statistical regularities found in their training data, rather than direct lived human experience, potentially leading to the amplification or suppression of certain effects [16].\u003c/p\u003e\u003ch2\u003eLimits and divergence patterns in LLM simulations\u003c/h2\u003e\u003cp\u003eAlthough the LLM simulations successfully reproduced the direction and statistical significance of key effects, several systematic divergences from human data were observed. Most notably, effect magnitudes varied across models. Gemini consistently produced larger effect sizes than those observed in human participants, suggesting a tendency toward effect amplification. In contrast, Perplexity yielded estimates more closely aligned with empirical data, while ChatGPT often fell between these extremes. These differences indicate that LLMs vary not only in linguistic fluency but also in how strongly they encode and express latent experimental manipulations.\u003c/p\u003e\u003cp\u003eA second divergence concerned variability. Across both studies, LLM-generated datasets exhibited reduced variance relative to human data. This likely reflects the tendency of language models to generate internally coherent and norm-consistent outputs, which may suppress natural variability present in human reasoning. As a result, LLM simulations may underestimate noise and overstate consistency in behavioral responses.\u003c/p\u003e\u003cp\u003eFinally, divergence patterns appeared to depend on task structure. LLMs aligned more closely with human data in the structured mathematical reasoning task than in the socially interpretive judgment task.\u003c/p\u003e\u003ch2\u003eLIMITATION\u003c/h2\u003e\u003cp\u003eFirst, LLM-simulated responses rely heavily on prompt design, lack genuine experience, and may fail to capture critical confounds present in real participants, which calls into question their ecological validity. Different LLMs can produce systematically varying levels of effect polarization or attenuation, as observed in Experiment 2 sometimes overstating treatment effects or underrepresenting the variability typical in human samples. Moreover, LLMs are highly sensitive to prompt structure and contextual framing; minor wording adjustments can produce substantially different outputs. This raises concerns about reproducibility and transparency unless prompts are fully standardized and disclosed like we did here in our work.\u003c/p\u003e\u003cp\u003eAnother limitation is that we did not conduct formal equivalence testing or correlation-based similarity metrics, as the goal of this study was not to establish statistical interchangeability between human and synthetic data. Instead, our objective in this study was to assess whether LLM-generated datasets preserve the qualitative structure of empirical findings, namely, effect direction, relative magnitude, and pattern consistency across conditions. Given that LLMs do not sample from the same generative process as human participants, traditional equivalent frameworks may give a misleading impression of comparability.\u003c/p\u003e\u003cp\u003eFinally, there are broader epistemological and ethical concerns regarding the use of synthetic data in scientific research. LLMs are best positioned as tools for hypothesis generation, estimate refinement, and early-stage design and not as replacements for confirmatory empirical testing.\u003c/p\u003e\u003ch2\u003eSUGGESTION FOR FUTURE STUDIES\u003c/h2\u003e\u003cp\u003eFuture work can build on these findings in several directions. First, replication across different domains and task types is necessary to determine the generalizability of LLM-simulated behavioural data. Tasks involving open-ended reasoning, emotion-laden judgments, or more complex social dynamics may reveal different levels of fidelity. Second, incorporating demographic conditioning (e.g., simulating responses from specific age groups or cultural backgrounds) could improve the ecological validity of the outputs and align them more closely with targeted participant samples. Third, examining whether simulations vary across different prompt structures or levels of prompting specificity could uncover best practices for achieving human-like outputs.\u003c/p\u003e\u003cp\u003eAdditionally, future research should investigate whether access to more powerful or updated models (e.g., GPT-4 Turbo, Gemini 1.5 Pro, Claude 3 Opus) improves alignment with real data. These models may have enhanced reasoning capabilities, larger context windows, or more consistent behaviour, which could affect the fidelity of their simulations. Comparing free vs. pro versions and open-source alternatives would help identify the most cost-effective and scientifically robust tools for simulation-based study design, effect size estimation, and power analysis.\u003c/p\u003e"},{"header":"CONCLUSION","content":"\u003cp\u003eThis study provides evidence that large language models (LLMs) can simulate participant-level data that meaningfully resemble human data in randomized controlled experiments. Across two studies, we found that LLM-generated data reproduced key statistical properties of human responses, including effect directions and patterns of significance. While not perfect replicas, these simulations aligned closely enough to support their use in methodological exploration.\u003c/p\u003e \u003cp\u003eLLMs offer a flexible and powerful complement to empirical research workflows. Their ability to generate datasets that mirror the statistical characteristics of human results hold promise for improving the efficiency, cost, and rigor of psychological research design and piloting. LLM simulations can assist with estimating plausible effect sizes, testing manipulations, assessing power, and refining analysis pipelines especially during early-stage development or in contexts where access to human samples is limited or ethically constrained.\u003c/p\u003e \u003cp\u003eHowever, researchers must remain vigilant regarding the interpretive limits of LLM-generated data. These models lack genuine experience, are sensitive to prompt design, and may omit critical real-world confounds. As such, findings from LLM simulations should always be validated with real human participants before drawing substantive conclusions.\u003c/p\u003e \u003cp\u003eIn sum, this study demonstrates that LLMs can serve as credible stand-ins for human subjects in the modelling and forecasting of experimental outcomes, provided that modelling assumptions and task specifications are well defined. When used responsibly, LLMs can become powerful tools for hypothesis generation and design prototyping not substitutes for empirical confirmation.\u003c/p\u003e "},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003e \u003cspan type=\"SmallCaps\" class=\"SmallCaps\" name=\"Emphasis\"\u003eConflict of Interest\u003c/span\u003e \u003c/h2\u003e \u003cp\u003eThe authors declare no conflict of interest\u003c/p\u003e \u003c/p\u003e\u003cp\u003e \u003ch2\u003e \u003cb\u003eConsent to Participate\u003c/b\u003e:\u003c/h2\u003e \u003cp\u003e Informed consent was obtained from all participants involved in the study.\u003c/p\u003e \u003c/p\u003e\u003cp\u003e \u003ch2\u003eClinical Trial Number\u003c/h2\u003e \u003cp\u003enot applicable.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003e \u003cb\u003eConsent to Publish\u003c/b\u003e:\u003c/strong\u003e \u003cp\u003enot applicable.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eEthics Declaration\u003c/strong\u003e \u003cp\u003e All procedures adhered to ethical research standards and open science principles of the College of Accounting Sciences Research Ethics Review Committee (RERC), University of South Africa, Pretoria, under reference number 10286.\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eFunding:\u003c/h2\u003e \u003cp\u003eThis research received no external funding.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eGL and EE conceptualised the study. GL conducted the research, analysed the data, and wrote the manuscript. EE provided continuous feedback throughout the project and suggested improvements from early development to final revision. All authors approved the final version.\u003c/p\u003e\u003ch2\u003eAcknowledgement\u003c/h2\u003e\u003cp\u003eThis research received no external funding.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eData and analysis scripts for all studies, together with pre-registration documents for each experiment, are publicly available on the Open Science Framework (OSF): [https://osf.io/7bmus/](https:/osf.io/7bmus) . To evaluate the capacity of large language models (LLMs) to reproduce human experimental data, we additionally simulated participant-level datasets for two pre-registered randomized controlled trials (Study 1: [https://osf.io/8ewgz/](https:/osf.io/8ewgz) and Study 2: [https://osf.io/vtwn6/](https:/osf.io/vtwn6) ). These studies, which are not yet published in journals, involved undergraduate participants completing reasoning or judgment tasks under experimentally manipulated conditions. The simulated datasets were generated using three state-of-the-art LLMs: ChatGPT ( [https://chatgpt.com/share/687fda3d-0448-800d-94b8-34dc7ed82ee6](https:/chatgpt.com/share/687fda3d-0448-800d-94b8-34dc7ed82ee6) ), Gemini ( [https://g.co/gemini/share/7da0fa1d4fb8](https:/g.co/gemini/share/7da0fa1d4fb8) ), and Perplexity ( [https://www.perplexity.ai/search/you-are-simulating-participant-w4nHtKv1Tj.rmTkGQpfOyA](https:/www.perplexity.ai/search/you-are-simulating-participant-w4nHtKv1Tj.rmTkGQpfOyA) ). Data and analysis scripts for all the studies, as well as pre-registration documents for all experiments, can be accessed on the Open Science Framework at [https://osf.io/7bmus/](https:/osf.io/7bmus) .\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eE. Hariton and J. J. Locascio, \u0026ldquo;Randomised controlled trials\u0026mdash;the gold standard for effectiveness research,\u0026rdquo; \u003cem\u003eBJOG : an international journal of obstetrics and gynaecology\u003c/em\u003e, vol. 125, no. 13, p. 1716, Dec. 2018, https://doi.org/10.1111/1471-0528.15199.\u003c/li\u003e\n\u003cli\u003eB. Brighton, M. Bhandari, Tornetta, and D. T. Felson, \u0026ldquo;Hierarchy of Evidence: From Case Reports to Randomized Controlled Trials,\u0026rdquo; \u003cem\u003eClinical Orthopaedics and Related Research\u0026reg;\u003c/em\u003e, vol. 413, p. 19, Aug. 2003, https://doi.org/10.1097/01.blo.0000079323.41006.12.\u003c/li\u003e\n\u003cli\u003eA. Griessbach \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;Resource use and costs of investigator-sponsored randomized clinical trials in Switzerland, Germany, and the United Kingdom: a metaresearch study,\u0026rdquo; \u003cem\u003eJournal of Clinical Epidemiology\u003c/em\u003e, vol. 176, p. 111536, Dec. 2024, https://doi.org/10.1016/j.jclinepi.2024.111536.\u003c/li\u003e\n\u003cli\u003eE. Peer, L. Brandimarte, S. Samat, and A. Acquisti, \u0026ldquo;Beyond the Turk: Alternative platforms for crowdsourcing behavioral research,\u0026rdquo; \u003cem\u003eJournal of Experimental Social Psychology\u003c/em\u003e, vol. 70, pp. 153\u0026ndash;163, May 2017, https://doi.org/10.1016/j.jesp.2017.01.006.\u003c/li\u003e\n\u003cli\u003eS. Balasubramanian, \u0026ldquo;EXPLORING THE CAPABILITIES OF CHATGPT IN NATURAL LANGUAGE PROCESSING TASKS,\u0026rdquo; vol. 2, pp. 7\u0026ndash;17, Dec. 2023, https://doi.org/10.17605/OSF.IO/XJYMQ.\u003c/li\u003e\n\u003cli\u003eS. J. Alghamdi, \u0026ldquo;Prediction of Concrete\u0026rsquo;s Compressive Strength via Artificial Neural Network Trained on Synthetic Data,\u0026rdquo; \u003cem\u003eEngineering, Technology \u0026amp; Applied Science Research\u003c/em\u003e, vol. 13, no. 6, pp. 12404\u0026ndash;12408, Dec. 2023, https://doi.org/10.48084/etasr.6560.\u003c/li\u003e\n\u003cli\u003eV. Cheung, M. Maier, and F. Lieder, \u0026ldquo;Large language models show amplified cognitive biases in moral decision-making,\u0026rdquo; \u003cem\u003eProceedings of the National Academy of Sciences\u003c/em\u003e, vol. 122, no. 25, p. e2412015122, June 2025, https://doi.org/10.1073/pnas.2412015122.\u003c/li\u003e\n\u003cli\u003eG. Simmons, \u0026ldquo;Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity,\u0026rdquo; \u003cem\u003earXiv.org\u003c/em\u003e, Sept. 24, 2022. https://arxiv.org/abs/2209.12106v2 (accessed Dec. 04, 2025).\u003c/li\u003e\n\u003cli\u003eL. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate, \u0026ldquo;Out of One, Many: Using Language Models to Simulate Human Samples,\u0026rdquo; \u003cem\u003ePolitical Analysis\u003c/em\u003e, vol. 31, no. 3, pp. 337\u0026ndash;351, July 2023, https://doi.org/10.1017/pan.2023.2.\u003c/li\u003e\n\u003cli\u003eB. Yang \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge,\u0026rdquo; \u003cem\u003eProc. ACM Interact. Mob. Wearable Ubiquitous Technol.\u003c/em\u003e, vol. 8, no. 4, p. 153:1-153:29, Nov. 2024, https://doi.org/10.1145/3699765.\u003c/li\u003e\n\u003cli\u003eS. Bubeck \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;Sparks of Artificial General Intelligence: Early experiments with GPT-4.\u0026rdquo; arXiv, Apr. 13, 2023, https://doi.org/10.48550/arXiv.2303.12712.\u003c/li\u003e\n\u003cli\u003eD. M. Goldenholz, S. R. Goldenholz, S. Habib, and M. B. Westover, \u0026ldquo;Inductive reasoning with large language models: A simulated randomized controlled trial for epilepsy,\u0026rdquo; \u003cem\u003eEpilepsy Research\u003c/em\u003e, vol. 211, p. 107532, Mar. 2025, https://doi.org/10.1016/j.eplepsyres.2025.107532.\u003c/li\u003e\n\u003cli\u003eZ. Cui, N. Li, and H. Zhou, \u0026ldquo;Can AI Replace Human Subjects? A Large-Scale Replication of Psychological Experiments with LLMs.\u0026rdquo; Social Science Research Network, Rochester, NY, Aug. 25, 2024, https://doi.org/10.2139/ssrn.4940173.\u003c/li\u003e\n\u003cli\u003eS. O. Sangoniyi, G. Lukumon, and A. Maharaj, \u0026ldquo;Harnessing AI-powered tools in mathematics education to improve problem-solving skills for economic and technological advancement,\u0026rdquo; \u003cem\u003eCOAST\u003c/em\u003e, vol. 7, no. 2, pp. 1351\u0026ndash;1357, Nov. 2025, Accessed: Jan. 01, 2026. [Online]. Available: https://www.ajol.info/index.php/coast/article/view/310489.\u003c/li\u003e\n\u003cli\u003eL. Ke, S. Tong, P. Cheng, and K. Peng, \u0026ldquo;Exploring the frontiers of LLMs in psychological applications: a comprehensive review,\u0026rdquo; \u003cem\u003eArtificial Intelligence Review\u003c/em\u003e, vol. 58, no. 10, p. 305, July 2025, https://doi.org/10.1007/s10462-025-11297-5.\u003c/li\u003e\n\u003cli\u003eG. Gui and O. Toubia, \u0026ldquo;The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective,\u0026rdquo; \u003cem\u003eSSRN Electronic Journal\u003c/em\u003e, 2023, https://doi.org/10.2139/ssrn.4650172.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"iran-journal-of-computer-science","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ircs","sideBox":"Learn more about [Iran Journal of Computer Science](https://link.springer.com/journal/42044)","snPcode":"42044","submissionUrl":"https://submission.nature.com/new-submission/42044/3","title":"Iran Journal of Computer Science","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"ChatGPT, Gemini, Perplexity, LLMs, Simulation, RCTs, AI-assisted research, virtual piloting","lastPublishedDoi":"10.21203/rs.3.rs-8827610/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8827610/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eCan large language models (LLMs) simulate participant-level datasets from experimental designs such that their statistical properties, such as effect directions, magnitudes, and significance, align with those of actual human data? In this work, we tested whether LLMs can generate simulated datasets that reproduce the core findings of real randomized controlled trials (RCTs) using only the information provided in a study\u0026rsquo;s pre-registration. We assessed whether this alignment generalizes across different LLMs (ChatGPT, Gemini, Perplexity) and across distinct experimental domains, including a math reasoning task comparing student performance and a social judgment task. We found that LLM-simulated datasets mirrored the real data in effect direction and successfully recovered the original patterns of statistical significance. All models correctly reproduced the direction of human effects, though effect magnitudes varied by model, with Gemini consistently overestimating effects and Perplexity showing the closest alignment to human data. While LLMs cannot replace empirical studies, our study offers a powerful and flexible complement capable of accelerating idea testing, refining study designs, and probing the robustness of research findings before conducting real-world experiments.\u003c/p\u003e","manuscriptTitle":"LLMs in the Lab: Can AI Predict What Real Participants Do?","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-08 08:31:47","doi":"10.21203/rs.3.rs-8827610/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-05-07T13:48:38+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"323270252734074188787415053329926987215","date":"2026-04-02T14:11:48+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-02T10:26:06+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-02T10:24:08+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-02-09T11:24:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"Iran Journal of Computer Science","date":"2026-02-09T07:55:20+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"iran-journal-of-computer-science","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ircs","sideBox":"Learn more about [Iran Journal of Computer Science](https://link.springer.com/journal/42044)","snPcode":"42044","submissionUrl":"https://submission.nature.com/new-submission/42044/3","title":"Iran Journal of Computer Science","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"394a2561-a91a-4b92-9873-06fda6fa6e9a","owner":[],"postedDate":"April 8th, 2026","published":true,"recentEditorialEvents":[{"type":"editorInvitedReview","content":"","date":"2026-05-07T13:48:38+00:00","index":15,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-04-08T08:31:47+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-08 08:31:47","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8827610","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8827610","identity":"rs-8827610","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}