Sex Differences in Model free and Model based Behavior

Sex Differences in Model free and Model based Behavior | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Sex Differences in Model free and Model based Behavior Marina Lukezic, Hao Chen, Maximilian Gilger, Philipp T. Neukam, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7021287/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 13 You are reading this latest preprint version Abstract Despite men and women nearing equal opportunities in education and career, we continue to identify sex-differences in decision-making. For instance, more women are found in social fields as nursing and medicine. One existing study found higher model-free behavior in the two-step task in women and another study found model-free behavior to be associated with prosocial behavior. We conducted an exploratory analysis to see (1) whether women differ regarding model-free and model-based behavioral control from men; (2) what brain responses underlie such differences. Participants (total 98; 37 female) performed the two-stage Markov task while in the fMRI scanner. Our analysis confirmed a previous finding of increased model-free behavior in women and we additionally observed an increased neural response in dorsolateral prefrontal cortex in women following not rewarded compared to rewarded trials. This difference was not present in men. Different decision-making patterns and neural responses observed between sexes in the two-step task are possibly able to predict existing real-world behavioral differences between men and women. Biological sciences/Neuroscience Biological sciences/Psychology Social science/Psychology model-based control model-free control cognitive-ability sex difference fMRI two-stage Markov decision task Figures Figure 1 Figure 2 Figure 3 Figure 4 Sex Differences between Model free and Model based Behavior Despite us nearing gender equality, women maintain high prosocial tendencies and men continue to be stronger represented in science and engineering fields 1 , 2 . In more gender-equal countries, this difference is even more apparent, with men expressing a higher interest and more joy for Science. The opposite was found for women, where greater gender-equality was associated with less interest in Science for women 2 . Differences in career-choice and other everyday choices between sexes, may be less a consequence of opportunity, but a result of internal decision-making. Studies have shown differences in risk-taking, social behavior and competition between men and women. Women are more easily influenced by their environment when making decisions, and generally favor less-competitive circumstances. In comparison men perform better than women when competing 3 . Differences in decision-making between men and women are observed not only in real life, but also in experimental tasks. One example of such a task is the dictator game, in which participants can choose to keep money for themselves or give it to others 4 . This task showed women to make more generous contributions compared to men 4 , 5 . This is one example of how decision-making in experimental tasks can mirror real-world behavior. Our everyday decisions are governed by model based and model free control. One task designed to tease apart these different decision-making patterns is Daw’s two step task. A balance of these two approaches regulates the way we make day-to-day choices 6 , 7 and may help us investigate sex differences in decision-making in more depth. On the one hand, there is model free behavior, which is characterized by low computational costs. This behavior relies solely on the history of reward and functions so that a previously rewarded action has a high likelihood to be repeated in future. A model free learner disregards the task structure or external environment and navigates based on previously received reward 6 – 10 . On the other hand, model based behavior is computationally more costly but a more flexible means of engaging with the environment 9 , 11 , 12 . This behavior is influenced not only by reward, but also by the task structure. A model based learner is constantly evaluating reward in relation to environment and updating the value of the reward as external circumstances shift. While a model based strategy has the advantage of adapting to changes, it is likewise very costly regarding time and brain resources and is too expensive to be used for all daily actions 13 . We were curious to find out whether sex differences in decision-making could be differentiated through aid of the two-step task. To the best of our knowledge only one study by Gillan et. al. identified differences in two-step behavior between men and women. Gillan et. al. found women displayed increased model-free behavior in the larger of two experiments. Other existing studies on the two-step task primarily match sex in the patient and control groups or use it as a control variable, this explains why there is not much to refer to in the literature 14 . A study by Oguchi et.al. found a correlation between increased prosocial behavior and the model free strategy in the two-step task, in comparison elevated proself behavior correlated with the model based strategy. This study assessed social orientation scores via an online slider measure, where participants could choose to allocate money to others or themselves. The scores on social orientation were then compared to decision-making strategies in a two-step Markov decision task. The findings of the study propose that the two-step task may predict resource allocation between others and one-self 15 . Some existing studies suggest that decision-making in women is more socially oriented than in men 16 . This supports the assumption that decision-making mechanisms extracted from the two-step task can reflect social decision-making and also aligns with increased female representation in prosocial fields and could help us understand sex-mechanisms driving career and other life choices. Diving into the animal world, evidence from mouse studies suggests that male and female mice use different strategies to make adaptive decisions 17 . However, the association between sex and decision-making strategies is complex, with some studies showing increased exploratory behavior in female mice and others in male mice 18 , 19 . Although the direction of behavior is not always uniform, males and females process environments differently and perceive the environmental structure in different ways, adapting their behavior accordingly. Even though behavioral data regarding decision-making in animal studies is inconclusive, there are clear trends in reward processing between male and female rodents 19 , 20 . Females show elevated neural response after negative feedback compared to males 17 , 19 , 20 . Females also displayed higher reactivity to immediate reward independent of reward size 21 , whereas males were more likely to seek out bigger 19 more risky rewards 21 – 23 . Some studies in humans have shown higher response to reward in men and punishment in women 16 . Differences in reward processing may be guiding decision-making in men and women. The lives of men and women continue to drift in different directions, with women being more represented in social sectors, as medicine and nursing. Prosocial qualities and female sex have been linked to model free behavior in the two-step task. We conducted an exploratory analysis on an existing data set to investigate sex-differences in the two-step task. Specifically, we compared the parameters for model free and model based control, and the computational parameters: \(\:{ß}_{1}\) , \(\:{ß}_{2}\) , \(\:{\alpha\:}_{1}\) , \(\:{\alpha\:}_{2}\) , λ, π and \(\:\omega\:\) . Additionally, we looked into neural correlates between sexes, while performing the two-step task, which few studies have done so far. Specifically, we looked into the ventral striatum (VS), ventral medial prefrontal cortex (vmPFC), and dorsolateral prefrontal cortex (dlPFC), these are the key areas linked to reward processing and main areas of focus in the original study 24 . Materials and Methods Participants Data for this study originated from project B04 of the Collaborative Research Center 940 on cognition and volitional control and looked into serotonergic modulation of decision-making in the two-step task 24 . Ethical approval for the study was obtained EK 42022012 (TU Dresden) and all experiments were performed in accordance with relevant guidelines/regulations in line with the Declaration of Helsinki. Participants were recruited from a randomized sample stratified by age and gender from the registration data and invited to participate in the study. Persons without MRI safety concerns, sufficient visual capability (corrected binocular visus lower than 0.8) and without a history or current diagnosis of mental disorder were eligible for participation (for more detail see original publication by Gilger et.al. 24 ). The final sample size for behavioral analysis included 98 participants (37 female), ages 20–42 (Mean: 32.2 years, SD = 6.1 years). Ten participants needed to be excluded from imaging analysis: five participants showed strong unrepairable artefacts (strong ghosting or spiking), four participants had more than 30% repaired slices (ArtRepair Toolbox) in one session and one participant had high scan-to-scan motion, this left a final sample size of 88 participants (36 female). All participants gave written informed consent for study participation prior to the study and received monetary reimbursement (fixed amount plus bonus adjusted to task performance) at the end of the study. (For detailed information on recruitment and inclusion/exclusion criteria please see supplement) Procedure and intervention Two-step paradigm Participants completed a sequential decision-making task according to Daw 13 in a MRI scanner. (Fig. 1 ) Response patterns for stay-switch probabilities According to Daw et. Al. 13 two strategies for task performance are observed: model free and model based learning-strategies, respectively. While pure model-free behavior is characterized by sole influence of reward on previous first stage choice, model-based behavior takes into account the interaction of reward and transition for the subsequent first stage choice. Previous research suggests that participants display a mix of model free and model based behavior 25 . Analysis of behavioral data Model-free (MF) and model-based (MB) scores for each participant were calculated according to their first stage choice probability across all trials. We then used the scores to compare stay-switch probabilities between both sexes. The scores were calculated according to Daw et.al. as follows: MF-Score = P (stay|rewarded common) + P (stay|rewarded rare) – P (stay|unrewarded common) – P (stay|unrewarded rare) / 2 MB-Score = P (stay|rewarded common) – P (stay|rewarded rare) – P (stay|unrewarded common) + P (stay|unrewarded rare) / 2 Behavioral statistical analysis was carried out with SPSS 28 (IBM-SPSS, Chicago, IL, USA) with significance set at p < .05. First, we calculated mean values of model free and model based behavior for each individual across all sessions and then compared the average values between men and women. The original analysis concluded with a null finding 24 , disputing any effects of the pharmacological intervention with tryptophan on first-stage switching behavior. We proceeded to compare basic sample characteristics (age, school achievement and fluid intelligence score) in order to exclude the influence of these variables on two step behavior. 2-step behavior between sexes was analysed via simple statistical test (t-test, X2 test). (For a detailed analysis on basic sample characteristics please see supplement) Computational Modelling We further applied the seven-parameter ( \(\:{ß}_{1}\) , \(\:{ß}_{2},\:\) \(\:{\alpha\:}_{1}\) , \(\:{\alpha\:}_{2}\) , \(\:,\:\omega\:\) , λ \(\:,\:{\pi\:}\) ) reinforcement-learning model by Daw et.al. 13 to our data (See supplement for computational model equations). On the conceptual level, the inverse temperature parameter \(\:{ß}_{i}\) provides insight into choice reliability between stages and tells us how deterministic the choices are, \(\:{ß}_{1}\) and \(\:{ß}_{2}\) refer to the first and second stage, respectively. The learning rate α determines to what extend an individual updates beliefs or expectation based on new information. \(\:{\alpha\:}_{1}\) refers to the first stage learning rate and, \(\:{\alpha\:}_{2}\) the second stage, respectively (0 < α < 1) 13 . The parameter \(\:{\omega\:}\) weights the influence of model based and model free values. In the specific case of a pure model-based learner \(\:{\omega\:}\:\) = 1, for a pure model-free learner \(\:{\omega\:}=0\) would be the case. The model also includes a stage-skipping value update parameter λ, which provides an update of first-stage values based on reward prediction errors (RPE) received after the second stage (0 < λ < 1). As the task has only two steps, λ reflects the influence of the RPE received after the second stage on the first stage value 13 . A λ of zero means only the current state is eligible for updating, λ equals one means all states that have ever been visited are eligible for updating 25 . The perseveration parameter π describes the propensity of repeating the previous first stage option (p > 0) or switching (p < 0) 13 . The effect of sex on all seven parameters was tested via simple statistical tests (T-test, Mann-Whitney Test). Analysis of fMRI data We were able to include 88 participants (36 females) in the fMRI analysis after quality control. We adopted the preprocessing steps and the first-level model from Gilger et.al 24 (please see supplement material for more information). All functional brain analyses was performed with SPM8. For the 1st level statistics, one onset regressor and three parametric modulators for previous transition, previous reward and interaction of reward and transition were defined. The first stage onset regressor on previous reward provides an estimate of model free behavior, while the first stage onset regressor evaluating the interaction of transition and reward reflects model-based behavior. An additional regressor described the second stage transition and a third regressor was selected for reward at outcome onset. Our three main regions of interest (ROI) were the VS, vmPFC, and dlPFC and we selected them with help of meta-analysis software (For a detailed description on how ROIs were selected, please see Kroemer et. al. 26 ). For second level analysis we extracted the neural responses from the regions of interest and performed a t-test with sex added as a covariate. Lastly, we concluded the imaging with a whole-brain analysis: applying sex as a covariate we compared model-free, model-based behavior, as well as differences in transition and reward for second stage onset. A voxel-based threshold p uncorr .<.001 was selected. Results Sex differences in model-based and model-free parameters We compared the probability of repeating the same first stage choice as in the previous trial in men and women (Fig. 2 a). If the selection of the same first stage choice was driven by reward obtained after a choice at the second stage, the behavior is termed model free. A strategy considering both reward and transition rates, is defined as model based. The Mann-Whitney U test revealed increased model-free behavior i.e. an increased effect of previous reward in women in contrast to men of our sample (U = 476, Z=-4.782, p = < .001, r=-0.48) (See Fig. 2 b). No significant difference between men and women was observed for the model-based strategy (U = 1036, Z=-0.678, p = .498, r=-0.068) (Fig. 2 b right). Despite the differences in strategies adopted, there was no difference in final average reward between men and women (t = 0.932, p = .353, d = 0.805; r = .364) (See Fig. 2 c). Sex comparison of computational model parameters To get a more precise idea on differences in decision-making between males and females, we investigated the seven parameters according to the computational model established by Daw, et al. 13 . As also reported in Gilger et.al. we had an average \(\:\omega\:\) of 0.41 (SD = 0.29) 24 . Similar results can be found in existing literature, revealing that overall a mixture of both model-free and model-based strategies is applied in the two-step task, our sample falling in line with the overall average. On closer inspection, we found no difference in average \(\:\omega\:\) value between males and females (U = 1059, Z=-.509, p = .611, r=-.051) (Fig. 3 g). However, we did find higher learning rates for the first stage learning rate \(\:{\alpha\:}_{1}\) , (t=-2.7, p = .009, d = 0,2) (Fig. 3 c) and stage-skipping value update parameter λ (U = 753, Z=-2.75, p = .006, r=-.28) in women (Fig. 3 e). The difference in the second stage learning rate \(\:{\alpha\:}_{2}\) , was insignificant between both sexes (t=-1.9, p = .058, d = 0.21) (Fig. 3 d). No statistically significant difference was detected for either inverse temperature ( \(\:{\text{ß}}_{1}\) ; U = 936, Z=-1.41, p = .158, r=-.142; \(\:{\text{ß}}_{2}\) ; U = 962, Z=-1.22, p = .222, r=-.123 (Fig. 3 a, b) or perseveration rate π (U = 1103, Z=-0.187, p = 0.852, r=-0.019) (Fig. 3 f). Sex differences on the brain level To check whether there was a difference in neural response between sexes during performance on the two-step task, we compared the first stage onset regressor with previous reward (model free) and the first stage onset regressor with interaction of transition and reward (model based control). We investigated the three predetermined ROIs (vmPFC, dlPFC und VS) and additionally performed an exploratory whole brain analysis. The ROI analysis showed an increased response in the dlPFC for model free control in rewarded compared to unrewarded trials in women in comparison to men (t = 2.727, p = .008, d = 0.468) (Fig. 4 a). The whole-brain analysis also showed increased neural response in dlPFC in men compared to women for the model free condition when rewarded trials were compared to unrewarded trials (Fig. 4 b). No other significant differences were detected for any of the other ROIs in any condition. Discussion We conducted an exploratory analysis on an existing data set to investigate sex-differences in model based and model free control. Females in our sample navigated through the task structure highly influenced by previously obtained reward, performing in a model free manner. Females also attributed higher values to the reward prediction error, as can be seen by higher learning rates (α) and stage-skipping update parameter (λ). Reward and the future estimated value of a reward were guiding decision-making in women throughout the two-step task. In the imaging analyses we saw a higher neural response in the dlPFC in women following unrewarded trials compared to rewarded. For men there was very little difference in neural response between trials with a previous reward and those without. No statistically relevant differences in other key regions of reward processing as VS or vmPFC were detected. We thus conclude increased MF control and increased variance in dlPFC response dependent on reward in women, showing a sex-difference in task approach and neural processing between men and women. Apart from increased MF behavior in women, we also observed an increased learning rate (α) for women in the first stage. The learning rate measures to what extend an individual updates beliefs or expectations based on new information. As the first stage is visited more frequently than either second stage (more information is obtained on the first stage throughout the task) it is sensible to differentiate two learning rates, one for each stage. The learning rate serves as a scaling factor for the reward prediction error, this is reflected in the computational model equation (see supplement), where the RPE in the model-free state action value is multiplied by the learning rate. A higher learning rate (maximum value of 1) amplifies the value of the reward prediction error accordingly, increasing the overall model free state action value. A higher learning rate also assigns a greater importance to more recent rewards 27 , consequently we may be able to say that because women were quick to adapt their task strategy according to recent events they may have anticipated a more volatile environment. The women of our sample took more care to update their expectations based on new information and took information from individual trials into greater consideration, whereas men were more likely to aggregate trials and assign outcomes to coincidence, finding trials to be less correlated to each other. Next to the elevated learning rate we also found increased values for the stage-skipping value update (λ) parameter in women compared to men. The stage-skipping value update parameter updates the first stage values according to reward prediction errors received in the second stage. Here, likewise a higher λ (maximum value of 1) amplifies the influence of the reward prediction error 13 (see equation supplement). Λ serves as a connecting factor over two steps, providing an estimate on how many time steps an agent considers while making a decisions. The closer to one, the more the previous second-stage RPE is influencing future first-step decisions 28 . Consequently, we might be able to say that a higher λ decreases the impact of the stage transitions. We found model-free behavior was correlated with λ. Studies clarifying how computational model parameters may relate to real-world decision-making are needed. However, women were more likely to update their first stage values according to RPE received after the second stage, assigning a higher reward connection between stages than men did. Despite differences in MF control and underlying computational parameters, the sexes did not differ in final payout (see Results section Fig. 2 c) – demonstrating that neither strategy bore a clear advantage in this task. From a cost-benefit perspective, we could even say that the model-free strategy achieved the same final gain at lower computational costs, terming model-free the more efficient strategy in this scenario 29 , 30 . Other studies have described the model-free and model-based strategies as quite similar in effectiveness and information processing 31 . Since there was no difference in monetary reward between sexes in our study, we cannot state either strategy to be superior. Studies on the two-step task have shown that model-based behavior is linked to IQ, working memory and verbal IQ 32 – 34 . Our sample showed no difference in fluid intelligence or school education between both sexes (see supplement), which suggests that the group differences in behavior are unlikely driven by these potential confounding factors. As done in Gillan et.al. future applications building off the two-step task should correct for sex as well as a palette (age, education, etc.) of other factors influencing decision-making 14 . The behavioral findings observed between sexes for model-free control correlated with a difference in dlPFC response. Women in comparison to men showed a higher response in dlPFC in unrewarded compared to rewarded trials (Fig. 4 a). Men’s dlPFC response did not differ between rewarded and unrewarded trials (Fig. 4 b, left). Studies on the Iowa Gambling Test in Humans have shown that the dlPFC response increases during decision-making and is important for processing negative emotions 35 . Greater fluctuations in dlPFC response have been correlated to higher top down control, underlining our finding of a higher learning rate in women. Further studies have demonstrated women to be more loss-aversive than men and men more sensitive to reward than women 16 . The neural findings of this study point in a similar direction, with women showing higher responses to unrewarded trials and men more neural response than women following rewarded trials. We could not detect a difference in VS or vmPFC between sexes, even though these areas are key regions in reward processing. The different behaviors applied in the two-step task might be reflecting real-world behavior carrying an advantage 30 . Model-free strategy for example has been correlated with social behavior 15 . A connection between reward and sociability can be traced in imaging data, where both behaviors activate the same brain regions 36 , 37 . Some studies have shown women to be more sensitive to social rewards than men 38 . Perhaps, there may be a link between social behavior and decision-making and reward processing. However, as our study incorporated no measure of sociability, we are not able to determine a clear directionality. A follow-up study may consider implementing a questionnaire to measure social behavior in participants or implement both monetary and social rewards in the task design. Limitations The study was performed with relatively few women (N = 36) in comparison to men (N = 61). Ideally, equal and larger numbers of men and women would be included in the study design. The analysis was conducted in an exploratory fashion on an existing data set originally designed to investigate whether tryptophan influenced decision-making in the two-step task 24 . A follow-up study to confirm this finding should have a specific hypothesis and would ideally consider the influence of hormones and tracking of the female cycle. The set-up may include assessments at two time points within the cycle – luteal and follicular – in women on natural cycles and compare decision-making to women taking oral contraceptives. Our study adopts a binary view on sex division, future studies may also move away from this dichotomy to gain a better understanding of sex differences. Conclusion We compared different behaviors in the two-step task between men and women, and found increased preference for the model-free strategy in the women of our sample. On the imaging side, we found increased neural response in dlPFC after reward omission compared to receiving a reward in women and similar responses for both outcomes in men. In conclusion, our study shows different behaviors between men/women correlated to different decision-making processes and these mechanisms are linked to different brain responses. We would like to emphasize the beauty of differences and how each individual and gender can contribute his/her/their very own talents without fear of being inferior or worse in spite of. Declarations Acknowledgments and Disclosures Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Deutsche Forschungsgemeinschaft (DFG, grants numbers 178833530 (SFB 940), 402170461 (TRR 265), 454245598 (IRTG 2773), and 521379614 (TRR 393)). The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. ORCID iDs: Marina Lukezic ORCID iD 0000-0002-9204-4814 Philipp T. Neukam ORCID iD 0000-0002-9442-0043 Michael N. Smolka ORCID iD 0000-0001-5398-5569 Yacila I. Deza-Lougovski ORCID iD 0000-0002-2624-077X Author contributions Concept, design and data acquisition (M.G., P.N., Y.I.D, M.N.S.), analysis and/or interpretation of data (M.L., H.C., M.N.S.), drafting the article (M.L.), reviewing critical version of the article (M.L., H.C., M.N.S.). All authors approved the final version of the manuscript. Data Availability Statement The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation ( [email protected] ). Additional Information None of the authors have any competing interest that must be disclosed. References Bolt, E. et al. Effects of a virtual gender swap on social and temporal decision-making. Sci. Rep. 11 , 15376. https://doi.org/10.1038/s41598-021-94869-z (2021). Stoet, G. & Geary, D. C. The Gender-Equality Paradox in Science, Technology, Engineering, and Mathematics Education. Psychol. Sci. 29 , 581–593. https://doi.org/10.1177/0956797617741719 (2018). Croson, R. & Gneezy, U. Gender Differences in Preferences. J. Econ. Lit. 47 , 448–474. https://doi.org/10.1257/jel.47.2.448 (2009). Holstein, C. B. & Irreversible Stepwise Sequence in the Development of Moral Judgment: A Longitudinal Study of Males and Females. Child Dev. 47 , 51–61. https://doi.org/10.2307/1128282 (1976). Tinghög, G. et al. Intuition and Moral Decision-Making - The Effect of Time Pressure and Cognitive Load on Moral Judgment and Altruistic Behavior. PLoS One . 11 , e0164012. https://doi.org/10.1371/journal.pone.0164012 (2016). Daw, N. D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8 , 1704–1711. https://doi.org/10.1038/nn1560 (2005). Dolan, R. J. & Dayan, P. Goals and habits in the brain. Neuron 80 , 312–325. https://doi.org/10.1016/j.neuron.2013.09.007 (2013). Thorndike, E. L. . J. Philos. Inc . 9 , 193–194. https://doi.org/10.2307/2013764 (1912). Balleine, B. W. D. N. a. O. D. J. . Neuroeconomics 367–387. https://doi.org/10.1016/B978-0-12-374176-9.00024-5 (2009). Balleine, B. W. & O'Doherty, J. P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology: official publication Am. Coll. Neuropsychopharmacol. 35 , 48–69. https://doi.org/10.1038/npp.2009.131 (2010). Tolman, E. C. Cognitive maps in rats and men. Psychol. Rev. 55 , 189–208. https://doi.org/10.1037/h0061626 (1948). Worbe, Y. et al. Valence-dependent influence of serotonin depletion on model-based choice strategy. Mol. Psychiatry . 21 , 624–629. https://doi.org/10.1038/mp.2015.46 (2016). Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P. & Dolan, R. J. Model-based influences on humans' choices and striatal prediction errors. Neuron 69 , 1204–1215. https://doi.org/10.1016/j.neuron.2011.02.027 (2011). Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A. & Daw, N. D. Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. eLife 5 (2016). https://doi.org/10.7554/eLife.11305 Oguchi, M. et al. Proselfs depend more on model-based than model-free learning in a non-social probabilistic state-transition task. Sci. Rep. 13 , 1419. https://doi.org/10.1038/s41598-023-27609-0 (2023). Dhingra, I. et al. Sex differences in neural responses to reward and the influences of individual reward and punishment sensitivity. BMC Neurosci. 22 , 12. https://doi.org/10.1186/s12868-021-00618-3 (2021). Orsini, C. A., Willis, M. L., Gilbert, R. J., Bizon, J. L. & Setlow, B. Sex differences in a rat model of risky decision making. Behav. Neurosci. 130 , 50–61. https://doi.org/10.1037/bne0000111 (2016). Koot, S., van den Bos, R., Adriani, W. & Laviola, G. Gender differences in delay-discounting under mild food restriction. Behav. Brain. Res. 200 , 134–143. https://doi.org/10.1016/j.bbr.2009.01.006 (2009). van den Bos, R., Jolles, J., van der Knaap, L., Baars, A. & de Visser, L. Male and female Wistar rats differ in decision-making performance in a rodent version of the Iowa Gambling Task. Behav. Brain. Res. 234 , 375–379. https://doi.org/10.1016/j.bbr.2012.07.015 (2012). Overman, W. H. et al. Performance on the IOWA card task by adolescents and adults. Neuropsychologia 42 , 1838–1851. https://doi.org/10.1016/j.neuropsychologia.2004.03.014 (2004). Islas-Preciado, D. et al. Risk-based decision making in rats: Modulation by sex and amphetamine. Horm. Behav. 125 , 104815. https://doi.org/10.1016/j.yhbeh.2020.104815 (2020). Wang, L. et al. Sex differences in neural substrates of risk taking: Implications for sex-specific vulnerabilities to internet gaming disorder. J. Behav. addictions . 11 , 778–795. https://doi.org/10.1556/2006.2022.00057 (2022). Gaillard, A., Fehring, D. J. & Rossell, S. L. A systematic review and meta-analysis of behavioural sex differences in executive control. Eur. J. Neurosci. 53 , 519–542. https://doi.org/10.1111/ejn.14946 (2021). Gilger, M. D. et al. Arbitration between model-free and model-based control is not affected by transient changes in tonic serotonin levels. J. Psychopharmacol. (Oxford, England) . 38 , 178–187. https://doi.org/10.1177/02698811231216325 (2024). Daw, N. D. in Neuroeconomics 299–320 (2014). Kroemer, N. B. et al. Balancing reward and work: anticipatory brain activation in NAcc and VTA predict effort differentially. NeuroImage 102 Pt . 2 , 510–519. https://doi.org/10.1016/j.neuroimage.2014.07.060 (2014). Nussenbaum, K. & Hartley, C. A. Reinforcement learning across development: What insights can we draw from a decade of research? Dev. Cogn. Neurosci. 40 , 100733. https://doi.org/10.1016/j.dcn.2019.100733 (2019). Satinder, P. & Singh, R. S. S. Reinforcement Learning with Replacing Eligibility Traces. Springer Link. 22 , 123–158 (1996). Feher da Silva, C. & Hare, T. A. A note on the analysis of two-stage task results: How changes in task structure affect what model-free and model-based strategies predict about the effects of reward and transition on the stay probability. PLoS One . 13 , e0195328. https://doi.org/10.1371/journal.pone.0195328 (2018). Kool, W., Gershman, S. J. & Cushman, F. A. Cost-Benefit Arbitration Between Multiple Reinforcement-Learning Systems. Psychol. Sci. 28 , 1321–1333. https://doi.org/10.1177/0956797617708288 (2017). Smid, C. R., Kool, W., Hauser, T. U. & Steinbeis, N. Computational and behavioral markers of model-based decision making in childhood. Dev. Sci. 26 , e13295. https://doi.org/10.1111/desc.13295 (2023). Culbreth, A. J., Westbrook, A., Daw, N. D., Botvinick, M. & Barch, D. M. Reduced model-based decision-making in schizophrenia. J. Abnorm. Psychol. 125 , 777–787. https://doi.org/10.1037/abn0000164 (2016). Sebold, M. et al. Model-based and model-free decisions in alcohol dependence. Neuropsychobiology 70 , 122–131. https://doi.org/10.1159/000362840 (2014). Eppinger, B., Walter, M., Heekeren, H. R. & Li, S. C. Of goals and habits: age-related and individual differences in goal-directed decision-making. Front. Neurosci. 7 , 253. https://doi.org/10.3389/fnins.2013.00253 (2013). van den Bos, R., Homberg, J. & de Visser, L. A critical review of sex differences in decision-making tasks: focus on the Iowa Gambling Task. Behav. Brain. Res. 238 , 95–108. https://doi.org/10.1016/j.bbr.2012.10.002 (2013). Lin, A., Adolphs, R. & Rangel, A. Social and monetary reward learning engage overlapping neural substrates. Soc. Cognit. Affect. Neurosci. 7 , 274–281. https://doi.org/10.1093/scan/nsr006 (2012). Izuma, K., Saito, D. N. & Sadato, N. Processing of social and monetary rewards in the human striatum. Neuron 58 , 284–294. https://doi.org/10.1016/j.neuron.2008.03.020 (2008). Spreckelmeyer, K. N. et al. Anticipation of monetary and social reward differently activates mesolimbic brain structures in men and women. Soc. Cognit. Affect. Neurosci. 4 , 158–165. https://doi.org/10.1093/scan/nsn051 (2009). Additional Declarations No competing interests reported. Supplementary Files LukezicetalSupplement20250703.docx Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 04 Dec, 2025 Reviews received at journal 12 Nov, 2025 Reviewers agreed at journal 10 Nov, 2025 Reviews received at journal 09 Nov, 2025 Reviewers agreed at journal 25 Oct, 2025 Reviewers agreed at journal 24 Oct, 2025 Reviewers agreed at journal 10 Oct, 2025 Reviewers agreed at journal 20 Sep, 2025 Reviewers invited by journal 28 Jul, 2025 Editor assigned by journal 28 Jul, 2025 Editor invited by journal 09 Jul, 2025 Submission checks completed at journal 06 Jul, 2025 First submitted to journal 06 Jul, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7021287","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":492767198,"identity":"29e727c7-8d75-40bf-a60e-ce4de5b0d900","order_by":0,"name":"Marina Lukezic","email":"","orcid":"","institution":"Technische Universität Dresden","correspondingAuthor":false,"prefix":"","firstName":"Marina","middleName":"","lastName":"Lukezic","suffix":""},{"id":492767199,"identity":"d6fa411e-7ae6-4ddf-85b7-6ad1c3225cc2","order_by":1,"name":"Hao Chen","email":"","orcid":"","institution":"Technische Universität Dresden","correspondingAuthor":false,"prefix":"","firstName":"Hao","middleName":"","lastName":"Chen","suffix":""},{"id":492767200,"identity":"d7049cb5-b781-4b09-8f30-986cdc64f6b7","order_by":2,"name":"Maximilian Gilger","email":"","orcid":"","institution":"Technische Universität Dresden","correspondingAuthor":false,"prefix":"","firstName":"Maximilian","middleName":"","lastName":"Gilger","suffix":""},{"id":492767201,"identity":"5c9a6736-701d-46b1-ac70-3dcf95e95059","order_by":3,"name":"Philipp T. Neukam","email":"","orcid":"","institution":"Technische Universität Dresden","correspondingAuthor":false,"prefix":"","firstName":"Philipp","middleName":"T.","lastName":"Neukam","suffix":""},{"id":492767202,"identity":"e59be574-b7f5-4e88-a2cf-b883e2403d76","order_by":4,"name":"Yacila l. Deza-Lougovski","email":"","orcid":"","institution":"Technische Universität Dresden","correspondingAuthor":false,"prefix":"","firstName":"Yacila","middleName":"l.","lastName":"Deza-Lougovski","suffix":""},{"id":492767203,"identity":"0596f638-e120-4378-af89-993bb3d69c0b","order_by":5,"name":"Michael N. Smolka","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABIElEQVRIiWNgGAWjYDACZgY2IGkBYjIebGCwATEMwOJQBi4tEmA2UEuaBGEtDKhaDhPWotvO/OzBjwoJBvlphw8cnFFzvs7gAPO2Dx/+WMuZMzBvfIBFi9lhNnPDnjMSDAa30xIObjh2W8LgAFvxzJlt6caWDWzF2KwxO8zDJsHbBtQinWNw8AEbUMv9N8bMvA2HEzcc4DGTwKFF8u8/oMNmg7T8Owe0hceY+c+fw/VALeY/cGiR5m0AmnYbqGVj2wGIFga2wwlAhhk27wP9YiYtc0yCB+yXmX3JkjOBfmHsbUs33HCYrRirw84ffib5psZGTn528sGHPd/s+PkOMG9m+PHHWt7gePPGD9isgQIeLGLMeNSPglEwCkbBKMALAFywY93ssWQTAAAAAElFTkSuQmCC","orcid":"","institution":"Technische Universität Dresden","correspondingAuthor":true,"prefix":"","firstName":"Michael","middleName":"N.","lastName":"Smolka","suffix":""}],"badges":[],"createdAt":"2025-07-01 14:23:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7021287/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7021287/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":88002938,"identity":"78e9e6a7-1695-4c50-abb4-f63278701870","added_by":"auto","created_at":"2025-07-31 10:29:36","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":39337,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eThe two-step paradigm.\u003c/strong\u003e The task consisted of 201 trials in total. Initially, participants needed to choose between one of the two grey boxes containing black ovals. Each grey box lead to either the paired blue or purple boxes with a fixed transition probability of common (70%) or rare (30%). The task instruction provided this information. In the second step, participants selected one of the blue/purple boxes and had a chance to win 20¢ (€0.20) with a reward probability ranging between 25-75%. This final reward probability fluctuated across the task according to Gaussian random walks. Participants were instructed to collect as much money as possible, as this was added to the final monetary reward.\u003c/p\u003e","description":"","filename":"Picture1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7021287/v1/d1cf724d5ab998de3371ba26.jpg"},{"id":88004507,"identity":"4b81c657-8250-496a-aeb8-dfbc88f1a01d","added_by":"auto","created_at":"2025-07-31 10:37:36","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":291846,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eComparison of first stage stay probabilities between men and women. \u003c/strong\u003eA) Stay probabilities of rewarded/unrewarded trials for common/rare transitions for men (left) and women (right) with error bars. Men display a mixture of model-free and model-based behavior with no clear tendency for a preferred behavior, in comparison women show increased stay probability after rewarded trials – both common and rare – favoring a model free strategy. \u0026nbsp;B) Boxplot comparing average model-free (left) and model based (right) effect between women (left) and men (right). The model free boxplots show increased model free effect in females (median (IQR) of 0.263(0.261)) compared to men (0.098(0.153)), whereas no clear trend of model based strategy is apparent between sexes (women 0.097(0.207), men 0.101(0.185)). C) Barplot with error bars showing no difference in final payout between women (mean 6.48 and SE 0.081) and men (mean 6.64 and SE 0.121).\u003c/p\u003e","description":"","filename":"Picture2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7021287/v1/540e95b45629ae57a087f5e3.jpg"},{"id":88002941,"identity":"861b0647-36c2-4f3f-833a-cf87b1263af6","added_by":"auto","created_at":"2025-07-31 10:29:36","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":36289,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of computational model parameters between sexes. Boxplots comparing average ß\u003csub\u003e1\u003c/sub\u003e (a), ß\u003csub\u003e2\u003c/sub\u003e (b), α\u003csub\u003e1\u003c/sub\u003e \u0026nbsp;(c), α\u003csub\u003e2\u003c/sub\u003e (d), λ (e), π (f) and ω (g) between women (pink) and men (blue). Showing an increased α\u003csub\u003e1\u003c/sub\u003e \u0026nbsp;\u0026nbsp;and λ in women, and insignificant increase in α\u003csub\u003e2\u003c/sub\u003e in women. However, no difference in the other parameters (Ω, ß\u003csub\u003e1\u003c/sub\u003e, ß\u003csub\u003e2\u003c/sub\u003e and π)\u003c/p\u003e","description":"","filename":"Picture3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7021287/v1/da8f0f6e51bf4d14510137e1.jpg"},{"id":88002945,"identity":"14ab8da9-f3ed-47ef-9182-f2899c93b86d","added_by":"auto","created_at":"2025-07-31 10:29:36","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":330665,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eNeural response in dlPFC men/women comparison. \u003c/strong\u003ea) Bar plot with error bars showing average dlPFC response in unrewarded (red) and rewarded (green) trials for men (left) and women (right). b) Increased neural response in men compared to women for MF control (1\u003csup\u003est\u003c/sup\u003e regressor 2nd parametric); displayed at threshold of p\u0026lt;.001.\u003c/p\u003e","description":"","filename":"Picture4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7021287/v1/3ee1eb69e7a4c7898c7116c3.jpg"},{"id":88006588,"identity":"b42ea3c7-557f-42db-afef-fb64171ba143","added_by":"auto","created_at":"2025-07-31 11:01:50","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1373034,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7021287/v1/2e1e5510-5df2-4817-9cb5-1e662510c8fa.pdf"},{"id":88004509,"identity":"aeb73161-a456-428f-b7b5-ad2057ebc61c","added_by":"auto","created_at":"2025-07-31 10:37:36","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":1538980,"visible":true,"origin":"","legend":"","description":"","filename":"LukezicetalSupplement20250703.docx","url":"https://assets-eu.researchsquare.com/files/rs-7021287/v1/62ee37c3f3c18ad0a055d99c.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Sex Differences in Model free and Model based Behavior","fulltext":[{"header":"Sex Differences between Model free and Model based Behavior","content":"\u003cp\u003eDespite us nearing gender equality, women maintain high prosocial tendencies and men continue to be stronger represented in science and engineering fields \u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. In more gender-equal countries, this difference is even more apparent, with men expressing a higher interest and more joy for Science. The opposite was found for women, where greater gender-equality was associated with less interest in Science for women \u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eDifferences in career-choice and other everyday choices between sexes, may be less a consequence of opportunity, but a result of internal decision-making. Studies have shown differences in risk-taking, social behavior and competition between men and women. Women are more easily influenced by their environment when making decisions, and generally favor less-competitive circumstances. In comparison men perform better than women when competing \u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eDifferences in decision-making between men and women are observed not only in real life, but also in experimental tasks. One example of such a task is the dictator game, in which participants can choose to keep money for themselves or give it to others \u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e. This task showed women to make more generous contributions compared to men\u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e,\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. This is one example of how decision-making in experimental tasks can mirror real-world behavior.\u003c/p\u003e\u003cp\u003eOur everyday decisions are governed by model based and model free control. One task designed to tease apart these different decision-making patterns is Daw\u0026rsquo;s two step task. A balance of these two approaches regulates the way we make day-to-day choices \u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e and may help us investigate sex differences in decision-making in more depth.\u003c/p\u003e\u003cp\u003eOn the one hand, there is model free behavior, which is characterized by low computational costs. This behavior relies solely on the history of reward and functions so that a previously rewarded action has a high likelihood to be repeated in future. A model free learner disregards the task structure or external environment and navigates based on previously received reward \u003csup\u003e\u003cspan additionalcitationids=\"CR7 CR8 CR9\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eOn the other hand, model based behavior is computationally more costly but a more flexible means of engaging with the environment \u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e,\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e,\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e. This behavior is influenced not only by reward, but also by the task structure. A model based learner is constantly evaluating reward in relation to environment and updating the value of the reward as external circumstances shift. While a model based strategy has the advantage of adapting to changes, it is likewise very costly regarding time and brain resources and is too expensive to be used for all daily actions \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eWe were curious to find out whether sex differences in decision-making could be differentiated through aid of the two-step task. To the best of our knowledge only one study by Gillan et. al. identified differences in two-step behavior between men and women. Gillan et. al. found women displayed increased model-free behavior in the larger of two experiments. Other existing studies on the two-step task primarily match sex in the patient and control groups or use it as a control variable, this explains why there is not much to refer to in the literature \u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eA study by Oguchi et.al. found a correlation between increased prosocial behavior and the model free strategy in the two-step task, in comparison elevated proself behavior correlated with the model based strategy. This study assessed social orientation scores via an online slider measure, where participants could choose to allocate money to others or themselves. The scores on social orientation were then compared to decision-making strategies in a two-step Markov decision task. The findings of the study propose that the two-step task may predict resource allocation between others and one-self \u003csup\u003e\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e. Some existing studies suggest that decision-making in women is more socially oriented than in men \u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. This supports the assumption that decision-making mechanisms extracted from the two-step task can reflect social decision-making and also aligns with increased female representation in prosocial fields and could help us understand sex-mechanisms driving career and other life choices.\u003c/p\u003e\u003cp\u003eDiving into the animal world, evidence from mouse studies suggests that male and female mice use different strategies to make adaptive decisions \u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e. However, the association between sex and decision-making strategies is complex, with some studies showing increased exploratory behavior in female mice and others in male mice \u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e,\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e. Although the direction of behavior is not always uniform, males and females process environments differently and perceive the environmental structure in different ways, adapting their behavior accordingly.\u003c/p\u003e\u003cp\u003eEven though behavioral data regarding decision-making in animal studies is inconclusive, there are clear trends in reward processing between male and female rodents \u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e,\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e. Females show elevated neural response after negative feedback compared to males \u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e,\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e,\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e. Females also displayed higher reactivity to immediate reward independent of reward size \u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e, whereas males were more likely to seek out bigger \u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e more risky rewards \u003csup\u003e\u003cspan additionalcitationids=\"CR22\" citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e. Some studies in humans have shown higher response to reward in men and punishment in women \u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. Differences in reward processing may be guiding decision-making in men and women.\u003c/p\u003e\u003cp\u003eThe lives of men and women continue to drift in different directions, with women being more represented in social sectors, as medicine and nursing. Prosocial qualities and female sex have been linked to model free behavior in the two-step task. We conducted an exploratory analysis on an existing data set to investigate sex-differences in the two-step task. Specifically, we compared the parameters for model free and model based control, and the computational parameters: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\u0026szlig;}_{1}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\u0026szlig;}_{2}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:}_{1}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:}_{2}\\)\u003c/span\u003e\u003c/span\u003e, λ, π and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\omega\\:\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e\u003cp\u003eAdditionally, we looked into neural correlates between sexes, while performing the two-step task, which few studies have done so far. Specifically, we looked into the ventral striatum (VS), ventral medial prefrontal cortex (vmPFC), and dorsolateral prefrontal cortex (dlPFC), these are the key areas linked to reward processing and main areas of focus in the original study \u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e"},{"header":"Materials and Methods","content":"\u003cp\u003e\u003cb\u003eParticipants\u003c/b\u003e\u003c/p\u003e\u003cp\u003eData for this study originated from project B04 of the Collaborative Research Center 940 on cognition and volitional control and looked into serotonergic modulation of decision-making in the two-step task \u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e. Ethical approval for the study was obtained EK 42022012 (TU Dresden) and all experiments were performed in accordance with relevant guidelines/regulations in line with the Declaration of Helsinki. Participants were recruited from a randomized sample stratified by age and gender from the registration data and invited to participate in the study. Persons without MRI safety concerns, sufficient visual capability (corrected binocular visus lower than 0.8) and without a history or current diagnosis of mental disorder were eligible for participation (for more detail see original publication by Gilger et.al. \u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e). The final sample size for behavioral analysis included 98 participants (37 female), ages 20\u0026ndash;42 (Mean: 32.2 years, SD\u0026thinsp;=\u0026thinsp;6.1 years). Ten participants needed to be excluded from imaging analysis: five participants showed strong unrepairable artefacts (strong ghosting or spiking), four participants had more than 30% repaired slices (ArtRepair Toolbox) in one session and one participant had high scan-to-scan motion, this left a final sample size of 88 participants (36 female). All participants gave written informed consent for study participation prior to the study and received monetary reimbursement (fixed amount plus bonus adjusted to task performance) at the end of the study. (For detailed information on recruitment and inclusion/exclusion criteria please see supplement)\u003c/p\u003e\u003cp\u003e\u003cb\u003eProcedure and intervention\u003c/b\u003e\u003c/p\u003e\u003cp\u003e\u003cspan type=\"BoldItalicUnderline\" class=\"BoldItalicUnderline\" name=\"Emphasis\"\u003eTwo-step paradigm\u003c/span\u003e\u003c/p\u003e\u003cp\u003eParticipants completed a sequential decision-making task according to Daw \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e in a MRI scanner. (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e)\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003cspan type=\"BoldItalicUnderline\" class=\"BoldItalicUnderline\" name=\"Emphasis\"\u003eResponse patterns for stay-switch probabilities\u003c/span\u003e\u003c/p\u003e\u003cp\u003eAccording to Daw et. Al. \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e two strategies for task performance are observed: model free and model based learning-strategies, respectively. While pure model-free behavior is characterized by sole influence of reward on previous first stage choice, model-based behavior takes into account the interaction of reward and transition for the subsequent first stage choice. Previous research suggests that participants display a mix of model free and model based behavior \u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003e\u003cb\u003eAnalysis of behavioral data\u003c/b\u003e\u003c/p\u003e\u003cp\u003eModel-free (MF) and model-based (MB) scores for each participant were calculated according to their first stage choice probability across all trials. We then used the scores to compare stay-switch probabilities between both sexes. The scores were calculated according to Daw et.al. as follows:\u003c/p\u003e\u003cp\u003eMF-Score\u0026thinsp;=\u0026thinsp;P (stay|rewarded common)\u0026thinsp;+\u0026thinsp;P (stay|rewarded rare) \u0026ndash; P (stay|unrewarded common) \u0026ndash; P (stay|unrewarded rare) / 2\u003c/p\u003e\u003cp\u003eMB-Score\u0026thinsp;=\u0026thinsp;P (stay|rewarded common) \u0026ndash; P (stay|rewarded rare) \u0026ndash; P (stay|unrewarded common)\u0026thinsp;+\u0026thinsp;P (stay|unrewarded rare) / 2\u003c/p\u003e\u003cp\u003eBehavioral statistical analysis was carried out with SPSS 28 (IBM-SPSS, Chicago, IL, USA) with significance set at p\u0026thinsp;\u0026lt;\u0026thinsp;.05. First, we calculated mean values of model free and model based behavior for each individual across all sessions and then compared the average values between men and women. The original analysis concluded with a null finding \u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e, disputing any effects of the pharmacological intervention with tryptophan on first-stage switching behavior. We proceeded to compare basic sample characteristics (age, school achievement and fluid intelligence score) in order to exclude the influence of these variables on two step behavior. 2-step behavior between sexes was analysed via simple statistical test (t-test, X2 test). (For a detailed analysis on basic sample characteristics please see supplement)\u003c/p\u003e\u003cp\u003e\u003cb\u003eComputational Modelling\u003c/b\u003e\u003c/p\u003e\u003cp\u003eWe further applied the seven-parameter (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\u0026szlig;}_{1}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\u0026szlig;}_{2},\\:\\)\u003c/span\u003e\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:}_{1}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:}_{2}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:,\\:\\omega\\:\\)\u003c/span\u003e\u003c/span\u003e, λ \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:,\\:{\\pi\\:}\\)\u003c/span\u003e\u003c/span\u003e) reinforcement-learning model by Daw et.al. \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e to our data (See supplement for computational model equations). On the conceptual level, the inverse temperature parameter \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\u0026szlig;}_{i}\\)\u003c/span\u003e\u003c/span\u003e provides insight into choice reliability between stages and tells us how deterministic the choices are, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\u0026szlig;}_{1}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\u0026szlig;}_{2}\\)\u003c/span\u003e\u003c/span\u003e refer to the first and second stage, respectively. The learning rate α determines to what extend an individual updates beliefs or expectation based on new information. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:}_{1}\\)\u003c/span\u003e\u003c/span\u003e refers to the first stage learning rate and, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:}_{2}\\)\u003c/span\u003e\u003c/span\u003e the second stage, respectively (0\u0026thinsp;\u0026lt;\u0026thinsp;α\u0026thinsp;\u0026lt;\u0026thinsp;1) \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. The parameter \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\omega\\:}\\)\u003c/span\u003e\u003c/span\u003e weights the influence of model based and model free values. In the specific case of a pure model-based learner \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\omega\\:}\\:\\)\u003c/span\u003e\u003c/span\u003e= 1, for a pure model-free learner \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\omega\\:}=0\\)\u003c/span\u003e\u003c/span\u003e would be the case. The model also includes a stage-skipping value update parameter λ, which provides an update of first-stage values based on reward prediction errors (RPE) received after the second stage (0\u0026thinsp;\u0026lt;\u0026thinsp;λ\u0026thinsp;\u0026lt;\u0026thinsp;1). As the task has only two steps, λ reflects the influence of the RPE received after the second stage on the first stage value \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. A λ of zero means only the current state is eligible for updating, λ equals one means all states that have ever been visited are eligible for updating \u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e. The perseveration parameter π describes the propensity of repeating the previous first stage option (p\u0026thinsp;\u0026gt;\u0026thinsp;0) or switching (p\u0026thinsp;\u0026lt;\u0026thinsp;0) \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. The effect of sex on all seven parameters was tested via simple statistical tests (T-test, Mann-Whitney Test).\u003c/p\u003e\u003cp\u003e\u003cb\u003eAnalysis of fMRI data\u003c/b\u003e\u003c/p\u003e\u003cp\u003e We were able to include 88 participants (36 females) in the fMRI analysis after quality control. We adopted the preprocessing steps and the first-level model from Gilger et.al \u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e (please see supplement material for more information). All functional brain analyses was performed with SPM8. For the 1st level statistics, one onset regressor and three parametric modulators for previous transition, previous reward and interaction of reward and transition were defined. The first stage onset regressor on previous reward provides an estimate of model free behavior, while the first stage onset regressor evaluating the interaction of transition and reward reflects model-based behavior. An additional regressor described the second stage transition and a third regressor was selected for reward at outcome onset. Our three main regions of interest (ROI) were the VS, vmPFC, and dlPFC and we selected them with help of meta-analysis software (For a detailed description on how ROIs were selected, please see Kroemer et. al. \u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e).\u003c/p\u003e\u003cp\u003eFor second level analysis we extracted the neural responses from the regions of interest and performed a t-test with sex added as a covariate. Lastly, we concluded the imaging with a whole-brain analysis: applying sex as a covariate we compared model-free, model-based behavior, as well as differences in transition and reward for second stage onset. A voxel-based threshold \u003cem\u003ep\u003c/em\u003e\u003csub\u003euncorr\u003c/sub\u003e.\u0026lt;.001 was selected.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e\u003cb\u003eSex differences in model-based and model-free parameters\u003c/b\u003e\u003c/p\u003e\u003cp\u003eWe compared the probability of repeating the same first stage choice as in the previous trial in men and women (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003ea). If the selection of the same first stage choice was driven by reward obtained after a choice at the second stage, the behavior is termed model free. A strategy considering both reward and transition rates, is defined as model based. The Mann-Whitney U test revealed increased model-free behavior i.e. an increased effect of previous reward in women in contrast to men of our sample (U\u0026thinsp;=\u0026thinsp;476, Z=-4.782, p\u0026thinsp;=\u0026thinsp;\u0026lt;\u0026thinsp;.001, r=-0.48) (See Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eb). No significant difference between men and women was observed for the model-based strategy (U\u0026thinsp;=\u0026thinsp;1036, Z=-0.678, p\u0026thinsp;=\u0026thinsp;.498, r=-0.068) (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eb right). Despite the differences in strategies adopted, there was no difference in final average reward between men and women (t\u0026thinsp;=\u0026thinsp;0.932, p\u0026thinsp;=\u0026thinsp;.353, d\u0026thinsp;=\u0026thinsp;0.805; \u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.364) (See Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003ec).\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eSex comparison of computational model parameters\u003c/b\u003e\u003c/p\u003e\u003cp\u003eTo get a more precise idea on differences in decision-making between males and females, we investigated the seven parameters according to the computational model established by Daw, et al. \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. As also reported in Gilger et.al. we had an average \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\omega\\:\\)\u003c/span\u003e\u003c/span\u003e of 0.41 (SD\u0026thinsp;=\u0026thinsp;0.29) \u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e. Similar results can be found in existing literature, revealing that overall a mixture of both model-free and model-based strategies is applied in the two-step task, our sample falling in line with the overall average. On closer inspection, we found no difference in average \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\omega\\:\\)\u003c/span\u003e\u003c/span\u003e value between males and females (U\u0026thinsp;=\u0026thinsp;1059, Z=-.509, p\u0026thinsp;=\u0026thinsp;.611, r=-.051) (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eg). However, we did find higher learning rates for the first stage learning rate \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:}_{1}\\)\u003c/span\u003e\u003c/span\u003e, (t=-2.7, p\u0026thinsp;=\u0026thinsp;.009, d\u0026thinsp;=\u0026thinsp;0,2) (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ec) and stage-skipping value update parameter λ (U\u0026thinsp;=\u0026thinsp;753, Z=-2.75, p\u0026thinsp;=\u0026thinsp;.006, r=-.28) in women (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ee). The difference in the second stage learning rate \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:}_{2}\\)\u003c/span\u003e\u003c/span\u003e, was insignificant between both sexes (t=-1.9, p\u0026thinsp;=\u0026thinsp;.058, d\u0026thinsp;=\u0026thinsp;0.21) (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ed). No statistically significant difference was detected for either inverse temperature (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\text{\u0026szlig;}}_{1}\\)\u003c/span\u003e\u003c/span\u003e; U\u0026thinsp;=\u0026thinsp;936, Z=-1.41, p\u0026thinsp;=\u0026thinsp;.158, r=-.142; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\text{\u0026szlig;}}_{2}\\)\u003c/span\u003e\u003c/span\u003e; U\u0026thinsp;=\u0026thinsp;962, Z=-1.22, p\u0026thinsp;=\u0026thinsp;.222, r=-.123 (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ea, b) or perseveration rate π (U\u0026thinsp;=\u0026thinsp;1103, Z=-0.187, p\u0026thinsp;=\u0026thinsp;0.852, r=-0.019) (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ef).\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eSex differences on the brain level\u003c/b\u003e\u003c/p\u003e\u003cp\u003eTo check whether there was a difference in neural response between sexes during performance on the two-step task, we compared the first stage onset regressor with previous reward (model free) and the first stage onset regressor with interaction of transition and reward (model based control). We investigated the three predetermined ROIs (vmPFC, dlPFC und VS) and additionally performed an exploratory whole brain analysis. The ROI analysis showed an increased response in the dlPFC for model free control in rewarded compared to unrewarded trials in women in comparison to men (t\u0026thinsp;=\u0026thinsp;2.727, p\u0026thinsp;=\u0026thinsp;.008, d\u0026thinsp;=\u0026thinsp;0.468) (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003ea). The whole-brain analysis also showed increased neural response in dlPFC in men compared to women for the model free condition when rewarded trials were compared to unrewarded trials (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eb). No other significant differences were detected for any of the other ROIs in any condition.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eWe conducted an exploratory analysis on an existing data set to investigate sex-differences in model based and model free control. Females in our sample navigated through the task structure highly influenced by previously obtained reward, performing in a model free manner. Females also attributed higher values to the reward prediction error, as can be seen by higher learning rates (α) and stage-skipping update parameter (λ). Reward and the future estimated value of a reward were guiding decision-making in women throughout the two-step task. In the imaging analyses we saw a higher neural response in the dlPFC in women following unrewarded trials compared to rewarded. For men there was very little difference in neural response between trials with a previous reward and those without. No statistically relevant differences in other key regions of reward processing as VS or vmPFC were detected. We thus conclude increased MF control and increased variance in dlPFC response dependent on reward in women, showing a sex-difference in task approach and neural processing between men and women.\u003c/p\u003e\u003cp\u003eApart from increased MF behavior in women, we also observed an increased learning rate (α) for women in the first stage. The learning rate measures to what extend an individual updates beliefs or expectations based on new information. As the first stage is visited more frequently than either second stage (more information is obtained on the first stage throughout the task) it is sensible to differentiate two learning rates, one for each stage. The learning rate serves as a scaling factor for the reward prediction error, this is reflected in the computational model equation (see supplement), where the RPE in the model-free state action value is multiplied by the learning rate. A higher learning rate (maximum value of 1) amplifies the value of the reward prediction error accordingly, increasing the overall model free state action value. A higher learning rate also assigns a greater importance to more recent rewards \u003csup\u003e\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e, consequently we may be able to say that because women were quick to adapt their task strategy according to recent events they may have anticipated a more volatile environment. The women of our sample took more care to update their expectations based on new information and took information from individual trials into greater consideration, whereas men were more likely to aggregate trials and assign outcomes to coincidence, finding trials to be less correlated to each other.\u003c/p\u003e\u003cp\u003eNext to the elevated learning rate we also found increased values for the stage-skipping value update (λ) parameter in women compared to men. The stage-skipping value update parameter updates the first stage values according to reward prediction errors received in the second stage. Here, likewise a higher λ (maximum value of 1) amplifies the influence of the reward prediction error \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e (see equation supplement). Λ serves as a connecting factor over two steps, providing an estimate on how many time steps an agent considers while making a decisions. The closer to one, the more the previous second-stage RPE is influencing future first-step decisions \u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e. Consequently, we might be able to say that a higher λ decreases the impact of the stage transitions. We found model-free behavior was correlated with λ. Studies clarifying how computational model parameters may relate to real-world decision-making are needed. However, women were more likely to update their first stage values according to RPE received after the second stage, assigning a higher reward connection between stages than men did.\u003c/p\u003e\u003cp\u003eDespite differences in MF control and underlying computational parameters, the sexes did not differ in final payout (see Results section Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003ec) \u0026ndash; demonstrating that neither strategy bore a clear advantage in this task. From a cost-benefit perspective, we could even say that the model-free strategy achieved the same final gain at lower computational costs, terming model-free the more efficient strategy in this scenario \u003csup\u003e\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e,\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e. Other studies have described the model-free and model-based strategies as quite similar in effectiveness and information processing \u003csup\u003e\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e. Since there was no difference in monetary reward between sexes in our study, we cannot state either strategy to be superior.\u003c/p\u003e\u003cp\u003eStudies on the two-step task have shown that model-based behavior is linked to IQ, working memory and verbal IQ \u003csup\u003e\u003cspan additionalcitationids=\"CR33\" citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e\u003c/sup\u003e. Our sample showed no difference in fluid intelligence or school education between both sexes (see supplement), which suggests that the group differences in behavior are unlikely driven by these potential confounding factors. As done in Gillan et.al. future applications building off the two-step task should correct for sex as well as a palette (age, education, etc.) of other factors influencing decision-making \u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eThe behavioral findings observed between sexes for model-free control correlated with a difference in dlPFC response. Women in comparison to men showed a higher response in dlPFC in unrewarded compared to rewarded trials (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003ea). Men\u0026rsquo;s dlPFC response did not differ between rewarded and unrewarded trials (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eb, left). Studies on the Iowa Gambling Test in Humans have shown that the dlPFC response increases during decision-making and is important for processing negative emotions \u003csup\u003e\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e\u003c/sup\u003e. Greater fluctuations in dlPFC response have been correlated to higher top down control, underlining our finding of a higher learning rate in women. Further studies have demonstrated women to be more loss-aversive than men and men more sensitive to reward than women \u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. The neural findings of this study point in a similar direction, with women showing higher responses to unrewarded trials and men more neural response than women following rewarded trials. We could not detect a difference in VS or vmPFC between sexes, even though these areas are key regions in reward processing.\u003c/p\u003e\u003cp\u003eThe different behaviors applied in the two-step task might be reflecting real-world behavior carrying an advantage \u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e. Model-free strategy for example has been correlated with social behavior \u003csup\u003e\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e. A connection between reward and sociability can be traced in imaging data, where both behaviors activate the same brain regions \u003csup\u003e\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e,\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e\u003c/sup\u003e. Some studies have shown women to be more sensitive to social rewards than men \u003csup\u003e\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e\u003c/sup\u003e. Perhaps, there may be a link between social behavior and decision-making and reward processing. However, as our study incorporated no measure of sociability, we are not able to determine a clear directionality. A follow-up study may consider implementing a questionnaire to measure social behavior in participants or implement both monetary and social rewards in the task design.\u003c/p\u003e\u003cp\u003e\u003cb\u003eLimitations\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThe study was performed with relatively few women (N\u0026thinsp;=\u0026thinsp;36) in comparison to men (N\u0026thinsp;=\u0026thinsp;61). Ideally, equal and larger numbers of men and women would be included in the study design. The analysis was conducted in an exploratory fashion on an existing data set originally designed to investigate whether tryptophan influenced decision-making in the two-step task \u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e. A follow-up study to confirm this finding should have a specific hypothesis and would ideally consider the influence of hormones and tracking of the female cycle. The set-up may include assessments at two time points within the cycle \u0026ndash; luteal and follicular \u0026ndash; in women on natural cycles and compare decision-making to women taking oral contraceptives. Our study adopts a binary view on sex division, future studies may also move away from this dichotomy to gain a better understanding of sex differences.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eWe compared different behaviors in the two-step task between men and women, and found increased preference for the model-free strategy in the women of our sample. On the imaging side, we found increased neural response in dlPFC after reward omission compared to receiving a reward in women and similar responses for both outcomes in men. In conclusion, our study shows different behaviors between men/women correlated to different decision-making processes and these mechanisms are linked to different brain responses. We would like to emphasize the beauty of differences and how each individual and gender can contribute his/her/their very own talents without fear of being inferior or worse in spite of.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eAcknowledgments and Disclosures\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding:\u003c/strong\u003e The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Deutsche Forschungsgemeinschaft (DFG, grants numbers 178833530 (SFB 940), 402170461 (TRR 265), 454245598 (IRTG 2773), and 521379614 (TRR 393)).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.\u003c/p\u003e\n\u003cp\u003eORCID iDs:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eMarina Lukezic ORCID iD \u003cu\u003e0000-0002-9204-4814\u003c/u\u003e\u003c/p\u003e\n\u003cp\u003ePhilipp T. Neukam ORCID iD 0000-0002-9442-0043\u003c/p\u003e\n\u003cp\u003eMichael N. Smolka ORCID iD 0000-0001-5398-5569\u003c/p\u003e\n\u003cp\u003eYacila I. Deza-Lougovski\u0026nbsp;ORCID iD \u003cu\u003e0000-0002-2624-077X\u003c/u\u003e\u003c/p\u003e\n\u003cp\u003eAuthor contributions\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConcept, design and data acquisition (M.G., P.N., Y.I.D, M.N.S.), analysis and/or interpretation of data (M.L., H.C., M.N.S.), drafting the article (M.L.), reviewing critical version of the article (M.L., H.C., M.N.S.). All authors approved the final version of the manuscript.\u003c/p\u003e\n\u003cp\u003eData Availability Statement\u003c/p\u003e\n\u003cp\u003eThe raw data supporting the conclusions of this article will be made available by the authors, without undue reservation ([email protected]).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAdditional Information\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNone of the authors have any competing interest that must be disclosed. \u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBolt, E. et al. Effects of a virtual gender swap on social and temporal decision-making. \u003cem\u003eSci. Rep.\u003c/em\u003e \u003cb\u003e11\u003c/b\u003e, 15376. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-021-94869-z\u003c/span\u003e\u003cspan address=\"10.1038/s41598-021-94869-z\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2021).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eStoet, G. \u0026amp; Geary, D. C. The Gender-Equality Paradox in Science, Technology, Engineering, and Mathematics Education. \u003cem\u003ePsychol. Sci.\u003c/em\u003e \u003cb\u003e29\u003c/b\u003e, 581\u0026ndash;593. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/0956797617741719\u003c/span\u003e\u003cspan address=\"10.1177/0956797617741719\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2018).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCroson, R. \u0026amp; Gneezy, U. Gender Differences in Preferences. \u003cem\u003eJ. Econ. Lit.\u003c/em\u003e \u003cb\u003e47\u003c/b\u003e, 448\u0026ndash;474. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1257/jel.47.2.448\u003c/span\u003e\u003cspan address=\"10.1257/jel.47.2.448\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2009).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHolstein, C. B. \u0026amp; Irreversible Stepwise Sequence in the Development of Moral Judgment: A Longitudinal Study of Males and Females. \u003cem\u003eChild Dev.\u003c/em\u003e \u003cb\u003e47\u003c/b\u003e, 51\u0026ndash;61. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2307/1128282\u003c/span\u003e\u003cspan address=\"10.2307/1128282\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (1976).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTingh\u0026ouml;g, G. et al. Intuition and Moral Decision-Making - The Effect of Time Pressure and Cognitive Load on Moral Judgment and Altruistic Behavior. \u003cem\u003ePLoS One\u003c/em\u003e. \u003cb\u003e11\u003c/b\u003e, e0164012. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1371/journal.pone.0164012\u003c/span\u003e\u003cspan address=\"10.1371/journal.pone.0164012\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2016).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDaw, N. D., Niv, Y. \u0026amp; Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. \u003cem\u003eNat. Neurosci.\u003c/em\u003e \u003cb\u003e8\u003c/b\u003e, 1704\u0026ndash;1711. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/nn1560\u003c/span\u003e\u003cspan address=\"10.1038/nn1560\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2005).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDolan, R. J. \u0026amp; Dayan, P. Goals and habits in the brain. \u003cem\u003eNeuron\u003c/em\u003e \u003cb\u003e80\u003c/b\u003e, 312\u0026ndash;325. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.neuron.2013.09.007\u003c/span\u003e\u003cspan address=\"10.1016/j.neuron.2013.09.007\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2013).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eThorndike, E. L. \u0026lt;animal by=\"edward=\" experimental=\"intelligence=\" l.=\"studies=\" thorndike.pdf=\"\"\u0026gt;\u0026lt;/animal\u0026gt;. \u003cem\u003eJ. Philos. Inc\u003c/em\u003e. \u003cb\u003e9\u003c/b\u003e, 193\u0026ndash;194. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2307/2013764\u003c/span\u003e\u003cspan address=\"10.2307/2013764\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (1912).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBalleine, B. W. D. N. a. O. D. J. \u0026lt;multiple and=\"dopamine.pdf=\" forms=\"function=\" learning=\"of=\" the=\"value=\"\u0026gt;\u0026lt;/multiple\u0026gt;. \u003cem\u003eNeuroeconomics\u003c/em\u003e 367\u0026ndash;387. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/B978-0-12-374176-9.00024-5\u003c/span\u003e\u003cspan address=\"10.1016/B978-0-12-374176-9.00024-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2009).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBalleine, B. W. \u0026amp; O'Doherty, J. P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. \u003cem\u003eNeuropsychopharmacology: official publication Am. Coll. Neuropsychopharmacol.\u003c/em\u003e \u003cb\u003e35\u003c/b\u003e, 48\u0026ndash;69. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/npp.2009.131\u003c/span\u003e\u003cspan address=\"10.1038/npp.2009.131\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2010).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTolman, E. C. Cognitive maps in rats and men. \u003cem\u003ePsychol. Rev.\u003c/em\u003e \u003cb\u003e55\u003c/b\u003e, 189\u0026ndash;208. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1037/h0061626\u003c/span\u003e\u003cspan address=\"10.1037/h0061626\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (1948).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWorbe, Y. et al. Valence-dependent influence of serotonin depletion on model-based choice strategy. \u003cem\u003eMol. Psychiatry\u003c/em\u003e. \u003cb\u003e21\u003c/b\u003e, 624\u0026ndash;629. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/mp.2015.46\u003c/span\u003e\u003cspan address=\"10.1038/mp.2015.46\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2016).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDaw, N. D., Gershman, S. J., Seymour, B., Dayan, P. \u0026amp; Dolan, R. J. Model-based influences on humans' choices and striatal prediction errors. \u003cem\u003eNeuron\u003c/em\u003e \u003cb\u003e69\u003c/b\u003e, 1204\u0026ndash;1215. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.neuron.2011.02.027\u003c/span\u003e\u003cspan address=\"10.1016/j.neuron.2011.02.027\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2011).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A. \u0026amp; Daw, N. D. Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. \u003cem\u003eeLife\u003c/em\u003e 5 (2016). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.7554/eLife.11305\u003c/span\u003e\u003cspan address=\"10.7554/eLife.11305\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eOguchi, M. et al. Proselfs depend more on model-based than model-free learning in a non-social probabilistic state-transition task. \u003cem\u003eSci. Rep.\u003c/em\u003e \u003cb\u003e13\u003c/b\u003e, 1419. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-023-27609-0\u003c/span\u003e\u003cspan address=\"10.1038/s41598-023-27609-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDhingra, I. et al. Sex differences in neural responses to reward and the influences of individual reward and punishment sensitivity. \u003cem\u003eBMC Neurosci.\u003c/em\u003e \u003cb\u003e22\u003c/b\u003e, 12. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s12868-021-00618-3\u003c/span\u003e\u003cspan address=\"10.1186/s12868-021-00618-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2021).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eOrsini, C. A., Willis, M. L., Gilbert, R. J., Bizon, J. L. \u0026amp; Setlow, B. Sex differences in a rat model of risky decision making. \u003cem\u003eBehav. Neurosci.\u003c/em\u003e \u003cb\u003e130\u003c/b\u003e, 50\u0026ndash;61. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1037/bne0000111\u003c/span\u003e\u003cspan address=\"10.1037/bne0000111\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2016).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKoot, S., van den Bos, R., Adriani, W. \u0026amp; Laviola, G. Gender differences in delay-discounting under mild food restriction. \u003cem\u003eBehav. Brain. Res.\u003c/em\u003e \u003cb\u003e200\u003c/b\u003e, 134\u0026ndash;143. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.bbr.2009.01.006\u003c/span\u003e\u003cspan address=\"10.1016/j.bbr.2009.01.006\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2009).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003evan den Bos, R., Jolles, J., van der Knaap, L., Baars, A. \u0026amp; de Visser, L. Male and female Wistar rats differ in decision-making performance in a rodent version of the Iowa Gambling Task. \u003cem\u003eBehav. Brain. Res.\u003c/em\u003e \u003cb\u003e234\u003c/b\u003e, 375\u0026ndash;379. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.bbr.2012.07.015\u003c/span\u003e\u003cspan address=\"10.1016/j.bbr.2012.07.015\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2012).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eOverman, W. H. et al. Performance on the IOWA card task by adolescents and adults. \u003cem\u003eNeuropsychologia\u003c/em\u003e \u003cb\u003e42\u003c/b\u003e, 1838\u0026ndash;1851. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.neuropsychologia.2004.03.014\u003c/span\u003e\u003cspan address=\"10.1016/j.neuropsychologia.2004.03.014\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2004).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eIslas-Preciado, D. et al. Risk-based decision making in rats: Modulation by sex and amphetamine. \u003cem\u003eHorm. Behav.\u003c/em\u003e \u003cb\u003e125\u003c/b\u003e, 104815. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.yhbeh.2020.104815\u003c/span\u003e\u003cspan address=\"10.1016/j.yhbeh.2020.104815\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2020).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWang, L. et al. Sex differences in neural substrates of risk taking: Implications for sex-specific vulnerabilities to internet gaming disorder. \u003cem\u003eJ. Behav. addictions\u003c/em\u003e. \u003cb\u003e11\u003c/b\u003e, 778\u0026ndash;795. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1556/2006.2022.00057\u003c/span\u003e\u003cspan address=\"10.1556/2006.2022.00057\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2022).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGaillard, A., Fehring, D. J. \u0026amp; Rossell, S. L. A systematic review and meta-analysis of behavioural sex differences in executive control. \u003cem\u003eEur. J. Neurosci.\u003c/em\u003e \u003cb\u003e53\u003c/b\u003e, 519\u0026ndash;542. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/ejn.14946\u003c/span\u003e\u003cspan address=\"10.1111/ejn.14946\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2021).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGilger, M. D. et al. Arbitration between model-free and model-based control is not affected by transient changes in tonic serotonin levels. \u003cem\u003eJ. Psychopharmacol. (Oxford, England)\u003c/em\u003e. \u003cb\u003e38\u003c/b\u003e, 178\u0026ndash;187. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/02698811231216325\u003c/span\u003e\u003cspan address=\"10.1177/02698811231216325\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2024).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDaw, N. D. in \u003cem\u003eNeuroeconomics\u003c/em\u003e 299\u0026ndash;320 (2014).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKroemer, N. B. et al. Balancing reward and work: anticipatory brain activation in NAcc and VTA predict effort differentially. \u003cem\u003eNeuroImage 102 Pt\u003c/em\u003e. \u003cb\u003e2\u003c/b\u003e, 510\u0026ndash;519. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.neuroimage.2014.07.060\u003c/span\u003e\u003cspan address=\"10.1016/j.neuroimage.2014.07.060\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2014).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eNussenbaum, K. \u0026amp; Hartley, C. A. Reinforcement learning across development: What insights can we draw from a decade of research? \u003cem\u003eDev. Cogn. Neurosci.\u003c/em\u003e \u003cb\u003e40\u003c/b\u003e, 100733. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.dcn.2019.100733\u003c/span\u003e\u003cspan address=\"10.1016/j.dcn.2019.100733\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2019).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSatinder, P. \u0026amp; Singh, R. S. S. Reinforcement Learning with Replacing Eligibility Traces. \u003cem\u003eSpringer Link.\u003c/em\u003e \u003cb\u003e22\u003c/b\u003e, 123\u0026ndash;158 (1996).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eFeher da Silva, C. \u0026amp; Hare, T. A. A note on the analysis of two-stage task results: How changes in task structure affect what model-free and model-based strategies predict about the effects of reward and transition on the stay probability. \u003cem\u003ePLoS One\u003c/em\u003e. \u003cb\u003e13\u003c/b\u003e, e0195328. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1371/journal.pone.0195328\u003c/span\u003e\u003cspan address=\"10.1371/journal.pone.0195328\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2018).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKool, W., Gershman, S. J. \u0026amp; Cushman, F. A. Cost-Benefit Arbitration Between Multiple Reinforcement-Learning Systems. \u003cem\u003ePsychol. Sci.\u003c/em\u003e \u003cb\u003e28\u003c/b\u003e, 1321\u0026ndash;1333. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/0956797617708288\u003c/span\u003e\u003cspan address=\"10.1177/0956797617708288\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSmid, C. R., Kool, W., Hauser, T. U. \u0026amp; Steinbeis, N. Computational and behavioral markers of model-based decision making in childhood. \u003cem\u003eDev. Sci.\u003c/em\u003e \u003cb\u003e26\u003c/b\u003e, e13295. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/desc.13295\u003c/span\u003e\u003cspan address=\"10.1111/desc.13295\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCulbreth, A. J., Westbrook, A., Daw, N. D., Botvinick, M. \u0026amp; Barch, D. M. Reduced model-based decision-making in schizophrenia. \u003cem\u003eJ. Abnorm. Psychol.\u003c/em\u003e \u003cb\u003e125\u003c/b\u003e, 777\u0026ndash;787. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1037/abn0000164\u003c/span\u003e\u003cspan address=\"10.1037/abn0000164\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2016).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSebold, M. et al. Model-based and model-free decisions in alcohol dependence. \u003cem\u003eNeuropsychobiology\u003c/em\u003e \u003cb\u003e70\u003c/b\u003e, 122\u0026ndash;131. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1159/000362840\u003c/span\u003e\u003cspan address=\"10.1159/000362840\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2014).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eEppinger, B., Walter, M., Heekeren, H. R. \u0026amp; Li, S. C. Of goals and habits: age-related and individual differences in goal-directed decision-making. \u003cem\u003eFront. Neurosci.\u003c/em\u003e \u003cb\u003e7\u003c/b\u003e, 253. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fnins.2013.00253\u003c/span\u003e\u003cspan address=\"10.3389/fnins.2013.00253\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2013).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003evan den Bos, R., Homberg, J. \u0026amp; de Visser, L. A critical review of sex differences in decision-making tasks: focus on the Iowa Gambling Task. \u003cem\u003eBehav. Brain. Res.\u003c/em\u003e \u003cb\u003e238\u003c/b\u003e, 95\u0026ndash;108. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.bbr.2012.10.002\u003c/span\u003e\u003cspan address=\"10.1016/j.bbr.2012.10.002\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2013).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLin, A., Adolphs, R. \u0026amp; Rangel, A. Social and monetary reward learning engage overlapping neural substrates. \u003cem\u003eSoc. Cognit. Affect. Neurosci.\u003c/em\u003e \u003cb\u003e7\u003c/b\u003e, 274\u0026ndash;281. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/scan/nsr006\u003c/span\u003e\u003cspan address=\"10.1093/scan/nsr006\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2012).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eIzuma, K., Saito, D. N. \u0026amp; Sadato, N. Processing of social and monetary rewards in the human striatum. \u003cem\u003eNeuron\u003c/em\u003e \u003cb\u003e58\u003c/b\u003e, 284\u0026ndash;294. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.neuron.2008.03.020\u003c/span\u003e\u003cspan address=\"10.1016/j.neuron.2008.03.020\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2008).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSpreckelmeyer, K. N. et al. Anticipation of monetary and social reward differently activates mesolimbic brain structures in men and women. \u003cem\u003eSoc. Cognit. Affect. Neurosci.\u003c/em\u003e \u003cb\u003e4\u003c/b\u003e, 158\u0026ndash;165. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/scan/nsn051\u003c/span\u003e\u003cspan address=\"10.1093/scan/nsn051\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2009).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"model-based control, model-free control, cognitive-ability, sex difference, fMRI, two-stage Markov decision task","lastPublishedDoi":"10.21203/rs.3.rs-7021287/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7021287/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eDespite men and women nearing equal opportunities in education and career, we continue to identify sex-differences in decision-making. For instance, more women are found in social fields as nursing and medicine. One existing study found higher model-free behavior in the two-step task in women and another study found model-free behavior to be associated with prosocial behavior. We conducted an exploratory analysis to see (1) whether women differ regarding model-free and model-based behavioral control from men; (2) what brain responses underlie such differences. Participants (total 98; 37 female) performed the two-stage Markov task while in the fMRI scanner. Our analysis confirmed a previous finding of increased model-free behavior in women and we additionally observed an increased neural response in dorsolateral prefrontal cortex in women following not rewarded compared to rewarded trials. This difference was not present in men. Different decision-making patterns and neural responses observed between sexes in the two-step task are possibly able to predict existing real-world behavioral differences between men and women.\u003c/p\u003e","manuscriptTitle":"Sex Differences in Model free and Model based Behavior","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-31 10:29:31","doi":"10.21203/rs.3.rs-7021287/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-12-04T08:18:24+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-11-12T17:20:53+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"308086683134622363635069004302060808742","date":"2025-11-10T08:33:19+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-11-09T09:51:00+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"124262417372653135318300715379261684810","date":"2025-10-25T07:34:47+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"161863390513706114031399611805063637709","date":"2025-10-24T23:22:17+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"161863390513706114031399611805063637709","date":"2025-10-10T16:53:06+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"155237187215153554640881728746420506893","date":"2025-09-21T01:31:43+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-07-28T17:42:51+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-07-28T17:20:52+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-07-09T09:56:26+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-07-06T11:19:05+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2025-07-06T11:16:16+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f9f8de70-8575-48f0-a994-61f942130eac","owner":[],"postedDate":"July 31st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[{"id":52325437,"name":"Biological sciences/Neuroscience"},{"id":52325438,"name":"Biological sciences/Psychology"},{"id":52325439,"name":"Social science/Psychology"}],"tags":[],"updatedAt":"2025-12-04T08:23:35+00:00","versionOfRecord":[],"versionCreatedAt":"2025-07-31 10:29:31","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7021287","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7021287","identity":"rs-7021287","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}