A Temporal Difference Approach To Relief

A Temporal Difference Approach To Relief | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A Temporal Difference Approach To Relief silvia papalini, Esther Krul, Tom Haber, Bram Vervliet This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6194403/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Subjective relief from threat omission is emerging as a proxy index of reward Prediction Error signaling during safety learning. Yet, the relation between relief and prediction error signaling remains poorly understood, limiting translational research. Here, we complemented our previous research in this field by providing further evidence of similarities between the emotion of relief and prediction error signaling. To this end, we enrolled fifty-one healthy participants, and applied a Temporal Difference Learning approach to subjective relief ratings collected during a classical fear extinction learning paradigm. If relief is a reliable index of reward Prediction Error signal then it should display the classical backpropagation from unexpected reward delivery (unexpected threat omission) to conditioned cue presentation, as the large literature in reward learning clearly demonstrated across species. We found that a TD model largely fits subjective relief ratings. Future studies could thereby use this TD model to understand if and how relief-PE drive fear extinction and its potential deficit. Biological sciences/Neuroscience Biological sciences/Psychology/Human behaviour Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. Introduction Reward Prediction Error (rPE) signals are computed when actual states of the world are better than expected 1 , 2 . For example, after having introduced the exact amount of cash into a vending machine, a rPE signal is generated if we receive more snacks than we expected. Decades of research conducted across different species recognize the critical role played by such error-based signals in the generation of new reward-based learning and the connected approach behaviors 1 – 3 . Such research established that rPEs are computed by the phasic responses of dopaminergic neurons in the mesolimbic/midbrain, which response’s magnitude directly depends on the error’s magnitude (or magnitude of the mismatch). These dopaminergic bursts are typically elevated in the early trials of a new experience and/or training. Then, with repeated exposures to the unexpected reward outcome, the accuracy of the predictions increase and the magnitude of these phasic bursts attenuate accordingly. When the obtainment of the reward becomes completely obvious, which typically emerges toward the end of a successful learning process, this dopaminergic-based rPE signal dissipates, giving rise to a new belief, and, likely, to a new prominent behavior (i.e. “I will always use this vending machine when hungry since it always delivers more snacks than I paid for”) 1 , 2 . To date, however, it has become increasingly clear that rPE signaling does not only govern reward-based learning, but also some forms of safety learning, such as fear extinction learning 4 , 5 . During a typical fear extinction paradigm, a cue (or Conditioned Stimulus [CS], e.g. the color of a light) that was previously learned to predict the occurrence of an unpleasant outcome (or Unconditioned stimulus [US], e.g. an unpleasant electrical stimulation) is no longer followed by this unpleasant US. Computationally, the surprisingly violation of the negative CS—US expectation is captured by the Rescorla-Wagner (RW) model 6 , a conceptual framework that uses rPEs to explain changes in conditional (fear) responding. rPEs from unpleasant US omissions, which represent better-than-expected outcomes, are thought to decrease such fearful response through the weakening of the association between a fear-conditioned CS and the unexpected absence of its outcome (US) or even by promoting the formation of a new safe memory of the CS that competes against the original threatening CS. Recent findings in rodents 7 , 8 , indicate that such omission rPEs are decoded by the dopaminergic mesolimbic/midbrain system. Similar to the typical rPE signal observed in appetitive learning paradigms, the dopaminergic burst exerted during such omissions evolves as a classical PE signal: it is elevated at the maximum magnitude of the mismatch between the expected threat and the confrontation with its absence and, with repeated exposures to these threat omissions, minimizes and dissipates over the fear extinction training, together with the fear-elicited response 7 , 8 . The same studies in rodents have also shown that blocking or accelerating the activity of these (midbrain) dopaminergic neurons at the exact time of the unexpected omission of the US, impairs or enhances extinction learning, respectively 7 , 8 . These findings suggest that threat omissions are processed as unexpected rewards that drive the new learning of safety. New learning of safety is the basis of exposure-based treatments for exaggerated fears in anxiety disorders. Therefore, translational research aiming to support the treatment of excessive anxiety is showing an increased interest in understanding the relation between rPE and reduction of clinical fear 9 . Yet, the advancement of this knowledge is hindered by the inability to investigate dopaminergic rPE signaling directly in humans, which would require highly invasive procedures to record the -in vivo- activity of dopaminergic neurons in deep nuclei of the human brain. Hence, there is a need for proximal indices that can indirectly track the development of rPE. One promising index is the subjective experience of relief, which we recently introduced in the fear conditioning procedure in humans. Relief is the pleasant feeling of surprise that is generated when an expected threat stays away 10 – 15 . We previously added a relief rating scale after each CS presentation in fear extinction and observed that the level of relief is high during the first unexpected omissions of the US and progressively reduces over the course of the omissions, in line with the expected course of rPE signaling during fear extinction learning 13 , 16 . This suggests that the course of relief during fear extinction could be used to probe the learning of safety, with a sharp decline in relief reflecting robust learning of safety. However, if relief is still relatively high at the end of fear extinction, this could indicate that little safety learning has been acquired. Hence, relief, together with the reduction in fear levels, is emerging as a potential index of fear extinction learning as well as a new potential target for assessing fear extinction-based sessions, such as those characterizing exposure therapy for clinical anxiety 9 . However, to provide more guidance to translational and clinical research it is important to further investigate if the subjective relief ratings we introduced in our paradigms reliably represents a proxy (indirect) index of a rPE signal. This deeper understanding can be achieved by investigating whether relief ratings also capture other well recognized aspects of dopaminergic PE signals, such as their backpropagation over the course of learning. Backpropagation of PEs is a cornerstone of the Temporal Difference learning (TD) model, an influential extension of the Rescorla-Wagner model 17 , 18 . While the RW model specifies the change in associative strength based on the whole trial’s result, the TD model computes changes in this strength within the trial. This model can do so since it distinguishes times/moments within the trial and allows the calculation of within-trial PEs. What is peculiar in the TD algorithm is that, over the course of the trials, it uses these within-trial PEs to update the value of the prediction of the reward, from the moment of the obtainment of the reward back to the onset of the CS (or back to the earliest possible cue that reliably predicts the future reward). Unless the reward contingencies from trial to trial is subjected to change, learning is fully acquired when the value of the prediction of the reward for each timepoint of the presentation of the CS is equal to the total reward available in the trial. In other words, with a TD learning model we can measure predictions about a future reward that do not uniquely occur at the exact time of the omission of a threat (e.g. at CS offset, as the RW model assumes), but also predictions about reward that emerge during cue presentations (e.g. CS onset), or avoidance actions taken during CS presentation (in the case of active avoidance trials). This measurement is key since within the context of fear extinction a TD algorithm allows a large comprehensive investigation of the role of omission PE in safety learning and consequent adjustment of the connected behavior. For example, differently from the RW model, the TD model can be used for explaining high-order conditioning as it allows us to promptly predict the presence of a now safe cue/situation and to engage the behavioral system to reduce the previously gained distance from this originally threatening cue. Hence, the utility of a backpropagation of an omission PE would consist in helping adjusting the value and prediction of the originally threatening but now safe cue, helping the safety learning process. To date, however, no studies investigated the potential role played by such temporal dynamic of threat omission PE on fear extinction in either animals or humans. Nonetheless, if dopaminergic rPEs are dynamic signals that do not remain anchored to the time of reward outcome 17 , 19 , 20 and subjective relief is a reliable indirect index of this dopaminergic PEs, then subjective relief ratings could also be used to provide information about such crucial temporal property. Hence, the aim of the present study was to test whether the decrease of relief that we previously observed at US omissions during fear extinction actually reflects a backpropagation from CS offsets to CS onsets, as predicted by TD, see prediction in Fig. 1 . To track the expected within-stimulus changes in relief, we added relief ratings at multiple time-points during CS presentations and we used computational modelling to estimate TD parameters. Additionally, we explored whether anxiety-related personality traits modulate this course in relief and computed TD parameters. Previously, we found that more anxious individuals report more relief at CS offsets (US omissions). Here, we explore whether this increased relief is paralleled by an absence of a temporal shift to CS onsets, which would reflect an inability to learn that the CS has become safe. -Fig. 1- 2. Methods 2.1 Participants To estimate the sample size, we used the data from our previous study on relief 13 . The reference was the decrease of relief pleasantness over the blocks of the extinction phase (ɳ 2 p = 0.187). The GLMMPSS software 21 provided a sample size of 38 subjects with alpha 0.05 and a power of 0.95. To make up for bad recording we enrolled fifty healthy participants. They were recruited from the local area (Leuven) via the Experiment Management System (ESM, http://psykuleuven.sona-systems.com/all_exp.aspx ). The protocol was approved by the Social and Societal Ethics Committee of KU Leuven and each participant provided written consent to the participation of the study in agreement with the local Ethical Committee (EC). Fifty-one healthy participants (average age 20.6, 39F: females and 12M: males) performed the task. 2.2 Stimuli and Apparatus The experiment took place in a laboratory room with dimmed light. All stimuli and questionnaires were presented on a computer screen using Affect 5 software 22 . Each trial started with an image of a desk lamp that lights up (after 4-5s) in one of three colors: red, blue, or yellow. These stimuli were taken from a previously validated task 23 . The three colors served as conditional stimuli (CSs, duration: starting after 3 sec. from the CS onset and continuing between 2 and 4 seconds after the relief rating). One color (CS+) was followed by the US, while the other two (CS-s) were not. The US (2ms electrical pulse) and US omissions were accompanied by an image of a lightning bolt and a lightning bolt with a cross through it, respectively (duration: 2s). The configurations of colors and CS types were pseudo-randomized between participants, such that all possible configurations were used with equal frequency. The unconditional stimulus was an uncomfortable, but not painful, electrical stimulation. The aversiveness of the US was individually calibrated via a work-up procedure. The participants rated gradually increasing pulse intensities until they reach a rating of “very uncomfortable but not painful”. The selected intensity of the electrical stimulation was used throughout the whole experiment without changes. A DS7 (Digitimer, Hertfordshire, UK) delivered the 2ms electrical pulse to the forearm of the dominant hand via two adjacent SensorMedics surface-electrodes with KY gel. 2.3 The paradigm Before the actual task, participants were familiarized with the stimuli and relief ratings through a short tutorial. At the start of the Pavlovian fear conditioning phase, they were then instructed as follow: “Try to see if there is a pattern between the pictures and the shocks”. The Pavlovian phase contained eight trials for each CSs- and twelve trials for the CS+ (twenty-eight trials in total). Next, participants performed a fear extinction learning phase. At the beginning of each phase, participants were instructed that they might or might not receive the electrical stimulation, a procedure that typically reintroduces some initial uncertainty about the CS-. This phase contained five trials for each CS- and eight trials for the CS+ (eighteen trials in total), which were visualized within the same conditioning context. Crucially, no electrical stimulations were delivered. As general rule, no more than two trials contained the same color. 2.4 Self-reports All participants completed three psychometric questionnaires at the start of the experiment. The scores at these questionnaires allowed us to explore the effect of individual differences in distress tolerance, anxiety traits, and positive and negative affect on relief and its temporal dynamics. The Distress Tolerance Scale (DTS) examined the participants’ perceived ability to tolerate emotional distress 24 . The State-Trait Anxiety Inventory (STAI-T) was used to measured anxiety traits 25 , and the Positive and Negative Affect Schedule (PANAS) was used to measure positive and negative affect 26 . During the Pavlovian fear conditioning and fear extinction learning phases, subjective relief intensity was measured at every CS presentation, and any time the US was not delivered (at CS offset). The measurement was both a binary (yes/no) and continuous (0-100) variable. Three seconds after each CS onset and US omission, participants received the question “Did this color make you feel relieved?” and “Did the absence of the shock make you feel relieved?”, respectively. The question appeared at the bottom of the screen along with two buttons “YES” and “NO” (forced-choice). After a “YES” response, participants rated the intensity of their relief on a VAS scale ranging from ‘not relieved’ (0) to ‘very relieved’ (100), Fig. 2 . After each learning phase, the participants indicated their US expectations for the first and last presentation of each CS on a 7-point Likert scale. -Fig. 2- 2.5 Procedures Upon arrival, participants were informed about the nature of the experiment (understanding emotional learning), followed by the judgment of the exclusion criteria and the informed consent procedure. Exclusion criteria were: pregnancy, cardiovascular and pulmonary diseases, neurological and/or psychiatric disorders, or any other serious medical condition, presence of an electronic implant, pain at hands or wrists, and a doctor’s request to avoid stressful situations. Next, participants completed three psychological questionnaires and were prompted to take a moment to think about the emotion of relief. Then, the electrodes for the electrical stimulations were attached. The intensity of the electrical stimulation was calibrated to match a subjective rating of ‘very uncomfortable but not painful’. The main task (duration 20–25 minutes) consisted of a Pavlovian fear conditioning and an fear extinction learning phase, modified from the previously validated paradigm of Vervliet et al. (2017). Finally, participants completed an adverse events form and debriefing was provided at the end of the session. 2.6 The Temporal Difference Model To model the backpropagation of the PE during Pavlovian fear conditioning and fear extinction learning, we fit a Temporal Difference Learning Model to the subjective relief ratings for each participant. The algorithm is defined as: V(s t ) ← V(s t ) + α[r t + 1 + γV(s t + 1 ) – V(s t )] where V(s t ) is the expected reward (US omission), one at the CS onset and one for the CS offset. This initial value is updated in each subsequent trial, where V(st) is the expected reward at the CS on/offset, α is the learning rate, r is the obtained reward at moment t + 1, and γ is the discount rate (gamma) of the value of future expected rewards. When the γ value approaches zero the backpropagation is weak whereas a value close to one indicates the presence of a strong temporal backpropagation of the PE (see Fig. 3 for visual representation of a simulated subject). -Fig. 3- In a final step, the score obtained from each psychological questionnaire was then added as a covariate to the TD model to investigate whether the discount parameter was different between participants with different levels of trait anxiety, distress tolerance, positive and negative affect, separately. To obtain such indication, we specifically looked at the covariate-parameter relationship (between gamma and the score at the questionnaires). One participant was excluded from all the analyses since they did not engage in the task (i.e., only providing relief ratings twice at CS presentation and never reporting relief at US). Hence, the final sample consisted of fifty participants. 2.7 TD parameters setting We fit the TD model with the subjective relief intensity rating (rescaling the simulated PEs to [0,100] to have the same range as for the relief ratings) experienced at the offset and onset of the CS + and at the offset and onset of the CS-. Since not all participants exhibited the expected learning, as indicated by a decrease in relief ratings at CS offset, we excluded such participants by fitting a linear model to the relief ratings at CS offset and constraining the slope (learning rate) to be smaller than − 1. This is similar to the strategy used by 27 , but we used a more stringent cutoff that only allowed learners (including small learning) to be included. This approach led to excluding 10 participants for CS + and 10 participants for CS-. 2.8 Statistical analysis A mixed effects model was used on retrospective US expectancy ratings to examine Pavlovian fear conditioning. The model included the fixed term Time (‘First’ and ‘Last’ presentation of the CS) and CS (‘CS+’ and ‘CS-’). A similar model was applied to the retrospective US expectancy ratings collected after fear extinction learning. It is worth to mention that Skin Conductance Levels were measured but not used given the suboptimal design for this measurement. For the statistical analysis related to relief ratings obtained from the CS onset and CS offset, we used a non-linear mixed effects model (nlme) using nlme in r studio 28 the CS + and the CS- separately. Alpha (α: learning rate), Gamma (γ: discount value), the initial reward expectancy at the onset of the extinction phase and the initial reward expectancy at the CS offset were included as fixed terms. The selection of the random components added to this model was performed by using the Bayesian Information Criterion (BIC) 29 , which provides a trade-off between model fit and model complexity. This resulted in two random components, one for alpha and one for gamma. Subsequently, the standardized scores from the psychometric questioners were added to the model to investigate for a potential fixed effect of each covariate on gamma and alpha parameters. Due to the non-linear nature of the model, the fitting procedure requires initial starting points for all parameters. For the initial general expectation of reward (omission of the US) at the beginning of the extinction phase, we used an adjusted value of the US-expectations from the end of Pavlovian conditioning phase of 0.5. The expectation to receive a reward at the CS + offset was set to 0.2 since this CS was previously conditioned to the US delivery while it was set to 0.8 for the CS- offset. We decided to maintain such a degree of uncertainty (adjusted value) in these expectations given the start of a new phase (Pavlovian ◊ Extinction) and the instructions we provide at the beginning of extinction (as by protocol: ”also during this phase you might or might not receive a shock”) typically reintroduce some degree of uncertainty about the CS ◊ US contingency. However, the results of the model fitting did not seem particularly sensitive to the starting values. The initial learning rate (alpha) and discount rate (gamma) were set to 0.7 and 0.9 respectively. Anonymized datasets and scripts for the statistical analysis can be found in https://github.com/silviapapalini/td . 3. Results Before proceeding with the TD model, we first ensure the presence of conditioning in both the learning phases (manipulation checks). 3.1 Retrospective expectancy ratings 3.1.1 Pavlovian conditioning The results from the linear mixed effects model showed a decrease in US expectancy from the first to the last presentation of the CS1- when the interaction between Time and CS + was added to the model. The same expected decrease was observed for the CS2- (Table 1 ), Fig. 4 left panel. Results from three post hoc (Bonferroni Corrected) comparisons run for each CS confirmed the expected reduction in US expectancy from the first to the last CS1- (Est-ꞵ = -2.53 SE = 0.28, t-value = -9.02, p -value < 0.001), a similar reduction for the CS2- (Est-ꞵ = -3.02, SE = 0.27, t-value = -11.03, p -value < 0.001), and an increase in US expectancy from the first to the last CS+ (Est-ꞵ = 1.49, SE = 0.20, t-value = 7.30, p -value < 0.001). Table 1 shows the results from the linear mixed effects model applied to the retrospective US expectancy ratings of the Pavlovian fear conditioning and fear extinction learning phase Pavlovian Extinction Fixed Effects (LMM) Est - ꞵ SE DF t-value p -value Est - ꞵ SE DF t-value p -value Time(last) 1.49 0.25 240 5.97 < 0.001 -3.65 0.29 240 -12.41 < 0.001 CS(CS1-) -1.49 0.25 240 -5.97 < 0.001 -1.59 0.29 240 -5.41 < 0.001 CS(CS2-) -0.88 0.25 240 -3.52 < 0.001 -1.65 0.29 240 -5.62 < 0.001 Time(last) x CS(CS1-) -4.02 0.35 240 -11.39 < 0.001 1.41 0.42 240 3.38 < 0.001 Time(last) x CS(CS2-) -4.51 0.35 240 -12.77 < 0.001 1.53 0.42 240 3.68 < 0.001 Reference levels: Time(first presentation) and CS(CS+) SE: Standard Error; DF: Degrees of Freedom 3.1.2 Fear extinction The results from the linear mixed effects model showed a decrease in US expectancy from the first to the last presentation of the CS + when the factor Time was added to the model. The same expected decrease was observed for the CS1- and CS2- (Table 1 ), since the new extinction phase probably reintroduced some uncertainty about the safety of the first CSs-, see Fig. 4 right panel. Results from three post hoc (Bonferroni Corrected) comparisons run for each CS confirmed the expected stronger reduction in US expectancy from the first to the last CS+ (Est-ꞵ = -3.65, SE = 0.31, t-value = -11.69, p -value < 0.001), from the first to the last CS1- (Est-ꞵ = -2.25, SE = 0.32, t-value = -6.96, p -value < 0.001), and from the first to the last CS2- (Est-ꞵ = -2.12, SE = 0.34, t-value = -6.29, p -value < 0.001). -Table 1- -Fig. 4- 3.2 TD approach to (continuous) relief ratings during fear extinction: Relief levels from each participant in response to CS onsets and CS offsets were fit with the TD model separately for each CS type. For the CS+ (Fig. 5 , left panel), the model provided a good fit for 40 participants. The results from the non-linear mixed effects model showed significant main effects of the learning rate and discount factors, the former indicating successful learning, and the latter indicating the presence of a temporal backpropagation. The expectation (Vs) of receiving a reward (US omission) was not significant at the initial CS onset but it was significant at the first offset (Table 2 ). Adding the scores obtained from the psychometric questionnaire to this model did not result in any significant interaction between the covariate and Alpha or Gamma (all p -values above 0.05). Table 2 shows the results from the nonlinear mixed effects model when the TD algorithm was applied to the CS+. TD model for the CS+ Fixed effects (NLME) Est - ꞵ SE DF t-value p -value α 0.30 0.04 597 7.25 < 0.001 γ 0.81 0.12 597 6.58 < 0.001 Onset 0.03 0.03 597 1.05 0.296 Offset 0.13 0.02 597 6.98 < 0.001 Random effects (NLME) SD α SD γ 0.25 0.67 Residual : 17.147 BIC : 5731.414 Reference levels: Onset (Initial) and Offset (Initial). SE: Standard Error; DF: Degrees of Freedom, SD: Standard Deviation -Table 2- For the CS- (Fig. 5 , right panel), the model provided a good fit for 40 participants. The results from the non-linear mixed effects model showed the presence of learning (significant main effect of alpha) in absence of a significant discount value (main effect of gamma). The expectations of receiving a reward (US omission) at initial onset and at the initial offset were also significant (Table 3 ). Adding the scores obtained from the psychometric questionnaire to this model did not result in any significant interaction between the covariate and Alpha or Gamma (all p -values above 0.05). Table 3 shows the results from the nonlinear mixed effects model when the TD algorithm was applied to the CS- (average relief ratings between the CS1- and the CS2-) TD model for the CS- (averaged relief between the CS1- and the CS2-) Fixed effects (NLME) Est - ꞵ SE DF t-value p -value α 0.23 0.04 357 5.21 < 0.001 γ -0.00 0.09 357 -0.02 0.987 Onset -0.37 0.04 357 -9.72 < 0.001 Offset 0.03 0.03 357 12.80 < 0.001 Random Effects (NLME) SD α SD γ 0.24 0.0 Residual : 21.031 BIC : 3702.37 Reference levels: Onset (Initial) and Offset (Initial). SE: Standard Error; DF: Degrees of Freedom, SD: Standard Deviation -Table 3- -Fig. 5- 4. Discussion This experiment was designed to investigate whether relief from threat omission during fear extinction in humans follows the typical backpropagation of reward prediction error during appetitive learning. To do so, this study utilized individual relief ratings obtained within and at the end of each trial of a fear extinction learning task. Specifically, we investigated whether relief backpropagates from threat omission to CS onset as extinction learning develops. We collected individual relief ratings at CS onset and CS offset (US omission), and we used these ratings to simulate learning-related parameters within a temporal difference model. We found that the TD algorithm produced a reasonable fit for about 80% of the participants. The subjective relief experienced at the time of the unexpected omission of the US backpropagated to the onset of the previously threat-conditioned CS+. The magnitude of this backpropagation was stimulus-dependent, as the discount factor (gamma) approached zero for the averaged safe cues (CS-) and one for the extinguished CS+. Regarding the learning rate (alpha) of the TD model, we found that alpha approached a value of 0.3 for the CS + and 0.2 for the CS-. This slight difference was expected since the CS- had already been learned as a safe cue during the previous Pavlovian phase. These results indicate that fear extinction is not exclusively governed by reward prediction error (rPE) and learning rate, as assumed by the Rescorla-Wagner (RW) model, but also by the gamma parameter, which represents the discount value that determines the importance of future rewards. The fact that gamma was high for the threatening stimulus under extinction (CS+) suggests that healthy participants adjusted their expectations and values in relation to the anticipated future omissions of the unconditioned stimulus (US). In contrast, the absence of a significant discount factor for the conditioned stimulus (CS-) may indicate that expected rewards do not significantly alter the agent's perception on the value of future (obvious) rewards associated with conditioned safety, especially since the safe cue was also linked to a lower learning rate. It is important to briefly mention that for this type of CS (CS-), we also typically observe some initial relief at CS offset since the experimental task we generally use consists of clearly separated learning phases. Essentially, the initiation of a new -extinction- learning phase after a Pavlovian fear conditioning phase reintroduces in the participant some initial uncertainty about the safety of the CS-. However, during the training, the relief for the CS- offset is usually lower and dissipates more rapidly than the relief linked to the threatening CS + offset 16 . For this reason, this relief is typically used as reference for comparison purposes, but nevertheless worthy investigating in separation from the CS+, as individuals with anxiety disorders seem to also be characterized by difficulties in differential learning and safety learning 30 , 31 , which primarily involve the CS-. The results of the present study did not indicate a significant link between the discount parameter and psychological traits relevant to anxiety/stress-related disorders such as state/trait anxiety, distress tolerance, and negative/positive affect. It is, however, important to note that our sample included only healthy participants. Future studies could therefore investigate whether slower backpropagation of omission PEs during fear extinction is associated with the presence of fear extinction deficits in clinical samples. It is indeed plausible to assume that heightened anxiety about future uncertainties could lead subjects to devalue long-term rewards and experience elevated relief even in a completely safe condition. Although this study focuses on fear extinction learning, these findings also have potential implications for our theory on relief and persistent avoidance 10 – 14 , 16 , 31 . The theory assumes that the pleasant emotion of relief arising from the active omission of a real threat acts as a reinforcer for avoidance behavior. Specifically, a higher feeling of relief during avoidance learning is thought to increase the likelihood of persisting in avoidance even after fear levels are reduced through extinction procedures 13 . It is important to clarify that we are referring here to the avoidance behaviors that still follow fear extinction. During original cue-avoidance conditioning, the prediction error is indeed dependent on the learned (effective) avoidance action of the subject (i.e., an avoidance action that abolishes the occurrence of an imminent real threat), while the threatening connotation or nature of the cue remains intact. It is important to note that a high gamma in approach learning promotes exploration, maximizing survival and adaptation across species. Indeed, an earlier phasic excitation can timely orient the attentive and behavioral system toward the earliest cues that predict the acquisition of the valued reward 1 , 2 , 32 , achieving better long-term outcomes even at the cost of missing short-term gains 1 , 2 , 33 . In the context of fear extinction, therefore, a high gamma might facilitate not only adjustments to predictions related to the threatening cue but also activate the motivational system, reducing the tendency to avoid the now-safe cue and encouraging long-term reward accumulation (e.g. promoting approach behaviors), thereby deepening the process of fear extinction. To our knowledge, a temporal difference model has only been successfully applied within the context of conditioned avoidance in rodents 34 , providing initial evidence of a strong similarity between PE backpropagation in reward-based learning and PE backpropagation in avoidance learning. Thus, it is appealing to consider the idea that the same backpropagation might also occur during avoidance learning in humans. Since this experiment did not include an instrumental component, a future step would be to investigate whether the backpropagation of relief during fear extinction is linked to the avoidance behaviors adopted toward the extinguished cue. If such a link exists, it would be interesting to explore whether individual variations in this backpropagation of relief correlate with excessive or diminished avoidance during a test for a return or renewal of avoidance behavior. It seems reasonable to assume that the absence of backpropagation in relief for the CS + may reflect a poor transfer of predictive error utility in updating previously established avoidance behaviors, ultimately leading to persistent avoidance. The present study also has limitations. First, given the exploratory nature of our experiment, it is important for future studies to replicate these preliminary findings in larger samples. Second, the presentation of the relief ratings three seconds after CS onsets and CS offsets made it impossible to reliably investigate backpropagation in the skin conductance responses, which could have added value to the present experiment. Future replications of this study could adapt the temporal presentation of CSs and ratings to allow for (neuro)physiological recording. This would also permit investigation into whether variations in CS duration (uncertainty/distance from the reward) affect the gamma parameter. In our experiment, the duration of the CS presentation after the relief rating only varied between 2 and 4 seconds and was thus not included as a covariate in our TD models. In summary, these results extend previous knowledge regarding relief from omission PE by showing that relief travels back from reward delivery to the onset of the CS that reliably predicted threat omission, similar to classical rPE within the context of appetitive learning tasks. Moving forward, such a relief-based TD approach could be used to explain conditioned reinforcement within the context of avoidance behaviors by adding relief ratings (and physiological or biological markers of relief) to classical avoidance paradigms. 5. Conclusions A TD-based approach to relief during fear extinction learning contributes to explaining within-trial effects of the temporal relationship between cues and outcomes during fear extinction learning. Declarations Conflict of interest The authors declare no competing financial interests. References Schultz W. Predictive reward signal of dopamine neurons. J Neurophysiol 1998; 80: 1–27. Schultz W. Updating dopamine reward signals. Current Opinion in Neurobiology 2013; 23: 229–238. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 2006; 442: 1042–1045. Kalisch R, Gerlicher AMV, Duvarci S. A Dopaminergic Basis for Fear Extinction. Trends Cogn Sci 2019; 23: 274–277. Papalini S, Beckers T, Vervliet B. Dopamine: from prediction error to psychotherapy. Transl Psychiatry 2020; 10: 1–13. Rescorla R, Wagner A. A theory of Pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement. Classical Conditioning: Current Research and Theory 1972. Luo R, Uematsu A, Weitemier A, Aquili L, Koivumaa J, McHugh TJ et al. A dopaminergic switch for fear to safety transitions. Nat Commun 2018; 9: 2483. Salinas-Hernández XI, Vogel P, Betz S, Kalisch R, Sigurdsson T, Duvarci S. Dopamine neurons drive fear extinction learning by signaling the omission of expected aversive outcomes. Elife 2018; 7: e38818. Craske MG, Treanor M, Conway C, Zbozinek T, Vervliet B. Maximizing Exposure Therapy: An Inhibitory Learning Approach. Behav Res Ther 2014; 58: 10–23. Leng L, Beckers T, Vervliet B. No joy - why bother? Higher anhedonia relates to reduced pleasure from and motivation for threat avoidance. Behav Res Ther 2022; 159: 104227. Leng L, Beckers T, Vervliet B. Anhedonia influences threat avoidance and relief: A conceptual replication. Journal of Mood & Anxiety Disorders 2024; 5: 100050. Leng L, Beckers T, Vervliet B. What a relief! The pleasure of threat avoidance. Emotion 2024; 24: 539–550. Papalini S, Ashoori M, Zaman J, Beckers T, Vervliet B. The role of context in persistent avoidance and the predictive value of relief. Behav Res Ther 2021; 138: 103816. San Martín C, Jacobs B, Vervliet B. Further characterization of relief dynamics in the conditioning and generalization of avoidance: Effects of distress tolerance and intolerance of uncertainty. Behaviour Research and Therapy 2020; 124: 103526. Willems AL, Oudenhove LV, Vervliet B. Omissions of Threat Trigger Subjective Relief and Prediction Error-Like Signaling in the Human Reward and Salience Systems. eLife 2024; 12. doi: 10.7554/eLife.91400.3 . Vervliet B, Lange I, Milad MR. Temporal dynamics of relief in avoidance conditioning and fear extinction: Experimental validation and clinical relevance. Behav Res Ther 2017; 96: 66–78. Sutton RS. Learning to predict by the methods of temporal differences. Mach Learn 1988; 3: 9–44. Sutton RS, Barto AG. Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review 1981; 88: 135–170. Farrell K, Lak A, Saleem AB. Midbrain dopamine neurons signal phasic and ramping reward prediction error during goal-directed navigation. Cell Rep 2022; 41: 111470. O’Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ. Temporal Difference Models and Reward-Related Learning in the Human Brain. Neuron 2003; 38: 329–337. Kreidler SM, Muller KE, Grunwald GK, Ringham BM, Coker-Dukowitz ZT, Sakhadeo UR et al. GLIMMPSE: Online Power Computation for Linear Models with and without a Baseline Covariate. J Stat Softw 2013; 54: i10. Spruyt A, Clarysse J, Vansteenwegen D, Baeyens F, Hermans D. Affect 4.0: a free software package for implementing psychological and psychophysiological experiments. Exp Psychol 2010; 57: 36–45. Milad MR, Orr SP, Pitman RK, Rauch SL. Context modulation of memory for fear extinction in humans. Psychophysiology 2005; 42: 456–464. Simons JS, Gaher RM. The Distress Tolerance Scale: Development and validation of a self-report measure. Motivation and Emotion 2005; 29: 83–102. Skapinakis P. Spielberger State-Trait Anxiety Inventory. In: Michalos AC (ed). Encyclopedia of Quality of Life and Well-Being Research . Springer Netherlands: Dordrecht, 2014, pp 6261–6264. Watson D, Clark LA, Tellegen A. Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology 1988; 54: 1063–1070. Abend R, Burk D, Ruiz SG, Gold AL, Napoli JL, Britton JC et al. Computational modeling of threat learning reveals links with anxiety and neuroanatomy in humans. eLife 2022; 11: e66169. Mixed-Effects Models in S and S-PLUS . Springer-Verlag: New York, 2000 doi: 10.1007/b98882 . Schwarz G. Estimating the Dimension of a Model. The Annals of Statistics 1978; 6: 461–464. Kong E, Monje FJ, Hirsch J, Pollak DD. Learning not to Fear: Neural Correlates of Learned Safety. Neuropsychopharmacol 2014; 39: 515–527. De Kleine RA, Hutschemaekers MHM, Hendriks GJ, Kampman M, Papalini S, Van Minnen A et al. Impaired action-safety learning and excessive relief during avoidance in patients with anxiety disorders. J Anxiety Disord 2023; 96: 102698. Berridge KC, Robinson TE. Liking, Wanting and the Incentive-Sensitization Theory of Addiction. Am Psychol 2016; 71: 670–679. Suri RE. TD models of reward predictive responses in dopamine neurons. Neural Networks 2002; 15: 523–533. Moutoussis M, Bentall RP, Williams J, Dayan P. A temporal difference account of avoidance learning. Network: Computation in Neural Systems 2008; 19: 137–160. Additional Declarations The authors have declared there is NO conflict of interest to disclose Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6194403","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":436254245,"identity":"59fad66c-83ee-495b-9ab6-4b51b6095d46","order_by":0,"name":"silvia papalini","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABIElEQVRIie3RMUvDQBTA8RcCcZKsD8T2KyQUilA/zB2CWdoiFOwiUglcNl0jCn4CQZfT8eQgLsE5YIfr0skhLqKLeJegKB52dbg/BI4Hv7sHAXC5/mO+PwPSHj1BADv60DfzdqZsxPsiYEjvJyEWApp8JvRHZ6tIN/MYqgMYh2epEovbreQmk1w9swG9zESgLCSSmpACJjgvIkFLHJ2Xu9P4lCWUl2QtshHfSyMS6H0qognDUQ7D/sY6k5RXEKBtsdSQd6AXVVIbkmD49DcB6R0pyoBeVcPmFYK44pWoIcc4ieflniFxjstpnD8kPV6aGyyLnUhx9/qyPe48ZteLN3bYxXCHq3p/sMnvZYG1ZbE2fRv+nn77ZZasxOVyuVxNH2AqcA3Bu0GrAAAAAElFTkSuQmCC","orcid":"","institution":"Donders Institute, Centre for Cognitive Neuroimaging, CNS, RadboudUMC","correspondingAuthor":true,"prefix":"","firstName":"silvia","middleName":"","lastName":"papalini","suffix":""},{"id":436254246,"identity":"b3b4c0d8-ec77-4af1-afb6-16789b36610d","order_by":1,"name":"Esther Krul","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Esther","middleName":"","lastName":"Krul","suffix":""},{"id":436254247,"identity":"e8dcb5d7-bf77-45f0-b41a-98a91cd45a38","order_by":2,"name":"Tom Haber","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Tom","middleName":"","lastName":"Haber","suffix":""},{"id":436254248,"identity":"06ef80e1-9c5c-46e0-b672-08c7e178712c","order_by":3,"name":"Bram Vervliet","email":"","orcid":"https://orcid.org/0000-0002-5904-4257","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Bram","middleName":"","lastName":"Vervliet","suffix":""}],"badges":[],"createdAt":"2025-03-10 10:20:48","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6194403/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6194403/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":80046524,"identity":"22df2a6e-7ea0-4ac6-9c9f-fe9f71eee6d3","added_by":"auto","created_at":"2025-04-07 09:51:45","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":21656,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eshows the predicted backpropagation of the rPE from outcome (noUS) to cue onset.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6194403/v1/7f9c9c2cfee63ed8afdfbb0b.jpg"},{"id":80046520,"identity":"05d0f9f3-a0d4-4c99-a4b5-dcad514e18f2","added_by":"auto","created_at":"2025-04-07 09:51:44","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":76592,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eshows the trial design. Subjective ratings of relief were collected at CS presentations as well as during US omissions (during both Pavlovian and fear extinction learning).\u003c/em\u003e\u003c/p\u003e","description":"","filename":"Screenshot20250404233614.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6194403/v1/db657b7dd3adb345d1e1c2af.jpg"},{"id":80046536,"identity":"ffbe98db-5891-47db-a0d1-9ada8b2faa7c","added_by":"auto","created_at":"2025-04-07 09:51:45","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":97516,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eshows the three simulated prediction error profiles across trials for changing α and γ. At the offset, α changes the steepness of the learning whereas at the onset α influences both the steepness of ramp up and ramp down. The effect of γ can be seen at the onset, where the strength of the backpropagation is increased or decreased.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"Figure3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6194403/v1/aeefb05666f07c99597d5a0e.jpg"},{"id":80046521,"identity":"a507c826-0e23-4898-8065-800363a6a48c","added_by":"auto","created_at":"2025-04-07 09:51:45","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":86231,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eshows the retrospective US expectancy ratings for each CS type after Pavlovian fear conditioning (left panel) and after fear extinction learning (right panel).\u003c/em\u003e\u003c/p\u003e","description":"","filename":"Figure4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6194403/v1/98ecbfb09867b11ccdbd8f6a.jpg"},{"id":80046523,"identity":"acbeaef7-82d8-4356-a007-221d570b5f59","added_by":"auto","created_at":"2025-04-07 09:51:45","extension":"jpg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":91674,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003edisplays the marginal means of the TD model fitted to the CS+ onset and offset (left panel, in red) and to the CS- (average between CS1- and CS2-) onset and offset (right panel, in blue).\u003c/em\u003e\u003c/p\u003e","description":"","filename":"Figure5.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6194403/v1/e2acde1a29ad7ba20103a7bc.jpg"},{"id":82049235,"identity":"cd42173d-582f-4dca-b868-12520e71cf8c","added_by":"auto","created_at":"2025-05-06 09:51:03","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1191319,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6194403/v1/7fd26e2c-bf51-46c7-9c85-9e2064b18eec.pdf"}],"financialInterests":"The authors have declared there is \u003cb\u003eNO\u003c/b\u003e conflict of interest to disclose","formattedTitle":"A Temporal Difference Approach To Relief","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eReward Prediction Error (rPE) signals are computed when actual states of the world are better than expected\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. For example, after having introduced the exact amount of cash into a vending machine, a rPE signal is generated if we receive more snacks than we expected. Decades of research conducted across different species recognize the critical role played by such error-based signals in the generation of new reward-based learning and the connected approach behaviors \u003csup\u003e\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e. Such research established that rPEs are computed by the phasic responses of dopaminergic neurons in the mesolimbic/midbrain, which response\u0026rsquo;s magnitude directly depends on the error\u0026rsquo;s magnitude (or magnitude of the mismatch). These dopaminergic bursts are typically elevated in the early trials of a new experience and/or training. Then, with repeated exposures to the unexpected reward outcome, the accuracy of the predictions increase and the magnitude of these phasic bursts attenuate accordingly. When the obtainment of the reward becomes completely obvious, which typically emerges toward the end of a successful learning process, this dopaminergic-based rPE signal dissipates, giving rise to a new belief, and, likely, to a new prominent behavior (i.e. \u0026ldquo;I will always use this vending machine when hungry since it always delivers more snacks than I paid for\u0026rdquo;)\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. To date, however, it has become increasingly clear that rPE signaling does not only govern reward-based learning, but also some forms of safety learning, such as fear extinction learning\u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e,\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eDuring a typical fear extinction paradigm, a cue (or Conditioned Stimulus [CS], e.g. the color of a light) that was previously learned to predict the occurrence of an unpleasant outcome (or Unconditioned stimulus [US], e.g. an unpleasant electrical stimulation) is no longer followed by this unpleasant US. Computationally, the surprisingly violation of the negative CS\u0026mdash;US expectation is captured by the Rescorla-Wagner (RW) model\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e, a conceptual framework that uses rPEs to explain changes in conditional (fear) responding. rPEs from unpleasant US omissions, which represent better-than-expected outcomes, are thought to decrease such fearful response through the weakening of the association between a fear-conditioned CS and the unexpected absence of its outcome (US) or even by promoting the formation of a new safe memory of the CS that competes against the original threatening CS. Recent findings in rodents\u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e,\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e, indicate that such omission rPEs are decoded by the dopaminergic mesolimbic/midbrain system. Similar to the typical rPE signal observed in appetitive learning paradigms, the dopaminergic burst exerted during such omissions evolves as a classical PE signal: it is elevated at the maximum magnitude of the mismatch between the expected threat and the confrontation with its absence and, with repeated exposures to these threat omissions, minimizes and dissipates over the fear extinction training, together with the fear-elicited response\u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e,\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e. The same studies in rodents have also shown that blocking or accelerating the activity of these (midbrain) dopaminergic neurons at the exact time of the unexpected omission of the US, impairs or enhances extinction learning, respectively\u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e,\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e. These findings suggest that threat omissions are processed as unexpected rewards that drive the new learning of safety.\u003c/p\u003e \u003cp\u003eNew learning of safety is the basis of exposure-based treatments for exaggerated fears in anxiety disorders. Therefore, translational research aiming to support the treatment of excessive anxiety is showing an increased interest in understanding the relation between rPE and reduction of clinical fear\u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e. Yet, the advancement of this knowledge is hindered by the inability to investigate dopaminergic rPE signaling directly in humans, which would require highly invasive procedures to record the -in vivo- activity of dopaminergic neurons in deep nuclei of the human brain. Hence, there is a need for proximal indices that can indirectly track the development of rPE. One promising index is the subjective experience of relief, which we recently introduced in the fear conditioning procedure in humans.\u003c/p\u003e \u003cp\u003eRelief is the pleasant feeling of surprise that is generated when an expected threat stays away\u003csup\u003e\u003cspan additionalcitationids=\"CR11 CR12 CR13 CR14\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e. We previously added a relief rating scale after each CS presentation in fear extinction and observed that the level of relief is high during the first unexpected omissions of the US and progressively reduces over the course of the omissions, in line with the expected course of rPE signaling during fear extinction learning\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e,\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. This suggests that the course of relief during fear extinction could be used to probe the learning of safety, with a sharp decline in relief reflecting robust learning of safety. However, if relief is still relatively high at the end of fear extinction, this could indicate that little safety learning has been acquired.\u003c/p\u003e \u003cp\u003eHence, relief, together with the reduction in fear levels, is emerging as a potential index of fear extinction learning as well as a new potential target for assessing fear extinction-based sessions, such as those characterizing exposure therapy for clinical anxiety\u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e. However, to provide more guidance to translational and clinical research it is important to further investigate if the subjective relief ratings we introduced in our paradigms reliably represents a proxy (indirect) index of a rPE signal. This deeper understanding can be achieved by investigating whether relief ratings also capture other well recognized aspects of dopaminergic PE signals, such as their backpropagation over the course of learning.\u003c/p\u003e \u003cp\u003eBackpropagation of PEs is a cornerstone of the Temporal Difference learning (TD) model, an influential extension of the Rescorla-Wagner model\u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e,\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e. While the RW model specifies the change in associative strength based on the whole trial\u0026rsquo;s result, the TD model computes changes in this strength within the trial. This model can do so since it distinguishes times/moments within the trial and allows the calculation of within-trial PEs. What is peculiar in the TD algorithm is that, over the course of the trials, it uses these within-trial PEs to update the value of the prediction of the reward, from the moment of the obtainment of the reward back to the onset of the CS (or back to the earliest possible cue that reliably predicts the future reward). Unless the reward contingencies from trial to trial is subjected to change, learning is fully acquired when the value of the prediction of the reward for each timepoint of the presentation of the CS is equal to the total reward available in the trial. In other words, with a TD learning model we can measure predictions about a future reward that do not uniquely occur at the exact time of the omission of a threat (e.g. at CS offset, as the RW model assumes), but also predictions about reward that emerge during cue presentations (e.g. CS onset), or avoidance actions taken during CS presentation (in the case of active avoidance trials). This measurement is key since within the context of fear extinction a TD algorithm allows a large comprehensive investigation of the role of omission PE in safety learning and consequent adjustment of the connected behavior. For example, differently from the RW model, the TD model can be used for explaining high-order conditioning as it allows us to promptly predict the presence of a now safe cue/situation and to engage the behavioral system to reduce the previously gained distance from this originally threatening cue. Hence, the utility of a backpropagation of an omission PE would consist in helping adjusting the value and prediction of the originally threatening but now safe cue, helping the safety learning process. To date, however, no studies investigated the potential role played by such temporal dynamic of threat omission PE on fear extinction in either animals or humans. Nonetheless, if dopaminergic rPEs are dynamic signals that do not remain anchored to the time of reward outcome\u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e,\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e,\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e and subjective relief is a reliable indirect index of this dopaminergic PEs, then subjective relief ratings could also be used to provide information about such crucial temporal property.\u003c/p\u003e \u003cp\u003eHence, the aim of the present study was to test whether the decrease of relief that we previously observed at US omissions during fear extinction actually reflects a backpropagation from CS offsets to CS onsets, as predicted by TD, see prediction in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. To track the expected within-stimulus changes in relief, we added relief ratings at multiple time-points during CS presentations and we used computational modelling to estimate TD parameters. Additionally, we explored whether anxiety-related personality traits modulate this course in relief and computed TD parameters. Previously, we found that more anxious individuals report more relief at CS offsets (US omissions). Here, we explore whether this increased relief is paralleled by an absence of a temporal shift to CS onsets, which would reflect an inability to learn that the CS has become safe.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003e-Fig.\u0026nbsp;1-\u003c/b\u003e \u003c/p\u003e"},{"header":"2. Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Participants\u003c/h2\u003e \u003cp\u003eTo estimate the sample size, we used the data from our previous study on relief\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. The reference was the decrease of relief pleasantness over the blocks of the extinction phase (ɳ\u003csup\u003e2\u003c/sup\u003e\u003csub\u003ep\u003c/sub\u003e = 0.187). The GLMMPSS software\u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e provided a sample size of 38 subjects with alpha 0.05 and a power of 0.95. To make up for bad recording we enrolled fifty healthy participants. They were recruited from the local area (Leuven) via the Experiment Management System (ESM, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://psykuleuven.sona-systems.com/all_exp.aspx\u003c/span\u003e\u003cspan address=\"http://psykuleuven.sona-systems.com/all_exp.aspx\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e).\u003c/span\u003e The protocol was approved by the Social and Societal Ethics Committee of KU Leuven and each participant provided written consent to the participation of the study in agreement with the local Ethical Committee (EC). Fifty-one healthy participants (average age 20.6, 39F: females and 12M: males) performed the task.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Stimuli and Apparatus\u003c/h2\u003e \u003cp\u003eThe experiment took place in a laboratory room with dimmed light. All stimuli and questionnaires were presented on a computer screen using Affect 5 software\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e. Each trial started with an image of a desk lamp that lights up (after 4-5s) in one of three colors: red, blue, or yellow. These stimuli were taken from a previously validated task\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e. The three colors served as conditional stimuli (CSs, duration: starting after 3 sec. from the CS onset and continuing between 2 and 4 seconds after the relief rating). One color (CS+) was followed by the US, while the other two (CS-s) were not. The US (2ms electrical pulse) and US omissions were accompanied by an image of a lightning bolt and a lightning bolt with a cross through it, respectively (duration: 2s). The configurations of colors and CS types were pseudo-randomized between participants, such that all possible configurations were used with equal frequency.\u003c/p\u003e \u003cp\u003eThe unconditional stimulus was an uncomfortable, but not painful, electrical stimulation. The aversiveness of the US was individually calibrated via a work-up procedure. The participants rated gradually increasing pulse intensities until they reach a rating of \u0026ldquo;very uncomfortable but not painful\u0026rdquo;. The selected intensity of the electrical stimulation was used throughout the whole experiment without changes. A DS7 (Digitimer, Hertfordshire, UK) delivered the 2ms electrical pulse to the forearm of the dominant hand via two adjacent SensorMedics surface-electrodes with KY gel.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 The paradigm\u003c/h2\u003e \u003cp\u003e Before the actual task, participants were familiarized with the stimuli and relief ratings through a short tutorial. At the start of the Pavlovian fear conditioning phase, they were then instructed as follow: \u0026ldquo;Try to see if there is a pattern between the pictures and the shocks\u0026rdquo;. The Pavlovian phase contained eight trials for each CSs- and twelve trials for the CS+ (twenty-eight trials in total). Next, participants performed a fear extinction learning phase. At the beginning of each phase, participants were instructed that they might or might not receive the electrical stimulation, a procedure that typically reintroduces some initial uncertainty about the CS-. This phase contained five trials for each CS- and eight trials for the CS+ (eighteen trials in total), which were visualized within the same conditioning context. Crucially, no electrical stimulations were delivered. As general rule, no more than two trials contained the same color.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e\u003cem\u003e2.4 Self-reports\u003c/em\u003e\u003c/h2\u003e \u003cp\u003eAll participants completed three psychometric questionnaires at the start of the experiment. The scores at these questionnaires allowed us to explore the effect of individual differences in distress tolerance, anxiety traits, and positive and negative affect on relief and its temporal dynamics. The Distress Tolerance Scale (DTS) examined the participants\u0026rsquo; perceived ability to tolerate emotional distress\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e. The State-Trait Anxiety Inventory (STAI-T) was used to measured anxiety traits\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e, and the Positive and Negative Affect Schedule (PANAS) was used to measure positive and negative affect\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eDuring the Pavlovian fear conditioning and fear extinction learning phases, subjective relief intensity was measured at every CS presentation, and any time the US was not delivered (at CS offset). The measurement was both a binary (yes/no) and continuous (0-100) variable. Three seconds after each CS onset and US omission, participants received the question \u0026ldquo;Did this color make you feel relieved?\u0026rdquo; and \u0026ldquo;Did the absence of the shock make you feel relieved?\u0026rdquo;, respectively. The question appeared at the bottom of the screen along with two buttons \u0026ldquo;YES\u0026rdquo; and \u0026ldquo;NO\u0026rdquo; (forced-choice). After a \u0026ldquo;YES\u0026rdquo; response, participants rated the intensity of their relief on a VAS scale ranging from \u0026lsquo;not relieved\u0026rsquo; (0) to \u0026lsquo;very relieved\u0026rsquo; (100), Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eAfter each learning phase, the participants indicated their US expectations for the first and last presentation of each CS on a 7-point Likert scale.\u003c/p\u003e \u003cp\u003e \u003cb\u003e-Fig.\u0026nbsp;2-\u003c/b\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e2.5 Procedures\u003c/h2\u003e \u003cp\u003eUpon arrival, participants were informed about the nature of the experiment (understanding emotional learning), followed by the judgment of the exclusion criteria and the informed consent procedure. Exclusion criteria were: pregnancy, cardiovascular and pulmonary diseases, neurological and/or psychiatric disorders, or any other serious medical condition, presence of an electronic implant, pain at hands or wrists, and a doctor\u0026rsquo;s request to avoid stressful situations. Next, participants completed three psychological questionnaires and were prompted to take a moment to think about the emotion of relief. Then, the electrodes for the electrical stimulations were attached. The intensity of the electrical stimulation was calibrated to match a subjective rating of \u0026lsquo;very uncomfortable but not painful\u0026rsquo;. The main task (duration 20\u0026ndash;25 minutes) consisted of a Pavlovian fear conditioning and an fear extinction learning phase, modified from the previously validated paradigm of Vervliet et al. (2017). Finally, participants completed an adverse events form and debriefing was provided at the end of the session.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e2.6 The Temporal Difference Model\u003c/h2\u003e \u003cp\u003eTo model the backpropagation of the PE during Pavlovian fear conditioning and fear extinction learning, we fit a Temporal Difference Learning Model to the subjective relief ratings for each participant. The algorithm is defined as:\u003c/p\u003e \u003cp\u003e \u003cem\u003eV(s\u003c/em\u003e \u003csub\u003e \u003cem\u003et\u003c/em\u003e \u003c/sub\u003e \u003cem\u003e) \u0026larr; V(s\u003c/em\u003e \u003csub\u003e \u003cem\u003et\u003c/em\u003e \u003c/sub\u003e \u003cem\u003e) + α[r\u003c/em\u003e \u003csub\u003e \u003cem\u003et + 1\u003c/em\u003e \u003c/sub\u003e\u0026thinsp;\u003cem\u003e+\u0026thinsp;γV(s\u003c/em\u003e\u003csub\u003e\u003cem\u003et + 1\u003c/em\u003e\u003c/sub\u003e\u003cem\u003e) \u0026ndash; V(s\u003c/em\u003e\u003csub\u003e\u003cem\u003et\u003c/em\u003e\u003c/sub\u003e\u003cem\u003e)]\u003c/em\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cem\u003eV(s\u003c/em\u003e\u003csub\u003e\u003cem\u003et\u003c/em\u003e\u003c/sub\u003e\u003cem\u003e)\u003c/em\u003e is the expected reward (US omission), one at the CS onset and one for the CS offset. This initial value is updated in each subsequent trial, where \u003cem\u003eV(st)\u003c/em\u003e is the expected reward at the CS on/offset, \u003cem\u003eα\u003c/em\u003e is the learning rate, \u003cem\u003er\u003c/em\u003e is the obtained reward at moment t\u0026thinsp;+\u0026thinsp;1, and \u003cem\u003eγ\u003c/em\u003e is the discount rate (gamma) of the value of future expected rewards. When the γ value approaches zero the backpropagation is weak whereas a value close to one indicates the presence of a strong temporal backpropagation of the PE (see Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e for visual representation of a simulated subject).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003e-Fig.\u0026nbsp;3-\u003c/b\u003e \u003c/p\u003e \u003cp\u003eIn a final step, the score obtained from each psychological questionnaire was then added as a covariate to the TD model to investigate whether the discount parameter was different between participants with different levels of trait anxiety, distress tolerance, positive and negative affect, separately. To obtain such indication, we specifically looked at the covariate-parameter relationship (between gamma and the score at the questionnaires).\u003c/p\u003e \u003cp\u003eOne participant was excluded from all the analyses since they did not engage in the task (i.e., only providing relief ratings twice at CS presentation and never reporting relief at US). Hence, the final sample consisted of fifty participants.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e2.7 TD parameters setting\u003c/h2\u003e \u003cp\u003eWe fit the TD model with the subjective relief intensity rating (rescaling the simulated PEs to [0,100] to have the same range as for the relief ratings) experienced at the offset and onset of the CS\u0026thinsp;+\u0026thinsp;and at the offset and onset of the CS-. Since not all participants exhibited the expected learning, as indicated by a decrease in relief ratings at CS offset, we excluded such participants by fitting a linear model to the relief ratings at CS offset and constraining the slope (learning rate) to be smaller than \u0026minus;\u0026thinsp;1. This is similar to the strategy used by\u003csup\u003e\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e, but we used a more stringent cutoff that only allowed learners (including small learning) to be included. This approach led to excluding 10 participants for CS\u0026thinsp;+\u0026thinsp;and 10 participants for CS-.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e2.8 Statistical analysis\u003c/h2\u003e \u003cp\u003eA mixed effects model was used on retrospective US expectancy ratings to examine Pavlovian fear conditioning. The model included the fixed term Time (\u0026lsquo;First\u0026rsquo; and \u0026lsquo;Last\u0026rsquo; presentation of the CS) and CS (\u0026lsquo;CS+\u0026rsquo; and \u0026lsquo;CS-\u0026rsquo;). A similar model was applied to the retrospective US expectancy ratings collected after fear extinction learning. It is worth to mention that Skin Conductance Levels were measured but not used given the suboptimal design for this measurement.\u003c/p\u003e \u003cp\u003eFor the statistical analysis related to relief ratings obtained from the CS onset and CS offset, we used a non-linear mixed effects model (nlme) using \u003cem\u003enlme\u003c/em\u003e in \u003cem\u003er\u003c/em\u003e studio\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e the CS\u0026thinsp;+\u0026thinsp;and the CS- separately. Alpha (α: learning rate), Gamma (γ: discount value), the initial reward expectancy at the onset of the extinction phase and the initial reward expectancy at the CS offset were included as fixed terms. The selection of the random components added to this model was performed by using the Bayesian Information Criterion (BIC)\u003csup\u003e\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e, which provides a trade-off between model fit and model complexity. This resulted in two random components, one for alpha and one for gamma. Subsequently, the standardized scores from the psychometric questioners were added to the model to investigate for a potential fixed effect of each covariate on gamma and alpha parameters.\u003c/p\u003e \u003cp\u003eDue to the non-linear nature of the model, the fitting procedure requires initial starting points for all parameters. For the initial general expectation of reward (omission of the US) at the beginning of the extinction phase, we used an adjusted value of the US-expectations from the end of Pavlovian conditioning phase of 0.5. The expectation to receive a reward at the CS\u0026thinsp;+\u0026thinsp;offset was set to 0.2 since this CS was previously conditioned to the US delivery while it was set to 0.8 for the CS- offset. We decided to maintain such a degree of uncertainty (adjusted value) in these expectations given the start of a new phase (Pavlovian \u0026loz; Extinction) and the instructions we provide at the beginning of extinction (as by protocol: \u0026rdquo;also during this phase you might or might not receive a shock\u0026rdquo;) typically reintroduce some degree of uncertainty about the CS \u0026loz; US contingency. However, the results of the model fitting did not seem particularly sensitive to the starting values. The initial learning rate (alpha) and discount rate (gamma) were set to 0.7 and 0.9 respectively.\u003c/p\u003e \u003cp\u003eAnonymized datasets and scripts for the statistical analysis can be found in \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/silviapapalini/td\u003c/span\u003e\u003cspan address=\"https://github.com/silviapapalini/td\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Results","content":"\u003cp\u003eBefore proceeding with the TD model, we first ensure the presence of conditioning in both the learning phases (manipulation checks).\u003c/p\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Retrospective expectancy ratings\u003c/h2\u003e \u003cdiv id=\"Sec13\" class=\"Section3\"\u003e \u003ch2\u003e3.1.1 Pavlovian conditioning\u003c/h2\u003e \u003cp\u003eThe results from the linear mixed effects model showed a decrease in US expectancy from the first to the last presentation of the CS1- when the interaction between Time and CS\u0026thinsp;+\u0026thinsp;was added to the model. The same expected decrease was observed for the CS2- (Table \u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e), Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e left panel. Results from three post hoc (Bonferroni Corrected) comparisons run for each CS confirmed the expected reduction in US expectancy from the first to the last CS1- (Est-ꞵ = -2.53 SE\u0026thinsp;=\u0026thinsp;0.28, t-value = -9.02, \u003cem\u003ep\u003c/em\u003e-value\u0026thinsp;\u0026lt;\u0026thinsp;0.001), a similar reduction for the CS2- (Est-ꞵ = -3.02, SE\u0026thinsp;=\u0026thinsp;0.27, t-value = -11.03, \u003cem\u003ep\u003c/em\u003e-value\u0026thinsp;\u0026lt;\u0026thinsp;0.001), and an increase in US expectancy from the first to the last CS+ (Est-ꞵ = 1.49, SE\u0026thinsp;=\u0026thinsp;0.20, t-value\u0026thinsp;=\u0026thinsp;7.30, \u003cem\u003ep\u003c/em\u003e-value\u0026thinsp;\u0026lt;\u0026thinsp;0.001).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eshows the results from the linear mixed effects model applied to the retrospective US expectancy ratings of the Pavlovian fear conditioning and fear extinction learning phase\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"11\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c11\" colnum=\"11\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePavlovian\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eExtinction\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c10\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c11\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eFixed Effects (LMM)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eEst\u003c/b\u003e\u003cb\u003e-\u003c/b\u003eꞵ\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eSE\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003eDF\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003et-value\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003ep\u003c/b\u003e\u003cb\u003e-value\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003eEst\u003c/b\u003e\u003cb\u003e-\u003c/b\u003eꞵ\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003eSE\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003eDF\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003et-value\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e\u003cb\u003ep\u003c/b\u003e\u003cb\u003e-value\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTime(last)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1.49\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e5.97\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-3.65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e-12.41\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCS(CS1-)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-1.49\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-5.97\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-1.59\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e-5.41\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCS(CS2-)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-0.88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-3.52\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-1.65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e-5.62\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTime(last) x CS(CS1-)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-4.02\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-11.39\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1.41\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.42\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e3.38\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTime(last) x CS(CS2-)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-4.51\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-12.77\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1.53\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.42\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e3.68\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"11\"\u003eReference levels: \u003cem\u003eTime(first presentation) and CS(CS+)\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"11\"\u003e\u003cem\u003eSE: Standard Error; DF: Degrees of Freedom\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section3\"\u003e \u003ch2\u003e3.1.2 Fear extinction\u003c/h2\u003e \u003cp\u003eThe results from the linear mixed effects model showed a decrease in US expectancy from the first to the last presentation of the CS\u0026thinsp;+\u0026thinsp;when the factor Time was added to the model. The same expected decrease was observed for the CS1- and CS2- (Table \u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e), since the new extinction phase probably reintroduced some uncertainty about the safety of the first CSs-, see Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e right panel. Results from three post hoc (Bonferroni Corrected) comparisons run for each CS confirmed the expected stronger reduction in US expectancy from the first to the last CS+ (Est-ꞵ = -3.65, SE\u0026thinsp;=\u0026thinsp;0.31, t-value = -11.69, \u003cem\u003ep\u003c/em\u003e-value\u0026thinsp;\u0026lt;\u0026thinsp;0.001), from the first to the last CS1- (Est-ꞵ = -2.25, SE\u0026thinsp;=\u0026thinsp;0.32, t-value = -6.96, \u003cem\u003ep\u003c/em\u003e-value\u0026thinsp;\u0026lt;\u0026thinsp;0.001), and from the first to the last CS2- (Est-ꞵ = -2.12, SE\u0026thinsp;=\u0026thinsp;0.34, t-value = -6.29, \u003cem\u003ep\u003c/em\u003e-value\u0026thinsp;\u0026lt;\u0026thinsp;0.001).\u003c/p\u003e \u003cp\u003e \u003cb\u003e-Table\u0026nbsp;1-\u003c/b\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003e-Fig.\u0026nbsp;4-\u003c/b\u003e \u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e3.2 TD approach to (continuous) relief ratings during fear extinction:\u003c/h2\u003e \u003cp\u003eRelief levels from each participant in response to CS onsets and CS offsets were fit with the TD model separately for each CS type. For the CS+ (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, left panel), the model provided a good fit for 40 participants. The results from the non-linear mixed effects model showed significant main effects of the learning rate and discount factors, the former indicating successful learning, and the latter indicating the presence of a temporal backpropagation. The expectation (Vs) of receiving a reward (US omission) was not significant at the initial CS onset but it was significant at the first offset (Table \u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Adding the scores obtained from the psychometric questionnaire to this model did not result in any significant interaction between the covariate and Alpha or Gamma (all \u003cem\u003ep\u003c/em\u003e-values above 0.05).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eshows the results from the nonlinear mixed effects model when the TD algorithm was applied to the CS+.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e \u003cp\u003eTD model for the CS+\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eFixed effects (NLME)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eEst\u003c/b\u003e\u003cb\u003e-\u003c/b\u003eꞵ\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eSE\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003eDF\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003et-value\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003ep\u003c/b\u003e\u003cb\u003e-value\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eα\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.30\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e597\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e7.25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eγ\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.81\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e597\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e6.58\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOnset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e597\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.296\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOffset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.02\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e597\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e6.98\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRandom effects (NLME)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eSD α\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eSD γ\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.67\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eResidual\u003c/b\u003e: 17.147\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eBIC\u003c/b\u003e: 5731.414\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"6\"\u003eReference levels: \u003cem\u003eOnset (Initial) and Offset (Initial). SE: Standard Error; DF: Degrees of Freedom, SD: Standard Deviation\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003e-Table\u0026nbsp;2-\u003c/b\u003e \u003c/p\u003e \u003cp\u003eFor the CS- (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, right panel), the model provided a good fit for 40 participants. The results from the non-linear mixed effects model showed the presence of learning (significant main effect of alpha) in absence of a significant discount value (main effect of gamma). The expectations of receiving a reward (US omission) at initial onset and at the initial offset were also significant (Table \u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Adding the scores obtained from the psychometric questionnaire to this model did not result in any significant interaction between the covariate and Alpha or Gamma (all \u003cem\u003ep\u003c/em\u003e-values above 0.05).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eshows the results from the nonlinear mixed effects model when the TD algorithm was applied to the CS- (average relief ratings between the CS1- and the CS2-)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"5\" nameend=\"c5\" namest=\"c1\"\u003e \u003cp\u003eTD model for the CS- (averaged relief between the CS1- and the CS2-)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eFixed effects (NLME)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eEst\u003c/b\u003e\u003cb\u003e-\u003c/b\u003eꞵ\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eSE\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003eDF\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003et-value\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003ep\u003c/b\u003e\u003cb\u003e-value\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eα\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e357\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e5.21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eγ\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.09\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e357\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-0.02\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.987\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOnset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-0.37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e357\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-9.72\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOffset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e357\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e12.80\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRandom Effects (NLME)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eSD α\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eSD γ\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eResidual\u003c/b\u003e: 21.031\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eBIC\u003c/b\u003e: 3702.37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"6\" nameend=\"c6\" namest=\"c1\"\u003e \u003cp\u003eReference levels: \u003cem\u003eOnset (Initial) and Offset (Initial). SE: Standard Error; DF: Degrees of Freedom, SD: Standard Deviation\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003e-Table\u0026nbsp;3-\u003c/b\u003e \u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003e-Fig.\u0026nbsp;5-\u003c/b\u003e \u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"4. Discussion","content":"\u003cp\u003eThis experiment was designed to investigate whether relief from threat omission during fear extinction in humans follows the typical backpropagation of reward prediction error during appetitive learning. To do so, this study utilized individual relief ratings obtained within and at the end of each trial of a fear extinction learning task. Specifically, we investigated whether relief backpropagates from threat omission to CS onset as extinction learning develops. We collected individual relief ratings at CS onset and CS offset (US omission), and we used these ratings to simulate learning-related parameters within a temporal difference model.\u003c/p\u003e \u003cp\u003e We found that the TD algorithm produced a reasonable fit for about 80% of the participants. The subjective relief experienced at the time of the unexpected omission of the US backpropagated to the onset of the previously threat-conditioned CS+. The magnitude of this backpropagation was stimulus-dependent, as the discount factor (gamma) approached zero for the averaged safe cues (CS-) and one for the extinguished CS+. Regarding the learning rate (alpha) of the TD model, we found that alpha approached a value of 0.3 for the CS\u0026thinsp;+\u0026thinsp;and 0.2 for the CS-. This slight difference was expected since the CS- had already been learned as a safe cue during the previous Pavlovian phase.\u003c/p\u003e \u003cp\u003eThese results indicate that fear extinction is not exclusively governed by reward prediction error (rPE) and learning rate, as assumed by the Rescorla-Wagner (RW) model, but also by the gamma parameter, which represents the discount value that determines the importance of future rewards. The fact that gamma was high for the threatening stimulus under extinction (CS+) suggests that healthy participants adjusted their expectations and values in relation to the anticipated future omissions of the unconditioned stimulus (US). In contrast, the absence of a significant discount factor for the conditioned stimulus (CS-) may indicate that expected rewards do not significantly alter the agent's perception on the value of future (obvious) rewards associated with conditioned safety, especially since the safe cue was also linked to a lower learning rate. It is important to briefly mention that for this type of CS (CS-), we also typically observe some initial relief at CS offset since the experimental task we generally use consists of clearly separated learning phases. Essentially, the initiation of a new -extinction- learning phase after a Pavlovian fear conditioning phase reintroduces in the participant some initial uncertainty about the safety of the CS-. However, during the training, the relief for the CS- offset is usually lower and dissipates more rapidly than the relief linked to the threatening CS\u0026thinsp;+\u0026thinsp;offset\u003csup\u003e16\u003c/sup\u003e. For this reason, this relief is typically used as reference for comparison purposes, but nevertheless worthy investigating in separation from the CS+, as individuals with anxiety disorders seem to also be characterized by difficulties in differential learning and safety learning\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e,\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e, which primarily involve the CS-.\u003c/p\u003e \u003cp\u003eThe results of the present study did not indicate a significant link between the discount parameter and psychological traits relevant to anxiety/stress-related disorders such as state/trait anxiety, distress tolerance, and negative/positive affect. It is, however, important to note that our sample included only healthy participants. Future studies could therefore investigate whether slower backpropagation of omission PEs during fear extinction is associated with the presence of fear extinction deficits in clinical samples. It is indeed plausible to assume that heightened anxiety about future uncertainties could lead subjects to devalue long-term rewards and experience elevated relief even in a completely safe condition.\u003c/p\u003e \u003cp\u003eAlthough this study focuses on fear extinction learning, these findings also have potential implications for our theory on relief and persistent avoidance\u003csup\u003e\u003cspan additionalcitationids=\"CR11 CR12 CR13\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e,\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e,\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e. The theory assumes that the pleasant emotion of relief arising from the active omission of a real threat acts as a reinforcer for avoidance behavior. Specifically, a higher feeling of relief during avoidance learning is thought to increase the likelihood of persisting in avoidance even after fear levels are reduced through extinction procedures\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. It is important to clarify that we are referring here to the avoidance behaviors that still follow fear extinction. During original cue-avoidance conditioning, the prediction error is indeed dependent on the learned (effective) avoidance action of the subject (i.e., an avoidance action that abolishes the occurrence of an imminent real threat), while the threatening connotation or nature of the cue remains intact. It is important to note that a high gamma in approach learning promotes exploration, maximizing survival and adaptation across species. Indeed, an earlier phasic excitation can timely orient the attentive and behavioral system toward the earliest cues that predict the acquisition of the valued reward\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e,\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e, achieving better long-term outcomes even at the cost of missing short-term gains\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e,\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e. In the context of fear extinction, therefore, a high gamma might facilitate not only adjustments to predictions related to the threatening cue but also activate the motivational system, reducing the tendency to avoid the now-safe cue and encouraging long-term reward accumulation (e.g. promoting approach behaviors), thereby deepening the process of fear extinction.\u003c/p\u003e \u003cp\u003eTo our knowledge, a temporal difference model has only been successfully applied within the context of conditioned avoidance in rodents\u003csup\u003e\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e\u003c/sup\u003e, providing initial evidence of a strong similarity between PE backpropagation in reward-based learning and PE backpropagation in avoidance learning. Thus, it is appealing to consider the idea that the same backpropagation might also occur during avoidance learning in humans. Since this experiment did not include an instrumental component, a future step would be to investigate whether the backpropagation of relief during fear extinction is linked to the avoidance behaviors adopted toward the extinguished cue. If such a link exists, it would be interesting to explore whether individual variations in this backpropagation of relief correlate with excessive or diminished avoidance during a test for a return or renewal of avoidance behavior. It seems reasonable to assume that the absence of backpropagation in relief for the CS\u0026thinsp;+\u0026thinsp;may reflect a poor transfer of predictive error utility in updating previously established avoidance behaviors, ultimately leading to persistent avoidance.\u003c/p\u003e \u003cp\u003eThe present study also has limitations. First, given the exploratory nature of our experiment, it is important for future studies to replicate these preliminary findings in larger samples. Second, the presentation of the relief ratings three seconds after CS onsets and CS offsets made it impossible to reliably investigate backpropagation in the skin conductance responses, which could have added value to the present experiment. Future replications of this study could adapt the temporal presentation of CSs and ratings to allow for (neuro)physiological recording. This would also permit investigation into whether variations in CS duration (uncertainty/distance from the reward) affect the gamma parameter. In our experiment, the duration of the CS presentation after the relief rating only varied between 2 and 4 seconds and was thus not included as a covariate in our TD models.\u003c/p\u003e \u003cp\u003eIn summary, these results extend previous knowledge regarding relief from omission PE by showing that relief travels back from reward delivery to the onset of the CS that reliably predicted threat omission, similar to classical rPE within the context of appetitive learning tasks. Moving forward, such a relief-based TD approach could be used to explain conditioned reinforcement within the context of avoidance behaviors by adding relief ratings (and physiological or biological markers of relief) to classical avoidance paradigms.\u003c/p\u003e"},{"header":"5. Conclusions","content":"\u003cp\u003eA TD-based approach to relief during fear extinction learning contributes to explaining within-trial effects of the temporal relationship between cues and outcomes during fear extinction learning.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003eConflict of interest\u003c/h2\u003e \u003cp\u003eThe authors declare no competing financial interests.\u003c/p\u003e \u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eSchultz W. Predictive reward signal of dopamine neurons. J Neurophysiol 1998; 80: 1\u0026ndash;27.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchultz W. Updating dopamine reward signals. Current Opinion in Neurobiology 2013; 23: 229\u0026ndash;238.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 2006; 442: 1042\u0026ndash;1045.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKalisch R, Gerlicher AMV, Duvarci S. A Dopaminergic Basis for Fear Extinction. Trends Cogn Sci 2019; 23: 274\u0026ndash;277.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePapalini S, Beckers T, Vervliet B. Dopamine: from prediction error to psychotherapy. Transl Psychiatry 2020; 10: 1\u0026ndash;13.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRescorla R, Wagner A. A theory of Pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement. Classical Conditioning: Current Research and Theory 1972.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLuo R, Uematsu A, Weitemier A, Aquili L, Koivumaa J, McHugh TJ \u003cem\u003eet al.\u003c/em\u003e A dopaminergic switch for fear to safety transitions. Nat Commun 2018; 9: 2483.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSalinas-Hern\u0026aacute;ndez XI, Vogel P, Betz S, Kalisch R, Sigurdsson T, Duvarci S. Dopamine neurons drive fear extinction learning by signaling the omission of expected aversive outcomes. Elife 2018; 7: e38818.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCraske MG, Treanor M, Conway C, Zbozinek T, Vervliet B. Maximizing Exposure Therapy: An Inhibitory Learning Approach. Behav Res Ther 2014; 58: 10\u0026ndash;23.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeng L, Beckers T, Vervliet B. No joy - why bother? Higher anhedonia relates to reduced pleasure from and motivation for threat avoidance. Behav Res Ther 2022; 159: 104227.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeng L, Beckers T, Vervliet B. Anhedonia influences threat avoidance and relief: A conceptual replication. Journal of Mood \u0026amp; Anxiety Disorders 2024; 5: 100050.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeng L, Beckers T, Vervliet B. What a relief! The pleasure of threat avoidance. Emotion 2024; 24: 539\u0026ndash;550.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePapalini S, Ashoori M, Zaman J, Beckers T, Vervliet B. The role of context in persistent avoidance and the predictive value of relief. Behav Res Ther 2021; 138: 103816.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSan Mart\u0026iacute;n C, Jacobs B, Vervliet B. Further characterization of relief dynamics in the conditioning and generalization of avoidance: Effects of distress tolerance and intolerance of uncertainty. Behaviour Research and Therapy 2020; 124: 103526.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWillems AL, Oudenhove LV, Vervliet B. Omissions of Threat Trigger Subjective Relief and Prediction Error-Like Signaling in the Human Reward and Salience Systems. \u003cem\u003eeLife\u003c/em\u003e 2024; 12. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.7554/eLife.91400.3\u003c/span\u003e\u003cspan address=\"10.7554/eLife.91400.3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVervliet B, Lange I, Milad MR. Temporal dynamics of relief in avoidance conditioning and fear extinction: Experimental validation and clinical relevance. Behav Res Ther 2017; 96: 66\u0026ndash;78.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSutton RS. Learning to predict by the methods of temporal differences. Mach Learn 1988; 3: 9\u0026ndash;44.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSutton RS, Barto AG. Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review 1981; 88: 135\u0026ndash;170.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFarrell K, Lak A, Saleem AB. Midbrain dopamine neurons signal phasic and ramping reward prediction error during goal-directed navigation. Cell Rep 2022; 41: 111470.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eO\u0026rsquo;Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ. Temporal Difference Models and Reward-Related Learning in the Human Brain. Neuron 2003; 38: 329\u0026ndash;337.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKreidler SM, Muller KE, Grunwald GK, Ringham BM, Coker-Dukowitz ZT, Sakhadeo UR \u003cem\u003eet al.\u003c/em\u003e GLIMMPSE: Online Power Computation for Linear Models with and without a Baseline Covariate. J Stat Softw 2013; 54: i10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSpruyt A, Clarysse J, Vansteenwegen D, Baeyens F, Hermans D. Affect 4.0: a free software package for implementing psychological and psychophysiological experiments. Exp Psychol 2010; 57: 36\u0026ndash;45.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMilad MR, Orr SP, Pitman RK, Rauch SL. Context modulation of memory for fear extinction in humans. Psychophysiology 2005; 42: 456\u0026ndash;464.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSimons JS, Gaher RM. The Distress Tolerance Scale: Development and validation of a self-report measure. Motivation and Emotion 2005; 29: 83\u0026ndash;102.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSkapinakis P. Spielberger State-Trait Anxiety Inventory. In: Michalos AC (ed). \u003cem\u003eEncyclopedia of Quality of Life and Well-Being Research\u003c/em\u003e. Springer Netherlands: Dordrecht, 2014, pp 6261\u0026ndash;6264.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWatson D, Clark LA, Tellegen A. Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology 1988; 54: 1063\u0026ndash;1070.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbend R, Burk D, Ruiz SG, Gold AL, Napoli JL, Britton JC \u003cem\u003eet al.\u003c/em\u003e Computational modeling of threat learning reveals links with anxiety and neuroanatomy in humans. eLife 2022; 11: e66169.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u003cem\u003eMixed-Effects Models in S and S-PLUS\u003c/em\u003e. Springer-Verlag: New York, 2000 doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/b98882\u003c/span\u003e\u003cspan address=\"10.1007/b98882\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchwarz G. Estimating the Dimension of a Model. The Annals of Statistics 1978; 6: 461\u0026ndash;464.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKong E, Monje FJ, Hirsch J, Pollak DD. Learning not to Fear: Neural Correlates of Learned Safety. Neuropsychopharmacol 2014; 39: 515\u0026ndash;527.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDe Kleine RA, Hutschemaekers MHM, Hendriks GJ, Kampman M, Papalini S, Van Minnen A \u003cem\u003eet al.\u003c/em\u003e Impaired action-safety learning and excessive relief during avoidance in patients with anxiety disorders. J Anxiety Disord 2023; 96: 102698.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBerridge KC, Robinson TE. Liking, Wanting and the Incentive-Sensitization Theory of Addiction. Am Psychol 2016; 71: 670\u0026ndash;679.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSuri RE. TD models of reward predictive responses in dopamine neurons. Neural Networks 2002; 15: 523\u0026ndash;533.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMoutoussis M, Bentall RP, Williams J, Dayan P. A temporal difference account of avoidance learning. Network: Computation in Neural Systems 2008; 19: 137\u0026ndash;160.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6194403/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6194403/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSubjective relief from threat omission is emerging as a proxy index of reward Prediction Error signaling during safety learning. Yet, the relation between relief and prediction error signaling remains poorly understood, limiting translational research. Here, we complemented our previous research in this field by providing further evidence of similarities between the emotion of relief and prediction error signaling. To this end, we enrolled fifty-one healthy participants, and applied a Temporal Difference Learning approach to subjective relief ratings collected during a classical fear extinction learning paradigm. If relief is a reliable index of reward Prediction Error signal then it should display the classical backpropagation from unexpected reward delivery (unexpected threat omission) to conditioned cue presentation, as the large literature in reward learning clearly demonstrated across species. We found that a TD model largely fits subjective relief ratings. Future studies could thereby use this TD model to understand if and how relief-PE drive fear extinction and its potential deficit.\u003c/p\u003e","manuscriptTitle":"A Temporal Difference Approach To Relief","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-07 09:51:40","doi":"10.21203/rs.3.rs-6194403/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ccac514a-2134-4a68-b8d8-31228c4f3471","owner":[],"postedDate":"April 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":46446782,"name":"Biological sciences/Neuroscience"},{"id":46446783,"name":"Biological sciences/Psychology/Human behaviour"}],"tags":[],"updatedAt":"2025-05-06T09:42:56+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-07 09:51:40","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6194403","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6194403","identity":"rs-6194403","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}