Reliability-Oriented Evaluation of Explanation Signals under Real and Synthetic Stress Tests

Reliability-Oriented Evaluation of Explanation Signals under Real and Synthetic Stress Tests | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Reliability-Oriented Evaluation of Explanation Signals under Real and Synthetic Stress Tests MOHAMMED ALSHANTI This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8554983/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This paper presents a reliability-oriented empirical study of global importance signals in machine learning models, focusing on stability, sensitivity, and cross-model consistency rather than proposing a new explanation algorithm. (i) Across two public real-world datasets and controlled synthetic stress tests, we show that explanation reliability can degrade even when predictive performance remains strong (often high); (ii) we provide empirical evidence that stability, sensitivity, and cross-model agreement capture complementary failure modes overlooked by accuracy-centric evaluation; and (iii) we discuss implications for reliability-aware XAI benchmarking under spurious correlations and retraining variability. These findings treat explanation reliability as a primary empirical property in our evaluated setting and offer practical guidance under the considered regimes for building and using future XAI benchmarks. Physical sciences/Mathematics and computing Biological sciences/Psychology Social science/Psychology Explainable AI reliability auditing stability sensitivity consistency Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Introduction Recent studies have shown that explanations may be unstable, overly sensitive to perturbations, or inconsistent across models, even when predictive accuracy is high [ 1 ], [ 2 ], [ 3 ]. In many high-stakes domains, such as healthcare, finance, and critical infrastructure, this mismatch between apparently strong performance and fragile explanations raises concrete governance concerns: auditors, domain experts, and regulators are increasingly asked to rely on explanation artifacts when validating models, yet the reliability of those artifacts is rarely quantified or stress-tested in a systematic way [ 4 ], [ 5 ]. A large body of work in explainable AI (XAI) has therefore shifted attention from proposing ever more explanation methods to examining whether existing methods behave in a trustworthy and predictable manner under realistic conditions [ 6 ], [ 7 ], [ 8 ], [ 9 ]. Early XAI research predominantly focused on interpretability mechanisms such as feature attribution, saliency maps, and surrogate models, often evaluated on a small number of benchmarks or illustrative case studies [ 10 ], [ 11 ]. More recent surveys and taxonomies highlight that explanation quality is multi-dimensional, spanning aspects such as robustness, faithfulness, stability, and human-centered usefulness [ 7 ], [ 8 ], [ 5 ]. Within this landscape, robustness-oriented studies have demonstrated that explanations can change markedly across retraining runs, small input perturbations, or shifts in the underlying data distribution, even when predictive metrics such as ROC-AUC or accuracy remain nearly constant [ 1 ], [ 2 ]–[ 3 ], [ 12 ]. These findings suggest that accuracy-centric evaluation is insufficient: without explicit checks on explanation behavior, practitioners may unknowingly deploy models whose explanations are unreliable, misleading, or overly dependent on spurious correlations. Despite this progress, existing empirical work on explanation reliability tends to examine individual dimensions in isolation—for example, stability under random seeds, sensitivity to input noise, or agreement between a specific explainer and a chosen reference method—often under narrow experimental conditions and without a unified protocol [ 7 ], [ 9 ], [ 5 ]. Benchmark efforts have begun to standardize datasets and tasks for XAI evaluation, but they frequently prioritize aggregate explanation scores or faithfulness metrics over reliability under stress, and they rarely provide operational decision rules for when an explanation should be considered too unstable or inconsistent to act upon [ 13 ]–[ 8 ]. As a result, there is still limited guidance on how to operationalize explanation reliability as a first-class empirical property that can be measured, stress-tested, and integrated into auditing workflows. This paper contributes to that gap by shifting focus from method-centric novelty toward systematic empirical assessment of explanation behavior under both real and synthetic stress regimes. We treat explanation reliability as a pre-use screening problem: the practical question is not which explainer is newest or most sophisticated, but whether the available explanation signal is stable enough, insensitive enough to controlled perturbations, and sufficiently consistent across model families to support auditing or decision support in a given deployment context [ 4 ], [ 5 ], [ 1 ]. To make this perspective concrete, we combine two public tabular benchmarks with controlled synthetic generators that inject spurious correlations and interaction-dominant proxies, allowing us to disentangle reliability phenomena from purely performance-driven effects. We keep the scope deliberately narrow: tabular benchmarks and global, model-specific importance. That is not a limitation by accident. It helps us separate explanation reliability from modality-specific preprocessing and from the extra randomness of local post-hoc explainers. Within this setting, we introduce and analyze three complementary reliability lenses—stability across retraining seeds, relative sensitivity (rESS) under perturbation and noise, and cross-model consistency via Top-k overlap—and study how they behave jointly across real and synthetic regimes [ 1 ], [ 2 ]–[ 3 ], [ 9 ]–[ 12 ]. Building on these signals, we derive a simple threshold-based “reliability gate” that can be plugged into XAI benchmarks and auditing pipelines to decide when not to act on an explanation: explanations that fail minimal stability, sensitivity, or cross-model agreement criteria are screened out before they influence an auditor’s judgment. The remainder of the paper formalizes these components, reports empirical findings across datasets, and discusses implications for reliability-aware XAI benchmarking and governance. The primary contributions of this study are empirical and methodological rather than algorithmic. Specifically, the paper makes the following key contributions: (1) It reframes explanation reliability as an empirically measurable property and demonstrates how stability, sensitivity, and cross-model agreement jointly reveal failure modes that accuracy-based evaluation misses. (2) It introduces a hybrid evaluation design combining public real-world data with synthetic stress tests, enabling controlled analysis of explanation behavior under spurious correlations. (3) This study provides an empirical analysis under the considered regimes in our evaluated setting showing that explanation instability can increase even as predictive performance remains strong (often high) on the evaluated benchmarks, challenging common assumptions in explainable AI practice. Operational addition: we contribute an operational reliability gate (a threshold rule over stability, sensitivity, and consistency) that can be reused as a plug-in stage in XAI benchmarks and auditing pipelines—deciding when not to act on an explanation. We formalize a deployable reliability gate as a benchmark plug-in stage that converts reliability signals into an operational reject/accept decision with explicit risk–coverage control. Related work and positioning. Early explainable AI research focused on interpretability mechanisms such as feature attribution and saliency methods. More recent work has emphasized explanation robustness, faithfulness, and consistency, revealing that explanations can vary significantly across retraining runs or data perturbations. Despite this progress, existing studies often evaluate reliability dimensions in isolation or under narrow experimental conditions. In contrast, this work aligns with recent calls for holistic empirical evaluation by jointly analyzing stability, sensitivity, and cross-model consistency within a unified experimental protocol. Recent work continues to refine robustness and faithfulness evaluation, including robust faithfulness metrics designed to reduce out-of-distribution artifacts in perturbation-based tests [ 14 ] and unified evaluation frameworks showing how explainer behavior depends on model architecture and dataset characteristics [ 15 ]. A practical implication is that reliability checks need to be cheap enough to run routinely, not only in one-off case studies. Cost matters in practice. A protocol that is too expensive to run routinely will not survive contact with real auditing workflows. A recurring challenge is that “explanation quality” is not a single construct: taxonomies distinguish transparency vs. post-hoc explanation, local vs. global scope, and multiple evaluation desiderata (faithfulness, stability, and usability) that can conflict in practice [ 16 ], [ 8 ], [ 5 ]. Recent empirical work therefore recommends separating the evaluation of explanation generation from the evaluation of explanation reliability under perturbations, because the latter can fail even when predictive performance is strong [ 13 ], [ 7 ]. Our protocol follows this line by treating reliability as a first-class experimental target and by reporting trade-offs explicitly, rather than collapsing them into a single “best” explainer score. Within robustness-focused XAI, stability is often operationalized as agreement of explanations across retraining runs, bootstrap samples, or small input perturbations. Early evidence shows that many popular explainers can be brittle under such changes, motivating robustness tests for explanations themselves rather than only for predictions [ 1 ], [ 2 ], [ 3 ]. At the same time, most robustness studies focus on local, post-hoc explainers (e.g., LIME/SHAP) [ 10 ], [ 11 ], whereas system auditing in tabular settings frequently relies on model-specific global importance signals that are cheaper to compute and easier to monitor over time [ 16 ], [ 7 ]. Our study therefore complements prior robustness analyses by centering on global, model-specific importance and by structuring stress tests that mimic proxy-driven “audit failures.” Another relevant thread connects explanation reliability to dataset shift and spurious correlations. Shortcut learning and correlated predictors can yield deceptively stable explanations in-distribution, while degrading under shifts that change which features act as shortcuts [ 17 ]. Recent venue papers on trust and reliability highlight that explainability claims can become fragile when evaluation ignores such regimes, especially in high-stakes contexts where explanations may be used to justify deployment decisions [ 9 ], [ 14 ]. These observations motivate stress gradients that are not intended to emulate a single real domain, but to probe failure modes systematically by controlling proxy strength and noise. Finally, there is growing consensus that interpretability should support operational decisions, not only retrospective narratives: calls to avoid over-reliance on post-hoc explanations emphasize the need for rigorous, reproducible evaluation and clear decision rules about when an explanation should not be acted upon [ 4 ], [ 5 ]. The reliability-gate perspective adopted here fits this operational framing: we focus on identifying regimes where explanation signals are sufficiently reliable to be used in model auditing, and we make the coverage–risk trade-off explicit. Design note: we focus on global, model-specific importance because it is cheap to compute, easy to log, and commonly used in tabular audit reports. We set k = 5 to match what auditors typically review in practice; Top-10 is reported as a sensitivity check (Supplementary Table S3). We also keep k and the perturbation schedule fixed across datasets to avoid any post-hoc tuning of explanations. In a real audit, these knobs are usually set once and reused for consistency. Contributions. The primary contributions of this study are empirical and methodological rather than algorithmic. Specifically, the paper makes the following key contributions: (1) It reframes explanation reliability as an empirically measurable property and demonstrates how stability, sensitivity, and cross-model agreement jointly reveal failure modes that accuracy-based evaluation misses. (2) It introduces a hybrid evaluation design combining public real-world data with synthetic stress tests, enabling controlled analysis of explanation behavior under spurious correlations. (3) This study provides an empirical analysis under the considered regimes in our evaluated setting showing that explanation instability can increase even as predictive performance remains strong (often high) on the evaluated benchmarks, challenging common assumptions in explainable AI practice. Results Figure 1 illustrates explanation stability across datasets. Linear models remain stable, while tree-based models show stability that varies with the data regime: on the easy synthetic generators (including Syn-SpuriousShift-Easy) stability can appear inflated, whereas on real data it can drop due to sampling variance and correlated predictors. This highlights why stability must be interpreted jointly with the stress condition. Interpretation: Logistic Regression (LR) exhibits consistently high stability across regimes—typically highest on the real benchmarks—because global attributions are directly tied to learned coefficients; repeated retraining under comparable splits yields highly consistent coefficient rankings. [16], [1] In contrast, RF stability varies by regime and can be affected by correlated predictors and sampling variance; synthetic low-noise settings may inflate stability. Accordingly, stability must be interpreted jointly with the stress condition rather than treated as a standalone guarantee of explanation robustness. [6], [1], [17] Interpretation: Relative sensitivity (rESS) for LR is lower on BreastCancer-UCI than on Wine-UCI (Supplementary Table S3), indicating that mild perturbations can redistribute attribution magnitudes on some real tabular tasks even when output changes remain modest. This should be interpreted as a warning sign for explanation fragility rather than a performance metric, consistent with robustness concerns raised in prior work. [1], [2] In the synthetic regimes, rESS values increase substantially, reflecting the controlled data-generating process and reduced noise. This contrast suggests why rESS should be treated as a diagnostic tool and compared primarily within a model family across conditions, not as an absolute cross-model score. [1], [2] Interpretation: Complete agreement in the simplified synthetic base regime can be trivial—when the data-generating signal is highly separable, different models naturally recover the same dominant features. Thus, high consistency here reflects dataset simplicity, not guaranteed reliability. [17] Agreement drops on the real datasets where correlated predictors and differing inductive biases yield multiple plausible feature subsets. By contrast, Syn-SpuriousShift-Easy intentionally yields high Top-k overlap (Supplementary Table S4) to illustrate trivial agreement; the harder non-triviality checks in Supplementary Table S5 show that agreement can deteriorate under interaction-dominant proxy structure even when accuracy remains competitive. Interpretation: Predictive performance is generally strong across the two real benchmarks, which helps isolate explanation reliability effects. On synthetic stress tests, performance can vary by design (Supplementary Table S5), enabling us to probe reliability under controlled difficulty rather than confounding reliability with arbitrary accuracy differences. Design note: our stress tests and auditing protocol are not intended to optimize AUC; they are constructed to isolate explanation reliability effects under controlled variability and proxy/noise shifts. Notably, a spurious-shift setup can maintain strong AUC while explanation behavior changes under stress. In our easy generator, agreement is intentionally high (Supplementary Table S4), whereas the harder variants in Supplementary Table S5 reduce agreement and can depress linear-model AUC—demonstrating that accuracy alone does not certify explanation reliability. Figure 2 reports explanation sensitivity (rESS). Notably, low rESS values indicate explanation fragility even when prediction changes are small, emphasizing that accuracy alone is insufficient to assess explanation quality. Why can rESS collapse while Δprediction remains small? Δprediction measures the average change in predicted probabilities, which can stay modest when the task is redundant (multiple correlated features support similar decisions). In contrast, rESS compares the distribution of global importance across features: under collinearity or near-equivalent feature sets, small changes in fitted parameters can redistribute attribution mass substantially without strongly moving the output. This coefficient/attribution instability has been repeatedly observed in robustness critiques of explanation methods and model interpretations [1], [2], [3]. Figure 3 shows cross-model consistency via Top-k overlap. On Syn-Base and the easy spurious-shift generator (Syn-SpuriousShift-Easy), agreement can remain high because the generator is highly separable—this is precisely the triviality risk: high agreement does not imply that explanations are decision-relevant. To demonstrate non-triviality, we additionally report a harder synthetic check in Supplementary Table S5, where agreement drops when the proxy is only exploitable through interaction (trees can exploit it; linear models cannot). Syn-SpuriousShift-Easy is designed to preserve trivial agreement (Supplementary Table S4); reliability degradation is demonstrated using the harder variants in Supplementary Table S5. Additional non-triviality check: the complete agreement observed in Syn-Base and Syn-SpuriousShift-Easy is intentionally retained as an example of “easy agreement” on highly separable generators. To show that our protocol does not force agreement, we also evaluate harder spurious-shift variants (Supplementary Table S5) where interaction and noise make the proxy partially exploitable; in these cases LR vs RF Top-5 agreement drops substantially and AUC can diverge, illustrating that high accuracy does not certify reliability. Figure 4 confirms that predictive performance remains strong on the real benchmarks and competitive overall, reinforcing that explanation reliability must be evaluated independently of accuracy. [Figure 1 about here] Expanded sensitivity reporting: in addition to rESS, we now report the underlying deviation E (mean±std) and a Top-10 variant (rESS_top10) that focuses on the most salient features (Supplementary Table S3). This directly addresses the concern that small Δprediction can coexist with large redistribution of attribution mass: rESS quantifies stability of the full importance distribution, while rESS_top10 provides a complementary sanity-check that concentrates on salient features. [Figure 2 about here] [Figure 3 about here] [Figure 4 about here] Reliability-gated auditing We simulate an auditing scenario with an impermissible proxy feature that is moderately correlated with the label during training, but is decorrelated at deployment (randomized). The auditor inspects global feature-importance signals and triggers a failure if the proxy appears among the Top-5 features. To create a non-trivial failure case, we (i) randomize proxy strength across seeds and (ii) add nuisance features to increase ranking variance. We repeat training across 20 seeds and two model families (LR/RF), yielding n=40 runs. Note: Table 2 follows the auditing protocol (fixed split, n=40 runs), whereas Tables A2–A4 report T=5 independent resampling runs. Uncertainty (bootstrap, 95% CI over runs; n=40): baseline audit error 42.5% [27.5, 57.5], gated audit error 0.0% [0.0, 0.0]. For AUC, baseline 0.995 [0.993, 0.997], gated 0.995 [0.992, 0.998]. Table 2 reports bootstrap CIs over run-level means under the fixed-split auditing protocol, whereas Supplementary Information (Supplementary Information B) reports Wilson CIs for run-level error proportions (including gated subsets). Interpretation (consistent with Table 2 and Table 3): under the Moderate gate (q=0.60; τS=0.92, τE=0.831, τC=0.25), the audit-error rate drops to 0.0% on the retained explanations while coverage decreases to 22.5%. This is an explicit safety–coverage trade-off: the gate rejects explanations that are unstable (low S), overly sensitive (low rESS@Top5), or poorly supported across model families (low C), before the auditor acts on them. Gate is a risk-control mechanism: it is expected to reduce coverage under stricter thresholds, and the operational choice is governed by an audit-error tolerance α. Low coverage is an expected outcome of risk control; practitioners should select (q, τC) via the audit error–coverage trade-off (Table 3) to meet a target tolerance α. We therefore treat coverage as a first-class outcome, not a nuisance. Thresholds are calibrated from reliability quantiles (Table 3) rather than optimized for audit error. Note: C is discrete when computed on Top-k overlaps; therefore small threshold changes can produce step-wise coverage shifts. We report three increasing regimes (Lenient/Moderate/Conservative) via quantiles on S and rESS while keeping τC aligned to each regime as shown in Table 3. Intermediate quantiles (e.g., q=0.65) can be used to densify the trade-off curve and are left for follow-up reporting. [Figure 5 about here] Across the two real datasets, Wine-UCI tends to show higher perturbation sensitivity (higher E and lower rESS/rESS_top10) than BreastCancer-UCI, even when AUC remains strong. This illustrates the paper’s central claim: explanation reliability should be assessed independently of predictive performance, because stable accuracy can mask unstable attribution patterns under mild shifts. Wine vs Breast Cancer: What changes? Across the same protocol (T=5 stratified resampling runs per dataset×model), Wine-UCI exhibits a more volatile reliability profile than BreastCancer-UCI. While both datasets sustain high ROC-AUC, the stability and sensitivity signals tend to degrade more on Wine, reflecting its smaller sample size and the fact that it originates from a multi-class problem (binarized here). Practically, this means feature-importance rankings on Wine are more sensitive to split and retraining randomness, and perturbation stress can redistribute attribution mass more noticeably even when Δprediction remains modest. Cross-model consensus also changes: BreastCancer can admit multiple plausible feature subsets due to correlated predictors, whereas Wine may concentrate importance on fewer features but with higher run-to-run variability. The key takeaway is that explanation reliability is dataset-conditional; thresholds (τS, τE, τC) should be calibrated per deployment regime rather than transferred from a single benchmark. Discussion Operationally, acting on unreliable explanations introduces tangible risks in model auditing and governance. The reliability-gated auditing results provide evidence: baseline auditing can fail by surfacing spurious proxy features despite stable AUC, while the gate reduces audit error by screening out unstable, overly sensitive, or cross-model-inconsistent explanations. This screening is a risk-control mechanism—coverage may decrease under stricter thresholds by design—and practitioners should choose (q, τC) using the audit error–coverage trade-off (Table 3) to satisfy an operational tolerance α. We intentionally keep the gate as a simple conjunctive rule so it can be implemented without training an additional classifier. In practice, this also makes threshold choices auditable. Very high cross-model agreement on simplified synthetic regimes is intentionally retained as a cautionary example: high agreement can be trivial when the data-generating signal is highly separable. This can create a false sense of trust, masking fragility once interaction-dominant proxies, nuisance noise, or regime shifts are introduced (Supplementary Table S5). In other words, agreement can be an artifact of task simplicity. That is exactly what the stress gradient is meant to reveal. The main takeaway is that reliability metrics expose failure modes that standard predictive evaluation does not, consistent with robustness critiques showing that global importance signals can be unstable or manipulable even when models are accurate [1], [2]–[5]. Accordingly, we advance one focused, testable claim: explanation reliability should be evaluated before explanation usefulness. To decouple reliability effects from accuracy, we combine high-AUC real benchmarks with complementary synthetic stress gradients (Supplementary Table S5) that systematically vary proxy interaction and nuisance noise. Implications for XAI Benchmarks Benchmark suites can adopt a reliability stage by explicitly reporting stability/sensitivity/consistency under stress tests, instead of treating a single explanation score as sufficient. This complements recent calls for transparent evaluation and comparable benchmarking practice [13]–[10]. Practical insertion into an auditing pipeline (high-level): (i) train the model and compute global, model-specific importance; (ii) repeat training under the resampling/seed protocol to estimate stability S and cross-model consistency C; (iii) run the perturbation/noise stress to estimate rESS@Top5; (iv) calibrate thresholds from reliability quantiles (Table 3) to match an operational audit-error tolerance α (rather than tuning for minimum error); (v) apply the reliability gate to screen explanations before an auditor acts on them; and (vi) log the resulting audit error–coverage trade-off and monitor it over time as data drift or retraining updates occur. This provides a lightweight governance artifact that can be reviewed alongside standard performance reports in applied IEEE settings. Failure modes and operational guidance for the reliability gate Failure modes (what the gate does NOT guarantee): (i) Trivial agreement—if models learn the same shortcut, C may be high while explanations remain misleading; (ii) Stability without robustness—S can be high even when attributions shift substantially under targeted perturbations (low rESS@Top5), especially under collinearity; (iii) Robustness without consensus—rESS may be high for a single model family yet C is low because alternative inductive biases emphasize different, equally predictive feature sets. Practitioner guidance: choose q and τC as a risk-control knob. Lower q (Lenient) increases coverage but tolerates more unreliable explanations; higher q (Conservative) reduces coverage and should be used when acting on explanations carries higher governance risk. We recommend reporting the full audit error–coverage curve (Table 3) and selecting (q, τC) to satisfy an operational constraint such as audit error ≤ α at maximum achievable coverage. 7. Limitations Reviewer-facing clarification Thresholds are calibrated from empirical reliability quantiles (Table 3) rather than optimized for audit error, and the gate is expected to trade coverage for reduced audit risk under stricter regimes. The reported high AUC on the real benchmarks is intentional to decouple explanation reliability from predictive performance; the synthetic stress gradient (Supplementary Table S5) is used to probe reliability under controlled difficulty shifts. This study is limited to tabular data and global attribution signals. While the observed reliability patterns are transferable, no claim of universal generalization across modalities is made. The selected real benchmarks yield high ROC-AUC, which helps isolate reliability effects but limits conclusions about low-accuracy regimes. We intentionally use LR and RF as canonical model families in the main protocol; extending the same reliability tests to boosting (e.g., gradient-boosted trees such as XGBoost) and neural models is left as future work. We focus on global, model-specific importance; local explainers and post-hoc methods are future work. We also note that the main experiments focus on model-specific importance signals; extending the same reliability protocol to a broader set of attribution methods (e.g., permutation-based or Shapley-style explainers) is an important next step. Conclusion In our evaluated setting, we show that explanation reliability is a distinct and empirically measurable dimension of model behavior. By combining public real-world benchmarks with synthetic stress gradients, our protocol exposes explanation failure modes that remain hidden under conventional accuracy-focused evaluation. We further show that a simple threshold-based reliability gate can act as an operational risk-control mechanism: it can reduce audit error by screening out unstable, overly sensitive, or cross-model-inconsistent explanation signals, at the expected cost of reduced coverage under stricter thresholds. From an applied perspective, this supports reliability-aware auditing in practice—for example, screening global feature-importance reports in a clinical triage model before changing treatment pathways, or in a credit decision system before issuing adverse-action rationales. Importantly, the empirical results reported here are specific to tabular settings using global, model-specific importance on LR/RF; we do not evaluate vision/text modalities or local/post-hoc explainers. Extending the same reliability protocol to additional model families, modalities, and explanation types is a natural direction for future work. Methods We treat explanation reliability as a pre-use screening problem: the goal is to decide whether an available explanation signal is stable enough to support auditing or decision support, rather than proposing a new explainer. This stance follows calls for rigorous, testable interpretability science [6], [4]. Our protocol uses three stress lenses—stability, sensitivity, and cross-model consistency—motivated by evidence that explanations can be fragile to perturbations and manipulations [1], [2]–[13], and that agreement can be trivial on overly simplified tasks [17]. Definitions Let f denote a trained predictor and E(f, x) produce an explanation vector e ∈ ℝ^d (e.g., feature attributions). Over repeated trials t=1…T: • Stability (S): average Spearman rank correlation between explanation vectors across trials; higher S implies less sensitivity to retraining randomness. We chose Spearman to compare rankings rather than magnitudes, which is stable under monotone rescaling. • Relative sensitivity (rESS): for baseline explanation e and perturbed ê, define the relative deviation E = ||e−ê||₁ / (||e||₁+ε). We report a bounded robustness score rESS = 1/(1+E) ∈ (0,1], where higher values indicate explanations change less under controlled perturbations. • Consistency (C): Jaccard overlap between top-k feature sets from explanations produced by alternative model families trained on the same task. We use Jaccard because it is simple, discrete, and directly interpretable as shared features. Threshold-based reliability rule Reliable(e) ⇔ (S ≥ τS) ∧ (rESS@Top5 ≥ τE) ∧ (C ≥ τC). For the auditing experiment (Wine-UCI) we use the Moderate operational setting (q=0.60; τS=0.92, τE=0.831, τC=0.25), estimated from quantiles over n=40 runs (20 seeds × 2 model families). Here rESS@Top5 is computed from the relative deviation on the baseline Top-5 importance scores under a stress perturbation that randomizes the proxy and injects feature noise. Composite Reliability Index (supplementary) For reporting only, we summarize the three dimensions as CRI = wS·S + wE·rESS + wC·C with wS+wE+wC=1. Decisions rely on the explicit threshold rule above. 4. Experimental Setup Specifically, we use the Wine dataset from the UCI Machine Learning Repository. The dataset contains 178 instances, 13 continuous physicochemical attributes, and three classes corresponding to wine cultivars. To align with the binary setting used in our reliability metrics and auditing rule, we form a binary task (class 1 vs. {0,2}), treating class 1 as the positive label (positive ratio ≈ 0.331, Table 1). All features are numeric with no missing values; features are standardized within each training split. [18], [19] Unlike the Breast Cancer dataset, Wine exhibits higher feature heterogeneity, class imbalance, and non-linear decision boundaries. This dataset therefore provides a more challenging and realistic testbed for evaluating explanation reliability under feature sparsity, encoding choices, and distributional complexity. The same evaluation protocol is applied without modification to ensure comparability. [18], [19] Reproducibility note: main-results tables/figures use T=5 independent stratified resampling runs per dataset×model (each run draws a new train/test split; seeds 0–4), reporting mean±std over runs. Auditing uses a fixed stratified split on Wine-UCI and varies only the random seed controlling model training randomness and proxy-strength generation; we train LR and RF over 20 seeds, giving n=40 independent model×seed runs. Bootstrap 95% confidence intervals are computed by resampling these runs with replacement and taking the mean audit-error (or AUC) across resampled runs. All auditing runs share the same split; only model randomness and proxy-strength differ. We keep the split fixed in auditing so that uncertainty reflects run-to-run randomness rather than changing data partitions. We picked T=5 because it provides a small but useful variance estimate while keeping the protocol lightweight. Practitioners can increase T if they have more compute or need tighter intervals. Declarations Code Availability: The code to reproduce all experiments, tables, and figures is maintained in a private repository during peer review. Access can be granted to the Editor and reviewers upon request. The repository will be made publicly available upon acceptance. We choose LR and RF as canonical linear vs. tree inductive biases; adding boosting/NNs is left as future work. This choice isolates contrasting inductive biases while keeping the protocol reproducible; extending to boosting/NNs is straightforward within the same protocol. Use of generative AI tools We used a large language model tool for language polishing and formatting assistance. All technical content, experimental choices, and reported numbers were reviewed and verified by the authors. Data Availability All real datasets used in this study are publicly available: the UCI Wine dataset and the Breast Cancer Wisconsin dataset. Synthetic datasets are generated from the procedures described in the Methods and the Supplementary Information. Code Availability The experimental code, dataset generators, and plotting scripts are maintained in a private repository during peer review and will be released publicly upon acceptance. (Repository URL: https://github.com/) Competing Interests The author declare no competing interests. Funding Declaration This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Author Contribution MOHAMMED ALSHANTI conceived the study, designed the experiments, implemented the protocol, analyzed the results, and wrote the manuscript. Acknowledgements We thank the reviewers for their constructive feedback. MOHAMMED ALSHANTI conceived the study, designed the experiments, implemented the protocol, analyzed the results, and wrote the manuscript. References Alvarez-Melis, D. & Jaakkola, T. S. On the Robustness of Interpretability Methods. In Proc. ICML Workshop on Human Interpretability in Machine Learning (WHI), Stockholm, Sweden, Jul. 2018. [Online]. Available: https://sites.google.com/view/whi2018/home. Accessed: Jan. 2026. (Preprint: arXiv:1806.08049.). Yeh, C.-K., Hsieh, C.-Y., Suggala, A. S., Inouye, D. I. & Ravikumar, P. On the (In)fidelity and Sensitivity of Explanations. In Proc. NeurIPS, 2019, pp. 10965–10976. Ghorbani, A., Abid, A. & Zou, J. Interpretation of Neural Networks Is Fragile. Proc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, https://doi.org/10.1609/aaai.v33i01.33013681. Rudin, C. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nat. Mach. Intell., vol. 1, pp. 206–215, 2019, https://doi.org/10.1038/s42256-019-0048-x. Vilone, G. & Longo, L. Notions of explainability and evaluation approaches for explainable artificial intelligence. Inf. Fusion, vol. 76, pp. 89–106, 2021, https://doi.org/10.1016/j.inffus.2021.05.009. Lisboa, P. J. G., Saralajew, S., Vellido, A., Fernández-Domenech, R. & Villmann, T. The coming of age of interpretable and explainable machine learning models. Neurocomputing, vol. 535, pp. 25–39, 2023, https://doi.org/10.1016/j.neucom.2023.02.040. Le, H., Ghazanfari, R., Cheung, S. C. & Salim, F. D. Benchmarking Explainable Artificial Intelligence: A Survey of Evaluation Methods and Practices. In Proc. IJCAI, 2023, pp. 6734–6742, https://doi.org/10.24963/ijcai.2023/750. Schwalbe, G. & Finzel, B. A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts. Data Mining Knowl. Discov., vol. 38, no. 5, pp. 3043–3101, 2024, https://doi.org/10.1007/s10618-022-00867-8. Vascotto, I. et al. When Can You Trust Your Explanations? A Robustness Analysis on Feature Importances. In Explainable Artificial Intelligence (xAI 2025), Commun. Comput. Inf. Sci., vol. 2578, pp. 225–249, Oct. 2025, https://doi.org/10.1007/978-3-032-08327-2_11. Ribeiro, M. T., Singh, S. & Guestrin, C. Why Should I Trust You?: Explaining the Predictions of Any Classifier. In Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD), 2016, pp. 1135–1144, https://doi.org/10.1145/2939672.2939778. Lundberg, S. M. & Lee, S.-I. A Unified Approach to Interpreting Model Predictions. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2017, pp. 4765–4774. Pruthi, D., Gupta, M., Dhingra, B., Neubig, G. & Lipton, Z. C. Learning to Deceive with Attention-Based Explanations. In Proc. 58th Annu. Meeting Assoc. Comput. Linguist. (ACL), 2020, pp. 4782–4793, https://doi.org/10.18653/v1/2020.acl-main.432. Agarwal, C., Ley, D., Krishna, S., Saxena, E., Pawelczyk, M., Johnson, N., Puri, I., Zitnik, M. & Lakkaraju, H. OpenXAI: Towards a Transparent Evaluation of Model Explanations. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), Datasets and Benchmarks Track, 2022. [Online]. Available: proceedings.neurips.cc/paper_files/paper/2022/hash/65398a0eba88c9b4a1c38ae405b125ef-Abstract-Datasets_and_Benchmarks.html. Accessed: Jan. 2026. Zheng, X., Shirani, F., Chen, Z., Lin, C., Cheng, W., Guo, W. & Luo, D. F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI. In Proc. Int. Conf. Learn. Representations (ICLR), 2025. [Online]. Available: openreview.net/forum?id=X0r4BN50Dv. Accessed: Jan. 2026. Dehdarirad, T. Evaluating explainability in language classification models: A unified framework incorporating feature attribution methods and key factors affecting faithfulness. Data and Information Management, vol. 9, no. 4, p. 100101, Dec. 2025, https://doi.org/10.1016/j.dim.2025.100101. Molnar, C. Interpretable Machine Learning, 2nd ed., 2022. [Online]. Available: christophm.github.io/interpretable-ml-book/. Accessed: Jan. 2026. Geirhos, R. et al. Shortcut Learning in Deep Neural Networks. Nat. Mach. Intell., vol. 2, no. 11, pp. 665–673, 2020, https://doi.org/10.1038/s42256-020-00257-z. Aeberhard, S. & Forina, M. Wine. UCI Machine Learning Repository, 1991. https://doi.org/10.24432/C5PC7J. Wolberg, W. H., Street, W. N. & Mangasarian, O. L. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1995. https://doi.org/10.24432/C5DW2B. Tables Table 1 | Datasets used in the study (real benchmarks and synthetic stress tests). Dataset Type Source Samples Features Positive ratio BreastCancer-UCI Real (public) UCI / scikit-learn 569 30 0.6274165202108963 Wine-UCI Real (public; binarized) UCI / scikit-learn 178 13 0.33146067415730335 Syn-Base Synthetic This study 2500 18 0.5 Syn-SpuriousShift-Easy Synthetic This study 2500 19 0.5 Table 2 | Reliability-gated auditing outcomes on Wine-UCI under proxy shift (mean over n=40 runs; 95% CI reported below). Condition Test ROC-AUC Audit error (%) Coverage (%) Baseline (no gate) 0.995 42.5 100.0 Reliability-gated 0.995 0.0 22.5 Table 3 | Threshold sensitivity analysis (audit error vs. coverage trade-off). Regime Quantile q τS τE (rESS) τC Gated audit error (%) Coverage (%) Gated AUC Lenient 0.50 0.91 0.815 0.20 16.7 30.0 0.996 Moderate 0.60 0.92 0.831 0.25 0.0 22.5 0.995 Conservative 0.70 0.93 0.848 0.30 0.0 15.0 0.995 Additional Declarations No competing interests reported. Supplementary Files SupplementaryInformationforReliabilityOrientedEvaluationofExplanationSignalsSupplementaryInformation.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8554983","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":575120605,"identity":"5936ccdb-71bb-4f1f-afb7-95e47f42a1f1","order_by":0,"name":"MOHAMMED ALSHANTI","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA50lEQVRIiWNgGAWjYFACHjApB8RsEIEDRGlJYDAmXUtiA9FadPvPHvxc+ONwev/sw88efGxjkOO7kcD84QceLWY38pKlZyQczp1xLs3ccGYbg7HkjQQ2yR68WngMpHmAWhrOMJhJ87YxJG4AaoGGCQ4t588Y/wZqSZc/w/4NpKUeqIX54x98Wg7kmIFsSTA4wwO2JcHgRgKDNF5bbuSYWfOkpRtuPMNTbjjjnIThzDMP26RlCDjsNo+NtbzcGfZtDz6U2cjzHU8+/PENHi1Q0AxjSAAxYwNhDQwMdcQoGgWjYBSMgpEKAIZwTZT03NSeAAAAAElFTkSuQmCC","orcid":"","institution":"Arab Open University","correspondingAuthor":true,"prefix":"","firstName":"MOHAMMED","middleName":"","lastName":"ALSHANTI","suffix":""}],"badges":[],"createdAt":"2026-01-08 20:53:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8554983/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8554983/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":100547326,"identity":"d2a62e11-7231-44e7-aca2-1365ca7badf5","added_by":"auto","created_at":"2026-01-19 08:15:16","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":311216,"visible":true,"origin":"","legend":"","description":"","filename":"ReliabilityOrientedEvaluationSciRepManuscriptMainv1FundingAdded.docx","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/4421a04b6d742c0773d2ea29.docx"},{"id":100546859,"identity":"253dd8ab-cf7c-4ea8-beed-b6e6322855a6","added_by":"auto","created_at":"2026-01-19 08:12:59","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3019,"visible":true,"origin":"","legend":"","description":"","filename":"0c01122222204d76b8a6ab560bea191e.json","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/c8d580551430431b9c2ed789.json"},{"id":100447204,"identity":"ee82ba3a-8f74-4ba0-9cc8-0285ec21ba3a","added_by":"auto","created_at":"2026-01-16 18:51:54","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":40767,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryInformationforReliabilityOrientedEvaluationofExplanationSignalsSupplementaryInformation.docx","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/9343fdb9d54b998d845984be.docx"},{"id":100547129,"identity":"76cba7d2-880a-409c-ab95-de4508c1833b","added_by":"auto","created_at":"2026-01-19 08:14:35","extension":"xml","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":81432,"visible":true,"origin":"","legend":"","description":"","filename":"0c01122222204d76b8a6ab560bea191e1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/8c649e2cd2a47a7287e6e662.xml"},{"id":100547354,"identity":"a4245e92-9a81-4b1e-aaed-1e49c7d983d1","added_by":"auto","created_at":"2026-01-19 08:15:20","extension":"xml","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":80849,"visible":true,"origin":"","legend":"","description":"","filename":"0c01122222204d76b8a6ab560bea191e1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/fc192eb8ea48660dafc3f446.xml"},{"id":100447208,"identity":"87b8f3f6-2273-4828-bf77-2dbc9f651504","added_by":"auto","created_at":"2026-01-16 18:51:55","extension":"html","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":91460,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/65bb7cc2c98f244274dc38f6.html"},{"id":100447197,"identity":"41134614-66e5-403e-9481-6ff3aed3e946","added_by":"auto","created_at":"2026-01-16 18:51:54","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":69478,"visible":true,"origin":"","legend":"\u003cp\u003eStability under seed and resampling variability. Higher values indicate explanations are less sensitive to retraining randomness.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/a4f8ace1ce523231ace735ab.png"},{"id":100547197,"identity":"3990207f-7a2b-424c-b052-6281d167908c","added_by":"auto","created_at":"2026-01-19 08:14:52","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":62773,"visible":true,"origin":"","legend":"\u003cp\u003eSensitivity under perturbation/noise stress tests (reported as rESS). Higher values indicate more robust explanations under controlled corruption.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/312b98a67fb00e93827037d8.png"},{"id":100447202,"identity":"3cd54392-8acb-44ff-9bcc-628d574945ae","added_by":"auto","created_at":"2026-01-16 18:51:54","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":79909,"visible":true,"origin":"","legend":"\u003cp\u003eCross-model consistency (Jaccard@k). Higher overlap reduces the risk of relying on model-specific artifacts.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/3a2a68fb46152340a9f77ddf.png"},{"id":100547358,"identity":"c5b028e7-bb79-4cc3-ba42-21eb91c6ad2b","added_by":"auto","created_at":"2026-01-19 08:15:20","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":64002,"visible":true,"origin":"","legend":"\u003cp\u003ePredictive ROC-AUC across conditions. Accuracy contextualizes reliability: high AUC does not imply reliable explanations.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/2555a3c0085afd114d6d12f8.png"},{"id":100447206,"identity":"f0d2c9c4-dae2-4cff-bbb2-b4f1910a2645","added_by":"auto","created_at":"2026-01-16 18:51:55","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":55734,"visible":true,"origin":"","legend":"\u003cp\u003eAudit error rate before and after applying the reliability gate (moderate thresholds, Wine-UCI; n=40 runs).\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/7609457db44b801aa3b3b26b.png"},{"id":105729302,"identity":"329b9c0e-5bfc-4dc6-8a69-200d827e1538","added_by":"auto","created_at":"2026-03-30 11:14:19","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1103794,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/86638e32-5abe-4bf4-89fd-e7ee680f85f1.pdf"},{"id":100447198,"identity":"a9e9b6a3-67b9-43a4-97cf-c9fa1a8294a1","added_by":"auto","created_at":"2026-01-16 18:51:54","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":40767,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryInformationforReliabilityOrientedEvaluationofExplanationSignalsSupplementaryInformation.docx","url":"https://assets-eu.researchsquare.com/files/rs-8554983/v1/b78f0753c81caf3b12ebf5e8.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Reliability-Oriented Evaluation of Explanation Signals under Real and Synthetic Stress Tests","fulltext":[{"header":"Introduction","content":"\u003cp\u003eRecent studies have shown that explanations may be unstable, overly sensitive to perturbations, or inconsistent across models, even when predictive accuracy is high [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e], [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. In many high-stakes domains, such as healthcare, finance, and critical infrastructure, this mismatch between apparently strong performance and fragile explanations raises concrete governance concerns: auditors, domain experts, and regulators are increasingly asked to rely on explanation artifacts when validating models, yet the reliability of those artifacts is rarely quantified or stress-tested in a systematic way [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e], [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. A large body of work in explainable AI (XAI) has therefore shifted attention from proposing ever more explanation methods to examining whether existing methods behave in a trustworthy and predictable manner under realistic conditions [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e], [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eEarly XAI research predominantly focused on interpretability mechanisms such as feature attribution, saliency maps, and surrogate models, often evaluated on a small number of benchmarks or illustrative case studies [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. More recent surveys and taxonomies highlight that explanation quality is multi-dimensional, spanning aspects such as robustness, faithfulness, stability, and human-centered usefulness [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Within this landscape, robustness-oriented studies have demonstrated that explanations can change markedly across retraining runs, small input perturbations, or shifts in the underlying data distribution, even when predictive metrics such as ROC-AUC or accuracy remain nearly constant [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e], [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]\u0026ndash;[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. These findings suggest that accuracy-centric evaluation is insufficient: without explicit checks on explanation behavior, practitioners may unknowingly deploy models whose explanations are unreliable, misleading, or overly dependent on spurious correlations.\u003c/p\u003e \u003cp\u003eDespite this progress, existing empirical work on explanation reliability tends to examine individual dimensions in isolation\u0026mdash;for example, stability under random seeds, sensitivity to input noise, or agreement between a specific explainer and a chosen reference method\u0026mdash;often under narrow experimental conditions and without a unified protocol [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e], [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Benchmark efforts have begun to standardize datasets and tasks for XAI evaluation, but they frequently prioritize aggregate explanation scores or faithfulness metrics over reliability under stress, and they rarely provide operational decision rules for when an explanation should be considered too unstable or inconsistent to act upon [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]\u0026ndash;[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. As a result, there is still limited guidance on how to operationalize explanation reliability as a first-class empirical property that can be measured, stress-tested, and integrated into auditing workflows.\u003c/p\u003e \u003cp\u003eThis paper contributes to that gap by shifting focus from method-centric novelty toward systematic empirical assessment of explanation behavior under both real and synthetic stress regimes. We treat explanation reliability as a pre-use screening problem: the practical question is not which explainer is newest or most sophisticated, but whether the available explanation signal is stable enough, insensitive enough to controlled perturbations, and sufficiently consistent across model families to support auditing or decision support in a given deployment context [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e], [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. To make this perspective concrete, we combine two public tabular benchmarks with controlled synthetic generators that inject spurious correlations and interaction-dominant proxies, allowing us to disentangle reliability phenomena from purely performance-driven effects.\u003c/p\u003e \u003cp\u003eWe keep the scope deliberately narrow: tabular benchmarks and global, model-specific importance. That is not a limitation by accident. It helps us separate explanation reliability from modality-specific preprocessing and from the extra randomness of local post-hoc explainers.\u003c/p\u003e \u003cp\u003eWithin this setting, we introduce and analyze three complementary reliability lenses\u0026mdash;stability across retraining seeds, relative sensitivity (rESS) under perturbation and noise, and cross-model consistency via Top-k overlap\u0026mdash;and study how they behave jointly across real and synthetic regimes [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e], [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]\u0026ndash;[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], [\u003cspan additionalcitationids=\"CR10 CR11\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]\u0026ndash;[\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Building on these signals, we derive a simple threshold-based \u0026ldquo;reliability gate\u0026rdquo; that can be plugged into XAI benchmarks and auditing pipelines to decide when not to act on an explanation: explanations that fail minimal stability, sensitivity, or cross-model agreement criteria are screened out before they influence an auditor\u0026rsquo;s judgment. The remainder of the paper formalizes these components, reports empirical findings across datasets, and discusses implications for reliability-aware XAI benchmarking and governance.\u003c/p\u003e \u003cp\u003eThe primary contributions of this study are empirical and methodological rather than algorithmic. Specifically, the paper makes the following key contributions:\u003c/p\u003e \u003cp\u003e(1) It reframes explanation reliability as an empirically measurable property and demonstrates how stability, sensitivity, and cross-model agreement jointly reveal failure modes that accuracy-based evaluation misses.\u003c/p\u003e \u003cp\u003e(2) It introduces a hybrid evaluation design combining public real-world data with synthetic stress tests, enabling controlled analysis of explanation behavior under spurious correlations.\u003c/p\u003e \u003cp\u003e(3) This study provides an empirical analysis under the considered regimes in our evaluated setting showing that explanation instability can increase even as predictive performance remains strong (often high) on the evaluated benchmarks, challenging common assumptions in explainable AI practice.\u003c/p\u003e \u003cp\u003eOperational addition: we contribute an operational reliability gate (a threshold rule over stability, sensitivity, and consistency) that can be reused as a plug-in stage in XAI benchmarks and auditing pipelines\u0026mdash;deciding when not to act on an explanation. We formalize a deployable reliability gate as a benchmark plug-in stage that converts reliability signals into an operational reject/accept decision with explicit risk\u0026ndash;coverage control.\u003c/p\u003e \u003cp\u003e \u003cb\u003eRelated work and positioning.\u003c/b\u003e Early explainable AI research focused on interpretability mechanisms such as feature attribution and saliency methods. More recent work has emphasized explanation robustness, faithfulness, and consistency, revealing that explanations can vary significantly across retraining runs or data perturbations.\u003c/p\u003e \u003cp\u003eDespite this progress, existing studies often evaluate reliability dimensions in isolation or under narrow experimental conditions. In contrast, this work aligns with recent calls for holistic empirical evaluation by jointly analyzing stability, sensitivity, and cross-model consistency within a unified experimental protocol. Recent work continues to refine robustness and faithfulness evaluation, including robust faithfulness metrics designed to reduce out-of-distribution artifacts in perturbation-based tests [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] and unified evaluation frameworks showing how explainer behavior depends on model architecture and dataset characteristics [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eA practical implication is that reliability checks need to be cheap enough to run routinely, not only in one-off case studies.\u003c/p\u003e \u003cp\u003eCost matters in practice. A protocol that is too expensive to run routinely will not survive contact with real auditing workflows.\u003c/p\u003e \u003cp\u003eA recurring challenge is that \u0026ldquo;explanation quality\u0026rdquo; is not a single construct: taxonomies distinguish transparency vs. post-hoc explanation, local vs. global scope, and multiple evaluation desiderata (faithfulness, stability, and usability) that can conflict in practice [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e], [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Recent empirical work therefore recommends separating the evaluation of explanation generation from the evaluation of explanation reliability under perturbations, because the latter can fail even when predictive performance is strong [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Our protocol follows this line by treating reliability as a first-class experimental target and by reporting trade-offs explicitly, rather than collapsing them into a single \u0026ldquo;best\u0026rdquo; explainer score.\u003c/p\u003e \u003cp\u003eWithin robustness-focused XAI, stability is often operationalized as agreement of explanations across retraining runs, bootstrap samples, or small input perturbations. Early evidence shows that many popular explainers can be brittle under such changes, motivating robustness tests for explanations themselves rather than only for predictions [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e], [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. At the same time, most robustness studies focus on local, post-hoc explainers (e.g., LIME/SHAP) [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e], whereas system auditing in tabular settings frequently relies on model-specific global importance signals that are cheaper to compute and easier to monitor over time [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e], [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Our study therefore complements prior robustness analyses by centering on global, model-specific importance and by structuring stress tests that mimic proxy-driven \u0026ldquo;audit failures.\u0026rdquo;\u003c/p\u003e \u003cp\u003eAnother relevant thread connects explanation reliability to dataset shift and spurious correlations. Shortcut learning and correlated predictors can yield deceptively stable explanations in-distribution, while degrading under shifts that change which features act as shortcuts [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Recent venue papers on trust and reliability highlight that explainability claims can become fragile when evaluation ignores such regimes, especially in high-stakes contexts where explanations may be used to justify deployment decisions [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e], [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. These observations motivate stress gradients that are not intended to emulate a single real domain, but to probe failure modes systematically by controlling proxy strength and noise.\u003c/p\u003e \u003cp\u003eFinally, there is growing consensus that interpretability should support operational decisions, not only retrospective narratives: calls to avoid over-reliance on post-hoc explanations emphasize the need for rigorous, reproducible evaluation and clear decision rules about when an explanation should not be acted upon [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e], [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. The reliability-gate perspective adopted here fits this operational framing: we focus on identifying regimes where explanation signals are sufficiently reliable to be used in model auditing, and we make the coverage\u0026ndash;risk trade-off explicit.\u003c/p\u003e \u003cp\u003eDesign note: we focus on global, model-specific importance because it is cheap to compute, easy to log, and commonly used in tabular audit reports.\u003c/p\u003e \u003cp\u003eWe set k\u0026thinsp;=\u0026thinsp;5 to match what auditors typically review in practice; Top-10 is reported as a sensitivity check (Supplementary Table S3).\u003c/p\u003e \u003cp\u003eWe also keep k and the perturbation schedule fixed across datasets to avoid any post-hoc tuning of explanations. In a real audit, these knobs are usually set once and reused for consistency.\u003c/p\u003e \u003cp\u003e \u003cb\u003eContributions.\u003c/b\u003e The primary contributions of this study are empirical and methodological rather than algorithmic. Specifically, the paper makes the following key contributions: (1) It reframes explanation reliability as an empirically measurable property and demonstrates how stability, sensitivity, and cross-model agreement jointly reveal failure modes that accuracy-based evaluation misses. (2) It introduces a hybrid evaluation design combining public real-world data with synthetic stress tests, enabling controlled analysis of explanation behavior under spurious correlations. (3) This study provides an empirical analysis under the considered regimes in our evaluated setting showing that explanation instability can increase even as predictive performance remains strong (often high) on the evaluated benchmarks, challenging common assumptions in explainable AI practice.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eFigure 1 illustrates explanation stability across datasets. Linear models remain stable, while tree-based models show stability that varies with the data regime: on the easy synthetic generators (including Syn-SpuriousShift-Easy) stability can appear inflated, whereas on real data it can drop due to sampling variance and correlated predictors. This highlights why stability must be interpreted jointly with the stress condition.\u003c/p\u003e\n\u003cp\u003eInterpretation: Logistic Regression (LR) exhibits consistently high stability across regimes\u0026mdash;typically highest on the real benchmarks\u0026mdash;because global attributions are directly tied to learned coefficients; repeated retraining under comparable splits yields highly consistent coefficient rankings. [16], [1]\u003c/p\u003e\n\u003cp\u003eIn contrast, RF stability varies by regime and can be affected by correlated predictors and sampling variance; synthetic low-noise settings may inflate stability. Accordingly, stability must be interpreted jointly with the stress condition rather than treated as a standalone guarantee of explanation robustness. [6], [1], [17]\u003c/p\u003e\n\u003cp\u003eInterpretation: Relative sensitivity (rESS) for LR is lower on BreastCancer-UCI than on Wine-UCI (Supplementary Table S3), indicating that mild perturbations can redistribute attribution magnitudes on some real tabular tasks even when output changes remain modest. This should be interpreted as a warning sign for explanation fragility rather than a performance metric, consistent with robustness concerns raised in prior work. [1], [2]\u003c/p\u003e\n\u003cp\u003eIn the synthetic regimes, rESS values increase substantially, reflecting the controlled data-generating process and reduced noise. This contrast suggests why rESS should be treated as a diagnostic tool and compared primarily within a model family across conditions, not as an absolute cross-model score. [1], [2]\u003c/p\u003e\n\u003cp\u003eInterpretation: Complete agreement in the simplified synthetic base regime can be trivial\u0026mdash;when the data-generating signal is highly separable, different models naturally recover the same dominant features. Thus, high consistency here reflects dataset simplicity, not guaranteed reliability. [17]\u003c/p\u003e\n\u003cp\u003eAgreement drops on the real datasets where correlated predictors and differing inductive biases yield multiple plausible feature subsets. By contrast, Syn-SpuriousShift-Easy intentionally yields high Top-k overlap (Supplementary Table S4) to illustrate trivial agreement; the harder non-triviality checks in Supplementary Table S5 show that agreement can deteriorate under interaction-dominant proxy structure even when accuracy remains competitive.\u003c/p\u003e\n\u003cp\u003eInterpretation: Predictive performance is generally strong across the two real benchmarks, which helps isolate explanation reliability effects. On synthetic stress tests, performance can vary by design (Supplementary Table S5), enabling us to probe reliability under controlled difficulty rather than confounding reliability with arbitrary accuracy differences.\u003c/p\u003e\n\u003cp\u003eDesign note: our stress tests and auditing protocol are not intended to optimize AUC; they are constructed to isolate explanation reliability effects under controlled variability and proxy/noise shifts.\u003c/p\u003e\n\u003cp\u003eNotably, a spurious-shift setup can maintain strong AUC while explanation behavior changes under stress. In our easy generator, agreement is intentionally high (Supplementary Table S4), whereas the harder variants in Supplementary Table S5 reduce agreement and can depress linear-model AUC\u0026mdash;demonstrating that accuracy alone does not certify explanation reliability.\u003c/p\u003e\n\u003cp\u003eFigure 2 reports explanation sensitivity (rESS). Notably, low rESS values indicate explanation fragility even when prediction changes are small, emphasizing that accuracy alone is insufficient to assess explanation quality.\u003c/p\u003e\n\u003cp\u003eWhy can rESS collapse while \u0026Delta;prediction remains small? \u0026Delta;prediction measures the average change in predicted probabilities, which can stay modest when the task is redundant (multiple correlated features support similar decisions). In contrast, rESS compares the distribution of global importance across features: under collinearity or near-equivalent feature sets, small changes in fitted parameters can redistribute attribution mass substantially without strongly moving the output. This coefficient/attribution instability has been repeatedly observed in robustness critiques of explanation methods and model interpretations [1], [2], [3].\u003c/p\u003e\n\u003cp\u003eFigure 3 shows cross-model consistency via Top-k overlap. On Syn-Base and the easy spurious-shift generator (Syn-SpuriousShift-Easy), agreement can remain high because the generator is highly separable\u0026mdash;this is precisely the triviality risk: high agreement does not imply that explanations are decision-relevant. To demonstrate non-triviality, we additionally report a harder synthetic check in Supplementary Table S5, where agreement drops when the proxy is only exploitable through interaction (trees can exploit it; linear models cannot).\u003c/p\u003e\n\u003cp\u003eSyn-SpuriousShift-Easy is designed to preserve trivial agreement (Supplementary Table S4); reliability degradation is demonstrated using the harder variants in Supplementary Table S5.\u003c/p\u003e\n\u003cp\u003eAdditional non-triviality check: the complete agreement observed in Syn-Base and Syn-SpuriousShift-Easy is intentionally retained as an example of \u0026ldquo;easy agreement\u0026rdquo; on highly separable generators. To show that our protocol does not force agreement, we also evaluate harder spurious-shift variants (Supplementary Table S5) where interaction and noise make the proxy partially exploitable; in these cases LR vs RF Top-5 agreement drops substantially and AUC can diverge, illustrating that high accuracy does not certify reliability.\u003c/p\u003e\n\u003cp\u003eFigure 4 confirms that predictive performance remains strong on the real benchmarks and competitive overall, reinforcing that explanation reliability must be evaluated independently of accuracy.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e[Figure 1 about here]\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eExpanded sensitivity reporting: in addition to rESS, we now report the underlying deviation E (mean\u0026plusmn;std) and a Top-10 variant (rESS_top10) that focuses on the most salient features (Supplementary Table S3). This directly addresses the concern that small \u0026Delta;prediction can coexist with large redistribution of attribution mass: rESS quantifies stability of the full importance distribution, while rESS_top10 provides a complementary sanity-check that concentrates on salient features.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e[Figure 2 about here]\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e[Figure 3 about here]\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e[Figure 4 about here]\u003c/em\u003e\u003c/p\u003e\n\u003ch2\u003eReliability-gated auditing\u003c/h2\u003e\n\u003cp\u003eWe simulate an auditing scenario with an impermissible proxy feature that is moderately correlated with the label during training, but is decorrelated at deployment (randomized). The auditor inspects global feature-importance signals and triggers a failure if the proxy appears among the Top-5 features. To create a non-trivial failure case, we (i) randomize proxy strength across seeds and (ii) add nuisance features to increase ranking variance. We repeat training across 20 seeds and two model families (LR/RF), yielding n=40 runs.\u003c/p\u003e\n\u003cp\u003eNote: Table 2 follows the auditing protocol (fixed split, n=40 runs), whereas Tables A2\u0026ndash;A4 report T=5 independent resampling runs.\u003c/p\u003e\n\u003cp\u003eUncertainty (bootstrap, 95% CI over runs; n=40): baseline audit error 42.5% [27.5, 57.5], gated audit error 0.0% [0.0, 0.0]. For AUC, baseline 0.995 [0.993, 0.997], gated 0.995 [0.992, 0.998]. Table 2 reports bootstrap CIs over run-level means under the fixed-split auditing protocol, whereas Supplementary Information (Supplementary Information B) reports Wilson CIs for run-level error proportions (including gated subsets).\u003c/p\u003e\n\u003cp\u003eInterpretation (consistent with Table 2 and Table 3): under the Moderate gate (q=0.60; \u0026tau;S=0.92, \u0026tau;E=0.831, \u0026tau;C=0.25), the audit-error rate drops to 0.0% on the retained explanations while coverage decreases to 22.5%. This is an explicit safety\u0026ndash;coverage trade-off: the gate rejects explanations that are unstable (low S), overly sensitive (low rESS@Top5), or poorly supported across model families (low C), before the auditor acts on them. Gate is a risk-control mechanism: it is expected to reduce coverage under stricter thresholds, and the operational choice is governed by an audit-error tolerance \u0026alpha;. Low coverage is an expected outcome of risk control; practitioners should select (q, \u0026tau;C) via the audit error\u0026ndash;coverage trade-off (Table 3) to meet a target tolerance \u0026alpha;. We therefore treat coverage as a first-class outcome, not a nuisance.\u003c/p\u003e\n\u003cp\u003eThresholds are calibrated from reliability quantiles (Table 3) rather than optimized for audit error.\u003c/p\u003e\n\u003cp\u003eNote: C is discrete when computed on Top-k overlaps; therefore small threshold changes can produce step-wise coverage shifts. We report three increasing regimes (Lenient/Moderate/Conservative) via quantiles on S and rESS while keeping \u0026tau;C aligned to each regime as shown in Table 3. Intermediate quantiles (e.g., q=0.65) can be used to densify the trade-off curve and are left for follow-up reporting.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e[Figure 5 about here]\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eAcross the two real datasets, Wine-UCI tends to show higher perturbation sensitivity (higher E and lower rESS/rESS_top10) than BreastCancer-UCI, even when AUC remains strong. This illustrates the paper\u0026rsquo;s central claim: explanation reliability should be assessed independently of predictive performance, because stable accuracy can mask unstable attribution patterns under mild shifts.\u003c/p\u003e\n\u003ch2\u003eWine vs Breast Cancer: What changes?\u003c/h2\u003e\n\u003cp\u003eAcross the same protocol (T=5 stratified resampling runs per dataset\u0026times;model), Wine-UCI exhibits a more volatile reliability profile than BreastCancer-UCI. While both datasets sustain high ROC-AUC, the stability and sensitivity signals tend to degrade more on Wine, reflecting its smaller sample size and the fact that it originates from a multi-class problem (binarized here). Practically, this means feature-importance rankings on Wine are more sensitive to split and retraining randomness, and perturbation stress can redistribute attribution mass more noticeably even when \u0026Delta;prediction remains modest. Cross-model consensus also changes: BreastCancer can admit multiple plausible feature subsets due to correlated predictors, whereas Wine may concentrate importance on fewer features but with higher run-to-run variability. The key takeaway is that explanation reliability is dataset-conditional; thresholds (\u0026tau;S, \u0026tau;E, \u0026tau;C) should be calibrated per deployment regime rather than transferred from a single benchmark.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eOperationally, acting on unreliable explanations introduces tangible risks in model auditing and governance. The reliability-gated auditing results provide evidence: baseline auditing can fail by surfacing spurious proxy features despite stable AUC, while the gate reduces audit error by screening out unstable, overly sensitive, or cross-model-inconsistent explanations. This screening is a risk-control mechanism\u0026mdash;coverage may decrease under stricter thresholds by design\u0026mdash;and practitioners should choose (q, \u0026tau;C) using the audit error\u0026ndash;coverage trade-off (Table 3) to satisfy an operational tolerance \u0026alpha;. We intentionally keep the gate as a simple conjunctive rule so it can be implemented without training an additional classifier. In practice, this also makes threshold choices auditable.\u003c/p\u003e\n\u003cp\u003eVery high cross-model agreement on simplified synthetic regimes is intentionally retained as a cautionary example: high agreement can be trivial when the data-generating signal is highly separable. This can create a false sense of trust, masking fragility once interaction-dominant proxies, nuisance noise, or regime shifts are introduced (Supplementary Table S5).\u003c/p\u003e\n\u003cp\u003eIn other words, agreement can be an artifact of task simplicity. That is exactly what the stress gradient is meant to reveal.\u003c/p\u003e\n\u003cp\u003eThe main takeaway is that reliability metrics expose failure modes that standard predictive evaluation does not, consistent with robustness critiques showing that global importance signals can be unstable or manipulable even when models are accurate [1], [2]\u0026ndash;[5]. Accordingly, we advance one focused, testable claim: explanation reliability should be evaluated before explanation usefulness. To decouple reliability effects from accuracy, we combine high-AUC real benchmarks with complementary synthetic stress gradients (Supplementary Table S5) that systematically vary proxy interaction and nuisance noise.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eImplications for XAI Benchmarks\u0026nbsp;\u003c/strong\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eBenchmark suites can adopt a reliability stage by explicitly reporting stability/sensitivity/consistency under stress tests, instead of treating a single explanation score as sufficient. This complements recent calls for transparent evaluation and comparable benchmarking practice [13]\u0026ndash;[10].\u003c/p\u003e\n\u003cp\u003ePractical insertion into an auditing pipeline (high-level): (i) train the model and compute global, model-specific importance; (ii) repeat training under the resampling/seed protocol to estimate stability S and cross-model consistency C; (iii) run the perturbation/noise stress to estimate rESS@Top5; (iv) calibrate thresholds from reliability quantiles (Table 3) to match an operational audit-error tolerance \u0026alpha; (rather than tuning for minimum error); (v) apply the reliability gate to screen explanations before an auditor acts on them; and (vi) log the resulting audit error\u0026ndash;coverage trade-off and monitor it over time as data drift or retraining updates occur. This provides a lightweight governance artifact that can be reviewed alongside standard performance reports in applied IEEE settings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFailure modes and operational guidance for the reliability gate\u0026nbsp;\u003c/strong\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFailure modes (what the gate does NOT guarantee): (i) Trivial agreement\u0026mdash;if models learn the same shortcut, C may be high while explanations remain misleading; (ii) Stability without robustness\u0026mdash;S can be high even when attributions shift substantially under targeted perturbations (low rESS@Top5), especially under collinearity; (iii) Robustness without consensus\u0026mdash;rESS may be high for a single model family yet C is low because alternative inductive biases emphasize different, equally predictive feature sets. Practitioner guidance: choose q and \u0026tau;C as a risk-control knob. Lower q (Lenient) increases coverage but tolerates more unreliable explanations; higher q (Conservative) reduces coverage and should be used when acting on explanations carries higher governance risk. We recommend reporting the full audit error\u0026ndash;coverage curve (Table 3) and selecting (q, \u0026tau;C) to satisfy an operational constraint such as audit error \u0026le; \u0026alpha; at maximum achievable coverage.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e7. Limitations\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eReviewer-facing clarification\u003c/p\u003e\n\u003cp\u003eThresholds are calibrated from empirical reliability quantiles (Table 3) rather than optimized for audit error, and the gate is expected to trade coverage for reduced audit risk under stricter regimes.\u003c/p\u003e\n\u003cp\u003eThe reported high AUC on the real benchmarks is intentional to decouple explanation reliability from predictive performance; the synthetic stress gradient (Supplementary Table S5) is used to probe reliability under controlled difficulty shifts.\u003c/p\u003e\n\u003cp\u003eThis study is limited to tabular data and global attribution signals. While the observed reliability patterns are transferable, no claim of universal generalization across modalities is made. The selected real benchmarks yield high ROC-AUC, which helps isolate reliability effects but limits conclusions about low-accuracy regimes.\u003c/p\u003e\n\u003cp\u003eWe intentionally use LR and RF as canonical model families in the main protocol; extending the same reliability tests to boosting (e.g., gradient-boosted trees such as XGBoost) and neural models is left as future work.\u003c/p\u003e\n\u003cp\u003eWe focus on global, model-specific importance; local explainers and post-hoc methods are future work. We also note that the main experiments focus on model-specific importance signals; extending the same reliability protocol to a broader set of attribution methods (e.g., permutation-based or Shapley-style explainers) is an important next step.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eIn our evaluated setting, we show that explanation reliability is a distinct and empirically measurable dimension of model behavior. By combining public real-world benchmarks with synthetic stress gradients, our protocol exposes explanation failure modes that remain hidden under conventional accuracy-focused evaluation. We further show that a simple threshold-based reliability gate can act as an operational risk-control mechanism: it can reduce audit error by screening out unstable, overly sensitive, or cross-model-inconsistent explanation signals, at the expected cost of reduced coverage under stricter thresholds.\u003c/p\u003e\n\u003cp\u003eFrom an applied perspective, this supports reliability-aware auditing in practice\u0026mdash;for example, screening global feature-importance reports in a clinical triage model before changing treatment pathways, or in a credit decision system before issuing adverse-action rationales. Importantly, the empirical results reported here are specific to tabular settings using global, model-specific importance on LR/RF; we do not evaluate vision/text modalities or local/post-hoc explainers. Extending the same reliability protocol to additional model families, modalities, and explanation types is a natural direction for future work.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eWe treat explanation reliability as a pre-use screening problem: the goal is to decide whether an available explanation signal is stable enough to support auditing or decision support, rather than proposing a new explainer. This stance follows calls for rigorous, testable interpretability science [6], [4].\u003c/p\u003e\n\u003cp\u003eOur protocol uses three stress lenses\u0026mdash;stability, sensitivity, and cross-model consistency\u0026mdash;motivated by evidence that explanations can be fragile to perturbations and manipulations [1], [2]\u0026ndash;[13], and that agreement can be trivial on overly simplified tasks [17].\u003c/p\u003e\n\u003ch2\u003eDefinitions\u003c/h2\u003e\n\u003cp\u003eLet f denote a trained predictor and E(f, x) produce an explanation vector e \u0026isin; ℝ^d (e.g., feature attributions). Over repeated trials t=1\u0026hellip;T:\u003c/p\u003e\n\u003cp\u003e\u0026bull; Stability (S): average Spearman rank correlation between explanation vectors across trials; higher S implies less sensitivity to retraining randomness. We chose Spearman to compare rankings rather than magnitudes, which is stable under monotone rescaling.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Relative sensitivity (rESS): for baseline explanation e and perturbed \u0026ecirc;, define the relative deviation E = ||e\u0026minus;\u0026ecirc;||₁ / (||e||₁+\u0026epsilon;). We report a bounded robustness score rESS = 1/(1+E) \u0026isin; (0,1], where higher values indicate explanations change less under controlled perturbations.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Consistency (C): Jaccard overlap between top-k feature sets from explanations produced by alternative model families trained on the same task. We use Jaccard because it is simple, discrete, and directly interpretable as shared features.\u003c/p\u003e\n\u003ch2\u003eThreshold-based reliability rule\u003c/h2\u003e\n\u003cp\u003eReliable(e) \u0026hArr; (S \u0026ge; \u0026tau;S) \u0026and; (rESS@Top5 \u0026ge; \u0026tau;E) \u0026and; (C \u0026ge; \u0026tau;C). For the auditing experiment (Wine-UCI) we use the Moderate operational setting (q=0.60; \u0026tau;S=0.92, \u0026tau;E=0.831, \u0026tau;C=0.25), estimated from quantiles over n=40 runs (20 seeds \u0026times; 2 model families). Here rESS@Top5 is computed from the relative deviation on the baseline Top-5 importance scores under a stress perturbation that randomizes the proxy and injects feature noise.\u003c/p\u003e\n\u003ch2\u003eComposite Reliability Index (supplementary)\u003c/h2\u003e\n\u003cp\u003eFor reporting only, we summarize the three dimensions as CRI = wS\u0026middot;S + wE\u0026middot;rESS + wC\u0026middot;C with wS+wE+wC=1. Decisions rely on the explicit threshold rule above.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4. Experimental Setup\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eSpecifically, we use the Wine dataset from the UCI Machine Learning Repository. The dataset contains 178 instances, 13 continuous physicochemical attributes, and three classes corresponding to wine cultivars. To align with the binary setting used in our reliability metrics and auditing rule, we form a binary task (class 1 vs. {0,2}), treating class 1 as the positive label (positive ratio \u0026asymp; 0.331, Table 1). All features are numeric with no missing values; features are standardized within each training split. [18], [19]\u003c/p\u003e\n\u003cp\u003eUnlike the Breast Cancer dataset, Wine exhibits higher feature heterogeneity, class imbalance, and non-linear decision boundaries. This dataset therefore provides a more challenging and realistic testbed for evaluating explanation reliability under feature sparsity, encoding choices, and distributional complexity. The same evaluation protocol is applied without modification to ensure comparability. [18], [19]\u003c/p\u003e\n\u003cp\u003eReproducibility note: main-results tables/figures use T=5 independent stratified resampling runs per dataset\u0026times;model (each run draws a new train/test split; seeds 0\u0026ndash;4), reporting mean\u0026plusmn;std over runs. Auditing uses a fixed stratified split on Wine-UCI and varies only the random seed controlling model training randomness and proxy-strength generation; we train LR and RF over 20 seeds, giving n=40 independent model\u0026times;seed runs. Bootstrap 95% confidence intervals are computed by resampling these runs with replacement and taking the mean audit-error (or AUC) across resampled runs. All auditing runs share the same split; only model randomness and proxy-strength differ. We keep the split fixed in auditing so that uncertainty reflects run-to-run randomness rather than changing data partitions. We picked T=5 because it provides a small but useful variance estimate while keeping the protocol lightweight. Practitioners can increase T if they have more compute or need tighter intervals.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eCode Availability:\u003c/h2\u003e\n\u003cp\u003eThe code to reproduce all experiments, tables, and figures is maintained in a private repository during peer review. Access can be granted to the Editor and reviewers upon request. The repository will be made publicly available upon acceptance.\u003c/p\u003e\n\u003cp\u003eWe choose LR and RF as canonical linear vs. tree inductive biases; adding boosting/NNs is left as future work. This choice isolates contrasting inductive biases while keeping the protocol reproducible; extending to boosting/NNs is straightforward within the same protocol.\u003c/p\u003e\n\u003ch2\u003eUse of generative AI tools\u003c/h2\u003e\n\u003cp\u003eWe used a large language model tool for language polishing and formatting assistance. All technical content, experimental choices, and reported numbers were reviewed and verified by the authors.\u003c/p\u003e\n\u003ch2\u003eData Availability\u003c/h2\u003e\n\u003cp\u003eAll real datasets used in this study are publicly available: the UCI Wine dataset and the Breast Cancer Wisconsin dataset. Synthetic datasets are generated from the procedures described in the Methods and the Supplementary Information.\u003c/p\u003e\n\u003ch2\u003eCode Availability\u003c/h2\u003e\n\u003cp\u003eThe experimental code, dataset generators, and plotting scripts are maintained in a private repository during peer review and will be released publicly upon acceptance. (Repository URL: https://github.com/)\u003c/p\u003e\n\u003ch2\u003eCompeting Interests\u003c/h2\u003e\n\u003cp\u003eThe author declare no competing interests.\u003c/p\u003e\n\u003ch2\u003eFunding Declaration\u003c/h2\u003e\n\u003cp\u003eThis research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.\u003c/p\u003e\n\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\n\u003cp\u003eMOHAMMED ALSHANTI conceived the study, designed the experiments, implemented the protocol, analyzed the results, and wrote the manuscript.\u003c/p\u003e\n\u003ch2\u003eAcknowledgements\u003c/h2\u003e\n\u003cp\u003eWe thank the reviewers for their constructive feedback.\u003c/p\u003e\n\u003cp\u003eMOHAMMED ALSHANTI conceived the study, designed the experiments, implemented the protocol, analyzed the results, and wrote the manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAlvarez-Melis, D. \u0026amp; Jaakkola, T. S. On the Robustness of Interpretability Methods. In Proc. ICML Workshop on Human Interpretability in Machine Learning (WHI), Stockholm, Sweden, Jul. 2018. [Online]. Available: https://sites.google.com/view/whi2018/home. Accessed: Jan. 2026. (Preprint: arXiv:1806.08049.).\u003c/li\u003e\n\u003cli\u003eYeh, C.-K., Hsieh, C.-Y., Suggala, A. S., Inouye, D. I. \u0026amp; Ravikumar, P. On the (In)fidelity and Sensitivity of Explanations. In Proc. NeurIPS, 2019, pp. 10965\u0026ndash;10976.\u003c/li\u003e\n\u003cli\u003eGhorbani, A., Abid, A. \u0026amp; Zou, J. Interpretation of Neural Networks Is Fragile. Proc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, https://doi.org/10.1609/aaai.v33i01.33013681.\u003c/li\u003e\n\u003cli\u003eRudin, C. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nat. Mach. Intell., vol. 1, pp. 206\u0026ndash;215, 2019, https://doi.org/10.1038/s42256-019-0048-x.\u003c/li\u003e\n\u003cli\u003eVilone, G. \u0026amp; Longo, L. Notions of explainability and evaluation approaches for explainable artificial intelligence. Inf. Fusion, vol. 76, pp. 89\u0026ndash;106, 2021, https://doi.org/10.1016/j.inffus.2021.05.009.\u003c/li\u003e\n\u003cli\u003eLisboa, P. J. G., Saralajew, S., Vellido, A., Fern\u0026aacute;ndez-Domenech, R. \u0026amp; Villmann, T. The coming of age of interpretable and explainable machine learning models. Neurocomputing, vol. 535, pp. 25\u0026ndash;39, 2023, https://doi.org/10.1016/j.neucom.2023.02.040.\u003c/li\u003e\n\u003cli\u003eLe, H., Ghazanfari, R., Cheung, S. C. \u0026amp; Salim, F. D. Benchmarking Explainable Artificial Intelligence: A Survey of Evaluation Methods and Practices. In Proc. IJCAI, 2023, pp. 6734\u0026ndash;6742, https://doi.org/10.24963/ijcai.2023/750.\u003c/li\u003e\n\u003cli\u003eSchwalbe, G. \u0026amp; Finzel, B. A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts. Data Mining Knowl. Discov., vol. 38, no. 5, pp. 3043\u0026ndash;3101, 2024, https://doi.org/10.1007/s10618-022-00867-8.\u003c/li\u003e\n\u003cli\u003eVascotto, I. et al. When Can You Trust Your Explanations? A Robustness Analysis on Feature Importances. In Explainable Artificial Intelligence (xAI 2025), Commun. Comput. Inf. Sci., vol. 2578, pp. 225\u0026ndash;249, Oct. 2025, https://doi.org/10.1007/978-3-032-08327-2_11.\u003c/li\u003e\n\u003cli\u003eRibeiro, M. T., Singh, S. \u0026amp; Guestrin, C. Why Should I Trust You?: Explaining the Predictions of Any Classifier. In Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD), 2016, pp. 1135\u0026ndash;1144, https://doi.org/10.1145/2939672.2939778.\u003c/li\u003e\n\u003cli\u003eLundberg, S. M. \u0026amp; Lee, S.-I. A Unified Approach to Interpreting Model Predictions. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2017, pp. 4765\u0026ndash;4774.\u003c/li\u003e\n\u003cli\u003ePruthi, D., Gupta, M., Dhingra, B., Neubig, G. \u0026amp; Lipton, Z. C. Learning to Deceive with Attention-Based Explanations. In Proc. 58th Annu. Meeting Assoc. Comput. Linguist. (ACL), 2020, pp. 4782\u0026ndash;4793, https://doi.org/10.18653/v1/2020.acl-main.432.\u003c/li\u003e\n\u003cli\u003eAgarwal, C., Ley, D., Krishna, S., Saxena, E., Pawelczyk, M., Johnson, N., Puri, I., Zitnik, M. \u0026amp; Lakkaraju, H. OpenXAI: Towards a Transparent Evaluation of Model Explanations. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), Datasets and Benchmarks Track, 2022. [Online]. Available: proceedings.neurips.cc/paper_files/paper/2022/hash/65398a0eba88c9b4a1c38ae405b125ef-Abstract-Datasets_and_Benchmarks.html. Accessed: Jan. 2026.\u003c/li\u003e\n\u003cli\u003eZheng, X., Shirani, F., Chen, Z., Lin, C., Cheng, W., Guo, W. \u0026amp; Luo, D. F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI. In Proc. Int. Conf. Learn. Representations (ICLR), 2025. [Online]. Available: openreview.net/forum?id=X0r4BN50Dv. Accessed: Jan. 2026.\u003c/li\u003e\n\u003cli\u003eDehdarirad, T. Evaluating explainability in language classification models: A unified framework incorporating feature attribution methods and key factors affecting faithfulness. Data and Information Management, vol. 9, no. 4, p. 100101, Dec. 2025, https://doi.org/10.1016/j.dim.2025.100101.\u003c/li\u003e\n\u003cli\u003eMolnar, C. Interpretable Machine Learning, 2nd ed., 2022. [Online]. Available: christophm.github.io/interpretable-ml-book/. Accessed: Jan. 2026.\u003c/li\u003e\n\u003cli\u003eGeirhos, R. et al. Shortcut Learning in Deep Neural Networks. Nat. Mach. Intell., vol. 2, no. 11, pp. 665\u0026ndash;673, 2020, https://doi.org/10.1038/s42256-020-00257-z.\u003c/li\u003e\n\u003cli\u003eAeberhard, S. \u0026amp; Forina, M. Wine. UCI Machine Learning Repository, 1991. https://doi.org/10.24432/C5PC7J.\u003c/li\u003e\n\u003cli\u003eWolberg, W. H., Street, W. N. \u0026amp; Mangasarian, O. L. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1995. https://doi.org/10.24432/C5DW2B.\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003e\u003cstrong\u003eTable 1 | Datasets used in the study (real benchmarks and synthetic stress tests).\u003c/strong\u003e\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eDataset\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eType\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eSource\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eSamples\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eFeatures\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePositive ratio\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eBreastCancer-UCI\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eReal (public)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eUCI / scikit-learn\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e569\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e30\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e0.6274165202108963\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eWine-UCI\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eReal (public; binarized)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eUCI / scikit-learn\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e178\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e13\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e0.33146067415730335\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eSyn-Base\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eSynthetic\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eThis study\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e2500\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e18\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e0.5\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eSyn-SpuriousShift-Easy\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eSynthetic\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eThis study\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e2500\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e19\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e0.5\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 2 | Reliability-gated auditing outcomes on Wine-UCI under proxy shift (mean over n=40 runs; 95% CI reported below).\u003c/strong\u003e\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCondition\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTest ROC-AUC\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eAudit error (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCoverage (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003eBaseline (no gate)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e0.995\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e42.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e100.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003eReliability-gated\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e0.995\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e22.5\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 3 | Threshold sensitivity analysis (audit error vs. coverage trade-off).\u003c/strong\u003e\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eRegime\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eQuantile q\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026tau;S\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026tau;E (rESS)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026tau;C\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGated audit error (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCoverage (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGated AUC\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003eLenient\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.91\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.815\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.20\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e16.7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e30.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.996\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003eModerate\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.92\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.831\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e22.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.995\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003eConservative\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.70\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.93\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.848\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.30\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e15.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e0.995\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Explainable AI, reliability, auditing, stability, sensitivity, consistency","lastPublishedDoi":"10.21203/rs.3.rs-8554983/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8554983/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis paper presents a reliability-oriented empirical study of global importance signals in machine learning models, focusing on stability, sensitivity, and cross-model consistency rather than proposing a new explanation algorithm. (i) Across two public real-world datasets and controlled synthetic stress tests, we show that explanation reliability can degrade even when predictive performance remains strong (often high); (ii) we provide empirical evidence that stability, sensitivity, and cross-model agreement capture complementary failure modes overlooked by accuracy-centric evaluation; and (iii) we discuss implications for reliability-aware XAI benchmarking under spurious correlations and retraining variability. These findings treat explanation reliability as a primary empirical property in our evaluated setting and offer practical guidance under the considered regimes for building and using future XAI benchmarks.\u003c/p\u003e","manuscriptTitle":"Reliability-Oriented Evaluation of Explanation Signals under Real and Synthetic Stress Tests","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-01-16 18:51:50","doi":"10.21203/rs.3.rs-8554983/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"75f5b877-14c4-4eab-bbe4-51d4b64c3986","owner":[],"postedDate":"January 16th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":61194969,"name":"Physical sciences/Mathematics and computing"},{"id":61194970,"name":"Biological sciences/Psychology"},{"id":61194971,"name":"Social science/Psychology"}],"tags":[],"updatedAt":"2026-03-30T05:56:48+00:00","versionOfRecord":[],"versionCreatedAt":"2026-01-16 18:51:50","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8554983","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8554983","identity":"rs-8554983","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}