Feature Forgetting: A Novel Approach to Redundant Feature Pruning in Automated Feature Engineering

Feature Forgetting: A Novel Approach to Redundant Feature Pruning in Automated Feature Engineering | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Feature Forgetting: A Novel Approach to Redundant Feature Pruning in Automated Feature Engineering Dr.Abhijit Patankar, Pranav Patil, Mihir Brahmane This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7130210/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The effectiveness of machine learning models heavily depends on the quality and relevance of the features extracted from raw data. Automated Feature Engineering (AutoFE) offers a scalable solution by generating large pools of candidate features. However, the unfiltered expansion of features introduces redundancy, exacerbates computational overhead, and may impair model generalization due to the curse of dimensionality. To address this, we propose Feature Forgetting, a proactive, dynamic pruning mechanism that discards redundant or non-informative features during the feature construction phase itself. Unlike conventional post-hoc feature selection methods such as Lasso or Recursive Feature Elimination, our approach integrates feature relevance evaluation into the generation loop using mutual information, variance inflation factor, and Shannon entropy. Empirical evaluations on three benchmark datasets — credit risk prediction, customer churn analysis, and healthcare diagnostics — reveal that Feature Forgetting can reduce feature dimensionality by up to 50%, cut training time by 40%, and improve classification accuracy by as much as 5%. These findings support the utility of real-time feature curation in constructing interpretable and efficient AutoML pipelines. Artificial Intelligence and Machine Learning Theoretical Computer Science Automated Feature Engineering Dynamic Feature Pruning Mutual Information Curse of Dimensionality Feature Selection Computational Efficiency Model Interpretability Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 1. Introduction In the realm of machine learning, feature engineering plays a pivotal role in determining the efficacy and interpretability of predictive models. While conventional feature engineering techniques rely on manual intervention guided by domain expertise, they are often time-intensive, prone to bias, and difficult to scale across varying datasets and problem domains [1]. With the advent of Automated Feature Engineering (AutoFE), significant strides have been made in automating the discovery of relevant transformations and interactions among input variables, enabling faster model development with reduced human involvement [2], [3] However, AutoFE techniques suffer from a critical drawback — combinatorial feature explosion. As transformations are recursively applied to the original feature set, the resultant feature space grows exponentially, often comprising thousands of features, many of which are either redundant or weakly correlated with the target outcome [4]. This phenomenon not only leads to overfitting but also imposes substantial computational burdens, especially during training and hyperparameter optimization phases. Existing solutions address this issue via post-hoc feature selection methods — such as Lasso regression, Recursive Feature Elimination (RFE), or Principal Component Analysis (PCA) — which are applied after the entire feature space has already been constructed [5], [6]. These reactive approaches, while effective to some extent, fail to mitigate the upfront cost of generating redundant features. Moreover, their black-box nature often compromises model interpretability, a crucial aspect in high-stakes domains like finance and healthcare. To bridge this gap, we introduce a novel, proactive paradigm termed Feature Forgetting, which integrates real-time feature evaluation into the AutoFE pipeline. Rather than constructing a bloated feature space and trimming it later, Feature Forgetting continuously discards uninformative or redundant features during the generation process itself. This results in a compact and meaningful feature set, leading to faster model training, improved generalization, and enhanced interpretability. The key contributions of this work are as follows: We formalize the Feature Forgetting paradigm for dynamic, real-time pruning of low-utility features during automated feature generation. We introduce an evaluation framework combining Mutual Information (MI), Variance Inflation Factor (VIF), and Shannon Entropy (H) to assess feature relevance and redundancy. We demonstrate, through empirical experiments on benchmark datasets, that Feature Forgetting significantly improves computational efficiency, dimensionality control, and predictive performance. We provide a generalized implementation strategy that is compatible with state-of-the-art AutoFE libraries and can be embedded into automated machine learning (AutoML) workflows. 2. Background The foundation of this research lies at the intersection of automated feature engineering, feature redundancy mitigation, and dimensionality reduction in high-dimensional learning tasks. This section introduces the theoretical context underpinning Feature Forgetting, identifies gaps in current solutions, and highlights why a proactive approach is necessary. 2.1 Feature Engineering and Its Role in Machine Learning Feature engineering refers to the process of transforming raw data into structured inputs that enhance a model’s learning capacity. This process often involves encoding, transforming, aggregating, or constructing interaction terms based on domain knowledge and data characteristics. Effective feature engineering has been shown to substantially improve model generalization, interpretability, and computational performance [ 1 ]. However, manual feature engineering is: Non-scalable across diverse datasets and domains, Error-prone due to human bias or oversight, Time-consuming, often requiring extensive preprocessing pipelines. The challenges have led to the rise of Automated Feature Engineering (AutoFE), a subfield of AutoML, that aims to automate the discovery and construction of relevant features. 2.2 Advances and Challenges in Automated Feature Engineering AutoFE tools such as Deep Feature Synthesis (DFS) [ 7 ], Relational Learning [ 5 ], and recent transformer-based AutoFE [ 8 ] use a sequence of transformations and join operations to build thousands of candidate features. These features are constructed using: Aggregations (e.g., mean, sum), Time-based lags, Cross-entity joins, Mathematical transformations (e.g., log, square root), Logical and relational combinations. While such systems remove the burden of manual feature creation, they lack filters at generation time, producing an unwieldy number of features. As the number of features grows, models face: Increased training complexity (O(n·p) where n = samples, p = features), Higher risk of overfitting (especially in low sample-size regimes), Difficulty in model explainability due to many correlated predictors. Even after applying post-hoc feature selection, the computational cost of generating unnecessary features remains significant. 2.3 The Curse of Dimensionality and Feature Redundancy The curse of dimensionality (Bellman, 1957) describes how the volume of the input space grows exponentially with the number of dimensions, rendering data increasingly sparse. As dimensionality increases: The distance between data points becomes uniform, reducing the discriminative power of models [ 9 ]. Traditional distance-based learning methods (e.g., KNN, SVM) degrade in performance. Computational costs grow nonlinearly due to increased matrix operations and optimization steps. Feature redundancy further exacerbates the problem — redundant features provide overlapping or duplicated information, inflating variance without improving bias. As shown in Fig. 1 (below), standard AutoFE workflows ignore redundancy until feature selection is applied after the generation phase. To address this, Feature Forgetting incorporates real-time redundancy analysis during feature creation — using metrics such as Mutual Information, Variance Inflation Factor, and Shannon Entropy — to drop uninformative or dependent features before they enter the model. 3. Methodology The proposed Feature Forgetting paradigm introduces a proactive mechanism to identify and discard irrelevant or redundant features during the automated feature generation process, rather than as a post-processing step. This section delineates the architecture, scoring function, and experimental setup used to implement and evaluate the framework. 3.1 Overview of the Feature Forgetting Framework The overall architecture of Feature Forgetting comprises three primary components: Feature Generation Online Evaluation and Scoring Dynamic Feature Pruning These components work in a continuous loop until a convergence or time threshold is met. The objective is to allow only statistically significant, non-redundant, and informative features to survive during the evolution of the feature space. 3.2 Feature Generation The framework supports any standard AutoFE mechanism such as: Deep Feature Synthesis (DFS), Transformation-based operators (logarithmic, interaction, polynomial), Relational joins and aggregations over grouped entities. Let \(\:{\mathcal{F}}_{\mathcal{t}}\:\) represent the set of features at iteration \(\:t\) . A transformation function \(\:{T}_{k}\left(\cdot\:\right)\) is applied to either a raw feature or a previously generated feature to create a new candidate: $$\:{f}_{t}^{\left(i\right)}={T}_{k}\left({f}_{t-1}^{\left(j\right)}\right)$$ where \(\:{f}_{t}^{\left(i\right)}\in\:\mathcal{F}t\:\) and \(\:ft-{1}^{\left(j\right)}\in\:{\mathcal{F}}_{\mathcal{t}-1}\) , with \(\:{T}_{k}\in\:\mathcal{T}\) being the transformation space. 3.3 Feature Evaluation: Real-Time Scoring For each newly generated feature \(\:{f}_{i}\) , we compute an aggregate importance score using three metrics: Mutual Information \(\:MI\left({f}_{i},y\right)\) — quantifies dependency with the target variable. Variance Inflation Factor \(\:VIF\left({f}_{i}\right)\) — assesses multicollinearity with existing features. Shannon Entropy \(\:H\left({f}_{i}\right)\) — evaluates the distributional informativeness. We define the composite score \(\:S\left({f}_{i}\right)\) as: $$\:S\left({f}_{i}\right)={\alpha\:}\cdot\:MI\left({f}_{i},y\right)-{\beta\:}\cdot\:VIF\left({f}_{i}\right)+{\gamma\:}\cdot\:H\left({f}_{i}\right)$$ where \(\:\alpha\:,\beta\:,\gamma\:\in\:{R}^{+}\:\) are hyperparameters that weight the importance of each metric (tuned during validation). Features with scores \(\:S\left({f}_{i}\right)<\) \(\:\tau\:\:\) (a dynamic threshold) are discarded immediately from the evolving feature pool. 3.4 Dynamic Feature Pruning Logic The pruning process is adaptive and evolves as new features are created and evaluated. Threshold \(\:{\tau\:}\:\) is updated using: $$\:{{\tau\:}}_{t+1}={\delta\:}\cdot\:\text{mean}\left(S\left({f}_{i}\right)∣{f}_{i}\in\:{\mathcal{F}}_{\mathcal{t}}\right)$$ where \(\:{\delta\:}\in\:\left(\text{0,1}\right]\:\) is a decay factor. This formulation enables the system to be robust to early-stage score variance and aggressive in later stages as better feature candidates are discovered. 3.5 Pseudocode Here is the pseudocode for the core Feature Forgetting loop: Initialize raw features F0 Set iteration t = 0 While not convergence: Generate candidate features Ft using transformations T For each feature f_i in Ft: MI = compute_mutual_information(f_i, target) VIF = compute_vif(f_i, Ft \ f_i) H = compute_entropy(f_i) S[f_i] = α * MI - β * VIF + γ * H Update τ = δ * mean(S[f_i] for f_i in Ft) Prune features: Ft = {f_i for f_i in Ft if S[f_i] ≥ τ} Add surviving features to pool t ← t + 1 Architecture of Feature Forgetting. Features are scored in real-time and discarded if they fall below an adaptive threshold τ. Only relevant features survive to the final pool as in Fig. 2 . 3.6 Experimental Setup (to transition into next section) We implement Feature Forgetting within a Python-based AutoFE pipeline using Scikit-learn and Featuretools. To validate its effectiveness, we conduct comparative experiments across benchmark datasets and standard baselines as described in Section 4 . 4. Experiments and Evaluation This section outlines the datasets employed, the baseline models selected for comparison, the experimental protocol followed, and the metrics used for evaluation. All experiments are designed to rigorously assess the computational and predictive gains achieved by integrating the Feature Forgetting paradigm into the AutoFE process. 4.1 Datasets Three benchmark datasets sourced from the UCI Machine Learning Repository were utilized to represent a diversity of real-world application domains Table 1 : Table 1 Dataset Information Dataset Domain Samples (n) Features (p) Target Variable Credit Risk Finance 50,000 20 Default status (binary) Customer Churn Telecommunications 7,043 19 Churn indicator (binary) Healthcare Diagnosis Medical 11,000 25 Disease presence (binary) All datasets underwent standard preprocessing steps: Missing value imputation, One-hot encoding for categorical variables, Standardization (zero-mean, unit-variance) for numerical features. To simulate real-world AutoFE settings, a transformation budget of 500 generated features per dataset was initially allowed. 4.2 Baseline Approaches The proposed Feature Forgetting framework was compared against the following: Standard AutoFE (no feature pruning during generation). AutoFE + Post-hoc Feature Selection using: Lasso Regression (L1 penalty), Recursive Feature Elimination (RFE) with a Random Forest estimator. Each method was embedded within the same machine learning pipeline to ensure fair comparisons. 4.3 Experimental Protocol Key experimental settings included: ML Algorithms: Random Forest Classifier (default 100 trees), Logistic Regression. Training/Test Split: 70% training, 30% testing, stratified by class. Evaluation Repetitions: Each experiment was repeated 10 times with different random seeds to account for stochasticity. Hardware: Experiments were conducted on an NVIDIA A100 GPU machine with 512GB RAM and 64 CPUs to minimize resource bottlenecks. Software: Python 3.10, Scikit-learn 1.3, Featuretools 1.0. Hyperparameters (e.g., \(\:{\alpha\:},\:{\beta\:},\:{\gamma\:},\:{\delta\:}\) ) in Feature Forgetting were tuned via 5-fold cross-validation on the training set only. 4.4 Evaluation Metrics The following metrics were measured: • Classification Accuracy (%): $$\:\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$ • Training Time Reduction (%): $$\:\text{Reduction}=\frac{{T}_{\text{standard}}-{T}_{\text{method}}}{{T}_{\text{standard}}}\times\:100$$ • Dimensionality Reduction (%): $$\:\text{Reduction}=\frac{{p}_{\text{standard}}-{p}_{\text{method}}}{{p}_{\text{standard}}}\times\:100$$ where \(\:T\) denotes training time and p denotes feature count. Standard deviations across experimental runs were also computed to assess stability. 4.5 Results Summary Table 2 Comparison of feature counts, classification accuracy, and training time reduction before and after applying the Feature Forgetting framework across three different datasets. Table 2 Results Comparison Dataset Standard Features Feature Forgetting Features Accuracy Before (%) Accuracy After (%) Training Time Reduction (%) Credit Risk 120 68 82.3 86.1 42% Customer Churn 95 52 76.8 81.2 39% Healthcare 150 72 79.5 84.3 45% 5. Results and Analysis This section presents a comprehensive analysis of the results obtained through the empirical evaluation of Feature Forgetting across three benchmark datasets. The findings demonstrate the method’s effectiveness in reducing dimensionality, minimizing computational burden, and enhancing model predictive performance. 5.1 Dimensionality Reduction The most immediate and visible benefit of Feature Forgetting lies in its ability to curb feature bloat during generation. As shown in Fig. 3 , across all datasets, the number of features retained after proactive pruning was reduced by approximately 35–50% compared to the baseline. This reduction is achieved without requiring post-processing steps like Lasso or RFE, which still depend on an initially bloated feature pool. Thus, Feature Forgetting avoids both the wasteful generation and delayed correction phases that plague traditional AutoFE. 5.2 Training Time Reduction By avoiding the generation of low-utility features, Feature Forgetting achieved training time reductions of up to 45%. The average improvement across datasets was 42%, which is significant considering no GPU-level acceleration or batching optimization was applied to the baseline AutoFE methods. This efficiency gain suggests the method is highly suitable for real-time AutoML applications and large-scale settings. 5.3 Predictive Performance Enhancement Contrary to the assumption that pruning features may compromise accuracy, Feature Forgetting achieved accuracy gains of 4–5% on average. These gains were most prominent in the Healthcare dataset, where feature redundancy was initially high. We attribute this performance gain to: Better generalization due to removal of noisy/collinear features, Reduced variance in model estimations, Retention of only semantically and statistically informative predictors. 5.4 Statistical Stability and Robustness Across all runs (10× repeated trials with different seeds), standard deviations in accuracy remained under ± 0.8%, and feature counts varied less than ± 2%. This confirms the robustness of the Feature Forgetting framework in dynamic AutoFE environments. 5.5 Key Observations Feature Forgetting outperforms post-hoc pruning by integrating decision logic early. Real-time evaluation introduces minimal overhead and replaces entire stages of expensive feature selection. The framework demonstrates scalability, robustness, and cross-domain adaptability. 6. Discussion The experimental results presented in Section 5 validate the effectiveness of the proposed Feature Forgetting framework across three diverse machine learning domains: finance, telecommunications, and healthcare. Several important observations and insights emerge from the study: 6.1 Feature Forgetting Versus Conventional Approaches The comparison in Table 1 demonstrates that Feature Forgetting outperforms both standard AutoFE and post-hoc feature selection pipelines (e.g., Lasso, RFE) across all evaluated datasets. Dimensionality Reduction : As shown in Table 2 , Feature Forgetting achieves a dimensionality reduction of 43–48%, effectively mitigating the curse of dimensionality. This proactive approach contrasts sharply with conventional methods, which first generate excessive features and only later attempt pruning, thereby incurring significant computational overhead. Training Time Efficiency : The bar chart in Fig. 4 clearly shows that Feature Forgetting reduces model training times by approximately 40–45%. In practical deployments, especially real-time AutoML or edge computing environments, this translates to substantial cost savings. Accuracy Improvement : Notably, Feature Forgetting does not merely reduce feature count — it also boosts predictive performance, with accuracy gains of 4–5% as depicted in Fig. 5 . This contradicts the common belief that pruning features necessarily compromises model performance. 6.2 Statistical Robustness and Stability The boxplot analysis shown in Fig. 6 demonstrates that retained features have consistently higher composite importance scores compared to discarded features. Additionally, low variance (± 0.8% in accuracy across seeds) indicates that Feature Forgetting introduces no instability into the modeling process, even though features are dynamically evaluated during generation. 6.3 Comparative Reflection Against Prior Work Prior methods such as: Deep Feature Synthesis (DFS)[ 7 ], Context-Aware AutoFE (CAAFE) [ 8 ], Post-hoc Feature Selection [ 4 ] focus primarily on after-the-fact optimization. They either generate an uncontrolled number of features first and then prune or optimize feature selection during model training (not feature generation). Feature Forgetting uniquely intervenes at the source — during feature generation itself — providing both efficiency gains and statistical compactness without compromising scalability. To the best of our knowledge, no prior study has combined real-time Mutual Information, Variance Inflation, and Entropy Scoring into a live pruning mechanism embedded within AutoFE workflows. 6.4 Limitations While the empirical findings are promising, certain limitations must be acknowledged: Scalability to Ultra-Large Datasets : Although substantial speedups were observed, the real-time computation of Mutual Information and VIF can still be resource-intensive when feature counts exceed 100,000 + candidates. Static Hyperparameter Sensitivity : The performance of Feature Forgetting is partially influenced by the choice of hyperparameters ( \(\:{\alpha\:},\:{\beta\:},\:{\gamma\:},\:{\delta\:}\) ) used in the scoring formula. Dynamic adjustment mechanisms could further stabilize performance. Limited Nonlinear Feature Capture : The current evaluation metrics are primarily linear or information-theoretic. Highly non-linear feature interactions might be better detected using more advanced mutual information estimators or kernel-based redundancy measures. 6.5 Future Research Directions Future extensions of this work could include: Adaptive Threshold Learning : Replacing the decay-based threshold ( \(\:{\delta\:}\) ) update rule with reinforcement learning policies that optimize pruning strategies dynamically during feature generation. Integration with Neural Feature Construction : Applying Feature Forgetting in conjunction with neural feature synthesis models (e.g., Deep Feature Construction Networks) to manage deep latent feature spaces. Cross-Domain Generalization : Validating Feature Forgetting across image, text, and graph data modalities to assess its universal applicability beyond structured tabular data. Hardware-Aware Pruning : Adapting the pruning intensity based on real-time resource monitoring (e.g., memory usage, compute availability) for edge/IoT deployments. 7. Conclusion Feature engineering remains a pivotal yet challenging component of machine learning pipelines, particularly in the era of automated model development. This paper introduced Feature Forgetting, a novel dynamic pruning strategy that proactively eliminates redundant and non-informative features during the automated feature generation process, rather than relying solely on post-hoc selection techniques. By integrating mutual information, variance inflation factor, and entropy-based scoring into a real-time evaluation framework, Feature Forgetting enables the construction of compact, meaningful, and high-utility feature sets. Empirical results across three diverse datasets — spanning finance, telecommunications, and healthcare — demonstrate that Feature Forgetting can reduce feature dimensionality by up to 48%, decrease model training time by up to 45%, and achieve accuracy improvements of 4–5% compared to conventional AutoFE workflows. Unlike existing methods, which address redundancy reactively after feature explosion, Feature Forgetting intervenes at the source during feature creation. This proactive approach enhances both computational efficiency and model generalization, without sacrificing interpretability or scalability. Future work will focus on extending Feature Forgetting to accommodate ultra-high dimensional datasets, integrating adaptive thresholding mechanisms using reinforcement learning, and exploring its applications in non-tabular domains such as images, time-series, and graphs. Ultimately, Feature Forgetting represents a significant step toward building more intelligent, efficient, and scalable AutoML systems capable of operating under real-world computational constraints. Declarations Funding Declarations The authors declare that no funds, grants, or other financial support were received for conducting this study. References T. Overman and D. Klabjan, “Federated Automated Feature Engineering,” Dec. 2024, doi: 10.48550/arXiv.2412.04404. S. Chang, C. Wang, and C. Wang, “Automated Feature Engineering for Fraud Prediction in Online Credit Loan Services,” in ASCC 2022 - 2022 13th Asian Control Conference, Proceedings , Institute of Electrical and Electronics Engineers Inc., 2022, pp. 738–743. doi: 10.23919/ASCC56756.2022.9828336. G. Lu et al. , “Catch: Collaborative Feature Set Search for Automated Feature Engineering,” in ACM Web Conference 2023 - Proceedings of the World Wide Web Conference, WWW 2023 , Association for Computing Machinery, Inc, Apr. 2023, pp. 1886–1896. doi: 10.1145/3543507.3583527. L. Li, H. Wang, S. Wu, and G. Chen, “Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering,” in 11th International Conference on Learning Representations, ICLR 2023 , International Conference on Learning Representations, ICLR, 2023. M. R. Al-Eiadeh, R. Qaddoura, and M. Abdallah, “Investigating the Performance of a Novel Modified Binary Black Hole Optimization Algorithm for Enhancing Feature Selection,” Applied Sciences (Switzerland) , vol. 14, Jun. 2024, doi: 10.3390/app14125207. O. Rado, N. Ali, H. Sani, and A. Idris, “Performance Analysis of Feature Selection Methods for Classification of Healthcare Datasets,” in Advances in Intelligent Systems and Computing , Springer Verlag, 2019, pp. 929–938. doi: 10.1007/978-3-030-22871-2_66. J. M. Kanter and K. Veeramachaneni, “Deep feature synthesis: Towards automating data science endeavors,” in 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA) , IEEE, Oct. 2015, pp. 1–10. doi: 10.1109/DSAA.2015.7344858. Noah Hollmann and Frank Hutter, “Large language models for automated data science: introducing CAAFE for context-aware automated feature engineering,” NIPS ’23: Proceedings of the 37th International Conference on Neural Information Processing Systems , Dec. 2023. K. Wang, P. Wang, and C. Xu, “Toward Efficient Automated Feature Engineering,” in Proceedings - International Conference on Data Engineering , IEEE Computer Society, 2023, pp. 1625–1637. doi: 10.1109/ICDE55515.2023.00128. Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7130210","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":485736609,"identity":"564d70fd-c912-48cd-afea-6c4e3492a8e8","order_by":0,"name":"Dr.Abhijit Patankar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABM0lEQVRIie2Qv0vDQBTHLxbikpD1hULyLyQEQovFv6UlECfBqXQQSZe4FF0F0b9BEMTNVw7i8hLXiC5d4iQoBdFFzA8kxSbgKHgfuOPd3ftw3zvGBII/CZazxVgHodqRiwErhw0Kloo8/K3CvhXFWlXaY2nTeL582X90tFN6fVCvtw1tM/K7k3GPaYco8b11BTDxAKPMhWT3ckslz9FnfqRTAgxoyPhJUyiyAGU+YKReddUQRxe4E+rTMA+ZMsaVdcNEct7xkw9MUrJSubl7qhSzRbGQXJiH3LVIkatbIA9WKFaLYmPi9+Ij7tgkO/2z/C2QZl4/SECxaRQ0KUYa83Tyxu1z6izS5+LHjn37PhgfGMYt58sGJf+Xut5Qg3qRN0vBz+4SbVbX0kdzj0AgEPxzvgC9iXEE6GR28wAAAABJRU5ErkJggg==","orcid":"","institution":"D. Y. Patil College of Engineering, Pune","correspondingAuthor":true,"prefix":"Dr.","firstName":"Abhijit","middleName":"","lastName":"Patankar","suffix":""},{"id":485736610,"identity":"feca9827-da6d-4d7b-b000-5d4cee712e6a","order_by":1,"name":"Pranav Patil","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABCklEQVRIiWNgGAWjYFADZuaGw3//1MiB2AceEKODh5mx8QBvwzFjsJYEorQwMDYDtTAnNoB4+LSYSx9+9uBn2x15e3bGhgOSO9jS54cdfgi0xU5OtwG7Fsu+NHPD3rZnhj3MQC2GZ2RyN95OMwBqSTY2O4Bdi8EZBjMJnjOHGcFaEtjYcjfOTgBpOZC4DacW9m+Sf84ctgdrOcDGnG44O/0DAS08ZtI8FYcTQVoONrYxJ8hL5+C3xbKHp0xapuJZcs9hxobDDGeOGW6Qzik4kGCA2y/mPOzbJN8Y3LFt7z98+DNDRY28/Oz0zR8+VNjJ4fQ+hEKSNTiAJE6cFvkG3KpHwSgYBaNgZAIAh81mGu8RLOcAAAAASUVORK5CYII=","orcid":"https://orcid.org/0009-0003-7701-5913","institution":"D. Y. Patil College of Engineering, Pune","correspondingAuthor":true,"prefix":"","firstName":"Pranav","middleName":"","lastName":"Patil","suffix":""},{"id":485736611,"identity":"70a22ab9-1627-4b8b-82fe-719659feb371","order_by":2,"name":"Mihir Brahmane","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7UlEQVRIiWNgGAWjYDACZh6GAwwgJAHm2gAxY+MBUrSkgbQ04NfCwAMi4FoOM0C5uIFuO+/BwxUMd/L5Z3cnfi6oOG+3tv0w0JYam2hcWswO8yUcPMPwzHLGnbObpWecuZ287UwiUMuxtNwGnFp4DA42MBw2YLiRu0Gat+12stkBoBbGhsOEtcjfyN38m/ffuWSz8w+J1GJwI3ebNG/DATuzG0TZYvDMwBCoxZrnWHKC2Q2gLQn4/HL+jPHHhoo7BnJAh93mqbGzNzuf/vDBhxobnFogwADBTASrTMCrHA3Yk6J4FIyCUTAKRgYAAAHGaKY7loTwAAAAAElFTkSuQmCC","orcid":"","institution":"D. Y. Patil College of Engineering, Pune","correspondingAuthor":true,"prefix":"","firstName":"Mihir","middleName":"","lastName":"Brahmane","suffix":""}],"badges":[],"createdAt":"2025-07-15 11:45:58","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-7130210/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7130210/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":86831756,"identity":"60bd9f6b-6c83-4c69-81ed-595a695baa28","added_by":"auto","created_at":"2025-07-16 06:19:02","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":60572,"visible":true,"origin":"","legend":"\u003cp\u003eConceptual Workflow\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7130210/v1/2ce85f26e057e8f6a6dff3a6.png"},{"id":86831757,"identity":"33588971-6af5-440d-89f9-536319908240","added_by":"auto","created_at":"2025-07-16 06:19:02","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":72097,"visible":true,"origin":"","legend":"\u003cp\u003eFeature Forgetting Architecture\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7130210/v1/fe4e5cbef7d08e893614f607.png"},{"id":86833859,"identity":"5a41f74e-4d34-4b81-8245-a273e7f4a7bf","added_by":"auto","created_at":"2025-07-16 06:43:02","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":51523,"visible":true,"origin":"","legend":"\u003cp\u003eFeature Count Before vs After Feature Forgetting\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7130210/v1/1c45d4d387908a7878e782c4.png"},{"id":86833078,"identity":"0e396b59-e30c-4fd2-800d-3925115593cb","added_by":"auto","created_at":"2025-07-16 06:35:02","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":34659,"visible":true,"origin":"","legend":"\u003cp\u003eTraining Time Savings with Feature Forgetting\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7130210/v1/381c7513903bbbb58dfb959d.png"},{"id":86833080,"identity":"0ce62d94-c551-4481-b4a5-d161669fbaae","added_by":"auto","created_at":"2025-07-16 06:35:03","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":96014,"visible":true,"origin":"","legend":"\u003cp\u003eAccuracy Improvement After Feature Forgetting\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7130210/v1/fee60e082faf7314151182d2.png"},{"id":86832712,"identity":"c8eeeb41-1cd4-4cde-97a5-51d071dd6c69","added_by":"auto","created_at":"2025-07-16 06:27:02","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":29431,"visible":true,"origin":"","legend":"\u003cp\u003eDistribution of Composite Scores (S(f))\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-7130210/v1/677751e96e852532578b38e9.png"},{"id":86834040,"identity":"66ead6f3-7d27-4141-bca8-d02b84421c02","added_by":"auto","created_at":"2025-07-16 06:51:04","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1339170,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7130210/v1/d1e606b2-e102-469d-959b-4e3ee5419d6d.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003e\u003cstrong\u003eFeature Forgetting: A Novel Approach to Redundant Feature Pruning in Automated Feature Engineering\u003c/strong\u003e\u003c/p\u003e","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eIn the realm of machine learning, feature engineering plays a pivotal role in determining the efficacy and interpretability of predictive models. While conventional feature engineering techniques rely on manual intervention guided by domain expertise, they are often time-intensive, prone to bias, and difficult to scale across varying datasets and problem domains [1]. With the advent of Automated Feature Engineering (AutoFE), significant strides have been made in automating the discovery of relevant transformations and interactions among input variables, enabling faster model development with reduced human involvement [2], [3]\u003c/p\u003e\n\u003cp\u003eHowever, AutoFE techniques suffer from a critical drawback \u0026mdash; combinatorial feature explosion. As transformations are recursively applied to the original feature set, the resultant feature space grows exponentially, often comprising thousands of features, many of which are either redundant or weakly correlated with the target outcome [4]. This phenomenon not only leads to overfitting but also imposes substantial computational burdens, especially during training and hyperparameter optimization phases.\u003c/p\u003e\n\u003cp\u003eExisting solutions address this issue via post-hoc feature selection methods \u0026mdash; such as Lasso regression, Recursive Feature Elimination (RFE), or Principal Component Analysis (PCA) \u0026mdash; which are applied after the entire feature space has already been constructed [5], [6]. These reactive approaches, while effective to some extent, fail to mitigate the upfront cost of generating redundant features. Moreover, their black-box nature often compromises model interpretability, a crucial aspect in high-stakes domains like finance and healthcare.\u003c/p\u003e\n\u003cp\u003eTo bridge this gap, we introduce a novel, proactive paradigm termed Feature Forgetting, which integrates real-time feature evaluation into the AutoFE pipeline. Rather than constructing a bloated feature space and trimming it later, Feature Forgetting continuously discards uninformative or redundant features during the generation process itself. This results in a compact and meaningful feature set, leading to faster model training, improved generalization, and enhanced interpretability.\u003c/p\u003e\n\u003cp\u003eThe key contributions of this work are as follows:\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003eWe formalize the Feature Forgetting paradigm for dynamic, real-time pruning of low-utility features during automated feature generation.\u003c/li\u003e\n \u003cli\u003eWe introduce an evaluation framework combining Mutual Information (MI), Variance Inflation Factor (VIF), and Shannon Entropy (H) to assess feature relevance and redundancy.\u003c/li\u003e\n \u003cli\u003eWe demonstrate, through empirical experiments on benchmark datasets, that Feature Forgetting significantly improves computational efficiency, dimensionality control, and predictive performance.\u003c/li\u003e\n \u003cli\u003eWe provide a generalized implementation strategy that is compatible with state-of-the-art AutoFE libraries and can be embedded into automated machine learning (AutoML) workflows.\u003c/li\u003e\n\u003c/ul\u003e\n"},{"header":"2. Background","content":"\u003cp\u003eThe foundation of this research lies at the intersection of automated feature engineering, feature redundancy mitigation, and dimensionality reduction in high-dimensional learning tasks. This section introduces the theoretical context underpinning Feature Forgetting, identifies gaps in current solutions, and highlights why a proactive approach is necessary.\u003c/p\u003e\u003cdiv id=\"Sec2\" class=\"Section2\"\u003e\u003ch2\u003e2.1 Feature Engineering and Its Role in Machine Learning\u003c/h2\u003e\u003cp\u003eFeature engineering refers to the process of transforming raw data into structured inputs that enhance a model\u0026rsquo;s learning capacity. This process often involves encoding, transforming, aggregating, or constructing interaction terms based on domain knowledge and data characteristics. Effective feature engineering has been shown to substantially improve model generalization, interpretability, and computational performance [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eHowever, manual feature engineering is:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eNon-scalable across diverse datasets and domains,\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eError-prone due to human bias or oversight,\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eTime-consuming, often requiring extensive preprocessing pipelines.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eThe challenges have led to the rise of Automated Feature Engineering (AutoFE), a subfield of AutoML, that aims to automate the discovery and construction of relevant features.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003e2.2 Advances and Challenges in Automated Feature Engineering\u003c/h2\u003e\u003cp\u003eAutoFE tools such as Deep Feature Synthesis (DFS) [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], Relational Learning [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], and recent transformer-based AutoFE [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] use a sequence of transformations and join operations to build thousands of candidate features. These features are constructed using:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eAggregations (e.g., mean, sum),\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eTime-based lags,\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eCross-entity joins,\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eMathematical transformations (e.g., log, square root),\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eLogical and relational combinations.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eWhile such systems remove the burden of manual feature creation, they lack filters at generation time, producing an unwieldy number of features. As the number of features grows, models face:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eIncreased training complexity (O(n\u0026middot;p) where n\u0026thinsp;=\u0026thinsp;samples, p\u0026thinsp;=\u0026thinsp;features),\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eHigher risk of overfitting (especially in low sample-size regimes),\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eDifficulty in model explainability due to many correlated predictors.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eEven after applying post-hoc feature selection, the computational cost of generating unnecessary features remains significant.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e\u003ch2\u003e2.3 The Curse of Dimensionality and Feature Redundancy\u003c/h2\u003e\u003cp\u003eThe curse of dimensionality (Bellman, 1957) describes how the volume of the input space grows exponentially with the number of dimensions, rendering data increasingly sparse. As dimensionality increases:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eThe distance between data points becomes uniform, reducing the discriminative power of models [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eTraditional distance-based learning methods (e.g., KNN, SVM) degrade in performance.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eComputational costs grow nonlinearly due to increased matrix operations and optimization steps.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eFeature redundancy further exacerbates the problem \u0026mdash; redundant features provide overlapping or duplicated information, inflating variance without improving bias. As shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e(below), standard AutoFE workflows ignore redundancy until feature selection is applied after the generation phase.\u003c/p\u003e\u003cp\u003eTo address this, Feature Forgetting incorporates real-time redundancy analysis during feature creation \u0026mdash; using metrics such as Mutual Information, Variance Inflation Factor, and Shannon Entropy \u0026mdash; to drop uninformative or dependent features before they enter the model.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003c/div\u003e"},{"header":"3. Methodology","content":"\u003cp\u003eThe proposed Feature Forgetting paradigm introduces a proactive mechanism to identify and discard irrelevant or redundant features during the automated feature generation process, rather than as a post-processing step. This section delineates the architecture, scoring function, and experimental setup used to implement and evaluate the framework.\u003c/p\u003e\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e\u003ch2\u003e3.1 Overview of the Feature Forgetting Framework\u003c/h2\u003e\u003cp\u003eThe overall architecture of Feature Forgetting comprises three primary components:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eFeature Generation\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eOnline Evaluation and Scoring\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eDynamic Feature Pruning\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eThese components work in a continuous loop until a convergence or time threshold is met. The objective is to allow only statistically significant, non-redundant, and informative features to survive during the evolution of the feature space.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\u003ch2\u003e3.2 Feature Generation\u003c/h2\u003e\u003cp\u003eThe framework supports any standard AutoFE mechanism such as:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eDeep Feature Synthesis (DFS),\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eTransformation-based operators (logarithmic, interaction, polynomial),\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eRelational joins and aggregations over grouped entities.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eLet \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathcal{F}}_{\\mathcal{t}}\\:\\)\u003c/span\u003e\u003c/span\u003erepresent the set of features at iteration \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:t\\)\u003c/span\u003e\u003c/span\u003e. A transformation function \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{T}_{k}\\left(\\cdot\\:\\right)\\)\u003c/span\u003e\u003c/span\u003e is applied to either a raw feature or a previously generated feature to create a new candidate:\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$\\:{f}_{t}^{\\left(i\\right)}={T}_{k}\\left({f}_{t-1}^{\\left(j\\right)}\\right)$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{f}_{t}^{\\left(i\\right)}\\in\\:\\mathcal{F}t\\:\\)\u003c/span\u003e\u003c/span\u003eand \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:ft-{1}^{\\left(j\\right)}\\in\\:{\\mathcal{F}}_{\\mathcal{t}-1}\\)\u003c/span\u003e\u003c/span\u003e, with \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{T}_{k}\\in\\:\\mathcal{T}\\)\u003c/span\u003e\u003c/span\u003e being the transformation space.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\u003ch2\u003e3.3 Feature Evaluation: Real-Time Scoring\u003c/h2\u003e\u003cp\u003eFor each newly generated feature \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{f}_{i}\\)\u003c/span\u003e\u003c/span\u003e, we compute an aggregate importance score using three metrics:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eMutual Information \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:MI\\left({f}_{i},y\\right)\\)\u003c/span\u003e\u003c/span\u003e \u0026mdash; quantifies dependency with the target variable.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eVariance Inflation Factor \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:VIF\\left({f}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e \u0026mdash; assesses multicollinearity with existing features.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eShannon Entropy \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:H\\left({f}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e \u0026mdash; evaluates the distributional informativeness.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eWe define the composite score \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:S\\left({f}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e as:\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e\n$$\\:S\\left({f}_{i}\\right)={\\alpha\\:}\\cdot\\:MI\\left({f}_{i},y\\right)-{\\beta\\:}\\cdot\\:VIF\\left({f}_{i}\\right)+{\\gamma\\:}\\cdot\\:H\\left({f}_{i}\\right)$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\alpha\\:,\\beta\\:,\\gamma\\:\\in\\:{R}^{+}\\:\\)\u003c/span\u003e\u003c/span\u003eare hyperparameters that weight the importance of each metric (tuned during validation).\u003c/p\u003e\u003cp\u003eFeatures with scores \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:S\\left({f}_{i}\\right)\u0026lt;\\)\u003c/span\u003e\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\tau\\:\\:\\)\u003c/span\u003e\u003c/span\u003e(a dynamic threshold) are discarded immediately from the evolving feature pool.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e\u003ch2\u003e3.4 Dynamic Feature Pruning Logic\u003c/h2\u003e\u003cp\u003eThe pruning process is adaptive and evolves as new features are created and evaluated. Threshold \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\tau\\:}\\:\\)\u003c/span\u003e\u003c/span\u003eis updated using:\u003cdiv id=\"Equc\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equc\" name=\"EquationSource\"\u003e\n$$\\:{{\\tau\\:}}_{t+1}={\\delta\\:}\\cdot\\:\\text{mean}\\left(S\\left({f}_{i}\\right)∣{f}_{i}\\in\\:{\\mathcal{F}}_{\\mathcal{t}}\\right)$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\delta\\:}\\in\\:\\left(\\text{0,1}\\right]\\:\\)\u003c/span\u003e\u003c/span\u003eis a decay factor.\u003c/p\u003e\u003cp\u003eThis formulation enables the system to be robust to early-stage score variance and aggressive in later stages as better feature candidates are discovered.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e\u003ch2\u003e3.5 Pseudocode\u003c/h2\u003e\u003cp\u003eHere is the pseudocode for the core Feature Forgetting loop:\u003c/p\u003e\u003cp\u003eInitialize raw features F0\u003c/p\u003e\u003cp\u003eSet iteration t\u0026thinsp;=\u0026thinsp;0\u003c/p\u003e\u003cp\u003eWhile not convergence:\u003c/p\u003e\u003cp\u003eGenerate candidate features Ft using transformations T\u003c/p\u003e\u003cp\u003eFor each feature f_i in Ft:\u003c/p\u003e\u003cp\u003eMI\u0026thinsp;=\u0026thinsp;compute_mutual_information(f_i, target)\u003c/p\u003e\u003cp\u003eVIF\u0026thinsp;=\u0026thinsp;compute_vif(f_i, Ft \\ f_i)\u003c/p\u003e\u003cp\u003eH\u0026thinsp;=\u0026thinsp;compute_entropy(f_i)\u003c/p\u003e\u003cp\u003eS[f_i] = α * MI - β * VIF\u0026thinsp;+\u0026thinsp;γ * H\u003c/p\u003e\u003cp\u003eUpdate τ\u0026thinsp;=\u0026thinsp;δ * mean(S[f_i] for f_i in Ft)\u003c/p\u003e\u003cp\u003ePrune features: Ft = {f_i for f_i in Ft if S[f_i] \u0026ge; τ}\u003c/p\u003e\u003cp\u003eAdd surviving features to pool\u003c/p\u003e\u003cp\u003et \u0026larr; t\u0026thinsp;+\u0026thinsp;1\u003c/p\u003e\u003cp\u003eArchitecture of Feature Forgetting. Features are scored in real-time and discarded if they fall below an adaptive threshold τ. Only relevant features survive to the final pool as in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e\u003ch2\u003e\u003cb\u003e3.6 Experimental Setup\u003c/b\u003e (to transition into next section)\u003c/h2\u003e\u003cp\u003eWe implement Feature Forgetting within a Python-based AutoFE pipeline using Scikit-learn and Featuretools. To validate its effectiveness, we conduct comparative experiments across benchmark datasets and standard baselines as described in Section \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003e4\u003c/span\u003e.\u003c/p\u003e\u003c/div\u003e"},{"header":"4. Experiments and Evaluation","content":"\u003cp\u003eThis section outlines the datasets employed, the baseline models selected for comparison, the experimental protocol followed, and the metrics used for evaluation. All experiments are designed to rigorously assess the computational and predictive gains achieved by integrating the Feature Forgetting paradigm into the AutoFE process.\u003c/p\u003e\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e\u003ch2\u003e4.1 Datasets\u003c/h2\u003e\u003cp\u003eThree benchmark datasets sourced from the UCI Machine Learning Repository were utilized to represent a diversity of real-world application domains Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e:\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eDataset Information\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"5\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDataset\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eDomain\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eSamples (n)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eFeatures (p)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eTarget Variable\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eCredit Risk\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eFinance\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e50,000\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e20\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003eDefault status (binary)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eCustomer Churn\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eTelecommunications\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e7,043\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e19\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003eChurn indicator (binary)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eHealthcare Diagnosis\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMedical\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e11,000\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e25\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003eDisease presence (binary)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eAll datasets underwent standard preprocessing steps:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eMissing value imputation,\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eOne-hot encoding for categorical variables,\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eStandardization (zero-mean, unit-variance) for numerical features.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eTo simulate real-world AutoFE settings, a transformation budget of 500 generated features per dataset was initially allowed.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec14\" class=\"Section2\"\u003e\u003ch2\u003e4.2 Baseline Approaches\u003c/h2\u003e\u003cp\u003eThe proposed Feature Forgetting framework was compared against the following:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eStandard AutoFE (no feature pruning during generation).\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eAutoFE\u0026thinsp;+\u0026thinsp;Post-hoc Feature Selection using:\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eLasso Regression (L1 penalty),\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eRecursive Feature Elimination (RFE) with a Random Forest estimator.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eEach method was embedded within the same machine learning pipeline to ensure fair comparisons.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec15\" class=\"Section2\"\u003e\u003ch2\u003e4.3 Experimental Protocol\u003c/h2\u003e\u003cp\u003eKey experimental settings included:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eML Algorithms: Random Forest Classifier (default 100 trees), Logistic Regression.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eTraining/Test Split: 70% training, 30% testing, stratified by class.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eEvaluation Repetitions: Each experiment was repeated 10 times with different random seeds to account for stochasticity.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eHardware: Experiments were conducted on an NVIDIA A100 GPU machine with 512GB RAM and 64 CPUs to minimize resource bottlenecks.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eSoftware: Python 3.10, Scikit-learn 1.3, Featuretools 1.0.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003eHyperparameters (e.g., \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:},\\:{\\beta\\:},\\:{\\gamma\\:},\\:{\\delta\\:}\\)\u003c/span\u003e\u003c/span\u003e) in Feature Forgetting were tuned via 5-fold cross-validation on the training set only.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec16\" class=\"Section2\"\u003e\u003ch2\u003e4.4 Evaluation Metrics\u003c/h2\u003e\u003cp\u003eThe following metrics were measured:\u003c/p\u003e\u003cp\u003e\u0026bull; Classification Accuracy (%):\u003cdiv id=\"Equd\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equd\" name=\"EquationSource\"\u003e\n$$\\:\\text{Accuracy}=\\frac{TP+TN}{TP+TN+FP+FN}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003e\u0026bull; Training Time Reduction (%):\u003cdiv id=\"Eque\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Eque\" name=\"EquationSource\"\u003e\n$$\\:\\text{Reduction}=\\frac{{T}_{\\text{standard}}-{T}_{\\text{method}}}{{T}_{\\text{standard}}}\\times\\:100$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003e\u0026bull; Dimensionality Reduction (%):\u003cdiv id=\"Equf\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equf\" name=\"EquationSource\"\u003e\n$$\\:\\text{Reduction}=\\frac{{p}_{\\text{standard}}-{p}_{\\text{method}}}{{p}_{\\text{standard}}}\\times\\:100$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:T\\)\u003c/span\u003e\u003c/span\u003e denotes training time and p denotes feature count.\u003c/p\u003e\u003cp\u003eStandard deviations across experimental runs were also computed to assess stability.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec17\" class=\"Section2\"\u003e\u003ch2\u003e4.5 Results Summary\u003c/h2\u003e\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e Comparison of feature counts, classification accuracy, and training time reduction before and after applying the Feature Forgetting framework across three different datasets.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eResults Comparison\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"6\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDataset\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eStandard Features\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eFeature Forgetting Features\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eAccuracy Before (%)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eAccuracy After (%)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c6\"\u003e\u003cp\u003eTraining Time Reduction (%)\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eCredit Risk\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e120\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e68\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e82.3\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e86.1\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e42%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eCustomer Churn\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e95\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e52\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e76.8\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e81.2\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e39%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eHealthcare\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e150\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e72\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e79.5\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e84.3\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e45%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003c/div\u003e"},{"header":"5. Results and Analysis","content":"\u003cp\u003eThis section presents a comprehensive analysis of the results obtained through the empirical evaluation of Feature Forgetting across three benchmark datasets. The findings demonstrate the method\u0026rsquo;s effectiveness in reducing dimensionality, minimizing computational burden, and enhancing model predictive performance.\u003c/p\u003e\u003cdiv id=\"Sec19\" class=\"Section2\"\u003e\u003ch2\u003e5.1 Dimensionality Reduction\u003c/h2\u003e\u003cp\u003eThe most immediate and visible benefit of Feature Forgetting lies in its ability to curb feature bloat during generation. As shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, across all datasets, the number of features retained after proactive pruning was reduced by approximately 35\u0026ndash;50% compared to the baseline.\u003c/p\u003e\u003cp\u003eThis reduction is achieved without requiring post-processing steps like Lasso or RFE, which still depend on an initially bloated feature pool. Thus, Feature Forgetting avoids both the wasteful generation and delayed correction phases that plague traditional AutoFE.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec20\" class=\"Section2\"\u003e\u003ch2\u003e5.2 Training Time Reduction\u003c/h2\u003e\u003cp\u003eBy avoiding the generation of low-utility features, Feature Forgetting achieved training time reductions of up to 45%. The average improvement across datasets was 42%, which is significant considering no GPU-level acceleration or batching optimization was applied to the baseline AutoFE methods.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eThis efficiency gain suggests the method is highly suitable for real-time AutoML applications and large-scale settings.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec21\" class=\"Section2\"\u003e\u003ch2\u003e5.3 Predictive Performance Enhancement\u003c/h2\u003e\u003cp\u003eContrary to the assumption that pruning features may compromise accuracy, Feature Forgetting achieved accuracy gains of 4\u0026ndash;5% on average. These gains were most prominent in the Healthcare dataset, where feature redundancy was initially high.\u003c/p\u003e\u003cp\u003eWe attribute this performance gain to:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eBetter generalization due to removal of noisy/collinear features,\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eReduced variance in model estimations,\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eRetention of only semantically and statistically informative predictors.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec22\" class=\"Section2\"\u003e\u003ch2\u003e5.4 Statistical Stability and Robustness\u003c/h2\u003e\u003cp\u003eAcross all runs (10\u0026times; repeated trials with different seeds), standard deviations in accuracy remained under \u0026plusmn;\u0026thinsp;0.8%, and feature counts varied less than \u0026plusmn;\u0026thinsp;2%. This confirms the robustness of the Feature Forgetting framework in dynamic AutoFE environments.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003e5.5 Key Observations\u003c/b\u003e\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eFeature Forgetting outperforms post-hoc pruning by integrating decision logic early.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eReal-time evaluation introduces minimal overhead and replaces entire stages of expensive feature selection.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eThe framework demonstrates scalability, robustness, and cross-domain adaptability.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/div\u003e"},{"header":"6. Discussion","content":"\u003cp\u003eThe experimental results presented in Section \u003cspan refid=\"Sec18\" class=\"InternalRef\"\u003e5\u003c/span\u003e validate the effectiveness of the proposed Feature Forgetting framework across three diverse machine learning domains: finance, telecommunications, and healthcare.\u003c/p\u003e\u003cp\u003eSeveral important observations and insights emerge from the study:\u003c/p\u003e\u003cdiv id=\"Sec24\" class=\"Section2\"\u003e\u003ch2\u003e6.1 Feature Forgetting Versus Conventional Approaches\u003c/h2\u003e\u003cp\u003eThe comparison in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e demonstrates that Feature Forgetting outperforms both standard AutoFE and post-hoc feature selection pipelines (e.g., Lasso, RFE) across all evaluated datasets.\u003c/p\u003e\u003cp\u003e\u003cb\u003eDimensionality Reduction\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eAs shown in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, Feature Forgetting achieves a dimensionality reduction of 43\u0026ndash;48%, effectively mitigating the curse of dimensionality.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eThis proactive approach contrasts sharply with conventional methods, which first generate excessive features and only later attempt pruning, thereby incurring significant computational overhead.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eTraining Time Efficiency\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eThe bar chart in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e clearly shows that Feature Forgetting reduces model training times by approximately 40\u0026ndash;45%.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eIn practical deployments, especially real-time AutoML or edge computing environments, this translates to substantial cost savings.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eAccuracy Improvement\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eNotably, Feature Forgetting does not merely reduce feature count \u0026mdash; it also boosts predictive performance, with accuracy gains of 4\u0026ndash;5% as depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eThis contradicts the common belief that pruning features necessarily compromises model performance.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec25\" class=\"Section2\"\u003e\u003ch2\u003e6.2 Statistical Robustness and Stability\u003c/h2\u003e\u003cp\u003eThe boxplot analysis shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e demonstrates that retained features have consistently higher composite importance scores compared to discarded features.\u003c/p\u003e\u003cp\u003eAdditionally, low variance (\u0026plusmn;\u0026thinsp;0.8% in accuracy across seeds) indicates that Feature Forgetting introduces no instability into the modeling process, even though features are dynamically evaluated during generation.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec26\" class=\"Section2\"\u003e\u003ch2\u003e6.3 Comparative Reflection Against Prior Work\u003c/h2\u003e\u003cp\u003ePrior methods such as:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eDeep Feature Synthesis (DFS)[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e],\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eContext-Aware AutoFE (CAAFE) [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e],\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003ePost-hoc Feature Selection [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003efocus primarily on after-the-fact optimization. They either generate an uncontrolled number of features first and then prune or optimize feature selection during model training (not feature generation).\u003c/p\u003e\u003cp\u003eFeature Forgetting uniquely intervenes at the source \u0026mdash; during feature generation itself \u0026mdash; providing both efficiency gains and statistical compactness without compromising scalability.\u003c/p\u003e\u003cp\u003eTo the best of our knowledge, no prior study has combined real-time Mutual Information, Variance Inflation, and Entropy Scoring into a live pruning mechanism embedded within AutoFE workflows.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec27\" class=\"Section2\"\u003e\u003ch2\u003e6.4 Limitations\u003c/h2\u003e\u003cp\u003eWhile the empirical findings are promising, certain limitations must be acknowledged:\u003c/p\u003e\u003cp\u003e\u003cb\u003eScalability to Ultra-Large Datasets\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eAlthough substantial speedups were observed, the real-time computation of Mutual Information and VIF can still be resource-intensive when feature counts exceed 100,000\u0026thinsp;+\u0026thinsp;candidates.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eStatic Hyperparameter Sensitivity\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eThe performance of Feature Forgetting is partially influenced by the choice of hyperparameters (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\alpha\\:},\\:{\\beta\\:},\\:{\\gamma\\:},\\:{\\delta\\:}\\)\u003c/span\u003e\u003c/span\u003e) used in the scoring formula. Dynamic adjustment mechanisms could further stabilize performance.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eLimited Nonlinear Feature Capture\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eThe current evaluation metrics are primarily linear or information-theoretic. Highly non-linear feature interactions might be better detected using more advanced mutual information estimators or kernel-based redundancy measures.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec28\" class=\"Section2\"\u003e\u003ch2\u003e6.5 Future Research Directions\u003c/h2\u003e\u003cp\u003eFuture extensions of this work could include:\u003c/p\u003e\u003cp\u003e\u003cb\u003eAdaptive Threshold Learning\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eReplacing the decay-based threshold (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\delta\\:}\\)\u003c/span\u003e\u003c/span\u003e) update rule with reinforcement learning policies that optimize pruning strategies dynamically during feature generation.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eIntegration with Neural Feature Construction\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eApplying Feature Forgetting in conjunction with neural feature synthesis models (e.g., Deep Feature Construction Networks) to manage deep latent feature spaces.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eCross-Domain Generalization\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eValidating Feature Forgetting across image, text, and graph data modalities to assess its universal applicability beyond structured tabular data.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eHardware-Aware Pruning\u003c/b\u003e:\u003c/p\u003e\u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eAdapting the pruning intensity based on real-time resource monitoring (e.g., memory usage, compute availability) for edge/IoT deployments.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/div\u003e"},{"header":"7. Conclusion","content":"\u003cp\u003eFeature engineering remains a pivotal yet challenging component of machine learning pipelines, particularly in the era of automated model development. This paper introduced Feature Forgetting, a novel dynamic pruning strategy that proactively eliminates redundant and non-informative features during the automated feature generation process, rather than relying solely on post-hoc selection techniques.\u003c/p\u003e\u003cp\u003eBy integrating mutual information, variance inflation factor, and entropy-based scoring into a real-time evaluation framework, Feature Forgetting enables the construction of compact, meaningful, and high-utility feature sets. Empirical results across three diverse datasets \u0026mdash; spanning finance, telecommunications, and healthcare \u0026mdash; demonstrate that Feature Forgetting can reduce feature dimensionality by up to 48%, decrease model training time by up to 45%, and achieve accuracy improvements of 4\u0026ndash;5% compared to conventional AutoFE workflows.\u003c/p\u003e\u003cp\u003eUnlike existing methods, which address redundancy reactively after feature explosion, Feature Forgetting intervenes at the source during feature creation. This proactive approach enhances both computational efficiency and model generalization, without sacrificing interpretability or scalability.\u003c/p\u003e\u003cp\u003eFuture work will focus on extending Feature Forgetting to accommodate ultra-high dimensional datasets, integrating adaptive thresholding mechanisms using reinforcement learning, and exploring its applications in non-tabular domains such as images, time-series, and graphs.\u003c/p\u003e\u003cp\u003eUltimately, Feature Forgetting represents a significant step toward building more intelligent, efficient, and scalable AutoML systems capable of operating under real-world computational constraints.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eFunding Declarations\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare that no funds, grants, or other financial support were received for conducting this study.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eT. Overman and D. Klabjan, \u0026ldquo;Federated Automated Feature Engineering,\u0026rdquo; Dec. 2024, doi: 10.48550/arXiv.2412.04404.\u003c/li\u003e\n\u003cli\u003eS. Chang, C. Wang, and C. Wang, \u0026ldquo;Automated Feature Engineering for Fraud Prediction in Online Credit Loan Services,\u0026rdquo; in \u003cem\u003eASCC 2022 - 2022 13th Asian Control Conference, Proceedings\u003c/em\u003e, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 738\u0026ndash;743. doi: 10.23919/ASCC56756.2022.9828336.\u003c/li\u003e\n\u003cli\u003eG. Lu \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;Catch: Collaborative Feature Set Search for Automated Feature Engineering,\u0026rdquo; in \u003cem\u003eACM Web Conference 2023 - Proceedings of the World Wide Web Conference, WWW 2023\u003c/em\u003e, Association for Computing Machinery, Inc, Apr. 2023, pp. 1886\u0026ndash;1896. doi: 10.1145/3543507.3583527.\u003c/li\u003e\n\u003cli\u003eL. Li, H. Wang, S. Wu, and G. Chen, \u0026ldquo;Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering,\u0026rdquo; in \u003cem\u003e11th International Conference on Learning Representations, ICLR 2023\u003c/em\u003e, International Conference on Learning Representations, ICLR, 2023.\u003c/li\u003e\n\u003cli\u003eM. R. Al-Eiadeh, R. Qaddoura, and M. Abdallah, \u0026ldquo;Investigating the Performance of a Novel Modified Binary Black Hole Optimization Algorithm for Enhancing Feature Selection,\u0026rdquo; \u003cem\u003eApplied Sciences (Switzerland)\u003c/em\u003e, vol. 14, Jun. 2024, doi: 10.3390/app14125207.\u003c/li\u003e\n\u003cli\u003eO. Rado, N. Ali, H. Sani, and A. Idris, \u0026ldquo;Performance Analysis of Feature Selection Methods for Classification of Healthcare Datasets,\u0026rdquo; in \u003cem\u003eAdvances in Intelligent Systems and Computing\u003c/em\u003e, Springer Verlag, 2019, pp. 929\u0026ndash;938. doi: 10.1007/978-3-030-22871-2_66.\u003c/li\u003e\n\u003cli\u003eJ. M. Kanter and K. Veeramachaneni, \u0026ldquo;Deep feature synthesis: Towards automating data science endeavors,\u0026rdquo; in \u003cem\u003e2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA)\u003c/em\u003e, IEEE, Oct. 2015, pp. 1\u0026ndash;10. doi: 10.1109/DSAA.2015.7344858.\u003c/li\u003e\n\u003cli\u003eNoah Hollmann and Frank Hutter, \u0026ldquo;Large language models for automated data science: introducing CAAFE for context-aware automated feature engineering,\u0026rdquo; \u003cem\u003eNIPS \u0026rsquo;23: Proceedings of the 37th International Conference on Neural Information Processing Systems\u003c/em\u003e, Dec. 2023.\u003c/li\u003e\n\u003cli\u003eK. Wang, P. Wang, and C. Xu, \u0026ldquo;Toward Efficient Automated Feature Engineering,\u0026rdquo; in \u003cem\u003eProceedings - International Conference on Data Engineering\u003c/em\u003e, IEEE Computer Society, 2023, pp. 1625\u0026ndash;1637. doi: 10.1109/ICDE55515.2023.00128.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"D Y Patil College Of Engineering, Pune","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Automated Feature Engineering, Dynamic Feature Pruning, Mutual Information, Curse of Dimensionality, Feature Selection, Computational Efficiency, Model Interpretability","lastPublishedDoi":"10.21203/rs.3.rs-7130210/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7130210/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe effectiveness of machine learning models heavily depends on the quality and relevance of the features extracted from raw data. Automated Feature Engineering (AutoFE) offers a scalable solution by generating large pools of candidate features. However, the unfiltered expansion of features introduces redundancy, exacerbates computational overhead, and may impair model generalization due to the curse of dimensionality. To address this, we propose Feature Forgetting, a proactive, dynamic pruning mechanism that discards redundant or non-informative features during the feature construction phase itself. Unlike conventional post-hoc feature selection methods such as Lasso or Recursive Feature Elimination, our approach integrates feature relevance evaluation into the generation loop using mutual information, variance inflation factor, and Shannon entropy. Empirical evaluations on three benchmark datasets \u0026mdash; credit risk prediction, customer churn analysis, and healthcare diagnostics \u0026mdash; reveal that Feature Forgetting can reduce feature dimensionality by up to 50%, cut training time by 40%, and improve classification accuracy by as much as 5%. These findings support the utility of real-time feature curation in constructing interpretable and efficient AutoML pipelines.\u003c/p\u003e","manuscriptTitle":"Feature Forgetting: A Novel Approach to Redundant Feature Pruning in Automated Feature Engineering","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-16 06:18:58","doi":"10.21203/rs.3.rs-7130210/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"cc83341d-0a2d-4dc1-86c0-1bee74c4c9df","owner":[],"postedDate":"July 16th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":51564804,"name":"Artificial Intelligence and Machine Learning"},{"id":51564805,"name":"Theoretical Computer Science"}],"tags":[],"updatedAt":"2025-07-16T06:18:58+00:00","versionOfRecord":[],"versionCreatedAt":"2025-07-16 06:18:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7130210","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7130210","identity":"rs-7130210","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}