A call to ensure reproducibility of machine learning applications in industrial ecology

A call to ensure reproducibility of machine learning applications in industrial ecology | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A call to ensure reproducibility of machine learning applications in industrial ecology Keagan Hudson Rankin, Franco Donati, Qingshi Tu, Jesse Ward-Bond, and 7 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9270723/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 8 You are reading this latest preprint version Abstract Machine learning (ML) usage in industrial ecology (IE) has grown nearly tenfold in the last decade. In other fields, similar increases in ML adoption have led to the widespread publication of results that cannot be reproduced. This uptick in irreproducibility, driven by a failure to follow best-practices when creating and reporting models - undermines the conclusions and credibility of science. Industrial ecologists have not yet determined whether reproducibility is becoming an issue of concern in their applications of ML. In order to assess this risk, we audited 50 recent IE studies against a ML reproducibility ontology. We find that 84% of surveyed studies suffer from computational reproducibility issues, and 28% exhibit methodological flaws that could introduce data leakage and invalidate findings. Yet, bibliometric analysis shows these potentially irreproducible studies are cited as or more frequently than their non-ML counterparts, which could embed flawed results into the scientific literature. Our findings serve as a call to action for the IE community. We suggest multi-level interventions, including that journals adopt reproducibility checklists and that reviewers prioritize key reproducibility errors over performance metrics, to safeguard the field and maximize the reproducibility of future ML-driven IE research. Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. INTRODUCTION Industrial ecologists, like many other domain researchers, are increasingly using supervised machine learning (ML) models in their research. The number of industrial ecology (IE) related papers that use just the keyword “machine learning” has increased from less than 20 papers per year in 2014 to an average of 170 papers per year since 2020 (Gong et al., 2025 ). This increase has been driven by the growth of IE data availability as well as an increased awareness and use of machine learning methods in applied science (Hao et al., 2026 ). Supervised ML has the potential to accelerate the research and practical goals of IE. Recent use-cases of ML in IE research include (among others): Gap-filling and prediction of emission inventories where data is unavailable (Cullen et al., 2024 ). Speeding up or automating traditionally repetitive tasks in life-cycle assessment (LCA) and complex models like IAMs (Balaji et al., 2023 ,2025; Li et al. 2025). Predicting properties of buildings (e.g. height, material content) and stocks in the built-environment (Huang et al., 2025 ). Supporting material recovery and circular economy practices by identifying materials and components in waste streams (Sarswat et al., 2024 ). These papers show how ML can inform policies, create data, and support decision making. However, recent literature also shows that increasing use of ML must be met with caution. Experts in data science have raised concerns about a growing “reproducibility crisis” in scientific research applying ML models (Kapoor & Narayanan, 2023 ; Kaufman et al., 2012 ; Lones, 2024 ; Lucasius et al., 2025 ). For example, an ongoing study by Kapoor & Narayanan ( 2026 ) finds that the conclusions of over 600 recent applied-science papers rely on ML models whose outputs could not be reproduced. When re-created, most of these models performed worse than originally claimed. Irreproducibility issues have forced many scientific fields, like psychology (Bogdan, 2025 ; Kristal et al., 2020 ), economics (Camerer et al., 2016 ), and medicine (Greenwald, 1975 ), to create new research standards, but the problem has not yet risen to prominence in ML-based science. Simply put, reproducibility is the ability of independent researchers to obtain the same results by using the same data in the same conditions (ISO, 2023 ; National Academies, 2019 ). Kapoor & Narayanan find that ML research specifically is reproducible “if the code and data used to obtain the findings are available and the data are correctly analyzed” (2023; p. 2). They identify two common reasons why studies using ML models may be irreproducible. First, models can suffer from “data leakage”: where information that should be unavailable to a model at prediction time is inadvertently introduced while preparing data or training an ML model, leading to spurious relationships between independent and dependent variables. If a study uses a model that has been impacted by data leakage, its results rely on data that does not accurately reflect the hypothesis of the study. Second, many studies do not adequately describe the hardware, software, and data required to re-create an ML model (which we refer to as factors contributing to ‘computational reproducibility’ going forward). Without this information, it is not possible to re-create the experimental conditions in order to confirm or reject a study’s original conclusions. Critically, the mechanisms of ML reproducibility are well understood in both cases above; irreproducibility is caused by a failure to follow established best practices when building models and reporting results. An inability to reproduce results is a failure of scientific research. Scientific knowledge is advanced when researchers use “reproducible evidence” from a critical experiment to confirm and reject hypotheses (Platt, 1964 ; Popper, 1959 ), but this knowledge is impossible to validate when the evidence cannot be empirically replicated. A lack of reproducibility weakens the overall evidence for hypotheses in a field, undermining the validity of important conclusions (Open Science Collaboration, 2015 ). Irreproducible results can also propagate erroneous conclusions forward in the scientific literature. Researchers might pursue a hypothesis based on promising evidence that turns out to be irreproducible. This can cost millions of dollars, years of potential scientific progress, or even human lives (Begley & Ellis, 2012 ). Finally, irreproducibility undermines the public credibility of science. This is especially important for industrial ecologists, as public beliefs related to climate change and sustainability (the problems of primary interest for IE) are strongly motivated by trust in a scientific consensus (Hornsey et al., 2016 ) when compared to other fields (Stoetzer & Zimmermann, 2024 ). It is therefore critical for IE researchers, reviewers, and journal editors to be able to identify, avoid, and correct irreproducible results. A growing number of papers have examined the ways that ML is used in IE and broader sustainability research (Bozeman et al., 2024 ; Donati et al., 2022 ; Gohr et al., 2025 ; Rolnick et al., 2023 ; Zhao et al., 2024 ; Zhu et al., 2023 ). However, the IE community has not evaluated its use of ML to determine whether reproducibility is becoming an issue of concern. We address this gap by surveying 50 studies, published in the last five years, that use ML models to answer IE-related hypotheses. We perform the following analysis: First, we code each study against an ontology of ML reproducibility to determine if their reported research design could lead to data leakage and computational reproducibility errors. We highlight where IE research is meeting best practices for reproducibility, and where it is failing. Next, we perform a bibliometric analysis to determine where and how often potentially irreproducible studies are being cited. Finally, we offer recommendations to researchers, journals, and reviewers, backed by data science and ML literature, on how to minimize irreproducibility of ML research in IE. 2. METHODS 2.1 Literature survey and study selection We survey industrial ecology studies that use ML models as a prominent part of their methods. By ‘ML models’, we are referring specifically to the use of supervised ML models. These models seek to fit a function f that maps a target variable y to a set of related features X , ( \(\:f:X\to\:y\) ) by minimizing an objective function (LeCun et al., 2015 ). Researchers measure the performance of their models using a metric specific to the task at hand (e.g. F1 score for classification, root mean square error for regression). Supervised ML models are ‘trained’ on some known data \(\:\left[\right({x}_{1},{y}_{1}),\:...,\:({x}_{n},{y}_{n}\left)\right]\) , before being evaluated on unseen ‘test’ data. Examples of model architectures ( \(\:f\) ) include ensemble regressions like random forest and gradient boosting algorithms, shallow and deep neural-networks, support vector machines, and traditional linear models. In the studies we have selected, researchers use the ML models to explicitly or implicitly address two types of predictive scientific hypothesis: 1. As a tool for predicting y when it is difficult/time consuming to collect data for the variable, but data for X is known and abundant. A hypothesis for this use-case might be: Nighttime light intensity can be used to predict material stocks in a city. 2. As a tool for determining why X predicts y , and what features might be more predictive than others. A hypothesis for this use-case might be: The life-cycle emissions of urban agriculture are most strongly correlated with the lifetime and circularity of a farm. We do not include studies focusing on unsupervised ML in our survey. Unsupervised models aim to find patterns in unlabelled data (LeCun et al., 2015 ). Traditionally, unsupervised models are used for exploratory hypotheses in the pursuit of a deeper understanding of relationships between a variety of variables; while reproducibility remains important in these cases, the core challenge we address - data leakage - is not relevant. We also avoid evaluating studies which prompt generative AI like large language models as part of a predictive workflow (e.g. Balaji et al., 2023 ). Although we discuss the implication of the emergence of generative AI in Section 4 , these tools are, to date, less common in IE compared to other forms of ML research. By ‘a prominent part of their methods’, we mean that we select studies that highlight the use of an ML model in their title or abstract, or studies where the methods focus solely or overwhelmingly on the training and testing of a supervised ML model. Some studies use supervised ML as a step towards achieving a broader goal (e.g. gap-filling missing data for a larger material flow model). These types of studies are not included in our survey because they are harder to identify and tend to provide less information about their models, though ML may be just as critical to reproducibility of these studies. We collected 50 studies published by 5 different journals from 2021 to the end of 2025. We expect articles published within the last 5 years to be more representative of current practices in IE research, and to use more widespread tools for building ML models (e.g. ScikitLearn). Our method for finding studies was as follows: Based on expert judgement, we selected a subset of journals that publish IE research: Resources, Conservation, and Recycling (RCR) (n = 20); Journal of Industrial Ecology (JIE) (n = 15); Environmental Science & Technology (ES&T) (n = 7); Environmental Research Letters (ERL) (n = 5); Frontiers in Sustainability (FIS) (n = 3). These are journals whose aims and scopes explicitly mention IE or related methods (e.g. LCA, MFA) and whose editorial boards contain a large number of IE researchers. Starting with journal issues published in December 2025, we worked backwards through each issue, manually identifying studies whose titles or abstracts focus on the use of ‘machine learning’, ‘artificial intelligence’, or related methods. We then refined this initial pool of studies. We selected only the subset of studies whose hypothesis and methods fit the described selection criteria. We also filtered for studies which focus on a problem related to industrial ecology based on keywords and author judgement. The final studies cover a range of IE topics, model types, and dataset sizes (Fig. 1 ). We note that our survey is not a comprehensive review of all ML studies in IE research. Our goal is not to perform a full review of the literature (i.e. collect all important citations into one-place, study the evolution of ML use in IE, determine a field-wide consensus). Instead, our aim is to ‘audit’ the current practices of IE researchers using ML. 2.2 Evaluating studies for reproducibility We compare each study in our survey to an ontology of ML best practices for reproducibility (Fig. 2 ). We adapt the ontology introduced in Kapoor & Naranyanan (2023), who identify eight types of errors (grouped into three categories dubbed L1 , L2 , and L3 ) researchers make when creating ML models that lead to leakage and irreproducibility. A full description of the ontology, and examples of errors, are included in Supplementary File 1 (SI1). We note that Kapoor & Naranyanan identify errors that occur in three stages when building an ML model: A) the initial data processing stage, B) the feature selection and training stage, and C) the interpretation and dissemination stage. We make the following modifications to tailor Kapoor & Naranyanan’s ontology for our survey: ● To the error category L3 : Test set is not drawn from the distribution of scientific interest , we add an additional error L3.4, geographic leakage . This error occurs when the test set gains information about the training set due to some geospatial correlation, or otherwise does not reflect the “distribution of interest” for the study’s hypothesis. Geospatial analysis is common in IE research, which could lead to an outsized appearance of these errors compared to other scientific fields. ● We elevate Kapoor & Naranyanan’s check for “computational reproducibility issues” into its own error category, dubbed L4 . We define five distinct errors in this category which lead to irreproducibility: L4.1 : computing resources (hardware) not given , L4.2 hyperparameters not given , L4.3 data not open , L4.4 model not open , and L4.5 improper metric choice . Our analysis benefits from greater detail to clarify whether authors provided all, some, or none of the information required to computationally reproduce their ML models. 2.3 Coding studies Using the ontology, five authors coded the studies as follows. After joint conversations and practice review, the lead author created a shared reference document for the ontology based on Kapoor & Naranyanan’s original paper (included in SI1). We then evaluated a subset of the surveyed studies. We compared each study to the reference document, providing a score of 0 (follows best practices), 0.5 (uncertain/unclear), or 1 (does not follow best practices) for each error in the ontology, based on the description in the text (Fig. 2 ). For example, we reviewed a study and found that it does not describe the creation or use of a hold-out test set in the main text or supplementary information. For category L1.1 no test set , we gave the study a score of 1. We took steps to ensure reliable coding between researchers. Before beginning independent coding, each researcher coded the same 1–5 studies alone for inter-reliability training, and then met as a group to discuss and resolve discrepancies. For subsequent studies, we had at least two researchers check scores and the notes of the original coder to ensure agreement with the reference document. At the conclusion of coding, we met as a group to reach a consensus about final coding uncertainties before analyzing the data. The criteria for assigning scores to each error is outlined in SI1. We include intercoder reliability scores for two papers, coded independently post-training, in Supplementary Information SI2. Here we clarify some important study design choices. First, our coding does not fully confirm and correct irreproducible results in the surveyed papers. Rather, we have identified failures in best practices which could and would likely lead to a lack of reproducibility, based on our interpretation of the text, figures and supporting information. In the case of category L3 errors ( test set is not drawn from distribution of scientific interest ) specifically, confirming irreproducibility requires fully re-creating and re-training the models presented in each paper, which would be time-consuming and difficult given many papers with L3 errors also have data and model transparency issues. Additionally, we note that some errors lead more obviously to irreproducibility than others (see Discussion section 4.1 ), and some errors are difficult to identify with our coding method. For example, error L1.4 : duplicates across train and test set is difficult to confirm unless the authors explicitly describe a pre-processing or data collection technique that would duplicate values before creating the hold-out set (and would be impossible to validate if the dataset is not given). Following this, we have anonymized the studies included in our survey. Our goal is not to single out specific authors or papers for their failure to follow best practices. Instead, we aim to identify common and persistent errors in the use of ML for IE research in order to caution the community and set a path to future widespread reproducibility. We have anonymized all of the data and results for this paper and released them in a Zenodo repository (See Data Availability). 2.4 Bibliometric analysis We use an open source citation database to see where and how often the studies in our survey have been cited. We performed this analysis using OpenAlex (OpenAlex, 2026 ). OpenAlex is a continuously updated, non-profit research database that graphs the connections between papers, authors, and institutions. We queried the OpenAlex graph to find the papers that have cited our surveyed studies (forward citations) between the beginning of 2022 and February of 2026. We extracted the following information about studies in our survey and the works that cite them: ● Publication date ● Article type ● Publication venue (i.e. journal name) ● Field-weighted citation percentile. We then linked this bibliometric information to our best-practice coding. We analyze the citation trends of studies based on their score in each ontology error category. Full info on how we extracted our results from the OpenAlex API can be found in the Zenodo repository. 3. RESULTS 3.1 Coding of studies Across the 50 coded studies, we identify 115 unique descriptions of errors that could lead to irreproducibility (score of 1.0); 23 of these errors occur in the initial data processing stage, 2 occur during feature selection and training, and 90 occur when results are being presented and disseminated. Additionally, we find 103 instances where we are uncertain, based on the information given in the text, if an error occurred which could lead to irreproducibility (score of 0.5) (Fig. 3 ). Of the 50 studies, we identify 7 which have no clear errors in any category (score of 0.0 or 0.5 across the ontology), and 2 which we are able to fully confirm follow all best practices (score of 0.0 across the ontology). Transparency and computational reproducibility errors are far more common across the studies than clear instances of data leakage. Out of the 50 studies, 14 described a reproducibility error that would cause data leakage (L1, L2, or L3 errors). Conversely, 42 studies have a computational reproducibility issue (L4 errors). Common issues include partial disclosure of model hyper-parameters (e.g., ranges instead of final optimal values), no disclosure of the hardware that models are trained on, links to data and model repositories (e.g., github) that are broken/inoperable, and a lack of data disclosure without a stated reason. Some studies use confidential data that cannot be released; while these studies received a score of 0.5 or 1 for L4.3, this is not an indictment of the quality of these studies. Our results suggest that, in general, ML reproducibility issues in IE may be more a result of publication standards and communication rather than methodological failures by authors. However, this lack of disclosure is still an important issue. Broad computational reproducibility issues make it harder to assess whether subtle leakage mistakes have been made, especially for L3 errors. This may result in an under-estimation of data leakage across IE studies. Despite being less common, we identify a concerning pattern of reproducibility errors which could cause data leakage. L1.2 errors ( pre-processing before split ) are the most common leakage issue. 12 studies explicitly describe data pre-processing techniques (most notably normalization and gap-filling across the entire dataset) that would give information about the test set to the training set, before splitting their data. For an additional 12 studies, we are uncertain whether data is being properly pre-processed due to a lack of clarity in the methods. Though less common, three studies make L1.1 errors: they evaluate their model’s performance on the data it is trained on, instead of held-out data. This error is critical. It leaves the predictive hypothesis unaddressed and is contrary to widespread practices in supervised ML. Conversely, most studies deal with spatial auto-correlation well. We find no instances of confirmable L3.4 errors. Two studies present ML models which aim to predict future time-series values, but introduce temporal leakage by training the models with randomly split data. This means the model is trained using data “from the future” (Kapoor & Narayanan, 2023 ), invalidating the hypothesis that it can predict future values. All of these errors are the same as those that have been identified repeatedly as causing reproducibility issues in other scientific fields; in other words, IE is not uniquely immune to the ‘reproducibility crisis’ of applied ML research. 3.2 Citations of studies The studies that make reproducibility errors have been cited hundreds of times in the past 4–5 years (Fig. 4 ). These citations are spread across 193 unique journals/publication venues (Fig. 4 d). While a majority of these citations are of studies with computational reproducibility issues (400 instances), around 100 citations are of studies which describe leakage-related errors in their methodology. To get a sense of how material from potentially irreproducible studies is being used, we read the articles published in the journal Scientific Data that cite studies in our survey. Scientific Data is of particular concern because it publishes descriptions of research datasets. If outputs of studies with data leakage are being cited and used in these datasets, this could directly propagate false results forward. Five unique articles in Scientific Data cite studies that make leakage-related errors (specifically L1.1, L1.2, and L1.3 errors). Luckily, none of these studies use data from the outputs of these models. However, they do use the studies as supporting evidence of their predictive hypothesis; i.e. that features X can predict a target variable y . This raises concerns about the propagation of erroneous conclusions forward, as discussed in the introduction. The overall evidence of supporting citations in these Scientific Data articles is being undermined by reproducibility errors in the original studies. We also found an article in Scientific Data which cited a surveyed study to highlight that it had computational reproducibility issues, and that these issues were a motivation for their own work. This raises another consequence of reproducibility errors: a lack of open data and models is slowing down science. These errors force researchers to invest time and funding into recreating results which could have been openly available in the first place. When considering the overall citation graph of the studies (Fig. 4 e), it is not clear that studies or citations cluster based on how well they adhere to best practices. Several citations draw evidence from both best-practice and error-filled articles. This raises the question: are studies which make ML reproducibility errors cited less often than those which follow best practices? We compare the field-weighted citation percentile (FWCP) of studies to see (Fig. 5 a). We find little evidence that making reproducibility errors results in less citations. Where differences in citation percentile exist between the citations of studies making errors vs. following best practices, we only have a few samples (e.g. two data points for L3.1 ). When comparing the mean FWCP across all categories, there is no statistical evidence that studies following best practices (open data/models, no data leakage) are cited more often (Welch's unequal variance t-test p-value = 0.13–0.78). In other words, researchers are not rewarded for making their studies reproducible. We expand our analysis to compare citations of all the ML studies we surveyed to the citations of the average article in their respective journals (Fig. 5 bc ). Beyond 1–2 years since publication, IE studies that use supervised ML may accrue more citations than their peer non-ML articles in some journals. In the Journal of Industrial Ecology , the average ML study in our survey receives, on average, 17.5 citations in just over three years, compared to 12.7 citations across all articles in the journal. Surveyed studies exceed the mean expected citations by around three-quarters of a standard deviation. Recent research corroborates the finding that studies using ML tend to be cited more often (Hao et al., 2026 ). This means that as ML-based research becomes more common, and if steps are not taken to limit reproducibility errors in this research, an outsized amount of the IE literature could become irreproducible. 4. DISCUSSION AND RECOMMENDATIONS The results of our survey justify a call to action for ensuring reproducibility in IE research using ML. While other types of errors not addressed in this paper can affect the reproducibility of ML and non-ML research equally (e.g., p-hacking, unintentional errors of experimental procedures or code), we argue that issues outlined here - data leakage and computational reproducibility - are uniquely important to address. These reproducibility issues will increase in prevalence as ML is used and cited more frequently in IE research. This is likely to be accelerated by access to generative AI tools for coding, which will lower the barrier to entry and therefore minimum domain knowledge required to use ML for applied science. Increased use of large-language models (LLM) in predictive workflows could also make it more difficult to report the computational details of ML studies (due to the off-the-shelf, third-party nature of these models and their training details) and increase the chance that test data has been seen by the LLM during prior training, which would induce data leakage. Scientists have been studying reproducibility for decades (Baker, 2016 ; McNutt, 2014 ), and ML researchers specifically have a significant corpus focused on reducing data-leakage and improving ML reproducibility (Kapoor & Narayanan, 2023 ; Kaufman et al., 2012 ; Lones, 2024 ; Lucasius et al., 2025 ). We draw on these sources to make recommendations for improving reproducibility of ML in IE in the following sections. 4.1 Recommendations for researchers using machine learning The points summarize suggestions from the literature and can act as a starting point for industrial ecologists interested in working with machine learning. Ground your work in best practices from the outset . There are many high-quality guidelines for structuring a machine learning study to avoid leakage (Kaufman et al., 2012 ; Lones, 2024 ; Mitchell et al., 2018 ) and to maximize data and model transparency (Chen et al., 2019 ; Nosek et al., 2015 ; Nosek & Lakens, 2014 ). We recommend starting with Kapoor & Narayanan ‘model info sheets’, which can be used to self-evaluate a study against the reproducibility ontology in our survey (2023). The first step by any IE researcher who plans to use ML should be to create a reproducible workflow based on these resources. Even so, complex workflows might introduce subtle sources of leakage that are hard for non-experts to recognize. In many cases, IE researchers may also benefit from consulting directly with domain experts in ML, having these experts suggest or build a model while the IE researcher weighs in on assumptions, constraints, and design evaluation. Taking a reproducibility-first approach is particularly important for the kinds of data-leakage errors that were most frequent in our survey; namely, not cleanly separating and isolating a test set at the beginning of the model-building process. Describe best practices in writing . Many studies in the survey had vague method descriptions, which made it difficult to determine whether they made reproducibility errors. In contrast, the few studies following best practices all had plain-language descriptions of the steps they took to avoid reproducibility errors (e.g. “we split the data, and then performed pre-processing”, “we used time-series cross validation to avoid temporal leakage”). We recommend that authors plainly describe, as much as is reasonably possible, their reproducible workflows in writing. This will allow readers to quickly confirm the quality of a study and promote reproducible work. Be clear why data or models must be withheld . Unexplained withholding of data and broken repository links were common in our survey. Data and model transparency should be the default assumption for ML studies. If this is not possible for reasons such as the use of proprietary data, researchers should strive to be transparent about why data or computational details are withheld and where similar data could reasonably be obtained. Maximizing the availability of models and data helps avoid the need for repeated work in the future, and also makes evaluating a study for data leakage easier. Don’t overfit the problem . In our survey, we were often unable to confirm best practices for studies which used complex, multi-model workflows. It is not always clear what value complex workflows add to a predictive hypothesis, especially when datasets are small (the mean dataset size was < 1000 for studies in our survey). We recommend reporting results for the simplest model, usually a linear regression, before moving on to a more complex model. These models have simpler diagnostics and are less likely to have unnoticed reproducibility errors. Additionally, some of the best-practice studies we surveyed found that linear models performed nearly as well as more complex models for their use case. These linear models can serve as a baseline for benchmarking. We also recommend that researchers consider whether ML is necessary to test their hypothesis in the first place. ML reproducibility errors can be avoided if empirical data are collected rather than predicted. Hunt for assumptions . Scientific research should consider the null hypothesis — that \(\:f\) does not map \(\:X\to\:y\) — an acceptable outcome (Greenwald, 1975 ). Measuring the success of an ML study solely by test metrics like R 2 makes it easy to overlook reproducibility errors. Because of this, we encourage IE researchers to adopt an “assumption-hunting” attitude (Saltelli & Funtowicz, 2014 ) when engaging in and with ML research. Similar to how industrial ecologists perform sensitivity analysis on their LCA or MFA, researchers must look for data and process assumptions that condition the results of an ML model. When scrutinizing their work, researchers must ask themselves: am I using data which reflects the process I am trying to model? Are there hidden assumptions in my process which could be giving me a misleading result? Have I been transparent enough for my model to be used by other researchers? 4.2 Recommendations for publishers and reviewers of machine learning research We briefly provide recommendations for how those publishing or reviewing IE papers using ML can help foster reproducible research. For journals publishing research. Journals could make the reproducibility check-lists discussed above mandatory for ML papers under review (Kapoor & Narayanan, 2026 ; Mitchell et al., 2018 ). These could be required as self-reported supporting information, similar to existing standards like data badges (Hertwich et al., 2018 ). Going a step further, editorial offices or peer reviewers could be asked to conduct similar checks explicitly as part of the publication process - ideally by a reviewer with machine-learning expertise. Data confidentiality will likely continue to limit the ability of papers to meet L4 best practices. We do not suggest that this should prevent the publishing of papers, but the reasons for withholding the data or models in an ML study should always be justified. Journals can also encourage authors to host models and data on persistent services (e.g., hosting on Google Colab) to encourage computational transparency and quick identification and correction of reproducibility errors. For reviewers . Some reproducibility errors are more disqualifying than others, and reviewers should keep the overall quality of a paper in mind when evaluating a study. Non-expert reviewers assigned to papers applying ML should look for easy-to-spot reproducibility errors that are totally disqualifying - specifically, a lack of a hold-out test set for evaluating models, temporal leakage, or normalization and gap-filling before splitting data into training and test sets. These errors do not require ML expertise to identify, and they put any hypothetical results into serious dispute. Confidential datasets or models may not be disqualifying if the rest of the study follows best practice. If reviewers are unsure if reproducibility errors are present (i.e. would code an error as 0.5 based on the ontology), it is likely that future readers will probably be as well. In these instances, we recommend reviewers ask for a revision of the article to clarify reproducibility of study. 5. CONCLUSION The power of machine learning (ML) to analyze old questions faster and generate new questions is energizing industrial ecology (IE) researchers. However, IE's rapid adoption of ML has outpaced its commitment to scientific reproducibility. This paper has identified the common challenges of reproducibility within IE papers using ML. These challenges are highlighted by the rapid growth of ML usage by industrial ecologists, and the continued citation of works which make mistakes that can lead to irreproducibility. Crises in other disciplines show that these mistakes can muddy the evidence base of literature and slow down the progress towards answering important scientific questions. The IE community stands to benefit from acting now to improve the reproducibility of its ML applications. Declarations Competing Interests Author FD serves as associate editor for the Journal of Industrial Ecology. Our survey includes papers co-authored by S Saxe and QT. These authors were not involved in coding their own papers, nor were they shown the final results for their own papers. Author Contribution **Conceptualization** : KHR, FD, QT, JWB, SvL, S Suh, S Saxe, IDP, JKH; **Methodology** : KHR, FD, QT, SvL, S Suh, S Saxe, IDP, JKH; **Data collection** : KHR, JWB, NR, BH, JKH; **Visualization** : KHR; **Validation** : KHR, JWB, NR, BH, JKH; **Writing (original draft, review and editing)** : KHR, FD, QT, JWB, SvL, S Suh, S Saxe, IDP, JKH; **Funding acquisition** : KHR, QT, JKH Acknowledgement This research was undertaken, in part, thanks to funding from the Natural Sciences and Engineering Research Council of Canada (NSERC) Vanier scholarship held by KHR, NSERC funding [reference number RGPIN-2021-02841] held by QT, and University of Wyoming School of Computing funding held by JKH. Data Availability All data and code that support the findings of this study are available via Zenodo at [https://doi.org/10.5281/zenodo.19153101](https:/doi.org/10.5281/zenodo.19153101) . We have anonymized the identities of all surveyed papers and authors in the repository. References Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature , 533 (7604), 452–454. https://doi.org/10.1038/533452a Balaji, B., Vunnava, G., Domingo, N., Gupta, S., Gupta, H., Guest, G., & Srinivasan, A. (2023). Flamingo: Environmental impact factor matching for life cycle assessment with zero-shot machine learning . https://www.amazon.science/publications/flamingo-environmental-impact-factor-matching-for-life-cycle-assessment-with-zero-shot-machine-learning Begley, C. G., & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature , 483 (7391), 531–533. https://doi.org/10.1038/483531a Bogdan, P. C. (2025). One Decade Into the Replication Crisis, How Have Psychological Results Changed? Advances in Methods and Practices in Psychological Science , 8 (2), 25152459251323480. https://doi.org/10.1177/25152459251323480 Bozeman, J. F., Hollauer, C., Ramshankar, A. T., Nakkasunchi, S., Jambeck, J., Hicks, A., Bilec, M., McCauley, D., & Heidrich, O. (2024). Embed systemic equity throughout industrial ecology applications: How to address machine learning unfairness and bias. Journal of Industrial Ecology , 28 (6), 1362–1376. https://doi.org/10.1111/jiec.13509 Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science , 351 (6280), 1433–1436. https://doi.org/10.1126/science.aaf0918 Chen, X., Dallmeier-Tiessen, S., Dasler, R., Feger, S., Fokianos, P., Gonzalez, J. B., Hirvonsalo, H., Kousidis, D., Lavasa, A., Mele, S., Rodriguez, D. R., Šimko, T., Smith, T., Trisovic, A., Trzcinska, A., Tsanaktsidis, I., Zimmermann, M., Cranmer, K., Heinrich, L., … Neubert, S. (2019). Open is not enough. Nature Physics , 15 (2), 113–119. https://doi.org/10.1038/s41567-018-0342-2 Cullen, L., Marinoni, A., & Cullen, J. (2024). Machine learning for gap-filling in greenhouse gas emissions databases. Journal of Industrial Ecology , 28 (4), 636–647. https://doi.org/10.1111/jiec.13507 Donati, F., Dente, S. M. R., Li, C., Vilaysouk, X., Froemelt, A., Nishant, R., Liu, G., Tukker, A., & Hashimoto, S. (2022). The future of artificial intelligence in the context of industrial ecology. Journal of Industrial Ecology , 26 (4), 1175–1181. https://doi.org/10.1111/jiec.13313 Gohr, C., Rodríguez, G., Belomestnykh, S., Berg-Moelleken, D., Chauhan, N., Engler, J.-O., Heydebreck, L. V., Hintz, M. J., Kretschmer, M., Krügermeier, C., Meinberg, J., Rau, A.-L., Schwenck, C., Aoulkadi, I., Poll, S., Frank, E., Creutzig, F., Lemke, O., Maushart, M., … Von Wehrden, H. (2025). Artificial intelligence in sustainable development research. Nature Sustainability , 8 (8), 970–978. https://doi.org/10.1038/s41893-025-01598-6 Gong, Y., Ma, F., Wang, H., Tzachor, A., Sun, W., Zhu, J., Liu, G., & Schandl, H. (2025). The evolution of research at the intersection of industrial ecology and artificial intelligence. Journal of Industrial Ecology , 29 (2), 440–457. https://doi.org/10.1111/jiec.13612 Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin , 82 (1), 1–20. https://doi.org/10.1037/h0076157 Hao, Q., Xu, F., Li, Y., & Evans, J. (2026). Artificial intelligence tools expand scientists’ impact but contract science’s focus. Nature , 649 (8099), 1237–1243. https://doi.org/10.1038/s41586-025-09922-y Hertwich, E., Heeren, N., Kuczenski, B., Majeau-Bettez, G., Myers, R. J., Pauliuk, S., Stadler, K., & Lifset, R. (2018). Nullius in Verba1: Advancing Data Transparency in Industrial Ecology. Journal of Industrial Ecology , 22 (1), 6–17. https://doi.org/10.1111/jiec.12738 Hornsey, M. J., Harris, E. A., Bain, P. G., & Fielding, K. S. (2016). Meta-analyses of the determinants and outcomes of belief in climate change. Nature Climate Change , 6 (6), 622–626. https://doi.org/10.1038/nclimate2943 Huang, W., Olson, A. W., Khalil, E. B., & Saxe, S. (2025). Image-based prediction of residential building attributes with deep learning. Journal of Industrial Ecology , 29 (1), 81–95. https://doi.org/10.1111/jiec.13591 ISO. (2023). ISO 5725-1 . https://www.iso.org/obp/ui/#iso:std:iso:5725 :-1:ed-2:v1:en Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns , 4 (9), 100804. https://doi.org/10.1016/j.patter.2023.100804 Kapoor, S., & Narayanan, A. (2026). Leakage and the Reproducibility Crisis in ML-based Science . https://reproducible.cs.princeton.edu/#citation Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data , 6 (4), 1–21. https://doi.org/10.1145/2382577.2382579 Kristal, A. S., Whillans, A. V., Bazerman, M. H., Gino, F., Shu, L. L., Mazar, N., & Ariely, D. (2020). Signing at the beginning versus at the end does not decrease dishonesty. Proceedings of the National Academy of Sciences , 117 (13), 7103–7107. https://doi.org/10.1073/pnas.1911695117 LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature , 521 (7553), 436–444. https://doi.org/10.1038/nature14539 Lones, M. A. (2021). How to avoid machine learning pitfalls: A guide for academic researchers . https://doi.org/10.48550/ARXIV.2108.02497 Lones, M. A. (2024). Avoiding common machine learning pitfalls. Patterns , 5 (10), 101046. https://doi.org/10.1016/j.patter.2024.101046 Lucasius, C., Ali, M., Patel, T., Kundur, D., Szatmari, P., Strauss, J., & Battaglia, M. (2025). A procedural overview of why, when and how to use machine learning for psychiatry. Nature Mental Health , 3 (1), 8–18. https://doi.org/10.1038/s44220-024-00367-2 McNutt, M. (2014). Reproducibility. Science , 343 (6168), 229–229. https://doi.org/10.1126/science.1250475 Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2018). Model Cards for Model Reporting . https://doi.org/10.48550/ARXIV.1810.03993 National Academies. (2019). Reproducibility and Replicability in Science . https://www.nationalacademies.org/projects/DBASSE-BBCSS-17-03 Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., … Yarkoni, T. (2015). Promoting an open research culture. Science , 348 (6242), 1422–1425. https://doi.org/10.1126/science.aab2374 Nosek, B. A., & Lakens, D. (2014). Registered Reports: A Method to Increase the Credibility of Published Results. Social Psychology , 45 (3), 137–141. https://doi.org/10.1027/1864-9335/a000192 Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science , 349 (6251), aac4716. https://doi.org/10.1126/science.aac4716 OpenAlex. (2026). OpenAlex [Dataset]. https://openalex.org/ Platt, J. R. (1964). Strong Inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others. Science , 146 (3642), 347–353. https://doi.org/10.1126/science.146.3642.347 Popper, K. (1959). The logic of scientific discovery . https://psycnet.apa.org/record/1961-02882-000 Rolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K., Lacoste, A., Sankaran, K., Ross, A. S., Milojevic-Dupont, N., Jaques, N., Waldman-Brown, A., Luccioni, A. S., Maharaj, T., Sherwin, E. D., Mukkavilli, S. K., Kording, K. P., Gomes, C. P., Ng, A. Y., Hassabis, D., Platt, J. C., … Bengio, Y. (2023). Tackling Climate Change with Machine Learning. ACM Computing Surveys , 55 (2), 1–96. https://doi.org/10.1145/3485128 Saltelli, A., & Funtowicz, S. (2014). When All Models are Wrong. Issues in Science and Technology , 30 (2). https://issues.org/andrea/ Sarswat, P. K., Singh, R. S., & Pathapati, S. V. S. H. (2024). Real time electronic-waste classification algorithms using the computer vision based on Convolutional Neural Network (CNN): Enhanced environmental incentives. Resources, Conservation and Recycling , 207 , 107651. https://doi.org/10.1016/j.resconrec.2024.107651 Stoetzer, L. S., & Zimmermann, F. (2024). A representative survey experiment of motivated climate change denial. Nature Climate Change , 14 (2), 198–204. https://doi.org/10.1038/s41558-023-01910-2 Tu, Q., Guo, J., Li, N., Qi, J., & Xu, M. (2024). Mitigating Grand Challenges in Life Cycle Inventory Modeling through the Applications of Large Language Models. Environmental Science & Technology , 58 (44), 19595–19603. https://doi.org/10.1021/acs.est.4c07634 Zhao, B., Yu, Z., Wang, H., Shuai, C., Qu, S., & Xu, M. (2024). Data Science Applications in Circular Economy: Trends, Status, and Future. Environmental Science & Technology , 58 (15), 6457–6474. https://doi.org/10.1021/acs.est.3c08331 Zhu, J.-J., Yang, M., & Ren, Z. J. (2023). Machine Learning in Environmental Research: Common Pitfalls and Best Practices. Environmental Science & Technology , 57 (46), 17671–17689. https://doi.org/10.1021/acs.est.3c00026 26 Additional Declarations Competing interest reported. Author FD serves as associate editor for the Journal of Industrial Ecology. Our survey includes papers co-authored by S Saxe and QT. These authors were not involved in coding their own papers, nor were they shown the final results for their own papers. Supplementary Files SI130032026.docx SI230032026.docx Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 22 Apr, 2026 Reviewers agreed at journal 13 Apr, 2026 Reviewers agreed at journal 09 Apr, 2026 Reviewers agreed at journal 08 Apr, 2026 Reviewers invited by journal 07 Apr, 2026 Editor assigned by journal 31 Mar, 2026 Submission checks completed at journal 31 Mar, 2026 First submitted to journal 30 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9270723","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":620659141,"identity":"36d44a84-9c1e-4839-91b7-f3617676148c","order_by":0,"name":"Keagan Hudson Rankin","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Keagan","middleName":"Hudson","lastName":"Rankin","suffix":""},{"id":620659142,"identity":"3a53d5aa-ae2c-44c3-913c-ceb6e14f2641","order_by":1,"name":"Franco Donati","email":"","orcid":"","institution":"Leiden University","correspondingAuthor":false,"prefix":"","firstName":"Franco","middleName":"","lastName":"Donati","suffix":""},{"id":620659143,"identity":"39cf3065-9e19-481e-9dd0-7fb65a492ae6","order_by":2,"name":"Qingshi Tu","email":"","orcid":"","institution":"University of British Columbia","correspondingAuthor":false,"prefix":"","firstName":"Qingshi","middleName":"","lastName":"Tu","suffix":""},{"id":620659144,"identity":"8a26be95-2abb-4319-92b0-25eb4585a4fb","order_by":3,"name":"Jesse Ward-Bond","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Jesse","middleName":"","lastName":"Ward-Bond","suffix":""},{"id":620659145,"identity":"4c290a95-f6f8-4bb3-8249-de53e9e376bc","order_by":4,"name":"Nolan Reitz","email":"","orcid":"","institution":"University of Wyoming","correspondingAuthor":false,"prefix":"","firstName":"Nolan","middleName":"","lastName":"Reitz","suffix":""},{"id":620659146,"identity":"fc24291f-f4d9-4bd2-97d2-3a07dd24e50d","order_by":5,"name":"Brianna Hiser","email":"","orcid":"","institution":"University of Wyoming","correspondingAuthor":false,"prefix":"","firstName":"Brianna","middleName":"","lastName":"Hiser","suffix":""},{"id":620659147,"identity":"5c5b6ad1-6eff-4918-90b9-bd81640f88d0","order_by":6,"name":"Simon van Lierde","email":"","orcid":"","institution":"Leiden University","correspondingAuthor":false,"prefix":"","firstName":"Simon","middleName":"van","lastName":"Lierde","suffix":""},{"id":620659148,"identity":"5313ac98-dd39-4f8a-8a41-4b3c591a4858","order_by":7,"name":"Sangwon Suh","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Sangwon","middleName":"","lastName":"Suh","suffix":""},{"id":620659149,"identity":"b61a7eef-4ff7-471f-a1ee-5e5c707ba27f","order_by":8,"name":"Shoshanna Saxe","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Shoshanna","middleName":"","lastName":"Saxe","suffix":""},{"id":620659150,"identity":"1f33cd12-a4f8-4bb3-95ed-e8a9bba4d596","order_by":9,"name":"I. Daniel Posen","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"I.","middleName":"Daniel","lastName":"Posen","suffix":""},{"id":620659151,"identity":"355e8747-8af7-4fa9-8c61-632906f5d5bd","order_by":10,"name":"Jason K Hawes","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAs0lEQVRIiWNgGAWjYBACxhlgygbKZSNeSxoJWhgkwORhErQwz25+JvHhz/nEDTeSDzB8KDtMWAvjnGNmkjPbbgO1pCUwzjhHjJYZCWa3eRtuJ267kWPAzNtGlJb0b7f//DkH0fKXOC05ZrcZ2A5AtDASqaX8Z29bsvH+M88SDvacSyesxXBG+maDH3/sZGe2Jx988KPMmggtDTCWQALDAcLqgUAezuInTsMoGAWjYBSMQAAAMuFDOkKrH3gAAAAASUVORK5CYII=","orcid":"","institution":"University of Wyoming","correspondingAuthor":true,"prefix":"","firstName":"Jason","middleName":"K","lastName":"Hawes","suffix":""}],"badges":[],"createdAt":"2026-03-30 17:54:50","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9270723/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9270723/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106960482,"identity":"d261f4d5-9c00-4398-809d-70e8ae7432bc","added_by":"auto","created_at":"2026-04-15 09:21:23","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":234816,"visible":true,"origin":"","legend":"\u003cp\u003eSummary statistics of surveyed studies. a. Journals of surveyed studies. b. Sample sizes in studies, organized by model type. c. Industrial ecology topic of interest to studies (authors judgement). d. Types of data used in studies.\u003c/p\u003e","description":"","filename":"image1.png","url":"https://assets-eu.researchsquare.com/files/rs-9270723/v1/ff6d25301450ef3230c74813.png"},{"id":106960209,"identity":"ca98abbc-20ad-4103-9ddd-7544ff69d518","added_by":"auto","created_at":"2026-04-15 09:19:23","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":526193,"visible":true,"origin":"","legend":"\u003cp\u003eAn ontology of best-practice errors for ML reproducibility, adapted from Kapoor \u0026amp; Naranyanan \u003ca href=\"https://www.zotero.org/google-docs/?x205ng\"\u003e(2023)\u003c/a\u003e. We code each study in our survey, giving them a score of 0 (follows best practice), 0.5 (uncertain or unclear), and 1 (does not follow best practice) for every type of error in the ontology (Section 2.3); the scores shown are illustrative, not a synopsis of results.\u003c/p\u003e","description":"","filename":"image2.png","url":"https://assets-eu.researchsquare.com/files/rs-9270723/v1/3ef2895d7bd869ad75f56a8e.png"},{"id":106821075,"identity":"9939856b-f2c4-4151-a358-70403b3e8156","added_by":"auto","created_at":"2026-04-13 18:46:47","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":280518,"visible":true,"origin":"","legend":"\u003cp\u003eReproducibility coding of studies in our survey. We split results based on the ontology presented in the methods section. Each bar denotes an error which could lead to irreproducibility. Each grouping of bars denotes a stage of model development from the ontology.\u003c/p\u003e","description":"","filename":"image3.png","url":"https://assets-eu.researchsquare.com/files/rs-9270723/v1/d26dc4c16f484048cef8993c.png"},{"id":106960519,"identity":"df16c5a7-b348-4203-95fe-5008197c49d4","added_by":"auto","created_at":"2026-04-15 09:21:35","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":589145,"visible":true,"origin":"","legend":"\u003cp\u003eOverall citations of studies in our survey. The plots on the left show the cumulative, non-unique citations of studies with at least a single score of 1 or 0.5 for an error occurring in a. The initial data processing stage (\u003cem\u003eL1 \u003c/em\u003eerrors), b. The feature selection and training stage (\u003cem\u003eL2\u003c/em\u003e,\u003cem\u003e L3 \u003c/em\u003eerrors) and c. the results interpretation and dissemination stage (\u003cem\u003eL4\u003c/em\u003e errors). d. The total non-unique citation occurrences of studies which make a data-leakage (\u003cem\u003eL1, L2, or L3\u003c/em\u003e)\u003cem\u003e \u003c/em\u003eerror, organized by the journal that the citing study is published in. The top 15 journals are shown. e. network of the surveyed studies and the articles cite them. Arrows point to citing articles. A total of 193 journals and 381 unique articles cite studies which make a reproducibility error.\u003c/p\u003e","description":"","filename":"image4.png","url":"https://assets-eu.researchsquare.com/files/rs-9270723/v1/b20550949082ff94408bd97d.png"},{"id":106821077,"identity":"8e8aaed7-cb84-4778-a362-fbe48714e37c","added_by":"auto","created_at":"2026-04-13 18:46:47","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":284891,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of citations across best practices. a. Comparison of field weighted citation percentile (FWCP) across studies, grouped by each error in the ontology. Missing bars are for categories where we found no best-practice errors. The legend and plots on the right compare the average accumulation of citations for studies in our survey compared to all articles published during the same period in b. the \u003cem\u003eJournal of Industrial Ecology\u003c/em\u003e and c. \u003cem\u003eResource, Conservation, and Recycling\u003c/em\u003e. We omit other journals due to low sample size. The range shows the standard deviation (σ) around the mean.\u003c/p\u003e","description":"","filename":"image5.png","url":"https://assets-eu.researchsquare.com/files/rs-9270723/v1/b892b9fe30bf39c6a0ec3808.png"},{"id":106994124,"identity":"62ed0f2f-598f-4d83-a554-cb6f2044b82a","added_by":"auto","created_at":"2026-04-15 15:04:30","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2225640,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9270723/v1/c42905f2-d081-4d69-add9-5ccba056a9ba.pdf"},{"id":106821071,"identity":"7eca1f25-0e1d-418d-acb9-fd9b6a3fee38","added_by":"auto","created_at":"2026-04-13 18:46:47","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":35605,"visible":true,"origin":"","legend":"","description":"","filename":"SI130032026.docx","url":"https://assets-eu.researchsquare.com/files/rs-9270723/v1/d1bf85c218fb82262345b34e.docx"},{"id":106821073,"identity":"8b6f6c41-5fee-4995-84ad-d728d31c2a66","added_by":"auto","created_at":"2026-04-13 18:46:47","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":24569,"visible":true,"origin":"","legend":"","description":"","filename":"SI230032026.docx","url":"https://assets-eu.researchsquare.com/files/rs-9270723/v1/825865c43a1f7fcf53dcbe73.docx"}],"financialInterests":"Competing interest reported. Author FD serves as associate editor for the Journal of Industrial Ecology. Our survey includes papers co-authored by S Saxe and QT. These authors were not involved in coding their own papers, nor were they shown the final results for their own papers.","formattedTitle":"A call to ensure reproducibility of machine learning applications in industrial ecology","fulltext":[{"header":"1. INTRODUCTION","content":"\u003cp\u003eIndustrial ecologists, like many other domain researchers, are increasingly using supervised machine learning (ML) models in their research. The number of industrial ecology (IE) related papers that use just the keyword \u0026ldquo;machine learning\u0026rdquo; has increased from less than 20 papers per year in 2014 to an average of 170 papers per year since 2020 (Gong et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). This increase has been driven by the growth of IE data availability as well as an increased awareness and use of machine learning methods in applied science (Hao et al., \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2026\u003c/span\u003e). Supervised ML has the potential to accelerate the research and practical goals of IE. Recent use-cases of ML in IE research include (among others):\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eGap-filling and prediction of emission inventories where data is unavailable (Cullen et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eSpeeding up or automating traditionally repetitive tasks in life-cycle assessment (LCA) and complex models like IAMs (Balaji et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2023\u003c/span\u003e,2025; Li et al. 2025).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003ePredicting properties of buildings (e.g. height, material content) and stocks in the built-environment (Huang et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eSupporting material recovery and circular economy practices by identifying materials and components in waste streams (Sarswat et al., \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eThese papers show how ML can inform policies, create data, and support decision making. However, recent literature also shows that increasing use of ML must be met with caution. Experts in data science have raised concerns about a growing \u0026ldquo;reproducibility crisis\u0026rdquo; in scientific research applying ML models (Kapoor \u0026amp; Narayanan, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Kaufman et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2012\u003c/span\u003e; Lones, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Lucasius et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). For example, an ongoing study by Kapoor \u0026amp; Narayanan (\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) finds that the conclusions of over 600 recent applied-science papers rely on ML models whose outputs could not be reproduced. When re-created, most of these models performed worse than originally claimed. Irreproducibility issues have forced many scientific fields, like psychology (Bogdan, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Kristal et al., \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), economics (Camerer et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2016\u003c/span\u003e), and medicine (Greenwald, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e1975\u003c/span\u003e), to create new research standards, but the problem has not yet risen to prominence in ML-based science.\u003c/p\u003e \u003cp\u003eSimply put, reproducibility is the ability of independent researchers to obtain the same results by using the same data in the same conditions (ISO, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; National Academies, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). Kapoor \u0026amp; Narayanan find that ML research specifically is reproducible \u0026ldquo;if the code and data used to obtain the findings are available and the data are correctly analyzed\u0026rdquo; (2023; p. 2). They identify two common reasons why studies using ML models may be irreproducible. First, models can suffer from \u0026ldquo;data leakage\u0026rdquo;: where information that should be unavailable to a model at prediction time is inadvertently introduced while preparing data or training an ML model, leading to spurious relationships between independent and dependent variables. If a study uses a model that has been impacted by data leakage, its results rely on data that does not accurately reflect the hypothesis of the study. Second, many studies do not adequately describe the hardware, software, and data required to re-create an ML model (which we refer to as factors contributing to \u0026lsquo;computational reproducibility\u0026rsquo; going forward). Without this information, it is not possible to re-create the experimental conditions in order to confirm or reject a study\u0026rsquo;s original conclusions. Critically, the mechanisms of ML reproducibility are well understood in both cases above; irreproducibility is caused by a failure to follow established best practices when building models and reporting results.\u003c/p\u003e \u003cp\u003eAn inability to reproduce results is a failure of scientific research. Scientific knowledge is advanced when researchers use \u0026ldquo;reproducible evidence\u0026rdquo; from a critical experiment to confirm and reject hypotheses (Platt, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e1964\u003c/span\u003e; Popper, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e1959\u003c/span\u003e), but this knowledge is impossible to validate when the evidence cannot be empirically replicated. A lack of reproducibility weakens the overall evidence for hypotheses in a field, undermining the validity of important conclusions (Open Science Collaboration, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2015\u003c/span\u003e). Irreproducible results can also propagate erroneous conclusions forward in the scientific literature. Researchers might pursue a hypothesis based on promising evidence that turns out to be irreproducible. This can cost millions of dollars, years of potential scientific progress, or even human lives (Begley \u0026amp; Ellis, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2012\u003c/span\u003e). Finally, irreproducibility undermines the public credibility of science. This is especially important for industrial ecologists, as public beliefs related to climate change and sustainability (the problems of primary interest for IE) are strongly motivated by trust in a scientific consensus (Hornsey et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2016\u003c/span\u003e) when compared to other fields (Stoetzer \u0026amp; Zimmermann, \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). It is therefore critical for IE researchers, reviewers, and journal editors to be able to identify, avoid, and correct irreproducible results.\u003c/p\u003e \u003cp\u003eA growing number of papers have examined the ways that ML is used in IE and broader sustainability research (Bozeman et al., \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Donati et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Gohr et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Rolnick et al., \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Zhao et al., \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Zhu et al., \u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). However, the IE community has not evaluated its use of ML to determine whether reproducibility is becoming an issue of concern. We address this gap by surveying 50 studies, published in the last five years, that use ML models to answer IE-related hypotheses. We perform the following analysis: First, we code each study against an ontology of ML reproducibility to determine if their reported research design could lead to data leakage and computational reproducibility errors. We highlight where IE research is meeting best practices for reproducibility, and where it is failing. Next, we perform a bibliometric analysis to determine where and how often potentially irreproducible studies are being cited. Finally, we offer recommendations to researchers, journals, and reviewers, backed by data science and ML literature, on how to minimize irreproducibility of ML research in IE.\u003c/p\u003e"},{"header":"2. METHODS","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Literature survey and study selection\u003c/h2\u003e \u003cp\u003eWe survey industrial ecology studies that use ML models as a prominent part of their methods. By \u0026lsquo;ML models\u0026rsquo;, we are referring specifically to the use of supervised ML models. These models seek to fit a function \u003cem\u003ef\u003c/em\u003e that maps a target variable \u003cem\u003ey\u003c/em\u003e to a set of related features \u003cem\u003eX\u003c/em\u003e, (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:f:X\\to\\:y\\)\u003c/span\u003e\u003c/span\u003e) by minimizing an objective function (LeCun et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2015\u003c/span\u003e). Researchers measure the performance of their models using a metric specific to the task at hand (e.g. F1 score for classification, root mean square error for regression). Supervised ML models are \u0026lsquo;trained\u0026rsquo; on some known data \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\left[\\right({x}_{1},{y}_{1}),\\:...,\\:({x}_{n},{y}_{n}\\left)\\right]\\)\u003c/span\u003e\u003c/span\u003e, before being evaluated on unseen \u0026lsquo;test\u0026rsquo; data. Examples of model architectures (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:f\\)\u003c/span\u003e\u003c/span\u003e) include ensemble regressions like random forest and gradient boosting algorithms, shallow and deep neural-networks, support vector machines, and traditional linear models. In the studies we have selected, researchers use the ML models to explicitly or implicitly address two types of \u003cem\u003epredictive\u003c/em\u003e scientific hypothesis:\u003c/p\u003e \u003cp\u003e1. As a tool for predicting \u003cem\u003ey\u003c/em\u003e when it is difficult/time consuming to collect data for the variable, but data for \u003cem\u003eX\u003c/em\u003e is known and abundant. A hypothesis for this use-case might be: \u003cem\u003eNighttime light intensity can be used to predict material stocks in a city.\u003c/em\u003e\u003c/p\u003e \u003cp\u003e2. As a tool for determining why \u003cem\u003eX\u003c/em\u003e predicts \u003cem\u003ey\u003c/em\u003e, and what features might be more predictive than others. A hypothesis for this use-case might be: \u003cem\u003eThe life-cycle emissions of urban agriculture are most strongly correlated with the lifetime and circularity of a farm.\u003c/em\u003e\u003c/p\u003e \u003cp\u003eWe do not include studies focusing on unsupervised ML in our survey. Unsupervised models aim to find patterns in unlabelled data (LeCun et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2015\u003c/span\u003e). Traditionally, unsupervised models are used for exploratory hypotheses in the pursuit of a deeper understanding of relationships between a variety of variables; while reproducibility remains important in these cases, the core challenge we address - data leakage - is not relevant. We also avoid evaluating studies which prompt generative AI like large language models as part of a predictive workflow (e.g. Balaji et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Although we discuss the implication of the emergence of generative AI in Section \u003cspan refid=\"Sec10\" class=\"InternalRef\"\u003e4\u003c/span\u003e, these tools are, to date, less common in IE compared to other forms of ML research.\u003c/p\u003e \u003cp\u003eBy \u0026lsquo;a prominent part of their methods\u0026rsquo;, we mean that we select studies that highlight the use of an ML model in their title or abstract, or studies where the methods focus solely or overwhelmingly on the training and testing of a supervised ML model. Some studies use supervised ML as a step towards achieving a broader goal (e.g. gap-filling missing data for a larger material flow model). These types of studies are not included in our survey because they are harder to identify and tend to provide less information about their models, though ML may be just as critical to reproducibility of these studies.\u003c/p\u003e \u003cp\u003eWe collected 50 studies published by 5 different journals from 2021 to the end of 2025. We expect articles published within the last 5 years to be more representative of current practices in IE research, and to use more widespread tools for building ML models (e.g. ScikitLearn). Our method for finding studies was as follows: Based on expert judgement, we selected a subset of journals that publish IE research: Resources, Conservation, and Recycling (RCR) (n\u0026thinsp;=\u0026thinsp;20); Journal of Industrial Ecology (JIE) (n\u0026thinsp;=\u0026thinsp;15); Environmental Science \u0026amp; Technology (ES\u0026amp;T) (n\u0026thinsp;=\u0026thinsp;7); Environmental Research Letters (ERL) (n\u0026thinsp;=\u0026thinsp;5); Frontiers in Sustainability (FIS) (n\u0026thinsp;=\u0026thinsp;3). These are journals whose aims and scopes explicitly mention IE or related methods (e.g. LCA, MFA) and whose editorial boards contain a large number of IE researchers. Starting with journal issues published in December 2025, we worked backwards through each issue, manually identifying studies whose titles or abstracts focus on the use of \u0026lsquo;machine learning\u0026rsquo;, \u0026lsquo;artificial intelligence\u0026rsquo;, or related methods. We then refined this initial pool of studies. We selected only the subset of studies whose hypothesis and methods fit the described selection criteria. We also filtered for studies which focus on a problem related to industrial ecology based on keywords and author judgement. The final studies cover a range of IE topics, model types, and dataset sizes (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eWe note that our survey is not a comprehensive review of all ML studies in IE research. Our goal is not to perform a full review of the literature (i.e. collect all important citations into one-place, study the evolution of ML use in IE, determine a field-wide consensus). Instead, our aim is to \u0026lsquo;audit\u0026rsquo; the current practices of IE researchers using ML.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Evaluating studies for reproducibility\u003c/h2\u003e \u003cp\u003eWe compare each study in our survey to an ontology of ML best practices for reproducibility (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). We adapt the ontology introduced in Kapoor \u0026amp; Naranyanan (2023), who identify eight types of errors (grouped into three categories dubbed \u003cem\u003eL1\u003c/em\u003e, \u003cem\u003eL2\u003c/em\u003e, and \u003cem\u003eL3\u003c/em\u003e) researchers make when creating ML models that lead to leakage and irreproducibility. A full description of the ontology, and examples of errors, are included in Supplementary File 1 (SI1). We note that Kapoor \u0026amp; Naranyanan identify errors that occur in three stages when building an ML model: A) the initial data processing stage, B) the feature selection and training stage, and C) the interpretation and dissemination stage. We make the following modifications to tailor Kapoor \u0026amp; Naranyanan\u0026rsquo;s ontology for our survey:\u003c/p\u003e \u003cp\u003e● To the error category \u003cem\u003eL3\u003c/em\u003e: \u003cem\u003eTest set is not drawn from the distribution of scientific interest\u003c/em\u003e, we add an additional error \u003cem\u003eL3.4, geographic leakage\u003c/em\u003e. This error occurs when the test set gains information about the training set due to some geospatial correlation, or otherwise does not reflect the \u0026ldquo;distribution of interest\u0026rdquo; for the study\u0026rsquo;s hypothesis. Geospatial analysis is common in IE research, which could lead to an outsized appearance of these errors compared to other scientific fields.\u003c/p\u003e \u003cp\u003e● We elevate Kapoor \u0026amp; Naranyanan\u0026rsquo;s check for \u0026ldquo;computational reproducibility issues\u0026rdquo; into its own error category, dubbed \u003cem\u003eL4\u003c/em\u003e. We define five distinct errors in this category which lead to irreproducibility: \u003cem\u003eL4.1\u003c/em\u003e: \u003cem\u003ecomputing resources (hardware) not given\u003c/em\u003e, \u003cem\u003eL4.2 hyperparameters not given\u003c/em\u003e, \u003cem\u003eL4.3 data not open\u003c/em\u003e, \u003cem\u003eL4.4 model not open\u003c/em\u003e, and \u003cem\u003eL4.5 improper metric choice\u003c/em\u003e. Our analysis benefits from greater detail to clarify whether authors provided all, some, or none of the information required to computationally reproduce their ML models.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Coding studies\u003c/h2\u003e \u003cp\u003eUsing the ontology, five authors coded the studies as follows. After joint conversations and practice review, the lead author created a shared reference document for the ontology based on Kapoor \u0026amp; Naranyanan\u0026rsquo;s original paper (included in SI1). We then evaluated a subset of the surveyed studies. We compared each study to the reference document, providing a score of 0 (follows best practices), 0.5 (uncertain/unclear), or 1 (does not follow best practices) for each error in the ontology, based on the description in the text (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). For example, we reviewed a study and found that it does not describe the creation or use of a hold-out test set in the main text or supplementary information. For category \u003cem\u003eL1.1 no test set\u003c/em\u003e, we gave the study a score of 1. We took steps to ensure reliable coding between researchers. Before beginning independent coding, each researcher coded the same 1\u0026ndash;5 studies alone for inter-reliability training, and then met as a group to discuss and resolve discrepancies. For subsequent studies, we had at least two researchers check scores and the notes of the original coder to ensure agreement with the reference document. At the conclusion of coding, we met as a group to reach a consensus about final coding uncertainties before analyzing the data. The criteria for assigning scores to each error is outlined in SI1. We include intercoder reliability scores for two papers, coded independently post-training, in Supplementary Information SI2.\u003c/p\u003e \u003cp\u003eHere we clarify some important study design choices. First, our coding does not fully confirm and correct irreproducible results in the surveyed papers. Rather, we have identified failures in best practices which \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003ecould and would likely\u003c/span\u003e lead to a lack of reproducibility, based on our interpretation of the text, figures and supporting information. In the case of category \u003cem\u003eL3\u003c/em\u003e errors (\u003cem\u003etest set is not drawn from distribution of scientific interest\u003c/em\u003e) specifically, confirming irreproducibility requires fully re-creating and re-training the models presented in each paper, which would be time-consuming and difficult given many papers with \u003cem\u003eL3\u003c/em\u003e errors also have data and model transparency issues. Additionally, we note that some errors lead more obviously to irreproducibility than others (see Discussion section \u003cspan refid=\"Sec11\" class=\"InternalRef\"\u003e4.1\u003c/span\u003e), and some errors are difficult to identify with our coding method. For example, error \u003cem\u003eL1.4\u003c/em\u003e: \u003cem\u003eduplicates across train and test set\u003c/em\u003e is difficult to confirm unless the authors explicitly describe a pre-processing or data collection technique that would duplicate values before creating the hold-out set (and would be impossible to validate if the dataset is not given).\u003c/p\u003e \u003cp\u003eFollowing this, we have anonymized the studies included in our survey. Our goal is not to single out specific authors or papers for their failure to follow best practices. Instead, we aim to identify common and persistent errors in the use of ML for IE research in order to caution the community and set a path to future widespread reproducibility. We have anonymized all of the data and results for this paper and released them in a Zenodo repository (See Data Availability).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4 Bibliometric analysis\u003c/h2\u003e \u003cp\u003eWe use an open source citation database to see where and how often the studies in our survey have been cited. We performed this analysis using OpenAlex (OpenAlex, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2026\u003c/span\u003e). OpenAlex is a continuously updated, non-profit research database that graphs the connections between papers, authors, and institutions. We queried the OpenAlex graph to find the papers that have cited our surveyed studies (forward citations) between the beginning of 2022 and February of 2026. We extracted the following information about studies in our survey and the works that cite them:\u003c/p\u003e \u003cp\u003e● Publication date\u003c/p\u003e \u003cp\u003e● Article type\u003c/p\u003e \u003cp\u003e● Publication venue (i.e. journal name)\u003c/p\u003e \u003cp\u003e● Field-weighted citation percentile.\u003c/p\u003e \u003cp\u003eWe then linked this bibliometric information to our best-practice coding. We analyze the citation trends of studies based on their score in each ontology error category. Full info on how we extracted our results from the OpenAlex API can be found in the Zenodo repository.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. RESULTS","content":"\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Coding of studies\u003c/h2\u003e \u003cp\u003eAcross the 50 coded studies, we identify 115 unique descriptions of errors that could lead to irreproducibility (score of 1.0); 23 of these errors occur in the initial data processing stage, 2 occur during feature selection and training, and 90 occur when results are being presented and disseminated. Additionally, we find 103 instances where we are uncertain, based on the information given in the text, if an error occurred which could lead to irreproducibility (score of 0.5) (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Of the 50 studies, we identify 7 which have no clear errors in any category (score of 0.0 or 0.5 across the ontology), and 2 which we are able to fully confirm follow all best practices (score of 0.0 across the ontology).\u003c/p\u003e \u003cp\u003eTransparency and computational reproducibility errors are far more common across the studies than clear instances of data leakage. Out of the 50 studies, 14 described a reproducibility error that would cause data leakage (L1, L2, or L3 errors). Conversely, 42 studies have a computational reproducibility issue (L4 errors). Common issues include partial disclosure of model hyper-parameters (e.g., ranges instead of final optimal values), no disclosure of the hardware that models are trained on, links to data and model repositories (e.g., github) that are broken/inoperable, and a lack of data disclosure without a stated reason. Some studies use confidential data that cannot be released; while these studies received a score of 0.5 or 1 for L4.3, this is not an indictment of the quality of these studies. Our results suggest that, in general, ML reproducibility issues in IE may be more a result of publication standards and communication rather than methodological failures by authors. However, this lack of disclosure is still an important issue. Broad computational reproducibility issues make it harder to assess whether subtle leakage mistakes have been made, especially for L3 errors. This may result in an under-estimation of data leakage across IE studies.\u003c/p\u003e \u003cp\u003eDespite being less common, we identify a concerning pattern of reproducibility errors which could cause data leakage. L1.2 errors (\u003cem\u003epre-processing before split\u003c/em\u003e) are the most common leakage issue. 12 studies explicitly describe data pre-processing techniques (most notably normalization and gap-filling across the entire dataset) that would give information about the test set to the training set, before splitting their data. For an additional 12 studies, we are uncertain whether data is being properly pre-processed due to a lack of clarity in the methods. Though less common, three studies make L1.1 errors: they evaluate their model\u0026rsquo;s performance on the data it is trained on, instead of held-out data. This error is critical. It leaves the predictive hypothesis unaddressed and is contrary to widespread practices in supervised ML. Conversely, most studies deal with spatial auto-correlation well. We find no instances of confirmable L3.4 errors. Two studies present ML models which aim to predict future time-series values, but introduce temporal leakage by training the models with randomly split data. This means the model is trained using data \u0026ldquo;from the future\u0026rdquo; (Kapoor \u0026amp; Narayanan, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), invalidating the hypothesis that it can predict future values. All of these errors are the same as those that have been identified repeatedly as causing reproducibility issues in other scientific fields; in other words, IE is not uniquely immune to the \u0026lsquo;reproducibility crisis\u0026rsquo; of applied ML research.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Citations of studies\u003c/h2\u003e \u003cp\u003eThe studies that make reproducibility errors have been cited hundreds of times in the past 4\u0026ndash;5 years (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). These citations are spread across 193 unique journals/publication venues (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003ed). While a majority of these citations are of studies with computational reproducibility issues (400 instances), around 100 citations are of studies which describe leakage-related errors in their methodology.\u003c/p\u003e \u003cp\u003eTo get a sense of how material from potentially irreproducible studies is being used, we read the articles published in the journal \u003cem\u003eScientific Data\u003c/em\u003e that cite studies in our survey. \u003cem\u003eScientific Data\u003c/em\u003e is of particular concern because it publishes descriptions of research datasets. If outputs of studies with data leakage are being cited and used in these datasets, this could directly propagate false results forward. Five unique articles in \u003cem\u003eScientific Data\u003c/em\u003e cite studies that make leakage-related errors (specifically L1.1, L1.2, and L1.3 errors). Luckily, none of these studies use data from the outputs of these models. However, they do use the studies as supporting evidence of their predictive hypothesis; i.e. that features \u003cem\u003eX\u003c/em\u003e can predict a target variable \u003cem\u003ey\u003c/em\u003e. This raises concerns about the propagation of erroneous conclusions forward, as discussed in the introduction. The overall evidence of supporting citations in these \u003cem\u003eScientific Data\u003c/em\u003e articles is being undermined by reproducibility errors in the original studies. We also found an article in \u003cem\u003eScientific Data\u003c/em\u003e which cited a surveyed study to highlight that it had computational reproducibility issues, and that these issues were a motivation for their own work. This raises another consequence of reproducibility errors: a lack of open data and models is slowing down science. These errors force researchers to invest time and funding into recreating results which could have been openly available in the first place.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eWhen considering the overall citation graph of the studies (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003ee), it is not clear that studies or citations cluster based on how well they adhere to best practices. Several citations draw evidence from both best-practice and error-filled articles. This raises the question: are studies which make ML reproducibility errors cited less often than those which follow best practices? We compare the field-weighted citation percentile (FWCP) of studies to see (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003ea). We find little evidence that making reproducibility errors results in less citations. Where differences in citation percentile exist between the citations of studies making errors vs. following best practices, we only have a few samples (e.g. two data points for \u003cem\u003eL3.1\u003c/em\u003e). When comparing the mean FWCP across all categories, there is no statistical evidence that studies following best practices (open data/models, no data leakage) are cited more often (Welch's unequal variance t-test p-value\u0026thinsp;=\u0026thinsp;0.13\u0026ndash;0.78). In other words, researchers are not rewarded for making their studies reproducible. We expand our analysis to compare citations of all the ML studies we surveyed to the citations of the average article in their respective journals (Fig.\u0026nbsp;5\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003ebc\u003c/span\u003e). Beyond 1\u0026ndash;2 years since publication, IE studies that use supervised ML may accrue more citations than their peer non-ML articles in some journals. In the \u003cem\u003eJournal of Industrial Ecology\u003c/em\u003e, the average ML study in our survey receives, on average, 17.5 citations in just over three years, compared to 12.7 citations across all articles in the journal. Surveyed studies exceed the mean expected citations by around three-quarters of a standard deviation. Recent research corroborates the finding that studies using ML tend to be cited more often (Hao et al., \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2026\u003c/span\u003e). This means that as ML-based research becomes more common, and if steps are not taken to limit reproducibility errors in this research, an outsized amount of the IE literature could become irreproducible.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"4. DISCUSSION AND RECOMMENDATIONS","content":"\u003cp\u003eThe results of our survey justify a call to action for ensuring reproducibility in IE research using ML. While other types of errors not addressed in this paper can affect the reproducibility of ML and non-ML research equally (e.g., p-hacking, unintentional errors of experimental procedures or code), we argue that issues outlined here - data leakage and computational reproducibility - are uniquely important to address. These reproducibility issues will increase in prevalence as ML is used and cited more frequently in IE research. This is likely to be accelerated by access to generative AI tools for coding, which will lower the barrier to entry and therefore minimum domain knowledge required to use ML for applied science. Increased use of large-language models (LLM) in predictive workflows could also make it more difficult to report the computational details of ML studies (due to the off-the-shelf, third-party nature of these models and their training details) and increase the chance that test data has been seen by the LLM during prior training, which would induce data leakage.\u003c/p\u003e \u003cp\u003eScientists have been studying reproducibility for decades (Baker, \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2016\u003c/span\u003e; McNutt, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2014\u003c/span\u003e), and ML researchers specifically have a significant corpus focused on reducing data-leakage and improving ML reproducibility (Kapoor \u0026amp; Narayanan, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Kaufman et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2012\u003c/span\u003e; Lones, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Lucasius et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). We draw on these sources to make recommendations for improving reproducibility of ML in IE in the following sections.\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Recommendations for researchers using machine learning\u003c/h2\u003e \u003cp\u003eThe points summarize suggestions from the literature and can act as a starting point for industrial ecologists interested in working with machine learning.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGround your work in best practices from the outset\u003c/b\u003e. There are many high-quality guidelines for structuring a machine learning study to avoid leakage (Kaufman et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2012\u003c/span\u003e; Lones, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Mitchell et al., \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) and to maximize data and model transparency (Chen et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Nosek et al., \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Nosek \u0026amp; Lakens, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2014\u003c/span\u003e). We recommend starting with Kapoor \u0026amp; Narayanan \u0026lsquo;model info sheets\u0026rsquo;, which can be used to self-evaluate a study against the reproducibility ontology in our survey (2023). The first step by any IE researcher who plans to use ML should be to create a reproducible workflow based on these resources. Even so, complex workflows might introduce subtle sources of leakage that are hard for non-experts to recognize. In many cases, IE researchers may also benefit from consulting directly with domain experts in ML, having these experts suggest or build a model while the IE researcher weighs in on assumptions, constraints, and design evaluation. Taking a reproducibility-first approach is particularly important for the kinds of data-leakage errors that were most frequent in our survey; namely, not cleanly separating and isolating a test set at the beginning of the model-building process.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDescribe best practices in writing\u003c/b\u003e. Many studies in the survey had vague method descriptions, which made it difficult to determine whether they made reproducibility errors. In contrast, the few studies following best practices all had plain-language descriptions of the steps they took to avoid reproducibility errors (e.g. \u0026ldquo;we split the data, and then performed pre-processing\u0026rdquo;, \u0026ldquo;we used time-series cross validation to avoid temporal leakage\u0026rdquo;). We recommend that authors plainly describe, as much as is reasonably possible, their reproducible workflows in writing. This will allow readers to quickly confirm the quality of a study and promote reproducible work.\u003c/p\u003e \u003cp\u003e \u003cb\u003eBe clear why data or models must be withheld\u003c/b\u003e. Unexplained withholding of data and broken repository links were common in our survey. Data and model transparency should be the default assumption for ML studies. If this is not possible for reasons such as the use of proprietary data, researchers should strive to be transparent about why data or computational details are withheld and where similar data could reasonably be obtained. Maximizing the availability of models and data helps avoid the need for repeated work in the future, and also makes evaluating a study for data leakage easier.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDon\u0026rsquo;t overfit the problem\u003c/b\u003e. In our survey, we were often unable to confirm best practices for studies which used complex, multi-model workflows. It is not always clear what value complex workflows add to a predictive hypothesis, especially when datasets are small (the mean dataset size was \u0026lt;\u0026thinsp;1000 for studies in our survey). We recommend reporting results for the simplest model, usually a linear regression, before moving on to a more complex model. These models have simpler diagnostics and are less likely to have unnoticed reproducibility errors. Additionally, some of the best-practice studies we surveyed found that linear models performed nearly as well as more complex models for their use case. These linear models can serve as a baseline for benchmarking. We also recommend that researchers consider whether ML is necessary to test their hypothesis in the first place. ML reproducibility errors can be avoided if empirical data are collected rather than predicted.\u003c/p\u003e \u003cp\u003e \u003cb\u003eHunt for assumptions\u003c/b\u003e. Scientific research should consider the null hypothesis \u0026mdash; that \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:f\\)\u003c/span\u003e\u003c/span\u003e does not map \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:X\\to\\:y\\)\u003c/span\u003e\u003c/span\u003e \u0026mdash; an acceptable outcome (Greenwald, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e1975\u003c/span\u003e). Measuring the success of an ML study solely by test metrics like R\u003csup\u003e2\u003c/sup\u003e makes it easy to overlook reproducibility errors. Because of this, we encourage IE researchers to adopt an \u0026ldquo;assumption-hunting\u0026rdquo; attitude (Saltelli \u0026amp; Funtowicz, \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2014\u003c/span\u003e) when engaging in and with ML research. Similar to how industrial ecologists perform sensitivity analysis on their LCA or MFA, researchers must look for data and process assumptions that condition the results of an ML model. When scrutinizing their work, researchers must ask themselves: \u003cem\u003eam I using data which reflects the process I am trying to model? Are there hidden assumptions in my process which could be giving me a misleading result? Have I been transparent enough for my model to be used by other researchers?\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Recommendations for publishers and reviewers of machine learning research\u003c/h2\u003e \u003cp\u003eWe briefly provide recommendations for how those publishing or reviewing IE papers using ML can help foster reproducible research.\u003c/p\u003e \u003cp\u003e \u003cb\u003eFor journals publishing research.\u003c/b\u003e Journals could make the reproducibility check-lists discussed above mandatory for ML papers under review (Kapoor \u0026amp; Narayanan, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2026\u003c/span\u003e; Mitchell et al., \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). These could be required as self-reported supporting information, similar to existing standards like data badges (Hertwich et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). Going a step further, editorial offices or peer reviewers could be asked to conduct similar checks explicitly as part of the publication process - ideally by a reviewer with machine-learning expertise. Data confidentiality will likely continue to limit the ability of papers to meet \u003cem\u003eL4\u003c/em\u003e best practices. We do not suggest that this should prevent the publishing of papers, but the reasons for withholding the data or models in an ML study should always be justified. Journals can also encourage authors to host models and data on persistent services (e.g., hosting on Google Colab) to encourage computational transparency and quick identification and correction of reproducibility errors.\u003c/p\u003e \u003cp\u003e \u003cb\u003eFor reviewers\u003c/b\u003e. Some reproducibility errors are more disqualifying than others, and reviewers should keep the overall quality of a paper in mind when evaluating a study. Non-expert reviewers assigned to papers applying ML should look for easy-to-spot reproducibility errors that are totally disqualifying - specifically, a lack of a hold-out test set for evaluating models, temporal leakage, or normalization and gap-filling before splitting data into training and test sets. These errors do not require ML expertise to identify, and they put any hypothetical results into serious dispute. Confidential datasets or models may not be disqualifying if the rest of the study follows best practice. If reviewers are unsure if reproducibility errors are present (i.e. would code an error as 0.5 based on the ontology), it is likely that future readers will probably be as well. In these instances, we recommend reviewers ask for a revision of the article to clarify reproducibility of study.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. CONCLUSION","content":"\u003cp\u003eThe power of machine learning (ML) to analyze old questions faster and generate new questions is energizing industrial ecology (IE) researchers. However, IE's rapid adoption of ML has outpaced its commitment to scientific reproducibility. This paper has identified the common challenges of reproducibility within IE papers using ML. These challenges are highlighted by the rapid growth of ML usage by industrial ecologists, and the continued citation of works which make mistakes that can lead to irreproducibility. Crises in other disciplines show that these mistakes can muddy the evidence base of literature and slow down the progress towards answering important scientific questions. The IE community stands to benefit from acting now to improve the reproducibility of its ML applications.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eCompeting Interests\u003c/strong\u003e\u003cp\u003eAuthor FD serves as associate editor for the Journal of Industrial Ecology. Our survey includes papers co-authored by S Saxe and QT. These authors were not involved in coding their own papers, nor were they shown the final results for their own papers.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003e**Conceptualization** : KHR, FD, QT, JWB, SvL, S Suh, S Saxe, IDP, JKH; **Methodology** : KHR, FD, QT, SvL, S Suh, S Saxe, IDP, JKH; **Data collection** : KHR, JWB, NR, BH, JKH; **Visualization** : KHR; **Validation** : KHR, JWB, NR, BH, JKH; **Writing (original draft, review and editing)** : KHR, FD, QT, JWB, SvL, S Suh, S Saxe, IDP, JKH; **Funding acquisition** : KHR, QT, JKH\u003c/p\u003e\u003ch2\u003eAcknowledgement\u003c/h2\u003e\u003cp\u003eThis research was undertaken, in part, thanks to funding from the Natural Sciences and Engineering Research Council of Canada (NSERC) Vanier scholarship held by KHR, NSERC funding [reference number RGPIN-2021-02841] held by QT, and University of Wyoming School of Computing funding held by JKH.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eAll data and code that support the findings of this study are available via Zenodo at [https://doi.org/10.5281/zenodo.19153101](https:/doi.org/10.5281/zenodo.19153101) . We have anonymized the identities of all surveyed papers and authors in the repository.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBaker, M. (2016). 1,500 scientists lift the lid on reproducibility. \u003cem\u003eNature\u003c/em\u003e, \u003cem\u003e533\u003c/em\u003e(7604), 452\u0026ndash;454. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/533452a\u003c/span\u003e\u003cspan address=\"10.1038/533452a\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBalaji, B., Vunnava, G., Domingo, N., Gupta, S., Gupta, H., Guest, G., \u0026amp; Srinivasan, A. (2023). \u003cem\u003eFlamingo: Environmental impact factor matching for life cycle assessment with zero-shot machine learning\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.amazon.science/publications/flamingo-environmental-impact-factor-matching-for-life-cycle-assessment-with-zero-shot-machine-learning\u003c/span\u003e\u003cspan address=\"https://www.amazon.science/publications/flamingo-environmental-impact-factor-matching-for-life-cycle-assessment-with-zero-shot-machine-learning\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBegley, C. G., \u0026amp; Ellis, L. M. (2012). Raise standards for preclinical cancer research. \u003cem\u003eNature\u003c/em\u003e, \u003cem\u003e483\u003c/em\u003e(7391), 531\u0026ndash;533. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/483531a\u003c/span\u003e\u003cspan address=\"10.1038/483531a\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBogdan, P. C. (2025). One Decade Into the Replication Crisis, How Have Psychological Results Changed? \u003cem\u003eAdvances in Methods and Practices in Psychological Science\u003c/em\u003e, \u003cem\u003e8\u003c/em\u003e(2), 25152459251323480. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/25152459251323480\u003c/span\u003e\u003cspan address=\"10.1177/25152459251323480\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBozeman, J. F., Hollauer, C., Ramshankar, A. T., Nakkasunchi, S., Jambeck, J., Hicks, A., Bilec, M., McCauley, D., \u0026amp; Heidrich, O. (2024). Embed systemic equity throughout industrial ecology applications: How to address machine learning unfairness and bias. \u003cem\u003eJournal of Industrial Ecology\u003c/em\u003e, \u003cem\u003e28\u003c/em\u003e(6), 1362\u0026ndash;1376. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/jiec.13509\u003c/span\u003e\u003cspan address=\"10.1111/jiec.13509\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCamerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., \u0026amp; Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. \u003cem\u003eScience\u003c/em\u003e, \u003cem\u003e351\u003c/em\u003e(6280), 1433\u0026ndash;1436. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1126/science.aaf0918\u003c/span\u003e\u003cspan address=\"10.1126/science.aaf0918\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen, X., Dallmeier-Tiessen, S., Dasler, R., Feger, S., Fokianos, P., Gonzalez, J. B., Hirvonsalo, H., Kousidis, D., Lavasa, A., Mele, S., Rodriguez, D. R., Šimko, T., Smith, T., Trisovic, A., Trzcinska, A., Tsanaktsidis, I., Zimmermann, M., Cranmer, K., Heinrich, L., \u0026hellip; Neubert, S. (2019). Open is not enough. \u003cem\u003eNature Physics\u003c/em\u003e, \u003cem\u003e15\u003c/em\u003e(2), 113\u0026ndash;119. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41567-018-0342-2\u003c/span\u003e\u003cspan address=\"10.1038/s41567-018-0342-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCullen, L., Marinoni, A., \u0026amp; Cullen, J. (2024). Machine learning for gap-filling in greenhouse gas emissions databases. \u003cem\u003eJournal of Industrial Ecology\u003c/em\u003e, \u003cem\u003e28\u003c/em\u003e(4), 636\u0026ndash;647. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/jiec.13507\u003c/span\u003e\u003cspan address=\"10.1111/jiec.13507\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDonati, F., Dente, S. M. R., Li, C., Vilaysouk, X., Froemelt, A., Nishant, R., Liu, G., Tukker, A., \u0026amp; Hashimoto, S. (2022). The future of artificial intelligence in the context of industrial ecology. \u003cem\u003eJournal of Industrial Ecology\u003c/em\u003e, \u003cem\u003e26\u003c/em\u003e(4), 1175\u0026ndash;1181. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/jiec.13313\u003c/span\u003e\u003cspan address=\"10.1111/jiec.13313\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGohr, C., Rodr\u0026iacute;guez, G., Belomestnykh, S., Berg-Moelleken, D., Chauhan, N., Engler, J.-O., Heydebreck, L. V., Hintz, M. J., Kretschmer, M., Kr\u0026uuml;germeier, C., Meinberg, J., Rau, A.-L., Schwenck, C., Aoulkadi, I., Poll, S., Frank, E., Creutzig, F., Lemke, O., Maushart, M., \u0026hellip; Von Wehrden, H. (2025). Artificial intelligence in sustainable development research. \u003cem\u003eNature Sustainability\u003c/em\u003e, \u003cem\u003e8\u003c/em\u003e(8), 970\u0026ndash;978. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41893-025-01598-6\u003c/span\u003e\u003cspan address=\"10.1038/s41893-025-01598-6\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGong, Y., Ma, F., Wang, H., Tzachor, A., Sun, W., Zhu, J., Liu, G., \u0026amp; Schandl, H. (2025). The evolution of research at the intersection of industrial ecology and artificial intelligence. \u003cem\u003eJournal of Industrial Ecology\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e(2), 440\u0026ndash;457. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/jiec.13612\u003c/span\u003e\u003cspan address=\"10.1111/jiec.13612\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGreenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. \u003cem\u003ePsychological Bulletin\u003c/em\u003e, \u003cem\u003e82\u003c/em\u003e(1), 1\u0026ndash;20. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1037/h0076157\u003c/span\u003e\u003cspan address=\"10.1037/h0076157\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHao, Q., Xu, F., Li, Y., \u0026amp; Evans, J. (2026). Artificial intelligence tools expand scientists\u0026rsquo; impact but contract science\u0026rsquo;s focus. \u003cem\u003eNature\u003c/em\u003e, \u003cem\u003e649\u003c/em\u003e(8099), 1237\u0026ndash;1243. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41586-025-09922-y\u003c/span\u003e\u003cspan address=\"10.1038/s41586-025-09922-y\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHertwich, E., Heeren, N., Kuczenski, B., Majeau-Bettez, G., Myers, R. J., Pauliuk, S., Stadler, K., \u0026amp; Lifset, R. (2018). Nullius in Verba1: Advancing Data Transparency in Industrial Ecology. \u003cem\u003eJournal of Industrial Ecology\u003c/em\u003e, \u003cem\u003e22\u003c/em\u003e(1), 6\u0026ndash;17. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/jiec.12738\u003c/span\u003e\u003cspan address=\"10.1111/jiec.12738\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHornsey, M. J., Harris, E. A., Bain, P. G., \u0026amp; Fielding, K. S. (2016). Meta-analyses of the determinants and outcomes of belief in climate change. \u003cem\u003eNature Climate Change\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e(6), 622\u0026ndash;626. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/nclimate2943\u003c/span\u003e\u003cspan address=\"10.1038/nclimate2943\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang, W., Olson, A. W., Khalil, E. B., \u0026amp; Saxe, S. (2025). Image-based prediction of residential building attributes with deep learning. \u003cem\u003eJournal of Industrial Ecology\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e(1), 81\u0026ndash;95. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/jiec.13591\u003c/span\u003e\u003cspan address=\"10.1111/jiec.13591\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eISO. (2023). \u003cem\u003eISO 5725-1\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.iso.org/obp/ui/#iso:std:iso:5725\u003c/span\u003e\u003cspan address=\"https://www.iso.org/obp/ui/#iso:std:iso:5725\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e:-1:ed-2:v1:en\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKapoor, S., \u0026amp; Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. \u003cem\u003ePatterns\u003c/em\u003e, \u003cem\u003e4\u003c/em\u003e(9), 100804. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.patter.2023.100804\u003c/span\u003e\u003cspan address=\"10.1016/j.patter.2023.100804\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKapoor, S., \u0026amp; Narayanan, A. (2026). \u003cem\u003eLeakage and the Reproducibility Crisis in ML-based Science\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://reproducible.cs.princeton.edu/#citation\u003c/span\u003e\u003cspan address=\"https://reproducible.cs.princeton.edu/#citation\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKaufman, S., Rosset, S., Perlich, C., \u0026amp; Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. \u003cem\u003eACM Transactions on Knowledge Discovery from Data\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e(4), 1\u0026ndash;21. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/2382577.2382579\u003c/span\u003e\u003cspan address=\"10.1145/2382577.2382579\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKristal, A. S., Whillans, A. V., Bazerman, M. H., Gino, F., Shu, L. L., Mazar, N., \u0026amp; Ariely, D. (2020). Signing at the beginning versus at the end does not decrease dishonesty. \u003cem\u003eProceedings of the National Academy of Sciences\u003c/em\u003e, \u003cem\u003e117\u003c/em\u003e(13), 7103\u0026ndash;7107. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1073/pnas.1911695117\u003c/span\u003e\u003cspan address=\"10.1073/pnas.1911695117\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeCun, Y., Bengio, Y., \u0026amp; Hinton, G. (2015). Deep learning. \u003cem\u003eNature\u003c/em\u003e, \u003cem\u003e521\u003c/em\u003e(7553), 436\u0026ndash;444. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/nature14539\u003c/span\u003e\u003cspan address=\"10.1038/nature14539\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLones, M. A. (2021). \u003cem\u003eHow to avoid machine learning pitfalls: A guide for academic researchers\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/ARXIV.2108.02497\u003c/span\u003e\u003cspan address=\"10.48550/ARXIV.2108.02497\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLones, M. A. (2024). Avoiding common machine learning pitfalls. \u003cem\u003ePatterns\u003c/em\u003e, \u003cem\u003e5\u003c/em\u003e(10), 101046. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.patter.2024.101046\u003c/span\u003e\u003cspan address=\"10.1016/j.patter.2024.101046\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLucasius, C., Ali, M., Patel, T., Kundur, D., Szatmari, P., Strauss, J., \u0026amp; Battaglia, M. (2025). A procedural overview of why, when and how to use machine learning for psychiatry. \u003cem\u003eNature Mental Health\u003c/em\u003e, \u003cem\u003e3\u003c/em\u003e(1), 8\u0026ndash;18. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s44220-024-00367-2\u003c/span\u003e\u003cspan address=\"10.1038/s44220-024-00367-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcNutt, M. (2014). Reproducibility. \u003cem\u003eScience\u003c/em\u003e, \u003cem\u003e343\u003c/em\u003e(6168), 229\u0026ndash;229. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1126/science.1250475\u003c/span\u003e\u003cspan address=\"10.1126/science.1250475\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., \u0026amp; Gebru, T. (2018). \u003cem\u003eModel Cards for Model Reporting\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/ARXIV.1810.03993\u003c/span\u003e\u003cspan address=\"10.48550/ARXIV.1810.03993\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNational Academies. (2019). \u003cem\u003eReproducibility and Replicability in Science\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.nationalacademies.org/projects/DBASSE-BBCSS-17-03\u003c/span\u003e\u003cspan address=\"https://www.nationalacademies.org/projects/DBASSE-BBCSS-17-03\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., \u0026hellip; Yarkoni, T. (2015). Promoting an open research culture. \u003cem\u003eScience\u003c/em\u003e, \u003cem\u003e348\u003c/em\u003e(6242), 1422\u0026ndash;1425. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1126/science.aab2374\u003c/span\u003e\u003cspan address=\"10.1126/science.aab2374\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNosek, B. A., \u0026amp; Lakens, D. (2014). Registered Reports: A Method to Increase the Credibility of Published Results. \u003cem\u003eSocial Psychology\u003c/em\u003e, \u003cem\u003e45\u003c/em\u003e(3), 137\u0026ndash;141. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1027/1864-9335/a000192\u003c/span\u003e\u003cspan address=\"10.1027/1864-9335/a000192\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOpen Science Collaboration. (2015). Estimating the reproducibility of psychological science. \u003cem\u003eScience\u003c/em\u003e, \u003cem\u003e349\u003c/em\u003e(6251), aac4716. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1126/science.aac4716\u003c/span\u003e\u003cspan address=\"10.1126/science.aac4716\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOpenAlex. (2026). \u003cem\u003eOpenAlex\u003c/em\u003e [Dataset]. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://openalex.org/\u003c/span\u003e\u003cspan address=\"https://openalex.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePlatt, J. R. (1964). Strong Inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others. \u003cem\u003eScience\u003c/em\u003e, \u003cem\u003e146\u003c/em\u003e(3642), 347\u0026ndash;353. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1126/science.146.3642.347\u003c/span\u003e\u003cspan address=\"10.1126/science.146.3642.347\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePopper, K. (1959). \u003cem\u003eThe logic of scientific discovery\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://psycnet.apa.org/record/1961-02882-000\u003c/span\u003e\u003cspan address=\"https://psycnet.apa.org/record/1961-02882-000\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K., Lacoste, A., Sankaran, K., Ross, A. S., Milojevic-Dupont, N., Jaques, N., Waldman-Brown, A., Luccioni, A. S., Maharaj, T., Sherwin, E. D., Mukkavilli, S. K., Kording, K. P., Gomes, C. P., Ng, A. Y., Hassabis, D., Platt, J. C., \u0026hellip; Bengio, Y. (2023). Tackling Climate Change with Machine Learning. \u003cem\u003eACM Computing Surveys\u003c/em\u003e, \u003cem\u003e55\u003c/em\u003e(2), 1\u0026ndash;96. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3485128\u003c/span\u003e\u003cspan address=\"10.1145/3485128\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSaltelli, A., \u0026amp; Funtowicz, S. (2014). When All Models are Wrong. \u003cem\u003eIssues in Science and Technology\u003c/em\u003e, \u003cem\u003e30\u003c/em\u003e(2). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://issues.org/andrea/\u003c/span\u003e\u003cspan address=\"https://issues.org/andrea/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSarswat, P. K., Singh, R. S., \u0026amp; Pathapati, S. V. S. H. (2024). Real time electronic-waste classification algorithms using the computer vision based on Convolutional Neural Network (CNN): Enhanced environmental incentives. \u003cem\u003eResources, Conservation and Recycling\u003c/em\u003e, \u003cem\u003e207\u003c/em\u003e, 107651. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.resconrec.2024.107651\u003c/span\u003e\u003cspan address=\"10.1016/j.resconrec.2024.107651\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eStoetzer, L. S., \u0026amp; Zimmermann, F. (2024). A representative survey experiment of motivated climate change denial. \u003cem\u003eNature Climate Change\u003c/em\u003e, \u003cem\u003e14\u003c/em\u003e(2), 198\u0026ndash;204. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41558-023-01910-2\u003c/span\u003e\u003cspan address=\"10.1038/s41558-023-01910-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTu, Q., Guo, J., Li, N., Qi, J., \u0026amp; Xu, M. (2024). Mitigating Grand Challenges in Life Cycle Inventory Modeling through the Applications of Large Language Models. \u003cem\u003eEnvironmental Science \u0026amp; Technology\u003c/em\u003e, \u003cem\u003e58\u003c/em\u003e(44), 19595\u0026ndash;19603. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1021/acs.est.4c07634\u003c/span\u003e\u003cspan address=\"10.1021/acs.est.4c07634\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhao, B., Yu, Z., Wang, H., Shuai, C., Qu, S., \u0026amp; Xu, M. (2024). Data Science Applications in Circular Economy: Trends, Status, and Future. \u003cem\u003eEnvironmental Science \u0026amp; Technology\u003c/em\u003e, \u003cem\u003e58\u003c/em\u003e(15), 6457\u0026ndash;6474. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1021/acs.est.3c08331\u003c/span\u003e\u003cspan address=\"10.1021/acs.est.3c08331\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhu, J.-J., Yang, M., \u0026amp; Ren, Z. J. (2023). Machine Learning in Environmental Research: Common Pitfalls and Best Practices. \u003cem\u003eEnvironmental Science \u0026amp; Technology\u003c/em\u003e, \u003cem\u003e57\u003c/em\u003e(46), 17671\u0026ndash;17689. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1021/acs.est.3c00026 26\u003c/span\u003e\u003cspan address=\"10.1021/acs.est.3c00026 26\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"journal-of-industrial-ecology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"44498","submissionUrl":"https://submission.springernature.com/new-submission/44498/3","title":"Journal of Industrial Ecology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-9270723/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9270723/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eMachine learning (ML) usage in industrial ecology (IE) has grown nearly tenfold in the last decade. In other fields, similar increases in ML adoption have led to the widespread publication of results that cannot be reproduced. This uptick in irreproducibility, driven by a failure to follow best-practices when creating and reporting models - undermines the conclusions and credibility of science. Industrial ecologists have not yet determined whether reproducibility is becoming an issue of concern in their applications of ML. In order to assess this risk, we audited 50 recent IE studies against a ML reproducibility ontology. We find that 84% of surveyed studies suffer from computational reproducibility issues, and 28% exhibit methodological flaws that could introduce data leakage and invalidate findings. Yet, bibliometric analysis shows these potentially irreproducible studies are cited as or more frequently than their non-ML counterparts, which could embed flawed results into the scientific literature. Our findings serve as a call to action for the IE community. We suggest multi-level interventions, including that journals adopt reproducibility checklists and that reviewers prioritize key reproducibility errors over performance metrics, to safeguard the field and maximize the reproducibility of future ML-driven IE research.\u003c/p\u003e","manuscriptTitle":"A call to ensure reproducibility of machine learning applications in industrial ecology","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-13 18:46:43","doi":"10.21203/rs.3.rs-9270723/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-04-23T02:07:01+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"94907551613054911709151392455403093216","date":"2026-04-13T16:57:12+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"245714262327468444229028880750498799889","date":"2026-04-10T03:47:23+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"215031753350353967906371042019570263338","date":"2026-04-08T11:13:32+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-07T11:03:01+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-01T03:43:29+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-31T07:40:55+00:00","index":"","fulltext":""},{"type":"submitted","content":"Journal of Industrial Ecology","date":"2026-03-30T17:42:22+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"journal-of-industrial-ecology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"44498","submissionUrl":"https://submission.springernature.com/new-submission/44498/3","title":"Journal of Industrial Ecology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"9e1c7670-9d63-415a-b0cf-c7e13933ba46","owner":[],"postedDate":"April 13th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-04-13T18:46:43+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-13 18:46:43","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9270723","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9270723","identity":"rs-9270723","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}