Reusability report: A unified pre-trained deep learning framework for cross-task reaction performance prediction and synthesis planning

Reusability report: A unified pre-trained deep learning framework for cross-task reaction performance prediction and synthesis planning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Reusability report: A unified pre-trained deep learning framework for cross-task reaction performance prediction and synthesis planning Kuangbiao Liao, Chonghuan Zhang, Majian Zeng, Qianghua Lin This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8237411/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract Deep learning has substantially advanced reaction-yield prediction and synthesis-planning methodologies, yet achieving a unified architecture capable of transferring across these tasks remains a central challenge in chemical machine learning. RXNGraphormer introduces such a framework by combining a pretrained graph–transformer encoder with a delta-molecular reaction representation designed to support cross-task generalization. In this reusability report, we independently assess the reproducibility and practical applicability of RXNGraphormer using the released implementation, pretrained checkpoint and benchmark datasets. All major regression and sequence-generation results reported in the original study were consistently reproduced, including the relative difficulty patterns in out-of-sample evaluations, demonstrating the stability and transparency of the published workflow. To evaluate reusability, we examined the model’s transfer to multiple high-throughput datasets generated under standardized experimental conditions. In these settings, the pretrained encoder adapted efficiently and delivered strong predictive performance with minimal fine-tuning. When applied to a heterogeneous literature-derived benchmark, performance decreased, reflecting the inherent variability and structural noise characteristic of uncurated reaction corpora. Overall, our findings indicate that RXNGraphormer constitutes a reproducible and practically reusable chemical foundation model, capable of supporting both reaction-performance prediction and synthesis-planning tasks across diverse settings. These results further highlight the importance of harmonized reaction representations, curated experimental data and domain-specific refinement. Looking forward, continued progress in large-scale pretraining, interpretable reaction embeddings and standardized reaction corpora will be essential for extending the reach of unified chemical models to broader and more complex reaction spaces. Physical sciences/Chemistry/Cheminformatics Physical sciences/Mathematics and computing/Computational science Figures Figure 1 Figure 2 Figure 3 Introduction Machine-learning approaches have transformed modern chemical synthesis by enabling data-driven prediction of reaction outcomes, condition optimisation and route design¹ - ³. Graph-based and transformer-based models have demonstrated remarkable progress in capturing structure–reactivity relationships and predicting yields across diverse chemical spaces⁴ - ⁶. In parallel, algorithms for retrosynthetic and forward synthesis planning have evolved from template-based heuristics to policy-learning and generative frameworks that operate at scale⁷ - ⁹. Together, these developments mark a transition from task-specific systems toward integrated, data-centric paradigms for reaction modelling. Despite this progress, most frameworks remain specialised for either performance regression or synthesis planning, limiting transferability between tasks. Achieving a unified architecture capable of supporting both remains an open challenge. RXNGraphormer, recently introduced by Cao et al .¹⁰, addresses this by coupling a pre-trained graph–transformer backbone with a delta-molecular (ΔMol) representation to unify cross-task reaction learning. The model’s pretraining across millions of reactions aims to bridge reactivity prediction and synthesis sequence generation through a shared chemical representation. In this Reusability Report, we independently evaluate the reproducibility and reusability of RXNGraphormer. As shown in Figure 1, using the authors’ released code and datasets, we reproduce the reported results for reactivity, selectivity and synthesis tasks and extend the assessment to new datasets, including homogenous high-throughput experimentation (HTE) reaction benchmarks and a heterogeneous literature-derived benchmark. We verify the reported performance, test adaptability to new data sources, and assess the model’s potential as a reusable foundation for chemical reaction prediction and synthesis planning. Reproducibility 1. Regression benchmarks All four regression benchmarks across multiple aspects of reaction performance including reactivity, regioselectivity, and enantioselectivity were successfully reproduced. The full reproduction experiment results can be found in “Experiment Results” section of our GitHub repository and shown in Figure 2. On Buchwald–Hartwig and Suzuki–Miyaura coupling reactions, reproduced test-set R² values of 0.970 and 0.871 closely matched the reported 0.971 and 0.876. The radical C–H functionalization benchmark yielded a reproduced R² of 0.992 and MAE of 0.273 kcal mol⁻¹, consistent with the original 0.992 and 0.266 kcal mol⁻¹. For asymmetric thiol addition, reproduced R² reached 0.916, essentially identical to the reported 0.915. These experiments confirm that the ΔMol encoder and task-specific prediction heads maintain stable numerical behaviour across distinct reaction classes. 2. Out-of-sample evaluation The out-of-sample protocols for Buchwald–Hartwig coupling reproduced the same hierarchy. Bromide remained the most stable subset (R² of 0.869), whereas chloride again produced a negative value (R² of –0.377), consistent with its compressed yield distribution. Additive-based OOS splits reproduced the reported division between easy and difficult subsets (R² of 0.815–0.897 vs 0.651–0.643). The component-combination split reproduced the difficulty with R² of 0.732. For the asymmetric thiol-addition dataset, the reproduced substrate-held-out, catalyst-held-out and combined-held-out results preserved the published trend, yielding R² values of 0.915, 0.804 and 0.732, respectively. This confirms that catalyst-novelty continues to impose the strongest distributional shift to the latent reactivity space. 3. External regression datasets Three additional regression datasets were used to assess model transferability beyond the reaction domains examined in the RXNGraphormer publication. The pretrained model produced R² of 0.739 on asymmetric hydrogenation and R² of 0.900 on pallada-electrocatalyzed C–H activation. On the more structurally diverse NICOLit hydrogenation dataset, performance decreased to R² of 0.209, although class-level discrimination remained intact as reflected in accuracy and precision values. These trends mirror the distributional effects highlighted in the original study. 4. Sequence-generation tasks Retrosynthesis on USPTO-50k reproduced the reported top-k accuracies, with results of 50.3 %, 69.3 %, 73.7 % and 78.0 % for k = 1, 3, 5 and 10, which closely matched the published 51.0 %, 69.0 %, 73.0 % and 79.2 %. Re-evaluation of the USPTO-full model produced results nearly identical to the published values, as expected for an inference-only checkpoint. For forward prediction on USPTO-480k and USPTO-STEREO, reproduced top-1 accuracies differed by no more than 1.5 percentage points. Convergence curves followed the same progression as in the original work, confirming that the pretrained backbone transfers reliably across both regression and sequence-generation objectives. Across repeated runs, numerical variance remained minimal, and all reproduced visualizations—such as parity plots, distribution comparisons and learning-curve behaviour—aligned with the patterns shown in the published figures. Together, these results indicate that the released model and codebase are fully sufficient to reproduce RXNGraphormer’s findings. Reusability 1. High-throughput amide-coupling experiments To assess the reusability of RXNGraphormer under experimental realistic conditions, we first focused on HTE reaction-yied prediction tasks. Yield tasks benefit from systematically generated datasets and consistent experimental protocols, making them well suited for evaluating how a pretrained model transfers across reaction systems. The model was evaluated on three HTE datasets previously developed in our research group, 12-14 each designed with a distinct evaluation protocol. The full reusiblility experiment results can be found in “Experiment Results” section of our GitHub repository and shown in Figure 3. The first benchmark, the iridium-catalyzed cross-dimerization of sulfoxonium ylides, 12 comprises 600 standardized reactions collected under a single train–test split, suitable for testing generalization under limited data. When RXNGraphormer was applied under the original partitioning, the model achieved an R² of 0.60 with an MAE of 9.46%, performing comparably to the neural-network baseline and below the best tree-based model reported in the paper. 12 This indicates that the graph-transformer backbone can capture the relatively smooth reactivity landscape of sulfoxonium ylides without substantial loss of predictive power. The discrepancy between the higher training-set R² and comparatively lower test-set R² suggests a degree of overfitting under this limited data regime. Incorporating a targeted regularization strategy or early-stopping criterion may therefore help stabilize model generalization when RXNGraphormer is applied to small, single-condition HTE campaigns. The second benchmark, the hindered meta-C–H arylation of o-alkylaryl ketones, 13 contains 1,032 HTE reactions and was originally evaluated using an independent test (withholding four ketones) and a stricter independent test (withholding both ketones and aryltrifluoroborates). This reaction introduces structured distribution shifts through substrate and reagent novelty. RXNGraphormer followed the same ordering of difficulty: performance remained stable on the independent test and decreased substantially under the strict independent test, consistent with the increased combined novelty of both substrates and reagents. Under this more challenging setting, the reproduced R² became negative, reflecting the severely shifted distribution and narrow yield range characterizing this subset. Such behaviour aligns with the expectation that variance-based metrics can invert when applied to small, structurally specialized HTE regimes, even when directional trends remain partially preserved. The third benchmark is the amide-coupling HTE study, 14 which contains approximately 47,000 reactions spanning 96 reaction conditions. Since the original study demonstrated that explicitly modelling mechanistic intermediates can improve performance, 14 we further used this benchmark to probe RXNGraphormer’s built-in intermediate-generation capability and its impact on yield prediction. At the “full HTE” level, the original analysis 14 trained models directly on substrate and condition descriptors without explicit intermediates. In our re-analysis, we compared RXNGraphormer with and without intermediate information against this baseline. In the Full HTE (with NMI intermediate) setting, RXNGraphormer was supplied with intermediates generated by the RXNGraphormer, whereas the benmark model 14 remained purely end-to-end. In the Full HTE (without intermediate) setting, both models ignored explicit intermediates. Across these two regimes, RXNGraphormer delivered competitive performance to the original global HTE models: using automatically generated intermediates slightly improved its accuracy relative to the no-intermediate variant but did not fully close the gap to the best condition-specific baselines, whereas removing intermediates led to a noticeable but controlled loss of performance. In addition to the global model, the original study introduced six representative reaction conditions, for which separate models were built using manually curated reaction templates and intermediates tailored to each reaction condition. The design was mirrored by feeding the same manually curated intermediates into RXNGraphormer and training condition-specific models. Across these six single-condition settings, the model’s performance varied more widely than in the full-HTE regime. For some conditions, RXNGraphormer reached accuracy comparable to the original intermediate-based models, whereas for others its predictive quality deteriorated substantially, falling well below the condition-specific baselines. In these more challenging cases, the combination of reduced data size and highly localized structure–reactivity patterns appeared to limit the benefit of a large, shared encoder. Overall, the three HTE benchmarks collectively demonstrate that RXNGraphormer can ingest both automatically generated and manually designed intermediates, but also highlight that its reusability on narrowly defined single-condition campaigns is more sensitive to data scale and reaction-specific complexity than on large, globally trained HTE models. 2. Heterogeneous literature-derived amide-coupling data To assess RXNGraphormer on literature-derived amide-coupling data, complementing the amide-coupling HTE benchmark, 14 we followed the benchmark protocol introduced by Isayev et al ., 15 which is based on a curated collection of amide-coupling reactions licensed from Reaxys. Because the original Reaxys records cannot be redistributed, only the processed benchmark split —consisting of a training subset and a single held-out test set—was accessible for our evaluation. This test set represents a fixed compilation of literature reactions spanning diverse electrophiles, nucleophiles, bases and solvents, and constitutes the only publicly usable evaluation split available from the benchmark. Using a literature-dervied amide-coupling reaction dataset of approximately 60k reactions, RXNGraphormer was finetuned on the designated training portion and subsequently evaluated it on the held-out literature test set, yielding an of approximately 0.35. This value is lower than the benchmarked performance reported by Isayev et al ., and all reported metrics reflect performance on this unified test set, as no additional Reaxys-derived evaluation splits are available for comparison. Together, these analyses confirm that the publicly released RXNGraphormer model and codebase can be reused for reaction-performance prediction across new, well-defined chemical systems, particularly those generated under standardized high-throughput conditions. At the same time, the results indicate that reusability remains sensitive to distributional shifts and descriptor inconsistencies, suggesting that future applications may benefit from more harmonized data formats and reaction representations when transferring the pretrained model to diverse or heterogeneous reaction domains. Discussion RXNGraphormer represents one of the first unified pre-trained architectures for reaction modelling that seeks to bridge performance prediction and synthesis planning within a single representational framework. A major methodological innovation of the work lies in its pretraining strategy, which constructs fictitious reactions from real ones under chemically constrained rules and trains a graph–transformer backbone using ΔMol representations. Unlike conventional reaction models that act on structures alone, the ΔMol representation encodes chemical change directly, offering a lens through which reactivity patterns can be learned in a task-agnostic manner. The generation of fictitious reactions—guided by atom-mapping consistency, bond-change validity, reaction balancing and structural filtering—further ensures that the negative samples used for weakly supervised pretraining remain chemically plausible. In our reproduction, manual inspection of a subset of generated fictitious reactions did not reveal obvious false negatives, corroborating the authors’ design that such noise is unlikely to dominate the learned manifold. Together, these elements constitute a genuinely novel pretraining paradigm within the field of AI for chemistry, and our independent experiments confirm that the released weights faithfully encode transferable structure–reactivity information across a broad range of downstream tasks. Our reproducibility assessment demonstrates that the original results can be recovered with high fidelity. Regression benchmarks, out-of-sample partitions and sequence-generation tasks all reproduced the reported performance within expected numerical tolerances. Minor discrepancies between reproduced and published metrics are consistent with known sources of nondeterminism in modern deep-learning frameworks, including stochastic GPU kernels, dependency version drift, data-loader variability and floating-point precision effects. These factors are intrinsic to current compute architectures and do not detract from the overall reproducibility of the authors’ pipeline. The near-identity between our reproduced figures and the published visual analyses further underscores the internal consistency of the released codebase. Beyond strict reproduction, our evaluation of reusability highlights both the strengths and boundaries of RXNGraphormer as a pre-trained foundation model for chemical reactivity. On three HTE datasets generated in our research group, the model consistently delivered strong performance when evaluated under broad, well-balanced training distributions. In the sulfoxonium ylide 12 and meta-C–H arylation 13 systems, RXNGraphormer achieved performance broadly comparable to the baseline models, with some conditions showing close agreement and others displaying moderate deviations. These outcomes suggest that the ΔMol encoder captures the principal structure–reactivity patterns underlying these HTE datasets while still exhibiting sensitivity to local distributional differences. The amide-coupling full-HTE benchmark, 14 comprising 47,000 reactions across 96 conditions, further demonstrated the advantages of large-scale pretraining: RXNGraphormer achieved competitive performance with or without automatically generated intermediates, and modest gains were observed when incorporating intermediates derived from the model’s own intermediate-generation mechanism. These trends support the view that pretraining imparts a form of chemical prior that is particularly effective when the target distribution is diverse and the reactivity landscape broad. The limitations of pretraining emerge more clearly in narrowly defined or mechanistically fine-grained settings. In the six single-condition amide-coupling models from our benchmark study, where expert-curated reaction templates and manually constructed intermediates were used, RXNGraphormer exhibited variable behaviour: performance remained competitive for some conditions but deteriorated sharply for others. These cases feature highly localized reactivity patterns and limited data volume, creating conditions under which a general-purpose pre-trained encoder cannot fully substitute for vertical models tailored to specific mechanistic regimes. The contrast between the full-HTE and condition-specific scenarios suggests that RXNGraphormer is particularly well suited for broad HTE landscapes but less effective when asked to infer subtle mechanistic signals in small, specialized domains. This is not a shortcoming of the architecture per se, but rather a reflection of the underlying trade-off between model generality and domain specialization. A similar pattern arises in the literature-derived amide-coupling benchmark, 15 where the model achieved an R² of approximately 0.35 on the curated literature test set. The performance is lower than that of the benchmarked models in the original study yet remains within the typical range observed for yield prediction on heterogeneous, incompletely annotated literature corpora. In this setting, reaction descriptions exhibit substantial variability in reagent identity, stoichiometry, solvent specification and yield reporting. Although RXNGraphormer retains trend-level consistency within chemically coherent subsets, its absolute accuracy is constrained by the granularity and noise inherent to text-mined datasets. These findings parallel the challenges documented by Isayev et al. 15 and emphasise that literature benchmarks probe model robustness under conditions of maximal data heterogeneity, where pretraining alone cannot compensate for missing chemical context. Beyond the benchmarks examined here, our findings illustrate how unified reaction representations may support more integrated pipelines for AI-driven reaction optimisation and synthesis planning. As larger and more standardized reaction corpora emerge—from HTE platforms to curated literature collections—pretrained models such as RXNGraphormer could increasingly function as transferable backbones across diverse modelling tasks. At the same time, several open challenges remain, particularly in low-resource or mechanistically specialized domains where transferable structure–reactivity patterns are harder to learn. Addressing these limitations through improved uncertainty calibration, intermediate modelling and domain-adaptive training strategies will be crucial for extending the impact of unified reaction models. Taken together, our reproduction and reuse experiments demonstrate that RXNGraphormer is a technically reliable and methodologically transparent framework. The close agreement between reproduced and reported results indicates that the released implementation, datasets and configuration files are sufficient to recover all major findings, reflecting a well-engineered pretraining pipeline and a robust ΔMol representation. The ability to verify nearly the entire computational workflow—from architectural components to downstream evaluations—illustrates the strong reproducibility of the original study and sets an encouraging precedent for future chemical foundation models. Our reusability assessment further shows that the pretrained backbone transfers effectively to new experimental settings, particularly in broad, well-balanced high-throughput reaction campaigns where minimal fine-tuning yields strong predictive performance. By contrast, reduced accuracy on narrowly defined or heterogeneous literature-derived datasets highlights the continued importance of data harmonization, mechanistic resolution and standardized reaction representation when adapting large pretrained models to specialized reaction spaces. As the reaction performance models increasingly shift toward unified and pre-trained architectures, sustained commitments to open data pipelines, transparent documentation and accessible pretraining corpora will be essential for ensuring both reproducibility and practical reuse. The present study illustrates how such openness directly facilitates methodological validation and accelerates scientific progress, providing a model for future developments in machine-learning-driven reaction prediction and synthesis planning. Methods 1. Codebase and computational environment All reproduction experiments were conducted using the official RXNGraphormer repository released by the original authors, 11 together with an independent reproduction framework available in our GitHub archive. Computations were performed in PyTorch 2.1 with CUDA 12.2 on NVIDIA A100 GPUs. Molecular parsing, atom mapping and graph construction were implemented using RDKit 2023.03 and PyTorch Geometric. Random seeds were fixed throughout, and deterministic data-loading options were enabled when available. Minor dependency differences from the original environment, such as updated CUDA kernels or RDKit subversions, did not affect the model architecture or training workflow. The publicly released pretrained checkpoint served as the initialization for all downstream evaluations. The original large-scale pretraining was not re-executed due to computational cost, although deterministic forward passes were performed to verify the stability of the ΔMol representation and the integrity of the released encoder. 2. Data sources and preprocessing All datasets from the RXNGraphormer publication, including four regression benchmarks and three sequence-generation benchmarks, were accessed through the authors’ release. Reaction SMILES were canonicalized using RDKit, atom mappings were standardized, and invalid records were filtered according to the described criteria. Reaction graphs were constructed using the ΔMol formulation, in which product and reactant molecular graphs are combined into a unified difference representation. For sequence-generation tasks, reaction strings were preprocessed using the same tokenization rules provided in the official repository. Three high-throughput experimentation datasets were included to evaluate reusability. These datasets comprise an iridium-catalyzed sulfoxonium-ylide cross-coupling campaign, a hindered meta-C–H arylation study, and a 47,000-reaction amide-coupling platform generated under 96 standardized conditions. Raw experimental tables were extracted from the original publications and reprocessed using a uniform canonicalization and mapping pipeline to ensure consistency across datasets. For the amide-coupling benchmark, both global full-HTE data and six condition-specific subsets were reconstructed following the original definitions. Curated mechanistic intermediates, when available, were incorporated as additional molecular graphs; intermediate-free variants omitted these fields. RXNGraphormer-generated intermediates were produced using the intermediate-generation module included in the official codebase. Evaluation on literature-derived reaction data followed the protocol established in the benchmark constructed by Isayev et al . Only the processed split made available with the benchmark, consisting of a training set and a single held-out test set, was used. Because the underlying Reaxys records cannot be redistributed, no additional data were added. Canonicalization, atom mapping and ΔMol conversion were applied identically to the procedures described above. 3. Model training and evaluation procedures For the reproduction of the RXNGraphormer benchmarks, all hyperparameters, including optimizer settings, learning-rate schedules, batch sizes, model depths and inference configurations, followed the parameter files released by the authors. Training, validation and test partitions were identical to those used in the original study. In the case of the USPTO-full sequence model, the pretrained checkpoint was evaluated directly, consistent with the published workflow. No modifications were introduced to the model structure or training logic. For the HTE benchmarks, models were fine-tuned using dataset-specific training protocols defined in the original experimental studies. The sulfoxonium dataset used a single train–test split. The meta-C–H arylation dataset was evaluated under an independent-test setting in which four ketones were held out, as well as a stricter setting in which both ketones and aryltrifluoroborates were withheld. The amide-coupling study involved full-HTE training on the entire 47,000-reaction dataset as well as condition-specific training on each of the six curated subsets. Models incorporating curated or automatically generated intermediates were trained by augmenting the ΔMol input graphs with intermediate information, while intermediate-free variants omitted this augmentation. For the literature-derived amide-coupling dataset, the model was fine-tuned on the released training subset and evaluated on the held-out test subset using the same optimization schedule applied to the HTE tasks. No additional resampling, balancing or hyperparameter tuning was applied. Regression tasks were evaluated using the coefficient of determination, mean absolute error and mean squared error, computed on the designated test sets. Classification tasks used accuracy, precision, recall and F1 score. Retrosynthesis was evaluated using top-k accuracy, and forward reaction prediction using top-1 accuracy, with beam-search and decoding configurations matching those of the RXNGraphormer study. All metrics were computed using standard definitions without modification. 4. Data analysis and visualization Parity plots, distribution comparisons and top-k performance plots were generated using Python and Matplotlib based on the predictions and ground-truth labels from the reproduced experiments and reusability evaluations. All analysis and plotting scripts strictly consumed the datasets and model outputs described above and did not introduce additional data transformations beyond standard normalization and axis scaling. Scripts responsible for data preparation, model training, evaluation and visualization are provided in the reproduction repository. Declarations Code availability All reproduction scripts, model checkpoints, preprocessing utilities and visualization notebooks used in this study are available at the reproduction repository: https://github.com/MJ-Zeng/RXNGraphormer-Reproduction. The original RXNGraphormer implementation and pretrained weights released by the authors can be accessed from their official repository. No modifications were made to the model code during reproduction. Data availability Processed datasets and supplementary evaluation files generated for the reusability experiments are deposited on Figshare: https://figshare.com/articles/thesis/_b_RXNGraphormer_Reproduction_b_/30498368/2. All data used from previously published studies were obtained directly from their respective supporting information. No proprietary or licensed raw reaction data were redistributed. References Coley, C.W., Green, W.H. and Jensen, K.F., 2018. Machine learning in computer-aided synthesis planning. Accounts of chemical research , 51 (5), pp.1281-1289. Ahneman, D.T., Estrada, J.G., Lin, S., Dreher, S.D. and Doyle, A.G., 2018. Predicting reaction performance in C–N cross-coupling using machine learning. Science , 360 (6385), pp.186-190. Segler, M.H., Preuss, M. and Waller, M.P., 2018. Planning chemical syntheses with deep neural networks and symbolic AI. Nature , 555 (7698), pp.604-610. Schwaller, P., Laino, T., Gaudin, T., Bolgar, P., Hunter, C.A., Bekas, C. and Lee, A.A., 2019. Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction. ACS central science , 5 (9), pp.1572-1583. Qiu, J., Xie, J., Su, S., Gao, Y., Meng, H., Yang, Y. and Liao, K., 2022. Selective functionalization of hindered meta-C–H bond of o-alkylaryl ketones promoted by automation and deep learning. Chem , 8 (12), pp.3275-3287. Tu, Z. and Coley, C.W., 2022. Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction. Journal of chemical information and modeling , 62 (15), pp.3503-3513. Schreck, J.S., Coley, C.W. and Bishop, K.J., 2019. Learning retrosynthetic planning through simulated experience. ACS central science , 5 (6), pp.970-981. Dai, H., Li, C., Coley, C., Dai, B. and Song, L., 2019. Retrosynthesis prediction with conditional graph logic network. Advances in Neural Information Processing Systems , 32 . Wang, Y., Pang, C., Wang, Y., Jin, J., Zhang, J., Zeng, X., Su, R., Zou, Q. and Wei, L., 2023. Retrosynthesis prediction with an interpretable deep-learning framework based on molecular assembly tasks. Nature Communications , 14 (1), p.6155. Xu, L.C., Tang, M.J., An, J., Cao, F. and Qi, Y., 2025. A unified pre-trained deep learning framework for cross-task reaction performance prediction and synthesis planning. Nature Machine Intelligence , pp.1-11. licheng-xu-echo. 2025. RXNGraphormer . GitHub repository. Available at: https://github.com/licheng-xu-echo/RXNGraphormer (Accessed: 18 November 2025). Xu, Y., Gao, Y., Su, L., Wu, H., Tian, H., Zeng, M., Xu, C., Zhu, X. and Liao, K., 2023. High‐Throughput Experimentation and Machine Learning‐Assisted Optimization of Iridium‐Catalyzed Cross‐Dimerization of Sulfoxonium Ylides. Angewandte Chemie , 135 (48), p.e202313638. Qiu, J., Xie, J., Su, S., Gao, Y., Meng, H., Yang, Y. and Liao, K., 2022. Selective functionalization of hindered meta-C–H bond of o-alkylaryl ketones promoted by automation and deep learning. Chem , 8 (12), pp.3275-3287. Zhang, C., Lin, Q., Yang, C., Kong, Y., Yu, Z. and Liao, K., 2025. Intermediate knowledge enhanced the performance of the amide coupling yield prediction model. Chemical Science . Liu, Z., Moroz, Y.S. and Isayev, O., 2023. The challenge of balancing model sensitivity and robustness in predicting yields: a benchmarking study of amide coupling reactions. Chemical Science , 14 (39), pp.10835-10846. Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8237411","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":560962036,"identity":"b6d2387f-dada-4e03-b29e-224e45bfd32b","order_by":0,"name":"Kuangbiao Liao","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFUlEQVRIiWNgGAWjYBACPmYQeQBE8IAoGwYGdgY2vFrY0LSkMTAwE9LCgKQFCA4ToYWd+eEDhjOH5c351x488KPivN38ZuZnDxhq7Bj4ZzfgcBibsQHDjcOGO2e8SzjYc+Z2cmMzm7kBw7FkBok7B3D5xUyC4cNhxg03zhgc4G27ncwMFmE7wGAgkYBDC/s3kBZ7kJaDf9vOJUNE/uHTwgM088bhxA3newwO87YdsOMBiTC24dVSbMBwJj15ww2+hMMyZ5ITJJh5yg0S+5J5JG5g18LPf3zjA4Zj1rYbzp89/PFNhZ29fHv7tgcfvtnJ8c/ArgUEmP+ASKgzEhtAZAI0mvAD/gNgyp6wylEwCkbBKBhpAADp71ugDjuJdgAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0001-9089-0569","institution":"Guangzhou National Laboratory","correspondingAuthor":true,"prefix":"","firstName":"Kuangbiao","middleName":"","lastName":"Liao","suffix":""},{"id":560962037,"identity":"5f82fded-65ba-4cea-a566-a0ac46308020","order_by":1,"name":"Chonghuan Zhang","email":"","orcid":"","institution":"Guangzhou National Laboratory","correspondingAuthor":false,"prefix":"","firstName":"Chonghuan","middleName":"","lastName":"Zhang","suffix":""},{"id":560962038,"identity":"0cb2aedd-05ba-46d1-a6de-6d3c8555c5b7","order_by":2,"name":"Majian Zeng","email":"","orcid":"","institution":"Guangzhou National Laboratory","correspondingAuthor":false,"prefix":"","firstName":"Majian","middleName":"","lastName":"Zeng","suffix":""},{"id":560962039,"identity":"11074837-415b-4e6e-9895-3e4b534aeca5","order_by":3,"name":"Qianghua Lin","email":"","orcid":"","institution":"AIChemEco Inc.","correspondingAuthor":false,"prefix":"","firstName":"Qianghua","middleName":"","lastName":"Lin","suffix":""}],"badges":[],"createdAt":"2025-11-29 13:45:56","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8237411/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8237411/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":98374550,"identity":"5605bcba-ce61-450c-94b9-bc522286f45a","added_by":"auto","created_at":"2025-12-17 06:42:31","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":787518,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eOverall workflow of this Reusability Report: (a) RXNGraphormer architecture integrating GNN molecule encoders, an inter-molecular transformer, and task-specific heads for reactivity/selectivity prediction and synthesis planning; (b) reproduction workflow demonstrating successful replication of all core tasks—including reactivity, regioselectivity, enantioselectivity, and synthesis planning—using the released code and checkpoint; and (c) reusability workflow evaluating the pretrained encoder on external high-throughput and literature yield datasets, showing strong transferability on HTE data and expected degradation on heterogeneous literature benchmarks.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-8237411/v1/42667b87d8be68f2d0c1ddd5.png"},{"id":98374549,"identity":"cd9df813-56f1-497e-9f7e-a5ae99c182d0","added_by":"auto","created_at":"2025-12-17 06:42:31","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":376376,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eReproduction of RXNGraphormer across the downstream tasks: (a) average R² comparison on the in-distribution reactivity benchmark (Buchwald–Hartwig \u0026amp; Suzuki–Miyaura yield prediction), the OOS (radical C–H regioselectivity \u0026amp; thiol-addition enantioselectivity), and the external literature reaction yield datasets; (b) MAE distribution across all evaluated tasks; (c) correlation of R² across all downstream tasks; and (d) retrosynthesis and forward top-k accuracy curves on USPTO-50k, USPTO-full, and USPTO-STEREO benchmarks under matched decoding settings.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-8237411/v1/23e398d6b3ca11ae74630e2c.png"},{"id":98374551,"identity":"a48b48a6-bda5-4c5b-adc1-447d59defbd8","added_by":"auto","created_at":"2025-12-17 06:42:31","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":321407,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003e\u003cstrong\u003eReaction-yield prediction tasks:\u003c/strong\u003e\u003c/em\u003e\u003cem\u003e \u003c/em\u003e\u003cem\u003e\u003cstrong\u003e(a)\u003c/strong\u003e\u003c/em\u003e\u003cem\u003e R² correlation on sulfoxonium reactions, meta-C–H transformations, amide-coupling HTE, and amide-coupling literature datasets shown in a parity plot; \u003c/em\u003e\u003cem\u003e\u003cstrong\u003e(b)\u003c/strong\u003e\u003c/em\u003e\u003cem\u003e corresponding R² comparison shown as a histogram; \u003c/em\u003e\u003cem\u003e\u003cstrong\u003e(c)\u003c/strong\u003e\u003c/em\u003e\u003cem\u003e MAE comparison across the same sets of tasks; and \u003c/em\u003e\u003cem\u003e\u003cstrong\u003e(d)\u003c/strong\u003e\u003c/em\u003e\u003cem\u003e reagent-level R² differences for six commonly used amide-coupling reagents under random, partial-novelty, and full-novelty split settings.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-8237411/v1/424afe2969a3c8e8b0f6aa9b.png"},{"id":98622985,"identity":"442b12ed-9101-42ea-a521-bcd519a61639","added_by":"auto","created_at":"2025-12-19 17:03:53","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1942377,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8237411/v1/a09e139b-2e14-49bd-a1da-7be0a219d59d.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Reusability report: A unified pre-trained deep learning framework for cross-task reaction performance prediction and synthesis planning","fulltext":[{"header":"Introduction","content":"\u003cp\u003eMachine-learning approaches have transformed modern chemical synthesis by enabling data-driven prediction of reaction outcomes, condition optimisation and route design\u0026sup1;\u003csup\u003e-\u003c/sup\u003e\u0026sup3;. Graph-based and transformer-based models have demonstrated remarkable progress in capturing structure\u0026ndash;reactivity relationships and predicting yields across diverse chemical spaces⁴\u003csup\u003e-\u003c/sup\u003e⁶. In parallel, algorithms for retrosynthetic and forward synthesis planning have evolved from template-based heuristics to policy-learning and generative frameworks that operate at scale⁷\u003csup\u003e-\u003c/sup\u003e⁹. Together, these developments mark a transition from task-specific systems toward integrated, data-centric paradigms for reaction modelling.\u003c/p\u003e\n\u003cp\u003eDespite this progress, most frameworks remain specialised for either performance regression or synthesis planning, limiting transferability between tasks. Achieving a unified architecture capable of supporting both remains an open challenge. RXNGraphormer, recently introduced by Cao \u003cem\u003eet al\u003c/em\u003e.\u0026sup1;⁰, addresses this by coupling a pre-trained graph\u0026ndash;transformer backbone with a delta-molecular (\u0026Delta;Mol) representation to unify cross-task reaction learning. The model\u0026rsquo;s pretraining across millions of reactions aims to bridge reactivity prediction and synthesis sequence generation through a shared chemical representation.\u003c/p\u003e\n\u003cp\u003eIn this Reusability Report, we independently evaluate the reproducibility and reusability of RXNGraphormer. As shown in Figure 1, using the authors\u0026rsquo; released code and datasets, we reproduce the reported results for reactivity, selectivity and synthesis tasks and extend the assessment to new datasets, including homogenous high-throughput experimentation (HTE) reaction benchmarks and a heterogeneous literature-derived benchmark. We verify the reported performance, test adaptability to new data sources, and assess the model\u0026rsquo;s potential as a reusable foundation for chemical reaction prediction and synthesis planning.\u003c/p\u003e\n\u003ch3\u003eReproducibility\u003c/h3\u003e\n\u003ch2\u003e1. \u003cstrong\u003eRegression benchmarks\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eAll four regression benchmarks across multiple aspects of reaction performance including reactivity, regioselectivity, and enantioselectivity were successfully reproduced. The full reproduction experiment results can be found in \u0026ldquo;Experiment Results\u0026rdquo; section of our GitHub repository and shown in Figure 2. On Buchwald\u0026ndash;Hartwig and Suzuki\u0026ndash;Miyaura coupling reactions, reproduced test-set R\u0026sup2; values of 0.970 and 0.871 closely matched the reported 0.971 and 0.876. The radical C\u0026ndash;H functionalization benchmark yielded a reproduced R\u0026sup2; of 0.992 and MAE of 0.273 kcal mol⁻\u0026sup1;, consistent with the original 0.992 and 0.266 kcal mol⁻\u0026sup1;. For asymmetric thiol addition, reproduced R\u0026sup2; reached 0.916, essentially identical to the reported 0.915. These experiments confirm that the \u0026Delta;Mol encoder and task-specific prediction heads maintain stable numerical behaviour across distinct reaction classes.\u003c/p\u003e\n\u003ch2\u003e2. \u003cstrong\u003eOut-of-sample evaluation\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eThe out-of-sample protocols for Buchwald\u0026ndash;Hartwig coupling reproduced the same hierarchy. Bromide remained the most stable subset (R\u0026sup2; of 0.869), whereas chloride again produced a negative value (R\u0026sup2; of \u0026ndash;0.377), consistent with its compressed yield distribution. Additive-based OOS splits reproduced the reported division between easy and difficult subsets (R\u0026sup2; of 0.815\u0026ndash;0.897 vs 0.651\u0026ndash;0.643). The component-combination split reproduced the difficulty with R\u0026sup2; of 0.732.\u003c/p\u003e\n\u003cp\u003eFor the asymmetric thiol-addition dataset, the reproduced substrate-held-out, catalyst-held-out and combined-held-out results preserved the published trend, yielding R\u0026sup2; values of 0.915, 0.804 and 0.732, respectively. This confirms that catalyst-novelty continues to impose the strongest distributional shift to the latent reactivity space.\u003c/p\u003e\n\u003ch2\u003e3. \u003cstrong\u003eExternal regression datasets\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eThree additional regression datasets were used to assess model transferability beyond the reaction domains examined in the RXNGraphormer publication. The pretrained model produced R\u0026sup2; of 0.739 on asymmetric hydrogenation and R\u0026sup2; of 0.900 on pallada-electrocatalyzed C\u0026ndash;H activation. On the more structurally diverse NICOLit hydrogenation dataset, performance decreased to R\u0026sup2; of 0.209, although class-level discrimination remained intact as reflected in accuracy and precision values. These trends mirror the distributional effects highlighted in the original study.\u003c/p\u003e\n\u003ch2\u003e4. \u003cstrong\u003eSequence-generation tasks\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eRetrosynthesis on USPTO-50k reproduced the reported top-k accuracies, with results of 50.3 %, 69.3 %, 73.7 % and 78.0 % for k = 1, 3, 5 and 10, which closely matched the published 51.0 %, 69.0 %, 73.0 % and 79.2 %. Re-evaluation of the USPTO-full model produced results nearly identical to the published values, as expected for an inference-only checkpoint. For forward prediction on USPTO-480k and USPTO-STEREO, reproduced top-1 accuracies differed by no more than 1.5 percentage points. Convergence curves followed the same progression as in the original work, confirming that the pretrained backbone transfers reliably across both regression and sequence-generation objectives.\u003c/p\u003e\n\u003cp\u003eAcross repeated runs, numerical variance remained minimal, and all reproduced visualizations\u0026mdash;such as parity plots, distribution comparisons and learning-curve behaviour\u0026mdash;aligned with the patterns shown in the published figures. Together, these results indicate that the released model and codebase are fully sufficient to reproduce RXNGraphormer\u0026rsquo;s findings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eReusability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e1.\u0026nbsp; \u0026nbsp;High-throughput amide-coupling experiments\u003c/p\u003e\n\u003cp\u003eTo assess the reusability of RXNGraphormer under experimental realistic conditions, we first focused on HTE reaction-yied prediction tasks. Yield tasks benefit from systematically generated datasets and consistent experimental protocols, making them well suited for evaluating how a pretrained model transfers across reaction systems. The model was evaluated on three HTE datasets previously developed in our research group,\u003csup\u003e12-14\u003c/sup\u003e each designed with a distinct evaluation protocol. The full reusiblility experiment results can be found in \u0026ldquo;Experiment Results\u0026rdquo; section of our GitHub repository and shown in Figure 3. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe first benchmark, the iridium-catalyzed cross-dimerization of sulfoxonium ylides,\u003csup\u003e12\u003c/sup\u003e comprises 600 standardized reactions collected under a single train\u0026ndash;test split, suitable for testing generalization under limited data. When RXNGraphormer was applied under the original partitioning, the model achieved an R\u0026sup2; of 0.60 with an MAE of 9.46%, performing comparably to the neural-network baseline and below the best tree-based model reported in the paper.\u003csup\u003e\u0026nbsp;12\u003c/sup\u003e This indicates that the graph-transformer backbone can capture the relatively smooth reactivity landscape of sulfoxonium ylides without substantial loss of predictive power. The discrepancy between the higher training-set R\u0026sup2; and comparatively lower test-set R\u0026sup2; suggests a degree of overfitting under this limited data regime. Incorporating a targeted regularization strategy or early-stopping criterion may therefore help stabilize model generalization when RXNGraphormer is applied to small, single-condition HTE campaigns.\u003c/p\u003e\n\u003cp\u003eThe second benchmark, the hindered meta-C\u0026ndash;H arylation of o-alkylaryl ketones,\u003csup\u003e13\u003c/sup\u003e contains 1,032 HTE reactions and was originally evaluated using an independent test (withholding four ketones) and a stricter independent test (withholding both ketones and aryltrifluoroborates). This reaction introduces structured distribution shifts through substrate and reagent novelty. RXNGraphormer followed the same ordering of difficulty: performance remained stable on the independent test and decreased substantially under the strict independent test, consistent with the increased combined novelty of both substrates and reagents. Under this more challenging setting, the reproduced R\u0026sup2; became negative, reflecting the severely shifted distribution and narrow yield range characterizing this subset. Such behaviour aligns with the expectation that variance-based metrics can invert when applied to small, structurally specialized HTE regimes, even when directional trends remain partially preserved.\u003c/p\u003e\n\u003cp\u003eThe third benchmark is the amide-coupling HTE study,\u003csup\u003e14\u003c/sup\u003e which contains approximately 47,000 reactions spanning 96 reaction conditions. Since the original study demonstrated that explicitly modelling mechanistic intermediates can improve performance,\u003csup\u003e14\u003c/sup\u003e we further used this benchmark to probe RXNGraphormer\u0026rsquo;s built-in intermediate-generation capability and its impact on yield prediction. At the \u0026ldquo;full HTE\u0026rdquo; level, the original analysis\u003csup\u003e14\u003c/sup\u003e trained models directly on substrate and condition descriptors without explicit intermediates. In our re-analysis, we compared RXNGraphormer with and without intermediate information against this baseline. In the \u003cstrong\u003eFull HTE (with NMI intermediate)\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003esetting, RXNGraphormer was supplied with intermediates generated by the RXNGraphormer, whereas the benmark model\u003csup\u003e14\u003c/sup\u003e remained purely end-to-end. In the \u003cstrong\u003eFull HTE (without intermediate)\u003c/strong\u003e setting, both models ignored explicit intermediates. Across these two regimes, RXNGraphormer delivered competitive performance to the original global HTE models: using automatically generated intermediates slightly improved its accuracy relative to the no-intermediate variant but did not fully close the gap to the best condition-specific baselines, whereas removing intermediates\u0026nbsp;led to a noticeable but controlled loss of performance.\u003c/p\u003e\n\u003cp\u003eIn addition to the global model, the original study introduced six representative reaction conditions, for which separate models were built using manually curated reaction templates and intermediates tailored to each reaction condition. The design was mirrored by feeding the same manually curated intermediates into RXNGraphormer and training condition-specific models. Across these six single-condition settings, the model\u0026rsquo;s performance varied more widely than in the full-HTE regime. For some conditions, RXNGraphormer reached accuracy comparable to the original intermediate-based models, whereas for others its predictive quality deteriorated substantially, falling well below the condition-specific baselines. In these more challenging cases, the combination of reduced data size and highly localized structure\u0026ndash;reactivity patterns appeared to limit the benefit of a large, shared encoder. Overall, the three HTE benchmarks collectively demonstrate that RXNGraphormer can ingest both automatically generated and manually designed intermediates, but also highlight that its reusability on narrowly defined single-condition campaigns is more sensitive to data scale and reaction-specific complexity than on large, globally trained HTE models.\u003c/p\u003e\n\u003cp\u003e2.\u0026nbsp; \u0026nbsp;Heterogeneous literature-derived amide-coupling data\u003c/p\u003e\n\u003cp\u003eTo assess RXNGraphormer on literature-derived amide-coupling data, complementing the amide-coupling HTE benchmark,\u003csup\u003e14\u003c/sup\u003e we followed the benchmark protocol introduced by Isayev \u003cem\u003eet al\u003c/em\u003e.,\u003csup\u003e15\u003c/sup\u003e which is based on a curated collection of amide-coupling reactions licensed from Reaxys. Because the original Reaxys records cannot be redistributed, only the processed \u003cstrong\u003ebenchmark split\u003c/strong\u003e\u0026mdash;consisting of a training subset and a single held-out test set\u0026mdash;was accessible for our evaluation. This test set represents a fixed compilation of literature reactions spanning diverse electrophiles, nucleophiles, bases and solvents, and constitutes the only publicly usable evaluation split available from the benchmark. Using a literature-dervied amide-coupling reaction dataset of approximately 60k reactions, RXNGraphormer was finetuned on the designated training portion and subsequently evaluated it on the held-out literature test set, yielding an\u0026nbsp;\u0026nbsp;of approximately 0.35. This value is lower than the benchmarked performance reported by Isayev \u003cem\u003eet al\u003c/em\u003e., and all reported metrics reflect performance on this unified test set, as no additional Reaxys-derived evaluation splits are available for comparison.\u003c/p\u003e\n\u003cp\u003eTogether, these analyses confirm that the publicly released RXNGraphormer model and codebase can be reused for reaction-performance prediction across new, well-defined chemical systems, particularly those generated under standardized high-throughput conditions. At the same time, the results indicate that reusability remains sensitive to distributional shifts and descriptor inconsistencies, suggesting that future applications may benefit from more harmonized data formats and reaction representations when transferring the pretrained model to diverse or heterogeneous reaction domains.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eRXNGraphormer represents one of the first unified pre-trained architectures for reaction modelling that seeks to bridge performance prediction and synthesis planning within a single representational framework. A major methodological innovation of the work lies in its pretraining strategy, which constructs fictitious reactions from real ones under chemically constrained rules and trains a graph\u0026ndash;transformer backbone using \u0026Delta;Mol representations. Unlike conventional reaction models that act on structures alone, the \u0026Delta;Mol representation encodes chemical change directly, offering a lens through which reactivity patterns can be learned in a task-agnostic manner. The generation of fictitious reactions\u0026mdash;guided by atom-mapping consistency, bond-change validity, reaction balancing and structural filtering\u0026mdash;further ensures that the negative samples used for weakly supervised pretraining remain chemically plausible. In our reproduction, manual inspection of a subset of generated fictitious reactions did not reveal obvious false negatives, corroborating the authors\u0026rsquo; design that such noise is unlikely to dominate the learned manifold. Together, these elements constitute a genuinely novel pretraining paradigm within the field of AI for chemistry, and our independent experiments confirm that the released weights faithfully encode transferable structure\u0026ndash;reactivity information across a broad range of downstream tasks.\u003c/p\u003e\n\u003cp\u003eOur reproducibility assessment demonstrates that the original results can be recovered with high fidelity. Regression benchmarks, out-of-sample partitions and sequence-generation tasks all reproduced the reported performance within expected numerical tolerances. Minor discrepancies between reproduced and published metrics are consistent with known sources of nondeterminism in modern deep-learning frameworks, including stochastic GPU kernels, dependency version drift, data-loader variability and floating-point precision effects. These factors are intrinsic to current compute architectures and do not detract from the overall reproducibility of the authors\u0026rsquo; pipeline. The near-identity between our reproduced figures and the published visual analyses further underscores the internal consistency of the released codebase.\u003c/p\u003e\n\u003cp\u003eBeyond strict reproduction, our evaluation of reusability highlights both the strengths and boundaries of RXNGraphormer as a pre-trained foundation model for chemical reactivity. On three HTE datasets generated in our research group, the model consistently delivered strong performance when evaluated under broad, well-balanced training distributions. In the sulfoxonium ylide\u003csup\u003e12\u003c/sup\u003e and meta-C\u0026ndash;H arylation\u003csup\u003e13\u003c/sup\u003e systems, RXNGraphormer achieved performance broadly comparable to the baseline models, with some conditions showing close agreement and others displaying moderate deviations. These outcomes suggest that the \u0026Delta;Mol encoder captures the principal structure\u0026ndash;reactivity patterns underlying these HTE datasets while still exhibiting sensitivity to local distributional differences. The amide-coupling full-HTE benchmark,\u003csup\u003e14\u003c/sup\u003e comprising 47,000 reactions across 96 conditions, further demonstrated the advantages of large-scale pretraining: RXNGraphormer achieved competitive performance with or without automatically generated intermediates, and modest gains were observed when incorporating intermediates derived from the model\u0026rsquo;s own intermediate-generation mechanism. These trends support the view that pretraining imparts a form of chemical prior that is particularly effective when the target distribution is diverse and the reactivity landscape broad.\u003c/p\u003e\n\u003cp\u003eThe limitations of pretraining emerge more clearly in narrowly defined or mechanistically fine-grained settings. In the six single-condition amide-coupling models from our benchmark study, where expert-curated reaction templates and manually constructed intermediates were used, RXNGraphormer exhibited variable behaviour: performance remained competitive for some conditions but deteriorated sharply for others. These cases feature highly localized reactivity patterns and limited data volume, creating conditions under which a general-purpose pre-trained encoder cannot fully substitute for vertical models tailored to specific mechanistic regimes. The contrast between the full-HTE and condition-specific scenarios suggests that RXNGraphormer is particularly well suited for broad HTE landscapes but less effective when asked to infer subtle mechanistic signals in small, specialized domains. This is not a shortcoming of the architecture per se, but rather a reflection of the underlying trade-off between model generality and domain specialization.\u003c/p\u003e\n\u003cp\u003eA similar pattern arises in the literature-derived amide-coupling benchmark,\u003csup\u003e15\u003c/sup\u003e where the model achieved an R\u0026sup2; of approximately 0.35 on the curated literature test set. The performance is lower than that of the benchmarked models in the original study yet remains within the typical range observed for yield prediction on heterogeneous, incompletely annotated literature corpora. In this setting, reaction descriptions exhibit substantial variability in reagent identity, stoichiometry, solvent specification and yield reporting. Although RXNGraphormer retains trend-level consistency within chemically coherent subsets, its absolute accuracy is constrained by the granularity and noise inherent to text-mined datasets. These findings parallel the challenges documented by Isayev \u003cem\u003eet al.\u003c/em\u003e\u003csup\u003e15\u003c/sup\u003e and emphasise that literature benchmarks probe model robustness under conditions of maximal data heterogeneity, where pretraining alone cannot compensate for missing chemical context.\u003c/p\u003e\n\u003cp\u003eBeyond the benchmarks examined here, our findings illustrate how unified reaction representations may support more integrated pipelines for AI-driven reaction optimisation and synthesis planning. As larger and more standardized reaction corpora emerge\u0026mdash;from HTE platforms to curated literature collections\u0026mdash;pretrained models such as RXNGraphormer could increasingly function as transferable backbones across diverse modelling tasks. At the same time, several open challenges remain, particularly in low-resource or mechanistically specialized domains where transferable structure\u0026ndash;reactivity patterns are harder to learn. Addressing these limitations through improved uncertainty calibration, intermediate modelling and domain-adaptive training strategies will be crucial for extending the impact of unified reaction models.\u003c/p\u003e\n\u003cp\u003eTaken together, our reproduction and reuse experiments demonstrate that RXNGraphormer is a technically reliable and methodologically transparent framework. The close agreement between reproduced and reported results indicates that the released implementation, datasets and configuration files are sufficient to recover all major findings, reflecting a well-engineered pretraining pipeline and a robust \u0026Delta;Mol representation. The ability to verify nearly the entire computational workflow\u0026mdash;from architectural components to downstream evaluations\u0026mdash;illustrates the strong reproducibility of the original study and sets an encouraging precedent for future chemical foundation models.\u003c/p\u003e\n\u003cp\u003eOur reusability assessment further shows that the pretrained backbone transfers effectively to new experimental settings, particularly in broad, well-balanced high-throughput reaction campaigns where minimal fine-tuning yields strong predictive performance. By contrast, reduced accuracy on narrowly defined or heterogeneous literature-derived datasets highlights the continued importance of data harmonization, mechanistic resolution and standardized reaction representation when adapting large pretrained models to specialized reaction spaces. As the reaction performance models increasingly shift toward unified and pre-trained architectures, sustained commitments to open data pipelines, transparent documentation and accessible pretraining corpora will be essential for ensuring both reproducibility and practical reuse. The present study illustrates how such openness directly facilitates methodological validation and accelerates scientific progress, providing a model for future developments in machine-learning-driven reaction prediction and synthesis planning.\u003c/p\u003e"},{"header":"Methods","content":"\u003ch2\u003e1. \u003cstrong\u003eCodebase and computational environment\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eAll reproduction experiments were conducted using the official RXNGraphormer repository released by the original authors,\u003csup\u003e11\u003c/sup\u003e together with an independent reproduction framework available in our GitHub archive. Computations were performed in PyTorch 2.1 with CUDA 12.2 on NVIDIA A100 GPUs. Molecular parsing, atom mapping and graph construction were implemented using RDKit 2023.03 and PyTorch Geometric. Random seeds were fixed throughout, and deterministic data-loading options were enabled when available. Minor dependency differences from the original environment, such as updated CUDA kernels or RDKit subversions, did not affect the model architecture or training workflow. The publicly released pretrained checkpoint served as the initialization for all downstream evaluations. The original large-scale pretraining was not re-executed due to computational cost, although deterministic forward passes were performed to verify the stability of the \u0026Delta;Mol representation and the integrity of the released encoder.\u003c/p\u003e\n\u003ch2\u003e2. \u003cstrong\u003eData sources and preprocessing\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eAll datasets from the RXNGraphormer publication, including four regression benchmarks and three sequence-generation benchmarks, were accessed through the authors\u0026rsquo; release. Reaction SMILES were canonicalized using RDKit, atom mappings were standardized, and invalid records were filtered according to the described criteria. Reaction graphs were constructed using the \u0026Delta;Mol formulation, in which product and reactant molecular graphs are combined into a unified difference representation. For sequence-generation tasks, reaction strings were preprocessed using the same tokenization rules provided in the official repository.\u003c/p\u003e\n\u003cp\u003eThree high-throughput experimentation datasets were included to evaluate reusability. These datasets comprise an iridium-catalyzed sulfoxonium-ylide cross-coupling campaign, a hindered meta-C\u0026ndash;H arylation study, and a 47,000-reaction amide-coupling platform generated under 96 standardized conditions. Raw experimental tables were extracted from the original publications and reprocessed using a uniform canonicalization and mapping pipeline to ensure consistency across datasets. For the amide-coupling benchmark, both global full-HTE data and six condition-specific subsets were reconstructed following the original definitions. Curated mechanistic intermediates, when available, were incorporated as additional molecular graphs; intermediate-free variants omitted these fields. RXNGraphormer-generated intermediates were produced using the intermediate-generation module included in the official codebase.\u003c/p\u003e\n\u003cp\u003eEvaluation on literature-derived reaction data followed the protocol established in the benchmark constructed by Isayev \u003cem\u003eet al\u003c/em\u003e. Only the processed split made available with the benchmark, consisting of a training set and a single held-out test set, was used. Because the underlying Reaxys records cannot be redistributed, no additional data were added. Canonicalization, atom mapping and \u0026Delta;Mol conversion were applied identically to the procedures described above.\u003c/p\u003e\n\u003ch2\u003e3. \u003cstrong\u003eModel training and evaluation procedures\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eFor the reproduction of the RXNGraphormer benchmarks, all hyperparameters, including optimizer settings, learning-rate schedules, batch sizes, model depths and inference configurations, followed the parameter files released by the authors. Training, validation and test partitions were identical to those used in the original study. In the case of the USPTO-full sequence model, the pretrained checkpoint was evaluated directly, consistent with the published workflow. No modifications were introduced to the model structure or training logic.\u003c/p\u003e\n\u003cp\u003eFor the HTE benchmarks, models were fine-tuned using dataset-specific training protocols defined in the original experimental studies. The sulfoxonium dataset used a single train\u0026ndash;test split. The meta-C\u0026ndash;H arylation dataset was evaluated under an independent-test setting in which four ketones were held out, as well as a stricter setting in which both ketones and aryltrifluoroborates were withheld. The amide-coupling study involved full-HTE training on the entire 47,000-reaction dataset as well as condition-specific training on each of the six curated subsets. Models incorporating curated or automatically generated intermediates were trained by augmenting the \u0026Delta;Mol input graphs with intermediate information, while intermediate-free variants omitted this augmentation.\u003c/p\u003e\n\u003cp\u003eFor the literature-derived amide-coupling dataset, the model was fine-tuned on the released training subset and evaluated on the held-out test subset using the same optimization schedule applied to the HTE tasks. No additional resampling, balancing or hyperparameter tuning was applied.\u003c/p\u003e\n\u003cp\u003eRegression tasks were evaluated using the coefficient of determination, mean absolute error and mean squared error, computed on the designated test sets. Classification tasks used accuracy, precision, recall and F1 score. Retrosynthesis was evaluated using top-k accuracy, and forward reaction prediction using top-1 accuracy, with beam-search and decoding configurations matching those of the RXNGraphormer study. All metrics were computed using standard definitions without modification.\u003c/p\u003e\n\u003ch2\u003e4. \u003cstrong\u003eData analysis and visualization\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eParity plots, distribution comparisons and top-k performance plots were generated using Python and Matplotlib based on the predictions and ground-truth labels from the reproduced experiments and reusability evaluations. All analysis and plotting scripts strictly consumed the datasets and model outputs described above and did not introduce additional data transformations beyond standard normalization and axis scaling. Scripts responsible for data preparation, model training, evaluation and visualization are provided in the reproduction repository.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003e\u003cstrong\u003eCode availability\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eAll reproduction scripts, model checkpoints, preprocessing utilities and visualization notebooks used in this study are available at the reproduction repository:\u003cbr\u003ehttps://github.com/MJ-Zeng/RXNGraphormer-Reproduction.\u003c/p\u003e\n\u003cp\u003eThe original RXNGraphormer implementation and pretrained weights released by the authors can be accessed from their official repository. No modifications were made to the model code during reproduction.\u003c/p\u003e\n\u003ch2\u003e\u003cstrong\u003eData availability\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eProcessed datasets and supplementary evaluation files generated for the reusability experiments are deposited on Figshare:\u003cbr\u003ehttps://figshare.com/articles/thesis/_b_RXNGraphormer_Reproduction_b_/30498368/2.\u003c/p\u003e\n\u003cp\u003eAll data used from previously published studies were obtained directly from their respective supporting information. No proprietary or licensed raw reaction data were redistributed.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eColey, C.W., Green, W.H. and Jensen, K.F., 2018. Machine learning in computer-aided synthesis planning. \u003cem\u003eAccounts of chemical research\u003c/em\u003e, \u003cem\u003e51\u003c/em\u003e(5), pp.1281-1289.\u003c/li\u003e\n\u003cli\u003eAhneman, D.T., Estrada, J.G., Lin, S., Dreher, S.D. and Doyle, A.G., 2018. Predicting reaction performance in C\u0026ndash;N cross-coupling using machine learning. \u003cem\u003eScience\u003c/em\u003e, \u003cem\u003e360\u003c/em\u003e(6385), pp.186-190.\u003c/li\u003e\n\u003cli\u003eSegler, M.H., Preuss, M. and Waller, M.P., 2018. Planning chemical syntheses with deep neural networks and symbolic AI. \u003cem\u003eNature\u003c/em\u003e, \u003cem\u003e555\u003c/em\u003e(7698), pp.604-610.\u003c/li\u003e\n\u003cli\u003eSchwaller, P., Laino, T., Gaudin, T., Bolgar, P., Hunter, C.A., Bekas, C. and Lee, A.A., 2019. Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction. \u003cem\u003eACS central science\u003c/em\u003e, \u003cem\u003e5\u003c/em\u003e(9), pp.1572-1583.\u003c/li\u003e\n\u003cli\u003eQiu, J., Xie, J., Su, S., Gao, Y., Meng, H., Yang, Y. and Liao, K., 2022. Selective functionalization of hindered meta-C\u0026ndash;H bond of o-alkylaryl ketones promoted by automation and deep learning. \u003cem\u003eChem\u003c/em\u003e, \u003cem\u003e8\u003c/em\u003e(12), pp.3275-3287.\u003c/li\u003e\n\u003cli\u003eTu, Z. and Coley, C.W., 2022. Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction. \u003cem\u003eJournal of chemical information and modeling\u003c/em\u003e, \u003cem\u003e62\u003c/em\u003e(15), pp.3503-3513.\u003c/li\u003e\n\u003cli\u003eSchreck, J.S., Coley, C.W. and Bishop, K.J., 2019. Learning retrosynthetic planning through simulated experience. \u003cem\u003eACS central science\u003c/em\u003e, \u003cem\u003e5\u003c/em\u003e(6), pp.970-981.\u003c/li\u003e\n\u003cli\u003eDai, H., Li, C., Coley, C., Dai, B. and Song, L., 2019. Retrosynthesis prediction with conditional graph logic network. \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e, \u003cem\u003e32\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eWang, Y., Pang, C., Wang, Y., Jin, J., Zhang, J., Zeng, X., Su, R., Zou, Q. and Wei, L., 2023. Retrosynthesis prediction with an interpretable deep-learning framework based on molecular assembly tasks. \u003cem\u003eNature Communications\u003c/em\u003e, \u003cem\u003e14\u003c/em\u003e(1), p.6155.\u003c/li\u003e\n\u003cli\u003eXu, L.C., Tang, M.J., An, J., Cao, F. and Qi, Y., 2025. A unified pre-trained deep learning framework for cross-task reaction performance prediction and synthesis planning. \u003cem\u003eNature Machine Intelligence\u003c/em\u003e, pp.1-11. \u003c/li\u003e\n\u003cli\u003elicheng-xu-echo. 2025. \u003cem\u003eRXNGraphormer\u003c/em\u003e. GitHub repository. Available at: https://github.com/licheng-xu-echo/RXNGraphormer (Accessed: 18 November 2025).\u003c/li\u003e\n\u003cli\u003eXu, Y., Gao, Y., Su, L., Wu, H., Tian, H., Zeng, M., Xu, C., Zhu, X. and Liao, K., 2023. High‐Throughput Experimentation and Machine Learning‐Assisted Optimization of Iridium‐Catalyzed Cross‐Dimerization of Sulfoxonium Ylides. \u003cem\u003eAngewandte Chemie\u003c/em\u003e, \u003cem\u003e135\u003c/em\u003e(48), p.e202313638.\u003c/li\u003e\n\u003cli\u003eQiu, J., Xie, J., Su, S., Gao, Y., Meng, H., Yang, Y. and Liao, K., 2022. Selective functionalization of hindered meta-C\u0026ndash;H bond of o-alkylaryl ketones promoted by automation and deep learning. \u003cem\u003eChem\u003c/em\u003e, \u003cem\u003e8\u003c/em\u003e(12), pp.3275-3287.\u003c/li\u003e\n\u003cli\u003eZhang, C., Lin, Q., Yang, C., Kong, Y., Yu, Z. and Liao, K., 2025. Intermediate knowledge enhanced the performance of the amide coupling yield prediction model. \u003cem\u003eChemical Science\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eLiu, Z., Moroz, Y.S. and Isayev, O., 2023. The challenge of balancing model sensitivity and robustness in predicting yields: a benchmarking study of amide coupling reactions. \u003cem\u003eChemical Science\u003c/em\u003e, \u003cem\u003e14\u003c/em\u003e(39), pp.10835-10846.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8237411/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8237411/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eDeep learning has substantially advanced reaction-yield prediction and synthesis-planning methodologies, yet achieving a unified architecture capable of transferring across these tasks remains a central challenge in chemical machine learning. RXNGraphormer introduces such a framework by combining a pretrained graph–transformer encoder with a delta-molecular reaction representation designed to support cross-task generalization. In this reusability report, we independently assess the reproducibility and practical applicability of RXNGraphormer using the released implementation, pretrained checkpoint and benchmark datasets. All major regression and sequence-generation results reported in the original study were consistently reproduced, including the relative difficulty patterns in out-of-sample evaluations, demonstrating the stability and transparency of the published workflow.\u003c/p\u003e\n\u003cp\u003eTo evaluate reusability, we examined the model’s transfer to multiple high-throughput datasets generated under standardized experimental conditions. In these settings, the pretrained encoder adapted efficiently and delivered strong predictive performance with minimal fine-tuning. When applied to a heterogeneous literature-derived benchmark, performance decreased, reflecting the inherent variability and structural noise characteristic of uncurated reaction corpora.\u003c/p\u003e\n\u003cp\u003eOverall, our findings indicate that RXNGraphormer constitutes a reproducible and practically reusable chemical foundation model, capable of supporting both reaction-performance prediction and synthesis-planning tasks across diverse settings. These results further highlight the importance of harmonized reaction representations, curated experimental data and domain-specific refinement. Looking forward, continued progress in large-scale pretraining, interpretable reaction embeddings and standardized reaction corpora will be essential for extending the reach of unified chemical models to broader and more complex reaction spaces.\u003c/p\u003e","manuscriptTitle":"Reusability report: A unified pre-trained deep learning framework for cross-task reaction performance prediction and synthesis planning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-17 06:42:27","doi":"10.21203/rs.3.rs-8237411/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-machine-intelligence","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"natmachintell","sideBox":"Learn more about [Nature Machine Intelligence](http://www.nature.com/natmachintell/)","snPcode":"","submissionUrl":"","title":"Nature Machine Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Research","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"8b72420a-29de-4c52-8f2d-d28f6bc605fa","owner":[],"postedDate":"December 17th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":59722459,"name":"Physical sciences/Chemistry/Cheminformatics"},{"id":59722460,"name":"Physical sciences/Mathematics and computing/Computational science"}],"tags":[],"updatedAt":"2026-05-13T06:15:50+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-17 06:42:27","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8237411","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8237411","identity":"rs-8237411","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}