AlloyGPT: End-to-end prediction and design of additively manufacturable alloys using an autoregressive language model

AlloyGPT: End-to-end prediction and design of additively manufacturable alloys using an autoregressive language model | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article AlloyGPT: End-to-end prediction and design of additively manufacturable alloys using an autoregressive language model Bo Ni, Benjamin Glaser, S. Mohadeseh Taheri-Mousavi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6067058/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 26 Sep, 2025 Read the published version in npj Computational Materials → Version 1 posted 11 You are reading this latest preprint version Abstract Rapid progress in additive manufacturing of alloys opens opportunities in controlling compositions and microstructures at voxel-size resolution in complex geometries, thus unlocking unprecedented design and performance in various critical engineering applications. However, to fully exploit such potential, capable yet efficient models for navigating the vast design spaces of alloy compositions, structures and properties are of great research interest. Here, we present AlloyGPT, an autoregressive alloy-specific language model, that learns the composition-structure-property relationship and generates novel designs for additively manufacturable alloys. Specifically, we develop efficient grammar to convert physics-rich alloy datasets into readable text records for both forward prediction and inverse design tasks. Then, we construct a customized tokenizer and generative pre-trained transformer (GPT) model to master this alloy-specific language through autoregressive training. At deployment, our model can accurately predict multiple phase structures and properties based on given alloy compositions, achieving R2 values ranging from 0.86 to 0.99 for the test set. When tested beyond the learned composition domain, this performance only degrades gradually in a stable manner. Given the desired properties and structures, the same model can suggest multiple alloy compositions that meet the design goals. And the balance between composition diversity and design accuracy can be further tuned stably. Our AlloyGPT model presents a novel way of integrating comprehensive knowledge of alloys in terms of language and can simultaneously solve forward prediction and inverse design tasks with accuracy, diversity and robustness. This fundamental language model will open new avenues to accelerate knowledge integration and material design for pure or gradient structural alloys manufactured by traditional and additive manufacturing. Physical sciences/Materials science/Structural materials/Metals and alloys Physical sciences/Materials science/Theory and computation/Computational methods Alloy design generative deep learning composition-structure-property relationship language model alloy language gradient alloys alloy diversity Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Teaser Generative AI model learning an alloy-specific language can predict and design additively manufacturable alloys with accuracy, diversity, and robustness. 1. Introduction Structural alloys are indispensable in modern engineering, serving critical roles across industries such as aerospace 1 – 3 , automotive 4 – 6 , and energy 7 – 9 . There is a continuous pursuit for novel alloys with enhanced properties 10 – 15 , including strength 10 , 11 , toughness 12 , 13 and resistance to failure under various working environments 14 , 15 . Recent developments in complex alloys, e.g., high entropy alloys 16 , 17 or multi-principal element alloys 18 , 19 , are actively enlarging the composition palette for alloy design, pushing the boundaries of traditional compositions. Concurrently, rapid advances in additive manufacturing (AM) have unlocked unprecedented control over alloy compositions, microstructures and geometries at voxel-scale resolution while shortening supply chain of manufacturing, thus enabling further tailored solutions for demanding applications 20 , 21 . Together, these advancements are opening an expansive design space that consists of numerous composition choices and manufacturing procedures and hold promises for achieving combinations of superior properties. However, experimental fabrication and testing across the whole design space are extremely costly in resources as well as time-cosuming 22 , 23 . Therefore, efficient yet affordable methods to navigate this broad design space for desired performances are of great interest, not only to advance scientific understanding but also to enable novel engineering applications with next-generation super alloys. Recent progress in artificial intelligence (AI) has rapidly expanded the horizon of solving scientific problems and engineering applications using a data-driven methodology. Large language models (LLMs), e.g., GPT-4 from openAI 24 and Llama models from Meta 25 , trained on general language corpora have demonstrated intelligent capabilities in perceiving, understanding and applying natural languages 26 , 27 as well as coding languages 28 , 29 , thus assisting human researchers in accelerating and automating workflows in various tasks 30 – 32 . For solving scientific challenges, AI-based models, such as Alphafold 2 33–35 and RoseTTAFold 36 , 37 , can predict the atomic structures of proteins based on their sequences, reaching experimental accuracy but at a much-reduced cost 34 . On material engineering, end-to-end generative AI (GAI) models have been developed to design novel protein sequences that meet the desired structures or properties 38 – 40 . Similar explorations have been actively developed for various material systems, including functional molecules 41 , 42 , polymers 43 , 44 , inorganic crystals 45 , 46 as well as metallic alloys 47 , 48 . For structural alloys, data on experimentally measured mechanical properties remains relatively sparse due to their time-consuming and costly nature. Since language models have unique strength of integrating data from various sources as general text and further being multi-modal, they are highly interesting specifically in the field of alloy design while in-depth studies remain rare. In this work, we explore the possibility of solving the alloy design challenge from a language modeling perspective by leveraging GAI and alloy data. Specifically, we encode physics-rich alloy data as one-dimensional (1D) sequences of text and develop an attention-based 49 LLM to capture the underlying patterns in this alloy-specific language. Our model, AlloyGPT, at deployment can accurately predict structures and properties for the given compositions and achieve R2 values ranging from 0.86 to 0.99 on the test set. When tested with compositions beyond the learned domain, we observe the accuracy tends to degrade but in a gradual and stable manner with respect to the distance in the composition space. The same model can also tackle alloy design challenges. For the same given property target, it generates various composition choices while maintaining high design accuracy. Through a sampling parameter, we can further boost the creativity of the model and tune the balance between the diversity of the suggested compositions and the accuracy of the delivered properties. Those results of our numerical experiments demonstrate that language modeling with the LLM specialized for alloys can provide accurate and robust representation of alloy physics. As a probabilistic model by nature, our model is particularly suitable for alloy design tasks with high degeneracy. For example, it can be exploited for designing gradient composition or microstructure in alloys. At each voxel of design, depending on constraints, the possible solutions will be further restricted to lead to higher overall performance and design metrics. These capabilities can be valuable for accelerating alloy discoveries in AM and reducing time and resources costs. Our methodology is expected to be generalizable for new alloys and other materials with broad design spaces, potentially leading to a foundation language model with comprehensive material knowledge and suitable for integrated material design in the future. 2. Result and discussions An alloy-specific language dataset and AlloyGPT model Natural languages are powerful tools that enable human communications. In scientific communities, research can be conducted in various forms, including experiments, simulations or theories ( Figure 1 A). Communication and documentation of the research are dominantly taken in the form of languages so far, either as written text or oral speech. Inspired by this universal flexibility, here we explore the possibility of using language-like 1D sequences to directly represent physics-rich data for alloys. Note that the intended speakers of such languages are LLMs, instead of humans. Without loss of generality, in this work we adopt a database of Al-based alloys 50 as an example of scientific data. As shown in Figure 1 B, entries from this database include numerical values for various physical quantities, including the compositions in terms of mol% of Al and the five alloying elements (i.e., Ni, Er, Zr, Y and Yb), structure information of the key phases (L1 2 , ternary, Al 3 Ni and Al 3 Zr phases) in the as-built and aged conditions, as well as related properties (diffusion resistivity, misfit, coarsening rate metric, freezing range, crack susceptibility coefficient and hot cracking susceptibility). This database is based on CALPHAD (CALculation of PHAse Diagrams) simulations and has been validated in several experiments 50,51 . Detailed information on this dataset can be found in the Materials and Method section. To reformat individual entries as “sentences” of an alloy-specific language, we group information based on underlying physics and order the information blocks based on the nature of the tasks. As shown in Figure 1 B, locally we group the physical quantities as blocks on compositions (in the red dash rectangle), structures (green) and properties (blue) separately, given that compositions lead to specific phases and phase distribution affects the properties. Globally, in the sentence structure, we order those blocks by putting the given information ahead of the queried. For example, for forward prediction, compositions are listed before structures and properties ( Figure 1 C, sentence structure). While in inverse design, the sequence starts with properties which are followed by structures and compositions ( Figure 1 D, sentence structure). To distinguish different tasks, we add a task block at the beginning of the sentences to label the task name. We also include simple signs to improve the readability of the sentences for human researchers ( Figure 1 CD, examples). Curly brackets enclose the individual blocks; square brackets enclose the list of physical quantities inside the block. The physical quantities are documented as individual pairs of “Key: Value”, where the key is a readable name of the quantity and the numerical value are expressed in a scientific format. To connect the blocks, we use the directional sign, “=>”, to indicate the reading or inferring direction and equal sign “=” for equivalence. With these simple rules, we can efficiently convert the numerical dataset of Al-based alloy into the corpus of sentences for different tasks. The dataset of this alloy-specific language on Al-based alloys can be found in the SI. We expect these rules can be straightforwardly generalized to include more information blocks or applied to other alloys or material systems. It should be noted that the new alloy-specific language dataset keeps almost all the information of the original numerical dataset. The key difference is that it enables us to view alloy prediction and design challenges as the same sentence completion task. Specifically, given only the beginning blocks, to accurately complete these sentences will be equivalent to predict alloy structures and properties based on the given composition, or to design alloy compositions that fulfill the given properties. To handle such language tasks, we develop an attention-based 49 autoregressive LLM, AlloyGPT, and test its performance for both forward prediction and inverse design tasks for alloys. AlloyGPT model adopts an architecture similar to GPT-2 52 with a tokenizer 53 customized for the alloy language. We train AlloyGPT from scratch by using the curated alloy-specific language database. Sentences for both forward prediction and inverse design are both included and randomly mixed in the train batches. More details of the model architecture and training can be found in the Materials and Methods section. Learn the composition-structure-property (C-S-P) relationships with AlloyGPT To investigate whether our AlloyGPT model can learn the underlying composition-structure-property relationship through the alloy-specific language and autoregressive training, here we apply AlloyGPT model to solve forward prediction problems for Al-based alloys. Distinguishing from conventional methods, here we pose the problem as a language task. As shown in Figure 2 A, we provide a text prompt with the information blocks of the task type and the composition as the input. Our model is able to continue the sentence in the correct format of the alloy-specific language and finish it at the right sequence length. This behavior indicates that AlloyGPT can learn the grammar rules via training. Quantitative results then can be extracted from the text. For the case picked in Figure 2 A, the predicted values of the alloy phases and properties are close to the ground truth, thus solving this composition-to-structure and property (C-to-SP) task. Following this procedure, we performed this C-to-SP task for the whole test set and observed consistent high accuracy. Representative results of examples from the structure and property blocks are shown in Figure 2 B (as-built L1 2 phase) and C (hot cracking susceptibility) respectively. Good agreements are observed for both the overall distribution as well as the individual values between the prediction and the ground truth. R2 values larger than 0.95 are observed for as-built L1 2 phase mole fraction and hot cracking susceptibility. The positive results observed here indicate that language modeling can be competitive to conventional numerical methods in capturing the hidden pattern between composition, structures and properties in this Al-based alloy family, thus potentially opening new avenues to alloy research. As an autoregressive language model 54 , AlloyGPT may process length dependence of its predicting accuracy and present convenient strategies to enhance its performance. Specifically, AlloyGPT completes the given sentences by regressively predicting the next token. Thus, the predicting error can accumulate and influence the accuracy of the content predicted at a later stage. In Figure 2 C, we plot the R2 values (blue bars for C-to-SP tasks) of all the physical quantities in their predicting order from left to right. Our model achieves high accuracy across various physical quantities with R2 values ranging from 0.86 to 0.99 (See Table S1 for details). At the same time, the property block (shaded in blue) predicted later in the output shows overall smaller R2 values compared with the earlier predicted structure block (shaded in green). Therefore, to improve the accuracy of property predictions, one straightforward way could be to increase the input prompt length. For example, providing not only composition but also the structure blocks for property prediction (i.e., a CS-to-P task) can reduce the final prediction length for the language model. As shown in Figure 2 C, indeed, such CS-to-P tasks performed by the same model show improved accuracy with larger R2 values (red bars). This strategy also aligns with a physics-based consideration. Because composition and structures together may better define the alloy and determine its properties. It should also be noted that there could exist other factors that affect the prediction accuracy. For example, by definition, CSC values are determined by the whole solidification process 55 , which makes it relatively challenging to predict. And we observe CSC has the lowest R2 value while it is not predicted at last. Design Al-based alloys for target properties using AlloyGPT As a language model of a probabilistic nature, AlloyGPT provides unique advantages in addressing the inverse design challenges for alloys. On one hand, AlloyGPT model unifies the forward predicting and inverse designing tasks under the same language task. From the format point of view, the language modeling task of sentence completion developed for the forward predicting tasks in the previous section can be straightforwardly applied to the inverse design tasks. As shown in Figure 3 A (left panel), the updated input prompt can include only the property information for a property-to-structure and composition (P-to-SC) design task, or both the property and structure blocks for a property and structure-to-composition (PS-to-C) design. Our model is tasked to complete the remaining of the sentence with the composition information included. On the other hand, the probability-based prediction of AlloyGPT model can be leveraged to capture degenerated solutions in alloy design challenges, which will be discussed later in this section. To evaluate the performance of this unified pathway for solving design challenges, we perform P-to-SC and PS-to-C design tasks based on the test set using AlloyGPT. By setting the properties of the data entries from the test set as the goal, we make sure there exists at least one composition as the solution. The R2 values calculated using the known compositions from the test set and those suggested by our model are shown in Figure 3 A (right penal). When both properties and structures are provided as the input prompt or design goal, in the PS-to-C tasks the suggested compositions achieve high R2 values (red bars in Figure 3 A) (similar to those observed in the previous section for forward predicting tasks) and agrees well with the known compositions from the test set (taking Y in Figure 3 C as the weakest example). This result demonstrates that AlloyGPT can “rediscover” compositions known to the test set based on conditioning of properties and structures. However, when only properties are prescribed as the design goal in the P-to-SC tasks, the suggested physical quantities, for both structures and compositions, achieve low R2 values (blue bars in Figure 3 A). As shown in Figure 3 B, the suggested mol% of Y tends to diverge from the known records and spread broadly. Thus, AlloyGPT can also suggest compositions different from the known ones. It should be clarified that the R2 values of the compositions in Figure 3 A (right panel, shaded in red) should not be taken as the indicator of design accuracy. By definition, it represents the composition recoverability with respect to the test set, i.e., how similar the proposed compositions are to the known ones from the test set ( Figure 4 A). Design accuracy, instead, should be evaluated by comparing the targeted goal and the delivered performance, i.e., the prescribed properties and the achieved values for the P-to-SC design task ( Figure 4 A). To truly estimate the design accuracy, we first obtain the property and structure results of the designed compositions by applying the same protocol used for the initial dataset creation, and then calculate the R2 values between these and the input design target. As shown in Figure 4 B, high R2 values (blue bars) are obtained between the input property goals and the achieved properties, indicating actually AlloyGPT also achieves high design accuracy for the P-to-SC design tasks. Comparisons of some example properties, including coarsening metric and hot cracking susceptibility, are shown in Figure 4 C and D. Good agreement in both distributions and individual values are observed. At the same time, the low composition recoverability (red bars in Figure 4 B) suggests for the given properties, there could be different compositions as the design solutions. By achieving high design accuracy and low composition recoverability, AlloyGPT has demonstrated promising potential in addressing the challenges of degenerated solutions in inverse design problems and maintaining design accuracy and diversity at the same time. It should also be noted that AlloyGPT addresses the inverse design tasks in an efficient end-to-end manner. Thus, it bypasses the iterative searching steps that are usually required and can be time-consuming in the conventional design methods using optimization 50,56,57 . By solving forward prediction and inverse design tasks with the same model, our AlloyGPT presents unique strength compared to traditional machine learning techniques 58 . 3. Discussions The results above all together have demonstrated that language modeling can provide a unified format to address the forward prediction and inverse design tasks in alloy research. Trained on physics-rich alloy data, our AlloyGPT model can capture the underlying composition-structure-property relationships and accurately predict the microstructural phases and properties at as-built and aged conditions. For inverse design, AlloyGPT model can not only produce accurate designs but also generate diverse compositions. As an initial step towards a language modeling powered material research direction, we expect our model and methodology to be integrated with many engineering applications and continuously improve with the growing dataset, enhanced language model architecture and deepened scientific understanding. To do that, some discussions on its robustness, tunability and expandability are in order. Go beyond the learned domain As a data-driven method, the capability of our AlloyGPT learned is strongly affected by the available data. For example, the data set adopted in the current AlloyGPT model covers a finite hypercube domain in the composition space 50 . For many practical applications, it is important to evaluate how the model performs when being pushed beyond this learned domain. To do so, we intentionally sample compositions beyond the learned domain and perform C-to-SP prediction tasks to evaluate the accuracy. As shown in Figure 5 A, we enlarge the mol% of Yb, Zr and Er by one-fold and randomly sample compositions from this enlarged region. Since our model has not been trained on any data from this domain, we borrow from biology and refer to these compositions as de novo compositions. We define the L2 distance of the sampled composition to the nearest boundary the learned domain as the mutation distance, d M . As the mutation distance increases, we observe the variance of the relative L1 errors of the predicted properties gradually grow ( Figure 5 B), indicating that the model loses the predicting capability when moving away from the learned domain. However, it should be noted that the variance increases in a gradual and stable manner. Therefore, in the thin neighborhood of the learned domain (e.g., d M < 0.5), the model is expected to still behave relatively well without dramatic increases in error. It is recommended to use caution when making predictions beyond this region. In the long term, retraining the model with a growing dataset for an enlarged composition space is expected to better address this issue. Tune design accuracy and composition diversity Contracting to the conventional deterministic models, AlloyGPT predicts the probability distribution for the next token based on the previous text and then samples it under this distribution 52 . Therefore, depending on the shape of the probability distribution, different outcomes in the token level can be observed when repeating the generation process with the same input or prompt. At the same time, this probability distribution can be rescaled by diving it with a sampling temperature parameter, which is termed as prediction temperature, T p , in our AlloyGPT model. A T p larger than 1 tends to flatten the probability distribution and boost diverse predictions. As shown in Figure S3 A, repeating the same generation tasks can lead slight different responses (rows 2 and 3); By increasing T p to a higher value of 3.0, the generated sentence fails to follow the grammar rules of the alloy specific language and becomes unusable for quantity extraction. Given those observations, it is important to understand the stability of AlloyGPT’s predictions and how to potentially leverage it for design purposes. We randomly sample 200 data entries from the test set to perform P-to-SC design tasks. For each input prompt, we repeatedly generate 20 designs at the same prediction temperature. This procedure is then performed for a series of prediction temperatures ranging from 0.001 to 3. It is observed that a prediction temperature, , can produce readable outputs with a high probability () and is recommended for real applications. For designs under such conditions, we analyze the diversity of the suggested compositions as well as the accuracy of the achieved properties using their coefficient of variation, CV , and relative L1 error, , respectively (the definition can be found in Materials and Method section). To investigate the effect of prediction temperature, we grouped the results for different design targets at the same prediction temperature and show the mean values in Figure 6 . It is demonstrated that, as the prediction temperature increases, the mean CV values for all alloying elements generally increase ( Figure 6 A), indicating boosted diversity of the composition designed. At the same time, the mean values of for the achieved properties also increase ( Figure 6 B), suggesting the design accuracy decreases with the prediction temperature. It should be noted that, for many properties (excepting crack susceptibility coefficient and hot cracking susceptibility), the normalized error increases with the prediction temperature with a small and stable slope. These trends together indicate that AlloyGPT can generate diverse designs by its probabilistic nature. On one hand, with a suitable predicting temperature (e.g., T p =1.0 adopted in Figure 4 ), various compositions can be suggested to achieve accurate design. On the other hand, the prediction temperature can be used to intentionally boost the “creativity” of the model and tune the balance between composition diversity and design accuracy in a controllable manner. Expand for other alloy and materials systems Our current protype AlloyGPT model has been trained on Al-based alloys. Given the generality of the alloy-specific language, more numerical data of relevant physics and other alloys systems can be easily reformatted as new text corps with no or little updates. GPT architecture and models similar to our AlloyGPT have demonstrated promising scaling law 59,60 in learning from more training data and achieving better performances. Combining these two strengths, we expect our AlloyGPT model to be further improved by training with enlarged text corpora for different alloys and materials. For example, it would be important to include data on processing conditions 61,62 for AM alloy design. Not only numerical data but also text statements or descriptions 56 can be integrated at the same time in this language modelling process. This mixture may help to take advantage as much as possible of the available data in different formats and with various fidelity. Going beyond Al-based alloy, it will be interesting to let the AlloyGPT model learn multiple alloy systems and investigate its extrapolation capabilities in prediction or design for novel hybrids of alloys as well as alloys with spatial gradients 63 , thus building towards a foundation model for alloys We take these as promising directions in language modeling-based material research and design. 4. Conclusions In this study, we introduce AlloyGPT, an autoregressive language model specifically tailored to alloy design and prediction tasks. By encoding comprehensive alloy data into an alloy-specific language, our approach bridges the gap between traditional numerical modeling and modern generative AI techniques. AlloyGPT successfully learns the intricate C-S-P relationships within alloys, demonstrating high predictive accuracy and robust inverse design capabilities for Al-based alloys. Beyond its ability to discover known compositions, the model excels in generating diverse alloy designs that achieve targeted properties with high accuracy, effectively addressing the degeneracy challenge in inverse design. Our results highlight the versatility of AlloyGPT in navigating the vast compositional design space of additively manufacturable alloys, with gradual degradation in performance observed only when operating outside the learned domain. The probabilistic nature of the model allows multiple composition designs to achieve the given design goal. And through parameters such as prediction temperature, the “creativity” of the model can be further boosted by tuning the balance between design diversity and accuracy, thus providing a flexible tool for alloy discovery. These findings underscore the potential of language modeling as a unified framework for both forward prediction and inverse design in materials science. This work establishes a foundation for integrating generative AI into alloy and material design. Future efforts may focus on expanding the training datasets to include diverse alloy systems and processing parameters, enhancing model architectures, and integrating textual and numerical data for a more comprehensive understanding. By leveraging the scalability of language models and the generalizability of the alloy-specific language, AlloyGPT is expected to serve as a cornerstone in the development of next-generation alloys, accelerating innovation and reducing resource-intensive experimental workflows. 5. Materials and Methods The Al-based alloy database As a prototype case study, in this work, we adopt a database on Al-based alloys with five alloying elements, including Ni (0 ~ 4%), Er (0 ~ 2%), Zr (0 ~ 2%), Y (0 ~ 1%) and Yb (0 ~ 1%), based on CALPHAD simulations using Thermo-Calc 64 . In a previous study 50 , this database has been used to design printable Al alloys with high strength, which have been validated via experiments. Focusing on AM applications, we document microstructures and properties for both as-built and fully aged conditions. The key microstructure features include mol% of L1 2 phase, ternary phase, Al 3 Ni phase and Al 3 Zr phase, while the properties cover diffusion resistivity, misfit, coarsening rate metric, freezing range, crack susceptibility coefficient (CSC) and hot cracking susceptibility (HCS). For these physical quantities with nontrivial units, we normalize their values with respect to those of the benchmark alloy, i.e., “Alloy 1” in Ref 50 . The distributions of these quantities are shown in Figure S1 . More details can be found in Ref 50 . AlloyGPT model and training AlloyGPT includes 36 layers of multi-head attention layers and ~ 400 M parameters. The full structure of the model is shown in Figure S2A . We augment a character-level tokenizer to include words for all elements and signs used in the alloy-specific language. The alloy-specific language dataset includes sentences for both forward prediction and inverse designs. We divide it randomly into a training set (90%) and a test set (10%). We train the AlloyGPT model from scratch on the training set for 4 epochs with an AdamW optimizer 65 using a NVIDIA A40 GPU. No overfitting has been observed ( Figure S2B ). We developed our code based on nanoGPT 66 . Design accuracy and diversity evaluation We use relative L1 errors to measure the design accuracy for the properties. $$\:{L}_{1}^{rela}\left[x,\:y\right]=\frac{\left|y-x\right|}{\left|x\right|}$$ 1 where x is the ground truth or input value, and y is the achieved value based on the designed composition. For the repeated designs with the same input target and a fixed prediction temperature, we use the coefficient of variance to measure the diversity of the suggested composition. $$\:CV=\frac{\sigma\:}{\mu\:}$$ 2 where \(\:\sigma\:\) is the standard deviation and \(\:\mu\:\) is the mean of the designed mol% of the individual elements. Software versions and hardware We use Python 3.10.13, PyTorch 2.2.0 + cu121with CUDA (CUDA version 12.1) 67 , and an NVIDIA A40 with 48 GB VRAM for training and inference. Declarations Conflict of interest The authors declare no conflict of interest. Data and materials availability Source code and script examples, for training and inference, are available on GitHub https://github.com/Taheri-Mousavi-Laboratory/AlloyGPT . The training dataset is generated using ThermoCalc. Due to the restrictions of ThermoCalc End User License Agreement 68 , the dataset and the trained model will only be made available based on reasonable requests. Author contributions: Conceptualization: M.T.-M. and B.N. Investigation: B.N. and M.T.-M. Methodology: B.N. and M.T.-M. Resources: M.T.-M. Funding acquisition: M.T.-M. Data curation: M.T.-M., B.N. and B.G. Validation: M.T.-M. and B.N. Supervision: M.T.-M. Formal analysis: B.N. and M.T.-M. Software: B.N. and M.T.-M Project administration: M.T.-M. Visualization: B.N. Writing—original draft: B.N. Writing—review and editing: B.N., M.T.-M. and B.G. Acknowledgement We acknowledge support from Naval Nuclear Laboratory (NNL) award No. 1047622. This research was conducted using the Tartan Research Advanced Computing Environment (TRACE). The authors would like to gratefully acknowledge the College of Engineering at Carnegie Mellon University for making this shared high-performance computing resource available to its community. References Gialanella, S. & Malandruccolo, A. Aerospace Alloys . (Springer International Publishing, Cham, 2020). doi:10.1007/978-3-030-24440-8. Li, S. et al. Development and applications of aluminum alloys for aerospace industry. J. Mater. Res. Technol. 27 , 944–983 (2023). Bai, J. et al. Applications of magnesium alloys for aerospace: A review. J. Magnes. Alloys 11 , 3609–3619 (2023). Czerwinski, F. Current Trends in Automotive Lightweighting Strategies and Materials. Materials 14 , 6631 (2021). Liu, B. et al. Development and application of magnesium alloy parts for automotive OEMs: A review. J. Magnes. Alloys 11 , 15–47 (2023). Li, Y. et al. Al Alloys and Casting Processes for Induction Motor Applications in Battery-Powered Electric Vehicles: A Review. Metals 12 , 216 (2022). Cann, J. L. et al. Sustainability through alloy design: Challenges and opportunities. Prog. Mater. Sci. 117 , 100722 (2021). Raabe, D. The Materials Science behind Sustainable Metals and Alloys. Chem. Rev. 123 , 2436–2608 (2023). Wilms, M. B., Rittinghaus, S.-K., Goßling, M. & Gökce, B. Additive manufacturing of oxide-dispersion strengthened alloys: Materials, synthesis and manufacturing. Prog. Mater. Sci. 133 , 101049 (2023). Han, S. Z., Choi, E.-A., Lim, S. H., Kim, S. & Lee, J. Alloy design strategies to increase strength and its trade-offs together. Prog. Mater. Sci. 117 , 100720 (2021). Kang, L. & Yang, C. A Review on High-Strength Titanium Alloys: Microstructure, Strengthening, and Properties. Adv. Eng. Mater. 21 , 1801359 (2019). Li, J. et al. Progress on improving strength-toughness of ultra-high strength martensitic steels for aerospace applications: a review. J. Mater. Res. Technol. 23 , 172–190 (2023). Jin, Z.-Z. et al. Alloying design and microstructural control strategies towards developing Mg alloys with enhanced ductility. J. Magnes. Alloys 10 , 1191–1206 (2022). Darolia, R. Development of strong, oxidation and corrosion resistant nickel-based superalloys: critical review of challenges, progress and prospects. Int. Mater. Rev. 64 , 355–380 (2019). Li, W., Liaw, P. K. & Gao, Y. Fracture resistance of high entropy alloys: A review. Intermetallics 99 , 69–83 (2018). George, E. P., Raabe, D. & Ritchie, R. O. High-entropy alloys. Nat. Rev. Mater. 4 , 515–534 (2019). Yeh, J.-W. et al. Nanostructured High-Entropy Alloys with Multiple Principal Elements: Novel Alloy Design Concepts and Outcomes. Adv. Eng. Mater. 6 , 299–303 (2004). Birbilis, N., Choudhary, S., Scully, J. R. & Taheri, M. L. A perspective on corrosion of multi-principal element alloys. Npj Mater. Degrad. 5 , 14 (2021). Miracle, D. B. & Senkov, O. N. A critical review of high entropy alloys and related concepts. Acta Mater. 122 , 448–511 (2017). Armstrong, M., Mehrabi, H. & Naveed, N. An overview of modern metal additive manufacturing technology. J. Manuf. Process. 84 , 1001–1029 (2022). Gardner, L. Metal additive manufacturing in structural engineering – review, advances, opportunities and outlook. Structures 47 , 2178–2193 (2023). Thomas-Seale, L. E. J., Kirkman-Brown, J. C., Attallah, M. M., Espino, D. M. & Shepherd, D. E. T. The barriers to the progression of additive manufacture: Perspectives from UK industry. Int. J. Prod. Econ. 198 , 104–118 (2018). Baumers, M., Dickens, P., Tuck, C. & Hague, R. The cost of additive manufacturing: machine productivity, economies of scale and technology-push. Technol. Forecast. Soc. Change 102 , 193–201 (2016). OpenAI et al. GPT-4 Technical Report. Preprint at https://doi.org/10.48550/arXiv.2303.08774 (2024). Touvron, H. et al. LLaMA: Open and Efficient Foundation Language Models. Preprint at https://doi.org/10.48550/arXiv.2302.13971 (2023). Baktash, J. A. & Dawodi, M. Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing. Preprint at https://doi.org/10.48550/arXiv.2305.03195 (2023). Bubeck, S. et al. Sparks of Artificial General Intelligence: Early experiments with GPT-4. Preprint at https://doi.org/10.48550/arXiv.2303.12712 (2023). Kashefi, A. & Mukerji, T. ChatGPT FOR PROGRAMMING NUMERICAL METHODS. J. Mach. Learn. Model. Comput. 4 , (2023). Rozière, B. et al. Code Llama: Open Foundation Models for Code. Preprint at https://doi.org/10.48550/arXiv.2308.12950 (2024). Ni, B. & Buehler, M. J. MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge. Extreme Mech. Lett. 67 , 102131 (2024). Jignasu, A. et al. Towards Foundational AI Models for Additive Manufacturing: Language Models for G-Code Debugging, Manipulation, and Comprehension. Preprint at https://doi.org/10.48550/arXiv.2309.02465 (2023). Buehler, M. J. MechGPT, a Language-Based Strategy for Mechanics and Materials Modeling That Connects Knowledge Across Scales, Disciplines, and Modalities. Appl. Mech. Rev. 76 , (2024). Jumper, J. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596 , 583–589 (2021). Varadi, M. et al. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Res. 50 , D439–D444 (2022). Abramson, J. et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature 630 , 493–500 (2024). Baek, M. et al. Accurate prediction of protein structures and interactions using a three-track neural network. Science 373 , 871–876 (2021). Krishna, R. et al. Generalized biomolecular modeling and design with RoseTTAFold All-Atom. Science 384 , eadl2528 (2024). Ni, B., Kaplan, D. L. & Buehler, M. J. Generative design of de novo proteins based on secondary-structure constraints using an attention-based diffusion model. Chem 9 , 1828–1849 (2023). Ni, B., Kaplan, D. L. & Buehler, M. J. ForceGen: End-to-end de novo protein generation based on nonlinear mechanical unfolding responses using a language diffusion model. Sci. Adv. 10 , eadl4000 (2024). Ni, B. & Buehler, M. J. Agentic End-to-End De Novo Protein Design for Tailored Dynamics Using a Language Diffusion Model. Preprint at https://doi.org/10.48550/arXiv.2502.10173 (2025). Tong, X. et al. Generative Models for De Novo Drug Design. J. Med. Chem. 64 , 14011–14027 (2021). Sanchez-Lengeling, B. & Aspuru-Guzik, A. Inverse molecular design using machine learning: Generative models for matter engineering. Science 361 , 360–365 (2018). Tran, H. et al. Design of functional and sustainable polymers assisted by artificial intelligence. Nat. Rev. Mater. 9 , 866–886 (2024). Gurnani, R. et al. polyG2G: A Novel Machine Learning Algorithm Applied to the Generative Design of Polymer Dielectrics. Chem. Mater. 33 , 7008–7016 (2021). Park, H., Li, Z. & Walsh, A. Has generative artificial intelligence solved inverse materials design? Matter 7 , 2355–2367 (2024). Chen, Y. et al. MatterGPT: A Generative Transformer for Multi-Property Inverse Design of Solid-State Materials. Preprint at https://doi.org/10.48550/arXiv.2408.07608 (2024). Roy, A. et al. Rapid discovery of high hardness multi-principal-element alloys using a generative adversarial network model. Acta Mater. 257 , 119177 (2023). Li, Z., Li, S. & Birbilis, N. A machine learning-driven framework for the property prediction and generative design of multiple principal element alloys. Mater. Today Commun. 38 , 107940 (2024). Vaswani, A. et al. Attention is All you Need. in Advances in Neural Information Processing Systems vol. 30 (Curran Associates, Inc., 2017). Taheri-Mousavi, S. M. et al. Additively manufacturable high-strength aluminum alloys with thermally stable microstructures enabled by hybrid machine learning-based design. Preprint at https://doi.org/10.48550/arXiv.2406.17457 (2024). TAHERI-MOUSAVI, S. M., HART, A. & Olson, G. Methodologies for formulating compositions, including aluminum alloys with high-temperature strength. (2024). Radford, A. et al. Language models are unsupervised multitask learners. OpenAI Blog 1 , 9 (2019). Byte-Pair Encoding tokenization - Hugging Face NLP Course. https://huggingface.co/learn/nlp-course/en/chapter6/5. Schuurmans, D., Dai, H. & Zanini, F. Autoregressive Large Language Models are Computationally Universal. Preprint at https://doi.org/10.48550/arXiv.2410.03170 (2024). Yan, X. & Lin, J. C. Prediction of hot tearing tendency for multicomponent aluminum alloys. Metall. Mater. Trans. B 37 , 913–918 (2006). Chaudhari, A., Guntuboina, C., Huang, H. & Farimani, A. B. AlloyBERT: Alloy Property Prediction with Large Language Models. Preprint at http://arxiv.org/abs/2403.19783 (2024). Khatamsaz, D. et al. Bayesian optimization with active learning of design constraints using an entropy-based approach. Npj Comput. Mater. 9 , 1–14 (2023). Debnath, A. et al. Comparing forward and inverse design paradigms: A case study on refractory high-entropy alloys. J. Mater. Res. 38 , 4107–4117 (2023). [2001.08361] Scaling Laws for Neural Language Models. https://arxiv.org/abs/2001.08361. [2404.05405] Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. https://arxiv.org/abs/2404.05405. Sames, W. J., List, F. A., Pannala, S., Dehoff, R. R. & Babu, S. S. The metallurgy and processing science of metal additive manufacturing. Int. Mater. Rev. 61 , 315–360 (2016). Brennan, M. C., Keist, J. S. & Palmer, T. A. Defects in Metal Additive Manufacturing Processes. J. Mater. Eng. Perform. 30 , 4808–4818 (2021). Dan, X. et al. Exceptional strength and ductility in heterogeneous multi-gradient TiAl alloys through additive manufacturing. Acta Mater. 281 , 120395 (2024). Andersson, J.-O., Helander, T., Höglund, L., Shi, P. & Sundman, B. Thermo-Calc & DICTRA, computational tools for materials science. Calphad 26 , 273–312 (2002). Loshchilov, I. & Hutter, F. Decoupled Weight Decay Regularization. Preprint at https://doi.org/10.48550/arXiv.1711.05101 (2019). Andrej. karpathy/nanoGPT. (2024). Paszke, A. et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Preprint at https://doi.org/10.48550/arXiv.1912.01703 (2019). Terms and Conditions for Purchase. Thermo-Calc Software https://thermocalc.com/terms-and-conditions-for-the-purchase/. Additional Declarations No competing interests reported. Supplementary Files SI.docx Cite Share Download PDF Status: Published Journal Publication published 26 Sep, 2025 Read the published version in npj Computational Materials → Version 1 posted Editorial decision: Revision requested 14 Mar, 2025 Reviews received at journal 14 Mar, 2025 Reviews received at journal 11 Mar, 2025 Reviews received at journal 03 Mar, 2025 Reviewers agreed at journal 24 Feb, 2025 Reviewers agreed at journal 24 Feb, 2025 Reviewers agreed at journal 23 Feb, 2025 Reviewers invited by journal 23 Feb, 2025 Editor assigned by journal 23 Feb, 2025 Submission checks completed at journal 20 Feb, 2025 First submitted to journal 19 Feb, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6067058","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":424720730,"identity":"f69108ca-eda3-40b9-b852-fe565b588a83","order_by":0,"name":"Bo Ni","email":"","orcid":"","institution":"Carnegie Mellon University","correspondingAuthor":false,"prefix":"","firstName":"Bo","middleName":"","lastName":"Ni","suffix":""},{"id":424720731,"identity":"d7589273-8146-4980-b710-dc325faf356b","order_by":1,"name":"Benjamin Glaser","email":"","orcid":"","institution":"Carnegie Mellon University","correspondingAuthor":false,"prefix":"","firstName":"Benjamin","middleName":"","lastName":"Glaser","suffix":""},{"id":424720732,"identity":"562370f5-ceb9-4d4c-b040-04d88e4c4c40","order_by":2,"name":"S. Mohadeseh Taheri-Mousavi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7ElEQVRIiWNgGAWjYJACZiDmgbIl5CA0G34NKFqMidYCB4kNhLTITzt/8HNBxT0Z3fYe4w8/2yzS+6edMWD4UHYYpxaD28nM0jPOFPOYnTljJtnbJpE743aOAeOMc3i0SCczSPO2JfCY3cgxY+A5I5G7QTrHgJm3DbcW+dnJzL95/4G1GH/8c0Yi3QCk5S8eLQy3k9mkeRvAWgykeSokEsBaGPFoAfrFzJrnGFDLmWNl0jIVEoYzbqcVHOw5l47HYYmPb/PUJNibHW/e/PGNQZ08/+zkjQ9+lFnjdhhWcIBE9aNgFIyCUTAK0AAALoVOaT59TJsAAAAASUVORK5CYII=","orcid":"","institution":"Carnegie Mellon University","correspondingAuthor":true,"prefix":"","firstName":"S.","middleName":"Mohadeseh","lastName":"Taheri-Mousavi","suffix":""}],"badges":[],"createdAt":"2025-02-19 21:38:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6067058/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6067058/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41524-025-01768-2","type":"published","date":"2025-09-26T15:56:59+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":77947810,"identity":"bc55da7f-71c9-4404-b133-3f47f20a87b2","added_by":"auto","created_at":"2025-03-07 06:52:05","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":2042297,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eConvert physics-rich data as sentences of an alloy-specific language. \u003c/strong\u003e(A) Curate scientific data on alloys from various sources, including simulations, experiments and theories. (B) Example entries in the Al-based alloy dataset include quantitative information on the compositions (red), structures (green) and properties (blue) of the alloy samples. (C-D) Reformat the example data entry as a 1D sequence of information blocks, which is referred as a “sentence” of this alloy-specific language. Inside each block, the physical quantity is presented as a pair of key and value. (C) For forward prediction, the sentence is stated in the order of task type, composition, structure and property. (D) For inverse design, the sentence follows the order of task type, property, structure and composition.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-6067058/v1/474e4f0a16cf779294434b52.png"},{"id":77947818,"identity":"eab04eb3-c5d2-47ac-8064-a25874381c58","added_by":"auto","created_at":"2025-03-07 06:52:06","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":1623693,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eSolve forward prediction tasks for the composition-structure-property relationship in alloys using AlloyGPT. \u003c/strong\u003e(A) Pose the alloy predicting problem as a language task of sentence completion and solve it using AlloyGPT. (B-C) Compare the predicted values and the ground truth for example physical quantities from the structure block (B) and the property block (C) using the test set. Better accuracy is indicated by a larger R2, smaller mean squared error (MSE) and smaller mean absolute error (MAE). (D) Compare the predicting accuracy of the property block using different input prompt length in the C-to-SP and CS-to-P tasks.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-6067058/v1/25d4d20a480712683655acde.png"},{"id":77947811,"identity":"42018774-2f2f-4c46-b759-62f09fc38a13","added_by":"auto","created_at":"2025-03-07 06:52:05","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":2857809,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eGenerate alloy compositions based on property and structure information using AlloyGPT. \u003c/strong\u003e(A) R2 values between the known physical quantities and the suggested ones for different inverse design tasks. When both property and structure information are provided as the input (i.e., PS-to-C tasks), higher R2 values are observed (red bars), and (C) the compositions suggested by AlloyGPT are similar to the known cases in the test set. However, when only property target is provided in a P-to-SC task, the suggested compositions can be very different from the known ones (B). (B-C) Compare Y mol% suggested by AlloyGPT model and the known values from the test set for the P-to-SC (B) and the PS-to-C (C) design tasks respectively.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-6067058/v1/057d71c4916c9d4bc67f8053.png"},{"id":77948696,"identity":"4a24eaeb-e11e-48c1-a539-1c6f25137716","added_by":"auto","created_at":"2025-03-07 07:00:06","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":2201952,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eEvaluate the design accuracy of AlloyGPT model in the P-to-SC design tasks. \u003c/strong\u003e(A) Distinguish composition recoverability and design accuracy for alloy design tasks. We clarify the similarity between the suggested composition and the known one from the test set as composition recoverability, not design accuracy. Instead, design accuracy is evaluated by comparing the design goals and the delivered performances, i.e., the prescribed properties and the achieved properties for P-to-SC tasks. (B) Design accuracy and composition recoverability of AlloyGPT for the P-to-SC task. R2 values of the targeted properties and achieved properties (blue bars) are high and indicate good design accuracy. While the R2 values of the known composition and the designed ones (red bars) are low, suggesting low composition recoverability with respect to the test set. (C-D) Comparison of the targeted values and the achieved values for some example properties, including coarsening metric (C) and hot cracking susceptibility (D).\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-6067058/v1/aa59d1ef5baf0fd81306ee10.png"},{"id":77947819,"identity":"e6a24b0b-8d31-4b92-bacc-f5702e136051","added_by":"auto","created_at":"2025-03-07 06:52:07","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":710699,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eTest AlloyGPT beyond the learned domain with the prediction tasks. \u003c/strong\u003e(A) Sample composition points beyond the learned domain as de novo test points and define the mutation distance. (B) Means and variances of the relative L1 error of property predictions for the de novo compositions of a increasing mutation distance.\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-6067058/v1/f815836e6b85b96fb990a1d0.png"},{"id":77947813,"identity":"d6761f9f-ee6f-4589-bdac-94755913194a","added_by":"auto","created_at":"2025-03-07 06:52:05","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":1137811,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eThe effects of prediction temperature on the diversity of compositions (A) and the accuracy of the achieved properties (B) in solving P-to-SC design tasks using AlloyGPT.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-6067058/v1/0adae20c0ca752d75a979080.png"},{"id":92430433,"identity":"ad7c6068-10b6-47c7-be73-689b61a6f777","added_by":"auto","created_at":"2025-09-29 16:03:57","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":12355584,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6067058/v1/10fcee30-dd60-4ec1-9897-c2b3115354ec.pdf"},{"id":77947815,"identity":"22d49216-2b1a-408c-9f39-81a8cbda89bb","added_by":"auto","created_at":"2025-03-07 06:52:05","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":7879319,"visible":true,"origin":"","legend":"","description":"","filename":"SI.docx","url":"https://assets-eu.researchsquare.com/files/rs-6067058/v1/c1126d17c2fbf52300d80193.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"AlloyGPT: End-to-end prediction and design of additively manufacturable alloys using an autoregressive language model","fulltext":[{"header":"Teaser","content":"\u003cp\u003eGenerative AI model learning an alloy-specific language can predict and design additively manufacturable alloys with accuracy, diversity, and robustness.\u003c/p\u003e"},{"header":"1. Introduction","content":"\u003cp\u003eStructural alloys are indispensable in modern engineering, serving critical roles across industries such as aerospace\u003csup\u003e\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e, automotive\u003csup\u003e\u003cspan additionalcitationids=\"CR5\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e, and energy\u003csup\u003e\u003cspan additionalcitationids=\"CR8\" citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e. There is a continuous pursuit for novel alloys with enhanced properties\u003csup\u003e\u003cspan additionalcitationids=\"CR11 CR12 CR13 CR14\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e, including strength\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e,\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e, toughness\u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e,\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e and resistance to failure under various working environments\u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e,\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e. Recent developments in complex alloys, e.g., high entropy alloys\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e,\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e or multi-principal element alloys\u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e,\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e, are actively enlarging the composition palette for alloy design, pushing the boundaries of traditional compositions. Concurrently, rapid advances in additive manufacturing (AM) have unlocked unprecedented control over alloy compositions, microstructures and geometries at voxel-scale resolution while shortening supply chain of manufacturing, thus enabling further tailored solutions for demanding applications\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e,\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e. Together, these advancements are opening an expansive design space that consists of numerous composition choices and manufacturing procedures and hold promises for achieving combinations of superior properties. However, experimental fabrication and testing across the whole design space are extremely costly in resources as well as time-cosuming\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e,\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e. Therefore, efficient yet affordable methods to navigate this broad design space for desired performances are of great interest, not only to advance scientific understanding but also to enable novel engineering applications with next-generation super alloys.\u003c/p\u003e \u003cp\u003eRecent progress in artificial intelligence (AI) has rapidly expanded the horizon of solving scientific problems and engineering applications using a data-driven methodology. Large language models (LLMs), e.g., GPT-4 from openAI\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e and Llama models from Meta\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e, trained on general language corpora have demonstrated intelligent capabilities in perceiving, understanding and applying natural languages\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e,\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e as well as coding languages\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e,\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e, thus assisting human researchers in accelerating and automating workflows in various tasks\u003csup\u003e\u003cspan additionalcitationids=\"CR31\" citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e. For solving scientific challenges, AI-based models, such as Alphafold 2\u003csup\u003e33\u0026ndash;35\u003c/sup\u003e and RoseTTAFold\u003csup\u003e\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e,\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e\u003c/sup\u003e, can predict the atomic structures of proteins based on their sequences, reaching experimental accuracy but at a much-reduced cost\u003csup\u003e\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e\u003c/sup\u003e. On material engineering, end-to-end generative AI (GAI) models have been developed to design novel protein sequences that meet the desired structures or properties\u003csup\u003e\u003cspan additionalcitationids=\"CR39\" citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e\u003c/sup\u003e. Similar explorations have been actively developed for various material systems, including functional molecules\u003csup\u003e\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e,\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e\u003c/sup\u003e, polymers\u003csup\u003e\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e,\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e\u003c/sup\u003e, inorganic crystals\u003csup\u003e\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e,\u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e\u003c/sup\u003e as well as metallic alloys\u003csup\u003e\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e,\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e48\u003c/span\u003e\u003c/sup\u003e. For structural alloys, data on experimentally measured mechanical properties remains relatively sparse due to their time-consuming and costly nature. Since language models have unique strength of integrating data from various sources as general text and further being multi-modal, they are highly interesting specifically in the field of alloy design while in-depth studies remain rare.\u003c/p\u003e \u003cp\u003eIn this work, we explore the possibility of solving the alloy design challenge from a language modeling perspective by leveraging GAI and alloy data. Specifically, we encode physics-rich alloy data as one-dimensional (1D) sequences of text and develop an attention-based\u003csup\u003e\u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e49\u003c/span\u003e\u003c/sup\u003e LLM to capture the underlying patterns in this alloy-specific language. Our model, AlloyGPT, at deployment can accurately predict structures and properties for the given compositions and achieve R2 values ranging from 0.86 to 0.99 on the test set. When tested with compositions beyond the learned domain, we observe the accuracy tends to degrade but in a gradual and stable manner with respect to the distance in the composition space. The same model can also tackle alloy design challenges. For the same given property target, it generates various composition choices while maintaining high design accuracy. Through a sampling parameter, we can further boost the creativity of the model and tune the balance between the diversity of the suggested compositions and the accuracy of the delivered properties. Those results of our numerical experiments demonstrate that language modeling with the LLM specialized for alloys can provide accurate and robust representation of alloy physics. As a probabilistic model by nature, our model is particularly suitable for alloy design tasks with high degeneracy. For example, it can be exploited for designing gradient composition or microstructure in alloys. At each voxel of design, depending on constraints, the possible solutions will be further restricted to lead to higher overall performance and design metrics. These capabilities can be valuable for accelerating alloy discoveries in AM and reducing time and resources costs. Our methodology is expected to be generalizable for new alloys and other materials with broad design spaces, potentially leading to a foundation language model with comprehensive material knowledge and suitable for integrated material design in the future.\u003c/p\u003e"},{"header":"2. Result and discussions","content":"\u003cp\u003e\u003cstrong\u003eAn alloy-specific language dataset and AlloyGPT model\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNatural languages are powerful tools that enable human communications. In scientific communities, research can be conducted in various forms, including experiments, simulations or theories (\u003cstrong\u003eFigure 1\u003c/strong\u003eA). Communication and documentation of the research are dominantly taken in the form of languages so far, either as written text or oral speech. Inspired by this universal flexibility, here we explore the possibility of using language-like 1D sequences to directly represent physics-rich data for alloys. Note that the intended speakers of such languages are LLMs, instead of humans.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWithout loss of generality, in this work we adopt a database of Al-based alloys\u003csup\u003e50\u003c/sup\u003e as an example of scientific data. As shown in \u0026nbsp;\u003cstrong\u003eFigure 1\u003c/strong\u003eB, entries from this database include numerical values for various physical quantities, including the compositions in terms of mol% of Al and the five alloying elements (i.e., Ni, Er, Zr, Y and Yb), structure information of the key phases (L1\u003csub\u003e2\u003c/sub\u003e, ternary, Al\u003csub\u003e3\u003c/sub\u003eNi and Al\u003csub\u003e3\u003c/sub\u003eZr phases) in the as-built and aged conditions, as well as related properties (diffusion resistivity, misfit, coarsening rate metric, freezing range, crack susceptibility coefficient and hot cracking susceptibility). This database is based on CALPHAD (CALculation of PHAse Diagrams) simulations and has been validated in several experiments\u003csup\u003e50,51\u003c/sup\u003e. Detailed information on this dataset can be found in the Materials and Method section.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTo reformat individual entries as \u0026ldquo;sentences\u0026rdquo; of an alloy-specific language, we group information based on underlying physics and order the information blocks based on the nature of the tasks. As shown in\u0026nbsp;\u003cstrong\u003eFigure 1\u003c/strong\u003eB, locally we group the physical quantities as blocks on compositions (in the red dash rectangle), structures (green) and properties (blue) separately, given that compositions lead to specific phases and phase distribution affects the properties. Globally, in the sentence structure, we order those blocks by putting the given information ahead of the queried. For example, for forward prediction, compositions are listed before structures and properties (\u003cstrong\u003eFigure 1\u003c/strong\u003eC, sentence structure). While in inverse design, the sequence starts with properties which are followed by structures and compositions (\u003cstrong\u003eFigure 1\u003c/strong\u003eD, sentence structure). To distinguish different tasks, we add a task block at the beginning of the sentences to label the task name. We also include simple signs to improve the readability of the sentences for human researchers (\u003cstrong\u003eFigure 1\u003c/strong\u003eCD, examples). Curly brackets enclose the individual blocks; square brackets enclose the list of physical quantities inside the block. The physical quantities are documented as individual pairs of \u0026ldquo;Key: Value\u0026rdquo;, where the key is a readable name of the quantity and the numerical value are expressed in a scientific format. To connect the blocks, we use the directional sign, \u0026ldquo;=\u0026gt;\u0026rdquo;, to indicate the reading or inferring direction and equal sign \u0026ldquo;=\u0026rdquo; for equivalence. With these simple rules, we can efficiently convert the numerical dataset of Al-based alloy into the corpus of sentences for different tasks. The dataset of this alloy-specific language on Al-based alloys can be found in the SI. We expect these rules can be straightforwardly generalized to include more information blocks or applied to other alloys or material systems.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIt should be noted that the new alloy-specific language dataset keeps almost all the information of the original numerical dataset. The key difference is that it enables us to view alloy prediction and design challenges as the same sentence completion task. Specifically, given only the beginning blocks, to accurately complete these sentences will be equivalent to predict alloy structures and properties based on the given composition, or to design alloy compositions that fulfill the given properties.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTo handle such language tasks, we develop an attention-based\u003csup\u003e49\u003c/sup\u003e autoregressive LLM, AlloyGPT, and test its performance for both forward prediction and inverse design tasks for alloys. AlloyGPT model adopts an architecture similar to GPT-2\u003csup\u003e52\u003c/sup\u003e with a tokenizer\u003csup\u003e53\u003c/sup\u003e customized for the alloy language. We train AlloyGPT from scratch by using the curated alloy-specific language database. Sentences for both forward prediction and inverse design are both included and randomly mixed in the train batches. More details of the model architecture and training can be found in the Materials and Methods section.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLearn the composition-structure-property (C-S-P) relationships with AlloyGPT\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo investigate whether our AlloyGPT model can learn the underlying composition-structure-property relationship through the alloy-specific language and autoregressive training, here we apply AlloyGPT model to solve forward prediction problems for Al-based alloys. Distinguishing from conventional methods, here we pose the problem as a language task. As shown in\u0026nbsp;\u003cstrong\u003eFigure 2\u003c/strong\u003eA, we provide a text prompt with the information blocks of the task type and the composition as the input. Our model is able to continue the sentence in the correct format of the alloy-specific language and finish it at the right sequence length. This behavior indicates that AlloyGPT can learn the grammar rules via training.\u003c/p\u003e\n\u003cp\u003eQuantitative results then can be extracted from the text. For the case picked in \u003cstrong\u003eFigure 2\u003c/strong\u003eA, the predicted values of the alloy phases and properties are close to the ground truth, thus solving this composition-to-structure and property (C-to-SP) task. Following this procedure, we performed this C-to-SP task for the whole test set and observed consistent high accuracy. Representative results of examples from the structure and property blocks are shown in \u003cstrong\u003eFigure 2\u003c/strong\u003eB (as-built L1\u003csub\u003e2\u003c/sub\u003e phase) and C (hot cracking susceptibility) respectively. Good agreements are observed for both the overall distribution as well as the individual values between the prediction and the ground truth. R2 values larger than 0.95 are observed for as-built L1\u003csub\u003e2\u003c/sub\u003e phase mole fraction and hot cracking susceptibility. The positive results observed here indicate that language modeling can be competitive to conventional numerical methods in capturing the hidden pattern between composition, structures and properties in this Al-based alloy family, thus potentially opening new avenues to alloy research.\u003c/p\u003e\n\u003cp\u003eAs an autoregressive language model\u003csup\u003e54\u003c/sup\u003e, AlloyGPT may process length dependence of its predicting accuracy and present convenient strategies to enhance its performance. Specifically, AlloyGPT completes the given sentences by regressively predicting the next token. Thus, the predicting error can accumulate and influence the accuracy of the content predicted at a later stage. In \u003cstrong\u003eFigure 2\u003c/strong\u003eC, we plot the R2 values (blue bars for C-to-SP tasks) of all the physical quantities in their predicting order from left to right. Our model achieves high accuracy across various physical quantities with R2 values ranging from 0.86 to 0.99 (See Table S1 for details). At the same time, the property block (shaded in blue) predicted later in the output shows overall smaller R2 values compared with the earlier predicted structure block (shaded in green). Therefore, to improve the accuracy of property predictions, one straightforward way could be to increase the input prompt length. For example, providing not only composition but also the structure blocks for property prediction (i.e., a CS-to-P task) can reduce the final prediction length for the language model. As shown in \u003cstrong\u003eFigure 2\u003c/strong\u003eC, indeed, such CS-to-P tasks performed by the same model show improved accuracy with larger R2 values (red bars). This strategy also aligns with a physics-based consideration. Because composition and structures together may better define the alloy and determine its properties. It should also be noted that there could exist other factors that affect the prediction accuracy. For example, by definition, CSC values are determined by the whole solidification process\u003csup\u003e55\u003c/sup\u003e, which makes it relatively challenging to predict. And we observe CSC has the lowest R2 value while it is not predicted at last. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDesign Al-based alloys for target properties using AlloyGPT\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAs a language model of a probabilistic nature, AlloyGPT provides unique advantages in addressing the inverse design challenges for alloys. On one hand, AlloyGPT model unifies the forward predicting and inverse designing tasks under the same language task. From the format point of view, the language modeling task of sentence completion developed for the forward predicting tasks in the previous section can be straightforwardly applied to the inverse design tasks. As shown in\u0026nbsp;\u003cstrong\u003eFigure 3\u003c/strong\u003eA (left panel), the updated input prompt can include only the property information for a property-to-structure and composition (P-to-SC) design task, or both the property and structure blocks for a property and structure-to-composition (PS-to-C) design. Our model is tasked to complete the remaining of the sentence with the composition information included. On the other hand, the probability-based prediction of AlloyGPT model can be leveraged to capture degenerated solutions in alloy design challenges, which will be discussed later in this section.\u003c/p\u003e\n\u003cp\u003eTo evaluate the performance of this unified pathway for solving design challenges, we perform P-to-SC and PS-to-C design tasks based on the test set using AlloyGPT. By setting the properties of the data entries from the test set as the goal, we make sure there exists at least one composition as the solution. The R2 values calculated using the known compositions from the test set and those suggested by our model are shown in\u0026nbsp;\u003cstrong\u003eFigure 3\u003c/strong\u003eA (right penal). When both properties and structures are provided as the input prompt or design goal, in the PS-to-C tasks the suggested compositions achieve high R2 values (red bars in\u0026nbsp;\u003cstrong\u003eFigure 3\u003c/strong\u003eA) (similar to those observed in the previous section for forward predicting tasks) and agrees well with the known compositions from the test set (taking Y in\u0026nbsp;\u003cstrong\u003eFigure 3\u003c/strong\u003eC as the weakest example). This result demonstrates that AlloyGPT can \u0026ldquo;rediscover\u0026rdquo; compositions known to the test set based on conditioning of properties and structures.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eHowever, when only properties are prescribed as the design goal in the P-to-SC tasks, the suggested physical quantities, for both structures and compositions, achieve low R2 values (blue bars in\u0026nbsp;\u003cstrong\u003eFigure 3\u003c/strong\u003eA). As shown in\u0026nbsp;\u003cstrong\u003eFigure 3\u003c/strong\u003eB, the suggested mol% of Y tends to diverge from the known records and spread broadly. Thus, AlloyGPT can also suggest compositions different from the known ones.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIt should be clarified that the R2 values of the compositions in\u0026nbsp;\u003cstrong\u003eFigure 3\u003c/strong\u003eA (right panel, shaded in red) should not be taken as the indicator of design accuracy. By definition, it represents the composition recoverability with respect to the test set, i.e., how similar the proposed compositions are to the known ones from the test set (\u003cstrong\u003eFigure 4\u003c/strong\u003eA). Design accuracy, instead, should be evaluated by comparing the targeted goal and the delivered performance, i.e., the prescribed properties and the achieved values for the P-to-SC design task (\u003cstrong\u003eFigure 4\u003c/strong\u003eA).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTo truly estimate the design accuracy, we first obtain the property and structure results of the designed compositions by applying the same protocol used for the initial dataset creation, and then calculate the R2 values between these and the input design target. As shown in\u0026nbsp;\u003cstrong\u003eFigure 4\u003c/strong\u003eB, high R2 values (blue bars) are obtained between the input property goals and the achieved properties, indicating actually AlloyGPT also achieves high design accuracy for the P-to-SC design tasks. Comparisons of some example properties, including\u0026nbsp;coarsening metric and hot cracking susceptibility, are shown in \u003cstrong\u003eFigure 4\u003c/strong\u003eC and D. Good agreement in both distributions and individual values are observed. At the same time, the low composition recoverability (red bars in\u0026nbsp;\u003cstrong\u003eFigure 4\u003c/strong\u003eB) suggests for the given properties, there could be different compositions as the design solutions. By achieving high design accuracy and low composition recoverability, AlloyGPT has demonstrated promising potential in addressing the challenges of degenerated solutions in inverse design problems and maintaining design accuracy and diversity at the same time.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIt should also be noted that AlloyGPT addresses the inverse design tasks in an efficient end-to-end manner. Thus, it bypasses the iterative searching steps that are usually required and can be time-consuming in the conventional design methods using optimization\u003csup\u003e50,56,57\u003c/sup\u003e. By solving forward prediction and inverse design tasks with the same model, our AlloyGPT presents unique strength compared to traditional machine learning techniques\u003csup\u003e58\u003c/sup\u003e.\u003c/p\u003e"},{"header":"3. Discussions","content":"\u003cp\u003eThe results above all together have demonstrated that language modeling can provide a unified format to address the forward prediction and inverse design tasks in alloy research. Trained on physics-rich alloy data, our AlloyGPT model can capture the underlying composition-structure-property relationships and accurately predict the microstructural phases and properties at as-built and aged conditions. For inverse design, AlloyGPT model can not only produce accurate designs but also generate diverse compositions. As an initial step towards a language modeling powered material research direction, we expect our model and methodology to be integrated with many engineering applications and continuously improve with the growing dataset, enhanced language model architecture and deepened scientific understanding. To do that, some discussions on its robustness, tunability and expandability are in order.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eGo beyond the learned domain\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAs a data-driven method, the capability of our AlloyGPT learned is strongly affected by the available data. For example, the data set adopted in the current AlloyGPT model covers a finite hypercube domain in the composition space\u003csup\u003e50\u003c/sup\u003e. For many practical applications, it is important to evaluate how the model performs when being pushed beyond this learned domain. To do so, we intentionally sample compositions beyond the learned domain and perform C-to-SP prediction tasks to evaluate the accuracy. As shown in \u003cstrong\u003eFigure 5\u003c/strong\u003eA, we enlarge the mol% of Yb, Zr and Er by one-fold and randomly sample compositions from this enlarged region. Since our model has not been trained on any data from this domain, we borrow from biology and refer to these compositions as \u003cem\u003ede novo\u003c/em\u003e compositions. We define the L2 distance of the sampled composition to the nearest boundary the learned domain as the mutation distance, \u003cem\u003ed\u003csub\u003eM\u003c/sub\u003e\u003c/em\u003e\u003csub\u003e.\u003c/sub\u003e As the mutation distance increases, we observe the variance of the relative L1 errors of the predicted properties gradually grow (\u003cstrong\u003eFigure 5\u003c/strong\u003eB), indicating that the model loses the predicting capability when moving away from the learned domain. However, it should be noted that the variance increases in a gradual and stable manner. Therefore, in the thin neighborhood of the learned domain (e.g., \u003cem\u003ed\u003csub\u003eM\u0026nbsp;\u003c/sub\u003e\u003c/em\u003e\u0026lt; 0.5), the model is expected to still behave relatively well without dramatic increases in error. It is recommended to use caution when making predictions beyond this region. In the long term, retraining the model with a growing dataset for an enlarged composition space is expected to better address this issue.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTune design accuracy and composition diversity\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eContracting to the conventional deterministic models, AlloyGPT predicts the probability distribution for the next token based on the previous text and then samples it under this distribution\u003csup\u003e52\u003c/sup\u003e. Therefore, depending on the shape of the probability distribution, different outcomes in the token level can be observed when repeating the generation process with the same input or prompt. At the same time, this probability distribution can be rescaled by diving it with a sampling temperature parameter, which is termed as prediction temperature, T\u003csub\u003ep\u003c/sub\u003e, in our AlloyGPT model. A T\u003csub\u003ep\u003c/sub\u003e larger than 1 tends to flatten the probability distribution and boost diverse predictions. As shown in \u003cstrong\u003eFigure S3\u003c/strong\u003eA, repeating the same generation tasks can lead slight different responses (rows 2 and 3); By increasing T\u003csub\u003ep\u003c/sub\u003e to a higher value of 3.0, the generated sentence fails to follow the grammar rules of the alloy specific language and becomes unusable for quantity extraction.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eGiven those observations, it is important to understand the stability of AlloyGPT\u0026rsquo;s predictions and how to potentially leverage it for design purposes. We randomly sample 200 data entries from the test set to perform P-to-SC design tasks. For each input prompt, we repeatedly generate 20 designs at the same prediction temperature. This procedure is then performed for a series of prediction temperatures ranging from 0.001 to 3. It is observed that a prediction temperature,\u0026nbsp;, can produce readable outputs with a high probability () and is recommended for real applications. For designs under such conditions, we analyze the diversity of the suggested compositions as well as the accuracy of the achieved properties using their coefficient of variation, \u003cem\u003eCV\u003c/em\u003e, and relative L1 error,\u0026nbsp;,\u0026nbsp;respectively (the definition can be found in Materials and Method section). To investigate the effect of prediction temperature, we grouped the results for different design targets at the same prediction temperature and show the mean values in \u003cstrong\u003eFigure 6\u003c/strong\u003e. It is demonstrated that, as the prediction temperature increases, the mean CV values for all alloying elements generally increase (\u003cstrong\u003eFigure 6\u003c/strong\u003eA), indicating boosted diversity of the composition designed. At the same time, the mean values of \u0026nbsp;\u0026nbsp;for the achieved properties also increase\u0026nbsp;(\u003cstrong\u003eFigure 6\u003c/strong\u003eB), suggesting the design accuracy decreases with the prediction temperature. It should be noted that, for many properties (excepting crack susceptibility coefficient and hot cracking susceptibility), the normalized error increases with the prediction temperature with a small and stable slope. These trends together indicate that AlloyGPT can generate diverse designs by its probabilistic nature. On one hand, with a suitable predicting temperature (e.g., T\u003csub\u003ep\u003c/sub\u003e=1.0 adopted in \u003cstrong\u003eFigure 4\u003c/strong\u003e), various compositions can be suggested to achieve accurate design. On the other hand, the prediction temperature can be used to intentionally boost the \u0026ldquo;creativity\u0026rdquo; of the model and tune the balance between composition diversity and design accuracy in a controllable manner.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eExpand for other alloy and materials systems\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOur current protype AlloyGPT model has been trained on Al-based alloys. Given the generality of the alloy-specific language, more numerical data of relevant physics and other alloys systems can be easily reformatted as new text corps with no or little updates. GPT architecture and models similar to our AlloyGPT have demonstrated promising scaling law\u003csup\u003e59,60\u003c/sup\u003e in learning from more training data and achieving better performances. Combining these two strengths, we expect our AlloyGPT model to be further improved by training with enlarged text corpora for different alloys and materials. For example, it would be important to include data on processing conditions\u003csup\u003e61,62\u003c/sup\u003e for AM alloy design. Not only numerical data but also text statements or descriptions\u003csup\u003e56\u003c/sup\u003e can be integrated at the same time in this language modelling process. This mixture may help to take advantage as much as possible of the available data in different formats and with various fidelity. Going beyond Al-based alloy, it will be interesting to let the AlloyGPT model learn multiple alloy systems and investigate its extrapolation capabilities in prediction or design for novel hybrids of alloys as well as alloys with spatial gradients\u003csup\u003e63\u003c/sup\u003e, thus building towards a foundation model for alloys We take these as promising directions in language modeling-based material research and design.\u0026nbsp;\u003c/p\u003e"},{"header":"4. Conclusions","content":"\u003cp\u003eIn this study, we introduce AlloyGPT, an autoregressive language model specifically tailored to alloy design and prediction tasks. By encoding comprehensive alloy data into an alloy-specific language, our approach bridges the gap between traditional numerical modeling and modern generative AI techniques. AlloyGPT successfully learns the intricate C-S-P relationships within alloys, demonstrating high predictive accuracy and robust inverse design capabilities for Al-based alloys. Beyond its ability to discover known compositions, the model excels in generating diverse alloy designs that achieve targeted properties with high accuracy, effectively addressing the degeneracy challenge in inverse design.\u003c/p\u003e \u003cp\u003eOur results highlight the versatility of AlloyGPT in navigating the vast compositional design space of additively manufacturable alloys, with gradual degradation in performance observed only when operating outside the learned domain. The probabilistic nature of the model allows multiple composition designs to achieve the given design goal. And through parameters such as prediction temperature, the \u0026ldquo;creativity\u0026rdquo; of the model can be further boosted by tuning the balance between design diversity and accuracy, thus providing a flexible tool for alloy discovery. These findings underscore the potential of language modeling as a unified framework for both forward prediction and inverse design in materials science.\u003c/p\u003e \u003cp\u003eThis work establishes a foundation for integrating generative AI into alloy and material design. Future efforts may focus on expanding the training datasets to include diverse alloy systems and processing parameters, enhancing model architectures, and integrating textual and numerical data for a more comprehensive understanding. By leveraging the scalability of language models and the generalizability of the alloy-specific language, AlloyGPT is expected to serve as a cornerstone in the development of next-generation alloys, accelerating innovation and reducing resource-intensive experimental workflows.\u003c/p\u003e"},{"header":"5. Materials and Methods","content":"\u003cp\u003e \u003cb\u003eThe Al-based alloy database\u003c/b\u003e \u003c/p\u003e \u003cp\u003eAs a prototype case study, in this work, we adopt a database on Al-based alloys with five alloying elements, including Ni (0\u0026thinsp;~\u0026thinsp;4%), Er (0\u0026thinsp;~\u0026thinsp;2%), Zr (0\u0026thinsp;~\u0026thinsp;2%), Y (0\u0026thinsp;~\u0026thinsp;1%) and Yb (0\u0026thinsp;~\u0026thinsp;1%), based on CALPHAD simulations using Thermo-Calc\u003csup\u003e\u003cspan citationid=\"CR64\" class=\"CitationRef\"\u003e64\u003c/span\u003e\u003c/sup\u003e. In a previous study\u003csup\u003e\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e\u003c/sup\u003e, this database has been used to design printable Al alloys with high strength, which have been validated via experiments. Focusing on AM applications, we document microstructures and properties for both as-built and fully aged conditions. The key microstructure features include mol% of L1\u003csub\u003e2\u003c/sub\u003e phase, ternary phase, Al\u003csub\u003e3\u003c/sub\u003eNi phase and Al\u003csub\u003e3\u003c/sub\u003eZr phase, while the properties cover diffusion resistivity, misfit, coarsening rate metric, freezing range, crack susceptibility coefficient (CSC) and hot cracking susceptibility (HCS). For these physical quantities with nontrivial units, we normalize their values with respect to those of the benchmark alloy, i.e., \u0026ldquo;Alloy 1\u0026rdquo; in Ref \u003csup\u003e\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e\u003c/sup\u003e. The distributions of these quantities are shown in \u003cb\u003eFigure \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e\u003c/b\u003e. More details can be found in Ref \u003csup\u003e\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e \u003cb\u003eAlloyGPT model and training\u003c/b\u003e \u003c/p\u003e \u003cp\u003eAlloyGPT includes 36 layers of multi-head attention layers and ~\u0026thinsp;400 M parameters. The full structure of the model is shown in \u003cb\u003eFigure S2A\u003c/b\u003e. We augment a character-level tokenizer to include words for all elements and signs used in the alloy-specific language. The alloy-specific language dataset includes sentences for both forward prediction and inverse designs. We divide it randomly into a training set (90%) and a test set (10%). We train the AlloyGPT model from scratch on the training set for 4 epochs with an AdamW optimizer\u003csup\u003e\u003cspan citationid=\"CR65\" class=\"CitationRef\"\u003e65\u003c/span\u003e\u003c/sup\u003e using a NVIDIA A40 GPU. No overfitting has been observed (\u003cb\u003eFigure S2B\u003c/b\u003e). We developed our code based on nanoGPT\u003csup\u003e\u003cspan citationid=\"CR66\" class=\"CitationRef\"\u003e66\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDesign accuracy and diversity evaluation\u003c/b\u003e \u003c/p\u003e \u003cp\u003eWe use relative L1 errors to measure the design accuracy for the properties.\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$$\\:{L}_{1}^{rela}\\left[x,\\:y\\right]=\\frac{\\left|y-x\\right|}{\\left|x\\right|}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cem\u003ex\u003c/em\u003e is the ground truth or input value, and \u003cem\u003ey\u003c/em\u003e is the achieved value based on the designed composition.\u003c/p\u003e \u003cp\u003eFor the repeated designs with the same input target and a fixed prediction temperature, we use the coefficient of variance to measure the diversity of the suggested composition.\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$$\\:CV=\\frac{\\sigma\\:}{\\mu\\:}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\sigma\\:\\)\u003c/span\u003e\u003c/span\u003e is the standard deviation and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\mu\\:\\)\u003c/span\u003e\u003c/span\u003e is the mean of the designed mol% of the individual elements.\u003c/p\u003e \u003cp\u003e \u003cb\u003eSoftware versions and hardware\u003c/b\u003e \u003c/p\u003e \u003cp\u003eWe use Python 3.10.13, PyTorch 2.2.0\u0026thinsp;+\u0026thinsp;cu121with CUDA (CUDA version 12.1)\u003csup\u003e\u003cspan citationid=\"CR67\" class=\"CitationRef\"\u003e67\u003c/span\u003e\u003c/sup\u003e, and an NVIDIA A40 with 48 GB VRAM for training and inference.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eConflict of interest\u003c/h2\u003e\n\u003cp\u003eThe authors declare no conflict of interest.\u003c/p\u003e\n\u003ch2\u003eData and materials availability\u003c/h2\u003e\n\u003cp\u003eSource code and script examples, for training and inference, are available on GitHub \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/Taheri-Mousavi-Laboratory/AlloyGPT\u003c/span\u003e\u003c/span\u003e. The training dataset is generated using ThermoCalc. Due to the restrictions of ThermoCalc End User License Agreement\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e68\u003c/span\u003e\u003c/sup\u003e, the dataset and the trained model will only be made available based on reasonable requests.\u003c/p\u003e\n\u003ch2\u003eAuthor contributions:\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eConceptualization: M.T.-M. and B.N.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eInvestigation: B.N. and M.T.-M.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eMethodology: B.N. and M.T.-M.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eResources: M.T.-M.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFunding acquisition: M.T.-M.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eData curation: M.T.-M., B.N. and B.G.\u003c/p\u003e\n\u003cp\u003eValidation: M.T.-M. and B.N.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eSupervision: M.T.-M.\u003c/p\u003e\n\u003cp\u003eFormal analysis: B.N. and M.T.-M.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eSoftware: B.N. and M.T.-M\u003c/p\u003e\n\u003cp\u003eProject administration: M.T.-M.\u003c/p\u003e\n\u003cp\u003eVisualization: B.N.\u003c/p\u003e\n\u003cp\u003eWriting\u0026mdash;original draft: B.N.\u003c/p\u003e\n\u003cp\u003eWriting\u0026mdash;review and editing: B.N., M.T.-M. and B.G.\u003c/p\u003e\n\u003ch2\u003eAcknowledgement\u003c/h2\u003e\n\u003cp\u003eWe acknowledge support from Naval Nuclear Laboratory (NNL) award No. 1047622. This research was conducted using the Tartan Research Advanced Computing Environment (TRACE). The authors would like to gratefully acknowledge the College of Engineering at Carnegie Mellon University for making this shared high-performance computing resource available to its community.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eGialanella, S. \u0026amp; Malandruccolo, A. \u003cem\u003eAerospace Alloys\u003c/em\u003e. (Springer International Publishing, Cham, 2020). doi:10.1007/978-3-030-24440-8.\u003c/li\u003e\n\u003cli\u003eLi, S. \u003cem\u003eet al.\u003c/em\u003e Development and applications of aluminum alloys for aerospace industry. \u003cem\u003eJ. Mater. Res. Technol.\u003c/em\u003e \u003cstrong\u003e27\u003c/strong\u003e, 944\u0026ndash;983 (2023).\u003c/li\u003e\n\u003cli\u003eBai, J. \u003cem\u003eet al.\u003c/em\u003e Applications of magnesium alloys for aerospace: A review. \u003cem\u003eJ. Magnes. Alloys\u003c/em\u003e \u003cstrong\u003e11\u003c/strong\u003e, 3609\u0026ndash;3619 (2023).\u003c/li\u003e\n\u003cli\u003eCzerwinski, F. Current Trends in Automotive Lightweighting Strategies and Materials. \u003cem\u003eMaterials\u003c/em\u003e \u003cstrong\u003e14\u003c/strong\u003e, 6631 (2021).\u003c/li\u003e\n\u003cli\u003eLiu, B. \u003cem\u003eet al.\u003c/em\u003e Development and application of magnesium alloy parts for automotive OEMs: A review. \u003cem\u003eJ. Magnes. Alloys\u003c/em\u003e \u003cstrong\u003e11\u003c/strong\u003e, 15\u0026ndash;47 (2023).\u003c/li\u003e\n\u003cli\u003eLi, Y. \u003cem\u003eet al.\u003c/em\u003e Al Alloys and Casting Processes for Induction Motor Applications in Battery-Powered Electric Vehicles: A Review. \u003cem\u003eMetals\u003c/em\u003e \u003cstrong\u003e12\u003c/strong\u003e, 216 (2022).\u003c/li\u003e\n\u003cli\u003eCann, J. L. \u003cem\u003eet al.\u003c/em\u003e Sustainability through alloy design: Challenges and opportunities. \u003cem\u003eProg. Mater. Sci.\u003c/em\u003e \u003cstrong\u003e117\u003c/strong\u003e, 100722 (2021).\u003c/li\u003e\n\u003cli\u003eRaabe, D. The Materials Science behind Sustainable Metals and Alloys. \u003cem\u003eChem. Rev.\u003c/em\u003e \u003cstrong\u003e123\u003c/strong\u003e, 2436\u0026ndash;2608 (2023).\u003c/li\u003e\n\u003cli\u003eWilms, M. B., Rittinghaus, S.-K., Go\u0026szlig;ling, M. \u0026amp; G\u0026ouml;kce, B. Additive manufacturing of oxide-dispersion strengthened alloys: Materials, synthesis and manufacturing. \u003cem\u003eProg. Mater. Sci.\u003c/em\u003e \u003cstrong\u003e133\u003c/strong\u003e, 101049 (2023).\u003c/li\u003e\n\u003cli\u003eHan, S. Z., Choi, E.-A., Lim, S. H., Kim, S. \u0026amp; Lee, J. Alloy design strategies to increase strength and its trade-offs together. \u003cem\u003eProg. Mater. Sci.\u003c/em\u003e \u003cstrong\u003e117\u003c/strong\u003e, 100720 (2021).\u003c/li\u003e\n\u003cli\u003eKang, L. \u0026amp; Yang, C. A Review on High-Strength Titanium Alloys: Microstructure, Strengthening, and Properties. \u003cem\u003eAdv. Eng. Mater.\u003c/em\u003e \u003cstrong\u003e21\u003c/strong\u003e, 1801359 (2019).\u003c/li\u003e\n\u003cli\u003eLi, J. \u003cem\u003eet al.\u003c/em\u003e Progress on improving strength-toughness of ultra-high strength martensitic steels for aerospace applications: a review. \u003cem\u003eJ. Mater. Res. Technol.\u003c/em\u003e \u003cstrong\u003e23\u003c/strong\u003e, 172\u0026ndash;190 (2023).\u003c/li\u003e\n\u003cli\u003eJin, Z.-Z. \u003cem\u003eet al.\u003c/em\u003e Alloying design and microstructural control strategies towards developing Mg alloys with enhanced ductility. \u003cem\u003eJ. Magnes. Alloys\u003c/em\u003e \u003cstrong\u003e10\u003c/strong\u003e, 1191\u0026ndash;1206 (2022).\u003c/li\u003e\n\u003cli\u003eDarolia, R. Development of strong, oxidation and corrosion resistant nickel-based superalloys: critical review of challenges, progress and prospects. \u003cem\u003eInt. Mater. Rev.\u003c/em\u003e \u003cstrong\u003e64\u003c/strong\u003e, 355\u0026ndash;380 (2019).\u003c/li\u003e\n\u003cli\u003eLi, W., Liaw, P. K. \u0026amp; Gao, Y. Fracture resistance of high entropy alloys: A review. \u003cem\u003eIntermetallics\u003c/em\u003e \u003cstrong\u003e99\u003c/strong\u003e, 69\u0026ndash;83 (2018).\u003c/li\u003e\n\u003cli\u003eGeorge, E. P., Raabe, D. \u0026amp; Ritchie, R. O. High-entropy alloys. \u003cem\u003eNat. Rev. Mater.\u003c/em\u003e \u003cstrong\u003e4\u003c/strong\u003e, 515\u0026ndash;534 (2019).\u003c/li\u003e\n\u003cli\u003eYeh, J.-W. \u003cem\u003eet al.\u003c/em\u003e Nanostructured High-Entropy Alloys with Multiple Principal Elements: Novel Alloy Design Concepts and Outcomes. \u003cem\u003eAdv. Eng. Mater.\u003c/em\u003e \u003cstrong\u003e6\u003c/strong\u003e, 299\u0026ndash;303 (2004).\u003c/li\u003e\n\u003cli\u003eBirbilis, N., Choudhary, S., Scully, J. R. \u0026amp; Taheri, M. L. A perspective on corrosion of multi-principal element alloys. \u003cem\u003eNpj Mater. Degrad.\u003c/em\u003e \u003cstrong\u003e5\u003c/strong\u003e, 14 (2021).\u003c/li\u003e\n\u003cli\u003eMiracle, D. B. \u0026amp; Senkov, O. N. A critical review of high entropy alloys and related concepts. \u003cem\u003eActa Mater.\u003c/em\u003e \u003cstrong\u003e122\u003c/strong\u003e, 448\u0026ndash;511 (2017).\u003c/li\u003e\n\u003cli\u003eArmstrong, M., Mehrabi, H. \u0026amp; Naveed, N. An overview of modern metal additive manufacturing technology. \u003cem\u003eJ. Manuf. Process.\u003c/em\u003e \u003cstrong\u003e84\u003c/strong\u003e, 1001\u0026ndash;1029 (2022).\u003c/li\u003e\n\u003cli\u003eGardner, L. Metal additive manufacturing in structural engineering \u0026ndash; review, advances, opportunities and outlook. \u003cem\u003eStructures\u003c/em\u003e \u003cstrong\u003e47\u003c/strong\u003e, 2178\u0026ndash;2193 (2023).\u003c/li\u003e\n\u003cli\u003eThomas-Seale, L. E. J., Kirkman-Brown, J. C., Attallah, M. M., Espino, D. M. \u0026amp; Shepherd, D. E. T. The barriers to the progression of additive manufacture: Perspectives from UK industry. \u003cem\u003eInt. J. Prod. Econ.\u003c/em\u003e \u003cstrong\u003e198\u003c/strong\u003e, 104\u0026ndash;118 (2018).\u003c/li\u003e\n\u003cli\u003eBaumers, M., Dickens, P., Tuck, C. \u0026amp; Hague, R. The cost of additive manufacturing: machine productivity, economies of scale and technology-push. \u003cem\u003eTechnol. Forecast. Soc. Change\u003c/em\u003e \u003cstrong\u003e102\u003c/strong\u003e, 193\u0026ndash;201 (2016).\u003c/li\u003e\n\u003cli\u003eOpenAI \u003cem\u003eet al.\u003c/em\u003e GPT-4 Technical Report. Preprint at https://doi.org/10.48550/arXiv.2303.08774 (2024).\u003c/li\u003e\n\u003cli\u003eTouvron, H. \u003cem\u003eet al.\u003c/em\u003e LLaMA: Open and Efficient Foundation Language Models. Preprint at https://doi.org/10.48550/arXiv.2302.13971 (2023).\u003c/li\u003e\n\u003cli\u003eBaktash, J. A. \u0026amp; Dawodi, M. Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing. Preprint at https://doi.org/10.48550/arXiv.2305.03195 (2023).\u003c/li\u003e\n\u003cli\u003eBubeck, S. \u003cem\u003eet al.\u003c/em\u003e Sparks of Artificial General Intelligence: Early experiments with GPT-4. Preprint at https://doi.org/10.48550/arXiv.2303.12712 (2023).\u003c/li\u003e\n\u003cli\u003eKashefi, A. \u0026amp; Mukerji, T. ChatGPT FOR PROGRAMMING NUMERICAL METHODS. \u003cem\u003eJ. Mach. Learn. Model. Comput.\u003c/em\u003e \u003cstrong\u003e4\u003c/strong\u003e, (2023).\u003c/li\u003e\n\u003cli\u003eRozi\u0026egrave;re, B. \u003cem\u003eet al.\u003c/em\u003e Code Llama: Open Foundation Models for Code. Preprint at https://doi.org/10.48550/arXiv.2308.12950 (2024).\u003c/li\u003e\n\u003cli\u003eNi, B. \u0026amp; Buehler, M. J. MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge. \u003cem\u003eExtreme Mech. Lett.\u003c/em\u003e \u003cstrong\u003e67\u003c/strong\u003e, 102131 (2024).\u003c/li\u003e\n\u003cli\u003eJignasu, A. \u003cem\u003eet al.\u003c/em\u003e Towards Foundational AI Models for Additive Manufacturing: Language Models for G-Code Debugging, Manipulation, and Comprehension. Preprint at https://doi.org/10.48550/arXiv.2309.02465 (2023).\u003c/li\u003e\n\u003cli\u003eBuehler, M. J. MechGPT, a Language-Based Strategy for Mechanics and Materials Modeling That Connects Knowledge Across Scales, Disciplines, and Modalities. \u003cem\u003eAppl. Mech. Rev.\u003c/em\u003e \u003cstrong\u003e76\u003c/strong\u003e, (2024).\u003c/li\u003e\n\u003cli\u003eJumper, J. \u003cem\u003eet al.\u003c/em\u003e Highly accurate protein structure prediction with AlphaFold. \u003cem\u003eNature\u003c/em\u003e \u003cstrong\u003e596\u003c/strong\u003e, 583\u0026ndash;589 (2021).\u003c/li\u003e\n\u003cli\u003eVaradi, M. \u003cem\u003eet al.\u003c/em\u003e AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. \u003cem\u003eNucleic Acids Res.\u003c/em\u003e \u003cstrong\u003e50\u003c/strong\u003e, D439\u0026ndash;D444 (2022).\u003c/li\u003e\n\u003cli\u003eAbramson, J. \u003cem\u003eet al.\u003c/em\u003e Accurate structure prediction of biomolecular interactions with AlphaFold 3. \u003cem\u003eNature\u003c/em\u003e \u003cstrong\u003e630\u003c/strong\u003e, 493\u0026ndash;500 (2024).\u003c/li\u003e\n\u003cli\u003eBaek, M. \u003cem\u003eet al.\u003c/em\u003e Accurate prediction of protein structures and interactions using a three-track neural network. \u003cem\u003eScience\u003c/em\u003e \u003cstrong\u003e373\u003c/strong\u003e, 871\u0026ndash;876 (2021).\u003c/li\u003e\n\u003cli\u003eKrishna, R. \u003cem\u003eet al.\u003c/em\u003e Generalized biomolecular modeling and design with RoseTTAFold All-Atom. \u003cem\u003eScience\u003c/em\u003e \u003cstrong\u003e384\u003c/strong\u003e, eadl2528 (2024).\u003c/li\u003e\n\u003cli\u003eNi, B., Kaplan, D. L. \u0026amp; Buehler, M. J. Generative design of \u003cem\u003ede novo\u003c/em\u003e proteins based on secondary-structure constraints using an attention-based diffusion model. \u003cem\u003eChem\u003c/em\u003e \u003cstrong\u003e9\u003c/strong\u003e, 1828\u0026ndash;1849 (2023).\u003c/li\u003e\n\u003cli\u003eNi, B., Kaplan, D. L. \u0026amp; Buehler, M. J. ForceGen: End-to-end de novo protein generation based on nonlinear mechanical unfolding responses using a language diffusion model. \u003cem\u003eSci. Adv.\u003c/em\u003e \u003cstrong\u003e10\u003c/strong\u003e, eadl4000 (2024).\u003c/li\u003e\n\u003cli\u003eNi, B. \u0026amp; Buehler, M. J. Agentic End-to-End De Novo Protein Design for Tailored Dynamics Using a Language Diffusion Model. Preprint at https://doi.org/10.48550/arXiv.2502.10173 (2025).\u003c/li\u003e\n\u003cli\u003eTong, X. \u003cem\u003eet al.\u003c/em\u003e Generative Models for De Novo Drug Design. \u003cem\u003eJ. Med. Chem.\u003c/em\u003e \u003cstrong\u003e64\u003c/strong\u003e, 14011\u0026ndash;14027 (2021).\u003c/li\u003e\n\u003cli\u003eSanchez-Lengeling, B. \u0026amp; Aspuru-Guzik, A. Inverse molecular design using machine learning: Generative models for matter engineering. \u003cem\u003eScience\u003c/em\u003e \u003cstrong\u003e361\u003c/strong\u003e, 360\u0026ndash;365 (2018).\u003c/li\u003e\n\u003cli\u003eTran, H. \u003cem\u003eet al.\u003c/em\u003e Design of functional and sustainable polymers assisted by artificial intelligence. \u003cem\u003eNat. Rev. Mater.\u003c/em\u003e \u003cstrong\u003e9\u003c/strong\u003e, 866\u0026ndash;886 (2024).\u003c/li\u003e\n\u003cli\u003eGurnani, R. \u003cem\u003eet al.\u003c/em\u003e polyG2G: A Novel Machine Learning Algorithm Applied to the Generative Design of Polymer Dielectrics. \u003cem\u003eChem. Mater.\u003c/em\u003e \u003cstrong\u003e33\u003c/strong\u003e, 7008\u0026ndash;7016 (2021).\u003c/li\u003e\n\u003cli\u003ePark, H., Li, Z. \u0026amp; Walsh, A. Has generative artificial intelligence solved inverse materials design? \u003cem\u003eMatter\u003c/em\u003e \u003cstrong\u003e7\u003c/strong\u003e, 2355\u0026ndash;2367 (2024).\u003c/li\u003e\n\u003cli\u003eChen, Y. \u003cem\u003eet al.\u003c/em\u003e MatterGPT: A Generative Transformer for Multi-Property Inverse Design of Solid-State Materials. Preprint at https://doi.org/10.48550/arXiv.2408.07608 (2024).\u003c/li\u003e\n\u003cli\u003eRoy, A. \u003cem\u003eet al.\u003c/em\u003e Rapid discovery of high hardness multi-principal-element alloys using a generative adversarial network model. \u003cem\u003eActa Mater.\u003c/em\u003e \u003cstrong\u003e257\u003c/strong\u003e, 119177 (2023).\u003c/li\u003e\n\u003cli\u003eLi, Z., Li, S. \u0026amp; Birbilis, N. A machine learning-driven framework for the property prediction and generative design of multiple principal element alloys. \u003cem\u003eMater. Today Commun.\u003c/em\u003e \u003cstrong\u003e38\u003c/strong\u003e, 107940 (2024).\u003c/li\u003e\n\u003cli\u003eVaswani, A. \u003cem\u003eet al.\u003c/em\u003e Attention is All you Need. in \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e vol. 30 (Curran Associates, Inc., 2017).\u003c/li\u003e\n\u003cli\u003eTaheri-Mousavi, S. M. \u003cem\u003eet al.\u003c/em\u003e Additively manufacturable high-strength aluminum alloys with thermally stable microstructures enabled by hybrid machine learning-based design. Preprint at https://doi.org/10.48550/arXiv.2406.17457 (2024).\u003c/li\u003e\n\u003cli\u003eTAHERI-MOUSAVI, S. M., HART, A. \u0026amp; Olson, G. Methodologies for formulating compositions, including aluminum alloys with high-temperature strength. (2024).\u003c/li\u003e\n\u003cli\u003eRadford, A. \u003cem\u003eet al.\u003c/em\u003e Language models are unsupervised multitask learners. \u003cem\u003eOpenAI Blog\u003c/em\u003e \u003cstrong\u003e1\u003c/strong\u003e, 9 (2019).\u003c/li\u003e\n\u003cli\u003eByte-Pair Encoding tokenization - Hugging Face NLP Course. https://huggingface.co/learn/nlp-course/en/chapter6/5.\u003c/li\u003e\n\u003cli\u003eSchuurmans, D., Dai, H. \u0026amp; Zanini, F. Autoregressive Large Language Models are Computationally Universal. Preprint at https://doi.org/10.48550/arXiv.2410.03170 (2024).\u003c/li\u003e\n\u003cli\u003eYan, X. \u0026amp; Lin, J. C. Prediction of hot tearing tendency for multicomponent aluminum alloys. \u003cem\u003eMetall. Mater. Trans. B\u003c/em\u003e \u003cstrong\u003e37\u003c/strong\u003e, 913\u0026ndash;918 (2006).\u003c/li\u003e\n\u003cli\u003eChaudhari, A., Guntuboina, C., Huang, H. \u0026amp; Farimani, A. B. AlloyBERT: Alloy Property Prediction with Large Language Models. Preprint at http://arxiv.org/abs/2403.19783 (2024).\u003c/li\u003e\n\u003cli\u003eKhatamsaz, D. \u003cem\u003eet al.\u003c/em\u003e Bayesian optimization with active learning of design constraints using an entropy-based approach. \u003cem\u003eNpj Comput. Mater.\u003c/em\u003e \u003cstrong\u003e9\u003c/strong\u003e, 1\u0026ndash;14 (2023).\u003c/li\u003e\n\u003cli\u003eDebnath, A. \u003cem\u003eet al.\u003c/em\u003e Comparing forward and inverse design paradigms: A case study on refractory high-entropy alloys. \u003cem\u003eJ. Mater. Res.\u003c/em\u003e \u003cstrong\u003e38\u003c/strong\u003e, 4107\u0026ndash;4117 (2023).\u003c/li\u003e\n\u003cli\u003e[2001.08361] Scaling Laws for Neural Language Models. https://arxiv.org/abs/2001.08361.\u003c/li\u003e\n\u003cli\u003e[2404.05405] Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. https://arxiv.org/abs/2404.05405.\u003c/li\u003e\n\u003cli\u003eSames, W. J., List, F. A., Pannala, S., Dehoff, R. R. \u0026amp; Babu, S. S. The metallurgy and processing science of metal additive manufacturing. \u003cem\u003eInt. Mater. Rev.\u003c/em\u003e \u003cstrong\u003e61\u003c/strong\u003e, 315\u0026ndash;360 (2016).\u003c/li\u003e\n\u003cli\u003eBrennan, M. C., Keist, J. S. \u0026amp; Palmer, T. A. Defects in Metal Additive Manufacturing Processes. \u003cem\u003eJ. Mater. Eng. Perform.\u003c/em\u003e \u003cstrong\u003e30\u003c/strong\u003e, 4808\u0026ndash;4818 (2021).\u003c/li\u003e\n\u003cli\u003eDan, X. \u003cem\u003eet al.\u003c/em\u003e Exceptional strength and ductility in heterogeneous multi-gradient TiAl alloys through additive manufacturing. \u003cem\u003eActa Mater.\u003c/em\u003e \u003cstrong\u003e281\u003c/strong\u003e, 120395 (2024).\u003c/li\u003e\n\u003cli\u003eAndersson, J.-O., Helander, T., H\u0026ouml;glund, L., Shi, P. \u0026amp; Sundman, B. Thermo-Calc \u0026amp; DICTRA, computational tools for materials science. \u003cem\u003eCalphad\u003c/em\u003e \u003cstrong\u003e26\u003c/strong\u003e, 273\u0026ndash;312 (2002).\u003c/li\u003e\n\u003cli\u003eLoshchilov, I. \u0026amp; Hutter, F. Decoupled Weight Decay Regularization. Preprint at https://doi.org/10.48550/arXiv.1711.05101 (2019).\u003c/li\u003e\n\u003cli\u003eAndrej. karpathy/nanoGPT. (2024).\u003c/li\u003e\n\u003cli\u003ePaszke, A. \u003cem\u003eet al.\u003c/em\u003e PyTorch: An Imperative Style, High-Performance Deep Learning Library. Preprint at https://doi.org/10.48550/arXiv.1912.01703 (2019).\u003c/li\u003e\n\u003cli\u003eTerms and Conditions for Purchase. \u003cem\u003eThermo-Calc Software\u003c/em\u003e https://thermocalc.com/terms-and-conditions-for-the-purchase/.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-computational-materials","isNatureJournal":false,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"npjcompumats","sideBox":"Learn more about [npj Computational Materials](http://www.nature.com/npjcompumats/)","snPcode":"41524","submissionUrl":"https://mts-npjcompumats.nature.com/","title":"npj Computational Materials","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Alloy design, generative deep learning, composition-structure-property relationship, language model, alloy language, gradient alloys, alloy diversity","lastPublishedDoi":"10.21203/rs.3.rs-6067058/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6067058/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eRapid progress in additive manufacturing of alloys opens opportunities in controlling compositions and microstructures at voxel-size resolution in complex geometries, thus unlocking unprecedented design and performance in various critical engineering applications. However, to fully exploit such potential, capable yet efficient models for navigating the vast design spaces of alloy compositions, structures and properties are of great research interest. Here, we present AlloyGPT, an autoregressive alloy-specific language model, that learns the composition-structure-property relationship and generates novel designs for additively manufacturable alloys. Specifically, we develop efficient grammar to convert physics-rich alloy datasets into readable text records for both forward prediction and inverse design tasks. Then, we construct a customized tokenizer and generative pre-trained transformer (GPT) model to master this alloy-specific language through autoregressive training. At deployment, our model can accurately predict multiple phase structures and properties based on given alloy compositions, achieving R2 values ranging from 0.86 to 0.99 for the test set. When tested beyond the learned composition domain, this performance only degrades gradually in a stable manner. Given the desired properties and structures, the same model can suggest multiple alloy compositions that meet the design goals. And the balance between composition diversity and design accuracy can be further tuned stably. Our AlloyGPT model presents a novel way of integrating comprehensive knowledge of alloys in terms of language and can simultaneously solve forward prediction and inverse design tasks with accuracy, diversity and robustness. This fundamental language model will open new avenues to accelerate knowledge integration and material design for pure or gradient structural alloys manufactured by traditional and additive manufacturing.\u003c/p\u003e","manuscriptTitle":"AlloyGPT: End-to-end prediction and design of additively manufacturable alloys using an autoregressive language model","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-03-07 06:52:00","doi":"10.21203/rs.3.rs-6067058/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-03-14T17:42:52+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-03-14T10:20:25+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-03-11T21:50:03+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-03-03T12:35:00+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"50710225820459726477466535391421955300","date":"2025-02-25T04:04:42+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"130228102554709153758517937036781382014","date":"2025-02-24T16:30:51+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"306569302426636544258073321195700433572","date":"2025-02-24T01:24:34+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-02-23T20:00:34+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-02-23T19:44:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-02-20T07:38:10+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Computational Materials","date":"2025-02-19T21:35:42+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-computational-materials","isNatureJournal":false,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"npjcompumats","sideBox":"Learn more about [npj Computational Materials](http://www.nature.com/npjcompumats/)","snPcode":"41524","submissionUrl":"https://mts-npjcompumats.nature.com/","title":"npj Computational Materials","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"08257ec1-5057-4077-9d1b-f2f2de4d6b67","owner":[],"postedDate":"March 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":45267503,"name":"Physical sciences/Materials science/Structural materials/Metals and alloys"},{"id":45267504,"name":"Physical sciences/Materials science/Theory and computation/Computational methods"}],"tags":[],"updatedAt":"2025-09-29T15:59:31+00:00","versionOfRecord":{"articleIdentity":"rs-6067058","link":"https://doi.org/10.1038/s41524-025-01768-2","journal":{"identity":"npj-computational-materials","isVorOnly":false,"title":"npj Computational Materials"},"publishedOn":"2025-09-26 15:56:59","publishedOnDateReadable":"September 26th, 2025"},"versionCreatedAt":"2025-03-07 06:52:00","video":"","vorDoi":"10.1038/s41524-025-01768-2","vorDoiUrl":"https://doi.org/10.1038/s41524-025-01768-2","workflowStages":[]},"version":"v1","identity":"rs-6067058","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6067058","identity":"rs-6067058","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}