RP3Net: a deep learning model for predicting recombinant protein production in Escherichia coli

Motivation The production of protein reagents is an essential part of the research and development process in the pharmaceutical and biotechnology industries. In drug discovery it is often a pre -requisite for screening and hit identification 1,2. In living tissues, the target protein may occur in very small amounts alongside numerous other biomolecules. To be used for high-throughput screening of drug candidates, structural determination and the development of functional assays, the target protein needs to be expressed in a cell culture and purified. The ability to express a protein depends on multiple factors. First and foremost is the protein itself, but other factors include the cloning vector, the species and strain of the host cells, the codon optimisation algorithm, the use of tags and fusion proteins, and other experimental conditions 3–21. The choice of these parameters is often influenced by the details of the downstream experiments ,22 making protein production time-consuming and error-prone, and often requiring multiple iterations and much trial and error. The purpose of this work is to develop a deep learning model to predict soluble protein expression in E. coli from the construct sequence, thus accelerating the timescales for protein production from months to weeks, cutting costs and reducing environmental impact. A recombinant protein production experimental pipeline involves several steps 23, including con- struct design, cloning, small -scale expression screening, progression of expressing constructs to large-scale purification and quality control (QC). Small-scale soluble expression screening, shown in Fig. 1, is crucial for assessing whether to progress the construct to large -scale production. First, cells are transfected with vectors (e.g. plasmids) carrying the cloned DNA of the protein of interest .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 3 (step 1). Recombinant protein production is performed in deep well format (step 2). Cells are spun down and lysed (step 3). The lysate contains the total amount of protein produced. After an addi- tional centrifugation step the soluble protein is found in the supernatant and the insoluble material is discarded in the pellet (step 3). Soluble protein is captured in a one -step purification using the histidine tag and immobilized metal affinity chromatography (IMAC24, step 4). The soluble protein yield and correct size are typically assessed by performing denaturing gel electrophoresis (SDS - PAGE) where yield and size are compared to a protein standard (step 5). The yield can be estimated by quantifying the amount of the target protein compared to the protein standard in the stained gel, Fig. 1 The experimental workflow for small-scale recombinant soluble protein production with one-step purification. After cloning, plasmids with the genetic material of the protein of interest are transfected into E. coli cells (step 1). The cells are grow n for 24 hours (step 2). Harvesting involves two centrifugation stages: first to pellet down the cells, then, after lysis, to isolate the soluble protein in the supernatant (step 3). This is followed by IMAC purification (step 4), yield estimation via densitometric analysis of SDS-PAGE gels (step 5) and, finally, data capture for further analysis and machine learning (ML, step 6). Image generated with BioRender.com. .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 4 using densitometric analysis. At this stage, it is important to record both positive (produced) and negative (failed to produce) experimental outcomes (step 6). Throughout the rest of this publication, unless stated explicitly, terms “protein production” and “protein expression” refer to this step of the experimental pipeline. Constructs that pass this small-scale screening are then typically progressed to large-scale purification and further downstream applications. Protein and DNA foundation models (FMs) have become ubiquitous tools for predicting structural and functional protein properties from amino acid and/or nucleic acid sequences 25–32. These FMs are typically trained on large corpora of sequences, such as UniRef33,34, GenBank35,36, MGnify37 or BFD38 (Table 1). During training a portion of the sequence is masked out, i.e. each residue is re- placed with a special "blank" character. The training objective then becomes to reconstruct the masked portion of the input, or "fill in the blanks". This technique, referred to as language modelling task, originates from natural language processing39. ESM25,26,40, ProtBert27 and ProteinBert28 are examples of protein FMs; DNABert30,31 is a popular DNA FM. These models are all based on Transformer deep learning architecture 41 with different number of layers, feature dimensions and other details. HyenaDna 29 is another DNA FM that uses a different architecture. The intermediate layers of foundational models yield a residue -level sequence representation that can be used to predict the protein property of interest, such as secondary or tertiary structure, binding affinity, fluorescence, thermodynamic stability, sol ubility, etc25,26,38,40,42–47. The experimental da- tasets that describe these properties typically contain orders of magnitude fewer entries when com- pared to the sequence corpora. This scarcity of experimental datasets often makes it unfeasible to .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 5 train large foundational models from scratch for predicting protein properties. Such models are usu- ally pre-trained with the language modelling objective on the large corpora first, and then further trained to predict the property of interest using the smaller dataset 42,43,47. This final training step is referred to as fine -tuning. RP3Net follows this architectural blueprint by encoding the biological sequence with a foundational model, feeding this encoding through an aggregation layer to obtain a global representation for the entire construct, and then applying a fully connected classific ation head to compute the predicted probability of recombinant expression in E. coli (Fig. 2A) as a binary outcome. Although, in theory, a soluble protein production fine-tuning dataset could be designed and A Fig. 2. A. Architecture diagram of RP3Net. The input biological sequence is encoded by the foundation model to obtain a sequence representation, where each residue/codon/nucleotide is represented by a vector. The aggregation layer builds a global protein rep- resentation vector from the sequence representation. The predicted probability of successful recombinant expression of the protein in E. coli is computed by the fully connected classification head from the protein representation. B. Training with meta label correc- tion on a mixture of clean and noisy data. The standard training setup, where the model loss on clean inputs and labels is minimised with gradient descent, is shown in the top row. A special “teacher” model is trained to predict the corrected labels from the noisy input and labels. These corrected labels, along with noisy inputs, serve as inputs for training the “student” model. The latter model has the same architecture and weights as the “clean” model. The bi-level optimisation algorithm that makes sure that the corrected labels do not deviate from the (unknown) clean labels, relies on using the cross entropy (CE) loss. Model components with trainable weights are shown as blue boxes. Training data is shown as yellow boxes. Images generated with BioRender.com. B .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 6 experimentally generated from scratch, in practice it would be too time -consuming and expensive. Moreover, there already exist publicly available datasets of protein expression that contain the re- sults of experiments worth millions of dollars and representing years of lab work48–50. For training and evaluating RP3Net, the internal AstraZeneca (AZ) small-scale expression screen data is com- bined with datasets from the Structural Genomics Consortium (SGC), specifically their sites in Stockholm4,51 and Toronto23,50. The experimental pipeline for generating data from AZ and SGC Stockholm has been already discussed above, see Fig. 1, step 6. SGC Toronto captures the results of large-scale protein purification. Existing work A number of models that predict soluble expression from construct sequence have been published in recent years. Most of these systems use datasets derived from the Protein Structure Initiative (PSI) compendium, also referred to as TargetTrac k 48,49. PSI was an experimental research effort run across multiple laboratories in 2000-2017, with the objective of determining protein structures and depositing them in the Protein Data Bank (PDB)52,53 . This dataset records the pipeline position, i.e. the experimental stage where the work was terminated, for each target and construct. For exam- ple, if a construct was selected and cloned, but could not be expressed, its pipeline position would be recorded as "cloned". For another construct that has been selected, cloned, expressed and puri- fied, but could not be crystallised, the pipeline position would be "purified", e tc. One limitation of using TargetTrack data in this work is that in this dataset a genuine inability to express the construct under given experimental conditions can be confused with stopping to pursue the construct for other reasons (for example there being another well-behaving construct for the same target ). Different labs that have provided data for TargetTack were using different experimental pipelines: sometimes .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 7 small-scale expression screening as shown in Fig. 1, but also large-scale purification results as SGC Toronto, or the results of running SDS-PAGE on unpurified cell lysate. It is important to make a distinction between solubility as a general physical property of the protein, which can be measured, for example, as peak concentration in the solution, and the ability to achieve soluble expression of the protein under given experimental conditions, which is a binary outcome that is modelled in this work. A protein that is generally soluble could still fail to express, for ex- ample because it is toxic for the host cells, or because the chaperones that are required for forming the correct structure are missing, or due to other reasons. Solubility is thus a necessary but insuffi- cient condition for soluble recombinant protein production. NetSolP44 uses PSI data to evaluate multiple transformer -based models available at the time for predicting soluble expression. NetSolP outputs two scores: solubility and "usability", the latter be- ing a combined predictor of solubility and the ability of a protein to be expressed. PLMC46 and SADeepCry45 also use data derived from TargetTrack and a Transformer architecture but output the pipeline position given the construct sequence. PPCPred54, PredPPCrys55, Crysalis56 DCFCrystal57 are examples of older, simpler models that predict pipeline position, trained on vari- ous subsets of TargetTrack. SoluProt58 uses a different PSI-based dataset with a Gradient Boosted Machine (GBM) model 59 and global features based on relative amino acid frequencies, predicted physicochemical properties, similarity to E. coli proteome and output of various other bioinformat- ics tools to predict soluble expression. CamSol60,61 is a well -established relative solubility prediction tool for libraries of similar protein sequences. There are many other solubility predictions that use deep neural networks, such as GPSFun62 and PLM_Sol63. A few methods exist for modelling expression and solubility of human antibodies, but their experimental protocols differ substantially from E. coli -based expression .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 8 analysed in this work64,65. .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 9

and discussion RP3Net with fixed foundation model weights outperforms decision trees with global protein features The RP3Net architecture with fixed foundation model weights and mean pooling (Fig. 2A) was used for selecting the best performing foundation model. This architecture is denoted as Model A (Table 2). The models were trained and evaluated on SGC Stockholm dataset, with five-fold cross validation. SGC Stockholm was used because this dataset is of medium size, compared to AZ, which is much smaller, and SGC Toronto, which is much larger. This data set is also the only one of the three that provides DNA sequences for all the constructs. A gradient boosted decision tree (XGBoost 59) with global protein features as inputs was used as a Fig. 3. Performance of RP3Net with fixed foundation model (FM) weights and mean pooling (Model A) on SGC Stockholm, along with FM parameter count. On the left y-axis each boxplot shows the area under Receiver Operator Curve (AUROC) of Model A with a particular FM, evaluated on SGC Stockholm test data, with five-fold cross validation. On the right y-axis, black squares show the number of trainable parameters of the FM, in log scale. ESM2 (650M) and CaLM were selected for further analysis based on perfor- mance, consistency, parameter count and licensing restrictions. “Random embeddings” means using random residue embeddings instead of a foundation model. The source data for all charts is available in the supplement. .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 10 baseline model. As shown in Fig. 3, Model A with any protein FM outperforms the baseline model. Out of all the tested DNA and codon FMs, only CaLM shows better results than the baseline. A plausible explanation for this observation is due to the datasets used for pre-training the FMs: CaLM was pre-trained on coding sequences from ENA, whereas other DNA FMs were pre -trained on a mixture of coding and non-coding sequences. RP3Net performance also varies depending on the training data subset that was used, sometimes dramatically. For example, for the more consistent FMs, such as ESM2 (650M) and CaLM, the difference between the best and the worst runs is 0.03 and 0.01, respectively, whereas for Hye- naDNA Medium the difference is 0.14. The number of trainable foundation model parameters is used to indicate the resource requirements for fine-tuning the FM (compute time, memory) . The foundation model for subsequent evaluation was chosen based on the pragmatic trade-off between performance, training complexity and licens- ing constraints (see Table 1). We selected ESM2 with 650 million parameters . The simple Model A training protocol, applied to this FM, achieves an average increase in AUROC of 0.03, compared to the baseline model. Performance on different data sources reveals dependency on dataset size. Model A performance on SGC Stockholm dataset can be improved by replacing the mean pooling aggregation layer with a more sophisticated set transformer pooling (STP 66,67). This configuration is denoted as Model B. The main difference between mean pooling and STP is that, whereas the former just takes an average across the sequence, giving each residue the same weight, STP uses context-dependent weights for residue represe ntations, by computing multiheaded attention (MHA41, see Methods) between a special parameter, called the seed vector, and the output of the foundation model. The seed vector is updated during training with gradient descent, along with the .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 11 rest of the model parameters. Model B gives an AUROC improvement of 0.01% over Model A when trained and evaluated on SGC Stockholm (Table 3). The performance of Model B on other data sources varies with an AU- ROC of 0.59 on the AZ dataset and of 0.84 on SGC Toronto. A plausible reason for this variation is dataset size. Training Model B on the combined AZ and SGC Stockholm data improves evalua- tion of AZ to 0.73, which is almost the same as evaluating the same model on SGC Stockholm. Adding SGC Toronto to the training data does not improve the evaluation results significantly for any data source. Meta label correction with purification data yields a 0.04 increase in AUROC on SGC Stockholm Both Model A and Model B are fine-tuned on soluble protein expression data with frozen parame- ters of the foundation model. Unfreezing these parameters (Model C) and training on the full dataset leads to overfitting: perfect performance is quickly achieved on the training data set (AUROC»1.0), but on the validation and test sets the AUROC remains below 0.75. Training Model C on individual data sources also leads to overfitting, as expected. This could be explained by the fact that the datasets contain the results of slightly different experi- ments. The SGC Toronto dataset reports results of large-scale purification, whereas both AZ and SGC Stockholm report small-scale expression testing captured with one-step purification. Although the exact experimental conditions, materials and methods used for purifications were not available during model development, it is safe to assume that the SGC Toronto conditions are quite different from the small-scale expression testing. A natural question arises: given the construct sequence from SGC Toronto, and its binary purification result, what would be the result of small-scale expression testing this construct under the conditions of SGC Stockholm or AZ? .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 12 We address this within the Meta Label Correction framework (MLC68,69), where a large, noisy data set is used to aid training the model on a small, clean set. Rather than adding the noisy data directly to the training set, a special model is trained to predict the corrected label from the noisy input and noisy label. This is referred to as the "teacher model". The corrected labels are used to train the "student model", along with clean inputs and clean labels ( Fig. 2B). In our setup, SGC Toronto large scale purification data set is used to train the teacher model, and a union of SGC Stockholm and AZ small-scale expression data is used to train the student model. Using MLC with large scale purification data (Model D) achieves in AUROC of 0.74 on AZ dataset. This is an improvement of 0.01 compared to the second-best result of Model B on AZ data. When evaluating Model D on SGC Stockholm, AU ROC reaches 0.77, which is an improvement of 0.04 over the next-best result. We have also observed that the MLC model is more robust across different sequence clusters – training, validation and testing – than other models, which tend to overfit the training data. The MLC framework thus allows utilising large scale purification data to improve modelling of small-scale expression testing, whereas simple transfer learning (Model B or Model C trained on all sources) fails to achieve that outcome. Prospective experimental validation of the model shows AUROC of 0.83 To establish the utility of RP3Net for drug discovery projects, in addition to the normal train-vali- date-test model development loop, we have conducted prospective model evaluation in a real-life scenario. A set of 46 proteins was curated from the human proteome to include viable drug targets, whilst avoiding proteins with prior published evidence of successful expression. We started by gen- erating two full length constructs per target (with a 6-His affinity tag placed at the N- or C-Terminal) and running RP3Net on them. If both constructs were predicted not to express, we generated .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 13 trimmed constructs, ran these through the model, and , if they were predicted to express , included them in the dataset (see Methods). This resulted in a total of ninety-seven constructs for the exper- imental validation dataset, eight of which were generated by the trimming process (see Methods). The constructs were cloned and expressed in E. coli at the AZ protein production facility. 54% of the constructs passed small -scale expression screening, including one-step purification by affinity chromatography. The remaining 46% were annotated as either “Ambiguous” or “Not Passed”. The performance of RP3Net models B and D was compared with the baseline model, and two third- party predictors: SoluProt58 and NetSolP44 (Table 4). The highest AUROC of 0.83 is achieved by RP3Net model D. This is 0.06 better than the next-best result (RP3Net B trained on AZ and SGC Stockholm), and 0.08 better than the best third -party predictor (NetSolP useability). RP3Net D showed an accuracy of 0.77 when the score cut-off of 0.5 was used, and accuracy of 0.81 with the cut-off set to 0.79. For the subset of eight trimmed constructs, RP3Net D shows an accuracy of 0.5 with score cut-off of 0.5, and accuracy of 0.62 with score cut -off of 0.79. This could be an artefact of the small eval- uation set, or that RP3Net does not consider if sequences will fold into stable protein domains. Curiously, the trimmed constructs that did result in soluble protein also contained degradation prod- ucts (supplementary figure 1). With the score cut -off of 0.5 t he model predicts all trimmed con- structs to express, whereas in fact only four out of eight were expressed successfully. Performance on trimmed constructs could thus be considered an area for improvement. Ho wever, considering the small number of trimmed constructs, and the model accuracy (0.77) and precision (0.73) on the larger experimental validation set, it could be argued that an experimental scientist would still find the modelling results helpful. An “overconfident”, high recall, model that predicts too many positives, which are then partly confirmed in the laboratory, is preferrable to a model that .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 14 misses out constructs that would have expressed in the lab. Model precision could be increased, at the expense of recall, by increasing the score cut-off threshold. .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 15

The recombinant production of proteins can require multiple experimental rounds of trial and error. To improve the efficiency of such experiments, we have developed RP3Net, an AI model of heter- ologous protein expression in E. coli. RP3Net predicts the results of protein expression as a binary outcome. It was built using the latest foundational models and was trained using a combination of internal experimental results from small -scale AZ expression screens, and publicly available data from the SGC. Using an STP aggregation layer and MLC with large scale purification data enables RP3Net to achieve state-of-the-art performance both on the take-out data from SGC Stockholm and AZ. RP3Net has been experimentally validated on a manually selected set of constructs for viable human drug targets and outperformed third party predictors on that set as well. Ablation studies show that there is no single method that achieves a large performance inc rease, but rather many small incremental improvements. This work also underscores the need for large and well curated datasets of soluble protein expression and for the scientific community to agree on how the data should be captured following the FAIR23,70 principles, and to establish a protein production ontology. Unfortunately, in the field of protein production there is not yet an equivalent of the PDB for structural biology. Significant time in this project was spent on data curation. The modelling results may also be further improved by making the model more aware of the exper- imental conditions, such as E. coli host strain, induction methods, time and temperature at which various experimental stages were performed, buffer formulations, etc. This information is largely missing from the currently available data sets. RP3Net is already deployed and used by the protein scientists at AZ. This publication and the .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 16 accompanying code repository at GitHub1 make the model available to the wider research commu- nity, both in industry and in academia. 1 www.github.com/RP3Net/RP3Net .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 17

We would like to thank Susanne Gräslund and Opher Gileadi from SGC Stockholm, and Matthieu Schapira and Peter Loppnau from SGC Toronto for sharing the ir respective datasets and helping with the curation. We would like to acknowledge colleagues from the Quantitative Biology and Protein Science departments at AstraZeneca for constructive discussions during this project, and David Öling from BioPharmaceuticals R&D at AZ for overseeing the cloning of the experimental constructs. We would like to acknowledge Mat thew Hall from the Industry Partnerships team at EMBL-EBI and Birgit Kerber and colleagues from EMBLEM for helping to organise the collabo- ration; and the EMBL-EBI IT team for maintaining the computational facilities used to train the models. We also acknowledge the funding from the Member States of the European Molecular Bi- ology Laboratory (ARL). .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 18 Tables Table 1. Foundational models. Model Type Architecture Training Data Sources Training Data Size Number of Parameters Comments ESM2 650M26 Protein Transformer UniRef50, UniRef90 65M 650M ESM2 3B 26 Protein Transformer UniRef50, UniRef90 65M 2.8B ESM371 Protein Transformer Uniref, MGnify, JGI IMG/M, OAS 2.78B 1.8B Licensing re- strictions apply ESMC 60072 Protein Transformer Uniref, MGnify, JGI IMG/M 2.3B 575M Licensing re- strictions apply ESMC 30072 Protein Transformer Uniref, MGnify, JGI IMG/M 2.3B 333M Licensing re- strictions apply ProtT5 XL27 Protein Transformer UniRef50 49M 1.2B Encoder only ProtBert27 Protein Transformer UniRef100 217M 420M Protein- Bert28 Protein Transformer Uniref90 106M 16M HyenaDNA Medium29 DNA Hyena Human Genome hg38 3.2B bp 24M Maximum se- quence length = 450K bases HyenaDNA Large 29 DNA Hyena Human Genome hg38 3.2B bp 46M Maximum se- quence length = 1M bases DNABert31 DNA Transformer Genomes from 136 species be- longing to 6 clas- ses 32.49B bp 117M CaLM32 Codon Transformer European Nucle- otide Archive, coding se- quences 8.7M 85M .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 19 Table 2. RP3Net training and architecture configurations. RP3Net Model Aggregation FM weights Meta Label Correction A Mean Frozen No B STP Frozen No C STP Fine-tuned, LoRA No D STP Fine-tuned, LoRA Yes .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 20 Table 3. Results of evaluating different RP3Net models trained on different data sources, versus the baseline model and third party predictors. Model Trained on Tested on AUROC Accuracy Recall Precision NetSolP Solubility AZ 0.64 0.61 0.44 0.64 NetSolP Useability AZ 0.64 0.57 0.13 0.87 B AZ AZ 0.59 0.52 0.85 0.50 B SGC Stockholm, AZ AZ 0.73 0.66 0.72 0.63 B SGC Stockholm, SGC Toronto, AZ AZ 0.71 0.66 0.50 0.72 C SGC Stockholm, SGC Toronto, AZ AZ 0.72 0.66 0.52 0.71 D SGC Stockholm, AZ, SGC Toronto AZ 0.74 0.70 0.71 0.69 NetSolP Solubility SGC Stockholm 0.48 0.66 0.14 0.42 NetSolP Useability SGC Stockholm 0.39 0.68 0.00 0.00 Baseline SGC Stockholm SGC Stockholm 0.62 0.62 0.39 0.40 A SGC Stockholm SGC Stockholm 0.70 0.63 0.74 0.45 B SGC Stockholm SGC Stockholm 0.72 0.63 0.74 0.45 B SGC Stockholm, AZ SGC Stockholm 0.73 0.62 0.78 0.45 B SGC Stockholm, SGC Toronto, AZ SGC Stockholm 0.70 0.60 0.64 0.42 C SGC Stockholm, SGC Toronto, AZ SGC Stockholm 0.68 0.62 0.56 0.43 D SGC Stockholm, AZ, SGC Toronto SGC Stockholm 0.77 0.73 0.54 0.59 B SGC Stockholm, SGC Toronto, AZ SGC Toronto 0.75 0.87 0.12 0.27 C SGC Stockholm, SGC Toronto, AZ SGC Toronto 0.76 0.88 0.08 0.33 .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 21 Table 4. Results of experimental validation of RP3Net, the baseline model, and third-party predictors. Model Trained on AUROC Accuracy Recall Precision Soluprot 0.64 0.57 0.83 0.57 NetSolp Solubility 0.66 0.60 0.79 0.59 NetSolp Useability 0.75 0.57 0.23 0.86 Baseline SGC Stockholm 0.67 0.65 0.58 0.71 RP3Net B AZ 0.69 0.65 0.88 0.62 RP3Net B AS, SGC Stockholm, 0.77 0.71 0.94 0.66 RP3Net B AZ, SGC Stockholm, SGC Toronto 0.76 0.69 0.65 0.74 RP3Net D AZ, SGC Stockholm, SGC Toronto MLC 0.83 0.77 0.92 0.73 .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 22

The dataset Protein production results from AZ, SGC Stockholm and SGC Toronto were used for training and evaluation of the models. AZ and SGC Stockholm report the results of small -scale protein expression testing, after one purification step. In the AZ dataset the outcome is reported as an expression yield category, manually estimated by the scientist who has expressed the protein. In addition, for a subset of constructs, an estimate of the absolute concentration value in mg/L is provided. The estimate is obtained by c omparing the size and intensity of the band on the SDS - PAGE gel for the protein of interest with the band for the reference protein of known concentration. This comparison is performed by internal image analysis software. The amino acid sequence of the construct includes affinity and solubility tags; DNA sequences are available for a subset of constructs. The dataset from SGC Stockholm contains genetic sequences, with tags, annotated with categorical outcomes. For the bulk of the SGC Toronto data, the outcome is reported as a pipeline position, similarly to PSI/TargetTrack. Importantly, there is no dedicated stage for expression screening: "cloned" is immediately followed by "purified". Although it can generally be assumed that a protein has to be expressed before it can be purified, it is sometimes the case that producing at larger scale (expression volume) can rescue a construct that failed to yield soluble protein at small scale. For a small subset of SGC Toronto data, small-scale expression screening outcome is also provided as a categorical variable, similarly to SGC Stockholm. Genetic sequences are available for a subset of observations, and tags are included in the constructs. Graphical overview of the datasets is shown in supplementary figure 2. There are a total of 67,055 unique sequences, covering 5,712 target proteins. Publicly available datasets are significantly larger .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 23 that the internal AZ data set, SGC Toronto being the largest. Datasets vary in terms of number of constructs per target, availability of genetic sequences vs protein sequences and imbalance between positive and negative outcomes. To normalise the data across multiple sources and to compute the outcome imbalance, the labels were converted to binary form, with "True" indicating successful production, and "False" failed production. For AZ this binary outcome was computed based on the existing category annotation, estimate of the absolute concentration and manual re-annotation. For SGC Stockholm the binary outcome was derived directly from the existing category annotation. For SGC Toronto, it was derived from the pipeline position. Historical AZ small -scale expression screening results were reported as concentration range estimates: "0 to 1", "1 to 10", "10 to 20", "20 to 50", and "above 50" mg/L. This data was converted to binary outcomes as follows. Results in "0 to 1 mg/L" concentration range were annotated as False (not produced). Results within "10 to 20", "20 to 50", and "above 50" mg/L were annotated as True (produced). Results in the "1 to 10" range were handled in a special manner. The experiments that had an estimate of absolute concentration were annotated as either True or False by comparing this value with the threshold of 3.5 mg/L. The experiments where the absolute value had not been estimated were re-annotated manually, by re-examining the captured image of the SDS-PAGE gel. AZ data that were collected after April 2023 do not contain manual estimates of the concentration range. Instead, each experiment outcome is manually classified by the scientist into three categories, based on the SDS-PAGE gel: "Passed", "Not passed" and "Ambiguous". Outcomes belonging to the "Passed" category were annotated as True, and those belonging to "Not passed" and "Ambiguous" categories as False. SGC Stockholm reports small -scale soluble expression screening outcomes with manual numeric .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 24 qualitative annotations by the lab scientist: 0 – no soluble expression, 1 – low soluble expression, 2 – medium, 3 – high and 4 – very high soluble expression. Outcomes from categories 0 and 1 were annotated as False, and with categories 2 and above – as True. Small-scale soluble expression results for SGC Toronto were annotated in the identical manner. For the SGC Toronto data with pipeline position outcome, results marked with "cloned" were annotated as False, and those with "purified" and beyond - as True. Cross validation To avoid bias towards any particular protein sequence motifs , five -fold cross validation was performed. All constructs were clustered using MMseqs2 v2568829073. The affinity and solubility tags were removed from the constructs, and the remaining "target" sequences were clustered. Sequence clusters that contain the AZ results recorded after 1st of September 2023, as well as the constructs used for the experimental validation, were grouped together to form the test set. The remaining constructs were divided into five cross validation subsets , such that each cluster is entirely contained within a single subset. Five-fold leave-one-out cross validation was performed on Model A with SGC Stockholm data, and the worst performing data split was chosen for reporting and for subsequent model development. The baseline model A gradient -boosted decision tree (XGBoost v2.1.3 59) was used as a baseline model. The input features for the tree were generated by analysing the sequences with ProtParam, as well as predicting global protein properties with Schrodinger API v2021-274, DisEMBL v2.0 75 and RaptorX76. The full list of features and methods used to compute them is given in supplementary table 1. For tools .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 25 that take protein sequence alignments as input, those were built against the Uniprot database downloaded in February 2016, clustered at 20% cutoff (uniprot20_2016_0277). Aggregation Two types of aggregation layer were tested in this work: mean pooling and Set Transformer Pooling (STP66,67). For notation, assume that for a sequence of length 𝑁, the output of the foundational model for each residue 𝑖 is represented with a column vector 𝑥! " from a 𝑑-dimensional space, 𝑥! " ∈ ℝ#×%. The matrix representation for the entire protein, 𝑋", is obtained by stacking these residue representations along the sequence dimension: 𝑋" ∈ ℝ#×&. In this notation, mean pooling, which is just averaging all these residue representations, can be written as 𝑋! " = 1 𝑁 % 𝑥# $ % #&' . (1) The advantage of mean pooling is that it is simple to interpret and fast to compute. The disadvantage is that it does not have any trainable parameters, or weights, so all training must happen upstream, in the foundational model, or downstream, in the clas sification head. STP is an example of an ag- gregation layer with trainable weights. Here, the global representation is the result of performing Multiheaded Attention (MHA41) with a seed vector, 𝑤' ∈ ℝ#, as a query, and residue representa- tions 𝑋" as keys and values. Thus, 𝑋'() * = MHA(𝑤', 𝑋" , 𝑋"). (2) Multi-headed attention Computing the MHA 41 involves weight matrices 𝑊+ ,, 𝑊+ -, 𝑊+ . and 𝑊/ for queries, keys, values and outputs, respectively, where ℎ = 1. . 𝐻, and 𝐻 is the number of heads: .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 26 MHA(𝑤', 𝑋" , 𝑋") = [concat(𝐴%, 𝐴0, … , 𝐴1)𝑊/]2, 𝐴+ = softmax 8𝑄+𝐾+ 2 ;𝑑3 < 𝑉+, 𝑄+ = 𝑤'2𝑊+ ,, 𝐾+ = 𝑋"2𝑊+ -, 𝑉+ = 𝑋"2𝑊+ .. (3) Here, the matrices 𝑊+ , ∈ ℝ#×#! are used to project 𝑤', into 𝑑3-dimensional space, 𝑊+ - ∈ ℝ#×#! and 𝑊+ . ∈ ℝ#×#! – to project 𝑋"into 𝑑3-dimensional space, and the matrix 𝑊/ ∈ ℝ+#!×# – to project the concatenated single head attention outputs back to the 𝑑-dimensional space. These ma- trices, as well as the seed vector 𝑤', are updated during training. The number of heads, 𝐻, as well as the inputs and outputs dimension, 𝑑, and the attention dimension, 𝑑3, are hyperparameters, that are chosen to maximise model performance on the validation dataset. Matrix transposition, denoted by ⊤, is required to keep the inputs and outputs in column form. Multi-headed attention is used for the STP aggregation layer in this work, as outlined above. It is also an important part of the transformer architecture, that underpins most of the foundation models. RP3Net implementation and training RP3Net was implemented with PyTorch 78. Foundation models were downloaded from HuggingFace79. Training loop was implemented with PyTorch Lightning 80. For models C and D, when the foundation model weights were fine -tuned during training, low -rank adaptation (LoRA81,82) was used. Early stopping criterion was used, whe re training is terminated if AUROC for the validation dataset does not improve for 10 epochs. Exact revisions of software packages and foundation models, as well as training run configurations with hyperparameter values, are available in the RP3Net GitHub Repo. .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 27 Meta label correction with purification data The Meta Label Correction (MLC 68,69) framework utilises a larger, noisy, poor-quality dataset to augment the training process of the model that would normally use only a smaller, clean, high- quality dataset. A separate, “teacher” model is trained to predict the corrected soft label from the noisy data and labels. These corrected labels, along with the clean inputs and labels, are used to train the original model, which in this setup is referred to as the “student” model ( Fig. 2B in the main text). Formally, we can denote the clean dataset as 𝐷 ≡ {𝑋, 𝑦}, where X is the input and y is the label. The noisy dataset can be denoted as 𝐷E ≡ {𝑋F, 𝑦G}, and the corrected labels – as 𝑦c. The student model that predicts the probability of the clean label 𝑦, based on the clean input 𝑋, is denoted as 𝑝4 (𝑋): 𝑃(𝑦|𝑋) ∼ 𝑝4(𝑋), where 𝑤 are the trainable parameters. In the normal deep learning framework this model is trained by minimising the loss function ℒ(𝑤) between the true labels and the predicted labels over the clean dataset: 𝑤∗ = arg  min4ℒ(𝑤), (4) For binary labels that take values of 0 and 1, and cross-entropy (CE) loss, we have ℒ(𝑤) ≡ ℒ(𝑝4 (𝑋), 𝑦) ≡ 𝐶𝐸W𝑦, 𝑝4 (𝑋)X = 𝑦 × logW𝑝4(𝑋)X + (1 − 𝑦) × logW1 − 𝑝4(𝑋)X (5) Simple transfer learning would work by substituting 𝑋F and 𝑦G and for 𝑋 and 𝑦, respectively, in equa- tions (4) and (5). Instead, in the MLC framework, the noisy labels 𝑦G are replaced by the corrected labels 𝑦c, modelled by the teacher model, based on the noisy sequences and the noisy labels: 𝑃W𝑦c|𝑋F, 𝑦GX ∼ 𝑞6W𝑋F, 𝑦GX, with parameters 𝛼. The loss function ℒa between the corrected labels and the noisy input is obtained by substituting the teacher model in place of 𝑦 in the equation (5): .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 28 ℒa(𝑤, 𝛼) ≡ 𝐶𝐸 b𝑞6 W𝑋F, 𝑦GX, 𝑝4W𝑋FXc . (6) The optimal parameters of the student model 𝑤∗ now depend on the parameters of the teacher model 𝛼: 𝑤∗(𝛼) = arg  min4ℒa(𝑤, 𝛼). (7) The optimal value 𝛼∗ needs to be determined, such that the corrected labels 𝑦7 are indeed meaning- ful in the context of the student model, or, in other words, that the student model trained on the noisy data with corrected labels p erforms well on the clean data. This can be done by substituting the optimal value of 𝑤 defined by equation (7) in the equation (4): 𝛼∗ = arg  min6ℒ(𝑤∗(𝛼)). (8) Equations (7) and (8) form the bi-level optimisation problem, that jointly determines the parameters of the teacher and the student models. In the context of this work, the clean data is a union of the AZ and SGC Stockholm data sets, and the noisy data is SGC Toronto with pipeline position labels. The noisy dataset is thus several times larger than the clean one. On each step of the algorithm several gradient steps through the noisy data (Eqn. 7) are followed by a single step through the clean data (Eqn . 8). The number of noisy steps per single clean step is a hyperparameter. Putting it all together, we get Algorithm 1 for com- puting 𝑤∗and 𝛼∗. The teacher parameters 𝛼 at step 𝑡 are updated by computing the gradient 𝑔6 (9) of the clean loss ℒ with respect to (w.r.t) 𝛼. This gradient can be approximated by a formula involving the gradient of the clean loss w.r.t student parameters 𝑤 at step 𝑡 + 1, 𝑔4 (9;%), and the matrices of second derivatives (Hessian matrices) of the noisy loss w.r.t 𝑤 and 𝛼 at previous steps, 𝐻46 (<) = =" =4 =6 ℒaW𝑤(<), 𝛼(<)X: .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 29 𝑔6 (>)2 ≈ −𝜂4𝑔4 (9;%)2 k (1 − 𝜂4 )9?< 9 <@9?A;% 𝐻46 < . (9) This assumes that gradients are represented as column vectors. For the special case of 𝑘 = 1, the Algorithm 1 Bi-level optimisation of teacher and student model parameters via stochastic gradient descent Input: Clean and noisy datasets 𝐷 and 𝐷E; number of training steps T; initial parameters 𝑤(0), 𝛼(0) ; learning rates 𝜂+ , 𝜂,, number of noisy steps per clean step k. Output: Optimised parameters 𝑤(T), 𝛼(T). 1 for t = 0, ..., T – 1 do 2 {𝑋, 𝑦} ← 𝑆𝑎𝑚𝑝𝑙𝑒(𝒟); {𝑋=, 𝑦>} ← 𝑆𝑎𝑚𝑝𝑙𝑒?𝒟@A // sample the minibatches of clean and noisy data 3 𝑤(./') ← 𝑤(.) − 𝜂+∇+ℒE?𝑤(.), 𝛼(.)A // update w by descending the noisy loss w.r.t. w 4 if t mod k = k − 1 then 5 𝑔, = ∇,ℒ H𝑤(./')(𝛼)I // unroll 𝑤./' and approximate the gradient of the clean loss w.r.t. 𝛼 6 𝛼(./') ← 𝛼(.) − 𝜂,𝑔, // update 𝛼 by descending the clean loss w.r.t. 𝛼 7 else 8 𝛼(./') ← 𝛼(.) 9 end if 10 end for sum in equation (9) is reduced to just 𝐻46 (9) ; 𝑔6 (>)2 = −𝜂4𝑔4 (9;%)2𝐻469 . The CE loss allows for efficient computation of the Hessian 𝐻46 , by expressing it point-wise as a product of Jacobians, and averaging over the minibatch: 𝐻46 = 1 𝑁 k[𝐽4(𝑖)]2 & !@% [𝐽6(𝑖)], (10) Here, 𝐽4 (𝑖) is the Jacobian (matrix of derivatives) of the student loss w.r.t 𝑤, and 𝐽6 (𝑖) – the Jaco- bian of the teacher loss w.r.t 𝛼 at input 𝑖, and 𝑁 is the size of the minibatch. Target selection for experimental validation The target set for experimental validation of the model was curated to include viable human drug targets and exclude proteins that are well known from literature to be successfully expressed. We made sure that neither the protein itself, nor its close homologs, have been deposited in the PDB52,53. We have also excluded the target from the validation set if it was referenced from ChEMBL 83. .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 30 OpenTargets84 was used to check for viability of a drug target. Twenty thousand human protein s from UniProt34 were narrowed down to 454 viable targets. These targets were further curated man- ually, to have distribution across different target classes and avoid too many DNA-binding proteins. In the end, 46 targets were selected for experimental validation. Two full length constructs were created per target: one with a TEV -cleavable 6His tag and a GS linker at the N-terminal (MHHHHHHENLYFQGS...), and another one with a GS linker and a 6His tag at the C -terminal (...GSHHHHHH). Soluble production of the full-length constructs was pre- dicted with RP3Net. For the targets where both full-length constructs were predicted to fail to be produced, trimmed constructs were generated by iteratively removing residues from N - and C-ter- mini, with a minimum construct length of 50. Trimmed constructs that were predicted to express successfully were included in the experimental validation set. 70% of the set comprised constructs that were predicted to be produced, with the remaining 30% as negative controls. A total of 97 constructs were available for expression testing after taking into account cost constraints and clon- ing errors. Experimental procedures for construct expression. All sequences were codon optimised for E. coli and synthesized as synthetic genes (Life Technolo- gies Europe BV) and cloned into backbone vector pET24a. One construct failed during the cloning process. Amino acid and nucleotide sequences of synthesized constructs are found in supplementary table x. For small-scale soluble expression screening, the plasmid DNA was transformed into competent phage resistant E. coli BL21(DE3) cells (New England Biolabs #C2527H) in 96 -well PCR plates. The transformation mix was used to directly inoculate 3mL LB media supplemented with 100ug/mL .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 31 kanamycin in 24 deep-well plates and left shaking at 37°C overnight. Protein expression was auto induced in rich ZYP-8012 media supplemented with 100ug/mL kanamycin in 24 deep-well plates, by inoculating 3mL with 50uL pre-culture and left shaking for 3 h at 37°C followed by 24 h at 18°C. After harvest (4000xg, 5min, 4°C), the pellets were lysed with 900uL lysis buffer (40mM HEPES, 300mM NaCl, 5mM imidazole, 10% glycerol, 1mM TCEP, 0.1% DDM, 0.2mg/mL lysozyme, DNAse & protease inhibitors) and freeze -thawed once. The lysate was cleared by centrifugation (4000xg, 30min, 4°C) before subjecting to a one -step Nickel affinity purification using an auto- mated bead-based platform. The protein was captured on the magnetic beads for 30min at 4°C, followed by two wash st eps to wash off unbound proteins (40mM HEPES, 300mM NaCl, 5mM imidazole, 10% glycerol, 1mM TCEP) and eluted in 100uL elution buffer (40mM HEPES, 300mM NaCl, 300mM imidazole, 10% glycerol, 1mM TCEP). 10uL of the elution was loaded onto Nu- PAGE Bis- Tris gels (Invitrogen) together with Novex Pre- stained protein marker (Invitrogen) and 5ug of an internal standard protein. The gels were stained in Der Blaue Jonas (GRP) and analysed using the densitometry software Image Lab (BioRad). The protein yield (mg/L) was estimated from the relative quantity. The “Passed”, “Not Passed” and “Ambiguous” outcome annotations were pro- vided manually by the lab scientist, based on the relative thickness and brightness of gel bands. Annotations from two separate biological replicates are shown in supplementary Table X. Gels from one of the two experiments are shown in Supplementary Figure X. The maximum yield from the two experimental runs was used as the ground truth for the model evaluation. .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 32

1. Zanders, E. D. The Science and Business of Drug Discovery: Demystifying the Jargon: Second Edition. The Science and Business of Drug Discovery: Demystifying the Jargon: Second Edition (Springer International Publishing, 2020). 2. Singh, N. et al. Drug discovery and development: introduction to the general public and patient groups. Frontiers in Drug Discovery 3, 1201419 (2023). 3. Structural Genomics Consortium et al. Protein production and purification. Nat Methods 5, 135–46 (2008). 4. Gräslund, S. et al. The use of systematic N- and C-terminal deletions to promote production and structural studies of recombinant proteins. Protein Expr Purif 58, 210–221 (2008). 5. Burgess-Brown, N. A. et al. Codon optimization can improve expression of human genes in Escherichia coli: A multi-gene study. Protein Expr Purif 59, 94–102 (2008). 6. Hayashi, K. & Kojima, C. pCold-GST vector: A novel cold-shock vector containing GST tag for soluble protein production. Protein Expr Purif 62, 120–127 (2008). 7. Gordon, E. et al. Effective high-throughput overproduction of membrane proteins in Escherichia coli. Protein Expr Purif 62, 1–8 (2008). 8. Haacke, A., Fendrich, G., Ramage, P. & Geiser, M. Chaperone over-expression in Escherichia coli: Apparent in- creased yields of soluble recombinant protein kinases are due mainly to soluble aggregates. Protein Expr Purif 64, 185–193 (2009). 9. Raymond, A. et al. Combined protein construct and synthetic gene engineering for heterologous protein expression and crystallization using Gene Composer. BMC Biotechnol 9, 1–15 (2009). 10. Francis, D. M. & Page, R. Strategies to Optimize Protein Expression in E. coli. Curr Protoc Protein Sci 61, 5.24.1- 5.24.29 (2010). 11. Zhong, N. et al. Optimizing Production of Antigens and Fabs in the Context of Generating Recombinant Antibodies to Human Proteins. PLoS One 10, e0139695 (2015). 12. Cooper, C. D. O. & Marsden, B. D. N- and C-Terminal Truncations to Enhance Protein Solubility and Crystalliza- tion: Predicting Protein Domain Boundaries with Bioinformatics Tools. in Methods in Molecular Biology vol. 1586 11–31 (Humana Press Inc., 2017). 13. Söderberg, J. J., Grgic, M., Hjerde, E. & Haugen, P. Aliivibrio wodanis as a production host: Development of ge- netic tools for expression of cold-active enzymes. Microb Cell Fact 18, 1–16 (2019). 14. Strain-Damerell, C., Mahajan, P., Fernandez-Cid, A., Gileadi, O. & Burgess-Brown, N. A. Screening and Produc- tion of Recombinant Human Proteins: Ligation-Independent Cloning. Methods in Molecular Biology 2199, 23–43 (2021). 15. Burgess-Brown, N. A. et al. Screening and Production of Recombinant Human Proteins: Protein Production in E. coli. Methods in Molecular Biology 2199, 45–66 (2021). 16. Chapple, S. D. & Dyson, M. R. High-Throughput Expression Screening in Mammalian Suspension Cells. Methods in Molecular Biology 2199, 117–125 (2021). 17. Mahajan, P. et al. Screening and Production of Recombinant Human Proteins: Protein Production in Insect Cells.

in Molecular Biology 2199, 67–94 (2021). 18. Mahajan, P. et al. Expression Screening of Human Integral Membrane Proteins Using BacMam. Methods in Molec- ular Biology 2199, 95–115 (2021). 19. Simm, D., Popova, B., Braus, G. H., Waack, S. & Kollmar, M. Design of typical genes for heterologous gene ex- pression. Sci Rep 12, 1–14 (2022). 20. Morão, L. G., Manzine, L. R., Clementino, L. O. D., Wrenger, C. & Nascimento, A. S. A scalable screening of E. coli strains for recombinant protein expression. PLoS One 17, e0271403 (2022). 21. Kurashiki, R. et al. Development of a thermophilic host–vector system for the production of recombinant proteins at elevated temperatures. Appl Microbiol Biotechnol 107, 7475–7488 (2023). 22. Acton, T. B. et al. Preparation of Protein Samples for NMR Structure, Function, and Small-Molecule Screening Studies. Methods Enzymol 493, 21–60 (2011). 23. Edfeldt, K. et al. A data science roadmap for open science organizations engaged in early-stage drug discovery. Na- ture Communications 2024 15:1 15, 1–10 (2024). .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 33 24. Sulkowski, E. Purification of proteins by IMAC. Trends Biotechnol 3, 1–7 (1985). 25. Lin, Z. et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science (1979) 379, 1123–1130 (2023). 26. Rives, A. et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc Natl Acad Sci U S A 118, e2016239118 (2021). 27. Elnaggar, A. et al. ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE Trans Pattern Anal Mach Intell 44, 7112–7127 (2022). 28. Brandes, N., Ofer, D., Peleg, Y., Rappoport, N. & Linial, M. ProteinBERT: a universal deep-learning model of pro- tein sequence and function. Bioinformatics 38, 2102–2110 (2022). 29. Nguyen, E. et al. HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution. ArXiv (2023). 30. Ji, Y., Zhou, Z., Liu, H. & Davuluri, R. V. DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics 37, 2112–2120 (2021). 31. Zhou, Z. et al. DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome. ArXiv (2023). 32. Outeiral, C. & Deane, C. M. Codon language embeddings provide strong signals for use in protein engineering. Nat Mach Intell 6, 170–179 (2024). 33. Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B. & Wu, C. H. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics 31, 926 (2015). 34. Bateman, A. et al. UniProt: the Universal Protein Knowledgebase in 2023. Nucleic Acids Res 51, D523–D531 (2023). 35. Benson, D. A. et al. GenBank. Nucleic Acids Res 41, D36–D42 (2012). 36. Sayers, E. W. et al. GenBank 2024 Update. Nucleic Acids Res 52, D134–D137 (2024). 37. Richardson, L. et al. MGnify: the microbiome sequence data analysis resource in 2023. Nucleic Acids Res 51, D753–D759 (2023). 38. Jumper, J. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589 (2021). 39. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1, 4171–4186 (2018). 40. Hayes, T. et al. Simulating 500 million years of evolution with a language model. bioRxiv (2024) doi:10.1101/2024.07.01.600583. 41. Vaswani, A. et al. Attention Is All You Need. Adv Neural Inf Process Syst 2017-December, 5999–6009 (2017). 42. Rao, R. et al. Evaluating Protein Transfer Learning with TAPE. in Advances in Neural Information Processing Sys- tems 9686–9698 (2019). 43. Dallago, C. et al. FLIP: Benchmark tasks in fitness landscape inference for proteins. in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021). 44. Thumuluri, V. et al. NetSolP: predicting protein solubility in Escherichia coli using language models. Bioinformat- ics 38, 941–946 (2022). 45. Wang, S. & Zhao, H. SADeepcry: a deep learning framework for protein crystallization propensity prediction using self-attention and auto-encoder networks. Brief Bioinform 23, (2022). 46. Xiong, D., U, K., Sun, J. & Cribbs, A. P. PLMC: Language Model of Protein Sequences Enhances Protein Crystalli- zation Prediction. Interdiscip Sci 1–12 (2024) doi:10.1007/s12539-024-00639-6. 47. Li, F.-Z., Amini, A. P., Yue, Y., Yang, K. K. & Lu, A. X. Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models. in Proceedings of the 41st International Conference on Machine Learning (eds. Salakhutdinov, R. et al.) vol. 235 27351–27375 (PMLR, 2024). 48. Berman, H. M. et al. Protein Structure Initiative - TargetTrack 2000-2017 - all data files. https://zenodo.org/rec- ords/821654 https://doi.org/10.5281/zenodo.821654 (2017) doi:10.5281/zenodo.821654. 49. Gabanyi, M. J. et al. The Structural Biology Knowledgebase: a portal to protein structures, sequences, functions, and methods. J Struct Funct Genomics 12, 45–54 (2011). 50. Protein production data from the SGC. Preprint at https://www.ebi.ac.uk/biostudies/SGC/studies/S-BSST681 .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 34 (2022). 51. Savitsky, P. et al. High-throughput production of human proteins for crystallization: The SGC experience. J Struct Biol 172, 3–13 (2010). 52. Berman, H. M. The Protein Data Bank. Nucleic Acids Res 28, 235–242 (2000). 53. Burley, S. K. et al. RCSB Protein Data Bank (RCSB.org): delivery of experimentally-determined PDB structures alongside one million computed structure models of proteins from artificial intelligence/machine learning. Nucleic Acids Res 51, D488–D508 (2023). 54. Mizianty, M. J. & Kurgan, L. Sequence-based prediction of protein crystallization, purification and production pro- pensity. Bioinformatics 27, i24–i33 (2011). 55. Wang, H. et al. PredPPCrys: Accurate Prediction of Sequence Cloning, Protein Production, Purification and Crys- tallization Propensity from Protein Sequences Using Multi-Step Heterogeneous Feature Fusion and Selection. PLoS One 9, e105902 (2014). 56. Wang, H. et al. Crysalis: an integrated server for computational analysis and design of protein crystallization. Sci Rep 6, 21383 (2016). 57. Zhu, Y.-H. et al. Accurate multistage prediction of protein crystallization propensity using deep-cascade forest with sequence-based features. Brief Bioinform 22, (2021). 58. Hon, J. et al. SoluProt: prediction of soluble protein expression in \textit{Escherichia coli}. Bioinformatics 37, 23– 28 (2021). 59. Friedman, J. H. Greedy function approximation: A gradient boosting machine. The Annals of Statistics 29, (2001). 60. Sormanni, P., Aprile, F. A. & Vendruscolo, M. The CamSol Method of Rational Design of Protein Mutants with Enhanced Solubility. J Mol Biol 427, 478–490 (2015). 61. Sormanni, P., Amery, L., Ekizoglou, S., Vendruscolo, M. & Popovic, B. Rapid and accurate in silico solubility screening of a monoclonal antibody library. Scientific Reports 2017 7:1 7, 1–9 (2017). 62. Yuan, Q. et al. GPSFun: geometry-aware protein sequence function predictions with language models. Nucleic Ac- ids Res 52, W248–W255 (2024). 63. Zhang, X. et al. PLM_Sol: predicting protein solubility by benchmarking multiple protein language models with the updated Escherichia coli protein solubility dataset. Brief Bioinform 25, (2024). 64. Basafa, M., Hashemi, A. & Behravan, A. Optimizing recombinant antibody fragment production: A comparison of artificial intelligence and statistical modeling. Biotechnol Appl Biochem 71, 1094–1104 (2024). 65. Zhang, J. H., Shan, L. L., Liang, F., Du, C. Y. & Li, J. J. Strategies and Considerations for Improving Recombinant Antibody Production and Quality in Chinese Hamster Ovary Cells. Front Bioeng Biotechnol 10, 856049 (2022). 66. Buterez, D., Janet, J. P., Kiddle, S. J., Oglic, D. & Liò, P. Graph Neural Networks with Adaptive Readouts. in Ad- vances in Neural Information Processing Systems (eds. Koyejo, S. et al.) vol. 35 19746–19758 (Curran Associates, Inc., 2022). 67. Lee, J. et al. Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks. in Pro- ceedings of the 36th International Conference on Machine Learning (eds. Chaudhuri, K. & Salakhutdinov, R.) vol. 97 3744–3753 (PMLR, 2019). 68. Taraday, M. K. & Baskin, C. Enhanced Meta Label Correction for Coping with Label Corruption. ArXiv (2023). 69. Zheng, G., Awadallah, A. H. & Dumais, S. Meta Label Correction for Noisy Label Learning. in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) (2021). 70. Wilkinson, M. D. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016). 71. Hayes, T. et al. Simulating 500 million years of evolution with a language model. Science (1979) 387, 850–858 (2025). 72. ESM Team. ESM Cambrian: Revealing the mysteries of proteins with unsupervised learning. Evolutionary Scale Website https://www.evolutionaryscale.ai/blog/esm-cambrian (2024). 73. Steinegger, M. & Söding, J. MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nat Biotechnol 35, 1026–1028 (2017). 74. Sankar, K. et al. A Descriptor Set for Quantitative Structure‐property Relationship Prediction in Biologics. Mol In- form 41, (2022). .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint 35 75. Linding, R. et al. Protein Disorder Prediction. Structure 11, 1453–1459 (2003). 76. Wang, S., Li, W., Liu, S. & Xu, J. RaptorX-Property: a web server for protein structure property prediction. Nucleic Acids Res 44, W430–W435 (2016). 77. Mirdita, M. et al. Uniclust databases of clustered and deeply annotated protein sequences and alignments. Nucleic Acids Res 45, D170–D176 (2016). 78. Paszke, A. et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. in Advances in Neural Information Processing Systems 32 (Curran Associates, Inc., 2019). 79. Wolf, T. et al. Transformers: State-of-the-Art Natural Language Processing. in EMNLP 2020 - Conference on Em- pirical Methods in Natural Language Processing, Proceedings of Systems Demonstrations 38–45 (Association for Computational Linguistics (ACL), 2020). doi:10.18653/V1/2020.EMNLP-DEMOS.6. 80. Falcon, W. & The PyTorch Lightning team. PyTorch Lightning. Preprint at (2019). 81. Yu, Y. et al. Low-Rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recogni- tion. in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) 1–8 (IEEE, 2023). doi:10.1109/ASRU57964.2023.10389632. 82. Mangrulkar, S. et al. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. Preprint at https://github.com/huggingface/peft (2022). 83. Zdrazil, B. et al. The ChEMBL Database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods. Nucleic Acids Res 52, D1180–D1192 (2024). 84. Ochoa, D. et al. The next-generation Open Targets Platform: reimagined, redesigned, rebuilt. Nucleic Acids Res 51, D1353–D1359 (2023). .CC-BY 4.0 International licenseavailable under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (whichthis version posted May 16, 2025. ; https://doi.org/10.1101/2025.05.13.652824doi: bioRxiv preprint