Comprehensive Evaluation of AlphaFold-Multimer, AlphaFold3 and ColabFold, and Scoring Functions in Predicting Protein-Peptide Complex Structures

14 Determining the three-dimensional structures of protein-peptide complexes is crucial for 15 elucidating biological processes and designing peptide-based drugs. Protein-peptide docking has 16 become essential for predicting complex structures. AlphaFold-Multimer, ColabFold and 17 AlphaFold3 provided groundbreaking tools to enhance the protein-peptide docking accuracy. 18 This study evaluates these three tools for predicting protein-peptide complex structures using 19 Template-Based (TB) and Template-Free (TF) methods. AlphaFold-Multimer excels in TB 20 predictions and performs moderately in TF scenarios in the prediction pool, but TF outperforms 21 TB in the first-ranked models. ColabFold demonstrates versatility in both TB and TF settings. 22 AlphaFold3 generates high-quality structures for more proteins, but the medium accuracy is not 23 as good as AlphaFold-Multimer using a large model pool. We also assessed the performance of 24 various scoring functions in ranking predicted protein-peptide complex structures. While the 25 scoring function built in AlphaFold demonstrates the best performance, some other scoring 26 functions, e.g., FoldX-Stability and HADDOCK-mdscore, provide complementary values. The 27 findings suggest the potential for enhancing scoring functions targeting AlphaFold-based 28 predictions by combining multiple scoring functions or using a consensus approach from many 29 prediction models. 30 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 2 31

AlphaFold-Multimer; ColabFold; AlphaFold3; protein-peptide complex structure; 32 scoring functions 33 34

35 Protein-peptide interactions have been recognized as playing crucial roles in a wide range of 36 biological processes. The participating peptides typically consist of fewer than 50 residues. 37 Approximately 15-40% of protein-protein interaction activities are estimated to involve protein-38 peptide complexes[1]. Consequently, peptides have increasingly attracted attention as promising 39 binding candidates for drug design, owing to their biochemical properties and relatively low 40 toxicity[2]. Acquiring three-dimensional (3D) structural data of protein complexes is crucial for 41 understanding their functional roles and malfunctions associated with diseases[3–6]. This 42 understanding is essential for elucidating molecular mechanisms underlying protein-peptide 43 recognition and advancing the development of peptide-based therapeutics[7]. To experimentally 44 determine 3D structures of protein-peptide complexes, common techniques include cryo-electron 45 microscopy and X-ray crystallography. While effective, these experimental methods often 46 require extensive labour and costs[8]. Additionally, the experimental characterization of protein-47 peptide complex structures is considerably hindered by the dynamic and transient nature of some 48 interactions[9]. Due to these challenges, computational methods, such as protein-peptide 49 docking, offer valuable information for protein-peptide complex structures. Docking algorithms 50 fall into two main categories: template-based (TB) and template-free (TF)[1, 10]. In TB docking, 51 known complex structures are used as templates to predict interactions between proteins and 52 peptides, which is effective when similar complex structures are accessible. In contrast, TF 53 docking, also referred to as ab initio docking, predicts interactions based solely on the physical 54 and chemical properties of the molecules[11]. 55 Previously, the prediction of protein complex structures was primarily achieved through 56 geometry docking[12]. However, the advent of artificial intelligence (AI), particularly through 57 applications like AlphaFold-Multimer (AFM)[13], has recently revolutionized the field by 58 cofolding protein-protein and protein-peptide complexes[14]. ColabFold (CF) simplifies the use 59 of AlphaFold and AlphaFold-Multimer with Google Colab, and it predicts five TB or TF models 60 for each protein complex. CF greatly improves the speed and ease of predicting protein 61 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 3 complexes, thus facilitating fast advancements in structural biology[15]. AlphaFold3 (AF3), the 62 latest version of AlphaFold, is accessible through its dedicated website. It is capable of 63 predicting protein complex structures within five models[16]. Numerous studies have assessed 64 the performance of AlphaFold; however, they concentrated on single protein or protein-protein 65 complexes[17, 18] (including antibody-antigen structures[19, 20]), rather than on protein-peptide 66 complexes. In this study, we focused on assessing the predicted structures of protein-peptide 67 complexes by AFM, CF, and AF3. 68 Another significant challenge in computational methods is the accurate identification of 69 near-native protein-peptide interaction conformations from a vast array of generated models, a 70 process commonly known as scoring[21]. Several methods have been introduced to rank protein 71 complex structures[22]. The evolution of scoring functions in protein structure prediction has 72 shifted from physics-based methods—exemplified by HADDOCK[23]–and empirical-based 73

like AutoDock Vina[24], to knowledge-based potentials, such as FoldX[25]. Physics-74 based scoring functions began with force-field methods based on fundamental physical 75 principles, incorporating solvent effects and charge features, but these methods are 76 computationally intensive[26]. On the other hand, empirical methods use simplified models of 77 molecular interactions derived from experimental data, resulting in faster computations[27]. 78 Knowledge-based functions use statistical potentials derived from known structural data, 79 balancing accuracy and speed[28]. 80 Scoring functions have further advanced with the adoption of deep learning (DL) strategies, 81 including DOVE[29], GNN_DOVE[30], DeepRank[31], and the graph neural-network-based 82

such as DeepRank-GNN[32], DeepRank-GNN-esm[33], and Interpeprank[34]. DL-83 based scoring functions have been reviewed and assessed in various studies, such as those 84 involving antibody-antigen and enzyme-inhibitor interactions[28]. Other studies have offered a 85 comprehensive comparison of DL-based scoring functions for virtual screening[35] and ranking 86 of the predicted protein-ligand complexes[36]. While numerous scoring functions for protein 87 complex structure assessment have been developed, their performance depends on the methods 88 of predicting protein-peptide complex structures, and systematic evaluation focused on the 89 AlphaFold family is notably lacking. This gap necessitates a thorough evaluation of leading 90 computational tools like AFM and CF, especially in the context of protein-peptide structure 91 predictions using multiple quality and scoring functions. A comparative analysis of scoring 92 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 4 functions will illuminate the relative strengths and pinpoint areas for potential enhancement in 93 the methods applied. 94 In this study, to systematically evaluate the performance of prediction tools and scoring 95 functions, we compiled a dataset of 60 native protein-peptide structures from the Protein Data 96 Bank (PDB)[37, 38]. We used AFM to predict 1,000 protein-peptide structures, and we 97 employed CF was employed to predict five complex structures using both TB and TF 98 approaches. Additionally, we used AF3 was used to predict five models. This dataset served as a 99

to assess the effectiveness of these three prediction tools and their scoring functions. 100 101

102 We conducted a comprehensive two-part evaluation and analysis of protein-peptide complex 103 structures (Figure 1) . The first part focused on assessing the quality of predicted structures, 104 while the second part aimed at analysing various scoring functions. To accomplish this, we 105 employed two quality measures, DockQ[39] and MolProbity[40, 41], and some distinct scoring 106 functions. The insights derived from this study have provided a nuanced understanding of the 107 accuracy, capabilities and limitations of AFM, CF and AF3, as well as of the efficacy of each 108 scoring function employed. 109 Quality Analysis 110 In this section, we provide a detailed analysis using DockQ to assess prediction accuracies 111 against our ground-truth dataset for three leading protein structure prediction tools—AFM, CF, 112 and AF3—across 60 samples. Additionally, we used Molprobity to diagnose structural problems 113 in the predicted models. For each protein, AFM produced 1,000 predictions, while CF and AF3 114 produced 5. We compared the predicted complex structural models with their corresponding 115 native structures using the DockQ metrics. We segmented the datasets into four categories based 116 on DockQs prediction quality: High, Medium, Acceptable, and Incorrect. This classification 117 evaluates the practicality of the predicted structures for subsequent biological and biochemical 118 analyses. We used MolProbity to assess the quality of complex structures in terms of their 119 potential issues without knowing the native structures. 120 DockQ 121 DockQ is a scoring tool used to evaluate the quality of protein-protein docking models by 122 comparing predicted structures with reference (native) structures. It provides a single metric that 123 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 5 combines interface root-mean-square deviation (iRMSD), ligand RMSD (LRMSD), and fraction 124 of native contacts (Fnat), facilitating the assessment of docking accuracy and model quality[39, 125 42]. DockQ scores, ranging from 0 to 1, assess the accuracy of predicted models against native 126 PDB structures. Scores above 0.23 are considered acceptable by the CAPRI standards[43] 127 (Figure 1). These scores indicate the quality of the interaction interface between the protein and 128 the peptide relative to the native structure. Figure 2 illustrates the evaluation and analysis 129 outcomes for TB and TF methods, depicting the qualities of the top-ranked models by AFM, CF 130 and AF3. 131 132 Figure 1. Study overview. a. Data preparation: The dataset comprised post-12/01/2023 protein-peptide 133 complexes from PDB with standard amino acids in peptides of 3–50 residues, unencountered by AF’s 134 training dataset. We removed all sequences containing nonstandard residues from the dataset. Data 135 preprocessing: We refined the dataset using CD-HIT to eliminate sequences with over 40% redundancy, 136 resulting in 60 diverse protein-peptide complex structures for our evaluation pipeline. All structures of 137 sample sequences were predicted by AFM, CF and AF3. b. We categorized the predicted structures into 138 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 6 template-based (TB) and template-free (TF) models based on the prediction method. We subsequently 139 evaluated and analysed them using various quality and scoring functions. c. We calculated the Spearman 140 correlation coefficient for the rankings produced by all scoring functions compared to the ranking by 141 DockQ. We defined the loss parameter to evaluate the accuracy of each scoring function in identifying the 142 most near-native structure from the pool. 143 144 With AFM, the TF approach primarily produced structures in the Medium category (Figure 2a). 145 The TB approach generated more high-quality predictions than TF, demonstrating the critical 146 role of template information in achieving optimal prediction accuracy. AF3 outperformed both 147 AFM and CF in predicting more accurate structures, with approximately 35% of the total 148 samples falling into the High category. CF’s performance, on the other hand, was more evenly 149 distributed between the TB and TF approaches. The TF method exhibited a slight decline in the 150 High category performance, achieving 20.3% vs. 21.3% in TB. Yet, it maintained similar 151 percentages across the remaining categories (30.3% TF vs. 30.6% TB Medium, 27.3% TF vs. 152 27.7% TB Acceptable, and 22.0% TF vs. 20.3% TB Incorrect). This pattern suggests a minor 153 difference between TB and TF approaches within CF. 154 The side-by-side comparison of AFM and CF unveils significant insights into their predictive 155 strengths. AFM demonstrated superior performance in predicting high and medium quality 156 structures compared to CF (26.5% AFM-TB vs. 21.3% CF-TB and 22.5% AFM-TF vs. 20.3% 157 CF-TF in the High category, 30.1% AFM-TB vs. 30.6% CF-TB (almost similar) and 35.2% 158 AFM-TF vs. 30.3% CF-TF in Medium category), which indicates that the convenience of using 159 CF has a cost. These results underscore the importance of selecting AFM to predict high-quality 160 structures over CF if possible. Nevertheless, the AFM’s performance is significantly surpassed 161 by that of AF3, which offers both high quality and convenience. 162 In protein structure prediction, importance is often placed on the quality of the first-ranked 163 structures selected by the prediction tools without knowing the native structures, as these 164 predictions are usually used in subsequent biological analyses and functional studies. 165 Consequently, we compared the quality of first-ranked protein structure predictions by AFM and 166 CF by employing DockQ scores across TB and TF approaches ( Figure 2b ). For AFM, TF 167 slightly outperforms TB in the High category (33.3% TF vs. 31.6% TB). Interestingly, although 168 TB's average performance is better than TF for all prediction models, the 1,000 models predicted 169 by TF are expected to be more diverse than TB, and AFM has a strong capacity to select the best 170 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 7 model from the pool. A substantial consistency between TB and TF approaches is demonstrated 171 by CF, with a near-equal distribution of High and Medium quality predictions observed, 172 although its TB approach is slightly better. This may be because CF predicted a small number of 173 models in both TB and TF, resulting in little diversity in the model pool. AF3 generated the most 174 high-quality structures, with 35.6% of the first-ranked models falling into this category, followed 175 by 25.4% in the Medium category. 176 Figures 2cd, 2ef and 2g illustrate the first-ranked protein-peptide structures predicted by AFM, 177 CF and AF3, respectively, using both TB and TF approaches for the same sample (8EBL used as 178 an example). To analyse the prediction accuracy of each method, we calculated global 179 superposition metric template modelling score (TM-score)[44] and RMSD[45] metrics for the 180 entire complex structure, and the protein and peptide components separately. Based on the 181 results, the accuracy of protein-peptide complex structure predictions by both AFM/AF3 and CF 182 was High, with TM-scores values around 0.97–0.99 and RMSD values around 0.67–1.23 in both 183 TB and TF approaches. For the protein the TM-score values were within the range of 0.98 to 184 0.99. However, RMSD values varied, with the lowest associated with AF3 predictions and the 185 highest recorded for AFM using the TB approach. The predicted peptide structure has worse 186 quality than the protein, with TM-scores decreased to 0.37/0.66/0.33 for AFM-TB, AF3 and CF, 187 respectively. AF3 outperformed both AFM and CF in terms of TM-score for the predicted 188 peptide. 189 The distribution of all DockQ parameters, including Fnat, iRMSD, ligand LRMSD, DockQ, and 190 intersection over union (IOU), was analysed in Supplementary Figs. S1-S5. While AF3 has the 191 highest proportion of high-quality scores (DockQ ≥ 0.80), its average prediction accuracy, as 192 indicated by the median and quartiles in Supplementary Fig. S4, is less favourable than other 193 tools. Additionally, the radar plot in Figure 3a analysed the median values of the DockQ 194 parameters. The superior performance of AFM-TB and AFM-TF in producing high-quality and 195 consistent protein-peptide docking predictions is underscored by this analysis. 196 197 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 8 198 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 9 Figure 2. Evaluation of predicted protein-peptide complex structures using AFM, CF and AF3. a. 199 Percentage of predicted structures by all three tools in different categories of DockQ. b. Percentage of 200 first-ranked predicted structures by all three tools in different categories of DockQ. c., d., e., f. 201 Comparative analysis of protein-peptide complex predictions for sample 8EBL by AFM and CF, using 202 TB and TF methods, respectively, assessed against the ground truth (GT) structure with TM-score and 203 RMSD metrics. g. Analysis of predicted structure with AF3 against the GT. 204 205 Figure 3. Radar plots for the prediction-accuracy strengths and the structure-quality strengths of 206 various methods on the first-ranked predicted protein-peptide models. In each metric, the best 207 performer takes the position of the most outer circle, the worst one takes the most inner circle, and the 208 others take positions using linear scales of the metric. a. The radar plot of median values for Fnat, 209 iRMSD, LRMSD, IOU, and DockQ parameters across five methods along with GT, which has a score of 210 1 for all metrics. AFM-TB and AFM-TF demonstrate high accuracy and consistency in predictions, while 211 AF3 performs moderately. CF-TB and CF-TF have lower prediction quality, highlighting the superiority 212 of AFM models. b. The radar plot of MolProbity metrics. The Clashscore is assessed by its median value 213 of the MolProbity parameter, the smaller the better. Rama_Z scores are assessed by their root mean 214 square values of the MolProbity parameters, also the smaller the better. GT shows the best overall 215 performance, with the lowest RMS values across Ra ma_Z metrics. AF3 struggles with higher RMS 216 values but performs well in Twisted Peptides and Clashscores. CF-TB and CF-TF excel in specific areas. 217 GT remains the most reliable for structural quality, with AF3 and CF-TF showing targeted strengths and 218 weaknesses. 219 MolProbity 220 MolProbity is a popular validation tool for assessing protein structure quality through 221 stereochemistry and atomic interactions. It examines amino-acid residue conformations, angles, 222 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 10 and bond lengths, pinpointing deviations from standard values. In this study, we used the 223 MolProbity score, along with analyses of twis ted peptides and cis non-proline conformations, 224 were used to evaluate protein-peptide complex structures. In Figure 4a, the framework for our 225 analysis is provided by MolProbity scores, which offer a detailed look at the quality of structural 226 predictions from AFM, CF, and AF3. These scores inform us of the accuracy distribution within 227 the TB or TF approach (for AFM and CF) and AF3, with higher structural quality indicated by 228 lower MolProbity scores. The MolProbity score combines the Clashscore, rotamer, and 229 Ramachandran evaluations into a single comprehensive score[41]. 230 The AF3 model demonstrates superior performance, with 30.5% of its predictions falling within 231 the 1.0–1.2 range. Additionally, 22.0% and 25.4% of predictions fall within the 1.2–1.4 and 1.4–232 1.6 ranges, respectively. AF3’s minimal representation in high MolProbity ranges illustrates its 233 effectiveness in generating high-quality structures. In contrast, the AFM-TB and AFM-TF 234 models show a broader distribution of MolProbity scores. AFM-TB's highest percentage (28.3%) 235 falls within the 1.6–1.8 range, while AFM-TF has a significant concentration (25.0%) in the 1.8–236 2.0 range. Both models also have a substantial presence in the 2.0–2.2 range, indicating that they 237 generate lower-quality structures more frequently than AF3. 238 The CF-TB and CF-TF models display even more varied distributions, with notable peaks in 239 higher MolProbity score ranges. CF-TB has significant proportions in the 1.6–1.8 (21.7%) and 240 2.0–2.2 (25.0%) ranges, while CF-TF peaks in the 2.0–2.2 range (28.3%), with slightly worse 241 quality than CF-TB. These models exhibit substantial percentages in ranges beyond 2.2, 242 indicating a higher incidence of lower-quality prediction. Overall, AF3 consistently produces 243 more high-quality predictions, while AFM-TB and AFM-TF show broader quality distributions. 244 The Colab models display the most varied distributions, with the poorest overall prediction 245 quality. 246 Significant differences between computational prediction methods were revealed by our 247 comparative evaluation of twisted peptides within protein-peptide complexes. By considering all 248 models of all samples (Figure 4b) , only 1.7% of the ground truth (GT) structures contain 249 twisted peptides. AF3, with a low percentage of 2.4%, performs closest to the GT, suggesting 250 high quality overall with a much smaller percentage than other methods. The percentage for CF-251 TB is 18.7%, which is large but still smaller than the percentages for AFM-TB, AFM-TF, and 252 CF-TF, which range from 21% to 26% (specifically, 25.2%, 26.2%, and 21.3%, respectively). It 253 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 11 is also important to note that AFM-TF and AFM-TB each have 1,000 predictions, whereas AF3 254 and CF each have only five models in their predictions, potentially influencing the results. 255 For the first-ranked models, comparing against the baseline, i.e., 1.7% of the GT structures with 256 twisted peptides, AFM-TB has 11.7%, followed by AFM-TF and CF-TB with 15.0% and 16.7%, 257 respectively. The lowest quality first-ranked models are produced by CF-TF, with the highest 258 percentage at 26.7%. Interestingly, a 0.0% is observed for AF3, implying that no twisted 259 peptides were identified in first-ranked models. We note that other than those generated by 260 AFM-TF, first-ranked models have fewer twisted peptides than all other models on average. 261 We conducted a comparative analysis on the presence of cis non-proline peptides ( Figure 4c, 262 with an example shown in Supplementary Fig. S6). As a baseline for evaluation, 3.4% of the GT 263 structures contain cis non-proline peptides. Models by AFM-TB and AFM-TF contain higher 264 percentages of cis non-proline peptides, at 7.1% for all models (1.7% for the first models) and 265 5.9% for all models (1.7% for the first models), respectively. CF-TB predicted 3.3% for both all 266 models and the first models, closely aligning with the GT structures of 3.4%, while CF-TF 267 models have a lower percentage of 2.7% for all models (5.0% for the first models). AF3 models 268 have more cis non-proline peptides, with a rate of 6.7% for both all models and the first models. 269 According to our evaluation, all observed cis non-proline peptides occurred within the proteins, 270 without any occurrence in the peptides. 271 All the distributions of Ramachandran scores[46] obtained from the MolProbity tool for first-272 ranked models are shown in Supplementary Figs. S7-S10, illustrating the distribution of Rama-273 Z-whole scores, Rama-Z-helix scores, Rama-Z-sheet scores, and Rama-Z-loop scores, 274 respectively. Rama-Z scores ranging from 0 to ±1 indicate that the dihedral angles are very 275 similar to those found in GT structures. Rama-Z scores at ±1 to ±2 show that the models slightly 276 deviate from the normal dihedral angle distributions but are still acceptable. Rama-Z scores 277 beyond ±2 mark significant deviations from the expected values, suggesting potential issues with 278 the protein models, such as unusual conformations that could be errors. All the methods 279 delivered similar performance in these Rama-Z scores, significantly worse than the GT. In 280 particular, the GT has a significantly different distribution of Rama-Z-helix scores from other 281 methods, with more models with Rama-Z-helix scores in the range 0 to ±1 than other methods 282 (mostly negative values for GT while mostly positive values for other methods). This suggests 283 that the helices generated by all models are not quite protein-like. AF3 has better Rama-Z-helix 284 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 12 and Rama-Z-sheet scores, but worse Rama-Z-loop scores than other methods. Additionally, the 285 distribution of Clashscore values for first-ranked models across different protein modelling 286

is shown in Supplementary Fig. S11. AF3 shows the least clashes among all other 287 models, close to the GT, while other methods have significantly higher clashes. In Figure 3b, the 288 radar plot related to the MolProbity parameters displays the strengths of each model for all first-289 ranked models across all protein-peptide samples. AF3 demonstrates the best overall geometry 290 for twisted peptides and Clashscores. 291 Figure 5 presents example protein-peptide structures predicted by various methods together with 292 their MolProbity metrics. Figure 5a displays the hydrogen bonding interactions between the 293 protein and peptide in the 8HLO sample. It demonstrates that the hydrogen bonding in the native 294 structure is stronger than predicted structures by AFM-TB, AFM-TF and AF3. In Figure 5b , the 295 cis non-proline conformations in the peptide portion of the 8EBL sample are depicted. As 296 illustrated, the AFM-TB models exhibit a cis non-proline conformation, indicated by ω° values 297 (a cis peptide is characterized by an ω angle between −30° and +30°, whereas a trans peptide is 298 defined by an ω angle greater than +150° or less than −1 50°[47]), whereas this conformation is 299 absent in the native, AFM-TF and AF3 peptide structures. In Figure 5c, the presence of twisted 300 peptides ( ω angles that deviate more than 30° from planar[47]) in the peptide portion of the 301 7Z7C sample is shown for both AFM-TB and AFM-TF peptide structures, while AF3 does not 302 show any twisted peptide in that region. In this instance, the AFM-TB model exhibits more 303 twisted peptides than the AFM-TF models, despite the benefit of using templates for predicting 304 the structures. In Figure 5d, the clashes in the peptide portion of the 8AFI sample are depicted. 305 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 13 306 Figure. 4. Molprobity, twisted peptides, cis non-proline peptide evaluation. a. MolProbity scores for 307 AFM and CF using TB and TF approaches and AF3, higher structural quality indicated by lower 308 MolProbity scores. b., c. Percentages of twisted peptides and cis non-proline conformations observed in 309 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 14 the predicted structures for all samples and the first-ranked model, using AFM, CF and AF3. Predicted 310 structures with fewer twisted peptides and cis non-proline conformation, or those closer to the GT 311 models, are preferred. 312 313 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 15 Figure 5. Hydrogen bonding, cis non-proline, twisted peptides, clash evaluation examples. a. 314 Hydrogen bonding across protein-peptide interface of 8HLO. b. Cis non-proline in 8EBL. An example of 315 a predicted structure ranked 511 was used for both TB and TF in this figure (top-ranked structures do not 316 have any cis non-proline peptides). Corresponding residues through all structures are indicated by pink 317 and cyan colours. c. Twisted peptide on peptide region of 7Z7C GT and predicted structures. Twisted 318 peptides, marked by unusual omega ( ω° ) dihedral angles, deviate from standard cis ( ω ≈ 0) or trans ( ω≈ 319 180°) conformations. These deviations may indicate structural anomalies or prediction errors. ω angles 320 such as 92.97°, 95.28°, -148.03 °, 142.33° on TB model, or 149.75° on TF model are neither close to the 321 cis nor the trans conformations, indicating twisted peptide bonds. d. The clashes on the peptide site of the 322 complexes, along with their corresponding Clashscore for the entire complex structure. The AF3 model 323 exhibited the fewest clashes on the peptide site in the 8AFI sample. 324 325 Evaluation of Scoring Functions 326 In this section, we demonstrate the effectiveness of various scoring functions in ranking AFM-327 predicted protein-peptide complex structures. Using the Spearman correlation coefficient[48], we 328 assessed the alignment of each scoring function with DockQ and other scoring functions. To 329 further refine the analysis, all TB and TF predicted structures by AFM were consolidated to 330 create a unique and comprehensive dataset (2,000 models for each protein-peptide complex). 331 This approach enabled the identification of a set of common candidates, selected across different 332 scoring functions, which potentially outperforms others in identifying near-native structures. 333 Spearman correlation matrix between scoring functions 334 In Figure 6a,b, Spearman correlation matrices are utilized to elucidate the relationships between 335 various scoring functions under both TB and TF modelling approaches. These heatmap plots 336 illustrate that each scoring function employs a distinct strategy and algorithm to score the 337 protein-peptide structures in our dataset. In both TB and TF models, the AlphaFold-Multimer 338 scoring function (AFM-Score) demonstrate moderate correlations with tools such as PyRosetta, 339 FoldX-Stability, and HADDOCK. However, the correlations observed among HADDOCK’s 340 variants are surprisingly modest, suggesting subtle methodological differences despite sharing a 341 common computational framework. The minimal correlations observed between tools may mean 342 they are complementary and have the potential to be combined for better ranking performance. 343 The last row in both heatmap plots displays the mean correlation values of each scoring function 344 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 16 with all other functions, excluding itself. This metric provides insight into the overall agreement 345 and consistency of each scoring function with the others, offering a comprehensive view of their 346 relative performance and reliability. We observe that AFM-Score exhibits the highest mean 347 values in both the TB and TF approaches, followed by PyRosetta and HADDOCK scoring 348 functions. This may indicate that the AFM-Score captures the strengths of other scoring 349 functions in the most comprehensive way. 350 Evaluation of scoring function performance via DockQ score loss 351 To precisely evaluate the performance of each scoring function in finding the most near-native 352 structure, we introduced a parameter to represent the difference between the DockQ scores of 353 two models: (1) the top-ranked model, found by DockQ, and (2) the top-ranked model found by 354 each scoring function. This parameter, termed LOSS, is defined as below: 355 356 LOSS = (DockQ score of the first-ranked structure found by DockQ) - (DockQ score of the first-357 ranked structure found by a scoring function) 358 359 We applied this analysis to both TB and TF methods. The results for TB and TF methods are 360 displayed in Figure 6c,d, with all boxes sorted based on the mean LOSS value. The red line in 361 each box represents the median value. The deviations in DockQ scores among various scoring 362 functions for TB and TF predicted structures were scrutinized and quantified by the LOSS 363 parameter. This metric provides insight into the precision of each function in ranking the top-364 scoring structures compared to the DockQ rankings. In Figure 6c, using box plots for TB protein 365 structure predictions, the LOSS parameter across various scoring functions is examined in 366 comparison to DockQ benchmarks on TB models. AFM-Score and HADDOCK-emscore exhibit 367 the lowest mean LOSS values, at 0.08 and 0.09, respectively, indicating high accuracy and 368 minimal variability. In contrast, DeepRank-GNN-esm and GNN_DOVE show the highest mean 369 LOSS values, around 0.16, with significant variability. PyRosetta and FoldX-Stability 370 demonstrate intermediate performance. 371 In Figure 6d, the LOSS value on TF models is illustrated. DeepRank-GNN-esm and GNN_Dove 372 have the highest mean LOSS values of 0.17, suggesting less accurate rankings, with IQRs 373 (Interquartile Ranges, representing the middle 50% of the data) of 0.27 for DeepRank-GNN-esm 374 and 0.28 for GNN_DOVE, reflecting significant variability. PyRosetta and FoldX-Interaction 375 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 17 show intermediate performance. Like in the TB case, AFM-Score still has the lowest mean 376 LOSS value of 0.10 and an IQR of 0.10 and an IQR of 0.11. This LOSS value is higher than the 377 one in TB (0.08), probably because the TF models are more diverse. The second-best performer 378 is FoldX-Stability, with a mean LOSS value of 0.11 and an IQR of 0.10. HADDOCK-mdscore 379 has the third-best performance, exhibiting the lowest mean LOSS values of 0.12. As shown in 380 Figure 6c,d, AFM-Score has excelled in structure prediction and aligning with the GT samples 381 based on the DockQ score. However, it has limitations, with eight outliers in either TB or TF 382 models. These outliers include 7udl, 7yue, 7ue2, 8ebl, 8ahs, and 8c2p in both TB and TF, as well 383 as 8fk3 and 7zx4 in TB, and 8b58 and 8dgm in TF. Outliers in AFM-Score predictions can be 384 attributed to factors such as protein complexity, data quality, model limitations, dynamics, and 385 errors in reference structures. Identifying these outliers aids in refining predictive models. 386 This comprehensive analysis underscores the diversity in performance among the scoring 387 functions, revealing that while several tools perform well, their consistency differs. The variation 388 in the LOSS values, as indicated by their IQRs and outliers, reflects the complexities of AFM 389 protein structure prediction. Nevertheless, the comparable median LOSS values of several 390 scoring functions to AFM-Score illustrate that multiple methods have the potential to offer 391 similar predictive capability, though with varying levels of dependability. For example, while the 392 medians of the three scoring functions, PyRosetta, FoldX-Stability and HADDOCK-emscore in 393 the TF plot are lower and better than those of AFM-Score, the variability of AFM-Score remains 394 better compared to those three. The details of the scoring functions values are shown in 395 Supplementary Tables S2 and S3. 396 It is interesting to note that AF3’s ranking of the five models may not indicate their relative 397 performance, as shown in Supplementary Fig. S12. AF predicts many nearer-native structures. 398 The number of top 10 ranking positions for predicted structures with AF3 is calculated for each 399 model as follows: First Rank—6, Second Rank—12, Third Rank—13, Fourth Rank—8, Fifth 400 Rank—10. However, the third- and second-ranked models of AF3 perform better than its first-401 ranked positions based on these numbers. This suggests that using all five AF3 models instead of 402 just the first-ranked one may be helpful in practical applications. 403 Spearman correlation between scoring function ranking and DockQ ranking for all models 404 To further evaluate the overall performance of each scoring function, we analysed all related 405 Spearman correlation coefficient[48] values for each scoring function compared with DockQ 406 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 18 rankings. In TB modelling (Figure 6e) , AFM-Score led the scoring functions with a median 407 Spearman correlation of 0.13 and a range from -0.80 to 0.91. A strong performance was also 408 exhibited by DeepRank-GNN-esm and GNN_DOVE, with medians around 0.12. In contrast, 409 more consistent but moderate performance was displayed by HADDOCK scores. Lower, 410 inconsistent results were exhibited by PyRosetta, FoldX-Stability and FoldX-Interaction. High 411 variability was shown in the results of Vina and Vinardo. 412 In TF modelling (Figure 6f), strong adaptability is indicated by AFM-Score, which excels with a 413 median Spearman correlation of around 0.18 (higher than that in TB, probably due to a more 414 diverse model pool in TF). The second performer is GNN_DOVE with a 0.12 median value. 415 Moderate effectiveness and stability are shown by HADDOCK scores. Challenges in non-416 template settings are highlighted by FoldX-Interaction and Vina, which struggle with near-zero 417 or negative medians. Conversely, Vinardo and DeepRank-GNN-esm demonstrate modest 418 performance. Overall, robust performance is given by AFM-Score and GNN_DOVE. All related 419 details and information about these two Spearman correlation plots are shown in Supplementary 420 Tables S4 and S5. 421 Additionally, the percentage of positive Spearman correlations across all scoring functions on 422 both TB and TF models are illustrated in Supplementary Fig. S13. Although some tools show 423 positive correlation and better median Spearman correlation coefficients, they all fail to align 424 well with the ground-truth DockQ ranking. For instance, despite a good overall Spearman 425 correlation between GNN_DOVE and DockQ, GNN_DOVEs performance with TF models in 426 identifying the best model for each protein-peptide complex is weaker than most other tools. 427 In TB and TF models, the highest median Spearman correlations are achieved by AFM-Score, 428 showcasing AFM-Score’s adaptability across diverse conditions. GNN_DOVE and DeepRank-429 GNN-esm display strong performances in TB models, but DeepRank-GNN-esm effectiveness is 430 slightly reduced in analysing the predicted structures without templates, indicating a reliance on 431 structured data. HADDOCK demonstrates a consistent scoring approach for both TB and TF 432 models. In less constrained environments, underperformance is highlighted by FoldX-Stability 433 and FoldX Interaction, showing their limitations. The variability of Vina and Vinardo is 434 consistent in both contexts; however, their unpredictability is more pronounced in TF 435 environments, emphasizing the associated risks. Overall, while some scoring functions adapt 436 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 19 well across different frameworks, others depend significantly on the presence of templates in the 437 prediction process, impacting their scope of application. 438 439 440 Figure 6. Assessing of scoring functions. a., b. Spearman correlation matrices on TB and TF, illustrating 441 the similarities between each pair of scoring functions. c., d. Box plot distributions of LOSS values for 442 TB and TF protein-structure predictions. These plots compare the mean and variability of AFM-Score and 443 other scoring functions against DockQ benchmarks. e., f. Spearman correlation values for all scoring 444 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 20 functions across all samples in both TB and TF models provide insight into the consistency and 445 agreement of the scoring functions. 446 447 448 449 Figure 7. Evaluation and comparison of scoring functions for predicting near-native structures. a. 450 Venn diagram illustrating the intersection of three scoring functions: AFM-Score, FoldX-Stability, and 451 HADDOCK-mdscore. The 'com-3' region indicates the common samples region representing the 452 intersection of the three scoring functions. b., c. Box plots for TB and TF models were analysed using the 453 three scoring functions and common samples. In both cases, common samples (Com-score-3) demonstrate 454 lower mean LOSS values, indicating higher accuracy in predictions . d., e. The intersection region among 455 the four scoring functions, including all previous functions along with PyRosetta, was also evaluated 456 against all other scoring functions in TB and TF models. This evaluation is represented as Com-score-4 in 457 the plots. Both consensus-TB and Consensus-TF approaches performed better than any individual scoring 458 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 21 function. The consensus method produced more consistent results and less variability in TB and TF 459 models. 460 461 Because the strong performance of the scoring functions—AFM-Score, FoldX-Stability and 462 HADDOCK-mdscore—was demonstrated for both TB and TF models, we selected these for 463 further evaluation. In Figure 6c,d, AFM-Score displayed superior overall performance in the TB 464 models, albeit with significant variability. Conversely, FoldX-Stability and HADDOCK-mdscore 465 exhibited higher LOSS values and wider spreads, indicating less robustness. Similarly, AFM-466 Score showed higher variability in TF models than TB models but maintained better 467 performance than FoldX-Stability and HADDOCK-mdscore, which demonstrated higher LOSS 468 values and more substantial spreads, underscoring AFM-Scores relative superiority despite its 469 high variability. 470 We combined all TB and TF models from each protein-peptide sample and extracted the top 10 471 models (including both TB and TF top-ranked models) from each sample. The Venn diagram in 472 Figure 7a demonstrated the overlap of top-ranked models among the scoring functions, 473 identifying common samples ranked in the top 10 by all three scoring functions. These common 474 samples from the three mentioned scoring functions (Com-score-3), characterized by 475 consistently lower LOSS values, emerged as reliable candidates for near-native structures against 476 the individual scoring functions. The intersection between FoldX and HADDOCK showed the 477 lowest mean LOSS value of 0.01 compared to all other scoring functions, as illustrated in Figure 478 7b,c, suggesting that these two scoring functions are highly complementary to achieving a good 479 ranking. The combinations of Com-score-3 recorded lower LOSS values, with a mean LOSS of 480 0.07, compared to 0.10, 0.11 and 0.12 for AFM-Score, FoldX-Stability, and HADDOCK-481 mdscore in TF models, and 0.08, 0.11 and 0.13 in TB models, respectively. As depicted in 482 Figure 7d,e, adding the PyRosetta scoring function to the combination, Com-score-4, with a 483 mean value of 0.06, shows improved performance compared to the combination of three scoring 484 functions. Both combinations performed better than the other scoring functions. 485 We also explore the potential of the consensus approach, which integrates multiple models to 486 achieve a more reliable and accurate ranking[49]. This approach leverages the strengths and 487 compensates for the weaknesses of individual models, leading to a more robust prediction of 488 protein structures. For this purpose, we selected eight protein-peptide complex samples from the 489 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 22 TB and TF datasets separately: 7wqq, 7xty, 8ese, 8ia5, 8i3g, 7prx, 8cir, 7r2m. These samples 490 were obtained from different quartiles of the LOSS values distribution to ensure a diverse 491 performance representation. Each of these protein-peptide complexes has 1,000 predicted 492 structures in both TB and TF approaches. We calculated the DockQ score for each of these 1,000 493 models against all other 999 structures in the pool. Finally, we added up all these DockQ scores 494 and extracted the sample with the maximum value of this sum. This consensus approach, 495 indicated by Consensus-TB and Consensus-TF on their corresponding plots, capitalizing on the 496 agreement among different scoring functions, exhibited superior performance, as evidenced by 497 the box plots ( Figure 7d,e). Notably, this approach showed the smallest spread on TB models 498 and the lowest mean LOSS values on both TB and TF models, indicating high accuracy and 499 reliability. The analysis highlights the advantage of using a consensus approach to identify near-500 native structures, showing that samples highly ranked by multiple scoring functions exhibited 501 superior performance. 502 503

504 Our research thoroughly examined AFM, CF and AF3, targeting their accuracy, protein-like 505 properties, and dependability in predicting protein-peptide complex structures—a research area 506 notably lacking in systematic studies. Using benchmarks and reference structures from PDB, our 507 work provided a detailed and comprehensive evaluation of the AlphaFold tool family and their 508 applicable scoring functions through widely adopted quality functions, including DockQ and 509 MolProbity, across both TB and TF methodologies. 510 Our analysis shows that AFM excels in TB predictions with templates but has moderate TF 511 precision in the prediction pool, but TF outperforms TB in the first-ranked models. CF, while 512 less accurate than AFM, is versatile and consistent in TB and TF, improving slightly with 513 templates. AF3 generated high-quality structures for more protein samples; however, the AF3 514 medium DockQ score of all protein samples is significantly less than that of AFM. This 515 highlights the importance of generating a deep pool for achieving high accuracy. As AF3 only 516 provides five models for each protein, the ongoing community initiatives to reproduce AF3 as an 517 open-source tool are meaningful. Such initiatives may not reach the performance of in-house 518 AF3, but it is likely to deliver a tool with higher accuracy than AlphaFold2, and with a deep pool 519 the tool can achieve better performance than the AF3 web service. 520 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 23 Our analysis of MolProbity scores for AFM, CF and AF3 reveals different trends in the quality 521 of structural prediction. The AFM-TB approach generally produces structures of moderate 522 quality, and the model quality remains commendable in the TF settings, even in the absence of 523 template data. CF, on the other hand, consistently generates structures with medium quality 524 across both TB and TF methods, with a marginal enhancement in the quality of structures when 525 template data is used. The AF3 model demonstrates superior performance compared to the other 526 two tools. In exploring specific cases like twisted peptides and cis non-proline residues, we note 527 that AF3 predicted structures with significantly fewer twisted peptides than AFM or CF. 528 However, in predicting structures with cis non-proline peptides, both AFM and CF were slightly 529 better than AF3. 530 It should be noted that all the evaluated structures on AF3, including those with cis non-proline 531 peptides, have these conformations located on the protein part rather than on the peptide site. The 532 analysis of hydrogen bonds in the GT and predicted structures revealed that the native structure 533 has stronger hydrogen bonding than the predicted models. In addition, the helices generated by 534 any model have a different distribution from the GT. Despite using templates, some TB models 535 showed more twisted peptides than the native and TF models. AF3 exhibited no clashes in 536 models, while other methods produced some clashes. Considering all factors, AF3 delivered 537 better protein-like properties than other methods, but still has room for improvement. 538 For ranking prediction models, AFM-Score is clearly better than other scoring functions. This 539 may be partly because AFM-Score is developed explicitly for AlphaFold models, while other 540 scoring functions are intended for general models. Some scoring functions, such as FoldX-541 Stability and HADDOCK-mdscore, also delivered good performance. The scoring functions are 542 highly complementary to each other, as shown by weak correlations between any of them. This 543 suggests the advantage of using a consensus approach to identify near-native structures, as 544 common samples in both three-function (com-score-3) and four-function (com-score-4) 545 combinations ranked highly by multiple scoring functions showed better performance with lower 546 LOSS values. Our consensus method, which extracts the structure with the highest summed 547 DockQ score among the structures in the pool, also demonstrated better performance compared 548 to AFM-Score and all other individual scoring functions in both TB and TF models. Given the 549 massive usage of AFM, it is worthwhile to design scoring functions specifically targeting 550 predicted structures by AFM. Our study suggests the potential to refine individual scoring 551 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 24 functions and integrate ensemble methods for improving the model selection of complex 552 structures predicted by AFM. 553 554

555 Our study provides a comprehensive evaluation of AFM, CF, and AF3 for predicting protein-556 peptide complex structures, highlighting both the strengths and limitations of each tool. AFM 557 excels in Template-Based (TB) predictions, while Template-Free (TF) models often surpass TB 558 models when ranked. CF shows consistent performance across TB and TF methodologies, and 559 AF3 demonstrates the potential for generating high-quality structures, although a deeper model 560 pool is needed to achieve greater accuracy. Analyzing structural quality metrics such as 561 MolProbity scores, hydrogen bonding, and specific features like twisted peptides and cis non-562 proline residues reinforces the performance differences among these tools. Notably, AF3 563 delivered better protein-like properties with fewer clashes and twisted peptides, though some 564 areas still show room for improvement. Our investigation into various scoring functions revealed 565 the potential for optimizing structure selection through consensus approaches, combining metrics 566 such as FoldX-Stability and HADDOCK-mdscore to improve accuracy. Future efforts should 567 focus on refining scoring functions tailored to AFM models and enhancing ensemble methods to 568 improve prediction reliability in protein-peptide docking. 569 570

and Materials 571 Benchmark Dataset 572 In this study we developed a comprehensive pipeline to evaluate the quality of predicted protein-573 peptide structures by AFM, CF and AF3. Additionally, this pipeline assesses the efficacy of 574 various scoring functions in accurately ranking these predicted structures, thereby assessing their 575 capability to represent protein-peptide interactions accurately. For this purpose, we employed 576 two quality functions alongside distinct scoring functions. 577 To facilitate a thorough evaluation of AFM and CF’s performance in predicting protein-peptide 578 complex structures, we initiated the systematic preparation of a benchmark dataset. We carefully 579 curated this dataset to include protein-peptide complex structures that feature at least one protein 580 and one peptide chain, with a released date after 12/01/2023 (the cutoff date for the training set 581 used by AFM). This criterion ensured the incorporation of novel structures not previously 582 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 25 encountered by the predictive models. Furthermore, we restricted our selection to peptide 583 sequences spanning three to 50 residues in length, excluding any sequences containing 584 nonstandard amino acids to maintain dataset quality. Following the collection of all pertinent 585 protein-peptide sequences, we employed the CD-HIT[50] tool to identify and remove sequences 586 exhibiting more than 40% redundancy, thus enhancing the dataset’s diversity. This thorough 587 selection process resulted in a dataset comprising 60 unique protein-peptide complex structures 588 ready to incorporate into our evaluation pipeline (Supplementary Table S1). 589 Model Generation 590 AFM models were generated using AlphaFold v2.3.1, employing both TB and TF approaches. 591 AFM was run on NVIDIA A100 80GB PCIe GPUs. Default parameters were used for mu ltiple 592 sequence alignment generation to maintain consistency across predictions. For each protein-593 peptide sample, 1,000 complex structures were predicted using the TB method and 1,000 594 structures using the TF method, ensuring a comprehensive analysis. We also processed all 595 samples using CF’s default settings to predict structures through TB and TF approaches. By 596 default, each approach yielded five ranked models. The complex structures of all samples were 597 predicted using the AF3 web server, which by default returned five predicted structures for each 598 sample, regardless of the presence of templates. One of the samples (8C2P) out of the 60 in our 599 dataset could not be predicted by AF3 due to a biological sequence restriction, resulting in a 600 “Sequence filtering encountered” error. 601 For the AFM-TB models, the ‘max-template-date’ parameter was set to approximately seven 602 days before the release date of the protein-peptide complex on the PDB website. This strategy 603 was designed to incorporate all relevant PDB structures available in the AlphaFold dataset while 604 explicitly excluding the target protein-peptide complex under investigation. Conversely, for TF 605 models, the ‘max-template-date’ parameter was adjusted to ‘1,000-01-01’, effectively excluding 606 all templates from the modelling process and ensuring a purely template-free approach. The 607 approach employed both TB and TF modelling to thoroughly evaluate CF’s and AFM’s 608 performance in predicting protein-peptide complex structures, deliberately omitting Amber’s 609 relaxation step to assess only the unrelaxed, ranked models. AFM was run as below. 610 Alphafold --use_gpu --data_dir= --fasta_paths= --output_dir= --max_template_date= --model_preset=multimer --613 num_multimer_predictions_per_model=200 --run_relax=false 614 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 26 AFM models were generated using full-length sequences from the PDB SEQRES section. 615 During preparation, we conducted a cleaning step on predicted PDB structures to remove any 616 residues not resolved in the experimental data, aligning only residues present in both the 617 modelled structures and the reference PDB ATOM section. This critical step, ensuring model 618 integrity and accurate alignment with PDB sequences, was essential for precise DockQ score 619 calculations and reliable comparison against existing PDB entries. 620 All structural images presented in this paper were generated using PyMOL software 621 (Schrödinger, LLC). 622 Evaluation with Quality Functions 623 We used DockQ and Molprobity to compare predicted complexes to native structures and to 624 assess their quality, respectively. DockQ scores, ranging from 0 to 1, assess the accuracy of 625 predicted models against native PDB structures. Scores above 0.23 are considered acceptable by 626 CAPRI standards. These scores indicate the quality of the interaction interface between the 627 protein and the peptide relative to the native structure. Figure 1 illustrates the evaluation and 628 analysis outcomes for TB and TF methods, depicting the qualification of the top-ranked models 629 by AFM, CF, and AF3. 630 ./DockQ/scripts/fix_numbering.pl 632 The output of fix_numbering.pl (above command) was then used to calculate the DockQ score as 633 follows: 634 python DockQ.py 635 The DockQ score between all predicted structures and the native structure was calculated and 636 considered in all analysis parts. 637 MolProbity is a tool for validating and analysing 3D protein and nucleic acid structures. It 638 assesses structural quality by evaluating parameters such as Clashscores, rotamer outliers, and 639 Ramachandran plot statistics, thereby ensuring model accuracy and reliability. To prepare the 640 predicted structures for analysis with MolProbity, the phenix.reduce command was executed to 641 add hydrogens, potentially flip side chains, and build missing atoms, thus correcting common 642 structural issues and ensuring completeness and accuracy. The following commands were used 643 to run MolProbity on the predicted structures. 644 phenix.reduce -FLIP -BUILD &> FH.pdb 645 phenix.molprobity FH.pdb 646 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 27 In these commands, is the input PDB file, and FH.pdb is the output file 647 where the results of the reduced operation are saved. 648 Evaluation with Scoring Functions 649 AFM uses a scoring function named ‘predicted TM-score’ (pTM) to rank predicted structural 650 models. The pTM score predicts the model’s similarity to the GT structure, aiming for a high 651 TM-score match. DockQ, which compares predicted structures to native models, served as a 652

for evaluating the ranking accuracy of the AFM, CF and AF3 scoring functions. All 653 1,000 predicted models from both TB and TF methods were reranked according to DockQ 654 scores. To assess the ranking effectiveness of AFM-Score and CF-Score, we calculated the 655 Spearman correlation coefficient between AFM’s rankings and those of DockQ. We employed a 656 range of scoring functions to rank the AFM-predicted structures in both TB and TF models. We 657 selected these scoring functions from various methods to analyse the predicted structures from 658 different perspectives. They included PyRosetta, Foldx (Stability, Interaction), HADDOCK 659 (emscore, mdscore), AutoDock Vina (Vina, Vinardo), GNN_DOVE, and DeepRank-GNN-esm 660 (Table 1). 661 662 Functions Scores Key Features URL HADDOCK3[23] emscoring, mdscoring Flexible docking, incorporates experimental data https://github.com/haddockin g/haddock3 FoldX[25] Stability, AnalyzeCom plex Rapid energy calculation, mutational effect prediction, protein stability analysis https://foldxsuite.crg.eu/ PyRosetta[51] PyRosetta Detailed energy manipulation, support protein design https://www.pyrosetta.org/do wnloads AutoDock[24] Vina Vina, Vinardo Uses gradient optimization techniques https://github.com/ccsb- scripps/AutoDock-Vina DOVE[29] Goap, IT Integrates GOAP and .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 28 score (ATOM40) IT-Score https://github.com/kiharalab/ DOVE GNN_DOVE[30] DOVE score Incorporates GNN for enhanced prediction accuracy https://github.com/kiharalab/ GNN_DOVE DeepRank[31] Wrong Rank, Near Native Focuses on ranking PPIs, uses deep learning https://github.com/DeepRank DeepRank-GNN- esm[33] DeepRank score Combined GNN with LLMs, advanced feature learning https://github.com/DeepRank/ DeepRank-GNN-esm InterPepRank[34] InterPepRan k score GNN-based, predict protein https://bitbucket.org/isaakh94/ interpeprank.git 663 PyRosetta is a customizable toolkit for protein structure prediction and design, and we used its 664 scoring function in this study. HADDOCK employs empirical (emscore) and molecular 665 dynamics (mdscore) scoring for flexible docking of biomolecules. AutoDock Vina and its variant 666 Vinardo provide an efficient scoring function based on energy calculations, considering a 5Å 667 dimension box around the peptide part to ensure accurate modelling of critical interactions 668 within the binding site while maintaining computational efficiency. GNN_DOVE and 669 DeepRank-GNN-esm are two other deep-learning-based scoring functions used for accurate 670 protein complex scoring. The Python codes for running and analysing the predicted structures 671 with these scoring functions are available in the GitHub repository. FoldX offers both stability 672 and interaction scoring functions to assess protein-peptide structures, and the commands used to 673 run it are provided below. To run FoldX on a protein structure, the following command was 674 executed sequentially. 675 foldx --command=RepairPDB --pdb=example.pdb 676 This command below repairs the input PDB file by adding missing atoms, optimizing the 677 structure, and fixing any structural issues. This step ensures that the structure is complete and 678 accurate, preparing it for further analysis. 679 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 29 foldx --command=Stability --pdb=_Repair.pdb 680 After repairing the PDB file, the above command calculates the stability of the protein structure. 681 It estimates the free energy of folding, referred to as FoldX-Stability in this study, which can be 682 used to assess the stability and reliability of the protein model. 683 Foldx --command=AnalyseComplex --pdb=_Repair.pdb --684 analyseComplexChains=A,B 685 This command analyses the interactions between specified chains (in this case, chains A and B) 686 in the repaired PDB file. It calculates various energy terms associated with the protein-peptide 687 complex, such as binding energy, providing insights into the interaction dynamics and stability 688 of the complex. This parameter is referred to as FoldX-Interaction in this study. 689 In addition to the previously mentioned scoring functions, we evaluated DOVE and 690 DeepRank[52] were also evaluated on all 60 samples. However, due to installation issues with 691 DOVE, which led to unsatisfactory outcomes (predominantly zero values for ATOM40), we 692 excluded its results from our evaluation as well. DeepRank, which relies on PSSM in its 693 algorithm, encountered difficulties in identifying necessary homologs for very short peptide 694 sequences. As a result, it was only applicable to about 28 out of 60 samples of our dataset, 695 leading to its exclusion from our evaluation system. Other scoring functions were also attempted 696 for installation and testing on our dataset but could not be used. GDockScore[53] was not 697 applicable since it only accepts peptides with more than 30 residues, while some of the peptides 698 in our dataset had fewer than 30 residues. InterEvScore[54] encountered an error when running, 699 and alignment only works with chains of more than 30 residues as well. PyDocks[54] download 700 link could not be accessed. 701 Evaluations 702 Our analysis evaluated various scoring functions using a dataset of protein-peptide complexes, 703 comparing predicted AFM structures (TB and TF) against native PDBs for true structural 704 fidelity. DockQ served as a reference by comparing and ranking predicted structures aligned with 705 their native counterparts, establishing a benchmark for assessing other scoring functions. We 706 used the Spearman correlation coefficient as the metric for this evaluation. In the below formula, 707 if /g1870 /g3036 is the rank of the /g1861 /g3047/g3035 observation in the first set, and s i is the rank of the /g1861 /g3047/g3035 observation in 708 the second set, then /g1856 /g3036 /g3404/g1870 /g3036 /g3398/g1871 /g3036 . 709 710 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 30 ρ = 1- /g2874 ∑ /g3031 /g3284 /g3118 /g3289 /g3284/g3128/g3117 /g2924/g4666/g2924² /g2879 /g2869/g4667 /g1856 /g3036 /g3404/g1870 /g3036 /g3398/g1871 /g3036 Eq. (1) 711 Where /g1856 /g3036 is the difference between the ranks of corresponding values, /g1866 is the number of 712 observations, and ∑ /g1856 /g3036 /g2870 is the sum of the squares of the rank differences[55]. We used the 713 Spearman correlation coefficient to determine the concordance level between the rankings 714 produced by an array of scoring functions and the established DockQ benchmark. A coefficient 715 nearing the value of 1 indicates a strong positive agreement, signifying that a scoring function’s 716 rankings are highly consistent with those from DockQ. Such a correlation is indicative of the 717 scoring function’s predictive reliability for the structural configurations of protein-peptide 718 complexes. 719 We note that not all scoring functions provided scores for all models. AutoDock Vina's scoring 720 function failed for four samples—8CK5, 8ESE, 8HDJ, and 8HEP—due to time and memory 721 limitations. Consequently, AutoDock Vina evaluated a total of 56 samples. GNN_DOVE and 722 FoldX also failed for samples 8HEP and 8ARE in both the TB and TF approaches due to time 723 constraints encountered during model execution. Consequently, these two scoring functions were 724 evaluated on only 59 samples. InterPepRank’s inference on a CPU typically took two to three 725 days per sample, indicating time inefficiency. Challenges included NaN values and uniform 726

due to incorrect weight loading, leading to prediction failures. Satisfactory results were 727 not obtained for half of the samples in both TB and TF models. Due to this data inequality, we 728 excluded InterPrepRank from our evaluations. 729 Consensus Approach 730 This consensus approach aims to find the predicted structure with the most similarities to other structures 731 in the pool and rank them as the best predictions. For a given protein, we calculated the DockQ score for 732 each of 1, 000 predicted structures in either TB or TF a pproaches against all othe r 999 structures in the 733 pool. We added up all these DockQ scores and extracted the sample with the maximum value of this sum. 734 Equations (2) and (3) show the mathematical formula of the process. 735 /g1845/g1873/g1865 /g3036 /g3404 /g3533/g1830 /g3036,/g3037 /g1858/g1867/g1870 /g1861 /g1488 /g4668 1, … , /g1866 /g4669 /g3041 /g3037/g2880/g2869 /g3037/g2999/g3036 Eq. (2) 736 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 31 /g1861 /g1499 /g3404a r g max /g3036/g1488 /g4668 /g2869,…,/g3041 /g4669 /g1845/g1873/g1865 /g3036 /g1838/g1867/g1871/g1871/g4666/g1861 /g1499 /g4667 Eq. (3) In equation (2), /g1830 /g3036,/g3037 represents the DockQ score between the /g1861 th and /g1862 th predicted structure in the 737 pool, and /g1861 * shows the maximum summation value. 738 739 Authors’ Contributions 740 D.X. conceptualized the study and methodology. N.M. performed the investigations, formal 741 analyses and data curation. N.M. and J.R. implemented the validation and checking process. All 742 authors contributed to running tools and conducting the experiments. N.M. wrote the first draft 743 of the manuscript and created all visualizations, with J.R. providing notes on the manuscript. 744 D.X. reviewed and edited the manuscript. Additionally, D.X. administered the project and 745 supervised the study. 746 747 Competing Interests 748 The authors declare no competing interests. 749 750

751 This work is supported by the National Institutes of Health grant R35-GM126985. 752 753 Availability of data and materials 754 Dataset 755 The data used for this evaluation were generated during this study. Source data associated with 756 this paper are provided in the GitHub repository (https://github.com/NeginManshour/PpEv). 757 Code 758 The source code of this evaluation is freely available at 759 https://github.com/NeginManshour/PpEv. 760 761

762 [1] Petsalaki E, Russell RB . Peptide-medi at ed interac tions in biological systems: n ew discoveries and 763 applications . Curren t Opi nion i n Biot ech n ology 2008 ; 19: 344–350. 764 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 32 [2] Lee ACL, Harris JL, Khanna KK, e t al. A co mprehensive r eview on curren t advance s in peptide dr ug 765 developmen t and design. In terna tio nal J ourn al of Molec ular Scie nces ; 20 . Epub a head of print 2 766 May 2019. DOI: 10.3390/ijms20102383. 767 [3] Wang D, Pourmirzaei M, Abbas UL, e t al. S-PLM: Structure-awa re Prot ein Languag e Model via 768 Contrastive Lea rning be tween Se quence and Struc ture . DOI: 10 .1101/2023.08.06. 552203. 769 [4] Manshour N , Ghaffari A, Ghas emi RH. Ki nema tic a nal yzin g for a series of amin o a cids as an arm 770 of nano- act ua tor . 2012 . 771 [5] Esmaili F, Pourmirzaei M, R amazi S, e t al. A Review of Machine Learning and Algor ithmic Meth ods 772 for Protein Phosphoryla tion Si te Predicti on. Ge nomi cs, Prote omics a nd Bi oinform atics 2023; 21: 773 1266–1285. 774 [6] Milon TI, Wang Y, Fon teno t RL, et al. Developmen t of a novel repr esen tati on of dr ug 3D 775 structur es and enh ancemen t of the TSR- based meth od for probing drug and targ et int erac tions. 776 Compu t Biol C hem 2024; 112: 108117. 777 [7] Manshour N , He F, W ang D, et al. Int egra ting Prot ein Stru ct ure Predic tion and Bay esian 778 Optimiz ati on for Pepti de Design . 779 [8] Dill KA, Ozkan SB, Shell MS , et al . The pro tein folding probl em. An nu al Review of B ioph ysics 2008; 780 37: 289–316. 781 [9] Weng G, Gao J , Wang Z, et al . Compreh e nsive Evaluation of Fourt een Docking Programs on 782 Protein-Peptid e Complex es. J Ch em Theo ry Comp ut 2020; 16: 3959–3969 . 783 [10] Manshour N , Yu Y, Qin W, et a l. Evaluatin g templat e-based and templa te-free p rot ein-peptid e 784 complex struc tur e predic tion using Alpha Fold2. Inst itu te of Electrical and Electr oni cs Engineers 785 (IEEE), 2023, pp. 3856–3856. 786 [11] Zhou P, Li B, Yan Y, et al. Hiera rchical Fle xible Peptid e Docking by Conformer Gen erati on and 787 Ensemble Docking of Peptides. J Che m In f Model 2018; 58: 1292–1302. 788 [12] Koukos PI, Bonvin AMJJ . In tegra tive Mod elling of Biomolecular Comple xes. Jour na l of Molecul ar 789 Biolog y 2020; 432: 2861–2881. 790 [13] Evans R, O’neill M, Prit zel A, et al . Protei n complex pred iction with Al phaFold-Mu ltimer. Epub 791 ahead of print 2022. DOI: 10 .1101/2021.10.04.463034. 792 [14] Jumper J , Evans R, Pritzel A , et al . Highly accurate p rot ein struc tur e predic tion wit h AlphaFold . 793 Nat ure 2021; 596: 583–589. 794 [15] Mirdita M , Schütz e K, Moriwaki Y, et al . ColabFold: making pro tein folding accessi ble to all . Na t 795 Meth ods 2022; 19: 679–682. 796 [16] Abramson J , Adle r J, Dunger J, et al. Accu rate s tructu re pr edictio n of biomolecular interac tions 797 with AlphaFoldHiID3 . Na ture . Epub ahe ad o f print 8 May 2024. DOI: 10.1038/s41586-024-07487-w. 798 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 33 [17] Zhu W, Shenoy A, Kundro tas P, et al . Evaluation of Alph aFold-Mult imer pr ediction on multi-chain 799 prote in complex es. Bioi nform atics ; 39. Epub ahead of print 1 July 2023. DOI: 800 10.1093/bioinformatics/btad424 . 801 [18] Fowler NJ , Williamson MP. The accu racy of protein st ructu res in soluti on det ermi ned by 802 AlphaFold and NMR . Struc ture 2022; 30: 925-933.e2. 803 [19] Polonsky K, Pupko T, Freund NT. Evaluati on of the Abili ty of AlphaFold t o Predict t he Three-804 Dimensional Struc tures of An tibodi es an d Epitopes. Th e Jo urnal of Imm unol ogy 2 023; 211: 1578–805 1588. 806 [20] Yin R, Pierce BG. Evalua tion of Alph aFold antibody– antigen mod eling with implicat ions for 807 improving predictive accuracy. Prot ein Sc ience ; 33. Epub ahe ad of print 1 Ja nuary 2024. DOI: 808 10.1002/pro.4865. 809 [21] Raveh B, London N , Schuele r-Furman O . Sub-angstrom modeling of complex es be tween flexi ble 810 peptid es and globular p rot eins. Protei ns: Struct ure, Fu nc tio n an d Bioinfor ma tics 2 010; 78: 2029–811 2040. 812 [22] Xue LC, Dobbs D, Bonvin AMJJ, et al . Computatio nal pre diction of pro tein in terfa ces: A review of 813 data driven me thods . FEBS Letters 2015; 589: 3516–3526. 814 [23] Dominguez C, Boelens R, Bonvin A MJ J. H ADDOCK: A protein-pr otei n docking appr oach based on 815 biochemical or biophysical informa tion. J Am Chem Soc 2003; 125: 1731 –1737. 816 [24] Rentzsch R, R enar d BY. Docking small peptides r emains a grea t challenge : An asse ssment using 817 AutoDock Vina. Brief Bioi nform 2015; 16: 1045–1056. 818 [25] Schymkowitz J, Borg J, S triche r F, et al . The FoldX web server : An online forc e fiel d. Nu cleic A cids 819 Res; 33. Epub ahea d of print July 2005. D OI: 10.1093/na r/gki387. 820 [26] Lensink MF, Wodak SJ . Docking, scoring, and affinity predictio n in CAPRI. Protei ns: Struct ure, 821 Func tio n an d Bioi nforma tics 2013; 81: 20 82–2095. 822 [27] Sapundzhi F, Proda nova K, Lazarova M. S urvey of the scoring functions for pro tein -ligand 823 docking. In: AIP Co nferenc e Proceedi ngs . American Ins titu te of Physics Inc., 2019. Epub ahead of 824 print 13 Novemb er 2019. DOI : 10.1063/1.5133601. 825 [28] Huang SY, Grint er SZ, Zou X. Scoring functions and th eir evalu ation me thods for p rotei n-ligand 826 docking: Recent a dvances and futur e dir ections. Physi cal Che mistry C hemic al Phy sics 2010; 12: 827 12899–12908. 828 [29] Wang X, Terashi G, Chris toffer CW, e t al. Protein docking model evalu ation by 3D deep 829 convolutional n eural n etworks. Bi oinfor mati cs 2020; 36: 2113–2118. 830 [30] Wang X, Flannery ST, Kihara D. Pro tein D ocking Model Evaluation by Gr aph N eura l Networks. 831 Front M ol Biosci ; 8. Epub ahe ad of print 25 May 2021. DOI: 10.3389/fmolb.2021. 647915. 832 [31] Renaud N , Ge ng C, Georgievska S, e t al. DeepRank: a d eep le arning framework fo r data mining 3D 833 prote in-prot ein int erfaces . Na t Comm un ; 12. Epub ahead of print 1 December 202 1. DOI: 834 10.1038/s41467-021-27396 -0. 835 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 34 [32] Réau M, R enaud N , Xue LC, et al. Dee pRa nk-GNN: a gr aph neu ral ne twork framework to lea rn 836 patt erns in pro tein –pro tein in te rfaces. Bi oinforma tics ; 39. Epub ahe ad of print 1 J anuary 2023. 837 DOI: 10.1093/bioinforma tics/btac759. 838 [33] Xu X, Bonvin AMJJ. De epRank-G NN-esm: A graph neur al netwo rk for scoring prot e in-protein 839 models using protei n language model . Bi oinforma tics A dva nces ; 4. Epub ahe ad of print 2024. 840 DOI: 10.1093/bioadv/vbad191. 841 [34] Johansson-Åkh e I, Mir abell o C, Wallner B . Int erPepRa nk: Assessmen t of Docked P eptide 842 Conformations by a Deep Gr aph N etwor k. Frontiers in B ioinfor mati cs ; 1. Epub ah ead of print 843 2021. DOI: 10.3389/fbinf.2021.763102. 844 [35] Li H, Sze KH, Lu G, et al . Machine-le arnin g scoring functions for structur e-based vi rtual scre ening. 845 Wiley I nter discipli nary R eviews: Com pu ta tion al Mole cul ar Scienc e ; 11. Epub ah ead of print 1 846 Janua ry 2021. DOI: 10.1002/wcms.1478. 847 [36] Su M, Yang Q, Du Y, et al. Compara tive A ssessment of Scoring Functions : The CASF-2016 Update. 848 J Chem I nf Model 2019; 59 : 895–913. 849 [37] Burley SK, Berma n HM, Bhikadiya C, et al . Protein Dat a Bank: The single glob al arc hive for 3D 850 macromolecular s tructu re da ta. Nuc leic Acids Res 2019; 47: D520–D528. 851 [38] Berman HM, West brook J , Feng Z, et al . The Protein Da ta B ank , 852 http://www.rcsb.org/pdb/sta tus.h tml (2000). 853 [39] Basu S, Walln er B. DockQ: A qu ality meas ure for pro tein-pr otei n docking models. PLoS One ; 11. 854 Epub ahead of print 1 August 2016. DOI : 10.1371/journ al.pone.0161879. 855 [40] Chen VB, Arenda ll WB, He add J J, e t al. Ch apter 21 .6. MolPro bit y: all-a tom str uc tur e valid atio n for 856 macro mole cul ar cryst allo grap hy , h ttp ://molprobity.bi ochem.duk e.e du (2012). 857 [41] Davis IW, Leaver-Fay A, Chen VB, et al. M olProbity: All-a tom contac ts and struc tur e validation for 858 prote ins and nucleic acids. Nucl eic A cids Res; 35. Epub ahea d of print July 2007. D OI: 859 10.1093/nar/gkm216. 860 [42] Mirabell o C, Wallne r B. DockQ v2: Impro ved automa tic quality measur e for pro tei n multimers, 861 nucleic acids, and small molecules . DOI: 10.1101/2024.05.28.596225. 862 [43] Janin J , Henrick K, Moul t J, e t al. CAPRI: A critical assessment of PRedic ted int erac t ions. Proteins : 863 Struct ure, Fu nc tio n an d Ge neti cs 2003; 52: 2–9. 864 [44] Zhang Y, Skolnick J. Scoring function for a utomat ed assessmen t of prot ein struc tur e templa te 865 quality. Protei ns: Stru ct ure, Fu nct ion and Gene tics 2004; 57: 702–710. 866 [45] Kabsch W, Sander C. Dic tion ary of Protei n Secon dary Str uct ure: Pa tter n Rec ogni ti on of Hydr oge n-867 Bond ed a nd G eome tric al Fea tures . 868 [46] A global Ramachand ran score id entifies prote in struc tures with unlik ely stere och emistry. 869 [47] Williams CJ, Headd JJ, M oriar ty NW, et al . MolProbity: M ore a nd bet te r refer ence data for 870 improved all-atom st ructur e validati on. Protein Scie nce 2018; 27 : 293–315. 871 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint 35 [48] Spearma n Ra nk Correlati on Coefficie nt . I n: The Con cise Enc yclo pedi a of Sta tistics. Springer, N ew 872 York, NY. ht tps://doi .or g/10.1007/978-0- 387-32833-1_379 . 873 [49] McGuffin LJ. Benchmarking consensus m odel quali ty assessment for pro tein fold r ecognition . 874 BMC Bioinfor mati cs ; 8. Epub ah ead of pri nt 18 Sept ember 2007. DO I: 10.1186/1471-2105-8-345. 875 [50] Fu L, Niu B, Zhu Z, et al. CD-HIT: Acceler a ted for cluste ring the n ex t-gener ation s e quencing data . 876 Bioinform ati cs 2012; 28: 3150–3152. 877 [51] Chaudhury S, Lyskov S, Gray JJ. PyRoset t a: A script-bas ed int erface for implemen t ing molecular 878 modeling algorithms using Rose tt a. Bioi n formati cs 2010; 26: 689–691. 879 [52] Renaud N , Ge ng C, Georgievska S, e t al. DeepRank: a d eep le arning framework fo r data mining 3D 880 prote in-prot ein int erfaces . Na t Comm un ; 12. Epub ahead of print 1 December 202 1. DOI: 881 10.1038/s41467-021-27396 -0. 882 [53] Mcfee M, Kim PM. GDockScore: A gr aph-based pro tein-pr otei n docking scoring function. 883 Bioinform ati cs Adv an ces ; 3. Epub ah ead of print 2023. DOI: 10 .1093/bioadv/vbad072. 884 [54] Andre ani J, Fau re G, Gu erois R. Int erEvScore: A n ovel coarse-grai ned int erface sco ring function 885 using a multi-body statistical p ote ntial co upled to evolu tion . Bioinfor ma tics 2013; 29: 1742–1749. 886 [55] Croux C, Dehon C. Influence functions of the Spe arman and Ken dall corr ela tion measures . Stat 887 Meth ods Ap pt 2010; 19: 497 –515. 888 889 .CC-BY-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 11, 2024. ; https://doi.org/10.1101/2024.11.11.622992doi: bioRxiv preprint