Quality-Aware Evaluation for Journal Recommendation

Quality-Aware Evaluation for Journal Recommendation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Quality-Aware Evaluation for Journal Recommendation Nir Roguin, Omri Dominsky, Yariv Yogev, Yoav Baruch This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9053132/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Objective: Journal recommendation systems are tools that analyze a manuscript’s content and suggest potential target journals. Most tools that suggest journals are evaluated using simple accuracy measures, such as Top-K accuracy metrics: the recommendation is considered correct only if the exact journal is predicted. All other recommendations are counted as errors. However, this approach ignores journal quality. Recommending a different journal of similar quality is very different from recommending a journal that is far below the appropriate quality level. We developed a Quality-Aware evaluation tool designed to assess whether journal recommendation tools suggest journals of an appropriate quality level, rather than simply whether they predict the exact target journal. Methods: We developed specialty-specific journal recommendation models for medical fields using transformer-based architectures trained on nearly one million PubMed articles (2020–2024; 2020–2025 for Cardiology). Using SCImago quartiles as a proxy for journal quality, we evaluated models using both standard metrics and novel Quality-Aware metrics (Quality Accuracy, Undersell Rate, Severe Undersell Rate). We examined the relationship between these metric families to characterize when recommendation lists can be considered reliable. Results: Across five specialties, mean accuracy@1 was 47.9% (range: 39.9%–55.0%), with Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG) of 0.60 and 0.68, respectively. However, Quality Accuracy@1 averaged 67.7%, exceeding raw accuracy by approximately 20 percentage points. This gap indicates that more than a third of all prediction “errors” under standard metrics involved journals of equivalent quality. Furthermore, Quality Consistency@3 averaged 54.0%, demonstrating that, on average, the majority of journals in the top-3 recommendations consistently aligned with the target journal’s quality tier. Severe Undersell Rates (recommendations two or more quartiles below the ground truth) averaged 7.9%, with the best-performing models achieving rates as low as 5.2%. Conclusion: Standard evaluation metrics for journal recommendation are insufficient because they treat all errors as equivalent. In this study, Quality Accuracy@1 substantially exceeded top@1 accuracy, indicating that a large proportion of apparent “errors” involved journals of equivalent quality. Quality-Aware metrics, combined with ranking quality measures, provide a more complete assessment of whether a system produces reliable recommendations. We propose that journal recommendation systems report Quality-Aware metrics alongside traditional accuracy to better characterize real-world utility. MSC Codes: 68T10 (Pattern recognition); 62P10 (Applications of statistics to biology and medical sciences) JEL Codes: O32 (Management of Technological Innovation and R&D) journal recommendation evaluation methodology information retrieval medical informatics scholarly publishing Statement of Significance Problem Journal recommendation systems are evaluated using Top-K accuracy, which treats all prediction errors equivalently. This binary formulation fails to distinguish between errors that recommend similar-quality journals versus errors that recommend substantially lower-quality journals. What is Already Known Prior systems from Jane (Schuemie & Kors, 2008) through transformer-based approaches (Nguyen, Huynh, Dinh, et al., 2022; Le et al., 2022) have progressively improved Top-K accuracy through architectural advances. However, evaluation methodology has remained unchanged, with all systems using the same binary accuracy metrics. What This Paper Adds We introduce Quality-Aware evaluation metrics that use journal quartiles as a proxy for quality. Combined with ranking metrics (MRR, NDCG), these metrics characterize whether recommendation lists are reliable. We validate this methodology across five medical specialties with over 95,000 test samples. Who Benefits Developers of journal recommendation systems who need evaluation methods that better reflect real-world utility; researchers evaluating recommendation systems in domains where labels have ordinal structure; users of recommendation tools who need to assess whether outputs are trustworthy. Introduction After completing a research project, selecting an appropriate journal can be challenging. With thousands of potential journals, researchers must identify journals whose scope matches their manuscript while also considering factors like quality, audience, and likelihood of acceptance. An unsuitable choice may result in substantial delays due to repeated submissions and rejections, potentially compromising the perceived originality of the study, and ultimately contributing to researcher fatigue and loss of motivation. These challenges have motivated the development of automated journal recommendation systems. The field has progressed substantially over the past two decades. Early journal recommendation approaches relied on simple word-based similarity between indexed abstracts (Schuemie & Kors, 2008 ). Subsequent work introduced machine learning with engineered features (Wang et al., 2018 ; Son & Kim, 2018 ), followed by deep learning approaches that demonstrated gains through learned representations (Feng et al., 2019 ). Most recently, transformer-based systems have achieved state-of-the-art performance through semantic modeling (Nguyen, Huynh, Dinh, et al., 2022 ; Le et al., 2022 ). Despite architectural advances, evaluation methodology has remained static. Nearly all systems report Top-K accuracy: the proportion of test cases where the ground-truth journal (the true journal in which the article was published) appears among the top K recommendations. This metric, established in early work and inherited by subsequent papers, embeds a problematic assumption: that all prediction errors are equivalent. Not all recommendation errors are equivalent: suggesting a journal of similar quality is acceptable, while suggesting a substantially lower-quality journal is not, yet standard accuracy metrics treat both identically. From a user’s perspective, the key question is whether the recommended list can be trusted. If only one suggested journal is appropriate while the rest are not, the list is difficult to use. This becomes particularly important when the correct journal is not ranked first. In contrast, when all top recommendations are of appropriate quality, the list becomes useful. Current evaluation provides no insight into this distinction. A system with 85% accuracy could be producing recommendation lists where errors are minor (similar-quality alternatives) or major (inappropriate journals). Developers optimizing for Top-K accuracy during development have no visibility into which pattern their system exhibits. In this paper, we present a Quality-Aware evaluation methodology for journal recommendation. Our approach uses journal quartiles (from SCImago) as a proxy for quality. We introduce metrics that measure whether recommendations preserve appropriate quality levels, and we examine how these metrics combine with standard ranking metrics to characterize recommendation reliability. We validate this methodology across five medical specialties, demonstrating consistent patterns that may help guide the evaluation of journal recommendation systems. Journal recommendation systems have evolved through distinct methodological phases. Schuemie and Kors ( 2008 ) established the task as article similarity aggregation using Term Frequency–Inverse Document Frequency (TF-IDF) over Medline abstracts. While limited by vocabulary mismatch, Jane demonstrated the feasibility of automated recommendation and remains a reference baseline. Classical machine learning approaches introduced feature engineering and discriminative classifiers. Wang et al. ( 2018 ) achieved 61% Top-1 accuracy using logistic regression with TF-IDF features. Son and Kim ( 2018 ) improved this using title, abstract, and keyword features with Multilayer Perceptron (MLP) classifiers. These works established Top-K accuracy as the standard evaluation metric. Deep learning approaches reduced reliance on feature engineering. Feng et al. ( 2019 ) used word2vec embeddings with Convolutional Neural Network (CNN) classification on 880,000 PubMed abstracts, achieving 87% Top-10 accuracy. Nguyen, Huynh, et al. ( 2021 ) systematically compared architectures and found that representation quality mattered more than specific architectural choices. Transformer-based systems represent the current state-of-the-art. Nguyen, Huynh, Dinh, et al. ( 2022 ) demonstrated that Bidirectional Encoder Representations from Transformers (BERT)-family encoders outperform prior approaches across all Top-K metrics. Le et al. ( 2022 ) further improved performance through contrastive learning with journal scope text. These systems frame recommendation as semantic alignment between manuscripts and journals. All systems report Top-K accuracy, with some including MRR or NDCG. None assess whether prediction errors preserve appropriate quality levels. None measure the magnitude or direction of errors in terms of journal quality. This represents a gap between what developers measure and what users need. A user receiving recommendations needs to know not just whether the system is accurate, but whether its errors are acceptable. Current metrics provide no insight into this question. Methods Data Collection. We constructed datasets for five medical specialties: Cardiology, Obstetrics and Gynecology (OB/GYN), Pediatrics, Radiology, and Surgery. For each specialty, we identified relevant journals through NLM catalog classifications and retrieved articles published between 2020 and 2024 (2020–2025 for Cardiology) via the PubMed E-utilities API. We extracted title, abstract, and MeSH terms for each article. Articles lacking abstracts or with abstracts shorter than 50 words were excluded. Data were split temporally: for training (80%), for validation (10%), and for testing (10%). Table 2 presents dataset characteristics. The total corpus comprises nearly one million articles published across approximately 1,500 journals. Table 1 Evaluation Approaches in Prior Work System Year Primary Metric Ranking Metrics Quality-Aware Error Analysis Jane (Schuemie & Kors, 2008 ) 2008 Recall No No No Wang et al. ( 2018 ) 2018 Accuracy@K No No No Pubmender (Feng et al., 2019 ) 2019 Accuracy@K No No No PSRMTE (Nguyen, Huynh, Dinh, et al., 2022 ) 2022 Accuracy@K No No No SimCPSR (Le et al., 2022 ) 2022 Accuracy@K No No No This work 2025 Accuracy@K MRR, NDCG Yes Yes Table 2 Dataset Characteristics Specialty Total Train Validation Test Cardiology* 241,391 193,304 24,027 24,060 OB/GYN 95,990 76,799 9,595 9,596 Pediatrics 143,782 115,127 14,311 14,344 Radiology 166,823 133,619 16,603 16,601 Surgery 307,686 246,196 30,779 30,711 Total 955,672 765,045 95,315 95,312 *Cardiology dataset includes articles through 2025; all other specialties include articles through 2024. Quality Proxy: Journal Quartiles. Assessing whether a recommendation preserves “appropriate quality” requires operationalizing quality. We use SCImago Journal Rank quartiles as a proxy. Quartiles (Q1–Q4) are computed annually based on citation metrics within subject categories, with Q1 representing the top 25% of journals. Model Development. We developed specialty-specific recommendation models using transformer-based architectures. Each model was trained on the corresponding specialty corpus using established techniques from the information retrieval literature. Models were optimized for ranking journals by relevance to input manuscripts. We evaluated our models using both standard and Quality-Aware metrics. Standard Metrics : Ground-truth journal refers to the journal in which the article was eventually published. Accuracy@K : The proportion of test cases where the ground-truth journal appears in the top K recommendations. We report K = 1, 3, 5, and 10. MRR : The average of 1/rank for the ground-truth journal across test cases. Higher MRR indicates the correct journal tends to appear earlier in the ranking. NDCG : A ranking quality metric that applies position-based discounting. Unlike MRR, which considers only the position of the correct answer, NDCG can incorporate graded relevance. Quality-Aware Metrics : We introduce metrics that assess whether recommendations preserve appropriate quality levels, using quartiles as a proxy for quality. Quality Accuracy@K : The proportion of test cases where at least one journal in the top K recommendations is in the same quartile as the ground-truth journal. This metric captures “acceptable” recommendations where the exact journal differs but quality is preserved. Quality Consistency@K : The average proportion of journals within the top K recommendations that belong to the same quartile as the ground-truth journal. This metric measures the consistency of the recommendations, quantifying what percentage of the suggested journals align with the quality tier of the target journal. Over/Undersell Rate : The proportion of test cases where the top-1 recommendation is in a higher or lower quartile than the ground-truth journal. Severe Over/Undersell Rate : The proportion of test cases where the top-1 recommendation is two or more quartiles above or below the ground-truth journal. This represents a substantial quality mismatch. We distinguish between undersell/oversell (one quartile difference) and severe undersell/oversell (two or more quartiles). Moderate quality shifts are less concerning; quartile boundaries are arbitrary cutoffs on a continuous distribution, and a journal at the bottom of Q1 may be practically indistinguishable from one at the top of Q2. Severe undersell, however, represents meaningful quality mismatch and the most consequential error type. Interpreting Metric Combinations. Individual metrics provide partial information. The combination of metrics provides a more comprehensive assessment of recommendation reliability. Table 3 summarizes what metrics prior systems report. Notably, none report the combination of ranking quality (MRR, NDCG) with quality-tier preservation metrics needed to assess list reliability. Table 3 Metrics Reported by Journal Recommendation Systems System Acc@1 Acc@10 P/R/F1 MRR NDCG Q.Acc QC@3 Over/Under Jane 23% — — — — — — — Pubmender 50% 86% ✓ — — — — — S2CFT 68% 99% — — — — — — PSRMTE 52% 97% — — — — — — SimCPSR 52% 95% — — — — — — This work (avg.) 47.9% 82.0% — 0.60 0.68 67.7 54.0 ✓ Note: Pubmender reports Macro/Micro Precision, Recall, and F1 at Top-K cutoffs. No system reports both ranking quality metrics (MRR, NDCG) and Quality-Aware metrics. High MRR + High NDCG + High Quality Accuracy: The system produces coherent recommendation lists. The correct journal ranks highly, and even when it is not first, alternatives are of appropriate quality. Users can trust the list as a whole. High MRR + High NDCG + Low Quality Accuracy: The system finds the correct journal but populates the list with inappropriate alternatives. Users cannot trust recommendations beyond the top result. Low MRR + High Quality Accuracy: The system understands quality but not topical fit. Recommendations are of appropriate quality but the correct journal is buried in the list. This framework allows developers to diagnose specific failure modes and users to assess whether recommendations are actionable. Results Standard Performance Metrics. Table 4 presents standard metrics for all five specialties. Models achieved an average accuracy@1 of 47.9% (ranging from 39.9% to 55.0%). Accuracy@10 consistently exceeded 76%, reaching 85.5% for OB/GYN. MRR ranged from 0.531 to 0.654, and NDCG ranged from 0.611 to 0.716 (Table 4 ). Table 4 Standard Performance Metrics Specialty Acc@1 Acc@3 Acc@5 Acc@10 MRR NDCG OB/GYN 53.2% 71.6% 78.1% 85.5% 0.645 0.714 Pediatrics 55.0% 72.2% 78.0% 84.2% 0.654 0.716 Cardiology 39.9% 61.4% 68.9% 76.3% 0.531 0.611 Radiology 42.6% 65.5% 73.3% 81.4% 0.562 0.648 Surgery 48.9% 69.6% 75.7% 82.8% 0.612 0.686 Acc@K: Frequency of correct journal appearing within the top K recommendations. MRR: Mean Reciprocal Rank. NDCG: Normalized Discounted Cumulative Gain. Quality-Aware Performance. Table 5 presents Quality-Aware metrics. Quality Accuracy@1 consistently exceeded raw accuracy. While the average Top-1 accuracy was 47.9%, the average Quality Accuracy@1 was 67.7%, representing a difference of approximately 20 percentage points. Furthermore, Quality Consistency@3 averaged 54.0%. Table 5 Quality-Specific Performance Metrics Specialty Q.Acc@1 Q.Acc@3 QC@3 QC@10 Under. Sev.Under. Over. OB/GYN 71.4% 90.4% 55.8% 43.0% 10.8% 5.5% 17.9% Pediatrics 68.7% 86.8% 51.4% 42.5% 19.0% 9.2% 12.3% Cardiology 67.3% 86.6% 58.4% 49.9% 17.8% 5.2% 14.9% Radiology 63.1% 89.4% 52.0% 40.7% 23.8% 11.5% 13.1% Surgery 68.2% 88.9% 52.5% 42.8% 18.4% 8.4% 13.5% Analysis of Quality-Level Errors. Beyond Quality Accuracy, we examined the direction and magnitude of quality-level errors. The system’s errors were categorized by their impact on journal quality. Severe Undersell Rates, which represent the most significant downward errors, averaged 7.9% across all specialties. Cardiology demonstrated the lowest Severe Undersell Rate at 5.2%, while Radiology exhibited a higher rate of 11.5%. General Undersell Rates ranged from 10.8% in OB/GYN to 23.8% in Radiology. Conversely, Oversell Rates, where the system suggested a higher-quality journal than the target, were relatively stable across specialties, peaking at 17.9% in OB/GYN and reaching a low of 12.3% in Pediatrics. Stratified Performance by Ground-Truth Quartile. Table 6 presents performance stratified by the ground-truth journal’s quartile tier, averaged across specialties. The system performed best on Q1 papers (Quality Accuracy@1: 77.0%, Quality Consistency@3: 64.8%), with decreasing performance for lower tiers. Notably, severe undersell was absent for Q3/Q4 papers by definition, while Q1 papers showed the highest Severe Undersell Rate (10.8%), reflecting the asymmetric risk for top-tier manuscripts. Oversell Rates showed the inverse pattern, with Q3/Q4 papers most affected (36.6%). Table 6 Stratified Performance by Ground-Truth Quartile (Averaged Across Specialties) GT Tier Q.Acc@1 Q.Acc@3 QC@3 QC@10 Under. Sev.Under. Over. Q1 77.0% 93.5% 64.8% 53.6% 23.0% 10.8% 0.0% Q2 60.3% 88.2% 45.6% 37.3% 17.9% 8.8% 21.8% Q3/Q4 57.2% 76.8% 40.8% 30.7% 6.2% 0.0% 36.6% Overall Reliability of the Recommendation List. Evaluation across specialties shows a relationship between ranking position and quality preservation. The mean MRR across all specialties was 0.60 and the mean NDCG was 0.68. These ranking scores occurred alongside a mean Quality Accuracy@1 of 67.7% and Quality Accuracy@3 that exceeded 86% in every specialty (Table 5 ). Discussion The Limitation of Binary Evaluation. Prior studies have treated journal recommendation as a classification problem with binary success criteria: either the system identified the exact ground-truth journal or it did not. This formulation ignores the ordinal structure of journal quality. In domains where labels have inherent ordering, treating all errors as equivalent discards information that matters for real-world utility. Our results demonstrate this concretely. Under the standard “binary” approach, the system appears to have a 52% error rate (based on 47.9% raw accuracy). However, when applying Quality-Aware metrics, this perceived failure rate drops to 32.3%. This shift occurs because more than a third of the mismatches identified by standard metrics actually involved journals of equivalent quality. From a user’s perspective, a recommendation for a different journal within the same quality tier is not a failure but a valid alternative. Standard metrics are blind to this nuance, making the system appear significantly less reliable than it is in real-world practice. It is important to note that moderate Undersell and Oversell Rates (10–20%) should be interpreted with caution. Many high-quality manuscripts can reasonably be published across adjacent journal tiers. However, the low Severe Undersell Rates in specialties like OB/GYN and Cardiology indicate that extreme mismatches, where a high-quality paper is recommended to significantly lower-quality venues, are relatively rare. Therefore, when evaluating recommendation quality, we focus primarily on Severe Undersell and Oversell Rates as more meaningful indicators of recommendation accuracy. Severe undersell (recommending journals significantly below the paper’s quality level) represents a greater practical concern, as it may lead authors to miss opportunities for higher-impact publication venues. Our system achieves Severe Undersell Rates below 6% for OB/GYN and Cardiology, indicating that extreme mismatches are rare for these specialties. The stratified analysis (Table 6) reveals an asymmetric risk profile. Q1 papers face the highest Severe Undersell Rate (10.8%), meaning top-tier manuscripts carry the greatest risk of being directed to substantially lower-quality venues—precisely the scenario most consequential for authors. Conversely, Q3/Q4 papers show the highest Oversell Rate (36.6%) but zero severe undersell, suggesting the system errs toward recommending higher-quality journals for lower-tier papers, a more benign failure mode. This asymmetry reinforces our recommendation to stratify evaluation by quality tier, as aggregate metrics mask these tier-specific risk profiles. Metrics as Proxies. The use of quartiles as a proxy for journal quality is imperfect. Quartiles are computed from citation metrics, which imperfectly capture quality and may fluctuate annually. To mitigate this temporal variability, we indexed each article according to the specific quartile assigned to the journal during its year of publication. Quartiles might not reflect subspecialty-specific hierarchies, and two journals in the same quartile may have substantially different scopes. However, the goal is not to perfectly measure quality but to approximate it well enough to distinguish acceptable errors from problematic ones. Quartiles provide a standardized, reproducible proxy that is widely understood. The methodology we present could be adapted to other quality proxies (impact factor percentiles, expert ratings) if preferred for specific contexts. Reliability as a First-Class Concern. A key contribution of this work is framing recommendation reliability as a first-class evaluation concern. When a user receives a recommendation list, they must decide whether to trust it. Current metrics provide no guidance. A system with 85% accuracy could produce lists where errors are minor or lists where errors are severe. The combination of ranking metrics (MRR, NDCG) with Quality-Aware metrics addresses this. High values across all metrics indicate that the system produces coherent lists: correct journals rank highly, and alternatives are of appropriate quality. This combination tells developers their system is ready for deployment and tells users their recommendations are actionable. Our findings also demonstrate that the system produces a highly reliable recommendation “environment.” This is evidenced by the synergy between high ranking metrics and high Quality Accuracy. Essentially, even when the ground-truth journal is not ranked first, the other journals in the Top-3 or Top-10 are almost always from the correct quality tier. From a researcher’s perspective, this means the entire list is applicable. The alternatives provided are not random “filler” but are clinically and academically appropriate. Without this Quality-Aware validation, a system could achieve high statistical accuracy while populating the list with misleading or low-quality alternatives, making the tool far less useful in a professional setting. Applicability Beyond Journal Recommendation. The methodology generalizes to any recommendation task where labels have ordinal structure. Grant program recommendation, conference selection, reviewer matching—many academic recommendation tasks involve label spaces with implicit quality ordering. Binary accuracy evaluation in such tasks discards information. Quality-Aware or analogous ordinal-aware metrics provide a more complete picture. Clinical Implications. Accurate journal targeting carries implications beyond academic productivity. In clinical medicine, timely dissemination of research findings, such as novel treatment protocols, safety signals, or diagnostic criteria, can directly influence patient care. When a recommendation system directs a clinically significant manuscript to a substantially lower-quality journal, the work may reach a smaller audience, receive fewer citations, and ultimately have reduced impact on clinical guidelines and practice. For individual researchers, particularly early-career clinicians balancing clinical duties with academic output, repeated submission to mismatched journals can delay publication by months, a period during which evolving clinical evidence may lose relevance or priority. Quality-Aware evaluation, therefore, serves not only as a methodological refinement but as a safeguard helping to ensure that recommendation tools support, rather than hinder, the translation of clinical research into practice. Strengths and Limitations. A major strength of this study is the introduction of a Quality-Aware evaluation framework validated across five medical specialties on over 95,000 test samples, providing a systematic methodology for assessing recommendation reliability beyond binary accuracy. The use of specialty-specific models trained on nearly one million PubMed articles further strengthens the generalizability of our findings within the evaluated domains. Several limitations should be acknowledged. Our evaluation is retrospective, assessing performance on already-published articles, which introduces survivorship bias. The ground truth assumes the publication venue represents the optimal target, whereas in practice, publication reflects strategic decisions, prior rejections, and author preferences—meaning some apparent “undersells” may reflect intentional choices rather than system errors. Our quality proxy relies on SCImago quartiles, which are citation-derived and may not fully capture journal quality or subspecialty-specific hierarchies; moreover, quartile boundaries are arbitrary cutoffs on a continuous distribution, potentially overstating quality mismatch in boundary cases. All models were trained within specialty-specific corpora, and generalizability to interdisciplinary manuscripts remains uncertain. Models were trained using only titles, abstracts, and MeSH terms rather than full manuscript text, which may omit methodological details that influence journal fit—though this reflects the practical reality that researchers typically seek recommendations before manuscripts are finalized. This study also lacks user-centered evaluation; whether Quality-Aware metrics correlate with researcher satisfaction or influence submission behavior remains to be tested. Finally, direct comparison with prior systems was not possible, as earlier tools were evaluated on different datasets or did not report the metrics we propose. Implications for Journal Recommendation System Evaluation. Based on this study, we propose a set of considerations for the evaluation of journal recommendation systems: 1. Reporting Quality-Aware metrics alongside accuracy. Quality Accuracy@1 reveals what proportion of “errors” are acceptable. This provides essential context for interpreting accuracy numbers. Quality Consistency@K provides insight into how consistent and reliable the list may be. 2. Reporting Undersell Rates. Undersell Rate and Severe Undersell Rate quantify how often systems recommend lower-quality journals than appropriate. These metrics capture failure modes invisible to standard accuracy. 3. Interpreting metrics in combination. High MRR + high NDCG + high Quality Accuracy indicates reliable recommendations. Reporting and interpreting these together rather than in isolation may provide a more informative evaluation. 4. Stratify by quality tier. Performance may vary across the quality spectrum. Consider evaluating and reporting stratified metrics, particularly for higher quality manuscripts where errors are most consequential. Conclusion Standard evaluation metrics for journal recommendation treat all prediction errors as equivalent. This binary formulation fails to capture whether errors preserve appropriate quality levels. We introduced Quality-Aware evaluation metrics using journal quartiles as a quality proxy and demonstrated how these metrics combine with ranking metrics to characterize recommendation reliability. Furthermore, the low rate of severe quality downgrading, combined with strong overall ranking performance, demonstrates that these models produce reliable recommendations. Even when the specific target journal is not the primary result, the suggested alternatives maintain high academic standards, fostering user trust. We recommend the adoption of Quality-Aware metrics as standard practice to better characterize real-world utility. Declarations Author Contributions Conceptualization: Nir Roguin; Methodology: Nir Roguin; Software: Nir Roguin; Formal analysis and investigation: Nir Roguin; Data curation: Nir Roguin; Writing — original draft preparation: Nir Roguin; Writing — review and editing: Nir Roguin, Omri Dominsky, Yariv Yogev, Yoav Baruch; Supervision: Yoav Baruch. Funding: The authors did not receive support from any organization for the submitted work. Competing Interests: Nir Roguin is the developer of PubGo, a freely available journal recommendation platform. The remaining authors declare no competing interests. Ethics Approval: Not applicable. This study used publicly available metadata from PubMed and did not involve human participants, animal research, or sensitive data. Consent to Participate: Not applicable. Consent to Publish: Not applicable. Data Availability: The datasets analyzed during the current study were derived from publicly available PubMed metadata. The evaluation framework code is available from the corresponding author upon reasonable request. References Feng, X., Zhang, H., Ren, Y., Shang, P., Zhu, Y., Liang, Y., Guan, R., & Xu, D. (2019). The deep learning-based recommender system “Pubmender” for choosing a biomedical publication venue: Development and validation study. Journal of Medical Internet Research, 21 (5), e12957. https://doi.org/10.2196/12957 Le, D. H., Doan, T. T., Huynh, S. T., & Nguyen, B. T. (2022). SimCPSR: Simple contrastive learning for paper submission recommendation system. In Asian Conference on Intelligent Information and Database Systems (pp. 51–63). Springer. arXiv:2205.05940 Nguyen, D., Huynh, S., Huynh, P., Dinh, C. V., & Nguyen, B. T. (2021). S2CFT: A new approach for paper submission recommendation. In T. Bureš et al. (Eds.), SOFSEM 2021: Theory and Practice of Computer Science. Lecture Notes in Computer Science (Vol. 12607, pp. 563–573). Springer. https://doi.org/10.1007/978-3-030-67731-2_41 Nguyen, D. H., Huynh, S. T., Dinh, C. V., Huynh, P. T., & Nguyen, B. T. (2022). PSRMTE: Paper submission recommendation using mixtures of transformer. Expert Systems with Applications, 202 , 117096. https://doi.org/10.1016/j.eswa.2022.117096 Schuemie, M. J., & Kors, J. A. (2008). Jane: Suggesting journals, finding experts. Bioinformatics, 24 (5), 727–728. https://doi.org/10.1093/bioinformatics/btn006 Son, J., & Kim, S. B. (2018). Academic paper recommender system using multilevel simultaneous citation networks. Decision Support Systems, 105 , 24–33. https://doi.org/10.1016/j.dss.2017.10.011 Wang, D., Liang, Y., Xu, D., Feng, X., & Guan, R. (2018). A content-based recommender system for computer science publications. Knowledge-Based Systems, 157 , 1–9. https://doi.org/10.1016/j.knosys.2018.05.001 Additional Declarations Competing interest reported. Nir Roguin is the developer of PubGo, a freely available journal recommendation platform. The remaining authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9053132","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":614697642,"identity":"563e1a77-1a82-4e93-bc98-f66111fe11ed","order_by":0,"name":"Nir Roguin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6ElEQVRIiWNgGAWjYLCCBAYbMM0MEzjwgLCWNLAGhJYEwvYcRtXCgE+LeXv7NYmHO87n8fOfP8Bc8OswA3/7AUa8tsicOVMmkXjmdrHkjGQG5pl9hxkkziTgd5iERE6aRGLb7cQNN4AO4+0BuvAGAb9AtZxL3HD+MESLPGEt6ceAWg4kbjgAdBjPj8MMBgS18JxhtkhsSwb5xeDwzIZ0HsMziQ34tbC3P7z5s80OGGIHHz4u+GMtJ3f88OEPH/BoYWDgMQCRYGMPMLYx8DAwMDbg1cDAwP4AroWB4Q8BxaNgFIyCUTAiAQBHRVCH9bejtQAAAABJRU5ErkJggg==","orcid":"","institution":"Tel Aviv Sourasky Medical Center","correspondingAuthor":true,"prefix":"","firstName":"Nir","middleName":"","lastName":"Roguin","suffix":""},{"id":614697643,"identity":"795761c9-98c2-4245-ad66-c68a637f103b","order_by":1,"name":"Omri Dominsky","email":"","orcid":"","institution":"Tel Aviv Sourasky Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Omri","middleName":"","lastName":"Dominsky","suffix":""},{"id":614697644,"identity":"d3e3c095-69c5-4513-bb59-da751b880ad6","order_by":2,"name":"Yariv Yogev","email":"","orcid":"","institution":"Tel Aviv Sourasky Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Yariv","middleName":"","lastName":"Yogev","suffix":""},{"id":614697645,"identity":"f543c30b-8f4c-4b96-866d-889405eaf2c1","order_by":3,"name":"Yoav Baruch","email":"","orcid":"","institution":"Tel Aviv Sourasky Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Yoav","middleName":"","lastName":"Baruch","suffix":""}],"badges":[],"createdAt":"2026-03-06 18:09:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9053132/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9053132/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":105904836,"identity":"2ae77f85-585b-4590-b695-53519904de80","added_by":"auto","created_at":"2026-04-01 10:10:45","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1124711,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9053132/v1/bd832eee-1ea8-4899-8896-a68927eb2d2b.pdf"}],"financialInterests":"Competing interest reported. Nir Roguin is the developer of PubGo, a freely available journal recommendation platform. The remaining authors declare no competing interests.","formattedTitle":"Quality-Aware Evaluation for Journal Recommendation","fulltext":[{"header":"Statement of Significance","content":"\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"624\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 147px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eProblem\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 477px;\"\u003e\n \u003cp\u003eJournal recommendation systems are evaluated using Top-K accuracy, which treats all prediction errors equivalently. This binary formulation fails to distinguish between errors that recommend similar-quality journals versus errors that recommend substantially lower-quality journals.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 147px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eWhat is Already Known\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 477px;\"\u003e\n \u003cp\u003ePrior systems from Jane (Schuemie \u0026amp; Kors, 2008) through transformer-based approaches (Nguyen, Huynh, Dinh, et al., 2022; Le et al., 2022) have progressively improved Top-K accuracy through architectural advances. However, evaluation methodology has remained unchanged, with all systems using the same binary accuracy metrics.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 147px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eWhat This Paper Adds\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 477px;\"\u003e\n \u003cp\u003eWe introduce Quality-Aware evaluation metrics that use journal quartiles as a proxy for quality. Combined with ranking metrics (MRR, NDCG), these metrics characterize whether recommendation lists are reliable. We validate this methodology across five medical specialties with over 95,000 test samples.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 147px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eWho Benefits\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 477px;\"\u003e\n \u003cp\u003eDevelopers of journal recommendation systems who need evaluation methods that better reflect real-world utility; researchers evaluating recommendation systems in domains where labels have ordinal structure; users of recommendation tools who need to assess whether outputs are trustworthy.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"Introduction","content":"\u003cp\u003eAfter completing a research project, selecting an appropriate journal can be challenging. With thousands of potential journals, researchers must identify journals whose scope matches their manuscript while also considering factors like quality, audience, and likelihood of acceptance. An unsuitable choice may result in substantial delays due to repeated submissions and rejections, potentially compromising the perceived originality of the study, and ultimately contributing to researcher fatigue and loss of motivation. These challenges have motivated the development of automated journal recommendation systems.\u003c/p\u003e \u003cp\u003eThe field has progressed substantially over the past two decades. Early journal recommendation approaches relied on simple word-based similarity between indexed abstracts (Schuemie \u0026amp; Kors, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2008\u003c/span\u003e). Subsequent work introduced machine learning with engineered features (Wang et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Son \u0026amp; Kim, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2018\u003c/span\u003e), followed by deep learning approaches that demonstrated gains through learned representations (Feng et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). Most recently, transformer-based systems have achieved state-of-the-art performance through semantic modeling (Nguyen, Huynh, Dinh, et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Le et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eDespite architectural advances, evaluation methodology has remained static. Nearly all systems report Top-K accuracy: the proportion of test cases where the ground-truth journal (the true journal in which the article was published) appears among the top K recommendations. This metric, established in early work and inherited by subsequent papers, embeds a problematic assumption: that all prediction errors are equivalent.\u003c/p\u003e \u003cp\u003eNot all recommendation errors are equivalent: suggesting a journal of similar quality is acceptable, while suggesting a substantially lower-quality journal is not, yet standard accuracy metrics treat both identically.\u003c/p\u003e \u003cp\u003eFrom a user\u0026rsquo;s perspective, the key question is whether the recommended list can be trusted. If only one suggested journal is appropriate while the rest are not, the list is difficult to use. This becomes particularly important when the correct journal is not ranked first. In contrast, when all top recommendations are of appropriate quality, the list becomes useful.\u003c/p\u003e \u003cp\u003eCurrent evaluation provides no insight into this distinction. A system with 85% accuracy could be producing recommendation lists where errors are minor (similar-quality alternatives) or major (inappropriate journals). Developers optimizing for Top-K accuracy during development have no visibility into which pattern their system exhibits.\u003c/p\u003e \u003cp\u003eIn this paper, we present a Quality-Aware evaluation methodology for journal recommendation. Our approach uses journal quartiles (from SCImago) as a proxy for quality. We introduce metrics that measure whether recommendations preserve appropriate quality levels, and we examine how these metrics combine with standard ranking metrics to characterize recommendation reliability. We validate this methodology across five medical specialties, demonstrating consistent patterns that may help guide the evaluation of journal recommendation systems.\u003c/p\u003e \u003cp\u003eJournal recommendation systems have evolved through distinct methodological phases. Schuemie and Kors (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2008\u003c/span\u003e) established the task as article similarity aggregation using Term Frequency\u0026ndash;Inverse Document Frequency (TF-IDF) over Medline abstracts. While limited by vocabulary mismatch, Jane demonstrated the feasibility of automated recommendation and remains a reference baseline.\u003c/p\u003e \u003cp\u003eClassical machine learning approaches introduced feature engineering and discriminative classifiers. Wang et al. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) achieved 61% Top-1 accuracy using logistic regression with TF-IDF features. Son and Kim (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) improved this using title, abstract, and keyword features with Multilayer Perceptron (MLP) classifiers. These works established Top-K accuracy as the standard evaluation metric.\u003c/p\u003e \u003cp\u003eDeep learning approaches reduced reliance on feature engineering. Feng et al. (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) used word2vec embeddings with Convolutional Neural Network (CNN) classification on 880,000 PubMed abstracts, achieving 87% Top-10 accuracy. Nguyen, Huynh, et al. (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) systematically compared architectures and found that representation quality mattered more than specific architectural choices.\u003c/p\u003e \u003cp\u003eTransformer-based systems represent the current state-of-the-art. Nguyen, Huynh, Dinh, et al. (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) demonstrated that Bidirectional Encoder Representations from Transformers (BERT)-family encoders outperform prior approaches across all Top-K metrics. Le et al. (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) further improved performance through contrastive learning with journal scope text. These systems frame recommendation as semantic alignment between manuscripts and journals.\u003c/p\u003e \u003cp\u003eAll systems report Top-K accuracy, with some including MRR or NDCG. None assess whether prediction errors preserve appropriate quality levels. None measure the magnitude or direction of errors in terms of journal quality. This represents a gap between what developers measure and what users need. A user receiving recommendations needs to know not just whether the system is accurate, but whether its errors are acceptable. Current metrics provide no insight into this question.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e\u003cstrong\u003eData Collection.\u003c/strong\u003e We constructed datasets for five medical specialties: Cardiology, Obstetrics and Gynecology (OB/GYN), Pediatrics, Radiology, and Surgery. For each specialty, we identified relevant journals through NLM catalog classifications and retrieved articles published between 2020 and 2024 (2020\u0026ndash;2025 for Cardiology) via the PubMed E-utilities API. We extracted title, abstract, and MeSH terms for each article. Articles lacking abstracts or with abstracts shorter than 50 words were excluded.\u003c/p\u003e\n\u003cp\u003eData were split temporally: for training (80%), for validation (10%), and for testing (10%). Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents dataset characteristics. The total corpus comprises nearly one million articles published across approximately 1,500 journals.\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eEvaluation Approaches in Prior Work\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"6\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eSystem\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003eYear\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003ePrimary Metric\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003eRanking Metrics\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003eQuality-Aware\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003eError Analysis\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eJane (Schuemie \u0026amp; Kors, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2008\u003c/span\u003e)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e2008\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003eRecall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eWang et al. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2018\u003c/span\u003e)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e2018\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003eAccuracy@K\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003ePubmender (Feng et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2019\u003c/span\u003e)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e2019\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003eAccuracy@K\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003ePSRMTE (Nguyen, Huynh, Dinh, et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e2022\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003eAccuracy@K\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eSimCPSR (Le et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e2022\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003eAccuracy@K\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003eNo\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003e\u003cstrong\u003eThis work\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e\u003cstrong\u003e2025\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003e\u003cstrong\u003eAccuracy@K\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e\u003cstrong\u003eMRR, NDCG\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003e\u003cstrong\u003eYes\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003e\u003cstrong\u003eYes\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cdiv class=\"gridtable\"\u003e\n \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003cbr\u003e\u003c/div\u003e\u0026nbsp;\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eDataset Characteristics\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"5\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eSpecialty\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003eTotal\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003eTrain\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003eValidation\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003eTest\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eCardiology*\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e241,391\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\n \u003cp\u003e193,304\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\n \u003cp\u003e24,027\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\n \u003cp\u003e24,060\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eOB/GYN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e95,990\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\n \u003cp\u003e76,799\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\n \u003cp\u003e9,595\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\n \u003cp\u003e9,596\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003ePediatrics\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e143,782\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\n \u003cp\u003e115,127\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\n \u003cp\u003e14,311\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\n \u003cp\u003e14,344\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eRadiology\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e166,823\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\n \u003cp\u003e133,619\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\n \u003cp\u003e16,603\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\n \u003cp\u003e16,601\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eSurgery\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e307,686\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\n \u003cp\u003e246,196\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\n \u003cp\u003e30,779\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\n \u003cp\u003e30,711\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e\u003cstrong\u003e955,672\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\n \u003cp\u003e\u003cstrong\u003e765,045\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\n \u003cp\u003e\u003cstrong\u003e95,315\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\n \u003cp\u003e\u003cstrong\u003e95,312\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003ctfoot\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"5\"\u003e\u003cem\u003e*Cardiology dataset includes articles through 2025; all other specialties include articles through 2024.\u003c/em\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tfoot\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cstrong\u003eQuality Proxy: Journal Quartiles.\u003c/strong\u003e Assessing whether a recommendation preserves \u0026ldquo;appropriate quality\u0026rdquo; requires operationalizing quality. We use SCImago Journal Rank quartiles as a proxy. Quartiles (Q1\u0026ndash;Q4) are computed annually based on citation metrics within subject categories, with Q1 representing the top 25% of journals.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eModel Development.\u003c/strong\u003e We developed specialty-specific recommendation models using transformer-based architectures. Each model was trained on the corresponding specialty corpus using established techniques from the information retrieval literature. Models were optimized for ranking journals by relevance to input manuscripts. We evaluated our models using both standard and Quality-Aware metrics.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStandard Metrics\u003c/strong\u003e: Ground-truth journal refers to the journal in which the article was eventually published. \u003cstrong\u003eAccuracy@K\u003c/strong\u003e: The proportion of test cases where the ground-truth journal appears in the top K recommendations. We report K\u0026thinsp;=\u0026thinsp;1, 3, 5, and 10. \u003cstrong\u003eMRR\u003c/strong\u003e: The average of 1/rank for the ground-truth journal across test cases. Higher MRR indicates the correct journal tends to appear earlier in the ranking. \u003cstrong\u003eNDCG\u003c/strong\u003e: A ranking quality metric that applies position-based discounting. Unlike MRR, which considers only the position of the correct answer, NDCG can incorporate graded relevance.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eQuality-Aware Metrics\u003c/strong\u003e: We introduce metrics that assess whether recommendations preserve appropriate quality levels, using quartiles as a proxy for quality. \u003cstrong\u003eQuality Accuracy@K\u003c/strong\u003e: The proportion of test cases where at least one journal in the top K recommendations is in the same quartile as the ground-truth journal. This metric captures \u0026ldquo;acceptable\u0026rdquo; recommendations where the exact journal differs but quality is preserved. \u003cstrong\u003eQuality Consistency@K\u003c/strong\u003e: The average proportion of journals within the top K recommendations that belong to the same quartile as the ground-truth journal. This metric measures the consistency of the recommendations, quantifying what percentage of the suggested journals align with the quality tier of the target journal. \u003cstrong\u003eOver/Undersell Rate\u003c/strong\u003e: The proportion of test cases where the top-1 recommendation is in a higher or lower quartile than the ground-truth journal. \u003cstrong\u003eSevere Over/Undersell Rate\u003c/strong\u003e: The proportion of test cases where the top-1 recommendation is two or more quartiles above or below the ground-truth journal. This represents a substantial quality mismatch.\u003c/p\u003e\n\u003cp\u003eWe distinguish between undersell/oversell (one quartile difference) and severe undersell/oversell (two or more quartiles). Moderate quality shifts are less concerning; quartile boundaries are arbitrary cutoffs on a continuous distribution, and a journal at the bottom of Q1 may be practically indistinguishable from one at the top of Q2. Severe undersell, however, represents meaningful quality mismatch and the most consequential error type.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eInterpreting Metric Combinations.\u003c/strong\u003e Individual metrics provide partial information. The combination of metrics provides a more comprehensive assessment of recommendation reliability. Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e summarizes what metrics prior systems report. Notably, none report the combination of ranking quality (MRR, NDCG) with quality-tier preservation metrics needed to assess list reliability.\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\" class=\"fr-table-selection-hover\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eMetrics Reported by Journal Recommendation Systems\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"9\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eSystem\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003eAcc@1\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003eAcc@10\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003eP/R/F1\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003eMRR\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003eNDCG\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c7\"\u003e\n \u003cp\u003eQ.Acc\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c8\"\u003e\n \u003cp\u003eQC@3\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c9\"\u003e\n \u003cp\u003eOver/Under\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eJane\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003e23%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c7\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c8\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c9\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003ePubmender\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003e50%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003e86%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e✓\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c7\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c8\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c9\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eS2CFT\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003e68%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003e99%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c7\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c8\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c9\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003ePSRMTE\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003e52%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003e97%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c7\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c8\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c9\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eSimCPSR\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003e52%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003e95%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c7\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c8\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c9\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003e\u003cstrong\u003eThis work (avg.)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003e\u003cstrong\u003e47.9%\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003e\u003cstrong\u003e82.0%\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e\u0026mdash;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c5\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.60\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c6\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.68\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c7\"\u003e\n \u003cp\u003e\u003cstrong\u003e67.7\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c8\"\u003e\n \u003cp\u003e\u003cstrong\u003e54.0\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c9\"\u003e\n \u003cp\u003e\u003cstrong\u003e✓\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003ctfoot\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"9\"\u003e\u003cem\u003eNote: Pubmender reports Macro/Micro Precision, Recall, and F1 at Top-K cutoffs. No system reports both ranking quality metrics (MRR, NDCG) and Quality-Aware metrics.\u003c/em\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tfoot\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cstrong\u003eHigh MRR + High NDCG + High Quality Accuracy:\u0026nbsp;\u003c/strong\u003eThe system produces coherent recommendation lists. The correct journal ranks highly, and even when it is not first, alternatives are of appropriate quality. Users can trust the list as a whole.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHigh MRR + High NDCG + Low Quality Accuracy:\u0026nbsp;\u003c/strong\u003eThe system finds the correct journal but populates the list with inappropriate alternatives. Users cannot trust recommendations beyond the top result.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLow MRR + High Quality Accuracy:\u0026nbsp;\u003c/strong\u003eThe system understands quality but not topical fit. Recommendations are of appropriate quality but the correct journal is buried in the list.\u003c/p\u003e\n\u003cp\u003eThis framework allows developers to diagnose specific failure modes and users to assess whether recommendations are actionable.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e \u003cb\u003eStandard Performance Metrics.\u003c/b\u003e Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e presents standard metrics for all five specialties. Models achieved an average accuracy@1 of 47.9% (ranging from 39.9% to 55.0%). Accuracy@10 consistently exceeded 76%, reaching 85.5% for OB/GYN. MRR ranged from 0.531 to 0.654, and NDCG ranged from 0.611 to 0.716 (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eStandard Performance Metrics\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpecialty\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAcc@1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAcc@3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAcc@5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eAcc@10\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eMRR\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eNDCG\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOB/GYN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e53.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e71.6%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e78.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e85.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.645\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.714\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePediatrics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e55.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e72.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e78.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e84.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.654\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.716\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCardiology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e39.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e61.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e68.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e76.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.531\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.611\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRadiology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e42.6%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e73.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e81.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.562\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.648\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSurgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e48.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e69.6%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e75.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e82.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.612\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.686\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"7\"\u003e\u003cem\u003eAcc@K: Frequency of correct journal appearing within the top K recommendations. MRR: Mean Reciprocal Rank. NDCG: Normalized Discounted Cumulative Gain.\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eQuality-Aware Performance.\u003c/b\u003e Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e presents Quality-Aware metrics. Quality Accuracy@1 consistently exceeded raw accuracy. While the average Top-1 accuracy was 47.9%, the average Quality Accuracy@1 was 67.7%, representing a difference of approximately 20 percentage points. Furthermore, Quality Consistency@3 averaged 54.0%.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eQuality-Specific Performance Metrics\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"8\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpecialty\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQ.Acc@1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eQ.Acc@3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eQC@3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eQC@10\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eUnder.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eSev.Under.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eOver.\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOB/GYN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e71.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e90.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e55.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e43.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e10.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e5.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e17.9%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePediatrics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e68.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e86.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e51.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e42.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e19.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e9.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e12.3%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCardiology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e67.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e86.6%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e58.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e49.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e17.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e5.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e14.9%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRadiology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e63.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e89.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e52.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e40.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e23.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e11.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e13.1%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSurgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e68.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e88.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e52.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e42.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e18.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e8.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e13.5%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eAnalysis of Quality-Level Errors.\u003c/b\u003e Beyond Quality Accuracy, we examined the direction and magnitude of quality-level errors. The system\u0026rsquo;s errors were categorized by their impact on journal quality. Severe Undersell Rates, which represent the most significant downward errors, averaged 7.9% across all specialties. Cardiology demonstrated the lowest Severe Undersell Rate at 5.2%, while Radiology exhibited a higher rate of 11.5%.\u003c/p\u003e \u003cp\u003eGeneral Undersell Rates ranged from 10.8% in OB/GYN to 23.8% in Radiology. Conversely, Oversell Rates, where the system suggested a higher-quality journal than the target, were relatively stable across specialties, peaking at 17.9% in OB/GYN and reaching a low of 12.3% in Pediatrics.\u003c/p\u003e \u003cp\u003e \u003cb\u003eStratified Performance by Ground-Truth Quartile.\u003c/b\u003e Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e6\u003c/span\u003e presents performance stratified by the ground-truth journal\u0026rsquo;s quartile tier, averaged across specialties. The system performed best on Q1 papers (Quality Accuracy@1: 77.0%, Quality Consistency@3: 64.8%), with decreasing performance for lower tiers. Notably, severe undersell was absent for Q3/Q4 papers by definition, while Q1 papers showed the highest Severe Undersell Rate (10.8%), reflecting the asymmetric risk for top-tier manuscripts. Oversell Rates showed the inverse pattern, with Q3/Q4 papers most affected (36.6%).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab6\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 6\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eStratified Performance by Ground-Truth Quartile (Averaged Across Specialties)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"8\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGT Tier\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQ.Acc@1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eQ.Acc@3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eQC@3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eQC@10\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eUnder.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eSev.Under.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eOver.\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQ1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e77.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e93.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e64.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e53.6%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e23.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e10.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0.0%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQ2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e60.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e88.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e45.6%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e37.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e17.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e8.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e21.8%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQ3/Q4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e57.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e76.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e40.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e30.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e6.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e36.6%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eOverall Reliability of the Recommendation List.\u003c/b\u003e Evaluation across specialties shows a relationship between ranking position and quality preservation. The mean MRR across all specialties was 0.60 and the mean NDCG was 0.68. These ranking scores occurred alongside a mean Quality Accuracy@1 of 67.7% and Quality Accuracy@3 that exceeded 86% in every specialty (Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e).\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003e\u003cstrong\u003eThe Limitation of Binary Evaluation.\u0026nbsp;\u003c/strong\u003ePrior studies have treated journal recommendation as a classification problem with binary success criteria: either the system identified the exact ground-truth journal or it did not. This formulation ignores the ordinal structure of journal quality. In domains where labels have inherent ordering, treating all errors as equivalent discards information that matters for real-world utility.\u003c/p\u003e\n\u003cp\u003eOur results demonstrate this concretely. Under the standard “binary” approach, the system appears to have a 52% error rate (based on 47.9% raw accuracy). However, when applying Quality-Aware metrics, this perceived failure rate drops to 32.3%. This shift occurs because more than a third of the mismatches identified by standard metrics actually involved journals of equivalent quality. From a user’s perspective, a recommendation for a different journal within the same quality tier is not a failure but a valid alternative. Standard metrics are blind to this nuance, making the system appear significantly less reliable than it is in real-world practice.\u003c/p\u003e\n\u003cp\u003eIt is important to note that moderate Undersell and Oversell Rates (10–20%) should be interpreted with caution. Many high-quality manuscripts can reasonably be published across adjacent journal tiers. However, the low Severe Undersell Rates in specialties like OB/GYN and Cardiology indicate that extreme mismatches, where a high-quality paper is recommended to significantly lower-quality venues, are relatively rare.\u003c/p\u003e\n\u003cp\u003eTherefore, when evaluating recommendation quality, we focus primarily on Severe Undersell and Oversell Rates as more meaningful indicators of recommendation accuracy. Severe undersell (recommending journals significantly below the paper’s quality level) represents a greater practical concern, as it may lead authors to miss opportunities for higher-impact publication venues. Our system achieves Severe Undersell Rates below 6% for OB/GYN and Cardiology, indicating that extreme mismatches are rare for these specialties.\u003c/p\u003e\n\u003cp\u003eThe stratified analysis (Table 6) reveals an asymmetric risk profile. Q1 papers face the highest Severe Undersell Rate (10.8%), meaning top-tier manuscripts carry the greatest risk of being directed to substantially lower-quality venues—precisely the scenario most consequential for authors. Conversely, Q3/Q4 papers show the highest Oversell Rate (36.6%) but zero severe undersell, suggesting the system errs toward recommending higher-quality journals for lower-tier papers, a more benign failure mode. This asymmetry reinforces our recommendation to stratify evaluation by quality tier, as aggregate metrics mask these tier-specific risk profiles.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMetrics as Proxies.\u0026nbsp;\u003c/strong\u003eThe use of quartiles as a proxy for journal quality is imperfect. Quartiles are computed from citation metrics, which imperfectly capture quality and may fluctuate annually. To mitigate this temporal variability, we indexed each article according to the specific quartile assigned to the journal during its year of publication. Quartiles might not reflect subspecialty-specific hierarchies, and two journals in the same quartile may have substantially different scopes.\u003c/p\u003e\n\u003cp\u003eHowever, the goal is not to perfectly measure quality but to approximate it well enough to distinguish acceptable errors from problematic ones. Quartiles provide a standardized, reproducible proxy that is widely understood. The methodology we present could be adapted to other quality proxies (impact factor percentiles, expert ratings) if preferred for specific contexts.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eReliability as a First-Class Concern.\u0026nbsp;\u003c/strong\u003eA key contribution of this work is framing recommendation reliability as a first-class evaluation concern. When a user receives a recommendation list, they must decide whether to trust it. Current metrics provide no guidance. A system with 85% accuracy could produce lists where errors are minor or lists where errors are severe.\u003c/p\u003e\n\u003cp\u003eThe combination of ranking metrics (MRR, NDCG) with Quality-Aware metrics addresses this. High values across all metrics indicate that the system produces coherent lists: correct journals rank highly, and alternatives are of appropriate quality. This combination tells developers their system is ready for deployment and tells users their recommendations are actionable. Our findings also demonstrate that the system produces a highly reliable recommendation “environment.” This is evidenced by the synergy between high ranking metrics and high Quality Accuracy. Essentially, even when the ground-truth journal is not ranked first, the other journals in the Top-3 or Top-10 are almost always from the correct quality tier. From a researcher’s perspective, this means the entire list is applicable. The alternatives provided are not random “filler” but are clinically and academically appropriate. Without this Quality-Aware validation, a system could achieve high statistical accuracy while populating the list with misleading or low-quality alternatives, making the tool far less useful in a professional setting.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eApplicability Beyond Journal Recommendation.\u0026nbsp;\u003c/strong\u003eThe methodology generalizes to any recommendation task where labels have ordinal structure. Grant program recommendation, conference selection, reviewer matching—many academic recommendation tasks involve label spaces with implicit quality ordering. Binary accuracy evaluation in such tasks discards information. Quality-Aware or analogous ordinal-aware metrics provide a more complete picture.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical Implications.\u0026nbsp;\u003c/strong\u003eAccurate journal targeting carries implications beyond academic productivity. In clinical medicine, timely dissemination of research findings, such as novel treatment protocols, safety signals, or diagnostic criteria, can directly influence patient care. When a recommendation system directs a clinically significant manuscript to a substantially lower-quality journal, the work may reach a smaller audience, receive fewer citations, and ultimately have reduced impact on clinical guidelines and practice. For individual researchers, particularly early-career clinicians balancing clinical duties with academic output, repeated submission to mismatched journals can delay publication by months, a period during which evolving clinical evidence may lose relevance or priority. Quality-Aware evaluation, therefore, serves not only as a methodological refinement but as a safeguard helping to ensure that recommendation tools support, rather than hinder, the translation of clinical research into practice.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStrengths and Limitations.\u0026nbsp;\u003c/strong\u003eA major strength of this study is the introduction of a Quality-Aware evaluation framework validated across five medical specialties on over 95,000 test samples, providing a systematic methodology for assessing recommendation reliability beyond binary accuracy. The use of specialty-specific models trained on nearly one million PubMed articles further strengthens the generalizability of our findings within the evaluated domains.\u003c/p\u003e\n\u003cp\u003eSeveral limitations should be acknowledged. Our evaluation is retrospective, assessing performance on already-published articles, which introduces survivorship bias. The ground truth assumes the publication venue represents the optimal target, whereas in practice, publication reflects strategic decisions, prior rejections, and author preferences—meaning some apparent “undersells” may reflect intentional choices rather than system errors. Our quality proxy relies on SCImago quartiles, which are citation-derived and may not fully capture journal quality or subspecialty-specific hierarchies; moreover, quartile boundaries are arbitrary cutoffs on a continuous distribution, potentially overstating quality mismatch in boundary cases. All models were trained within specialty-specific corpora, and generalizability to interdisciplinary manuscripts remains uncertain. Models were trained using only titles, abstracts, and MeSH terms rather than full manuscript text, which may omit methodological details that influence journal fit—though this reflects the practical reality that researchers typically seek recommendations before manuscripts are finalized. This study also lacks user-centered evaluation; whether Quality-Aware metrics correlate with researcher satisfaction or influence submission behavior remains to be tested. Finally, direct comparison with prior systems was not possible, as earlier tools were evaluated on different datasets or did not report the metrics we propose.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eImplications for Journal Recommendation System Evaluation.\u0026nbsp;\u003c/strong\u003eBased on this study, we propose a set of considerations for the evaluation of journal recommendation systems:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e1. Reporting Quality-Aware metrics alongside accuracy.\u0026nbsp;\u003c/strong\u003eQuality Accuracy@1 reveals what proportion of “errors” are acceptable. This provides essential context for interpreting accuracy numbers. Quality Consistency@K provides insight into how consistent and reliable the list may be.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2. Reporting Undersell Rates.\u0026nbsp;\u003c/strong\u003eUndersell Rate and Severe Undersell Rate quantify how often systems recommend lower-quality journals than appropriate. These metrics capture failure modes invisible to standard accuracy.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3. Interpreting metrics in combination.\u0026nbsp;\u003c/strong\u003eHigh MRR + high NDCG + high Quality Accuracy indicates reliable recommendations. Reporting and interpreting these together rather than in isolation may provide a more informative evaluation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4. Stratify by quality tier.\u0026nbsp;\u003c/strong\u003ePerformance may vary across the quality spectrum. Consider evaluating and reporting stratified metrics, particularly for higher quality manuscripts where errors are most consequential.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eStandard evaluation metrics for journal recommendation treat all prediction errors as equivalent. This binary formulation fails to capture whether errors preserve appropriate quality levels. We introduced Quality-Aware evaluation metrics using journal quartiles as a quality proxy and demonstrated how these metrics combine with ranking metrics to characterize recommendation reliability.\u003c/p\u003e \u003cp\u003eFurthermore, the low rate of severe quality downgrading, combined with strong overall ranking performance, demonstrates that these models produce reliable recommendations. Even when the specific target journal is not the primary result, the suggested alternatives maintain high academic standards, fostering user trust.\u003c/p\u003e \u003cp\u003eWe recommend the adoption of Quality-Aware metrics as standard practice to better characterize real-world utility.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAuthor Contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eConceptualization: Nir Roguin; Methodology: Nir Roguin; Software: Nir Roguin; Formal analysis and investigation: Nir Roguin; Data curation: Nir Roguin; Writing \u0026mdash; original draft preparation: Nir Roguin; Writing \u0026mdash; review and editing: Nir Roguin, Omri Dominsky, Yariv Yogev, Yoav Baruch; Supervision: Yoav Baruch.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding:\u0026nbsp;\u003c/strong\u003eThe authors did not receive support from any organization for the submitted work.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting Interests:\u0026nbsp;\u003c/strong\u003eNir Roguin is the developer of PubGo, a freely available journal recommendation platform. The remaining authors declare no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthics Approval:\u0026nbsp;\u003c/strong\u003eNot applicable. This study used publicly available metadata from PubMed and did not involve human participants, animal research, or sensitive data.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent to Participate:\u0026nbsp;\u003c/strong\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent to Publish:\u0026nbsp;\u003c/strong\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Availability:\u0026nbsp;\u003c/strong\u003eThe datasets analyzed during the current study were derived from publicly available PubMed metadata. The evaluation framework code is available from the corresponding author upon reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eFeng, X., Zhang, H., Ren, Y., Shang, P., Zhu, Y., Liang, Y., Guan, R., \u0026amp; Xu, D. (2019). The deep learning-based recommender system \u0026ldquo;Pubmender\u0026rdquo; for choosing a biomedical publication venue: Development and validation study. \u003cem\u003eJournal of Medical Internet Research, 21\u003c/em\u003e(5), e12957. https://doi.org/10.2196/12957\u003c/li\u003e\n\u003cli\u003eLe, D. H., Doan, T. T., Huynh, S. T., \u0026amp; Nguyen, B. T. (2022). SimCPSR: Simple contrastive learning for paper submission recommendation system. In \u003cem\u003eAsian Conference on Intelligent Information and Database Systems\u003c/em\u003e (pp. 51\u0026ndash;63). Springer. arXiv:2205.05940\u003c/li\u003e\n\u003cli\u003eNguyen, D., Huynh, S., Huynh, P., Dinh, C. V., \u0026amp; Nguyen, B. T. (2021). S2CFT: A new approach for paper submission recommendation. In T. Bure\u0026scaron; et al. (Eds.), \u003cem\u003eSOFSEM 2021: Theory and Practice of Computer Science. Lecture Notes in Computer Science\u003c/em\u003e (Vol. 12607, pp. 563\u0026ndash;573). Springer. https://doi.org/10.1007/978-3-030-67731-2_41\u003c/li\u003e\n\u003cli\u003eNguyen, D. H., Huynh, S. T., Dinh, C. V., Huynh, P. T., \u0026amp; Nguyen, B. T. (2022). PSRMTE: Paper submission recommendation using mixtures of transformer. \u003cem\u003eExpert Systems with Applications, 202\u003c/em\u003e, 117096. https://doi.org/10.1016/j.eswa.2022.117096\u003c/li\u003e\n\u003cli\u003eSchuemie, M. J., \u0026amp; Kors, J. A. (2008). Jane: Suggesting journals, finding experts. \u003cem\u003eBioinformatics, 24\u003c/em\u003e(5), 727\u0026ndash;728. https://doi.org/10.1093/bioinformatics/btn006\u003c/li\u003e\n\u003cli\u003eSon, J., \u0026amp; Kim, S. B. (2018). Academic paper recommender system using multilevel simultaneous citation networks. \u003cem\u003eDecision Support Systems, 105\u003c/em\u003e, 24\u0026ndash;33. https://doi.org/10.1016/j.dss.2017.10.011\u003c/li\u003e\n\u003cli\u003eWang, D., Liang, Y., Xu, D., Feng, X., \u0026amp; Guan, R. (2018). A content-based recommender system for computer science publications. \u003cem\u003eKnowledge-Based Systems, 157\u003c/em\u003e, 1\u0026ndash;9. https://doi.org/10.1016/j.knosys.2018.05.001\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"journal recommendation, evaluation methodology, information retrieval, medical informatics, scholarly publishing","lastPublishedDoi":"10.21203/rs.3.rs-9053132/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9053132/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eObjective: \u003c/strong\u003eJournal recommendation systems are tools that analyze a manuscript’s content and suggest potential target journals. Most tools that suggest journals are evaluated using simple accuracy measures, such as Top-K accuracy metrics: the recommendation is considered correct only if the exact journal is predicted. All other recommendations are counted as errors. However, this approach ignores journal quality. Recommending a different journal of similar quality is very different from recommending a journal that is far below the appropriate quality level. We developed a Quality-Aware evaluation tool designed to assess whether journal recommendation tools suggest journals of an appropriate quality level, rather than simply whether they predict the exact target journal.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods: \u003c/strong\u003eWe developed specialty-specific journal recommendation models for medical fields using transformer-based architectures trained on nearly one million PubMed articles (2020–2024; 2020–2025 for Cardiology). Using SCImago quartiles as a proxy for journal quality, we evaluated models using both standard metrics and novel Quality-Aware metrics (Quality Accuracy, Undersell Rate, Severe Undersell Rate). We examined the relationship between these metric families to characterize when recommendation lists can be considered reliable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults: \u003c/strong\u003eAcross five specialties, mean accuracy@1 was 47.9% (range: 39.9%–55.0%), with Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG) of 0.60 and 0.68, respectively. However, Quality Accuracy@1 averaged 67.7%, exceeding raw accuracy by approximately 20 percentage points. This gap indicates that more than a third of all prediction “errors” under standard metrics involved journals of equivalent quality. Furthermore, Quality Consistency@3 averaged 54.0%, demonstrating that, on average, the majority of journals in the top-3 recommendations consistently aligned with the target journal’s quality tier. Severe Undersell Rates (recommendations two or more quartiles below the ground truth) averaged 7.9%, with the best-performing models achieving rates as low as 5.2%.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion: \u003c/strong\u003eStandard evaluation metrics for journal recommendation are insufficient because they treat all errors as equivalent. In this study, Quality Accuracy@1 substantially exceeded top@1 accuracy, indicating that a large proportion of apparent “errors” involved journals of equivalent quality. Quality-Aware metrics, combined with ranking quality measures, provide a more complete assessment of whether a system produces reliable recommendations. We propose that journal recommendation systems report Quality-Aware metrics alongside traditional accuracy to better characterize real-world utility.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMSC Codes: \u003c/strong\u003e68T10 (Pattern recognition); 62P10 (Applications of statistics to biology and medical sciences)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eJEL Codes: \u003c/strong\u003eO32 (Management of Technological Innovation and R\u0026amp;D)\u003c/p\u003e","manuscriptTitle":"Quality-Aware Evaluation for Journal Recommendation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-31 18:42:56","doi":"10.21203/rs.3.rs-9053132/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"7b804a98-5541-4c4a-a1f9-45531ea8c1ef","owner":[],"postedDate":"March 31st, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-31T18:42:57+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-31 18:42:56","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9053132","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9053132","identity":"rs-9053132","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}