A Comparative Analysis of Sentence Transformer Models for Automated Journal Recommendation Using PubMed Metadata

We present an automated journal recommendation pipeline designed to evaluate the performance of five Sentence Transformer models— all-mpnet-base-v2 (Mpnet), all-MiniLM-L6-v2 (Minilm-l6), all-MiniLM-L12-v2 (Minilm-l12), multi-qa-distilbert-cos-v1 (Multi-qa-distilbert), and all-distilroberta-v1 (Roberta)—in identifying journals that align with a manuscript's thematic scope. The pipeline dynamically tailored its search space by extracting domain-relevant keywords from a manuscript’s title and abstract using KeyBERT, which were then used to query PubMed and retrieve a custom corpus of potentially related articles. Both the test manuscript and the retrieved articles were encoded into high-dimensional embeddings, enabling the computation of cosine similarity to rank articles and their publishing journals based on thematic alignment. Evaluations on 50 test articles revealed distinct strengths and trade-offs among the models. Mpnet consistently demonstrated the highest performance, with mean similarity scores of 0.71 ± 0.04 and strong alignment with relevant journals. Minilm-l12 and minilm-l6 displayed comparable precision but lower computational requirements, while multi-qa-distilbert and roberta provided broader recommendations, suitable for interdisciplinary research. The low Shannon entropy values (~3.24) across all models reflected concentrated and focused recommendations. These results highlight the overall flexibility of these models in journal selection, providing interpretable and data-driven insights which may be tailored to diverse research contexts.