LEMON: LanguagE MOdel for Negative Sampling of Knowledge Graph Embeddings

Knowledge Graph Embedding models have become an important area of machine learning. Those models provide a latent representation of entities and relations of a knowledge graph which can then be used in downstream machine learning tasks such as link prediction.The learning process of such models can be performed by contrasting positive and negative triples.While the triples of the underlying knowledge graph are considered positive, the generation of the negative samples has its own process. Therefore, the sampling procedures for obtaining the negative triples play a crucial role in the performance and effectiveness of Knowledge Graph Embedding models. Most of the existing techniques draw negative samples from a random distribution of entities of the underlying Knowledge Graph which often includes uninformative negative triples. Different works employ adversarial techniques or generative neural networks for negative sampling, which improve the performance of models with the cost of additional sophisticated mechanisms. In this paper, we propose an approach for generating informative negative samples by utilizing complementary knowledge about entities. Particularly, pre-trained language models are used to form neighborhood clusters by computing the distances between entities to obtain representations of symbolic entities via their complementary textual information. Our proposed approach achieved the biggest leaps in performance over the baseline models KBGAN, NSCaching, Uni-SANS with an absolute difference of +22.38, +6.45 and +7.16, respectively on WN18RR dataset.