Category-Aware Deep Multimodal Network with Hybrid Attention Mechanism for Fashion Compatibility Modeling

To address the problem of alignment representation and compatibility prediction in the multimodal feature fusion of fashion items, this paper designs a fashion compatibility prediction model based on a deep attention mechanism and category perception. The model comprises an attention mechanism fusion module and a category-aware module. The former is a fusion model based on the transformer cross-modal attention mechanism, which aligns visual features and text features for feature dimensionality and semantics. To utilize the category information of single products, the potential space of category embedding is constructed for different category pairings based on coarse-grained category partitioning pairs to build different mapping networks, and the compatibility scores between single products are calculated. The experimental simulation results show that compared with existing methods, the accuracy of clothing collocation and the accuracy of the fill-in-the-blank task on the publicly available Polyvore and Polyvore-D datasets are improved.