AMP: Multi-task Transfer Learning via Leveraging Attention Mechanism on Task Embeddings

Abstract The attention mechanism has been successfully used in a sequence consisted of a series of word embeddings to improve the representation of the sequence. Inspired by this, we leverage the attention mechanism on a set of tasks to implement a multi-task transfer learning method called AMP. First, we encode a task into a prompt as task representation called task embedding. Second, we learn an attention component on all task embeddings to generate a combined prompt for each taskwhich is an attention-weighted sum of task embeddings. Each combined prompt incorporates the knowledge of all other tasks. The word embedding is a vector, but the task embedding is a 2D matrix. The attention mechanism can be exploited on a set of vectors rather than on a set of matrices. The prior methods employ pooling or flattened method to transform the matrix to the vector for computing the attentions between matrices. We propose a method called DAM which can compute attentions between matrices directly without transforming. DAM method can more exactly compute the attentions between matrices. Wide experiments demonstrate that AMP outperforms prompt-tuning method and prior prompt transfer methods.