Probing BERT on Syntactic Representation Paradigms: Dependency vs Constituency Trees

On the quest for interpreting deep Neural Language Models (NLMs), linguistically annotated probing sets are essential tools for investigating models' linguistic abilities. Such a line of research is currently dominated by dependency-based syntactic annotation formalisms, and particularly by Universal Dependencies treebanks. In this work, we test whether different Syntactic Representation Paradigms (SRP) have an impact on the performance of NLMs. To this aim, we set up a multilingual study where we compare the scores obtained by BERT on a set of probing tasks performed on 10 treebanks annotated according to two paradigms, i.e. the dependency and the constituency one, which diverges on fundamental annotation principles. Results suggest that BERT performance is minimally but consistently affected by SRPs and that such impact mostly concerns specific linguistic phenomena.