Symbiosis of antiferromagnetism and ferrimagnetism in adjacent honeycomb layers

Abstract Ferro/ferri- and antiferromagnetic orders are typically exclusive in nature, thus, their co-existence in atomic-scale proximity is expected only in heterostructures. Breaking this paradigm and broadening the range of unconventional magnetic states, we report here on the observation of a new, atomic-scale hybrid spin state. This ordering is stabilized in three-dimensional crystals of the polar antiferromagnet Co$_2$Mo$_3$O$_8$ by magnetic fields applied perpendicular to the \emph{Co} honeycomb layers and possesses a spontaneous in-plane ferromagnetic moment. Our microscopic spin model, capturing the observed field dependence of the longitudinal and transverse magnetization as well as the magnetoelectric/elastic properties, reveals that this novel spin state is composed of an alternating stacking of antiferromagnetic and ferrimagnetic honeycomb layers. The strong intra-layer and the weak inter-layer exchange couplings together with competing anisotropies at octahedral and tetrahedral \emph{Co} sites are identified as the key ingredients to stabilize antiferromagnetic and ferrimagnetic layers in such close proximity. We show that the proper balance of magnetic interactions can extend the stability range of this hybrid phase down to zero magnetic field. The possibility to realize a layer-by-layer stacking of such distinct spin orders via suitable combinations of microscopic interactions opens a new dimension toward the nanoscale engineering of magnetic states.