H-SSST: Hierarchical-State Space Model with Spatial-Temporal Features

For very long sequences, transformer models encounter O(L2) memory and computation bottle necks. We suggest a hierarchical architecture called H-SSST (Hierarchical-State Space Model with Spatial-Temporal features), which effectively compresses and fuses features by utilizing dynamic gating, local encoding, and vector quantization (VQ). In this preprint, the viability of H-SSST and the mechanistic contributions of important modules are presented through structural validation and small-scale experiments. Additionally, we offer theoretical justifications for the stability and effectiveness of the architecture.