| Abstract: |
| Repeated patterns occur in biological sequences, neural activity, and in tokenized representations of data across many domains. Modeling such repetition is central to sequence analysis, but remains challenging for modern learning methods. While transformer architectures can capture long-range dependencies, they do not explicitly account for recurrence or cyclic structure, which are key features in many sequences.
We introduce a topological framework for learning on sequences with repetition using zigzag persistent homology. As a sliding window moves along a sequence, this construction detects and tracks repeated patterns through the appearance and persistence of loops. Building on this idea, we develop ZZFormer, a transformer architecture that incorporates zigzag topological structure directly into token embeddings, enabling end-to-end learning on variable-length sequences.
By encoding repetition explicitly, ZZFormer provides a way to incorporate topological structure into sequence models. This framework applies broadly to problems involving repeated or cyclic patterns. We demonstrate its effectiveness on tasks involving repetitive DNA, including classification of transposable element superfamilies and orders. |
|