Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization

• Published in: arXiv.org 2024-12
• Main Authors: Masserano, Luca, Ansari, Abdul Fatir, Boran Han, Zhang, Xiyuan, Faloutsos, Christos, Mahoney, Michael W, Andrew Gordon Wilson, Park, Youngsuk, Rangapuram, Syama, Maddix, Danielle C, Wang, Yuyang
• Format: Article
• Language: English
• Subjects: Autoregressive models; Datasets; Decomposition; Deep learning; Forecasting; Time series
• ISSN: 2331-8422

How to best develop foundational models for time series forecasting remains an important open question. Tokenization is a crucial consideration in this effort: what is an effective discrete vocabulary for a real-valued sequential input? To address this question, we develop WaveToken, a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies. Our method first scales and decomposes the input time series, then thresholds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to forecast coefficients for the forecast horizon. By decomposing coarse and fine structures in the inputs, wavelets provide an eloquent and compact language for time series forecasting that simplifies learning. Empirical results on a comprehensive benchmark, including 42 datasets for both in-domain and zero-shot settings, show that WaveToken: i) provides better accuracy than recently proposed foundation models for forecasting while using a much smaller vocabulary (1024 tokens), and performs on par or better than modern deep learning models trained specifically on each dataset; and ii) exhibits superior generalization capabilities, achieving the best average rank across all datasets for three complementary metrics. In addition, we show that our method can easily capture complex temporal patterns of practical relevance that are challenging for other recent pre-trained models, including trends, sparse spikes, and non-stationary time series with varying frequencies evolving over time.