HN
Hiep Nguyen ↗
Ph.D. Student
Works on multi-modal time-series forecasting, frequency-aware fusion, and retrieval-augmented forecasting. Leads Parts 1–2 and Hands-on 1.
ACML 2026
Advances in
Multi-Modal Time Series
From Forecasting to Reasoning
A practical tutorial on learning from time series together with text and other modalities—from multimodal forecasting to language-model reasoning and foundation models.
Slides coming soon
Code & Colab coming soon
Recording coming soon
Photo: Dusk over City of Melbourne by Chris Phutully · CC BY 2.0
Date1 December 2026
LocationRMIT University · Melbourne
FormatHalf day · 150 minutes
Hands-on2 Colab-ready sessions
Abstract
Why multi-modal time series?
Time-series data consists of sequences of observations collected over time, such as sensor measurements, financial prices, and activity logs, and is fundamental to many applications where understanding temporal patterns is essential for prediction and decision-making. Increasingly, however, time series are accompanied by additional modalities such as text, images, event logs, and contextual information. These complementary signals can provide richer semantic and event-level context, but they also introduce new challenges, including temporal alignment, modality heterogeneity, missing or noisy observations, effective cross-modal fusion, scalability, and evaluation. This tutorial provides a structured introduction to recent advances in multi-modal time-series learning, spanning forecasting, reasoning, question answering, and foundation models. We begin by formalising the main problem settings, tasks, representative benchmarks, and emerging challenges. We then examine multi-modal forecasting with modern deep neural networks, focusing on how textual and numerical information can be integrated through different fusion strategies, including late fusion, auxiliary-variable alignment, and frequency-domain fusion.
Moving beyond forecasting, we explore how pretrained language and vision-language models can be adapted to jointly understand temporal, textual, and visual information, enabling natural-language question answering and more general reasoning over time series. We then discuss the growing role of foundation models, from numerical time-series models such as TimesFM and Chronos to multimodal foundation models that jointly represent time series and text for forecasting, reasoning, and question answering. The tutorial combines conceptual lectures with two guided, Colab-ready hands-on sessions. Participants will build a paired time-series-and-text forecasting pipeline and experiment with large pretrained models for forecasting and question answering. By connecting model architectures, fusion mechanisms, pretrained models, practical implementation, and open research challenges, the tutorial aims to provide participants with both a coherent overview of the field and practical foundations for developing future multi-modal time-series systems.
Tutorial outline
From forecasting to reasoning
10 min
Opening
Introduction and Overview
Time-series fundamentals, multimodal settings, recent trends, use cases, and the tutorial roadmap.
Hung Le
15 min
Part 1
Problem Definition and Emerging Challenges
Forecasting, reasoning and QA; temporal alignment; missing/noisy data; modality heterogeneity; scalability; Time-MMD and Time-MQA.
Hiep Nguyen
25 min
Part 2
Multi-Modal Forecasting with Deep Neural Networks
Sequence backbones and three fusion views: late fusion, auxiliary-variable alignment, and frequency-domain fusion.
Hiep Nguyen
15 min
Hands-on 1
Build a Multi-Modal Forecasting Pipeline
From paired time-series-and-text data to training and evaluation in a Colab notebook.
Hiep Nguyen
20 min
Break
Coffee break
—
20 min
Part 3
Multi-Modal Reasoning and Question Answering
Pretrained language and vision-language models for temporal reasoning and QA, plus current limitations.
Van Dai Do
20 min
Part 4
Scaling with Foundation Models
Numerical TSFMs and multimodal foundation models, including unified tokenisation and cross-attention.
Van Dai Do
10 min
Hands-on 2
Large Models for Forecasting and QA
Apply pretrained large models to simple forecasting and question-answering examples.
Van Dai Do
15 min
Closing
Wrap-Up and Q&A
Key lessons, open challenges, future research directions, and discussion.
Hung Le
About the presenters
Deakin Applied AI Initiative
All presenters are from A2I2, Deakin University, Australia.
HL
Hung Le ↗
ARC DECRA Fellow · Senior Lecturer
Works on deep learning, reinforcement learning, multi-modal AI, and multi-modal time-series forecasting. Leads the tutorial introduction and Q&A.
VD
Van Dai Do ↗
Associate Research Fellow
Works on foundation models, language-model adaptation, multi-modal forecasting, and temporal reasoning. Leads Parts 3–4 and Hands-on 2.
Assistant
YW
Tutorial materials
Benchmark datasets
Data used throughout the tutorial
- Time-MMDMulti-domain multimodal time-series analysis
- Time-MQATime-series multi-task question answering
Demo code · Hands-on 1
Multimodal forecasting
Build a paired time-series + text forecasting pipeline, train a model, and evaluate it.
Demo code · Hands-on 2
Foundation models for forecasting & QA
Apply pretrained models to time-series forecasting and natural-language question answering.
Slides
Full tutorial deck
The complete ACML 2026 slide deck will be linked here before the tutorial.
Code & reading list
Implementations and papers
Companion code, selected implementations, and the references used across the four tutorial parts.
Recording
Tutorial video
A recording link will be added here if one is released after ACML 2026.
Selected references
- Liu et al. Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis. NeurIPS, 2024.
- Nie et al. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR, 2023.
- Wu et al. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. ICLR, 2023.
- Li et al. Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrative. ICLR, 2026.
- Nguyen et al. Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting. AISTATS, 2026.
- Zhong et al. Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting. ICML, 2025.
- Kong et al. Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement. ACL, 2025.
- Merrill et al. Language Models Still Struggle to Zero-shot Reason about Time Series. Findings of EMNLP, 2024.
- Das et al. A Decoder-Only Foundation Model for Time-Series Forecasting. ICML, 2024.
- Ansari et al. Chronos: Learning the Language of Time Series. TMLR, 2024.
- Wang et al. ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data. AAAI, 2025.
- Langer et al. OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data. 2026.