One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

Time-Series Retrieval for Grounding Multimodal Language Models in Remaining Useful Life

arXiv:2608.192182026-08-21

Showing an AI similar past breakdown cases makes its machine-failure predictions better

This study looks at using multimodal AI models (MLLMs, which can read both images and text) to predict how much longer a piece of equipment, like a jet engine, will keep running before it fails, a metric called Remaining Useful Life (RUL). The core idea is to retrieve past degradation cases similar to the equipment being tested and show them to the AI as a side-by-side comparison chart. Tested on the FD001 subset of the C-MAPSS jet-engine benchmark, this retrieval-based approach produced lower prediction error and more stable results than showing the AI randomly chosen past cases.

What they did

  1. Sensor time-series segments are converted into vectors using an LSTM-based encoder, and the segments most similar to a test engine's recent sensor readings are found via vector search from historical run-to-failure data.
  2. The retrieved historical trajectories and the test engine's trajectory are combined into a single comparison chart image, inserted into a structured prompt, and sent to three Gemini-based multimodal models (Flash-Lite, Flash, Pro), which return both a numerical RUL estimate and a text explanation.
  3. Using the same prompt structure, the study compared 'random past-case selection' (MLLM+Random) against 'retrieval-based case selection' (MLLM+RAG) across 10 repeated experiments.
  4. Across all three models, retrieval-based selection produced lower error and less variability than random selection, with the gains being largest for the more capable model (Pro), suggesting stronger models make better use of the retrieved evidence.
  5. Even the best retrieval-augmented setup still fell short of specialized deep-learning models built specifically for this task, and the evaluation was limited to FD001, the simplest subset of C-MAPSS.
Figure 1: Preprocessing workflow applied to the FD001 partition of the C-MAPSS dataset.
Figure 1: Preprocessing workflow applied to the FD001 partition of the C-MAPSS dataset.
Table 1: Overview of the main pipeline modules according to their inputs, transformations, and outputs.
ModuleInputTransformationOutput
Segment encoderNormalized sliding-window segmentEncode with an LSTM autoencoder.Fixed-dimensional embedding.
Retrieval memory builderTraining segments and RUL labelsSample by RUL bin and index embeddings.Balanced vector database.
Query retrieverLast observed test segmentEncode query and perform k-nearest-neighbor search.Top-k historical references.
Trajectory comparison builderQuery trajectory and retrieved referencesAlign trajectories and generate a multisensor plot.Trajectory-comparison image.
Prompt composerComparison image and task instructionsInsert visual evidence into a structured prompt.Multimodal prompt for RUL estimation.
Figure 2: Overview of the time‑series RAG mechanism: (a) offline indexing of encoded training segments; (b) online retrieval using the encoded test segment.
Figure 2: Overview of the time‑series RAG mechanism: (a) offline indexing of encoded training segments; (b) online retrieval using the encoded test segment.
Table 2: Configuration of the learned LSTM encoder used to generate retrieval embeddings.
AspectConfiguration
Input window lengthL=30 cycles
Input variablesSelected sensor features
Encoder typeLSTM encoder
Number of recurrent layers1
Embedding layerLinear projection
Embedding dimensiond=64
Hidden dimension128
Decoder typeFeed-forward head
Training objectiveFinal-step reconstruction
Loss functionMean squared error
OptimizerAdamW
Training epochs200
Batch size128
Learning rate10−3
Weight decay10−5
Embedding normalizationL2 normalization
(b) Query retriever. The last segment of the test trajectory is encoded and used to retrieve the nearest indexed train segments.
(b) Query retriever. The last segment of the test trajectory is encoded and used to retrieve the nearest indexed train segments.
Table 3: FD001 test-set comparison between baseline methods and Gemini-based RAG variants.
BaselinesGemini RAG
MetricMeanRandom predictorLSTM (?, ?)Flash-LiteFlashPro
RMSE41.98 ± 0.0057.19 ± 2.4914.5443.12 ± 2.7232.85 ± 3.3022.69 ± 1.44
MAE36.07 ± 0.0047.53 ± 2.66–32.91 ± 2.1922.47 ± 1.9917.91 ± 1.24
R2-0.021 ± 0.000-0.897 ± 0.165–-0.080 ± 0.1350.370 ± 0.1250.701 ± 0.037
PHM’08 Score20302 ± 0181531 ± 74903322.44127441 ± 6661027712 ± 158301488 ± 323
Figure 3: Multimodal prompt generation and example comparison artifact used for MLLM inference.
Figure 3: Multimodal prompt generation and example comparison artifact used for MLLM inference.

Why it matters

In maintenance settings, having a text explanation alongside a numerical prediction matters for trust and decision-making, and this approach shows that MLLMs can provide both together. It also demonstrates that general-purpose language models can be improved on a specialized industrial task simply by retrieving relevant reference cases, rather than requiring full retraining, pointing to a practical direction for applying such models to other equipment-monitoring problems.

(b) Example of the trajectory-comparison image.
(b) Example of the trajectory-comparison image.

Terms in this paper

  • RUL (Remaining Useful Life) · An estimate of how much operating time is left before a machine or part fails, used for maintenance planning
  • MLLM (multimodal large language model) · An AI model that can process and reason over multiple input types, such as images and text, together
  • RAG (retrieval-augmented generation) · A technique where an AI first retrieves relevant reference material before generating its answer
  • C-MAPSS / FD001 · A widely used simulated jet turbofan engine degradation benchmark dataset; FD001 is its simplest subset
  • LSTM encoder · A neural network structure that compresses a time-varying sensor sequence into a single fixed-size numerical vector
Figure 4: Predictive performance over 10 repetitions for the three evaluated Gemini MLLMs under the two inference settings, MLLM+Random and MLLM+RAG.
Figure 4: Predictive performance over 10 repetitions for the three evaluated Gemini MLLMs under the two inference settings, MLLM+Random and MLLM+RAG.

Original abstract (English)

Large language models (LLMs) and agentic AI systems are increasingly being explored for domain-specific maintenance and prognostics tasks, raising the question of whether they can effectively support prognostics and health management (PHM). In this paper, we investigate remaining useful life (RUL) estimation with multimodal large language models (MLLMs) grounded through time-series retrieval. We propose a framework in which historically similar degradation segments are retrieved from the training set and, together with the test trajectory, transformed into a visual comparison artifact that is processed by the MLLM through a structured multimodal prompt. The approach is evaluated on the FD001 partition of the C-MAPSS benchmark under repeated experiments comparing retrieval-based inference against a non-retrieval baseline based on random reference selection. The results show that time-series retrieval consistently improves MLLM-based RUL prediction across the evaluated models, yielding lower error and more stable performance. At the same time, the magnitude of the benefit depends on model capacity, indicating that retrieval is most effective when the underlying MLLM is able to exploit the retrieved evidence. Overall, the study shows that time-series RAG is a promising mechanism for improving multimodal prognostic reasoning, while also highlighting the current limitations of MLLM-based RUL estimation in practical PHM settings.

Authors · Valeriu Dimidov, Rapha\"el Frank

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Valeriu Dimidov et al., arXiv:2608.19218, arxiv-nonexclusive