Log-based anomaly detection for evolving software: An incremental deep-learning approach

Context: DevOps — an integrated approach that combines software development and operations — has been widely applied in various long-running software. A key issue with these systems is that any run-time anomalies must be observed and handled in a timely manner. Various log-based anomaly detection (LAD) techniques have been proposed. Using features extracted from logs, LAD techniques examine whether or not the continuously produced logs match historically normal ones. However, the continuous development of such systems inevitably leads to ever-changing logs, which, unfortunately, are often neglected by LAD techniques — they may generate a very large number of incorrect alert messages (false positives in anomaly detection). Objective: In this paper, we propose an incremental deep-learning approach called IDLLog to precisely detect run-time anomalies through adaptive and efficient mechanisms that incorporate newly emerging logs and historical ones without incurring significant overheads. Method: IDLLog first trains an initial anomaly-detection model (e.g., a Transformer model) using historical logs to capture features of normal logs. IDLLog then uses an incremental learning mechanism to process newly emerging logs through a partial fine-tuning of the initial model — this is triggered when the number of changed log templates in the newly emerging logs exceeds a predefined proportion of the historical ones. Furthermore, historical logs may become unavailable for fine-tuning due to data retention policies or storage constraints. Accordingly, IDLLog encompasses two scenarios: When historical logs are available, a herding-sampling strategy selects a representative subset that has a similar feature distribution, and then combines them with newly-emerging logs for fine-tuning; otherwise, a knowledge-distillation strategy transfers the features from the initial model to the updated model during the fine-tuning. Result: We evaluated IDLLog using both a public dataset (HDFS) and an industrial dataset from the Ray system. Experimental results show that IDLLog outperformed all the state-of-the-art techniques. Conclusion: We conclude that IDLLog provides a promising anomaly-detection technique for evolving software in the context of DevOps.