Log-based anomaly detection for distributed systems: State of the art, industry experience, and open issues

Xinjie Wei, Jie Wang, Chang ai Sun, Dave Towey, Shoufeng Zhang, Wanqing Zuo, Yiming Yu, Ruoyi Ruan, Guyang Song

Research output: Journal PublicationArticlepeer-review

Abstract

Distributed systems have been widely used in many safety-critical areas. Any abnormalities (e.g., service interruption or service quality degradation) could lead to application crashes or decrease user satisfaction. These things may cause serious economic losses. Among the various quality assurance approaches for distributed systems, log-based anomaly detection (LAD) has become a popular research topic. Its popularity relates to system logs being able to record and reveal important run-time information. This paper presents a general LAD framework for distributed systems. Log grouping and feature-pattern mining are two crucial LAD components that impact on the anomaly-detection effectiveness. We also present a systematic survey of techniques in these two directions; propose classification frameworks for log grouping and feature patterns; and summarize four log-grouping techniques and five feature patterns (which refer to invariant relationships among logs that can be used for anomaly detection). To evaluate their applicability, we report on the findings when applying existing techniques to Ray, a popular industrial distributed system. Based on these findings, several open issues are identified, which provide potential guidance for future research and development.

Original languageEnglish
Article numbere2650
JournalJournal of software: Evolution and Process
Volume36
Issue number8
DOIs
Publication statusAccepted/In press - 2024

Keywords

  • distributed systems
  • industry experience
  • log-based anomaly detection
  • quality assurance

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'Log-based anomaly detection for distributed systems: State of the art, industry experience, and open issues'. Together they form a unique fingerprint.

Cite this