Intelligent Anomaly Detection for Large Network Traffic with Optimized Deep Clustering (ODC) Algorithm

Annie Gilda Roselin, Priyadarsi Nanda, Surya Nepal, Xiangjian He

Research output: Journal PublicationArticlepeer-review

20 Citations (Scopus)


The availability of an enormous amount of unlabeled datasets drives the anomaly detection research towards unsupervised machine learning algorithms. Deep clustering algorithms for anomaly detection gain significant research attention in this era. We propose an intelligent anomaly detection for extensive network traffic analysis with an Optimized Deep Clustering (ODC) algorithm. Firstly, ODC does the optimization of the deep AutoEncoder algorithm by tuning the hyperparameters. Thereby we can achieve a reduced reconstruction error rate from the deep AutoEncoder. Secondly, ODC feeds the optimized deep AutoEncoder's latent view to the BIRCH clustering algorithm to detect the known and unknown malicious network traffic without human intervention. Unlike other deep clustering algorithms, ODC does not require to specify the number of clusters needed to analyze the network traffic dataset. We experiment ODC algorithm with the CoAP off-path dataset obtained from our testbed and the MNIST dataset to compare our algorithm's accuracy with state-of-art clustering algorithms. The evaluation results show ODC deep clustering method outperforms the existing deep clustering methods for anomaly detection.

Original languageEnglish
Article number9383226
Pages (from-to)47243-47251
Number of pages9
JournalIEEE Access
Publication statusPublished - 2021
Externally publishedYes


  • AutoEncoders
  • BIRCH clustering
  • Deep learning
  • anomaly detection
  • latent space view
  • regularization

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering


Dive into the research topics of 'Intelligent Anomaly Detection for Large Network Traffic with Optimized Deep Clustering (ODC) Algorithm'. Together they form a unique fingerprint.

Cite this