Zero-Shot Anomaly Detection via Self-Supervised Vision Transformer and Hybrid Density Modeling

Abstract

Pedestrian walkway surveillance requires robust anomaly detection for intelligent video analysis systems and public safety. Traditional supervised methods require labelled data for anomalous events, a costly, impractical expectation because anomalies are rare, unpredictable, and difficult to capture. This study proposes a zero-shot anomaly detection framework based on self-supervised Vision Transformers (ViT) that learns from normal data only. The insight is that an accurate representation of the normal is good enough to extend to anomalies, i.e., it is generalizable without training on the anomaly. The framework merges rotation prediction and contrastive learning objectives, yielding rich and generalizable representations of normal activity. For scoring anomalies, a hybrid detection strategy is used where Mahalanobis distance captures global distribution-based anomalies, and Local Outlier Factor (LOF) captures local density-based anomalies using a weighted fusion mechanism that incorporates these normalized scores into a weighted sum improves reliability. The attention maps from the ViT demonstrate interpretability in the localization of anomalies, revealing where unusual activity is happening in video frames. The effectiveness of this framework is demonstrated through extensive experiments on a synthetic pedestrian dataset and benchmark surveillance datasets, namely UCSDPed2 and ShanghaiTech, achieving AUROC scores of 0.9385, 0.968 and 0.742, respectively. The proposed framework outperforms those of supervised baselines, reconstruction-based methods, and zero-shot frameworks. Additionally, because only normal data is used for training the framework, it reduces annotation burden and makes anomaly detection feasible for real-world deployment in urban surveillance scenarios with better interpretability and efficiency.