Multimodal Intrusion Detection In Restricted Zones Using Thermal-RGB Fusion And RFID Authentication

Abstract

Traditional single-sensor monitoring systems are not effective in low-visibility situations, like darkness, fog, and smoke, and often produce false alarms, thus restricting their useful application in critical security situations. Previous studies have investigated visual fusion based on thermal cues or RGB cues separately but none of the existing solutions can be deployed at the edge and combine dual-stream visual fusion and hardware identity authentication for real-time intrusion classification. Fusing thermal imaging (MLX90640: 24x32) with RGB video (IP camera: 640x480) streams, this paper proposes a multimodal intrusion detection system that uses two independently operating YOLO detection models, whose results are checked for consistency by spatial cross-checking through the Intersection-over-Union (IoU) method and trigger MFRC522 RFID-based personnel authentication. No server side processing is performed and the whole pipeline runs on a Raspberry Pi 5 which makes it suitable for resource constrained industrial, defence and warehouse applications. The proposed system is tested with 1000 pairs of synchronized thermal-RGB images under daytime, nighttime, and low-light conditions for its classification accuracy, precision, recall, and F1-score results are 98.23%, 0.9910, 0.9888, and 0.9899 respectively. The IoU-based fusion mechanism can significantly reduce false positives due to environmental heat sources and non-human thermal signatures from single-modality baselines. This system is light in weight, has multiple means of redundancy and is designed with RFID-based authentication and verification, which is a feasible and deployable solution for autonomous perimeter security.