AI-Driven Fraud Detection in FinTech: A Hybrid Deep Learning Framework for RealTime Transaction Monitoring

Abstract

The rapid expansion of financial technology (FinTech) and digital payment services has significantly increased transaction volumes while creating greater exposure to fraudulent activities. Conventional rule-based and machine-learning fraud-detection approaches often struggle with evolving fraud patterns, highly imbalanced transaction data, and the computational requirements of real-time monitoring. This study proposes a hybrid deep-learning framework for real-time financial transaction fraud detection by integrating automated feature extraction with sequential transaction-pattern learning. The framework incorporates a class-imbalance handling strategy to improve recognition of minority fraudulent transactions while reducing excessive false alarms. Incoming transactions are processed to estimate fraud probability and support immediate classification decisions. Performance is evaluated using Precision, Recall, F1-score, PrecisionRecall Area Under the Curve (PR-AUC), Receiver Operating Characteristic Area Under the Curve (ROC-AUC), False Positive Rate (FPR), and Matthews Correlation Coefficient (MCC). Real-time feasibility is further assessed through inference latency per transaction and transaction-processing throughput. Comparative analysis against conventional machine-learning and standalone deeplearning models is used to determine the effectiveness of the proposed hybrid architecture. The framework is intended to provide reliable fraud identification, improved minority-class detection, lower false-positive behavior, and computationally efficient transaction monitoring for scalable FinTech security applications.