A Hybrid Vision–Language Framework For Unified Image–Video Captioning And Semantic Visual Question Answering

Abstract

While image captioning, video captioning, and Visual Question Answering (VQA) are often solved by separate architectures, the advances in the field of Vision-Language Learning (VL) have significantly improved the understanding of multimedia content, but these systems are often not coherent in terms of the integration of the semantic content. Recent advances in Vision-Language Learning (VL) have greatly advanced the understanding of multimedia content, however, image captioning, video captioning, and Visual Question Answering (VQA) are often solved by separate architectures, resulting in semantic incoherence and computational redundancy. In this paper, a Hybrid Vision–Language Framework (HVLF) is proposed which integrates these tasks in one end-to-end architecture. The framework uses VGG16, ResNet152 and InceptionV3 to generate complementary visual representation, and adaptive multimodal fusion and spatial–temporal attention to fuse the results. First, a hybrid Bi-LSTM–GRU decoder that produces contextually meaningful captions is used, which provide an intermediate semantic representation for transformer-based VQA reasoning. The framework was tested on the MS COCO, Flickr8k, Flickr30k, MSVD, MSR-VTT, VQA v2, GQA and TextVQA benchmarks. Empirical results show that the captions generated by the model achieve BLEU-4, METEOR, ROUGE-L, CIDEr and SPICE scores of 46, 34, 63, 155 and 27, respectively. The accuracy, precision, recall and F1 scores of VQA were 0.87, 0.87, 0.86, 0.86, respectively, which are superior to the evaluated vision–language baselines. The contribution of adaptive fusion, attention, recurrent decoding, and transformer reasoning were verified in ablation experiments. Furthermore, HVLF needs around 300 million parameters, 15 h of training time, 0.208 s per sample for inference, and 16 GB of GPUs for training. The number of parameters, the training time, and the inference time are all relatively low, while the memory consumption of the GPUs during training is relatively high. The results show that HVLF is a successful comprehensive solution for caption generation of multimedia data and semantic visual understanding.