Region-Based Attention With Hierarchical Bilstm For Image Captioning

Abstract

Image captioning requires expressing complex visual scenes in coherent, contextually relevant natural language, but traditional encoder-decoder frameworks rely on a single global feature and a unidirectional decoder, limiting fine-grained understanding. This study proposes a framework integrating region-based feature extraction with hierarchical bidirectional decoding to generate semantically rich captions, aimed at helping visually impaired individuals perceive their surroundings without relying solely on touch. A Region-based Convolutional Neural Network (RCNN) extracts object-level visual features through region proposals, refined using a visual attention mechanism that focuses on the most relevant regions during generation. Refined features are decoded using a two-level Hierarchical Bidirectional LSTM (H-Bi-LSTM), where the lower level models word-level linguistic dependencies and the higher level captures sentence-level semantics. The key novelty lies in this joint region-based attention and hierarchical bidirectional decoding, which explicitly separates word-level syntax from sentence-level semantics. Generated captions are further converted into audible speech in English, Hindi, and Marathi through a text-to-speech module, enabling multilingual, accessibility-oriented delivery. Evaluated on MSCOCO and Flickr30K, the model achieves BLEU@4 scores of 39.3 and 38.2, with corresponding improvements in METEOR and CIDEr over existing methods.