Hierarchical Software Requirement Classification Using Transformer-Based Language Models

Abstract

The automated classification of software requirements into Functional Requirements (FRs) and Non-Functional Requirements (NFRs) has become a significant research direction in Requirements Engineering due to the growing size and complexity of modern software specifications. Manual requirement classification is often inconsistent, subjective and laborious. This emphasizes the need for scalable automated approaches that improve the reliability of early-stage system analysis. In this paper, we present an in-depth analysis of the PROMISE_EXP dataset, a systematically collected corpus comprising 969 requirement statements, including 444 FRs and 525 NFRs, spanning twelve granular NFR categories, such as Security, Usability, Performance, Availability, Maintainability, Scalability, and Portability. Besides a detailed statistical characterization of the dataset and a report of the class imbalance characterizing real-world requirement distributions, this work also explores a two-level hierarchical classification strategy. In the first level, requirements are categorized into a binary model (FR vs. NFR). In the second level, all NFRs are further classified into twelve multi-class labels. We implement and benchmark a comprehensive suite of models to evaluate the performance of automated classification at both levels. These include traditional machine learning algorithms (SVM, Random Forest, and Logistic Regression), deep learning architectures (CNNs, LSTMs) and transformer-based language models, specifically BERT and its domain-adapted variants. Experimental results show that transformer-based models consistently outperform ML and DL counterparts in both binary and multi-class settings particularly in dealing with semantically complex NFR subcategories. The results demonstrate that domain-aware language representations significantly improve accuracy, robustness and generalization, especially for low-frequency NFR labels. This study provides a robust two-level classification framework and a comprehensive performance evaluation of ML, DL and BERT based classifiers. It further provides practical insights for advancing automated requirements engineering. The results support the development of intelligent, data-driven tools for software engineering automation, enable a deeper understanding of the variability in requirements semantics, and promote reproducibility.