Automated Test Case Generation For Ai-Intensive Systems: an Nlp-Based Comparative Evaluation of Large Language Models

Abstract

Modern software systems commonly integrate artificial intelligence (AI) components like trained models, data-preprocessing pipelines, inference wrappers, and orchestration logic. Their behavior is non-deterministic and depends on the input data, so the use of classical automated test-generation techniques, which rely on oracle hypotheses and structural-coverage heuristics, is now problematic at best. Large Language Models (LLMs) offer an effective solution to automate test-case generation. The fact that LLMs can reason with natural-language goals, rather than only a program's structure, makes them well suited to this task. Even so, it is not clear that AI models and prompting methods can produce compilable, comprehensive, and semantically meaningful tests for programs that rely strongly on AI. This paper presents a system-based comparative evaluation that makes use of natural language processing (NLP) of the effectiveness of five leading large language models, GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, DeepSeek V3.2, and Llama 4 Maverick, on the generation of JUnit tests. Each model is tested under three different prompting methods: zero-shot, few-shot, and chain-of-thought. The evaluation of generated test includes methods taken from the Methods2Test corpus and various aspects like compilation success and structural adequacy using JaCoCo line and branch coverage, fault-detection capability through PIT mutation analysis, and semantic and structural similarity with developer-written reference tests using the BLEU-4 score. In addition, EvoSuite a well-established search-based testing tool is used for comparison as a structural method. The experimental evaluations on non-parametric statistical testing and Cliff's delta as measures of effect sizes to distinguish between statistically significant differences that are practically relevant. Experimental results show that the GPT-5.5 model under chain-of-thought prompting achieves the strongest overall trade-off, attaining 73.1 percent line coverage, 67.4 percent branch coverage, a 70.2 percent mutation score and a BLEU-4 of 54.3. Measured against the EvoSuite baseline (78.4 percent line, 74.6 percent branch), the best LLM configuration falls short of structural coverage by 5.3 percentage points, while compilation success ranges from 63.1 percent to 85.7 percent across configurations and accounts for a substantial share of the residual gap. Prompting strategy helps every model in the same order, namely chain-of-thought, then few-shot, then zero-shot, but the size of the gain scales inversely with model strength, and every approach degrades as cyclomatic complexity rises, with the LLM to EvoSuite coverage gap widest on high-complexity focal methods even though the models' relative fault-detection advantage grows there. This work provides an actionable tip on how to implement a test generation workflow using LLM for testing AI-heavy systems as part of quality assurance practices.