AXIS-Bench: A Pilot Benchmark For Interactivity and Accessibility Conformance in Front-End Code Generation

Abstract

Large language models are increasingly used to translate visual designs and natural-language specifications into front-end code, yet existing design-to-code benchmarks score generated pages primarily on visual and structural fidelity to a reference screenshot, and existing accessibility-focused resources evaluate static prompt-code pairs rather than the behavioral, keyboard-operable interactions that most often break accessibility in practice. We present AXIS-Bench, a pilot benchmark that targets the intersection of interactivity and accessibility. Its first release comprises twelve interactive UI component specifications, including disclosure widgets, modal dialogs, custom selection controls, tabbed navigation, sortable data tables, a carousel, a live-validated form, and a keyboard-operable drag-and-drop list, each paired with an explicit WCAG 2.2 and WAI-ARIA Authoring Practices Guide (APG) interface contract. We evaluate three generation conditions: a naive non-semantic baseline, a single-pass ("zero-shot") large language model, and the same model prompted with the relevant APG pattern text ("guideline-grounded"). Evaluation uses two independent, fully automated harnesses: the axe-core accessibility engine, and a purpose-built Playwright test suite that locates every element exclusively through its accessibility-tree role and accessible name, so that a component which is not exposed correctly to assistive technology cannot pass its interactivity test by accident. On this pilot suite, the naive baseline passes 0 of 12 interactivity and ARIA-conformance tasks, the zero-shot condition passes 6 of 12 (McNemar exact p = 0.031 against naive), and the guideline-grounded condition passes all 12 (p = 0.031 against zero-shot; p < 0.001 against naive). We additionally find that axe-core's static rule set, run on isolated components, is dominated by page-scaffold checks that do not distinguish the two LLM conditions, motivating the paired functional harness. We release the task specifications, all 36 generated solutions, and both evaluation harnesses to support replication and extension.