Securing Enterprise AI Operations Assistants: A Pilot Red-Team Evaluation of Guardrails for LLM-Based Runbooks, Ticket Automation, and Production Support

Abstract

Enterprise "AI operations assistants" (LLM agents wired into ticketing systems, runbook knowledge bases, observability tools, and production remediation actions) are already being deployed to cut incident mean time to diagnose, following the same retrieval-augmented, tool-using agent pattern documented in recent industry practice. This convenience creates a concrete attack surface, since any content the agent reads while doing its job, such as a customer-submitted ticket, a runbook document, or a diagnostic log line, is a channel through which an adversary who never touches the chat interface can attempt to steer the agent. This pattern has already been demonstrated against general-purpose tool-using LLM agents in benchmarks such as AgentDojo, InjecAgent, and ToolEmu. We present GUARD-Ops, a pilot red-team evaluation of a deterministic, code-level guardrail architecture for this setting. We define a threat model of seven attack classes spanning direct instruction injection, three channels of indirect (tool-mediated) injection, excessive agentic initiative, data exfiltration, and tool-argument injection, and we author fifteen scenarios, twelve attacks and three benign controls, against a fully simulated enterprise operations environment; no real ticketing system, cloud account, or production host is touched at any point. We evaluate three conditions against the same underlying agent policy and the same scenario content: an unguarded agent, a prompted-only agent whose system prompt adds natural-language cautions against embedded instructions and unapproved destructive actions, and GUARD-Ops, which wraps the identical agent behavior with four deterministic layers, namely provenance quarantine, risk-tiered human-approval gating, schema and inventory allow-list validation, and output redaction with audit logging. On this pilot suite, the unguarded condition's attack succeeds on 12 of 12 scenarios. Prompting alone reduces this to 3 of 12 (McNemar exact p = 0.0039) but fails specifically on the two tool-argument-injection scenarios that require structural rather than semantic judgment. GUARD-Ops reduces attack success to 0 of 12 (p = 0.0005 against unguarded) with zero false blocks on the three benign controls. We additionally find that code-level guardrails and prompting address different failure modes: GUARD-Ops eliminates attack success without improving the agent's underlying diagnostic correctness, while prompting alone improves diagnostic correctness (task completion rises from 50% to 83%) yet leaves a residual, structurally predictable attack surface, which argues for defense in depth rather than either mechanism alone. We release the threat model, scenario suite, guardrail implementation, and evaluation harness to support replication and extension.