Leveraging AI Agents to Accelerate Software Quality Assurance

A Practical Outlook from Real Project Experience

As delivery cycles accelerate, the pressure on quality assurance to keep pace without becoming a bottleneck, cost driver or governance risk is growing. Teams are expected to release software more quickly, work across increasingly distributed system landscapes and maintain confidence in its quality, even though requirements, interfaces and documentation are constantly changing.

Most organisations already rely on established tools such as Jira, Azure DevOps, TestRail, Xray and Zephyr, as well as automation frameworks such as Selenium, Playwright, Cypress and RestAssured. However, despite this mature tool landscape, many QA processes are still based on operating models designed for more stable environments. Traditional test automation is effective when systems change in a predictable manner. But it becomes more difficult to scale when specifications are scattered, requirements change mid-sprint, and APIs are being further developed whilst acceptance criteria are updated. This leads to constant manual reworking of test resources.

This is precisely where AI agents can make all the difference. They do not merely automate individual QA tasks; they transform quality engineering from a manual, coordination-based process into a more adaptable, networked and evidence-based model.

In this article, we demonstrate how our team developed an agent workflow and deployed AI agents in an ongoing client project to minimise manual effort, improve the consistency of test coverage and promote responsible AI testing. Importantly, this is not a proof of concept. This approach was developed, tested and refined as part of an extensive migration programme for a major airline’s loyalty platform in a live production environment.

From Traditional Test Automation to AI Agent-Powered Quality Engineering

Traditional test automation remains valuable, but its limitations become increasingly visible in fast-moving delivery environments. The more frequently requirements evolve, the more manual effort is needed to keep automated assets aligned with the current state of the system.

This creates a structural problem. Quality depends on humans being available at every step: reading specifications, writing tests, updating scripts, connecting information across tools, interpreting results and compiling reports. Each of these activities adds coordination overhead, introduces delay and increases the risk of inconsistency.

Over time, teams encounter a familiar pattern. Coverage lags behind change. Test suites become harder to maintain. Engineers end up spending more time maintaining automation than using it to identify defects. The issue is not simply that test automation is too slow. Rather, the operating model behind it no longer scales efficiently.

AI Agent-powered Quality Engineering changes this model. Agents can absorb much of the repetitive and preparatory work, such as reading specifications, generating test cases, identifying duplicates, mapping to acceptance criteria and presenting results. Human expertise remains essential, but its role changes. Review becomes a focused quality gate rather than a manual throughput bottleneck.

This change affects more than just the efficiency of testing processes. It is also transforming the underlying operational model, from test creation and maintenance to their integration into end-to-end quality engineering workflows. The following comparison illustrates the differences between traditional test automation and AI-agent-based quality engineering.

Challenges We Solved in a Live Project

The challenges we addressed were not theoretical. They arose during a live client engagement within a large airline's platform migration programme.

One of the main issues was the fragmented nature of the information spread across various tools. Specifications were spread across Jira tickets, Confluence pages, screenshots and other artefacts. At the same time, the volume of changes made it increasingly difficult to sustain manual test authoring. AI agents helped to bridge this gap by reading ticket descriptions, automatically generating BDD scenarios, and connecting information across documentation sources within a single, coordinated pipeline.

A second challenge was achieving consistent coverage. Depending on available bandwidth, test depth and completeness often varied under sprint pressure. By generating test cases directly from the broader specification base, the agents created a more consistent level of coverage throughout the delivery cycle.

The third challenge involved duplicate and redundant test assets. In dynamic delivery environments, similar or overlapping test cases can easily accumulate across tickets and teams. The agent pipeline identified and flagged duplicates before they reached the test management processes further down the line.

Another bottleneck was the expansion of scenario outlines. This was a previously manual and time-consuming task, which often limited the number of variants and edge cases that could be covered. However, automating this step enabled the agents to increase scenario depth while reducing manual overheads.

The fifth major challenge was authoring at scale. The large number of Jira tickets meant there was little time to write test cases manually. By generating BDD scenarios directly from ticket content, the agents reduced the effort required for authoring while improving throughput.

In addition, tooling was often disconnected. Jira, Confluence and Zephyr Scale frequently operated in isolation from each other, creating avoidable handoffs between systems and teams. The agents helped to bridge these gaps and establish a more integrated testing workflow.

How We Applied AI Agents in Practice

Traditional QA typically treats documentation as something people read and then translate into action. AI Agents change that model. Specifications, user stories, and acceptance criteria become a live knowledge base that agents can retrieve from, reason over, and act on directly. In our implementation, five capabilities formed the operational core.

1. AI-Based Test Generation
   Agents read specifications and create structured, Jira-ready test cases automatically, including step definitions that can be used in CI/CD execution flows.

2. RAG Over Project Documentation
   Confluence pages, Jira tickets, and related artefacts are chunked, embedded, and retrieved so agents work from the most current available sources rather than from static snapshots. This is critical in delivery environments where documentation changes frequently and outdated context leads to poor testing decisions.

3. Conversational Access to Test Results
   Instead of manually searching through spreadsheets or waiting for consolidated reports, testers can ask questions about outcomes in plain English. This makes result interpretation faster and more accessible across stakeholder groups.

4. Multi-Tool Orchestration
   The agent architecture connects Jira, Confluence, Zephyr Scale, and security evaluation tooling in one coordinated flow. This reduces manual handoffs and helps convert fragmented QA activities into a more integrated operating model.

5. Scheduled, Event-Driven, and On-Demand Execution
   Depending on the use case, custom agents can run against defined systems on demand, on schedule, or through CI/CD-triggered events. This gives teams greater flexibility in how they operationalise automated testing.

Each agent fulfilled a clearly defined role, and outputs were designed to feed directly into subsequent steps in the pipeline.

The Four-Layer Architecture

To make this operating model scalable, we structured the solution across four connected layers. Together, these layers take a project from raw documentation to verified test results with minimal manual intervention.

1. Knowledge layer: Documentation and ticket content provide the source context for agent reasoning.

2. Generation layer: Agents create structured test cases, scenarios and execution-ready outputs.

3. Execution layer: Generated outputs move into test execution and evaluation flows.

4. Reporting and governance layer: Results, evidence and controls remain visible, reviewable and traceable.

This layered structure ensures that documentation, test generation, execution and reporting are not isolated activities, but instead function as part of a connected quality engineering model.

How the Pipeline Worked

The operational flow followed five structured steps:

1. Step 01: Ingestion
   The chatbot specifications, formal acceptance criteria, architecture documentation and prior research artefacts were all incorporated into the Retrieval-Augmented Generation (RAG) layer. This ensured that all AI agents worked from the latest approved documentation, maintained traceability to requirements and reduced the risk of generating outputs based on outdated or incomplete information.

2. Step 02: Generation
   The Test Generation Agent analysed the chatbot specification documents and automatically produced structured, requirement-aligned test cases. These test cases were then transferred directly into Jira, which significantly reduced the effort required for manual test design while improving consistency, coverage and traceability between requirements and tests.

3. Step 03: Evaluation
   The Safety Evaluation Agent assessed the resilience of the chatbot by executing 124 prompt injection attack variants spanning seven distinct attack vectors. This systematic evaluation identified security weaknesses and measured the chatbot's resistance to adversarial inputs. It also provided evidence of the chatbot's robustness against prompt injection attacks.

4. Step 04: Review
   Every AI-generated test case was independently validated by a dedicated Review Agent, who cross-checked it against the formal acceptance criteria. This verification step ensured that the generated tests accurately reflected the intended system behaviour, identified missing or incorrect test scenarios, and improved the overall quality and reliability of the test suite.

5. Step 05: Reporting
   Test results, execution summaries, and identified defects were made accessible through a conversational Reporting Agent. Testers can now query findings using natural language instead of manually searching reports, which reduces reporting effort and accelerates access to insights. This makes test outcomes easier to interpret and share with stakeholders.

Data security, privacy and responsible AI testing

For enterprise organisations, particularly those in regulated sectors such as aviation, banking, insurance and the public sector, data governance and responsible AI practices are essential. They are core requirements for any scalable testing model.
This was reflected in the design of the pipeline from the outset.

The solution ran on the client's own infrastructure. No test data, prompts, or results left the client environment. All test data was also anonymised or synthetic before entering the pipeline, whether for test generation, retrieval tasks or query interfaces. Real user data was not used in the testing environment.

Prompt injection was treated as a primary risk rather than an ancillary consideration. Testing was carried out systematically across defined attack vectors with clear targets and traceable results. This is important because enterprise AI quality is not just about functional performance. It also requires assurance around resilience, misuse scenarios and governance expectations.

Outcome and Business Value

The value delivered by the pipeline can be viewed across three dimensions: quality, speed, and operational impact.

Quality

Test cases were generated directly from specifications and validated against formal acceptance criteria before execution. This established a more consistent connection between requirements and testing outputs. In addition, prompt injection performance could also be measured against a defined target threshold, making AI safety testing more structured and auditable.

Speed

Activities that would previously have taken days, particularly documentation-based test case generation and review, were completed within hours. Because agents worked from current sources, teams no longer needed manual refresh cycles to align on the latest documentation. Integration with Jira also eliminated a critical handover between test creation and downstream test management.

Operational and Business Impact

The broader impact extended beyond efficiency gains. Earlier defect identification reduced the effort required for downstream corrections. Reporting became more evidence-based and easier for stakeholders to use. QA specialists were able to spend more time on exploratory and analytical work rather than repetitive preparation tasks. Furthermore, governance stakeholders gained clearer, traceable indicators for discussing AI-related quality and risk.

What Comes Next

The next stage of development is already underway. Planned enhancements include automated regression testing triggered by Jira ticket updates, broader safety evaluation beyond prompt injection, and more formalised data governance artefacts covering classification, retention, and auditability. In addition, deeper CI/CD integration is expected to extend from pipeline handoff to full regression triggering on deployment events.

The direction is clear: AI-assisted QA is evolving into AI-native Quality Engineering. Organisations that build these capabilities early will be better positioned to improve delivery speed, increase coverage consistency, and strengthen confidence in software quality at scale.

If your teams are dealing with fragmented documentation, increasing manual testing efforts, or the need to test AI-enabled systems more responsibly, we would be happy to discuss what an AI Agent-enabled QA setup could work in your organisation.

Just get in touch with us!