From automation to AI: lessons learned from old testing trends

There have always been promising trends in software testing since the 1990s. From the automation wave of the 1990s and the focus on process models to agility and DevOps - each development brought new hopes and expectations.

But how have these trends actually developed? Which initial assumptions came true, and where did reality deviate from the forecasts? And what does all of this mean for the current upheaval in the field of AI-supported testing?

There is a recurring pattern that is reminiscent of the so-called Jevons paradox: increases in efficiency in one area often do not lead to a reduction in overall effort, but to its shifting and sometimes even expansion - a phenomenon that is particularly evident in software testing, even against the backdrop of ever shorter development cycles and a steadily increasing amount of software produced.

The 1990s: The rise of test automation - expectations vs. reality

In the past, we often heard: "Test automation will make manual testers superfluous."

Automation did indeed bring huge efficiency gains, particularly in regression testing and repetitive tasks. At the same time, however, there was a new need for testers with automation skills for script creation and maintenance.

Manual testing remained essential for technical, experience-based, exploratory and usability aspects. As we know today, manual and automated testing have been combined more frequently, leading to productivity gains, but not necessarily to a reduction in the number of testers, but to an expansion of their tasks.

This is a classic example of the Jevons paradox: the efficiency gained was not translated into a reduction of the "testing" resource, but into an expansion of its application and complexity, as more and more in-depth testing could now be carried out.

If you have just stopped reading: The focus should not only be on IT-side tests in the development context. Complex business IT systems of software users with high costs on the specialist department side during integration and acceptance must also be taken into account.

The 2000s: The focus on process improvement and certifications - promises and results

Test automation became more established, as did the canonization of testing and the establishment of the industry. It was now assumed: "Process models such as TMMi and certifications such as ISTQB will significantly improve software quality."

It is true that process models and certifications established important standards and a common vocabulary in the testing sector and thus contributed to the professionalization of testing. In particular, test methods, test processes and test procedures have improved. Specialist literature was published.

However, it became apparent that strategies, concepts and models did not guarantee the hoped-for leaps in quality, as they fizzled out without a corresponding quality culture in the company.

Here, too, it can be seen that the efforts to achieve efficiency through standardization did not reduce the overall need for high-quality test work, but rather fueled the search for more effective ways - such as Agile soon afterwards - which underlines the persistence of the need for testing in the sense of the Jevons principle.

Possible solutions were offered by a new, supposed savior that was gaining in popularity. It went by the name of "Agile". In testing, too, the focus increasingly shifted to agile methods and greater integration of testing into the development process.

The 2010s: Agile and DevOps take hold - the vision and implementation

Claims like this can still be found on some websites: "Agile and DevOps make dedicated testers superfluous, as development teams take over testing in an integrated manner."

It is true that agile promoted collaboration and shared responsibility for quality across the entire team. Developers took on more testing tasks, testers were integrated earlier and more strongly into the development process in many places. The product came to the fore.

The "Whole Team Ownership of Quality" approach became established and also changed project work significantly. Agile could not abolish the silo: The expertise of test specialists remained crucial - not only as agile test coaches, quality assurance enablers and experts for complex test types, but in the "traditional" roles and in completely new ones, such as the Software Developer in Test (SDET).

The time "saved" by a broader distribution of simpler testing tasks was thus invested in more specialized and in-depth testing activities, in line with the Jevons paradox, which tended to increase the overall value and scope of testing. Fields of activity such as API testing, load and performance testing or test automation gained in depth and differentiation. Very few highly qualified people can master more than two of these areas.

The 2020s: AI-supported testing: a real turning point?

Now we are witnessing an exciting new trend: AI-powered testing and the widespread use of LLMs. The urgency for such advances is heightened by the unstoppable trend towards ever shorter development cycles and an exponentially growing amount of software features and products. Why might this trend have a more lasting impact than its predecessors?

• Intelligent automation: AI enables automation that goes beyond rigid rules. It can recognize patterns, identify anomalies, generate test cases and even provide suggestions for troubleshooting. This has the potential to significantly reduce repetitive and analysis-intensive tasks.
• Focus on relevant quality: AI can help steer testing efforts more intelligently and focus on high-risk areas. This could lead to higher quality with potentially fewer but more targeted testing activities - or more likely, deeper coverage of critical areas.
• New roles and competencies: The implementation and use of AI in the testing environment will require new roles (e.g. AI test engineers) and change existing ones. In view of the constantly growing volume of software, it is not only a question of making existing testers more productive, but often also of being able to cope with the increased test load at all. AI can provide the necessary increase in productivity here, while human testers shift their focus to strategy development, result interpretation and the validation of AI models.
• Democratization of testing: AI-supported tools could make the creation and execution of tests more accessible to people with less specific testing knowledge, potentially increasing test coverage.

The key question: Is AI fundamentally changing the testing landscape?

Although AI-supported tests offer enormous potential, it is unlikely that they will replace human testers - rather, they are needed to cope with the sheer volume of tests required in modern software projects and at the same time increase the efficiency of each individual tester.

The need for testing activities is growing faster than testers can be trained; AI is a tool to close this gap and improve quality at the same time.

Some observations on AI-assisted testing:

• The effectiveness of AI testing depends heavily on the quality and representativeness of the training data.
• The decisions made by AI models are not always transparent and comprehensible.
• AI test systems also require continuous maintenance and adaptation to changing applications, interfaces and system landscapes.
• Creativity, critical thinking, empathy for the user and an understanding of the business context remain human strengths that AI will not be able to fully imitate in the foreseeable future. Exploratory testing and usability assessments will continue to require human expertise.
• Etc.

Past trends have shown us: The human factor will always remain. With the right tools, and AI-supported testing tools are nothing else, people become better, faster, more accurate, but certainly not replaceable.

It's about a synergy where both more testing capacity (potentially also more testers) and higher individual productivity through AI are sought in order to meet the requirements.

Outlook

The history of software testing is characterized by trends whose actual impact often deviated from original expectations. AI-powered testing is undoubtedly a significant advancement and offers exciting opportunities to increase efficiency and improve quality.

We are facing another evolution, not a revolution. The testing professions will continue to evolve, with new skills and areas of focus becoming more important. In view of shorter development cycles and an exploding amount of software, this is absolutely necessary.

The Jevons paradox teaches us that efficiency gains promised by AI often lead to us using more of a resource - in the case of testing, this means that we will use the capacity gained to test more comprehensively, more intelligently and more deeply than was previously possible.

So we need both: more testing capacity, potentially through more testers, and a significant increase in the productivity of individuals through tools such as AI. However, people remain the central element in the testing process - especially when interacting with new technologies.

Companies should realistically assess the potential of AI-supported testing and make targeted investments in the further training of their teams in order to make the most of the opportunities - without sacrificing proven human expertise.

We would be happy to discuss our position on AI-supported testing with you - get in touch!