ISTQB
Certified Tester
AI Testing

Seminar contents

1. introduction to AI

• Definition of AI and AI effect
• Weak AI, general AI and super AI
• AI-based and conventional systems
• AI techniques
• AI development frameworks
• Hardware for AI-based systems
• AI-as-a-service (AlaaS)
• Pre-trained models
• Standards, regulations and AI

2. quality characteristics for AI-based systems

• Flexibility and adaptability
• Autonomy
• Evolution
• Bias
• Ethics
• Side effects and reward hacking
• Transparency, interpretability and explainability
• Functional safety and AI

3. machine learning (ML) - overview

• Types of ML
• ML workflow
• Selecting a type of ML
• Factors that play a role in the selection of ML algorithms
• Overfitting and underfitting

4. machine learning (ML) data

• Data preparation as part of the ML workflow
• Training, validation and test datasets in the ML workflow
• Problems with data set quality
• Data quality and its impact on the ML model
• Data labeling for supervised learning

5. functional performance metrics of ML

• Confusion matrix
• Additional functional performance metrics of ML for classification, regression and clustering
• Limitations of functional 5. 5. 5. performance metrics of ML
• Selection of functional performance metrics of ML
• Benchmark suites for ML

6. ML - Neural networks and testing

• Neural networks
• Coverage measures for neural networks

7. testing of AI-based systems at a glance

• Specification of AI-based systems
• Test levels for AI-based systems
• Test data for testing AI-based systems
• Testing for automation biases in AI-based systems
• Documenting an AI component
• Testing for concept drift
• Selecting a test approach for an ML system

8. testing AI-specific quality characteristics

• Challenges in testing self-learning systems
• Testing autonomous AI-based systems
• Testing for algorithmic, sampling and inappropriate biases
• Challenges in testing probabilistic and non-deterministic AI-based systems
• Challenges in testing complex AI-based systems
• Testing the transparency, interpretability and explainability of AI-based systems
• Test oracles for AI-based systems
• Test objectives and acceptance criteria

9. methods and procedures for testing AI-based systems

• Adversarial attacks and data contamination
• Pairwise testing
• Comparative testing
• A/B testing
• Metamorphic testing (MT)
• Experience-based testing of AI-based systems
• Selection of test procedures for AI-based systems

10. test environments for AI-based systems

• Test environments for AI-based systems
• Virtual test environments for testing AI-based systems

11. use of AI for testing

• AI techniques for testing
• Use of AI to analyze reported errors
• Use of AI for test case generation
• Use of AI for the optimization of regression test suites
• Use of AI for fault prediction
• Use of AI for testing user interfaces

12 Practical exercises