ISTQB CT-AI Tester chapter 3

Practice flashcards covering machine learning forms (Supervised, Unsupervised, Reinforcement) and the standard ML workflow based on the CT-AI syllabus.

Supervised Learning

A type of learning where the algorithm creates the ML model from labeled data during the training phase to infer the relationship between input pairs (e.g., image and label) and output labels.

Classification

A category of supervised learning where the problem requires an input to be assigned to one of several pre-defined classes, such as face recognition or object detection.

Regression

A category of supervised learning where the ML model predicts a numeric output, such as predicting a person's age or the future prices of stocks.

Unsupervised Learning

A type of learning where the algorithm creates the ML model from unlabeled data to infer patterns and assign inputs to different classes based on similarities.

Clustering

A category of unsupervised learning that identifies similarities in input data points to group them based on common characteristics, such as categorizing customers for marketing.

Association

A category of unsupervised learning used to identify interesting relationships or dependencies among data attributes, like utilized in product recommendation systems basing on shopping behavior

Reinforcement Learning

An approach where an "intelligent agent" learns from experience by interacting with an environment, receiving rewards for correct decisions and penalties for incorrect ones.

ML Workflow - Understand the Objectives

The activity where the purpose of the ML model is agreed with stakeholders to ensure alignment with business priorities and define acceptance criteria.

ML Workflow - Select a Framework

The choice of an AI development framework based on objectives, acceptance criteria, and business priorities.

ML Workflow - Select & Build the Algorithm

The activity of choosing an ML algorithm based on factors like objectives and available data, which may be manually coded or retrieved from a library.

ML Workflow - Data Preparation

A phase in the ML workflow comprising data acquisition, data pre-processing, and feature engineering.

ML Workflow - Exploratory data analysis (EDA)

An activity performed alongside data preparation to understand the data that will be used by the algorithm and model.

ML Workflow - Characteristic of Data used in Test & Model Generation

Data must be based on objectives and representative of operational data used by model

ML workflow -Train the model

A phase in the ML workflow uses training data to train the model

Epoch

An activity in Training the model phase, referring to each individual iteration of training on the training dataset

2 types of parameter in training model

model hyperparameters, algorithm hyperparameters

What are model hyperparameters?

parameters that govern the behavior of the machine learning algorithm and are set before the model training begins.

What are algorithm hyperparameters?

to parameters that control the learning process of the algorithm itself, including settings like learning rate, batch size, and the number of epochs.

ML Workflow - Evaluate the Model

A phase in the machine learning workflow where the trained model's performance is assessed using validation data to ensure it meets defined acceptance criteria.

ML Workflow - Tune the Model

The process of optimizing hyperparameters and making adjustments to the ML model to improve its performance based on evaluation results.

ML Workflow - Test the Model

A final assessment phase where the model is tested with independent data to before deployment.

What is Non-functional tests in phase Test the model?

including testing scenario: time to train model, time and resource using to provide prediction

ML Workflow - Deploy the Model

The process of integrating the trained and tested ML model into an operational environment.

ML Workflow - Using the Model

Once deployed, the model is part of a larger AI-based system and can perform scheduled batch predictions or run on request in real time.

ML Workflow - Monitoring and Tuning the Model

While in operation, the model's performance may drift, requiring regular evaluation against acceptance criteria to identify and manage drift.

Model Drift Management

To address model drift, it may be necessary to update settings or retrain with new data, potentially leading to the creation of a new model.

A/B Testing in Model Comparison

The new model created with updated training data can be compared against the existing model using A/B testing.

Iterative ML Workflow

The ML workflow is applied iteratively, requiring return to steps like training and data preparation after evaluation.

Integration with Non-ML Parts

ML models typically cannot be deployed in isolation; they must be integrated with non-ML parts of the overall system prior to deployment.

Data Pipeline in ML Systems

In applications such as vision, a data pipeline cleans and modifies data before it is submitted to the ML model.

Problem: Insufficient data for selected ML approach

Solution: Ensure availability of sufficient training and test data for the chosen ML method.

Problem: Unlabeled data for supervised learning

Solution: Require properly labeled data for supervised learning.

Problem: output labels are available

Solution: select supervised learning as the approach.

Problem: Output is discrete and categorical

Solution: Apply classification techniques for the model.

Problem: Output is numeric and continuous

Solution: Implement regression techniques for the analysis.

Problem: No output data available

Solution: Consider unsupervised learning methods.

Problem: Need to group similar data points

Solution: Utilize clustering techniques to achieve this.

Problem: Finding relationships in data

Solution: Employ association methods to identify co-occurring data items.

Problem: Context involves environment interaction

Solution: Choose reinforcement learning as the suitable approach.

Problem: involves notion of multiple states and require decision at each state

Solution: Use reinforcement learning for decision-making in this context.

Factors affected ML algorithm selection

• expected functionality

• expected quality characteristic

• type of data availability

• amount of data availability

• features of data (thuộc tính)

• expected number of classes clustering

• previous experience with any ML algorithm

• trial and error

In ML algorithm selection, what is included when mentioning expected quality characteristic

• accuracy

• constraints on available memory

• training & retraining speed

• prediction speed

• transparency, interpretability, explainability

Overfitting

Overfitting occurs when a model fits too closely to training data, including noise, resulting in poor predictions on new data.

Causes of Overfitting

It is caused by fitting all training data points, including outliers, and by insufficient training data.

Underfitting

Underfitting happens when a model is too simplistic, failing to capture patterns in the training data, leading to inaccurate predictions.

Causes of Underfitting

It can result from missing important features in the training data or using a model that doesn't fit the data correctly.