Orientation and Introduction

Pattern Recognition and Machine Learning

Week 1: Orientation, Introduction and Linear Regression

The entire lecture in one sentence

Pattern Recognition and Machine Learning is about learning useful patterns from data so that a system can make predictions or decisions and generalise to new, unseen situations.

What you should know after this lecture

• Explain the relationship between Artificial Intelligence, Machine Learning and Pattern Recognition.

• Distinguish supervised, unsupervised and reinforcement learning.

• Distinguish classification from regression.

• Describe the blocks in a typical pattern-recognition system.

• Explain the purpose and mathematical form of simple linear regression.

• Calculate and interpret MAE, MSE and RMSE at a basic level.

1. Unit Facts and Assessment

What the unit expects

This is a core Artificial Intelligence unit. You are assessed on both what you produce and what you understand.

You should be able to:

• Understand the concepts.

• Implement machine-learning methods.

• Interpret the results produced by a model.

• Clearly explain your decisions and work.

• Understand and explain any code you submit.

PRML compared with Soft Computing

Soft Computing is more applied and focuses on practical methods for reasoning, learning and optimisation.

Pattern Recognition and Machine Learning focuses more on the fundamental knowledge of machine learning and pattern recognition, including how algorithms work and how to design and use them.

Assessment overview

The project will be research-oriented. The final output will include:

• A report written in research-paper style.

• The relevant source code and implementation.

• A presentation explaining the project.

Technical environment

Python will be used throughout the unit.

The recommended environment is Jupyter Notebook.

Generative AI and academic integrity

Generative AI tools should support your learning, not replace it.

Appropriate uses

• Brainstorming ideas.

• Planning your solution.

• Understanding concepts.

• Completing minor support tasks.

• Using referenced support code.

Uses that require care

• Debugging assistance.

• Code suggestions.

• Improving your wording.

• Generating examples.

Always check the individual assessment rules.

Not acceptable

• Allowing AI to complete most of the work.

• Submitting code you cannot explain.

• Submitting shared work as your own.

• Hiding the use of external assistance.

• Copying work without referencing it.

Open-source code and code created with Generative AI support should be referenced.

You may be asked questions about your code, so you must understand and be able to explain it.

Sharing assignments is plagiarism.

2. Patterns, AI, ML and Pattern Recognition

What is a pattern?

A pattern is a meaningful regularity, structure or relationship found in data.

A mathematical expression, equation, formula or model can capture a pattern. Once a pattern has been identified, it can be used to predict future or unknown information.

Examples of simple patterns include:

• Number sequences.

• Fibonacci sequences.

• Logical patterns.

• Relationships between variables.

The Fibonacci sequence is defined as:

F₀ = 0

F₁ = 1

Fₙ = Fₙ₋₁ + Fₙ₋₂

This produces the sequence:

0, 1, 1, 2, 3, 5, 8, 13, 21, …

Example: predicting electricity demand

Suppose electricity demand rises as temperature rises.

A model can learn the relationship between temperature and electricity demand from previous examples. It can then estimate the expected demand at a new temperature, such as 35°C.

This is a regression problem because the predicted output, electricity demand, is a continuous numerical value.

What is pattern recognition?

Pattern recognition is the process of discovering characteristics in data that allow a system to recognise, classify, group or interpret observations.

A useful pattern does not have to be visually obvious to a human. A machine-learning model may detect relationships that humans cannot easily see.

Pattern-recognition applications

Although these applications use different types of data, they share the same fundamental goal:

Learn useful patterns from observations and use them to make predictions or decisions.

Challenges in pattern recognition

The same object or class can appear in many different ways.

For example, face-recognition systems must handle changes in:

• Lighting.

• Pose.

• Facial expression.

• Ageing.

• Glasses or other accessories.

Handwritten digits also vary greatly between people. Different examples of the same digit can look very different, but humans can still identify them as belonging to the same class.

A pattern-recognition system must learn which differences are important and which differences can be ignored.

Artificial Intelligence, Machine Learning and Pattern Recognition

Artificial Intelligence

Artificial Intelligence is the broad field of creating systems capable of tasks associated with human intelligence.

These tasks include:

• Perception.

• Reasoning.

• Planning.

• Language understanding.

• Decision-making.

Examples include autonomous robots, virtual assistants and game-playing systems.

Machine Learning

Machine Learning is a major approach within Artificial Intelligence.

Machine-learning systems learn patterns and relationships from data to make predictions or decisions instead of relying only on explicitly programmed rules.

Examples include:

• Supervised learning.

• Unsupervised learning.

• Reinforcement learning.

Deep Learning

Deep Learning is a subset of Machine Learning that uses deep neural networks.

Pattern Recognition

Pattern Recognition studies methods for identifying meaningful patterns or regularities in data.

These patterns can be used to:

• Recognise observations.

• Classify observations.

• Group similar observations.

• Detect anomalies.

• Interpret information.

Examples include face recognition, handwritten digit recognition, speech recognition, clustering and anomaly detection.

Relationship between the fields

• Artificial Intelligence is the broad ambition of building intelligent systems.

• Machine Learning provides a framework for learning intelligent behaviour from data.

• Deep Learning is a subset of Machine Learning based on deep neural networks.

• Pattern Recognition is a key task involving the discovery and use of meaningful patterns.

3. Different Learning Problems

Supervised learning

Supervised learning uses labelled datasets.

Each training example contains:

• An input.

• The correct output or label.

The model learns the relationship between the inputs and correct outputs. It then uses that relationship to predict the output for new inputs.

Supervised learning includes classification and regression.

Classification

Classification predicts a categorical label.

The output belongs to one of two or more classes.

Classification answers the question:

Which category or class does this input belong to?

Examples include:

• Spam or not spam.

• Recognising a person’s identity.

• Recognising handwritten digits from 0 to 9.

• Identifying whether a medical image is normal or abnormal.

Regression

Regression predicts a real or continuous numerical value.

Regression answers the question:

What numerical value should be predicted?

Examples include:

• House-price prediction.

• Salary prediction.

• Electricity-demand prediction.

• Predicting risk based on a numerical health measurement.

Classification versus regression

Class versus label

A class is a collection of similar objects or observations.

A label is the name or category assigned to an individual example.

For example:

• “Spam” is a class.

• An individual email may be given the label “spam”.

• The digit “5” is a class.

• A particular handwritten image may be labelled “5”.

Unsupervised learning

Unsupervised learning uses unlabelled data.

The model is not given correct outputs. Instead, it must discover patterns, structures or insights without explicit guidance.

Clustering

Clustering is an unsupervised-learning technique that divides unlabelled data into groups called clusters.

The groups are created based on similarities or differences between the observations.

Applications include:

• Customer grouping.

• Anomaly detection.

• Fraud detection.

• Discovering patterns in large datasets.

Reinforcement learning

Reinforcement learning involves an intelligent agent interacting with a dynamic environment.

The agent:

1. Observes the environment.

2. Selects an action.

3. Receives a reward or penalty.

4. Changes its strategy based on the feedback.

The goal is to learn actions that maximise the cumulative reward over time.

Examples include:

• Robot control.

• AlphaGo.

• AlphaZero.

• Game-playing agents.

• Autonomous decision-making.

Quick decision rules

• If correct answers are supplied during training, it is supervised learning.

• If the model searches for groups or structures without correct answers, it is unsupervised learning.

• If an agent performs actions and receives rewards, it is reinforcement learning.

• If supervised learning predicts a category, it is classification.

• If supervised learning predicts a continuous number, it is regression.

4. Typical Pattern-Recognition System

A pattern-recognition system converts raw observations into useful predictions or decisions through a sequence of functional blocks.

Basic pipeline

Data acquisition → Preprocessing → Feature extraction and selection → Model learning or prediction → Evaluation and deployment

1. Data acquisition

Data acquisition is the process of capturing raw input through sensors and converting it into numerical values or vectors.

Examples of input sources include:

• Images.

• Audio.

• Video.

• LiDAR.

• Medical scanners.

• Brain-signal sensors.

• Environmental sensors.

2. Data preprocessing

Preprocessing prepares the raw data before it is given to the model.

It may:

• Remove noise.

• Correct lighting problems.

• Correct sampling errors.

• Resize inputs.

• Standardise values.

• Crop important regions.

• Separate an object from the background.

• Isolate an area of interest.

Examples include cropping a face from an image or separating a handwritten digit from its background.

3. Feature extraction and selection

Feature extraction converts raw input into meaningful descriptions called features.

Examples include:

• Edges.

• Shapes.

• Image moments.

• Spectral coefficients.

• PCA embeddings.

• Neural-network feature maps.

Good features should remain useful even when irrelevant variations occur.

Feature selection is the optional process of keeping only the most important features.

4. Model learning

The model is trained using the processed training data.

During training, the system learns parameters that relate the input features to the expected outputs.

Once the model has been trained, it can perform:

• Classification.

• Regression.

• Other prediction tasks.

5. Model evaluation

Model evaluation measures how well the model performs.

The model should be evaluated on unseen data to determine whether it can generalise beyond its training examples.

Possible evaluation measures include:

• Classification accuracy.

• Regression error.

• Precision.

• Recall.

• Other task-specific metrics.

6. Deployment

Deployment places the model into a real system where its predictions cause an action or decision.

Examples include:

• Warning that a pedestrian has been detected.

• Rejecting an unclear face image.

• Flagging an email as spam.

• Producing a medical diagnostic suggestion.

Training versus testing

The training path uses processed examples and features to learn a model.

The testing path applies the same preprocessing and feature-extraction steps to new, unseen inputs. The trained model then produces predictions for those inputs.

Testing evaluates generalisation.

A good model should not simply memorise its training examples. It should perform well on data it has not previously seen.

Questions to ask when designing a pattern-recognition application

For any application, ask:

1. What are the input data?

2. Which sensors collect the data?

3. What output should the system produce?

4. What preprocessing is required?

5. Which features might be helpful?

6. What challenges or unwanted variations exist?

7. How will the model be evaluated?

8. What should happen after a prediction is produced?

These questions can be applied to:

• Face recognition.

• Handwriting recognition.

• Licence-plate recognition.

• Biometric recognition.

• Autonomous systems.

• Medical diagnosis.

• Land-cover classification.

• Bioinformatics.

5. Linear Regression

The regression task

Regression learns a relationship between input variables and a continuous output.

Given observed input-output pairs, regression approximates the underlying relationship so that it can predict a reasonable output for a new input.

The training examples can be written as:

(x⁽¹⁾, y⁽¹⁾), (x⁽²⁾, y⁽²⁾), …, (x⁽ᴹ⁾, y⁽ᴹ⁾)

The model assumes:

y⁽ⁱ⁾ ≈ f₍w₎(x⁽ⁱ⁾)

This means that the real output for the i-th example is approximately equal to the output predicted by the model.

Two important decisions

1. Choose a model family

We must decide what type of function might represent the relationship.

Examples include:

• Linear model.

• Polynomial model.

• Neural network.

2. Find the model parameters

The model’s parameters must be adjusted so that its predictions fit the observed data well.

Example: predicting house prices

Suppose we have information about houses in a city.

Possible inputs include:

• House size.

• Number of bedrooms.

• Distance from the central business district.

The output is the house price.

For a new house that is 150 square metres, has three bedrooms and is five kilometres from the CBD, the model predicts a continuous numerical price.

Therefore, this is a regression problem.

Simple linear regression

Simple linear regression assumes a straight-line relationship between one input and the output.

The model is:

f(x) = wx + b

It may also be written as:

ŷ = wx + b

Meaning of the symbols

Weight or slope

The weight, w, controls the slope of the line.

It describes how much the prediction changes when x increases by one unit.

Bias or intercept

The bias, b, is the predicted output when x = 0.

It shifts the entire line upwards or downwards.

Example: diabetes-risk prediction

A simple linear-regression model could predict the risk of type-2 diabetes using Body Mass Index as the input.

The goal is to find values of w and b that allow the line to fit the observed BMI and diabetes-risk data as well as possible.

This creates two questions:

1. How good is the current model?

2. How do we learn the best parameter values?

To answer the first question, we measure the model’s error.

To answer the second question, we find parameter values that minimise the error.

6. Measuring Regression Error

Prediction error

For each training example, we compare the prediction with the true output.

The prediction for example i is:

ŷᵢ

The true output is:

yᵢ

The difference is:

ŷᵢ − yᵢ

An error metric combines the individual errors into one overall score.

In this lecture, a smaller error represents a better-fitting model.

Mean Absolute Error

Mean Absolute Error calculates the average absolute difference between the predictions and true values.

MAE = (1/N) Σ |ŷᵢ − yᵢ|

Absolute values ensure that positive and negative errors do not cancel each other.

Interpretation

• Easy to understand.

• Measures the average size of the error.

• Does not square the errors.

• Is less dominated by very large errors than MSE.

Mean Squared Error

Mean Squared Error calculates the average of the squared prediction errors.

MSE = (1/N) Σ (ŷᵢ − yᵢ)²

Interpretation

• All errors become positive after being squared.

• Large errors receive a stronger penalty.

• Commonly used as the learning objective for linear regression.

• A smaller MSE indicates a better fit to the data.

Root Mean Squared Error

Root Mean Squared Error is the square root of Mean Squared Error.

RMSE = √[(1/N) Σ (ŷᵢ − yᵢ)²]

Interpretation

• RMSE still penalises large errors.

• It is expressed in the same unit as the predicted output.

• This can make it easier to interpret than MSE.

Comparison of error metrics

Worked example

Suppose the true values are:

3 and 7

The predictions are:

4 and 5

The individual errors are:

4 − 3 = 1

5 − 7 = −2

MAE

MAE = (|1| + |−2|) ÷ 2

MAE = (1 + 2) ÷ 2

MAE = 1.5

MSE

MSE = (1² + (−2)²) ÷ 2

MSE = (1 + 4) ÷ 2

MSE = 2.5

RMSE

RMSE = √2.5

RMSE ≈ 1.58

Learning objective

Training a linear-regression model means finding the parameter values that minimise the selected error measure.

In this lecture, the main objective is to find the weight and bias that produce the minimum MSE.

The term “arg min” means:

The argument, input or parameter value that produces the smallest value of a function.

It does not refer to the smallest error value itself. It refers to the parameter value that causes that smallest error.

Basic model-training process

1. Begin with values for the model parameters.

2. Use the current parameters to generate predictions.

3. Compare the predictions with the true outputs.

4. Calculate an error, such as MSE.

5. Adjust the parameters to reduce the error.

6. Repeat the process until an acceptable minimum is reached.

7. Quiz-Ready Recap

Self-Test Questions

1. What is the main goal shared by the Pattern Recognition and Machine Learning applications discussed in the lecture?

2. How are Artificial Intelligence, Machine Learning, Deep Learning and Pattern Recognition related?

3. Why is spam filtering normally considered a classification problem?

4. Why is electricity-demand prediction normally considered a regression problem?

5. What is the difference between supervised and unsupervised learning?

6. How is reinforcement learning different from supervised learning?

7. What is the difference between a class and a label?

8. What does data preprocessing achieve? Give two examples.

9. What is the purpose of feature extraction?

10. Why must model testing use unseen data?

11. What does generalisation mean?

12. In f(x) = wx + b, what do w and b represent?

13. What is the difference between MAE and MSE?

14. Why does MSE penalise large errors more strongly than MAE?

15. Why might RMSE be easier to interpret than MSE?

16. What does “arg min” mean when training a model?

17. What is the overall goal when training a linear-regression model?

Week 1 Summary

This lecture introduced:

• The unit structure and assessment procedure.

• Patterns and their use in prediction.

• Pattern Recognition and Machine Learning.

• The relationship between AI, ML, Deep Learning and Pattern Recognition.

• Supervised, unsupervised and reinforcement learning.

• Classification and regression.

• The typical pattern-recognition pipeline.

• Simple linear regression.

• MAE, MSE and RMSE.

• The idea of learning parameters by minimising error.

Next week

The next lecture will go deeper into:

• How linear-regression models learn.

• Modelling non-linear relationships.

• Classification.