CSCI218 - Foundations of AI

What is Artificial?

This is something synetic, man-made and often as a copy of something natural

What is Intelligence?

The ability to learn, understand, and make judgements or have opinions that are based on reason.

What subjects laid the foundation of AI?

• Philosophy - the mind works like a machine

• Mathematics - teaches AI how to think correctly

• Neuroscience - inspires AI through neurons, learning and adaptation

• Economics - teaches AI to choose the option with the best expected outcome

• Control Theory - feedback makes AI better

• Psychology - humans are viewed as info-processing systems like AI

• Linguistics - helps AI understand language, grammar and meaning (comm tool)

• Computer Engineering - built better hardawre to run AI

What is Artificial Intellgience?

• It makes computers think in the full and literal sense

• This concept creates computers that can perceive, reason, plan and act rationally.

What can AI do?

It can perceive, analyse, predict, and make decisions to solve real-world problems e.g.

• Healthcare: detect diseases from x-rays, predict patient outcomes, assist with diagnosis

• Recognition: face rec, identify people wearing masks

• Transportation: self-driving cars, traffic monitoring

• Automation: chatbots, smart factories

• Decision-making: financial trading, business recs

• Games: beat human experts

• Public safety: monitor crowds, detect social distancing

What content can AI create?

• Text (articles, essays, stories)

• Speech (voice narration)

• News and reports

• Images

• Artwork

• Photos

• Videos

• Virtual characters and avatars

• Music (and other creative content)

What is AI Ethics?

The study of building and using AI in a fair, responsible, honest and safe way.

Core principles:

• Fairness

• Responsibility

• Integrity

• Honesty

• Moral values

It matters in:

• Banking

• Medical diagnosis

• Self-driving cars

• Law enforcement and justice

• Recruitment

• Military

What is FAIR?

• Fairness

• Accountability (Responsibility)

• Integrity

• Respect & Honesty

What is the ethical dilemma of self-driving

• A self-driving car may face situations where every option could cause harm e.g. running over a baby or an elderly person.

• The issue would be how the AI should decide who to protect.

• What is the impact? Who is responsible for the decision? How can it be improved?

What is the Center for Artificial Intelligence (CAI)?

• A research centre that develops AI technologies and helps society benefit from AI

• Mission: Conduct AI research, develop new AI techniques and share AI knowledge with indistry, communities, and society.

• Vision: become a world-leading AI research centre and to work with different stakeholders to solve AI challenges

• Supports UNSDGs: Good Health & Well-being, Affordable & Clean Energy and Industry, Innovation & Infrastructure

What is the Center for AI Research focus?

• Computer vision

• Machine learning

• Multimedia Signal Analysis

• Natural Language Processing

• Multi-agent Systems

• Distributed computing

• Probabilistic model checking

• Big data and applications

What are the different dimensions of AI?

• Human vs. rational

• Thought vs. behaviour

What are the four approaches of AI?

• Acting humanly: Turing test approach

• Thinking humanly: Cognitive modelling approach

• Thinking rationally: “Laws of thought“ approach

• Acting rationally: Rational agent approach

What is the Turing test approach?

• This is for the version of AI to act humanly

Capabilities needed:

• Natural language processing

• Automated reasoning

• Machine learning

• Speech recognition

• Robotics

• Knowledge representation

• Computer vision

What is the Cognitive Modelling approach?

• This is for the version of AI to think humanly

• Human thought can be learnt through introspection (our own thoughts), psychological experiments (observing a person in action) and brain imaging (watching the brain in action)

• Cognitive science uses AI to model thinking and psychology to test those models and theories, helping us how to understand how the human mind works.

What is the “Laws of thought“ approach?

• This is for the version of AI to think rationally

• Uses formal logic to reach correct conclusions e.g. all men are moral → Socrates is a man → Socrates is mortal

• Probability is used when info is uncertain

Logical operators:

• ∧ = AND

• ∨ = OR

• → = Implies

• ↔ = If and only if

• ¬ = NOT

What is the Rational-agent approach?

• This is for the version of AI to act rationally.

• AI focuses on building agents that do the right thing

• The “right thing“ is determined by the objective (goal) given to the agent

• Chooses the best action to achieve its given objective

• Combines logical oundations (definite plans) and probability theory & machine learning (decision-making under uncertainty)

• Standard model: perfect rationality vs. limited (bounded) rationality

What are the benefits of AI?

• Free humanity from work

• Increase goods

• Accelerate research

Whst are the risks of misuse of AI?

• Weapons

• Surveillance

• Biasness

• Employment

• Safety

• Security

What is the future of AI looking like?

• As AI becomes more capable, they will take more roles

• Hence there will be the rise of importance of governance and, eventually, regulation.

What is an AI agent?

• It perceives its environment through sensors and acts upon it throught actuators

• The agent function maps percept sequences to actions

• It is generated by an agent program running on a machine

What is an AI agent’s environment?

Everything around the AI agent that it can sense and perceive, interact with or be affected by through the agent’s actions.

What is consequentialism in AI?

This evaluates an AI agent’s behaviour by its consequences (results).

How does an AI agent affect its environment?

Perception → Action → Environment → Change in the environment’s state

• The key question is whether the resulting environment states are desirable

What is a performance measure for an AI agentI?

• This captures the desirability of an agent’s behaviour and evaluates sequences of environment states.

• The designer or user of the agent deines this

• Goal when designing: design it according to what is actually desired in the environment, not how the agent should behave

What four things does rationality (good behaviour) of an AI agent depend on?

1. Performance measure

2. Agent’s prior knowledge of the environment

3. Actions the agent can perform

4. Agent’s percept sequence to date

What is the definition of a rational agent?

• For each percept sequence, a rational agent selects the actions expected to maximise its performance measure, based on its percepts and built-in knowledge.

• Rational choices made by them dependds only on the percept sequence to date.

• They also learn as much as possible from what it perceives.

• They are autonomous so they should learn enough to compensate for partial or incorrect prior knowledge

What is the difference between omniscience and rationality in AI?

The former knows the actual outcome of actions whereas the latter chooses the action with the best expected performance.

Is rationality the same as perfection in AI?

No as the former maximises expected performance whereas the latter maximises actual performance.

What is a task environment?

• The problem to which a rational agent is the solution.

• This should be specified as fully as possible when designing an agent

What does PEAS stand for in rational agents?

• Performance measure

• Environment

• Actuators

• Sensors

What are actuators in rational agents?

The component that allows an agent to take actions in its environments.

What are sensors in rational agents?

The components that allow an agent to perceive (observe) its environment.

What are the properties of task environments?

• Fully observable vs. partially observable

• Single-agent vs. multiagent

• Deterministic vs. nondeterministic

• Episodic (each decision is independent) vs. sequential (current decisions affect future decisions)

• Static (environment does not change while the agent is deciding) vs. dynamic (environment can change during decision-making)

• Discrete (finite, separate states or actions) vs. continuous (states or actions vary continuously)

• Known (the agent knows how its actions affect the environment) vs. unknown (the effects of actions are not fully known)

The hardest are the latter ones.

What is an agent made of?

Agent Architecture + Agent Program

What is agent architecture?

A computing device with physical sensors and actuators

What is agent program

The brains of the agent, it maps percepts to actions.

What is the job of AI when designing an agent?

Design an agent program tht implements the agent function which is the mapping from percepts to actions.

What is the input and output of an agent program.

• The input is the current percept from the sensors

• The output is actions sent to actuators

What is the key challenge for AI in terms of rationality?

Produce rational behaviour from a small program rather than from a vast lookup table which can become too large and impractical.

What are the four basic kinds of agent programs?

1. Simple reflex agents: react to current percept

2. Model-based relex agents: remembers internal state

3. Goal-bases agents: plans to achieve goal

4. Utility-based agents: compares best expected utility

What is a Simple reflex agent?

• Selects actions based only on the current percept, ignoring the percept history

• It uses condition-action rules: Current percept → Match rule → Perform action

• They work only if the environment is fully observable, so the current percept contains all the info needed to choose the correct action.

• Its simple to implement but has limited intelligence

• It only works when the correct decision can be made from the current percept alone

What is a Model-based reflex agent?

• Keeps track of the current state of the world using an internal model, then chooses an action like a reflex agent

• It handles partial observability by keeping track of parts of the world it cannot currently see.

• An internal state (memory) depends on the percept history and reflects unobserved aspects of the current state and it also requires transition and sensor models.

• This agent chooses actions by updating its internal state and using condition-action rules to choose an action

What is a Transition model in a Model-based reflex?

How the world changes over time

What is a Sensor model in a Model-based reflex?

How the state of the world is reflected in the agent’s percepts.

What is a Goal-based agent?

• It uses goal info to describe desirable situatinos and chose actions to achieve these goals.

• Objective: choose an action that will eventually lead to the achievement of its goals.

• It combines goal info with its model of the world to choose actions that achieve the goal.

• It achieves its goal by using search and planning to find action sequences that lead to the goal.

• This agent is more flexible because their decision-making knowledge is represented explicitly and can be modified.

• It keeps track of the current state of the world and the goals it is trying to achieve.

What is a Utility-based agent?

• It compares different actions and chooses the one with the highest utility

• A utility function is an internal representation of the performance measure and measures the agent’s preference among states of the world

• An expected utility is the average

• Its considered ration when its utility function agrees with the external performance measure, maximising utility results in rational behaviour

• It’s more flexible and can handle conflicting goals, uncertainty and expected utility.

What is a learning agent?

• An agent that can learn and improve its performance over time

• They allow the agent to operate in initially unknown environments and become more competent than its initial knowledge alone would allow

• Learning is the modification of each component of the agent using feedback to improve overall performance.

What are the four components of a learning agent?

1. Learning element: modifies the performance element to improve the agent’s performance

2. Critic: provides feedback based on a performance standard to help improve the agent

3. Performance element: represents the whole agent program and selects the agent’s actions

4. Problem generator: creates learning goals to help the agent gain new knowledge

What is Learning?

A process or improving the performance of an agent through experience.

What is Supervised Learning?

• Learn an unknown target function f from labelled examples (x, y) where y = f(x)

• The labelled examples form a training set

• Choose a hypothesis h from a hypothesis space H so that h ≈ f

• The goal is for h to predict well on unseen/test examples

• Possible hypotheses include linear models, logistic regression, neural networks, decision trees, nearest-neighbour examples, grammars, kernerlised separators etc

What is Classification?

• Learning f with a discrete output value.

• e.g.

  • Image → giraffe/llama

  • Email → spam/ham

  • Digit image → 0-9

  • Symptoms → disease

  • Account activity → fraud/no fraud

What is Regression?

• Learning f with real-valued output value.

• e.g. house → selling price

How does a classification system make predictions?

1. Collect labelled training examples

2. Each example has an input and a class label

3. Use features = attributes that help make the classification decision

4. Learn to predict the label of new inputs

• e.g. of features:

  • Spam emails: words, text patterns, sender/link info

  • Digit recognition: pixels and shape patterns

What does curve fitting show about regression?

• It learns a function that relates inputs to real-valued outputs

• A hypothesis/curve is fitted to the observed data points

• Different hypotheses can fit the same training examples differently i.e. straight line or more complex curved

• The learned function is then used to estimate outputs for new inputs.

What data and features are used in supervised learning?

• Data: Labelled instances e.g. emails labelled spam/ham

• This is divided into:

  • Training set → used to learn the model

  • Held-out/validation set → used to tune the model

  • Test set → used for final evaluation

• Features: attribute-value pairs that characterise each input x

What is the correct learning and evaluation process in Supervised Learning?

• Learn parameters from the training set.

• Tune hyperparameters using the held-out/validation set

  • For each hyperparameter value → train the model and test it on held-out data

  • Choose the best value

• Perform the final test on the test set

• Never “peek“ at the test set while training or tuning.

Key rule:

• Parameters → training data

• Hyperparameters → different/held-out data

How is a classifier evaluated in Supervised Learning?

• Accuracy: fraction of instances predicted correctly

• Generalisation: the goal is a classifier that performs well on test/unseen data

What is Overfitting in Supervised Learning?

This fits the training data very closely but does not generalise well

What is Underfitting in Supervised Learning?

This fits the training data poorly.

What does Linear Regression do in Supervised Learning?

• Uses a linear hypothesis family.

• Fits a straight line (or a hyperplane for multiple features) to data

• For one input features:

  • hw​(x) = w0​ + w1​x

• Where:

  • x = input

  • hw(x) = predicted output

  • w0 = intercept

  • w1 = slope/weight

• e.g. x = house size, y = house price

How is error measured in Linear Regression in Supervlised Learning?

• For one example:

  • error/residual = y - hw(x)

• Where:

  • y = actual observation

  • hw(x) = prediction

• Linear regression uses L2 / least-squares loss:

  • Loss=j∑​[yj​−hw​(xj​)]2

• For hw​(x)=w0​+w1​x:

  • Loss=j∑​[yj​−(w0​+w1​xj​)]2

• Goal: w∗=weights that minimise the loss

• At w∗, the derivatives of the loss with respect to each weight are 0.

How are the optimal linear-regression weights w* found in Supervised Learning?

What is the main difference between regression and classification?

• Regression predicts a numerical/real value whereas classification predicts a discrete class (fixed category or label that the model choose from e.g. spam)

• For binary classification: y∈{−1,1} so the output must be either -1 or 1.

How does a linear classifier use a function to turn a numerical output into a discrete class?

Start with the threshold perception function: hw​(x) = w0 ​+ w1​x

where:

• x = input

• w0, w1 = weights

The linear function itself can produce many numericals, so an activiation function g is used to convert that number into a class e.g. spam or ham (1 or -1)

The activation function uses a threshold of 0:

• If the value is >= 0, the output is 1

• If the value is < 0, the output is -1

What is a threshold perceptron in linear classification?

• A single unit that makes a binary classification

• It calculates w * x which defines a hyperplane.

  • Hyperplane separates is the wall separating the classes:

    • y = 1

    • y = -1

• Then applies this rule: y = hw(x)

  • = 1, if w * x >= 0 (positive)

  • = -1, if w * x < 0 (negative)

• Whether y is positive or negative determines the class.

What is the difference between an activation function g and a threshold perceptron?

Activation function g(z):

• A rule inside perceptron

• Takes the weighted sum z = w x as its input

• Converts that numerical value into a class

• Uses a threshold of 0

  • g(z)

    • = 1, if z >= 0 (positive)

    • = -1, if z < 0 (negative)

Threshold perceptron:

• The whole classification unit

• Calculates each input with the weight (w * x = z)

• Applies the activation function g: g(z)

• Produces the final class 1 or -1

When and how does a perceptron adjust its weights?

• Only when it makes an incorrect prediction: y* ≠ hw​(x)

  • Where y* = the true class

Incorrect predictions can be:

• False negatives

  • w * x < 0 (negative) but the true output should be positive

  • Increase weights on positive inputs (+w on +ve inputs)

  • Decrease weights on negative inputs (-w on -ve inputs)

• False positives

  • w * x > 0 (positive) but the true output should be negative

  • Decrease weights on positive inputs (-w on +ve inputs)

  • Increase weights on negative inputs (+w on -ve inputs)

How does the perceptron learning rule train the model?

1. Start with w = 0

2. Use the current weights to classify it

   1. If the classification is correct - no weight change

   2. If the classification is wrong → adjust the weight vector w using the example’s features and true label

Basic idea: w = w + y*x

Hence, training repeatedly changes w to improve the classifications.

What does it mean if training data is linearly separable?

It means this if there is a hyperplane/decision boundary that can perfectly separate the positive and negative examples

What is the perceptron convergence theorem?

• If the training data is linearly separable, repeatedly applying perceptron learning will eventually converge to a perfect seperator

  • This means it eventually finds weights w that correctly separate all training examples

What does non-separable data mean for a perceptron?

• This means no single linear decision boundary can perfectly separate the classes.

• This means even the best linear boundary will make at least one mistake

• So a perceptron cannot find a perfect separator for this type of data

What happens when perceptron learning uses a fixed vs decaying leaarning rate α with non-separable data?

For non-separable data:

Fixed α:

• The same learning rate is used every update

• The perceptron may never converge

e.g. α = 0.2

100,000 updates → no convergence

Decaying α:

• α becomes smaller as learning continues

• The perceptron can approach a minimum-error solution

e.g.

α = 1000/(1000 ​+ t)

where t increases over time

Why is a decaying learning rate useful for non-separable data?

• Because no perfect separator exists, the perceptron may keep changing its weights as it tries to correct different mistakes

• The learning rate α controls how much the weight change:

  • Large α → large weight updates

  • Small α → small weight updates

• By gradually decreasing α, the weight updates become smaller over time

• So it helps the perceptron fine-tune instead of continuously making large corrections

How do the linear classifiers Perceptron, Logistic Regression and Support Vector Machines *(SVM)?

Perceptron

• Uses a threshold

• Is satisfied once it separates the training data

Logistic Regression

• Uses a sigmoid activiation function: g sigmoid​(x) = 1 / (1+e−x)

SVM:

• Finds a separating boundary

• Tries to maximise the margin between the boundary and the nearest training points

Why is a perceptron unable to solve the XOR function?

• A Perceptron uses a single linear decision boundary

• For XOR, the positive and negative examples can’t be separated by one straight line

• Therefore, XOR is not linearly separable so a single perceptron can’t correctly classify XOR

What basic question must a learning method answer?

1. Which hypothesis space H should we choose?

• e.g. What type of model should we use? Choose a hypothesis space H

2. How do we measure how well h fits the data?

• e.g. How do we measure how good it is?

3. How do we balance fit vs complexity? → “Ockham’s razor“

• How do we avoid overfitting?

4. How do we find a good hypothesis h?

• How do we find the best model?

5. How do we know h will predict well on new data?

• How do we know it works on new data?

What is the classical statistics/machine learning approach to learning?

Main idea: minimise a loss function (measures how wrong a model’s predictions are)

• Which hypothesis space H to choose?

  • A linear function → hw​(x )= wTx

• How to measure degree of fit?

  • Use a loss function e.g. squared error → ∑​(yi​ − wTxi​)²

• How to trade off degree of fit vs. complexity?

  • Use regularisation = complex penality → ∣∣w∣∣²

• How do we find a good h?

  • Optimisation (closed-form, numerical) or discrete search

• How do we know if a good h will predict well?

  • Check prediction: try it on other data using cross-validation, bootstrap, etc

What is the probabilistic approach to learning?

Which hypothesis space H to choose?

• Use a probability model → P(y∣x,h)

How to measure degree of fit?

• Use data likelihood → i∏​P(yi​∣xi​,h)

How to trade off degree of fit vs. complexity?

• Use regularisation or a prior → arg max/h ​P(h) ∏​/i P(yi​∣xi​,h)

• This is MAP (Maximum a Posteriori)

How do we find a good h?

• Optimisation (closed-form, numerical); discrete search

How do we know if a good h will predict well?

• Empirical process theory and key assumption is (i)id

What is the Bayesian approach to learning?

Instead of choosing only one best hypothesis, this type of learning considers all hypotheses that have not already been ruled out as impossible

• Which hypothesis space H to choose?

  • All hypotheses with non-zero a priori probability (meaning all hypotheses that are considered possible before seeing the data)

• How to measure degree of fit?

  • Use data probability, as in MLE/MAP

• How to trade off degree of fit vs. complexity?

  • Use priority, as for MAP

• How do we find a good h?

  • Dont choose just one

  • Instead, use the Bayes predictor which combines predictions across hypotheses P(y∣x,D) using their probabilities

Key idea: This type of learning uses the posterior over hypotheses rather than relying on only one h.

How does an artificial neuron product an output?

• Input ai come from other nodes (i or j) or from outside the network

• Each input (link) has a weight w(i,j)

• There isi also a fixed bias input a0 = 1 with bias weight w0j

• The neuron calculates the total input: inj ​= i∑wi,j​ai​

• It passes this through an activation function g: aj​ = g(inj​) =g(i∑​wi,j​ai​)

  • aj becomes the neuron’s output and can be passed to other nodes

• Artificial networks are loosely inspired by biological neurons

What does an activation function g do, and what are 2 examples?

• It converts a neuron’s total input inj, into its output

e.g.

• Threshold: switches to an output such as +1 once the input passes a threshold

• Sigimoid: changes the output smoothly: g(x) = 1/ [1+e^(-x)]

• Differences: Threshold changes suddenly; sigimoid changes smoothly

How does a linear classifier use features and weights to make a prediction?

• Inputs are feature values fi(x)

• Each feature has a weight wi

• The weighted features are summed to calculate the activation

  • activationw​(x) = i∑​wi​ fi​(x) = w * f(x)

• If activation is:

  • Positive → output + 1

  • Negative → output -1

Simple idea:

• Features * weight → sum → check sign → classification

What is the connection between a linear classifier and an artificial neuron?

• Both first calculate a weighted sum of their inputs: (∑/i)​ wi​ xi

• A linear classifier uses the sign of this sum to decide the class

• An artificial neuron passes the weighted sum through an activation function g to produce its output

How does logistic regression turn a linear score into a probability?

• First calculate the linear score (weighted sum): z = w * f(x)

• Then pass z through the sigimoid function: ϕ(z) = 1/ [1+e^(-z)]

• This converts the score into a probability between 0 and 1:

  • Very positive z → probability approaches 1

  • Very negative z → probability approaches 0

• For binary class

How are the probabilities of the 2 classes calculated in binary logistic regression?

Find the probability of +1 using sigmoid, the probability of -1 is whatever is left.

How are the best weights w chosen in logistic regression?

• Use maximum likelihood estimation (check formula img)

• Choose the weights w that make the observed correct classes as probable as possible

Simple idea: Choose w that gives the training examples the highest likelihood.

How does multiclass logistic regression classify and produce probabilities?

• Each class y has its own weight vector: wy

• Calculate each class’s score (activation): zy = wy * f(x)

• Without probabilities, the highest score wins: y = arg max(y)wy * f(x)

• To turn all class scores into probabilities use softmax (check img formula)

How do linear, binary logistic and multiclass logistic classification relate?

• Features → weighted sum → class

• Features → weighted sum → sigimoid → probability

• Features → one weighted sum per class → softmax → class probabilities

Key idea:

• They all begin with weighted sums of features

• Logistic regression adds a function that converts those scores into probabilities.

What is optimisation trying to do in logistic regression?

• Find the weights w that maximise the likelihood of the observed training labels.

• Simple idea: Finds the best weights w

How does hill climbing optimisation work?

1. Start anywhere

2. Move to the best neighbouring state

3. Repeat

4. If no neighbour is better, stop

• For multiclass logistic regression, this is tricky because optimisation occurs in a continuous space, meaning there are infinitely many neighbours

• Simple idea: Keep moving in a better direction until there is nowhere better nearby

How can we choose which direction to move during 1-D optimisation?

• At the current weight w0, we could compare: g(w0 + h) and g(w0 - h) and move in the better direction.

• or Calculate the derivative (check formula img)

Simple idea:

• Derivative = tells us which direction improves the function

How does gradient ascent optimise multiple weights?

For a function i.e. g(w1, w2, …, wn):

• The gradient contains the derivative for every weight

  • Check img for the formula

• The gradient gives the steepest uphill direction

• Update the weights in that direction:

  • Gradient ascent = + gradient → maximise

  • α = learning rate, which controls the step size and must be chosen carefully

  • ∇g(w): gradient → points in the steepest uphill direction

  • Steeper slope/larger derivative → bigger update for that coordinate

What is gradient descent and how does it update weights?

• Repeatedly moves in the steepest downhill direction to minise a function

• w ← w - α∇g(w)

  • Gradient descent = − gradient → minimise

  • - ∇g(w): gradient → points in the steepest downhill direction

  • α = learning rate → controls the step size

What is the difference between batch, stochastic and mini-batch gradient ascent?

All update using gradients, but use different amounts of training data per update.

• Batch gradient ascent: uses all training examples before updating

• Stohastic gradient ascent: pick one random training exmaple j and update immediately

• Mini-batch gradient ascent: pick a small random subset J and update using that group

  • These can be computed in parallel

Simple comparison:

• Batch = all examples

• Stohastic = 1 example

• Mini-batch = small group of examples

How is multiclass logistic regression related to a neural network?

Multiclass regression is a special case of a neural network:

• It has a single layer

• It has no hidden layer

• Each class gets a score zy

• Softmax converts the scores into class probabilities

What is a Multi-layer Perceptron and how does each neuron calculate its value?

• An MLP contains: input layer → hidden layer(s) → output layer

• Each neuron receives values from the previous layer, calculates a weighted sum, then applies a nonlinear activation function g (checkimg formula)

  • k = current layer

  • W = weights between layers

  • g = nonlinear activation function

• The output layer can use softmax to produce class probabilities

Simple idea:

• Previous layer → weighted sum → nonlinear activationo → next layer

What are the common activation functions shown for neural networks?

How is a Multi-layer Perceptron trained?

• Training is similar to logistic regression: choose weights w that maximise the log likelihood (check img formula)

• The difference is that an MLP usually has a much larger weight vector because it contains many connections

• Use gradient ascent, with the gradients calculated using the back-propagation algorithm

Simple idea:

• MLP training = maximise likelihood using gradient ascent + back-propagation

What is the main difference between multiclass logistic regression and an Multi-layer Perceptron?

• MLR: input → weighted scores → softmax → probabilities

• MLP: input → hidden layer(s) with weighted sums + nonlinear activations → output scores → softmax → probabilities

Key idea:

• An MLP extends logistic regression by adding hidden layers and nonlinear activation functions before the output.