AI 2 (Information Theory)

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/12

encourage image

There's no tags or description

Looks like no tags are added yet.

13 Terms

1
New cards

What is the Predictive Model Structure?

Root or Internal Node: Represents a feature

Leaf Node: Represents the target value

Branch: represents a decision rule

2
New cards

What is a Classification Tree?

A tree where the target variable takes categorical values (e.g. Male, Female)

3
New cards

What are Regression trees?

Trees that the target variable takes continuous values (e.g. Temperature)

4
New cards

How do you construct a decision tree?

Given a dataset, the algorithm:

→ Searches for the best rules to split samples that are dissimilar

→ Repeats the process until the partitions (leaves) reach a certain degree homogeneity (the state of being uniform)

5
New cards

What are the two main methods to split samples?

Gini Index (Impurity)

Information Gain

6
New cards

What is Gini Index (Impurity)?

Used in Classification and Regression Trees (CART)

<p>Used in Classification and Regression Trees (CART)</p>
7
New cards

What is Information Gain?

knowt flashcard image
8
New cards

What is the formula for Entropy {H(Y)}?

knowt flashcard image
9
New cards

What is the formula for Conditional Entropy {H(Y|A)}?

knowt flashcard image
10
New cards

After calculating all Information gains, where do you split?

The feature that gives the largest Information Gain

11
New cards

What are limitations of Decision Trees?

Unstable: A small change in the data can lead to a large change in the structure of the obtained decision tree

Greedy Search: Tree construction has no backtracking, as searching in the space of ALL possible trees is unfeasible

Relatively Inaccurate: Many other models, such as Support Vector Machines and Neural Networks, often perform better on similar data

12
New cards

What is Mutual Information?

Information gain

13
New cards

What is the Mutual Information Feature Selection Algorithm?

knowt flashcard image