Quiz 4 ML

ensemble learning

aggregation of predictions from models to get better predictions

difference between decision tree and random forest

decision tree:

Random forest: is a ensemble of decision tree,

Hard voting

majority vote classifier, has higher accuracy than the single best classifier in the ensemble

Soft vote

all classifier estimate class probabilities and predict the class with the highest class probability averaged over all individual classifiers

bagging (bootstrap aggregation)

sampling with replacement to create samples for the model to learn from

pasting

sampling without replacement

out of bag evaluation

measure of prediction error for random forest. mean prediction error on each training sample x using the trees that did not have x in their bootstrap

two parameters of random forest classifier

ntree: number of decision trees to build before final prediction vote

max_feature: size of subset of random features given to each tree when trying to split nodes optimally

bagging and boosting difference in training

boosting: SEQUENTIAL TRAINING. train predictors sequentially each one tries to correct the one before it. Boosting combines weak learns into one strong learner

bagging: independent training

ada boost steps

1. Compute weighted votes for each class

2. choose class w/ max weighted sum

How does ada boost boost performance

by paying attention to the instances that were underfitted by the predecessor model so that the new predictor will focus on hard cases

How is gradient boost different from ada

ada boost tweaks the weight of every instance everytime but gradient boost fits new predictor to errors made by the previous ones

difference between supervised and unsupervised learning

supervised has labeled data set while unsupervised has no labels

4 unsupervised algorithms

k-means clustering, hierarchical cluster, knn, neural network, anomaly detection, singular value decomposition

clustering

create groups of like instances

how is clustering different from classification

clustering is unsupervised with no given labels, classification is supervised with labels

use cases of clustering

customer segmentation, recommendation systems, image analysis

3 categories of clustering algorithms

hierarchical - tree of clusters (agglomerative -bottom up, divisive- top down) , partition based, density based- arbitrary shaped clusters (DBSCAN), distance based- (based on Euclidean distance, continuous attributes)

centroid

center point of data found by taking mean value of the points, centroid can be a point not in the data point

k means cluster steps

1. choose k val

2. select k objects arbitrarily use as the initial set of k centroids

3. assign each object to cluster that is nearest to the centroid

4. recalculate the centroids of the k clusters

5. repeat 3 & 4 until centroids stop moving

calculate distance from all instances to our k number of centroids the instances will be clustered w the centroid they have the shortest distance to. update the centroid and repeat

how to evaluate clusters after performing k means clustering

external approach- compare clusters w ground truth if available

internal approach-

1. calculate how bad each cluster

2. average the distance between data points within a cluster

3. average distance between datapoints within a cluster

4. average distance between the points with their centroids could be a measurement

elbow method

1. execute k means on data set

2. for each k calculate wcss (within clusters sum of squares)

3. plot wcss values and k values

4. the sharp point of bend of the plot is the optimal k value

what is the use of elbow method in k means clustering

find the most optimal k value

hierarchical clustering

hierarchy of clusters where each node is a cluster consisting of its daughters

difference between hierarchical and kmeans

in k means u have to specify the number of clusters we dont have to do this in hierarchical because we have dendrogram we can ask when to stop

how visualize hierarchical clusters

dendrogram, heatmap, scatterplot

what are the two main different types of hierarchical clustering methods and how do they differ

agglomerative- bottom up

divisive- top down/dividing the cluster

hierarchical cluster steps

1. assign each object to its own one object cluster and calculate distance between each pair of clusters

2. choose closest pair of clusters and merge into single cluster reducing the total # of cluster by 1

3. calculate the distance between new clusters and each old cluster

4. repeat steps 2 and 3 until all objects in one single cluster

single linkage

minimum distance between clusters

complete linkage

maximum distance between clusters

average linkage

average distance between clusters

centroid linkage

distance between cluster centroids

drawbacks of kmeans and how dbscan over comes these drawbacks

drawbacks:

1. may cluster loosely related observations

2. every observation becomes part of some cluster eventually even if they are scattered far in vector space

3. clusters depend on mean values so each data point contributes in forming clusters

4. a slight change in data points can affect clustering outcome

5. need to specify k

6. includes outlier

dbscan doesnt need to specify # of clusters you just need a function to calculate distance between values and decide what is considered close. Produces more reasonable results than k-means access a variety of different distributions.

2 parameters of dbscan

epsilon- neighborhood radius

minpts- minpoint to form dense region

if min= 1 makes no sense every point is its own cluster

min<= 2 same as hierarchical single link metric w dendrogram cut as height e

rule of thumb for minpts is minpts = 2 x dimension

dbscan steps

1. set parameters ( e and min pts)

2. visit each point ( all points marked unvisited initially)

3. select unvisited point( mark as visited and retrieve epsilon neighbor)

4. check density conditions (if # of points in epsilon neighborhood greater than or equal to min pts start new cluster an expand it with density reachability . ELSE: mark point as noise temporarily because it could also be a border point)

5. expand the cluster ( recursively add all density reachable paths, include neighbors of neighbors if they satisfy density conditions)

6. repeat (until all points visited)

output of dbscan

cluster of dense regions and noise outlier points

challenges of nlp

ambiguous languages, figurative language, unstructured data

3 applications of NLP

spam filter- analyze text to stop spam

identify fake news- determine accuracy bias and if news source is trusted

trends and news- track news reports and comments

recruitment - resume screening

Health care - improve care delivery and diagnosis, improves clinical documentation

what is bag of words

model allowing you to count all words in a piece of text. creates occurrence matrix disregarding grammar and word order

limitation of bag of words

1. absence of context or meaning

2. all words treated independently

3. common words dominate

4. poor semantic understanding

TFIDF(Term frequency inverse document frequency) purpose

gives measure to a word that indicates importance based on the words usage in all documents. rescales word frequency by how often a word appears in all text not just the text being analyzed. penalizes scores for common words like “the”.

rewards unique terms and improves bag of words by weights

tokenization

splitting words by blank spaces can pose problems with abbreviations or texts w hyphens, parentheses, and punctuation

stop words

unwanted words w/ no analysis value like “ and” “the” “to” can remove info and context if not careful

stemming

slicing end or beginning of words to remove affixes (prefix/suffix). downside is it can have results that arent words or change meanings

caring→ car

Lemmatization

reduce word to base for and group together different forms of word (ran, running,, runs → run)

takes context into consideration

NER (named entity recognition)

identifies and classifies key elements

• dates

• times

• location

• person

lemmatizer and stemmer difference

stemmer disregards context when splitting words by affixes, while lemmatization brings words back to their single dictionary form and takes into context words with multiple meanings

inflectional affixes

only show grammar rules like tense or count

derivational affixes

create new words and change meanings or word types

library for bag of words in python

countvectorizer

What information do you need to provide to the lemmatizer to get a better answer?

part of speech tag, contextual text and dictionary mappings

perceptron model

single layer neural network giving a single output. it has inputs with weights going into a function with a added bias and gives us one output (y hat prediction)

What is the purpose of a perceptron or neuron or node in a neural network?

inputs that go into a function and get a result or pass a signal or value to the next layer of functions till we come to a result

input layer

receives raw data

hidden layer

does math(function) finds patterns

output

gives us the final answer y

non deep neural network

one hidden layer

deep neural network

2 or more hidden layers

purpose of activation function in neural network

to shrink our output value y into a rnage

binary step

if input value is above or below a certain threshild the neuron is activated and sends the same signal to the next layer.

between 0-1

useful for classification (0-1)

sigmoid

logistic regression activation function between 0-1

1/ 1+e^-x^T theta)

hyperbolic tangent tanh

similar to sigmoid but has better performance

non linear so layers are stackable and it is less likely to get stuck during training because strong negative inputs are mapped to negative outputs and only 0 valued inputs are mapped to near 0 outputs

range (-1, 1)

RELU

rand 0-infinity

gives output x if x is positive and 0 otherwise

becomes dominant

avoids vanishing gradient for positive values because gradient is always equal to 1 when input is positive

sparse activations because negatives become 0

For multi-class classification, what activation function could be used?

softmax function- calculates probability distribution of an instance across k different clsses, calculates the probability of each target class over all possible classes, range 9-1 and the sum of all probabilites is 1

For classification using a neural network, what cost function is often used?

cross entropy (log loss)

-(ylog(y hat) + (1-y hat)log(1-p))

backpropagation purpose

allows neural networks to learn like a feedback loop. adjusts weights and biases to improve predictors

adam optimizer

advanced adaptive version of stochastic gradient descent . updates model weights to minimize the loss function during training