describing data, cleaning data, finding structure in data, k-means clustering

variance

covariance

Pearson correlation (r) properties and how to calculate

scale invariant

cor(x,x) = 1

zwischen -1 bis +1

Problems with pearson correlation

korrelation muss linear sein, sonst nicht damit erkennbar

stark beeinflusst von Ausreissern

Spearman correlation How to

1. transform values to ranks
2. correlation between ranks

how to deal with unexpected values

remove NA values

replace (imputation) form


1. mean
2. random value
3. predict missing value from other variables

dealing with heavily skewed distributions

log transformation

clustering

grouping of similar samples

Principal componentn analysis

show variables with high variance

Euclidean distance

manhattan distance

correlation distance

maximum distance

k means clustering

distance = euclidean metric


1. define k random centers
2. assign each point to the clostest of the centers
3. determine center of gravity of the clusters → new centers
4. assign to new centers
5. start over

stop when: no object changes cluster assignment

interpreting k means clustering

repeat process with different starting points, if identical → very strong cluster structure

fluctuating → no clear cluster structure

k means problem

even if there is no group structure clusters will be identified

choosing k

elbow method or silhouette method

elbow method

if each point has 1 cluster WSS = 0

the more k the higher WSS

kink in curve → optimal number of k

silhouette method

mean distance ai to all members of his cluster, bi smallest average distance to members of all other clustrs

Interpretation:

si \~ 1 → object very well clustered

si \~ 0 ambigous object

si < 0 wrong assignment