3 Distance outliers

What is Euclidean distance?

d2(x,y) = sqrt(∑_j (x_j - y_j)²). It is the straight-line distance between numerical vectors.

What is Manhattan distance?

d1(x,y) = ∑_j |x_j - y_j|. It sums absolute coordinate differences.

What is the Minkowski distance family?

d_q(x,y) = (∑_j |x_j - y_j|^q)^(1/q), q ≥ 1. q = 1 gives Manhattan distance; q = 2 gives Euclidean distance.

Example: find Euclidean and Manhattan distance between (1,2) and (4,6).

Coordinate differences are 3 and 4. Euclidean distance = sqrt(9 + 16) = 5; Manhattan distance = 3 + 4 = 7.

What is squared Mahalanobis distance from a reference mean?

D²(x) = (x - μ)^T * Σ^(-1) * (x - μ), where μ is the reference mean and Σ the reference covariance matrix. Mahalanobis distance D is sqrt(D²).

Why can Mahalanobis and Euclidean distance rank unusualness differently?

Mahalanobis distance accounts for variable scales and covariance. A small deviation across a low-variation direction can be more unusual than a larger deviation along a high-variation direction.

What conditions does the ordinary Mahalanobis formula require of the covariance matrix?

It must be suitably estimated and invertible. Classical estimates can be distorted by outliers, so the resulting distances depend on the reference fit.

What does R’s mahalanobis function return?

Squared Mahalanobis distance D². Compare that value to a chi-square quantile on the same squared-distance scale, not to the quantile’s square root.

How is a chi-square cutoff commonly used with squared Mahalanobis distance?

Under suitable multivariate-normal reference assumptions, flag when D² > χ²_(p,1-α), where p is the number of numerical features. Estimated parameters affect calibration.

Does a theoretical 90th-percentile chi-square cutoff force 10% of sample rows to be flagged?

No. The observed fraction depends on the data and calibration. Selecting a fixed number of highest anomaly scores by rank is a different policy.

In binary-vector comparison, what do f11, f00, f10 and f01 mean?

f11 counts shared presences; f00 shared absences; f10 and f01 count the two kinds of mismatch.

What is simple matching similarity for binary vectors?

SMC = (f11 + f00) / (f11 + f00 + f10 + f01). Both matching presences and matching absences contribute.

What is Jaccard similarity for binary vectors?

J = f11 / (f11 + f10 + f01). Shared absences are excluded; the denominator must be positive.

When is Jaccard often more informative than simple matching?

For sparse presence data such as purchases, where many shared absences should not dominate the comparison.

Example: two binary vectors share 2 presences, have 3 mismatches and 95 shared absences. What are SMC and Jaccard?

SMC = (2 + 95)/100 = 0.97. Jaccard = 2/(2 + 3) = 0.40. Their different treatment of shared absence explains the difference.

What is cosine similarity?

cos(x,y) = (x^T y) / (||x|| * ||y||), for nonzero vectors. It compares direction rather than total vector magnitude.

What is multidimensional scaling trying to preserve?

MDS constructs lower-dimensional coordinates that approximate the original dissimilarities. A 2D picture can distort relationships; its axes are constructed features.

Why do rotations or reflections of an MDS map not change its distance interpretation?

They preserve pairwise distances. The sign or compass direction of a constructed axis has no intrinsic scientific meaning.

What is the standard 1.5-IQR boxplot outlier rule?

Flag values below Q1 - 1.5IQR or above Q3 + 1.5IQR. A flag identifies unusualness under that rule, not a proven data-entry error.

Example: Q1 = 10 and Q3 = 18. What are the usual boxplot fences?

IQR = 8. Lower fence = 10 - 12 = -2; upper fence = 18 + 12 = 30. A value of 34 is flagged.

Why can pooled and subgroup-specific boxplots flag different rows?

Their reference distributions, quartiles and spreads differ. A value ordinary in the pooled data can be unusual within a particular group.

What is the central idea of isolation forest?

Unusual observations tend to be separated by fewer random partitions and have shorter isolation paths. It is an anomaly detector, rather than the supervised random-forest classifier.

If higher isolation scores mean more anomalous, what does lowering a fixed score cutoff do?

It flags at least as many observations, holding scores fixed. The score is not automatically the probability that a measurement is wrong.

What is the difference between a large residual, high leverage and influence?

A large residual is unusual response error; high leverage is an unusual predictor combination; influence means removing the point changes the fitted model substantially.

Can an observation have high leverage but a small residual?

Yes. It can have unusual predictor values but lie near the fitted line, possibly helping determine that line.

Example: detector A flags 12 rows, B flags 10, and 7 overlap. How many unique rows are flagged?

The union has 12 + 10 - 7 = 15 rows. Five are A-only and three are B-only. Agreement does not establish ground truth.