Distributions and Skew | data mining!
Call Kai
Learn
Practice Test
Spaced Repetition
Match
Flashcards
Knowt Play1/22
There's no tags or description
Looks like no tags are added yet.
23 Terms
What is a distribution?
The shape of the data: where values fall and how often. A histogram shows it.
Is skew named after the tail or the bulk of the data?
The tail. A long tail of high values means right skewed.
Application: a histogram's bulk sits on the left, with a long tail stretching right. Which way is it skewed?
Right skewed, because skew is named after the tail.
Which way does skew pull the mean?
Toward the tail. Right skew pulls the mean up; left skew pulls it down.
Does the bulk of the data pull the mean toward it?
No. The tail pulls the mean; the median stays near the bulk.
In right skewed data, which is bigger, the mean or the median?
The mean.
In left skewed data, which is bigger, the mean or the median?
The median.
Application: on an easy exam, most students scored 90–100 and a few scored 40–60. Which is bigger, the mean or the median?
The median. The data is left skewed, so the low tail pulls the mean down.
Application: what are the mean and median of 1, 2, 3, 4, 100?
Median 3, mean 22. The 100 drags the mean toward the tail.
Why do engineers report p50 and p99 latency instead of the mean?
Response times are right skewed: most are fast, with a long tail of slow ones. The mean hides both the typical case and the slow tail.
What might two peaks in a histogram suggest?
Two different groups mixed together in the data.
What does the 68–95–99.7 rule say?
For roughly normal data, about 68% of values fall within 1 SD of the mean, 95% within 2 SD, and 99.7% within 3 SD.
In normal data, roughly what percentage of values fall within 1 SD of the mean?
About 68% (not 10%).
Application: heights are roughly normal with mean 70 inches and SD 3 inches. What range holds about 95% of people?
64 to 76 inches (mean ± 2 SD).
Application: with mean 70 and SD 3, how unusual is a 79 inch man?
He's 3 SDs above the mean, taller than about 99.85% of men. 99.7% fall within ±3 SD, leaving 0.15% in each tail.
Why can't you use the 68–95–99.7 rule on income data?
The rule needs a roughly normal shape. Income is right skewed, so mean ± SD gives impossible ranges (e.g. −402 to 1,068 thousand). The problem is the shape, not data quality.
What are the fences in the 1.5 × IQR outlier rule?
Lower fence = Q1 − 1.5 × IQR; upper fence = Q3 + 1.5 × IQR. Values outside the fences get flagged.
Why use the 1.5 × IQR rule instead of mean ± SD to find outliers in skewed data?
It's built on quartiles, which outliers can't pull, so it still works on skewed data.
Application: Q1 = 20 and Q3 = 40. Is 75 an outlier?
The IQR is 20, so the upper fence is 40 + 30 = 70 (start from Q3, not the middle). Yes, 75 is flagged.
What does df["x"].skew() return for right skewed data?
A positive number. A negative number means left skewed.
Why apply np.log1p to a right skewed column?
The log compresses the long tail, which many models handle badly. log1p computes log(1 + x), so zeros stay valid.