Frequency Distributions and Data Visualization - Comprehensive Notes

Frequency Distribution and Class Boundaries

Frequency distribution: groups data into classes (intervals) and counts how many items fall into each class (the frequency).
When a class has more items, data are more concentrated in that class.
The most important part of constructing a frequency distribution is deciding the number of classes (and their boundaries).
Class choice depends on data size:
- With many data points, you can use more classes and still reveal structure.
- With few data points, too many classes can obscure meaning.
Example setup (from the transcript):
- Data range roughly from 12 up to 42 (illustrative).
- Start with simple boundaries: 10–20, 20–30, 30–40, 40–50 (four classes).
- Suppose the data yield the following frequencies:
- f1 (10–20) = 3
- f2 (20–30) = 6
- f3 (30–40) = 5
- f4 (40–50) = 6
- Total data points: $N = \,=\; f1+f2+f3+f4 = 3+6+5+6 = 20.$
Key takeaway: you can choose more or fewer classes, but changing the number of classes can change the observed frequencies, especially with small data sets. With very large data sets (e.g., 1,000,000), class-width changes have a smaller impact on the overall picture.

From Frequency to Relative Frequency and Percent

Relative frequency for class i: $ri = \frac{fi}{N}$
Percent for class i: $pi = 100\times ri = 100\times\frac{f_i}{N}$
Example with the 20-data case:
- $r_1 = \frac{3}{20}=0.15\quad (15\%)$
- $r_2 = \frac{6}{20}=0.30\quad (30\%)$
- $r_3 = \frac{5}{20}=0.25\quad (25\%)$
- $r_4 = \frac{6}{20}=0.30\quad (30\%)$
- Corresponding cumulative totals in percent (see next section) will use these values.

Cumulative Frequency and Cumulative Percent (Probability)

Cumulative frequency up to class i: $CFi = \sum{j=1}^{i} f_j$
Cumulative percent up to class i: $CPi = 100\times\frac{CFi}{N} = \sum{j=1}^{i} pj$
Using the example data:
- CF1=f1=3⇒CP1=100×320=15%CF1=f1=3⇒CP1=100×203=15%
- CF2=f1+f2=3+6=9⇒CP2=100×920=45%CF2=f1+f2=3+6=9⇒CP2=100×209=45%
- CF3=f1+f2+f3=3+6+5=14⇒CP3=100×1420=70%CF3=f1+f2+f3=3+6+5=14⇒CP3=100×2014=70%
- CF4=f1+f2+f3+f4=20⇒CP4=100%.CF4=f1+f2+f3+f4=20⇒CP4=100%.
Interpretation: CP_i represents the probability that a data point is less than the upper boundary of class i (i.e., the distribution function up to that point).
Note: In the transcript, the cumulative percentages were described as 15%, 45%, 70%, and 100% for the four classes, aligning with the example above.

Why Use Frequency Distributions?

Condenses data so you can extract meaningful patterns about behavior or characteristics.
Helps identify where data are concentrated (which class intervals matter most).
Class width and the number of classes can influence the apparent concentration; different setups can yield different pictures, especially with small samples.
Large data sets tend to stabilize the observed concentration across reasonable class choices; small data sets can be more sensitive to the chosen boundaries.
Policy, business, and scientific decisions should be based on sufficiently large data to avoid misleading conclusions from a small sample.

Visualization Methods for Categorical Data

Bar chart: for categorical values; height represents frequency or relative frequency.
Pie chart: shows proportion of each category as a wedge of a circle; can be less precise for comparing small differences.
Donut chart: a donut-shaped variant of a pie chart with an inner hole; can use an extra ring to display more information or categories, but increases space usage.
Pareto chart: a bar chart with bars arranged in descending order, usually with a cumulative line (polygon) on top; helps identify the most important factors (vital few).
- Bars are ordered from largest to smallest frequency.
- The cumulative polygon shows the cumulative percentage across bars.
Two-category side-by-side bar chart: visualizes two categorical variables in one picture (e.g., Error vs No Error by invoice size: Large, Medium, Small).
- Useful for comparing distributions across two categorical dimensions in a single view.
Practical note: choose the visualization that communicates the pattern clearly; there is no single best chart for every situation. Don’t overcomplicate with unnecessary charts.

Pareto Chart: Example and Purpose

Objective: identify the most impactful reasons behind a problem.
Example setup (ATM transaction failures, 724 transactions, 7 reasons):
- Reason 1: Card jam — f1 = 364 occurrences → p1 = $\frac{364}{724} = 0.503\approx 50.41\%$
- Reason 2: Unreadable cards — f2 = 234 occurrences → p2 = $\frac{234}{724} \approx 0.323\approx 32.32\%$
- Remaining categories account for the rest (roughly 17.27% total).
Cumulative percentages:
- CP1 ≈ 50.41%
- CP2 ≈ 82.73%
- CP3, CP4, … continue by adding each category's percent until 100%.
Insight: the top couple of causes explain the majority of failures, so focus remediation on those (the "vital few").
Visual: Pareto chart shows vertical bars in descending order with a line representing the cumulative percentage atop the bars.

Two-Category Visualizations: Side-by-Side Bar Chart

When you have two categorical variables, a side-by-side bar chart helps compare the distribution across categories.
Example from the lecture: two categories such as Error vs No Error and invoice size (Large, Medium, Small).
Interpretation: the chart reveals whether one category tends to be associated with higher/lower values (e.g., more errors with a particular invoice size).
Rationale: helps readers who may not be statistic-savvy to grasp the pattern quickly; complements a table by providing a visual cue.

Visualizing Numerical Data: Histograms and Ordered Data

For numerical (ordinal/continuous) data, you typically do not use simple bar charts for raw data; instead you visualize distributions with histograms.
Histogram vs bar chart:
- Histograms are for numerical data and show frequencies (or densities) within continuous intervals (bins).
- Bars in a histogram touch each other (no gaps) because the data are continuous; the width of each bar corresponds to the bin width.
Other methods for ordered data include stem-and-leaf plots (described next).

Stem-and-Leaf Plot: Concept and Use

Purpose: a compact way to display the distribution of numerical data while preserving the actual data values.
Structure:
- Stem: the leading digits (the part that remains fixed across a range, e.g., the tens digit).
- Leaf: the trailing digits (the units within that stem).
Example interpretation (ages of students):
- Ages range from around 16 up to above 42.
- Stem 1 (tens digit is 1) covers 10–19; leaves show the units (e.g., 6, 7, 8, 9, etc.).
- Stem 2 covers 20–29; leaves show units 0–9 for those ages (e.g., 0,1,2,…9).
- Stem 3 covers 30–39; Stem 4 covers 40–49, etc.
Practical steps to read a stem-and-leaf: the first digit (the stem) is the common leading part; the following digits (the leaves) are the varying trailing digits.
In the classroom example, there were eight observations in the 10–19 range, seven in 20–29, and so on; such a stem-and-leaf plot would allow quick assessment of where most data lie and what the distribution looks like.
Takeaway: stem-and-leaf plots can be used to construct a quick frequency distribution for numerical data and facilitate a smooth transition to a histogram.

Practical Tips and Takeaways

Always consider the class boundaries carefully: different class widths can yield different pictures of the same data; check robustness with different choices, especially for small samples.
Prefer relative frequencies and percentages when communicating to policymakers or non-technical audiences; it makes interpretation easier.
Use cumulative percentages to discuss probabilities or to discuss thresholds (e.g., what proportion of data falls below a certain value).
When presenting data to a diverse audience, choose a visualization that communicates clearly without overwhelming the reader; the same data can be shown with different charts depending on the message you want to emphasize.
Remember that larger data sets generally yield more stable and reliable patterns; relying on 20 observations for a general conclusion can be misleading.
In practice, you often report relative frequencies or percentages rather than raw frequencies, though some presentations (e.g., in charts) may show counts.

Quick Recap: Key Formulas and Concepts

Class frequencies: $fi\quad (i=1,2,\dots,k)$ with total $N = \sum{i=1}^{k} f_i$
Relative frequency: $ri = \frac{fi}{N}$
Percent: $pi = 100\times ri = \frac{100\,f_i}{N}$
Cumulative frequency: $CFi = \sum{j=1}^{i} f_j$
Cumulative percent: $CPi = 100\times \frac{CFi}{N} = \sum{j=1}^{i} pj$
Example class setup (illustrative): 10–20, 20–30, 30–40, 40–50 with frequencies $f1=3, f2=6, f3=5, f4=6$ and $N=20$ gives $p1=15\%, p2=30\%, p3=25\%, p4=30\%$ and $CP1=15\%, CP2=45\%, CP3=70\%, CP4=100\%.$