Scientific Data Analysis: Data Manipulation, Descriptive Statistics, and Distributional Exploration

Software Environment, Data Organization, and Spreadsheet Operations

  • Structured File and Directory Management:

    • Standard scientific workflow requires dedicated, hierarchical directory structures on cloud storage (e.g., OneDrive) separated by course unit and individual instructional topics (e.g., Topic 1).

    • Weekly datasets must be retrieved directly from the repository environment and stored locally within corresponding topic directories to maintain data integrity.

  • Spreadsheet Menu Architecture:

    • Standard spreadsheet applications structure tools across a horizontal top ribbon organized into functional tabs:

    • File: Document management, saving, importing, and exporting.

    • Home: Formatting, cell styling, clipboard operations, and basic data manipulation tools (including Sort & Filter).

    • Insert: Insertion of tables, visual displays (charts, histograms, scatter plots), and specialized symbols.

    • Formulas: Function libraries, formula auditing, calculation options, and defined names.

    • Data: External data connections, data queries, advanced sorting, filtering, and data validation.

    • Help: Integrated documentation, function reference guides, and software support.

  • Syntax and Execution of Essential Summary Functions:

    • Formulae are inputted directly into the function bar preceding data sheets and require an initial equals sign (=) followed by the operator and targeted cell range arguments:

    • =COUNT(range)=\text{COUNT}(\text{range}): Calculates the total number of cells in a specified range containing numerical values. Empty cells or cells containing text strings are excluded. Full column references, such as `=COUNT(A:A)=\text{COUNT}(A:A) , evaluate all numerical rows while ignoring header labels.

    • =SUM(range)=\text{SUM}(\text{range}): Computes the additive total of all numerical values within a designated block (e.g., =SUM(C2:C5)=\text{SUM}(C2:C5) adds rows 22 through 55 in Column C) or an entire column (e.g., =SUM(C:C)=\text{SUM}(C:C)).

    • =AVERAGE(range)=\text{AVERAGE}(\text{range}): Determines the arithmetic mean of all numeric values across a specified array (e.g., =AVERAGE(D:D)=\text{AVERAGE}(D:D)).

    • =COUNTIF(range,condition)=\text{COUNTIF}(\text{range}, \text{condition}): Tallies the number of cells within an evaluated range that satisfy an exact logical criterion or string match. String criteria require double quotation marks (e.g., =COUNTIF(B:B,"blue")=\text{COUNTIF}(B:B, \text{"blue"})), as do relational mathematical comparisons (e.g., =COUNTIF(G:G,">165")=\text{COUNTIF}(G:G, \text{">165"})).

    • =MAX(range)=\text{MAX}(\text{range}): Returns the largest mathematical value within a series, ignoring text headers automatically (e.g., =MAX(C:C)=\text{MAX}(C:C) identifies the absolute maximum in Column C, while =MAX(D2:D11)=\text{MAX}(D2:D11) evaluates strictly rows 22 to 1111 of Column D).

    • =FREQUENCY(data_array,bins_array)=\text{FREQUENCY}(\text{data\_array}, \text{bins\_array}): Calculates how often values occur within specific numerical intervals (bins), returning a vertical array of counts.

  • Spreadsheet Manipulation Techniques:

    • Column Insertion and Dynamic Indexing: Inserting a new column shifts existing columns to the right. Dynamic sequential numbering (e.g., creating an identifier column ID) is executed by typing the initial values (11, 22) in consecutive rows, selecting both cells, and using the autofill handle (bottom right-hand corner of the active cell box) to drag downward.

    • Data Sorting and Selection Expansion: When sorting a single column of interest, adjacent records within the same table must maintain row-level integrity. If a single column is highlighted and sorted via Home > Sort & Filter > Sort Smallest to Largest, the software detects adjacent values and initiates a Sort Warning dialog box.

Excel Sort and Filter OptionsExcel Sort Warning Dialog Box
  • The user must select Expand the selection rather than Continue with the current selection; choosing the former reorders entire horizontal rows simultaneously based on the sorted column, preventing records from becoming mismatched.

Data Classification, Structure, and In-Class Collection

  • Core Statistical Terminology:

    • Observational Unit: The individual entity, subject, or object on which measurements are collected (e.g., an individual student, an oyster specimen, or a specific bird roosting location).

    • Variable: Any characteristic, attribute, or quantity recorded for an observational unit that varies from one entity to another.

  • Taxonomy of Data Types:

    • Qualitative (Categorical) Data: Variables that describe qualities, attributes, or categories rather than measurable quantities:

    • Nominal Data: Categories with no natural, objective ranking or order (e.g., Eye colour: blue, brown, green, grey; Hair colour: black, blonde, brown).

    • Ordinal Data: Categories that have a logical hierarchy, sequence, or relative scale, but lack uniform mathematical distances between categories (e.g., Size ratings: Small, Medium, Large).

    • Quantitative (Numerical) Data: Variables that represent measurable, numerical counts or physical quantities:

    • Discrete Data: Numerical values resulting from counting processes, taking on distinct, separate values (often integers) with no intermediate fractions possible (e.g., Number of siblings: 0,1,2,3,…0, 1, 2, 3, \dots).

    • Continuous Data: Numerical values resulting from physical measurements along an uninterrupted continuum, capable of taking any real value within a given interval depending on measurement precision (e.g., Handspan in cm\text{cm}, Height in cm\text{cm}, Oyster volume in cc\text{cc}, Weight in g\text{g}, Shoe size).

  • In-Class Biometric Dataset Structure:

    • Biometric measurements recorded across students define observational units and variables:

    • ID: Nominal integer identifier distinguishing individual students.

    • Eye colour: Qualitative nominal variable.

    • Hair colour: Qualitative nominal variable.

    • Siblings: Quantitative discrete variable.

    • Shoe size: Quantitative numerical variable.

    • Handspan: Quantitative continuous variable (measured in cm\text{cm} using standardized calibration tools such as https://www.ginifab.com/feeds/cm_to_inch/actual_size_ruler.html).

    • Height: Quantitative continuous variable (measured in cm\text{cm}).

Sample STAT1004 Student Dataset
  • Observed Sample Records (n=10n = 10 Subsample of Class):

    • ID 11: Eye colour = brown, Hair colour = brown, Siblings = 66, Shoe size = 8.58.5, Handspan = 17.9 cm17.9\,\text{cm}, Height = 165 cm165\,\text{cm}

    • ID 22: Eye colour = blue, Hair colour = brown, Siblings = 22, Shoe size = 12.512.5, Handspan = 21.0 cm21.0\,\text{cm}, Height = 196 cm196\,\text{cm}

    • ID 33: Eye colour = blue, Hair colour = blonde, Siblings = 33, Shoe size = 9.09.0, Handspan = 18.0 cm18.0\,\text{cm}, Height = 170 cm170\,\text{cm}

    • ID 44: Eye colour = blue, Hair colour = blonde, Siblings = 22, Shoe size = 9.09.0, Handspan = 18.0 cm18.0\,\text{cm}, Height = 176 cm176\,\text{cm}

    • ID 55: Eye colour = green, Hair colour = brown, Siblings = 22, Shoe size = 9.09.0, Handspan = 18.0 cm18.0\,\text{cm}, Height = 171 cm171\,\text{cm}

    • ID 66: Eye colour = blue, Hair colour = brown, Siblings = 22, Shoe size = 13.013.0, Handspan = 21.0 cm21.0\,\text{cm}, Height = 189 cm189\,\text{cm}

    • ID 77: Eye colour = blue, Hair colour = blonde, Siblings = 22, Shoe size = 13.013.0, Handspan = 21.0 cm21.0\,\text{cm}, Height = 193 cm193\,\text{cm}

    • ID 88: Eye colour = brown, Hair colour = black, Siblings = 22, Shoe size = 11.511.5, Handspan = 19.8 cm19.8\,\text{cm}, Height = 180 cm180\,\text{cm}

    • ID 99: Eye colour = brown, Hair colour = brown, Siblings = 22, Shoe size = 7.07.0, Handspan = 17.0 cm17.0\,\text{cm}, Height = 170 cm170\,\text{cm}

    • ID 1010: Eye colour = blue, Hair colour = brown, Siblings = 33, Shoe size = 11.011.0, Handspan = 18.5 cm18.5\,\text{cm}, Height = 180 cm180\,\text{cm}

    • Summary Statistics for Full Class Cohort (n=23n = 23):

  • Siblings: Mean = 1.911.91, Standard Error = 0.290.29, Median = 22, Mode = 22, Standard Deviation = 1.381.38, Sample Variance = 1.901.90, Kurtosis = 3.223.22, Skewness = 1.541.54, Range = 66, Minimum = 00, Maximum = 66, Sum = 4444, Count = 2323.

  • Shoe size: Mean = 10.5910.59, Standard Error = 0.360.36, Median = 1111, Mode = 1111, Standard Deviation = 1.751.75, Sample Variance = 3.063.06, Kurtosis = −0.77-0.77, Skewness = −0.40-0.40, Range = 66, Minimum = 77, Maximum = 1313, Sum = 243.5243.5, Count = 2323.

  • (cm\text{cm}): Mean = 19.47 cm19.47\,\text{cm}, Standard Error = 0.360.36, Median = 19.8 cm19.8\,\text{cm}, Mode = 21 cm21\,\text{cm}, Standard Deviation = 1.71 cm1.71\,\text{cm}, Sample Variance = 2.942.94, Kurtosis = −0.56-0.56, Skewness = −0.10-0.10, Range = 7 cm7\,\text{cm}, Minimum = 16 cm16\,\text{cm}, Maximum = 23 cm23\,\text{cm}, Sum = 447.9 cm447.9\,\text{cm}, Count = 2323.

  • Height (cm\text{cm}): Mean = 180.91 cm180.91\,\text{cm}, Standard Error = 1.791.79, Median = 182 cm182\,\text{cm}, Mode = 186 cm186\,\text{cm}, Standard Deviation = 8.60 cm8.60\,\text{cm}, Sample Variance = 73.9073.90, Kurtosis = −0.89-0.89, Skewness = −0.23-0.23, Range = 31 cm31\,\text{cm}, Minimum = 165 cm165\,\text{cm}, Maximum = 196 cm196\,\text{cm}, Sum = 4161 cm4161\,\text{cm}, Count = 2323.

Experimental Calibration and Computer Vision: The Oyster Dataset

  • Study Origin and Experimental Context:

    • Collected in 2001 at an R&D laboratory at AGRI-TECH Inc. in Woodstock, Virginia.

    • Published in Three-dimension Reconstruction for High-speed Volume Measurement, Proceedings of the International Society for Optical Engineering (2001).

    • Designed as a calibration experiment to estimate the volumetric size of oysters and evaluate/compare two computer vision imaging technologies (2-D area-based imaging versus 3-D volumetric reconstruction) for automated size classification based on pixel counts.

  • Variable Architecture:

    • Oyster_ID: Unique numerical identification code for each specimen.

    • Oyster_Weight_g: Mass of the specimen measured in grams (g\text{g}).

    • Oyster_Volume_cc: True physical volume of the specimen measured in cubic centimeters (cc\text{cc}).

    • Pixels_3D: Size metric obtained from the 3-D imaging system, quantified in volume pixels (voxels).

    • Pixels_2D: Size metric obtained from the 2-D computer vision system, quantified in pixel area.

  • Classification Rules and Semi-Automated Categorization:

    • Commercial classification bins oysters into three ordinal grades based on volume (VV):

    • Small: V∈(0,10] ccV \in (0, 10]\,\text{cc}

    • Medium: V∈[10,13) ccV \in [10, 13)\,\text{cc}

    • Large: V∈[13,∞) ccV \in [13, \infty)\,\text{cc}

    • Procedural workflow in Excel:

    • Sort records by Oyster_Volume_cc (Column C) ascending, using selection expansion.

    • Label Column F as Size.

    • Populate cell F2 with "Small" for rows where volume is ≤10 cc\le 10\,\text{cc}.

    • Click and hold the fill handle at the bottom-right corner of cell F2 and drag downward to row 1111 to propagate the classification across all qualifying records.

Fill Down Selection Handle in Excel
* Repeat labeling for `"Medium"` and `"Large"` intervals, verifying counts using `=COUNTIF(F:F,"Small")=\text{COUNTIF}(F:F, \text{"Small"})`.
  • Bivariate Graphical Modeling (Scatter Plots):

    • Implemented by highlighting paired numerical columns (e.g., Column C Oyster_Volume_cc and Column D Pixels_3D) and navigating to Insert > Charts > All Charts > X Y (Scatter).

    • Flipping Plot Axes: If variables are plotted on incorrect coordinates, right-click inside the plot area, click Select Data, and modify the cell ranges assigned to X values and Y values within the dialog box, updating the series name to match the corrected dependent variable.

Visual Tools for Categorical Data

  • Overview of Visual Tools by Data Type:

    • Categorical Data: Pie charts, Bar charts, Pareto charts.

    • Numerical Data: Frequency histograms, Frequency polygons, Cumulative frequency polygons (ogives), Box-and-whisker plots (boxplots).

  • Pie Charts (Proportional Composition):

    • Represents proportional or percentage breakdowns of a single categorical variable, where the entire circle equals 100%100\%.

    • Observed distribution of student eye colours (n=23n = 23 total observations):

    • Blue: 99 students (39%39\%)

    • Brown: 88 students (35%35\%)

    • Green: 55 students (22%22\%)

    • Grey: 11 student (4%4\%)

Count of Butterfly Species Pie Chart
  • Observed species counts in butterfly population sampling:

    • Coenonympha tullia: 6262

    • Glaucopsyche lygdamus: 1818

    • Ochlodes sylvanoides: 1717

    • Plebejus icarioides: 1212

    • Papilio zelicaon: 88

    • Bar Charts (Frequency Comparison):

  • Displays counts or relative frequencies across discrete, non-overlapping categories with distinct spaces between bars (indicating absence of continuity).

  • Student hair colour observations demonstrate categories organized along the horizontal axis (Black, Blonde, Brown) against absolute frequency on the vertical scale.

    • Pareto Charts (Quality Control and Ranked Analysis):

  • Developed by Italian economist Vilfredo Pareto; standardly employed in industrial engineering and quality management to isolate the "vital few" causes of process defects.

  • Dual-Axis Architecture:

    • Primary vertical axis (left): Ranks categories in descending order of absolute frequency (tallest bar on the left to shortest bar on the right).

    • Secondary vertical axis (right): Displays a continuous cumulative percentage line starting at the top of the first category and terminating at exactly 100%100\%.

  • Readily adapted to ecological and scientific analysis to distinguish dominant observational groupings.

Pareto Chart of Carnaby Black Cockatoo Roosting Sites

Measures of Central Tendency

  • Mean (xˉ\bar{x}):

    • The arithmetic average, calculated by dividing the sum of all numerical values by the sample size (nn):

xˉ=∑xn\bar{x} = \frac{\sum x}{n}

  • Most mathematically powerful measure of centre when data is symmetric and bell-shaped.

  • Highly sensitive to extreme values or distribution skewness, particularly in small samples, pulling the calculated center toward the extreme tail.

    • Median:

  • The physical middle observation of a dataset after sorting all values in ascending order.

  • Positional calculation depending on sample size (nn):

    • If nn is odd: The middle value situated at rank n+12\frac{n + 1}{2}.

    • If nn is even: The arithmetic average of the two central values located at ranks n2\frac{n}{2} and n2+1\frac{n}{2} + 1.

  • Resistant to outliers and extreme skewness; widely preferred for heavily skewed socioeconomic indicators such as individual income or suburban house prices.

    • Mode:

  • The most frequently occurring score, value, or category within a dataset.

  • Operates across both numerical and categorical data types (unlike mean or median).

  • A dataset can possess no mode, a single mode (unimodal), two distinct modes (bimodal), or multiple modes (multimodal).

Measures of Dispersion and the Garden Worm Study

  • The Biological Benchmark Study:

    • Derived from Introduction to Statistics for Biology (Trudy A. Watt, Robin H. McCleery, Tom Hart).

    • Evaluates physical body lengths of common garden earthworms to establish standard analytical protocols.

    • Sample Measurements (n=10n = 10 Backyard Worms, in cm\text{cm}):

    • Unordered array: 11.5,10.0,9.5,8.0,12.5,13.5,9.5,10.5,9.0,6.011.5, 10.0, 9.5, 8.0, 12.5, 13.5, 9.5, 10.5, 9.0, 6.0

    • Ordered array: 6.0,8.0,9.0,9.5,9.5,10.0,10.5,11.5,12.5,13.56.0, 8.0, 9.0, 9.5, 9.5, 10.0, 10.5, 11.5, 12.5, 13.5

Frequency Histogram of Worm Lengths
  • Histogram Visual Variation:

    • Bin width directly affects the visual perception of numerical distributions: the same dataset of 1010 worms yields different histogram profiles depending on chosen interval boundaries (e.g., bins [6, 9.5], (9.5, 13], (13, 16.5]).

  • Central Tendency Calculations for Worm Sample:

    • Mean:

xˉ=11.5+10.0+9.5+8.0+12.5+13.5+9.5+10.5+9.0+6.010=100.0010=10.00 cm\bar{x} = \frac{11.5 + 10.0 + 9.5 + 8.0 + 12.5 + 13.5 + 9.5 + 10.5 + 9.0 + 6.0}{10} = \frac{100.00}{10} = 10.00\,\text{cm}

  • Median: For n=10n = 10, average the 5th5^{\text{th}} (9.59.5) and 6th6^{\text{th}} (10.010.0) ranked values:

Median=9.5+10.02=9.75 cm\text{Median} = \frac{9.5 + 10.0}{2} = 9.75\,\text{cm}

  • Mode: The single score repeated most frequently is 9.50 cm9.50\,\text{cm} (frequency = 22).

    • Measures of Dispersion (Spread):

  • Range: The mathematical distance between the absolute maximum and minimum values:

Range=xmax−xmin=13.50−6.00=7.50 cm\text{Range} = x_{\text{max}} - x_{\text{min}} = 13.50 - 6.00 = 7.50\,\text{cm}

* Highly simplistic; measures the total boundaries of 100%100\% of observations without revealing the internal concentration of intermediate values.
  • Interquartile Range (IQRIQR): The difference between the third quartile (Q3Q_3, 75th75^{\text{th}} percentile) and the first quartile (Q1Q_1, 25th25^{\text{th}} percentile), measuring the spread of the central 50%50\% of ordered data:

    • Lower half values (55 lowest observations): 6.0,8.0,9.0,9.5,9.56.0, 8.0, 9.0, 9.5, 9.5. Median (Q1Q_1) = 9.0 cm9.0\,\text{cm}.

    • Upper half values (55 highest observations): 10.0,10.5,11.5,12.5,13.510.0, 10.5, 11.5, 12.5, 13.5. Median (Q3Q_3) = 11.5 cm11.5\,\text{cm}.

    • Calculation:

IQR=Q3−Q1=11.50−9.00=2.50 cmIQR = Q_3 - Q_1 = 11.50 - 9.00 = 2.50\,\text{cm}

* *Note:* Standard automated spreadsheet summary packages (such as Excel's *Descriptive Statistics* tool) do not include IQRIQR in their standard summary output.
  • Sample Variance (s2s^2) and Sample Standard Deviation (ss):

    • Quantifies dispersion relative to the arithmetic mean, penalizing scores that deviate substantially from the center.

Sample Standard Deviation Formula
  • Standard mathematical formulation for sample standard deviation:

s=∑(x−xˉ)2n−1s = \sqrt{\frac{\sum (x - \bar{x})^2}{n - 1}}

  • Manual Computation Matrix (xˉ=10.00 cm\bar{x} = 10.00\,\text{cm}, n=10n = 10):

    • Score x1=6.0 cmx_1 = 6.0\,\text{cm}: Deviation x−xˉ=−4.0x - \bar{x} = -4.0, Squared Deviation (x−xˉ)2=16.00(x - \bar{x})^2 = 16.00

    • Score x2=8.0 cmx_2 = 8.0\,\text{cm}: Deviation x−xˉ=−2.0x - \bar{x} = -2.0, Squared Deviation (x−xˉ)2=4.00(x - \bar{x})^2 = 4.00

    • Score x3=9.0 cmx_3 = 9.0\,\text{cm}: Deviation x−xˉ=−1.0x - \bar{x} = -1.0, Squared Deviation (x−xˉ)2=1.00(x - \bar{x})^2 = 1.00

    • Score x4=9.5 cmx_4 = 9.5\,\text{cm}: Deviation x−xˉ=−0.5x - \bar{x} = -0.5, Squared Deviation (x−xˉ)2=0.25(x - \bar{x})^2 = 0.25

    • Score x5=9.5 cmx_5 = 9.5\,\text{cm}: Deviation x−xˉ=−0.5x - \bar{x} = -0.5, Squared Deviation (x−xˉ)2=0.25(x - \bar{x})^2 = 0.25

    • Score x6=10.0 cmx_6 = 10.0\,\text{cm}: Deviation x−xˉ=0.0x - \bar{x} = 0.0, Squared Deviation (x−xˉ)2=0.00(x - \bar{x})^2 = 0.00

    • Score x7=10.5 cmx_7 = 10.5\,\text{cm}: Deviation x−xˉ=0.5x - \bar{x} = 0.5, Squared Deviation (x−xˉ)2=0.25(x - \bar{x})^2 = 0.25

    • Score x8=11.5 cmx_8 = 11.5\,\text{cm}: Deviation x−xˉ=1.5x - \bar{x} = 1.5, Squared Deviation (x−xˉ)2=2.25(x - \bar{x})^2 = 2.25

    • Score x9=12.5 cmx_9 = 12.5\,\text{cm}: Deviation x−xˉ=2.5x - \bar{x} = 2.5, Squared Deviation (x−xˉ)2=6.25(x - \bar{x})^2 = 6.25

    • Score x10=13.5 cmx_{10} = 13.5\,\text{cm}: Deviation x−xˉ=3.5x - \bar{x} = 3.5, Squared Deviation (x−xˉ)2=12.25(x - \bar{x})^2 = 12.25

    • Sum of Deviations: ∑(x−xˉ)=0.0\sum (x - \bar{x}) = 0.0

    • Sum of Squared Deviations: ∑(x−xˉ)2=42.50\sum (x - \bar{x})^2 = 42.50

  • Variance and Deviation Solutions:

s2=42.5010−1=42.509≈4.722…s^2 = \frac{42.50}{10 - 1} = \frac{42.50}{9} \approx 4.722\dots

s=42.509=4.722…≈2.17 cms = \sqrt{\frac{42.50}{9}} = \sqrt{4.722\dots} \approx 2.17\,\text{cm}

  • Comprehensive Output Table (Descriptive Statistics for Worm Length):

    • Mean = 10.0010.00

    • Standard Error = 0.690.69

    • Median = 9.759.75

    • Mode = 9.509.50

    • Standard Deviation = 2.172.17

    • Sample Variance = 4.724.72

    • Kurtosis = 0.240.24

    • Skewness = −0.15-0.15

    • Range = 7.507.50

    • Minimum = 6.006.00

    • Maximum = 13.5013.50

    • Sum = 100.00100.00

    • Count = 10.0010.00

Box-and-Whisker Plot of Worm Lengths
  • Sample Size Invariance of Standard Deviation:

    • Standard deviation estimates the underlying population spread. When random samples of different sizes (n=10n = 10, n=100n = 100, n=200n = 200) are repeatedly drawn from the same stable biological population, the calculated sample standard deviation remains approximately identical across samples, whereas sample sums and counts scale proportionally with sample size.

Distributional Shape, Outlier Detection, and Ogives

  • Morphology of Distribution Curves (Measures of Shape):

    • Skewness describes distribution asymmetry across a frequency histogram, polygon, or boxplot relative to where the majority of observations concentrate.

Comparison of Symmetrical, Positive, and Negative Skewness Distributions
  • Symmetric Distribution (Zero Skew):

    • Unimodal, classic bell-shaped profile (approximately normal distribution).

    • Observations concentrate evenly around the central axis, with values tapering identically into left and right tails.

    • Mean, median, and mode coincide at the same point:

Mean=Median=Mode\text{Mean} = \text{Median} = \text{Mode}

* Typical biological examples: Heights and weights of university students.
  • Positive Skew (Right-Hand Skew):

    • The bulk of data values cluster at the lower end of the measurement scale, while a small number of extremely high observations extend the tail toward the right.

    • Mathematical ordering of central tendency metrics:

Mode<Median<Mean\text{Mode} < \text{Median} < \text{Mean}

* Typical real-world example: Age distribution of undergraduate university students.
  • Negative Skew (Left-Hand Skew):

    • The majority of data values cluster at the upper end of the measurement scale, while rare, exceptionally low values extend the tail toward the left.

    • Mathematical ordering of central tendency metrics:

Mean<Median<Mode\text{Mean} < \text{Median} < \text{Mode}

* Typical real-world example: Academic grade distributions in competitive university courses.
  • Definite determination of skewness cannot be established reliably from tiny samples due to potential multimodality; large representative samples are necessary to identify true population shape.

    • Detection and Methodological Treatment of Statistical Outliers:

  • Outliers are atypical measurements that diverge markedly from the overarching pattern of the data.

  • Outliers do not necessarily denote erroneous measurements; they must first be validated against data collection protocols to determine whether they represent transcription errors, equipment malfunctions, or genuine natural extremes.

  • In standard visual exploration, box-and-whisker plots display outliers as discrete plotted points located beyond the outer fences (whiskers).

  • Statistical practice frequently computes dual evaluations: once with the complete dataset and once with verified outliers removed, evaluating the structural robustness of the reported parameters.

    • Cumulative Frequency Polygons (Ogives):

  • Graph of running cumulative totals or cumulative percentages plotted against upper interval boundaries.

  • In an earthworm sample of size n=75n = 75, converting absolute cumulative counts to relative cumulative percentages (0%0\% to 100%100\%) standardizes data to evaluate percentiles regardless of total sample size.

Systematic Case Study: Carnaby's Black Cockatoo

  • Ecological Background:

    • Field survey assessing roosting site preferences and flock sizes of the endangered Carnaby's Black Cockatoo (Calyptorhynchus latirostris) across Western Australia.

  • Categorical Analysis of Roosting Substrates:

    • Sunset monitoring of n=100n = 100 roosting events categorized by tree type and urban setting:

    • Urban Pine (UrbPine): 3434 sites (34%34\%)

    • Urban Eucalyptus (UrbEuca): 2626 sites (26%26\%)

    • Non-Urban Pine (NonUrbPine): 2222 sites (22%22\%)

    • Non-Urban Eucalyptus (NonUrbEuca): 1818 sites (18%18\%)

    • Total Observations (nn) = 100100

Carnaby Cockatoo Roosting Sites Frequency Table
  • Pareto Ranking Construction:

    • First ranked: UrbPine (3434, Cumulative = 34%34\%)

    • Second ranked: UrbEuca (2626, Cumulative = 60%60\%)

    • Third ranked: NonUrbPine (2222, Cumulative = 82%82\%)

    • Fourth ranked: NonUrbEuca (1818, Cumulative = 100%100\%)

    • Quantitative Roosting Flock Size Survey (n=100n = 100 Complete Dataset):

  • Raw Sunset Counts:

29,18,87,44,65,7,41,49,44,50,42,44,51,39,81,49,37,60,29,48,29, 18, 87, 44, 65, 7, 41, 49, 44, 50, 42, 44, 51, 39, 81, 49, 37, 60, 29, 48,

47,50,71,29,40,50,31,108,68,37,51,69,49,50,13,53,107,36,81,57,47, 50, 71, 29, 40, 50, 31, 108, 68, 37, 51, 69, 49, 50, 13, 53, 107, 36, 81, 57,

37,90,87,39,78,69,40,45,60,42,66,50,156,42,65,47,72,71,39,40,37, 90, 87, 39, 78, 69, 40, 45, 60, 42, 66, 50, 156, 42, 65, 47, 72, 71, 39, 40,

118,50,33,25,8,62,22,48,59,76,38,42,55,57,17,53,80,131,20,74,118, 50, 33, 25, 8, 62, 22, 48, 59, 76, 38, 42, 55, 57, 17, 53, 80, 131, 20, 74,

4,92,68,39,51,80,27,25,65,15,44,43,55,44,13,64,82,37,70,534, 92, 68, 39, 51, 80, 27, 25, 65, 15, 44, 43, 55, 44, 13, 64, 82, 37, 70, 53

  • Binned Frequency Distribution (n=100n = 100):

    • Bin 1–201\text{--}20: Frequency = 99, Cumulative Frequency = 99

    • Bin >20–40>20\text{--}40: Frequency = 2222, Cumulative Frequency = 3131

    • Bin >40–60>40\text{--}60: Frequency = 3838, Cumulative Frequency = 6969

    • Bin >60–80>60\text{--}80: Frequency = 1919, Cumulative Frequency = 8888

    • Bin >80–100>80\text{--}100: Frequency = 77, Cumulative Frequency = 9696

    • Bin >100–120>100\text{--}120: Frequency = 33, Cumulative Frequency = 9898

    • Bin >120–140>120\text{--}140: Frequency = 11, Cumulative Frequency = 9999

    • Bin >140–160>140\text{--}160: Frequency = 11, Cumulative Frequency = 100100

Carnaby Black Cockatoo Survey Boxplot with Outliers
  • Outlier Removal and Comparative Sensitivity Analysis:

    • Inspection of the boxplot identifies three distinct high-value statistical outliers situated above the upper whisker: 118118, 131131, and 156156.

    • Removing these three extreme observations reduces the cohort to n=97n = 97, modifying the highest frequency intervals (bin >100–120>100\text{--}120 drops from 33 to 22; bins >120–140>120\text{--}140 and >140–160>140\text{--}160 drop to 00).

Carnaby Black Cockatoo Survey Boxplot with Outliers Removed
  • Direct Comparison of Descriptive Parameters:

    • Sample Size (nn): 100100 (With Outliers) vs. 9797 (Without Outliers)

    • Mean (xˉ\bar{x}): 52.8552.85 (With Outliers) vs. 50.3150.31 (Without Outliers) — shifted downward by 2.542.54 units (4.8%4.8\% reduction).

    • Median: 49.5049.50 (With Outliers) vs. 49.0049.00 (Without Outliers) — minimal shift of 0.500.50 units, illustrating resistance to extreme values.

    • Mode: 5050 (With Outliers) vs. 5050 (Without Outliers) — entirely unaffected.

    • Range: 152152 (156−4156 - 4) vs. 104104 (108−4108 - 4) — reduced by 4848 units (31.6%31.6\% reduction).

    • Interquartile Range (IQRIQR): 27.5027.50 vs. 26.5026.50 — shifts by only 1.001.00 unit, demonstrating stability of the central 50%50\% core.

    • Standard Deviation (ss): 25.7525.75 vs. 21.4121.41 — contracted by 4.344.34 units (16.9%16.9\% decrease), reflecting reduced dispersion.

    • Sample Variance (s2s^2): 662.96662.96 vs. 458.49458.49 — reduced by 204.47204.47 units.

    • Kurtosis: 2.482.48 vs. 0.100.10 — drops toward baseline normal peakedness as outlier tails are removed.

    • Skewness: 1.061.06 vs. 0.260.26 — shifts from marked positive right skew toward an approximately symmetric profile.

    • Minimum Value: 44 vs. 44

    • Maximum Value: 156156 vs. 108108

    • Sum: 52855285 vs. 48804880

Real-World Applications: Property Market Economics

  • Skewness and Parameter Selection in Real Estate:

    • Housing markets typically display strong positive skewness caused by high-end property transactions, rendering arithmetic mean values misleadingly elevated.

    • Non-parametric positional measures (medians and quartiles) provide reliable representations of market conditions.

Scarborough Real Estate Price Chart Showing Median and Quartiles
  • Longitudinal Real Estate Analysis (Scarborough Market, 2016–2020):

    • Annual settled sales metrics (evaluated 1 April 2019 to 31 March 2020, settled as at 03 August 2020):

    • Upper Quartile (Q3Q_3): $830,000\$830{,}000

    • Median (Q2Q_2): $738,750\$738{,}750

    • Lower Quartile (Q1Q_1): $637,500\$637{,}500

    • Contextual metric: Metropolitan Perth overall median remained substantially below local Scarborough values, ranging near $500,000\$500{,}000

Scarborough Property Price Quartile Breakdown
  • Long-Term Longitudinal Shift (Settled Sales as at 25 July 2024):

    • Re-evaluation of market parameters demonstrates marked capital appreciation and expansion of the interquartile range:

    • Upper Quartile (Q3Q_3): $1,300,000\$1{,}300{,}000 ($1.3M\$1.3\text{M}, 75th75^{\text{th}} percentile)

    • Median (Q2Q_2): $1,060,000\$1{,}060{,}000 ($1.06M\$1.06\text{M}, 50th50^{\text{th}} percentile)

    • Lower Quartile (Q1Q_1): $875,000\$875{,}000 ($875K\$875\text{K}, 25th25^{\text{th}} percentile)

    • Interquartile Range Expansion:

IQR2020=$830,000−$637,500=$192,500IQR_{2020} = \$830{,}000 - \$637{,}500 = \$192{,}500

IQR2024=$1,300,000−$875,000=$425,000IQR_{2024} = \$1{,}300{,}000 - \$875{,}000 = \$425{,}000

  • The increase in IQRIQR indicates broader price divergence across middle-tier properties alongside rising median valuations.