Statistics and Field Experimentation I Study Notes

Meaning and Philosophical Origins of Statistics

• Conceptual Definitions:     * The term "statistics" carries varied meanings depending on the context of the user:         * Environmental Protection Agency (EPA) Administrator: Statistics refer to information regarding the quantity and type of pollutants released into the environment.         * Farm Manager: Statistics consist of daily reports on production, inventory levels, and absenteeism.         * University Student: Statistics are the grades achieved in all courses taken during a specific examination period.     * Broadly, statistics involve the collection of numerical information called Data, the analysis of that data, and the making of meaningful decisions based on that analysis.

• Historical Etymology:     * The word "STATISTIK" originates from the Italian word "STATISTA," which translates to "STATESMAN."     * It was first utilized by Gittifield Achenwall (1719–1772), a Professor at Marlborough and Gottingen.     * It was introduced to England by Dr EAW Ziminarman.     * The term was popularized by Sir John Sinclair in his seminal work titled STATISTICS ACCOUNT OF SCOTLAND (1791–1799).

• Ancient Statistical Practices:     * Quantitative information was in use long before the term was coined. Official government statistics are as old as recorded history.     * The Old Testament contains several accounts of census-taking.     * Ancient Babylonians, Egyptians, Romans, and Israelites gathered detailed information regarding population and resources.

Classifications of Statistical Methods

• Descriptive (Deductive) Statistics:     * This involves the collection of data and the tabulation of results using tables, charts, or graphs to make data manageable and meaningful.     * Examples:         * A Professor computing an average grade for a course taught last semester to describe the class performance.         * Recording daily meteorological data for a weather station (e.g., from Jan–Dec 1999) is an example of Secondary Data (data already collected by someone else).         * Primary Data is data collected directly by a researcher for their specific purpose, such as a researcher going to a farm to collect info on production and practices.

• Inferential (Inductive) Statistics:     * This is a more complex approach used to make predictions about a large group (the Population or Universe) based on a smaller subset (the Sample).     * Collecting data from an entire population (e.g., the average height of all students at the University of Ibadan) is often impossible due to constraints in time, money, and labor. Therefore, a representative sample is selected.     * Generalizations or inferences about the population are made from this sample, provided the sample is truly representative.

Data Types and Variables

• Quantitative Variables:     * Discrete: Variables that take on specific, distinct values (e.g., number of children, number of cars).     * Continuous: Variables that can take any value within a range (e.g., height, weight, temperature).

• Qualitative Variables:     * Nominal: Categories with no inherent order (e.g., gender, eye color).     * Ordinal: Categories with a logical rank or order (e.g., socio-economic status: low, medium, high).

Construction of Frequency Distributions

• The Frequency Distribution is the first step in processing a raw data set. It organizes data into classes and similar categories so that the number of observations in each category can be seen at a glance.

• Case Study: Age at First Marriage of 50 Women in Ibadan:     * Data: 20, 23, 21, 27, 24, 34, 19, 30, 25, 28, 17, 28, 18, 40, 19, 41, 23, 31, 23, 29, 24, 38, 21, 27, 22, 35, 23, 39, 17, 34, 18, 23, 23, 26, 19, 20, 25, 44, 24, 31, 16, 20, 22, 27, 24, 29, 23, 28, 18, 25.     * Formulas for Grouping:         * Range = $Max - Min$         * Range = $44 - 16 = 28$         * Width of Class Interval ($C$) = $\frac{(Max + 1) - Min}{\text{Total Number of Intervals}}$         * With a maximum value of 44 and minimum of 16, $(44 + 1) - 16 = 29$. If using 7 intervals, width is approximately 4.

• Frequency Table Components:     * Relative Frequency ($RF$): Shows the proportion of the distribution in a class interval.     * $RF = \frac{f_i}{\sum f_i}$     * Cumulative Frequency ($CF$): The running total of frequencies through the classes.     * Mid-class (Class Mark): The average of the upper and lower limits of the class interval.

Graphical Representation of Data

• Histogram: A graph of frequency distribution where each class interval is represented by a block/rectangle with no space between bars. Frequency is on the vertical axis, and class intervals are on the horizontal axis.

• Frequency Polygon: A line graph created by connecting the mid-points of the histogram bars. It starts and ends at the horizontal axis (requiring extra classes for extrapolation).

• Cumulative Frequency Curve (Ogive): A graph showing the cumulative totals of frequencies.

• Relative Frequency Curve: Similar in shape to the Ogive but plots relative frequency proportions on the vertical axis.

Skewness and Kurtosis

• Skewness: Measures the asymmetry of the probability distribution.     * Positively Skewed: The tail of the curve tilts toward the right.     * Negatively Skewed: The tail of the curve tilts toward the left.

• Kurtosis: Measures the "peakedness" of the curve. Different curves can have the same central location and dispersion but different degrees of kurtosis.

Measures of Central Tendency

• Arithmetic Mean ($\bar{x}$):     * Ungrouped Data:     * $\bar{x} = \frac{\sum x}{n}$     * Grouped Data:     * $\bar{x} = \frac{\sum f_i x_i}{\sum f_i}$     * Coding/Assumed Mean Method:     * $\bar{x} = X_0 + C \left( \frac{\sum u_i f_i}{n} \right)$     * Where $X_0$ is the class mark assigned the code zero, $C$ is the width of the interval, and $u$ is the assigned code.

• Weighted Mean ($\bar{x}_w$):     * Used to account for the relative importance of each value to the overall total.     * $\bar{x}_w = \frac{\sum w_i x_i}{\sum w_i}$     * Example Application: Calculating average labor cost per hour for different products using skilled and unskilled labor.

• Geometric Mean ($GM$):     * Used for determining the average growth rate or rate of change over time.     * $GM = \sqrt[n]{x_1 \times x_2 \times ... \times x_n}$     * Compound Interest/Growth Formula:     * $P_n = P_0 (1 + r)^n$     * Example: Rate of production increases by $25\%$ in Year 1 and $40\%$ in Year 2. The average growth rate is calculated by finding the $GM$ of the growth factors (1.25 and 1.40).

• Median:     * The middle-most item in an ordered array.     * Grouped Data Formula:     * $\text{Median} = L_m + C \left( \frac{\frac{n}{2} - F_b}{f_m} \right)$     * Where $L_m$ is the lower class limit of the median class, $F_b$ is the cumulative frequency before the median class, and $f_m$ is the frequency of the median class.

• Mode:     * The value that occurs most frequently.     * Grouped Data Formula:     * $\text{Mode} = L_{mo} + C \left( \frac{d_1}{d_1 + d_2} \right)$     * Where $d_1$ is the difference between the modal class frequency and the frequency of the preceding class, and $d_2$ is the difference between the modal class frequency and the frequency of the succeeding class.

Measures of Dispersion

• Distance Measures:     * Range: $Max - Min$. It is limited as it only considers extremes.     * Interfractile Range: Measures the difference between two fractiles (e.g., between the 1st and 2nd thirds of income). Fractiles describe proportions lying above or below a location: deciles (10 parts), quintiles (5 parts), quartiles (4 parts).

• Average Deviation Measures:     * Mean Deviation ($MD$):     * $\text{Ungrouped: } MD = \frac{\sum |x_i - \bar{x}|}{n}$     * $\text{Grouped: } MD = \frac{\sum f_i |x_i - \bar{x}|}{n}$     * Variance ($\sigma^2$):     * $\text{Ungrouped: } \sigma^2 = \frac{\sum (x_i - \bar{x})^2}{n}$     * $\sigma^2 = \frac{\sum x^2 - \frac{(\sum x)^2}{n}}{n}$     * Standard Deviation ($\sigma$): The square root of the variance.

• Coefficient of Variation ($CV$):     * A relative measure of dispersion, used to compare the variability of datasets with different units or scales.     * $CV = \frac{\sigma}{\bar{x}} \times 100\%$     * Higher $CV$ indicates lower efficiency or higher variability.

• Chebyshev's Theorem:     * Devised by P.L. Chebyshev (1821–1894).     * States that regardless of distribution shape, at least $75\%$ of values fall within $\pm 2\sigma$ from the mean, and at least $89\%$ fall within $\pm 3\sigma$.     * For bell-shaped (Normal) curves: $68\%$ within $\pm 1\sigma$, $95\%$ within $\pm 2\sigma$, and $99\%$ within $\pm 3\sigma$.

Probability Theory and Set Operations

• Foundations: Facilitated by set theory, developed by GEORGI CANTOUK (1845).

• Set Definitions:     * Subset: Set A is a subset of B ($A \subset B$) if all elements in A are in B.     * Proper Subset: B contains at least one element not in A.     * Universal Set ($U$): The set containing all objects of interest.     * Empty/Null Set ($\phi$): A set with no elements.

• Set Operations:     * Union ($A \cup B$): Contains all elements in A, B, or both.     * Intersection ($A \cap B$): Elements common to both A and B.     * Disjoint Sets: Sets with no common elements ($A \cap B = \phi$).     * Complement ($A'$): Elements in the Universal set that are not in A.

• Laws of Set Operations:     * Idempotent ($A \cup A = A$), Commutative ($A \cup B = B \cup A$), Associative, Distributive, De Morgan’s Law ($(A \cup B)' = A' \cap B'$).

Concepts of Probability

• Definitions:     * Experiment: A process like tossing a coin or rolling a die.     * Sample Space ($S$): Set of all possible outcomes (e.g., for two coins, $S = \{HH, HT, TH, TT\}$).     * Event: A subset of the sample space.

• Types of Probability:     * Subjective: Based on personal belief, evidence, or educated guesses.     * Objective: Divided into Classical ($apriori$) and Relative Frequency.         * Classical: $P(E) = \frac{n(E)}{n(S)}$. Outcomes are equally likely.         * Relative Frequency: Probability determined by the proportion of times an event occurs in the long run.

• Probability Rules:     * $- 0 \le P(E) \le 1$     * - If $P(E) = 0$, the event is impossible. If $P(E) = 1$, it is certain.     * - $P(\text{success}) + P(\text{failure}) = p + q = 1$.

• Event Relationships:     * Mutually Exclusive: Events that cannot occur simultaneously ($A \cap B = \phi$).     * Independent: The occurrence of one does not affect the other ($P(A \cap B) = P(A) \times P(B)$).     * Conditional Probability: The probability of B occurring given that A has already occurred.     * $P(B|A) = \frac{P(A \cap B)}{P(A)}$

• Bayes' Theorem:     * Deals with revising prior probabilities after additional information (likelihoods) becomes available, resulting in posterior probabilities.     * $P(A_i | B) = \frac{P(A_i) P(B | A_i)}{\sum P(A_j) P(B | A_j)}$

Combinatorial Analysis and Distributions

• Fundamental Principle: If one event can occur in $n_1$ ways and another in $n_2$ ways, both can occur in sequence in $n_1 \times n_2$ ways.

• Permutations ($nP_r$): Arranging $r$ objects out of $n$ where order matters.     * $nP_r = \frac{n!}{(n-r)!}$     * With Repetition: $\frac{n!}{n_1! n_2! ... n_k!}$ (Example: arrangements of the word "STATISTICS").

• Combinations ($nC_r$): Selection where order does not matter.     * $nC_r = \frac{n!}{r!(n-r)!}$

• Binomial (Bernoulli) Distribution:     * Discovered by James Bernoulli. Used for independent trials with two outcomes (success/failure).     * $P(x) = nC_x p^x q^{n-x}$     * Mean = $np$, Variance = $npq$.

• Poisson Distribution:     * Discovered by S.D. Poisson. Used for "rare events" where $n$ is large and $p$ is small ($np$ stays stable).     * $P(x) = \frac{\lambda^x e^{-\lambda}}{x!}$     * Where $\lambda = np$. Mean and Variance both equal $\lambda$.

• Normal Distribution:     * A continuous distribution represented by the bell-shaped curve.     * $f(x) = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2} \left( \frac{x - \mu}{\sigma} \right)^2}$     * Standardized variable: $z = \frac{x - \mu}{\sigma}$.

Sampling Theory and Techniques

• Population vs Sample:     * Population: All possible observations of a type. Measures are called Parameters (e.g., $\mu, \sigma^2$).     * Sample: A subset of the population. Measures are called Statistics (e.g., $\bar{y}, s^2$).

• Sampling Methods:     1. Convenience Sampling: Using readily available observations.     2. Judgment Sampling: Sampler deliberately selects items based on knowledge.     3. Random Sampling: Every element has an equal chance of selection.         * Simple Random: Ballot method or Table of random numbers.         * Systematic: Selection at regular intervals. Raising factor $k = \frac{N}{n}$.         * Stratified: Divide population into strata (low internal difference, high external difference) then sample randomly from each.         * Cluster: Divide population into heterogeneous blocks (clusters) then select clusters and sample within them (two-stage).

• Basic Estimators in Sampling:     * Sample Mean: $\bar{y} = \frac{\sum y_i}{n}$.     * Standard Error of the Mean ($SE_{\bar{y}}$):         * $V(\bar{y}) = (1 - f) \frac{s^2}{n}$, where $f = \frac{n}{N}$ is the sampling fraction.         * The Finite Population Correction (FPC) $(1 - f)$ is ignored if $f \le 0.05$.     * Confidence Interval for Mean: $\bar{y} \pm t \cdot SE_{\bar{y}}$.     * Proportions: $p = \frac{a}{n}$. Variance of proportion $V(p) = \frac{pq}{n-1}$.

Principles of Hypothesis Testing

• Definitions:     * Statistical Hypothesis: A numerical statement about a population parameter that needs verification.     * Null Hypothesis ($H_0$): Stated in the negative or "no difference" form (e.g., "There is no difference in achievement between boys and girls").     * Research/Alternative Hypothesis ($H_a$): Predicts an outcome in a positive form or specifies a difference.

• Errors in Testing:     * Type I Error: Rejecting $H_0$ when it should have been accepted ($P(\text{Type I}) = \alpha$).     * Type II Error: Accepting $H_0$ when it should have been rejected.

• Testing Parameters:     * Level of Significance ($\alpha$): Usually $0.05$ ($5\%$) or $0.01$ ($1\%$).     * One-tailed Test: directional (e.g., "New drug is better").     * Two-tailed Test: non-directional (e.g., "There is a difference").

• Z-Tests (Large Samples, $n \ge 30$):     * Single Mean: $z = \frac{\bar{x} - \mu}{\sigma/\sqrt{n}}$.     * Difference of Means: $z = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}}$.

• T-Tests (Small Samples, n < 30):     * Requires Degrees of Freedom ($df = n - 1$ or $n_1 + n_2 - 2$).     * Formula for difference of means uses pooled variance ($s^2$).

Chi-Square ($\chi^2$) Analysis

• Measures the discrepancy between Observed ($O$) and Expected ($E$) frequencies.     * $\chi^2 = \sum \frac{(O_i - E_i)^2}{E_i}$     * $df = k - 1$ (for one-way) or $(row - 1)(column - 1)$ (for contingency tables).

• Yate's Correction: Applied when $df = 1$.     * $\chi^2 = \sum \frac{(|O_i - E_i| - 0.5)^2}{E_i}$

Correlation and Regression

• Correlation ($r$): Investigates the degree to which variables move together.     * Positive: Both move in the same direction.     * Negative: Move in opposite directions.     * Zero: No linear relationship.     * Spearman Rank Correlation ($r_s$): $r_s = 1 - \frac{6 \sum d^2}{n(n^2 - 1)}$.

• Regression: Measures the amount of change in a dependent variable ($y$) caused by a unit change in an independent variable ($x$).     * Model: $y = a + bx + e$     * $b = \text{Regressor coefficient/Slope}$.     * $a = \text{Intercept/Constant (value of y when x = 0)}$.

Analysis of Variance (ANOVA)

• Derived by Sir Ronald Fisher (1923), the F-ratio ($F$) is used to compare three or more means simultaneously.     * $F = \frac{\text{Variance between groups}}{\text{Variance within groups}}$

• ANOVA Designs:     1. One-way (Completely Randomized Design - CRD): Tests effects of one factor.     2. Two-way (Randomized Complete Block Design - RCBD): Adds a second factor (blocking/education) to reduce error.         * With Replications: Allows testing of the interaction term between factors.     3. Latin Square Design: Efficient when there are two sources of external variation (e.g., fertility gradients in two directions). Each treatment appears once in each row and column.

• The ANOVA Table: Summarizes Source, Sum of Squares ($SS$), Degrees of Freedom ($df$), Mean Squares ($MS$), and the Calculated F-ratio ($F_{cal}$) against Critical F values ($F_{tab}$).