Comprehensive CBSE Grade 10 Data Science Handbook

Foundations of Data Science Curriculum and Framework

• Curriculum Context and Objectives:

  • Data Science is introduced by the Central Board of Secondary Education (CBSE) in collaboration with Microsoft India as a skill module of $12\text{ hours}$ duration in Class VIII and as a formal skill subject across Classes IX through XII.
  • The Class X Student Handbook (Version 1.0) builds a foundational understanding of data science, focusing on how data is systematically collected, analyzed, visualized, and utilized for problem-solving and decision-making.
  • It covers core theoretical concepts, practical examples, distribution models, pattern recognition, data merging, ethical data governance, and prerequisites for Artificial Intelligence (AI) applications.

• Curriculum Structure and Core Units:

  • Unit 1: Use of Statistics in Data Science (Subsets, Two-Way Tables, Mean, Median, Mean Absolute Deviation, Standard Deviation).
  • Unit 2: Distributions in Data Science (Probability Distributions, Discrete and Continuous Data, Statistical Problem-Solving Process).
  • Unit 3: Identifying Patterns (Biases in Data, Probability for Statistics, The Central Limit Theorem).
  • Unit 4: Data Merging (Relational Joins, Z-Scores, Percentiles, Quartiles, Deciles).
  • Unit 5: Ethics in Data Science (Data Governance, Ethical Guidelines, Secure Discarding of Digital and Physical Data).

Statistical Subsetting in Data Science

• Definition and Purpose of Subsetting:
  • A Subset is a smaller division or portion of data extracted from a larger dataset or data frame.
  • When working with massive quantities of raw data, the entire dataset is rarely required for every analytical task. Subsetting filters out extraneous variables and observations, allowing focused analysis on only the relevant elements.
  • In data analytics, subsetting functions as an indexing mechanism to access, retrieve, select, and filter specific object elements from a data frame.
  • Example: In a data table consisting of $100\text{ rows}$ and $100\text{ columns}$, isolating the first $5\text{ rows}$ and the first $5\text{ columns}$ creates a $5 \times 5$ table known as a subset.

• Core Methodologies for Subsetting Data:
  • Row-Based Subsetting:
  • Involves selecting specific rows from the upper (top) or lower (bottom) sections of a data table based on position or rank.
  • Example: Extracting the top $3\text{ rows}$ from an existing table containing $6\text{ rows}$ and $4\text{ columns}$.
  • Column-Based Subsetting:
  • Involves isolating specific vertical columns (variables or features) from a dataset containing many fields, ignoring unnecessary dimensions.
  • Used when specific attributes are required for a particular computation while others remain irrelevant.
  • Data-Based Subsetting:
  • Involves selecting rows and observations based on conditional logic, attributes, or predefined filtering criteria.
  • Example: Querying an automobile database to extract only records where the attribute Color equals Red.

Two-Way Frequency and Relative Frequency Tables

• Structure and Purpose of Two-Way Frequency Tables:

  • A Two-Way Frequency Table is a statistical matrix that displays the observed counts (frequencies) for two distinct categorical variables simultaneously.
  • One categorical variable defines the horizontal rows, while the second categorical variable defines the vertical columns.
  • Each internal cell demonstrates the intersectional frequency of data points falling into both specific categories.
  • Standard components include:
  • Categories listed in the far-left column and the uppermost row.
  • Intersectional counts recorded within the inner cells.
  • Marginal totals at the end of every row and column.
  • Grand total (sum of all recorded observations) positioned at the bottom-right corner.

• Worked Example: Chocolate Preference Survey:

  • Raw poll data collected from $10\text{ individuals}$ across three age brackets ($5-10\text{ years}$, $10-15\text{ years}$, and $15-20\text{ years}$):
  • Person 1 ($6\text{ years}$): Yes
  • Person 2 ($8\text{ years}$): Yes
  • Person 3 ($13\text{ years}$): Yes
  • Person 4 ($12\text{ years}$): Yes
  • Person 5 ($18\text{ years}$): No
  • Person 6 ($9\text{ years}$): No
  • Person 7 ($16\text{ years}$): Yes
  • Person 8 ($19\text{ years}$): No
  • Person 9 ($14\text{ years}$): No
  • Person 10 ($12\text{ years}$): Yes
  • Two-Way Table aggregation:
  • Age $5 - 10$: $2\text{ Like}$, $1\text{ Dislike}$
  • Age $10 - 15$: $3\text{ Like}$, $1\text{ Dislike}$
  • Age $15 - 20$: $1\text{ Like}$, $2\text{ Dislike}$
  • Interpretative metrics:
  • Total individuals surveyed: $10$
  • Total individuals who like chocolates: $2 + 3 + 1 = 6$
  • Age group with the highest chocolate preference: $10 - 15\text{ years}$ ($3\text{ individuals}$)

• Movie Genre Preference Survey ($80\text{ individuals}$):

  • Survey parameters: $40\text{ men}$ and $40\text{ women}$.
  • Observed preferences:
  • Men: $8\text{ Comedy}$, $18\text{ Action}$, $14\text{ Horror}$ (Total = $40$)
  • Women: $23\text{ Comedy}$, $10\text{ Action}$, $7\text{ Horror}$ (Total = $40$)
  • Summary totals:
  • Comedy Total = $31$, Action Total = $28$, Horror Total = $21$
  • Grand Total = $80$

• Automobile Ownership Survey ($120\text{ individuals}$):
  • Men: $35\text{ Own a car}$, $25\text{ Do not own a car}$ (Total = $60$)
  • Women: $40\text{ Own a car}$, $20\text{ Do not own a car}$ (Total = $60$)
  • Column totals: $75\text{ Own a car}$, $45\text{ Do not own a car}$; Grand Total = $120$.

• Two-Way Relative Frequency Tables:
  • Converts raw numerical frequencies into percentages (or proportions) of data points fitting each category.
  • Can be constructed using row relative frequencies, column relative frequencies, or total relative frequencies depending on analytical focus.
  • Especially useful for comparing group preferences when sample sizes are unequal.
  • Sports Preference Example ($100\text{ Girls}$, $100\text{ Boys}$):
  • Raw counts: Indoor Games (Girls: $70$, Boys: $20$); Outdoor Games (Girls: $30$, Boys: $80$).
  • Column Relative Frequency Table:
    • Girls: Indoor $70\%$, Outdoor $30\%$, Total $100\%$
    • Boys: Indoor $20\%$, Outdoor $80\%$, Total $100\%$

Measures of Central Tendency: Mean and Median

• Arithmetic Mean:
  • A primary measure of central tendency representing the simple average value around which numeric data is spread.
  • Every single data point is weighted equally in the calculation.
  • Mathematical definition:     $\text{Mean} = \frac{\sum_{i=1}^n x_i}{n}$
  • Discrete Example: Sequence of integers from $10$ to $20$ ($11\text{ numbers}$ total: $10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20$):     $\text{Sum} = 165$$\text{Mean} = \frac{165}{10} = 16.5$
  • Real-World Application (City Temperatures):
  • Recorded minimum temperatures: Mumbai ($21^\circ\text{C}$), Delhi ($13^\circ\text{C}$), Chennai ($24^\circ\text{C}$), Kolkata ($15^\circ\text{C}$), Bangalore ($20^\circ\text{C}$).     $\text{Mean} = \frac{21 + 13 + 24 + 15 + 20}{5} = \frac{93}{5} = 18.6^\circ\text{C}$

• Median:
  • The exact middle point of a data set that has been arranged in ascending or descending numerical order.
  • Computation depends on the sample size ($n$):
  • Odd number of observations: The median is the single value located at position $\frac{n + 1}{2}$.
    • Example: Array $[12, 34, 56, 89, 32]$ sorted becomes $[12, 32, 34, 56, 89]$. Position $3$ holds the median: $34$.
  • Even number of observations: The median is the arithmetic average of the two central numbers at positions $\frac{n}{2}$ and $\frac{n}{2} + 1$.
    • Example: Array $[12, 34, 56, 89, 32, 48]$ sorted becomes $[12, 32, 34, 48, 56, 89]$.
    • Central pair: $34$ and $48$.       $\text{Median} = \frac{34 + 48}{2} = \frac{82}{2} = 41$

• Mean vs. Median in the Presence of Outliers:
  • The arithmetic mean is sensitive to extreme values (outliers or erroneous data), which pull the center away from typical values.
  • The median is resistant (robust) to outliers, making it a more accurate descriptor of central tendency when datasets contain skewed or faulty readings.
  • Blood Pressure Example:
  • Weekly systolic blood pressure recordings: $141, 143, 147, 210, 146$ (where $210$ is an error value caused by a faulty device).
  • Sorted values: $141, 143, 146, 147, 210$.
  • Computation comparison:       $\text{Mean} = \frac{141 + 143 + 147 + 210 + 146}{5} = \frac{787}{5} = 157.4$$\text{Median} = 146$
  • Conclusion: The mean deviates significantly due to the single machine error, whereas the median ($146$) accurately represents normal central pressure.

Measures of Variability: Mean Absolute Deviation and Standard Deviation

• Mean Absolute Deviation (MAD):

  • The arithmetic average of the absolute distances between each individual data point and the mean of the dataset.
  • Quantifies dispersion, scattering, and variability without regard to the direction (sign) of deviations.
  • Step-by-Step Procedure:
  1. Calculate the arithmetic mean ($\bar{x}$) of the data set.
  2. Subtract the mean from each data point and take the absolute value: $|x_i - \bar{x}|$.
  3. Calculate the mean of these absolute differences.
  • Worked Example: Dataset $[12, 16, 10, 18, 11, 19]$
  • Step 1: Calculate Mean:       $\bar{x} = \frac{12 + 16 + 10 + 18 + 11 + 19}{6} = \frac{86}{6} \approx 14$
  • Step 2: Compute Absolute Distances:
    • $|12 - 14| = |-2| = 2$
    • $|16 - 14| = |2| = 2$
    • $|10 - 14| = |-4| = 4$
    • $|18 - 14| = |4| = 4$
    • $|11 - 14| = |-3| = 3$
    • $|19 - 14| = |5| = 5$
    • Total sum of distances = $2 + 2 + 4 + 4 + 3 + 5 = 20$
  • Step 3: Compute Mean Absolute Deviation:       $\text{MAD} = \frac{20}{6} \approx 3.33$

• Standard Deviation ($\sigma$):

  • The primary metric measuring how spread out numbers are around their arithmetic mean.
  • Indicates whether individual data points cluster tightly near the average or disperse widely above and below it.
  • Five-Step Calculation Process:
  1. Calculate the Mean ($\mu$): Sum all values and divide by total number of values ($n$).
  2. Calculate Deviations: Subtract the mean from each observation ($x_i - \mu$).
  3. Square the Differences: Square each deviation: $(x_i - \mu)^2$.
  4. Calculate Variance ($\sigma^2$): Compute the average of the squared deviations:        $\sigma^2 = \frac{\sum (x_i - \mu)^2}{n}$
  5. Calculate Standard Deviation ($\sigma$): Take the principal square root of the variance:        $\sigma = \sqrt{\sigma^2}$
  • Worked Example: Dataset $[1, 2, 3, 5, 8]$
  • Step 1 (Mean): $\mu = \frac{1 + 2 + 3 + 5 + 8}{5} = \frac{19}{5} = 3.8$
  • Step 2 (Deviations):       $1 - 3.8 = -2.8$$2 - 3.8 = -1.8$$3 - 3.8 = -0.8$$5 - 3.8 = 1.2$$8 - 3.8 = 4.2$
  • Step 3 (Squared Deviations):       $(-2.8)^2 = 7.84$$(-1.8)^2 = 3.24$$(-0.8)^2 = 0.64$$(1.2)^2 = 1.44$$(4.2)^2 = 17.64$
  • Step 4 (Variance):       $\text{Sum of Squares} = 7.84 + 3.24 + 0.64 + 1.44 + 17.64 = 30.8$$\sigma^2 = \frac{30.8}{5} = 6.16$
  • Step 5 (Standard Deviation):       $\sigma = \sqrt{6.16} \approx 2.48$

• Real-World Applications of Standard Deviation:
  • Grading Tests: Evaluates whether student performance across an academic class is uniform (low $\sigma$) or shows extreme disparities in comprehension (high $\sigma$).
  • Survey Sampling Reliability: Assesses consistency and dispersion within collected survey responses to gauge confidence when extrapolating to larger general populations.
  • Weather Forecasting: Analyzes temperature ranges across regions; lower standard deviations indicate stable, highly predictable weather conditions.

Distribution Variability Analysis: The Family Size Activity

• Investigative Setup:

  • Objective: Determine the average household family size of students within a school.
  • Statistical Questioning Parameters:
  • Intended Population: Households of all students in a teacher's classroom.
  • Variable of Interest: Number of individuals residing in a single household.
  • Anticipated Variability: Household sizes naturally differ across students.

• Data Collection and Physical Modeling (Snap Cubes):

  • A teacher selects $5\text{ students}$ who physically model their family size using snap cubes.
  • Raw collected sizes: $3, 2, 4, 2, 4$ (Total cubes = $15$).

• Data Analysis and Equal Share Concept:
  • The stacks are arranged in ascending order: $2, 2, 3, 4, 4$.

• Two equivalent conceptual methods establish an "equal share" (fair share / arithmetic mean):
  1. Disconnection and Redistribution: Disconnect all $15\text{ cubes}$ and distribute them evenly one-by-one across the $5\text{ students}$, yielding $5\text{ stacks}$ of $3\text{ cubes}$.
  2. Step-by-Step Leveling: Remove a cube from the tallest stack ($4$) and shift it to the lowest stack ($2$).

• Continuing this process produces $5\text{ uniform stacks}$ of $3\text{ cubes}$.

• Conclusion: The equal share value of $3$ represents the arithmetic mean of the distribution ($\frac{15}{5} = 3$).

  • Investigating Variability Beyond the Mean:

• Two distinct groups of $5\text{ students}$ each possess an identical equal share value of $6$ (requiring $30\text{ total cubes}$ each):

  • Group 1 Configuration: Stacks of $6, 6, 7, 4, 7$.
  • Group 2 Configuration: Stacks of $6, 6, 8, 4, 6$.

• Evaluating "Closeness to Equality":
  • Group 2 exhibits the highest frequency of stacks already at the target equal share of $6$ ($3\text{ stacks}$ vs. $2\text{ stacks}$ in Group 1).
  • Alternatively, evaluating the metric "number of steps to equal":
  • Group 1: Two stacks have $7$ ($1\text{ cube}$ excess each). Shifting these $2\text{ cubes}$ into the stack of $4$ requires moving $2\text{ cubes}$ a single step each ($2\text{ steps total}$).
  • Group 2: One stack has $8$ ($2\text{ cubes}$ excess). Shifting both cubes into the stack of $4$ requires moving $2\text{ cubes}$ a total distance of two units ($2\text{ steps total}$).
  • Conclusion: Both distributions possess equal variability from their mean despite differing structural distributions.

Probability Distributions and the Problem-Solving Process

• Definition of Distribution in Data Science:

  • A Distribution is a statistical representation showing all possible values for a variable alongside their probabilities of occurrence.
  • While mathematical probability computes precise theoretical likelihoods, distributions allow analysts to map, visualize, and model data behavior across an entire sample space.
  • Exhaustive distribution rule: The sum of probabilities for all mutually exclusive possible outcomes must equal $1.0$ ($100\%$). Any value outside this defined sample space has an occurrence probability of zero.

• Uniform Distribution Models:

  • A distribution where every possible outcome has an identical, constant probability of occurring.
  • Single Coin Toss Example: Sample space $S = \{\text{Head}, \text{Tail}\}$.
  • $P(\text{Head}) = 0.5$
  • $P(\text{Tail}) = 0.5$
  • $P(\text{Any other outcome}) = 0$

• Two Coin Toss Example: Sample space $S = \{\text{Head-Head}, \text{Head-Tail}, \text{Tail-Head}, \text{Tail-Tail}\}$.
  • $P(\text{Head-Head}) = 0.25$
  • $P(\text{Head-Tail}) = 0.25$
  • $P(\text{Tail-Head}) = 0.25$
  • $P(\text{Tail-Tail}) = 0.25$
  • Total sum of probabilities = $0.25 \times 4 = 1.00$

• Classification of Data Types:

  • Discrete Data: Can assume only specific, isolated values (often integers or distinct categories). No intermediate fractional values exist between distinct points.
  • Examples: Exam outcomes (pass or fail), number of bedrooms in a house, defects in a production line.
  • Continuous Data: Can assume any real numerical value within a given finite or infinite continuum or measurement scale.
  • Examples: Depth of an ocean, body weight of a person, vehicle speed, length of a road.

• The Four Components of the Statistical Problem-Solving Process:

1. Formulate Statistical Investigative Questions (Anticipating Variability):
   • Must anticipate variability across a population or group rather than seeking a single deterministic value for an individual.
   • Contrast: "How tall is that specific plant?" is deterministic (data-gathering). "Do plants exposed to more sunlight grow faster?" is a valid statistical investigative question.
   • Criteria for a sound investigative question:
     • Variables of interest are explicitly defined.
     • Focused population group is unambiguous.
     • Intent is defined (describing data, comparing groups, or evaluating variable association).
     • Purposeful and answerable via primary or secondary empirical data.
2. Collect / Consider the Data (Acknowledging Variability):
   • Designs data collection mechanisms that minimize bias and account for natural variability.
   • Utilizes techniques such as Statistical Process Control, random sampling, and Design of Experiments (DoE).
   • Random assignment of subjects to treatment groups balances confounding factors that are not directly controlled.
3. Analyze the Data (Accounting for Variability):
   • Employs graphical displays, comparative plots (dot plots, box plots), and numerical summaries to quantify spread, overlap, and shape.
   • Example: Comparing cricket batting distributions between the Indian and Australian national teams via comparative dot plots to evaluate overlap and separation.
4. Interpret the Results (Allowing for Variability):
   • Interprets statistical findings while explicitly recognizing the underlying sources of error and variability (such as individual variation and sampling error).
   • Bounds conclusions and generalizations within established confidence constraints.

Multivariate Categorical Data Analysis: School Dance Program Activity

• School-Wide Music Preference Investigation:

  • Scenario: Selecting musical genres and entertainment for an annual school dance program.
  • Research question: What type of music do the students in our grade/school like?
  • Data collection sampling strategy: While a single class census (e.g., Grade 10 English) can serve as an exploratory sample, broader sampling (randomly selecting students across all homerooms) prevents sampling bias.

• Multivariate Survey Questionnaire Design:

  • Question 1: Check "Yes" or "No" for liking each music type (Classical, Rap, Bollywood, Pop, Rock).
  • Question 2: What is your single most favorite type of music?
  • Question 3: What is your second favorite type of music?
  • Question 4: Would you prefer a Live Band or a DJ?

• Graphical Analysis of Survey Findings:

  • Genre Acceptance (Question 1): Displayed via side-by-side comparative bar charts across grades ($8, 9, 10$).
  • Across the aggregate sample, Rock generated the highest absolute affirmative responses, followed closely by Rap and Classical.

• Primary Preference (Question 2): Single-variable bar charts showed Bollywood receiving the highest single-genre count ($53\text{ votes}$), while Classical and Rock tied at $43\text{ votes}$ each, with Rap and Pop recording $25\text{ votes}$ each.

• Joint Preference Bivariate Analysis (Questions 2 and 3):
  • Mapped on a $6 \times 6$ two-way dot plot featuring $36\text{ discrete intersecting bins}$.
  • Each dot represents an individual student respondent.
  • Top-left bin contains $2\text{ dots}$: Students whose primary choice is Classical and secondary choice is Rock.
  • Key finding: $17\text{ students}$ (shaded lighter) selected Bollywood, Pop, or Rock in their top two choices. Only $2\text{ students}$ (shaded darker) selected options outside these three genres.

• Format Selection (Question 4):
  • Vertical dot plot comparing preferences for a Live Band vs. a DJ.
  • Results revealed exactly identical voter counts ($8\text{ votes}$ each), yielding a net preference difference of zero.

Identifying Patterns, Biases, and Empirical Probability

• Nature and Genesis of Bias in Data Science:

  • Bias is a systematic deviation of results or inferences from the expected ground truth or true population parameter.
  • Fundamentally represents an indistinct, often unnoticed error embedded within data pipelines.
  • Root causes: Incomplete sampling, non-random estimation, and reliance on "found data" (data collected for operational or administrative purposes rather than controlled scientific modeling).
  • Dangers of biased data:
  • Machine learning models know only the data provided during training.
  • Skewed inputs corrupt model accuracy, undermine fidelity, and institutionalize algorithmic discrimination against demographic subgroups.

• Taxonomy of Cognitive and Statistical Biases:

  • Selection Bias:
  • Occurs when collected sample data fails to accurately represent the true population the model encounters in production.
  • Highly prevalent in recommender systems, user polling, and personalized advertisements where data is captured only from active users who interact with displayed content.
  • Linearity Bias:
  • A cognitive distortion assuming that a change in one variable produces a directly proportional and constant change in another.
  • Arises from human perceptual tendency to project simple linear trends onto non-linear real-world phenomena.
  • Confirmation Bias (Observer Bias):
  • Occurs when researchers or data labelers allow preconceived subjective hypotheses to dictate how observations are filtered, categorized, or evaluated.
  • Results in data labelers misclassifying ambiguous records to support predetermined outcomes.
  • Recall Bias:
  • A measurement bias common during data labeling where identical or highly similar data points are labeled inconsistently across time.
  • Example: A hardware inspection team categorizes identical structural damage on laptop casings inconsistently across "damaged" and "partially damaged" tags.
  • Survivorship Bias:
  • Occurs when datasets focus exclusively on successful entities ("survivors") while entirely overlooking failed entities that exited the observation window.
  • Example: Modeling business success factors by analyzing only surviving corporations while ignoring bankrupt or liquidated enterprises.

• Differentiating Theoretical Probability from Empirical Statistics:

  • Problem 1 (Theoretical Mathematical Probability):
  • Setup: Assume a fair coin ($P(\text{Tail}) = 0.5$) is tossed $10\text{ times}$.
  • Question: How many times will tails appear?
  • Approach: Logically deduces the exact probability distribution across discrete possible counts ($0, 1, 2, \dots, 10$) using deductive combinatorial probability.
  • Problem 2 (Empirical Statistical Inference):
  • Setup: An unknown physical coin is picked up.
  • Question: Is this coin fair?
  • Approach: Conducts empirical experiments ($100\text{ trials}$ of $10\text{ tosses}$ each). Computes observed frequencies and compares them to theoretical binomial predictions.
  • If observed variance departs significantly from theoretical models beyond chance variation, the coin is inferred to be biased.

The Central Limit Theorem and Sampling Distributions

• Core Theorem Statement:

  • The Central Limit Theorem (CLT) states that given a population with mean $\mu$ and finite variance $\sigma^2$, the sampling distribution of the sample mean ($\bar{x}$) approaches a normal distribution as the sample size ($n$) increases, regardless of the underlying shape of the population distribution.

• Foundational Principles and Criteria:

  • Sample Size Threshold: Sample sizes greater than or equal to $30$ ($n \ge 30$) are considered sufficiently large for the normal approximation to hold reliably.
  • Expected Value of the Sample Mean: The mean of the sampling distribution of sample means ($\mu_{\bar{x}}$) is mathematically identical to the true population mean ($\mu$):     $\mu_{\bar{x}} = \mu$
  • Standard Error of the Mean ($\sigma_{\bar{x}}$\text{ or }$SE$): The standard deviation of the sampling distribution equals the population standard deviation divided by the square root of $n$:     $\sigma_{\bar{x}} = \frac{\sigma}{\sqrt{n}}$
  • As sample size increases, the standard error shrinks, causing sample means to cluster tightly around the true population mean.

• Conceptual Illustration (Neighborhood Weight Study):

  • Target: Calculate the average weight of residents across $50\text{ houses}$ ($5\text{ residents each}$, total population = $250$).
  • Measuring all individuals is labor-intensive and impractical at scale.
  • CLT Workaround Procedure:
  1. Draw multiple random samples of size $n \ge 30$.
  2. Calculate the individual mean ($\bar{x}$) for each sample.
  3. Calculate the mean of these sample averages.
  4. A histogram of these sample means forms a symmetrical, bell-shaped normal distribution centered directly on the true population mean.

• Mathematical Worked Problem (Indian Male Weight Distribution):

  • Given population parameters: $\mu = 68\text{ kg}$, $\sigma = 10\text{ kg}$.
  • Sample parameters: $n = 50\text{ males}$.
  • Sample Mean determination:
  • Since $n = 50 > 30$, the sample mean equals the population mean:       $\mu_{\bar{x}} = 68\text{ kg}$
  • Sample Standard Deviation (Standard Error) determination:     $\sigma_{\bar{x}} = \frac{\sigma}{\sqrt{n}} = \frac{10}{\sqrt{50}} = \frac{10}{7.071} \approx 1.41\text{ kg}$

• Practical Significance of the Central Limit Theorem:

  • Election Polling: Allows pollsters to survey a small fraction of voters and project general population voting intentions within bounded confidence intervals.
  • Socio-Economic Surveys: Enables accurate estimation of regional mean family incomes and poverty indices without performing an exhaustive census.

Relational Data Merging and Database Joins

• Overview of Data Merging:

  • Data Merging is the process of integrating two or more discrete datasets or data frames into a single unified analytical dataset.
  • Required when raw variables are distributed across multiple functional tables, physical spreadsheets, or distinct software systems.
  • Common operational hurdles:
  • Discrepant naming conventions and inconsistent column headers.
  • Heterogeneous data formatting (e.g., modern databases versus legacy flat files).
  • Divergent timeframes, reporting cadences, and differing data collection objectives.

• Relational Database Joins:

  • One-to-One Join:
  • Each record in the primary table corresponds to exactly one record in the secondary table, linked via a unique common key.
  • Example: Corporate HR System:
    • Employees table contains primary key Employee ID alongside names.
    • Contact Info table contains foreign key Employee ID alongside phone numbers and addresses.
    • A one-to-one join matches records where Employees.Employee ID $=$ Contact Info.Employee ID.

• One-to-Many Join:
  • A single record in the primary table links to multiple related records in the secondary table.
  • Example: School Library System:
  • Students table has primary key Student ID (unique for each student).
  • Library table records book borrowings; a single Student ID can appear multiple times alongside distinct Book ID records.
  • Merging retrieves all library checkouts associated with an individual student.

• Many-to-Many Join:
  • Multiple records in one table relate to multiple records in another table.
  • Example: Students and Academic Courses (a student enrolls in multiple courses; a course contains multiple students).
  • Structural Resolution via a Join Table:
  • Direct joins are prone to data anomalies; a third intermediary table (join table) converts the relationship into two separate one-to-many joins.
  • In a course registration database, Enrollments acts as the join table.
  • Primary tables: Students (Student ID primary key) and Courses (Course ID primary key).
  • The Enrollments table contains composite foreign keys (Student ID and Course ID) alongside its own unique identifier (Enrollment ID).

Relative Standing: Z-Scores, Percentiles, Quartiles, and Deciles

• The Standard Score (Z-Score):

  • Quantifies the position of a raw score by measuring its distance from the mean in standard deviation units.
  • Standardizing converts any normal distribution into a standard normal distribution characterized by $\mu = 0$ and $\sigma = 1$.
  • Mathematical definition:     $Z = \frac{x - \mu}{\sigma}$
  • Where $x = \text{raw score}$, $\mu = \text{population mean}$, and $\sigma = \text{population standard deviation}$.
  • For sample data with unknown population parameters, the sample mean ($\bar{x}$) and sample standard deviation ($s$) are substituted.
  • Interpretation of Z-Score signs and magnitudes:
  • $Z = 0$: The raw score is identical to the mean.
  • $Z > 0$ (Positive): The score lies above the mean (e.g., $Z = +2.0$ indicates $2\text{ standard deviations}$ above the mean).
  • $Z < 0$ (Negative): The score lies below the mean (e.g., $Z = -2.5$ indicates $2.5\text{ standard deviations}$ below the mean).
  • Worked Example (Child Weight Distribution: $\mu = 10\text{ kg}$, $\sigma = 2\text{ kg}$):
  • Find Z-score for $x = 12\text{ kg}$:       $Z = \frac{12 - 10}{2} = \frac{2}{2} = +1.0$
  • Find Z-score for $x = 5\text{ kg}$:       $Z = \frac{5 - 10}{2} = \frac{-5}{2} = -2.5$
  • Find weight ($x$) corresponding to $Z = +1.25$:       $1.25 = \frac{x - 10}{2} \implies 2.5 = x - 10 \implies x = 12.5\text{ kg}$

• Percentiles:

  • The $p^{\text{th}}$ percentile represents the value such that $p\%$ of ordered observations fall at or below it.
  • The maximum value in any data series represents the $100^{\text{th}}$ percentile.
  • Computation Procedure:
  • Dataset: $[10, 12, 15, 17, 13, 22, 16, 23, 20, 24]$ ($n = 10$).
  • Step 1: Sort in ascending order: $[10, 12, 13, 15, 16, 17, 20, 22, 23, 24]$.
  • Step 2: Determine count of observations at or below target element ($22$):
    • Values at or below $22$: $10, 12, 13, 15, 16, 17, 20, 22$ ($8\text{ values}$ total).
  • Step 3: Compute percentile:       $\text{Percentile} = \frac{8}{10} \times 100 = 80^{\text{th}}\text{ percentile}$

• Quartiles and the Interquartile Range (IQR):

  • Quartiles partition ordered datasets into four equal parts ($25\%$ intervals):
  • First Quartile ($Q_1$): $25^{\text{th}}$ percentile; the median of the lower half of data.
  • Second Quartile ($Q_2$): $50^{\text{th}}$ percentile; the overall median.
  • Third Quartile ($Q_3$): $75^{\text{th}}$ percentile; the median of the upper half of data.
  • Fourth Quartile ($Q_4$): $100^{\text{th}}$ percentile; the maximum data point.
  • Interquartile Range (IQR): Measures the spread of the central $50\%$ of data, ignoring the upper and lower $25\%$ tails:     $\text{IQR} = Q_3 - Q_1$
  • Worked Example:
  • Dataset: $[10, 20, 30, 40, 50, 60, 70, 80, 90, 100]$ ($n = 10$).
  • $Q_2\text{ (Median)} = \frac{50 + 60}{2} = 55$
  • Lower half: $[10, 20, 30, 40, 50] \implies Q_1 = 30$
  • Upper half: $[60, 70, 80, 90, 100] \implies Q_3 = 80$$\text{IQR} = 80 - 30 = 50$
  • Meteorological and physical applications: Used in daily temperature reporting because the IQR is resistant to extreme uncharacteristic weather spikes, providing a more robust measure of typical dispersion than the standard range.

• Deciles:
  • Partitions an ordered dataset into ten equal divisions ($10\%, 20\%, \dots, 100\%$), denoted as $D_1$ through $D_{10}$.
  • Position formula for the $i^{\text{th}}$ decile:     $D_i = \frac{i \times (n + 1)}{10}\text{th value}$
  • Calculation Algorithm:
  1. Determine sample size ($n$).
  2. Sort data in ascending order.
  3. Calculate the fractional position index for $D_i$.
  4. Apply linear interpolation between the surrounding integer indices.
  • Complete Worked Example ($23\text{ observations}$):
  • Raw data: $[24, 32, 27, 32, 23, 62, 45, 77, 60, 63, 36, 54, 57, 36, 72, 55, 51, 32, 56, 33, 42, 55, 30]$
  • Sorted array ($n = 23$):
    • $1: 23$
    • $2: 24$
    • $3: 27$
    • $4: 30$
    • $5: 32$
    • $6: 32$
    • $7: 32$
    • $8: 33$
    • $9: 36$
    • $10: 36$
    • $11: 42$
    • $12: 45$
    • $13: 51$
    • $14: 54$
    • $15: 55$
    • $16: 55$
    • $17: 56$
    • $18: 57$
    • $19: 60$
    • $20: 62$
    • $21: 63$
    • $22: 72$
    • $23: 77$
  • Base index multiplier: $\frac{n + 1}{10} = \frac{23 + 1}{10} = 2.4$
  • Interpolated decile values:
    • $D_1 = 1 \times 2.4 = 2.4^{\text{th}}\text{ position} = 24 + 0.4 \times (27 - 24) = 25.2$
    • $D_2 = 2 \times 2.4 = 4.8^{\text{th}}\text{ position} = 30 + 0.8 \times (32 - 30) = 31.6$
    • $D_3 = 3 \times 2.4 = 7.2^{\text{th}}\text{ position} = 32 + 0.2 \times (33 - 32) = 32.2$
    • $D_4 = 4 \times 2.4 = 9.6^{\text{th}}\text{ position} = 36 + 0.6 \times (36 - 36) = 36.0$
    • $D_5 = 5 \times 2.4 = 12.0^{\text{th}}\text{ position} = 45.0$
    • $D_6 = 6 \times 2.4 = 14.4^{\text{th}}\text{ position} = 54 + 0.4 \times (55 - 54) = 54.4$
    • $D_7 = 7 \times 2.4 = 16.8^{\text{th}}\text{ position} = 55 + 0.8 \times (56 - 55) = 55.8$
    • $D_8 = 8 \times 2.4 = 19.2^{\text{th}}\text{ position} = 60 + 0.2 \times (62 - 60) = 60.4$
    • $D_9 = 9 \times 2.4 = 21.6^{\text{th}}\text{ position} = 63 + 0.6 \times (72 - 63) = 68.4$
  • Real-world application: Deciles are widely used in educational testing and school benchmarking to group student cohorts, where the top decile ($D_9 - D_{10}$) receives academic honors and the bottom decile ($D_1$) receives targeted academic remediation.

Ethical Frameworks, Governance, and Data Discarding

• Data Governance Framework Principles:

  • A Data Governance Framework establishes structural methodologies, role assignments, and organizational processes to systematically standardize, integrate, secure, audit, and store data assets.
  • Ethical challenges escalate when organizations commercialize or monetize user data externally for purposes divergent from the initial consent context.

• Core Ethical Guidelines for Data Analysis:

  • Protect Customer Privacy and Confidentiality:
  • Privacy does not mean absolute obscurity, as systems must support regulatory audits.
  • Confidential user data collected under informed consent must never be exposed, leaked, or traded to unauthorized external entities.
  • Strict contractual boundaries must restrict third-party data sharing (e.g., medical, financial, or spatial-location records).
  • Users must maintain clear visibility and governance authority over how their personal telemetry flows across third-party networks.
  • Respect Human Autonomy and Agency:
  • Advanced predictive systems must not manipulate, coerce, or undermine autonomous human decision-making.
  • Predictive analytics should not institutionalize, reinforce, or amplify systemic societal biases (e.g., racism, sexism, or socioeconomic exclusion) by training models on uncorrected historical data.

• Mechanisms for Secure Data Discarding:

  • Digital Data Purging:
  • Digital records represent the most vulnerable corporate asset due to sophisticated cyber-attacks and persistence in unallocated storage.
  • Simple "soft deletion" moves files to a temporary recycling directory or marks index blocks as available, leaving the underlying raw bits intact and easily recoverable via forensic tools.
  • Best practices:
    • Encrypt data at rest so that unlinked leaked blocks remain cryptographically unreadable.
    • Perform permanent memory clean-out and cryptographic erasure.
    • Execute low-level disk formatting and multi-pass overwrites across the physical sectors of storage media.
  • Physical Document Destruction:
  • Physical documentation containing personally identifiable information (PII) or confidential client data requires irreversible destruction:
    • Industrial Shredding: Cross-cut shredding ensures fragments cannot be manually or digitally reconstructed.
    • Micro-Cutting: Slices individual documents into miniature, indecipherable geometric segments.
    • Incineration (Burning): Completely converts cellulose paper into ash, guaranteeing that physical records cannot be reconstituted.

Review Exercises, High-Order Thinking Skills, and Projects

• Unit 1 Review: Use of Statistics in Data Science:

  • Objective Analysis & Solutions:
  • 1. Extracting red cars from an automobile table: b) Data based subsetting (uses conditional value-filtering on attribute Color = 'Red').
  • 2. Accurate central tendency measure with outliers: b) Median (mean is distorted by extreme outliers).
  • 3. Mean absolute deviation identifies variability: a) Yes (quantifies data dispersion about the mean).
  • 4. MAD divided by coefficient of MAD calculates: c) Arithmetic Mean (by algebraic definition of the coefficient).
  • 5. Combined Arithmetic Mean problem:
    • Given: Unit A ($n_1 = 40$, $\bar{x}_1 = 6400$), Unit B ($n_2 = 30$, $\bar{x}_2 = 5500$).       $\bar{x}_{\text{comb}} = \frac{n_1 \bar{x}_1 + n_2 \bar{x}_2}{n_1 + n_2} = \frac{(40 \times 6400) + (30 \times 5500)}{40 + 30} = \frac{256000 + 165000}{70} = \frac{421000}{70} \approx 6014.29$
    • Correct option: d) 6014.29.
  • 6. Mean deviation about the mean for dataset $[5, 6, 7, 8, 6, 9, 13, 12, 15]$ ($n = 9$):
    • Sum = $81$, Mean $\bar{x} = \frac{81}{9} = 9$.
    • Absolute deviations: $|5-9|=4$, $|6-9|=3$, $|7-9|=2$, $|8-9|=1$, $|6-9|=3$, $|9-9|=0$, $|13-9|=4$, $|12-9|=3$, $|15-9|=6$.
    • Sum of deviations = $4 + 3 + 2 + 1 + 3 + 0 + 4 + 3 + 6 = 26$.
    • Mean Deviation = $\frac{26}{9} \approx 2.89$.
    • Correct option: c) 2.89.
  • 7. Arithmetic mean of numerical deviations from an average: d) Mean Deviation.
  • HOTS Computations:
  • Mean of $[56, 89, 76, 58, 58, 65]$ ($n = 6$):       $\text{Sum} = 402 \implies \bar{x} = \frac{402}{6} = 67$
  • Median of $[56, 89, 76, 58, 58, 65]$:
    • Sorted: $[58, 58, 56, 65, 76, 89]$ arranged properly is $[56, 58, 58, 65, 76, 89]$.
    • Central pair: $58$ and $65$.       $\text{Median} = \frac{58 + 65}{2} = \frac{123}{2} = 61.5$

• Unit 2 Review: Distributions in Data Science:

  • Objective Analysis & Solutions:
  • 1. Probability of drawing a $5$ or a $7$ from a standard deck ($52\text{ cards}$):
    • Deck contains four $5\text{s}$ and four $7\text{s}$, total $8\text{ favorable outcomes}$.       $P = \frac{8}{52} = \frac{2}{13}$
    • Correct option: c) 8/52.
  • 2. Condition for Uniform Distribution: a) Each value in the set of possible values has the exact same possibility of happening.
  • 3. Collection of one or more outcomes: c) Event.
  • 4. Types of distributions: c) Both of them (Continuous and Discrete).
  • 5. Non-discrete probability distribution: a) The sale or purchase price of a house (continuous financial scale).
  • 6. Representation of discrete probability distributions: d) All of these (tables, graphs, mathematical equations).
  • 7. Probability of drawing a ball at random from an unspecified jar: e) Cannot be determined from given information.
  • 8. Components of the statistical investigative process: e) All of the above.

• Unit 3 Review: Identifying Patterns & Central Limit Theorem:

  • Objective Analysis & Solutions:
  • 1. Data science term for partiality, preference, and prejudice: a) Bias.
  • 2. Non-existent bias category: d) Trial Bias.
  • 3. Invalid probability statement: d) It is the collection of several experiments.
  • 4. Sampling distribution normality condition: b) The sample size is large ($n \ge 30$).
  • 5. Mean of the sampling distribution of sample means: c) Exactly equal to the population mean ($\mu_{\bar{x}} = \mu$).
  • 6. Sample size $n = 25$ from population with $\mu = 40$, $\sigma = 7.5$: c) 40 (mean of the sampling distribution always equals $\mu$).
  • HOTS Problem: US Healthcare Prediction Algorithm:
  • The algorithm used health costs as a proxy for healthcare needs. Because historical spending on black patients was lower due to systemic wealth disparities and unequal access to insurance, the algorithm falsely inferred that black patients were healthier than equally ill white patients.
  • This represents a combination of Selection Bias (biased historical proxy metrics) and Systemic Confirmation Bias.
  • HOTS Problem: English Speaking Population in India:
  • Population parameters: $\mu = 62$, $\sigma = 5$, sample size $n = 50$.
  • Mean of sample: $\mu_{\bar{x}} = 62$
  • Standard Error of sample:       $\sigma_{\bar{x}} = \frac{5}{\sqrt{50}} = \frac{5}{7.071} \approx 0.707$

• Unit 4 Review: Data Merging, Z-Scores, and Positional Measures:

  • Objective Analysis & Solutions:
  • 1. Definition of the $p^{\text{th}}$ percentile: c) p percent of the observations fall at or below it.
  • 2. Function used for quantiles: a) Quantile.
  • 3. Z-score for a $6\text{ ft}$ ($72\text{ inches}$) woman ($\mu = 65.5$, $\sigma = 2.5$):       $Z = \frac{72 - 65.5}{2.5} = \frac{6.5}{2.5} = 2.60$
    • Correct option: a) 2.60.
  • 4. Definition of Z-score: a) It is the number of standard deviations a particular score lies above or below the mean of the set of scores.
  • 5. Classification of median, deciles, and percentiles: d) Averages of position.
  • 6. Position of the median in percentiles: c) 50th.
  • 7. Measure dividing distribution into $10\text{ parts}$: b) Deciles.
  • 8. Measure dividing distribution into $4\text{ parts}$: a) Quartiles.
  • HOTS Problems:
  • Height of a 2nd grader: $\mu = 48\text{ inches}$, $\sigma = 2\text{ inches}$, $x = 40\text{ inches}$.       $Z = \frac{40 - 48}{2} = \frac{-8}{2} = -4.0$
  • Plant Heights ($\mu = 13\text{ cm}$, $\sigma = 1.3\text{ cm}$):
    • a. Z-score for $9\text{ cm}$: $Z = \frac{9 - 13}{1.3} = \frac{-4}{1.3} \approx -3.08$
    • b. Z-score for $2\text{ cm}$: $Z = \frac{2 - 13}{1.3} = \frac{-11}{1.3} \approx -8.46$
    • c. Centimeters corresponding to $Z = +2.25$:         $2.25 = \frac{x - 13}{1.3} \implies x - 13 = 2.925 \implies x = 15.925\text{ cm}$
  • Applied Project: Infant Birth Weights:
  • Mean birth weight $\mu = 3.5\text{ kg}$, standard deviation $\sigma = 1.25\text{ kg}$.
  • Underweight threshold: $x < 2.5\text{ kg}$.
  • Z-score calculation:       $Z = \frac{2.5 - 3.5}{1.25} = \frac{-1.0}{1.25} = -0.80$
  • Using standard normal distribution tables, the probability corresponding to $Z < -0.80$ is approximately $0.2119$ ($21.19\%$ probability of an infant being born underweight).

• Unit 5 Review: Ethics in Data Science:

  • Objective Analysis & Solutions:
  • 1. Not a data governance framework principle: c) Never collect confidential data from users (collection is permissible with consent and security).
  • 2. Private information handling: a) True (must always be confidential).
  • 3. Action when done with confidential data: b) Effectively destroy it in a way that it is unreadable.
  • 4. Formats for confidential data storage: c) Both (Digital Data and Physical Copies).
  • 5. Data institutionalizing unfair biases: a) True (it must never institutionalize biases).
  • 6. Discarding digital confidential data: a) Formatting the drive in which data was stored.
  • 7. Inappropriate document discarding method: d) Crumbling the papers which contain confidential data and throwing it in the dustbin.