Introduction to Modern Data Analysis and Statistics and Analysis

Definitions and Importance of Data Analysis

• Data (Dictionary Definition): Factual information, measurements, or statistics used as a basis for reasoning, discussion, and calculation.

• Data (Modern/AI Context): Anything that can be analyzed to provide information, including numbers, statistics, categories, natural language text, digital images, and videos.

• Data Analysis (Wikipedia Definition): The process of inspecting, cleaning, transforming, and modeling data to discover useful information, inform conclusions, and support decision-making.

• Layman’s Definition: Using mathematics and statistics to extract meaning from large datasets to make more sensible decisions.

• Importance of Data:

  • Data validates opinions and ideas: "Without data, you're just another person with an opinion."

  • Data Explosion: In 2017, it was estimated that over $2.5 \times 10^{18}$ bytes (2.5 quintillion bytes) of digital data were created every day.

  • The challenge of modern AI is not finding data, but finding the specific answers hidden within the "digital chaos."

The Four Goals of Data Analysis

1. Descriptive Data Analysis: Summarizes what has already happened.

   • Example: Identifying that company sales dropped by $15\%$ last year.

2. Diagnostic Data Analysis: Investigates the cause or reason behind an event by looking for correlations between data attributes.

   • Example: Determining that the $15\%$ sales drop correlated with a $3\%$ increase in national unemployment.

3. Predictive Data Analysis: Predicts what might happen in the future by designing mathematical models and algorithms to forecast unknown values.

   • Example: Forecasting a $20\%$ loss in profit for the next year if current trends continue.

4. Prescriptive Data Analysis: Provides advice on what actions or decisions to take by combining findings from the other three types.

   • Example: Recommending a shift toward emphasizing low-priced goods (under $100) because that market remains stable.

Applications and Case Studies

• Business/E-commerce: Marketing, recommendations, and targeted advertising.

• Healthcare: Disease prevention, diagnostics via blood tests/analysis, and personalized treatment plans based on patient-specific data.

• Cybersecurity: Analyzing network traffic for suspicious behavior, detecting anomalies (e.g., bank fraud detection based on purchase history), and protecting against malicious actors.

• Education: Platforms like Canvas allow instructors to track student engagement (e.g., video watch time, link clicks) to identify at-risk students.

• Case Study: Netflix (2006):

  • Netflix offered a $1 million USD prize to anyone who could improve their movie recommendation model by $10\%$.

  • Recommendations create a personalized experience by assigning a percentage likelihood that a user will enjoy a specific title.

  • Netflix estimated that $75\%$ of viewer activity is driven by these algorithmic recommendations.

Related Fields and Terminology

• Data Science / Data Mining: Advanced versions of data analysis with a heavier emphasis on mathematics, machine learning, and predictive modeling (e.g., neural networks).

• Data Engineering: Focuses on the infrastructure of data. This involves designing and maintaining systems for the collection, cleaning, and organized storage (databases) of data so that it can be retrieved efficiently.

Step 1: Identifying the Research Question

• The first step in the five-step data analysis pipeline is defining a formal problem or question.

• Translation: A vague problem (e.g., "Profits are down") must be translated into an answerable research question (e.g., "Can I group customers into different advertising demographics?" or "How can we reduce production costs?").

• Supermarket Scenario: If customers report a bad experience, researchers must identify factors (food price, checkout times, lack of healthy options) and determine corresponding data to collect.

Step 2: Data Collection and Types

• Quality Metrics: Data must be Relevant (related to the question), Correct (robust and verifiable), and Representative (sampled from the intended demographic to avoid bias).

• Data Sources:

  • Records: Existing student records, published reports, or government census data.

  • Surveys: Opinions gathered from a population.

    • Comprehensive Survey: Asking every single person in a population.

    • Sample Survey: Taking a subset of the population (common, but must be representative).

  • Scientific Experiments: Controlled testing (e.g., A/B testing different teaching styles in different campuses).

• Business Categorization:

  • Primary (Internal): Data collected within the company (customer reports, internal metrics).

  • Secondary (External): Public reports, government statistics, or outside research.

• Data Types:

  • Quantitative (Numerical):

    • Continuous: Any value in a range (height, money spent, time).

    • Discrete: Defined intervals or counts (number of items, review score 1–5, dates).

  • Qualitative (Categorical):

    • Nominal: Categories without defined order (color, species).

    • Ordinal: Categories with an implicit order (e.g., "Very Bad" to "Very Good").

  • Binary: A category with only two possible values (Survived/Died, Male/Female).

Step 3: Data Cleaning and Formatting

• Storage Formats: Small datasets use CSV (Comma Separated Values) or Excel spreadsheets. Large-scale data uses relational databases.

• Dataset Structure:

  • Rows: Records or individual instances (e.g., one specific customer).

  • Columns: Attributes, features, or properties (e.g., age, price paid).

  • Unique ID: The first column (e.g., FANS number or Passenger ID) uniquely identifies each record.

• Dirty Data Issues:

  • Irrelevant Data: Removing attributes that do not affect the outcome (e.g., number of letters in a name for movie preference) to speed up analysis and avoid coincidental correlations.

  • Duplicate Data: Identifying the same record entered multiple times using unique IDs.

  • Anomalies and Outliers: Edge cases that fall outside the norm.

    • Errors: Incorrect entry (e.g., a student listed as 340 years old).

    • Extreme Cases: Real but rare occurrences (e.g., a single order for 15 burgers for a primary school class). These can skew results like the Mean.

  • Missing Entries: Handling blank values in a survey.

• Handling Missing Data:

  • Removal: Dropping the entire row (record) or entire column (attribute). Recommended to drop a column only if it has a high percentage of missing values.

  • Imputation: Estimating values.

    • Mean Value Imputation: Replace missing numerical values with the average of the attribute.

    • Most Frequent Imputation: Replace missing categorical values with the mode.

• Effort Distribution: Data analysts typically spend $70\text{--}80\%$ of their time on the cleaning step.

Step 4: Data Exploration and Summary Statistics

• Univariate Analysis: Collecting statistics about a single individual attribute (e.g., average final mark).

• Multivariate Analysis: Analyzing relationships between multiple attributes (e.g., relationship between Canvas interaction time and final grades).

• Common Statistics:

  • Mean ($\mu$): Sum of values divided by the number of records. Sensitive to outliers.

  • Median: The middle value in an ordered list. Robust against outliers.

  • Mode: The most frequent value. Used for discrete or qualitative data.

  • Variance ($\sigma^2$): Explains the spread of data.

    • $\sigma^2 = \frac{\sum (x - \mu)^2}{N}$

    • A low variance indicates High Central Tendency (data points cluster close to the mean).

    • High variance suggests data is spread out and may contain outliers.

Step 5: Data Visualization

• Bar Graphs: Used for qualitative/categorical data to show frequency.

• Histograms: Used for numerical data. Data is grouped into "bins" or ranges. Bin size is a critical variable: too large lacks detail; too small becomes noisy.

• Box and Whisker Plots: Displays Min, Max, Median, and the Interquartile Range (upper and lower quartiles). Excellent for identifying skewness and outliers.

• Pie Charts: Used to compare frequencies of a whole. Note: Some analysts consider them harder to read than bar graphs and generally advise against them.