BUSINESS FOUNDATION QUIZ 1 - MIDTERM

Exploratory Data Analysis (EDA)

___is a crucial step in the data analysis process that involves examining and visualizing data to understand its main characteristics, uncover patterns, identify anomalies, and formulate hypotheses

is to gain insights into the underlying structure of the data and to guide the subsequent steps of statistical analysis or modeling

The primary goal of EDA__

Data Summarization:

• Computing basic summary statistics such as mean, median, mode, range, and standard deviation.

• Understanding the distribution of data through measures like skewness and kurtosis.

Data Visualization

• Creating visual representations of the data to reveal patterns and trends.

• Using histograms, box plots, scatter plots, and other charts to understand the distribution and relationships within the data

Handling Missing Data:

• Identifying and addressing missing values in the dataset.

• Assessing the impact of missing data on the analysis

Outlier Detection:

• Identifying and examining outliers that may significantly impact statistical analysis.

• Understanding the potential causes and deciding how to handle outliers.

Data Transformation:

• Applying transformations to variables, such as logarithmic or square root transformations, to achieve a more normal distribution.

• Standardizing or normalizing variables to ensure comparability.

Understanding Relationships:

• Exploring relationships between variables through correlation analysis.

• Investigating patterns in bivariate or multivariate distributions

Exploring Patterns in Time Series Data:

• Analyzing temporal patterns in data for time series datasets.

• Using line plots, seasonal decomposition, and autocorrelation functions.

Interactive Exploration:

• Utilizing interactive tools and dashboards for dynamic exploration of data.

• Platforms like Tableau, Power BI, or Jupyter notebooks with interactive visualizations.

Formulating Hypotheses:

• Based on the insights gained, formulating initial hypotheses about the relationships or patterns observed in the data.

• These hypotheses can guide further statistical testing or modeling

Communication of Findings

• Effectively communicating findings and insights to stakeholders, team members, or decision-makers.

• Preparing visualizations and reports to convey the story told by the data.

Understanding the Data

• EDA helps analysts become familiar with the dataset, including its structure, size, and content.

• It provides a preliminary understanding of the variables, their types, and the overall distribution of data.

Identifying Patterns and Trends

• EDA involves visualizing data through charts, graphs, and statistical summaries to identify patterns and trends.

• Analysts can detect relationships between variables, observe clusters, and recognize potential outliers or anomalies.

Handling Missing Data

• EDA helps identify missing values in the dataset and assess their impact on analysis.

• Analysts can make informed decisions about whether to impute missing data, exclude certain observations, or adopt other strategies

Outlier Detection:

• EDA aids in identifying outliers or extreme values that may significantly influence statistical analysis.

• Understanding the nature of outliers helps decide whether to retain, transform, or remove them

Data Transformation

• EDA guides decisions about data transformations, such as normalization or standardization, to meet assumptions of statistical methods.

• Transformation techniques may be applied to achieve a more normal distribution or to improve comparability between variables

Variable Selection

• EDA assists in selecting relevant variables for analysis by examining their distribution, relationship with the target variable, and potential collinearity.

• Analysts can prioritize features that contribute the most to the understanding of the problem.

Formulating Hypotheses:

• Based on observed patterns, EDA allows analysts to formulate initial hypotheses about relationships within the data.

• These hypotheses guide further statistical testing or modeling approaches.

Choosing Modeling Techniques:

• EDA informs decisions about appropriate modeling techniques by providing insights into the nature of the data.

• Analysts can choose regression, classification, clustering, or other methods based on the characteristics uncovered

Validating Assumptions

• EDA helps validate assumptions underlying statistical tests or models.

• Checking assumptions ensures the reliability and validity of subsequent analyses.

Communicating Results:

• EDA facilitates effective communication of findings to stakeholders, team members, or decision-makers.

• Visualizations and insights generated during EDA contribute to reports and presentations.

Iterative Analysis

• EDA is often an iterative process, where insights gained lead to further exploration or refinement of analysis techniques.

• Analysts may revisit EDA as they progress through different stages of analysis

Data preprocessing

__is the process of transforming raw data into a clean, organized, and usable format before feeding it into a data analysis or machine learning model. It's a critical first step in the data science workflow to ensure high-quality results

Raw data collected from real-world sources (like surveys, databases, sensors, or social media) often has

• Missing values

• Incorrect or inconsistent formats

• Duplicate records

• Irrelevant features

• Outliers or noise

Data Cleaning:

This involves handling issues that can distort analysis or model performance.

Handling Missing Values:

Addressing incomplete data entries by either removing rows/columns with missing values or imputing (filling in) them using various strategies (e.g., mean, median, mode, or more advanced methods)

Removing Duplicates

Identifying and eliminating redundant records to ensure each entry is unique and relevant

Correcting Inconsistent Formats

Standardizing data formats (e.g., date formats, string cases, units of measurement) to maintain consistency.

Handling Outliers:

Identifying and addressing extreme values that can significantly skew results or model training. This might involve removing them, transforming them, or treating them specially

Data Transformation:

This involves converting data into a format that is more suitable for analysis or modeling.

Normalization and Scaling

Adjusting numerical features to a standard range or distribution. This is important for algorithms that are sensitive to the magnitude of features

Min-Max Scaling

normalizing to a [0, 1] range

Standardization

scaling to zero mean and unit variance

Encoding Categorical Variables

Converting non-numerical (text-based) categorical data into a numerical representation that machine learning algorithms can understand. Common methods include One-Hot Encoding (creating new binary columns for each category) and Label Encoding (assigning a unique integer to each category)

Feature Engineering

Creating new features from existing ones to better capture underlying patterns and improve model performance. This often leverages domain knowledge

Data Aggregation:

Combining multiple data points into summary forms (e.g., averages, sums) to simplify analysis or reduce dimensionality

Data Integration

When data comes from multiple sources, this involves combining them into a unified dataset.

Schema Matching

Aligning fields and data structures from different sources

Data Deduplication

Identifying and removing duplicate entries across integrated datasets

Data Reduction

Reducing the volume of data while retaining essential information to improve efficiency and reduce computational costs

Dimensionality Reduction

Reducing the number of features (variables) in a dataset. Techniques like Principal Component Analysis (PCA) are used to create new, lower-dimensional representations of the data.

Feature Selection

Choosing a subset of the most relevant features to improve model performance and reduce overfitting

graphical representation of information and data

Data visualization is the ___. It's the art and science of translating data into visual forms like charts, graphs, maps, and dashboards, making it easier for the human brain to understand and identify trends, patterns, and insights

Bar Charts

What they show: Comparisons between discrete categories or changes over a

period.

• When to use them:

• To compare quantities across different groups (e.g., sales figures for different product

lines, student enrollment by major).

• To show changes over time when the time intervals are distinct (e.g., monthly website

visitors, quarterly revenue).

• Key features: Rectangular bars with lengths proportional to the values they

represent. Can be vertical or horizontal

Line Charts

What they show: Trends or changes over a continuous period. Excellent for

illustrating progression and movement.

• When to use them:

• To visualize data over time (e.g., stock prices over a year, daily temperature fluctuations,

website traffic trends over months).

• To show relationships between two continuous variables where one influences the other

over time.

• Key features: Data points connected by lines, representing continuous data

Pie Charts / Donut Charts

What they show: Proportions or parts of a whole (composition). Each slice

represents a category's contribution to the total.

• When to use them:

• To show the relative size of different categories that sum up to 100% (e.g., market share

breakdown, budget allocation by department)

Key features: A circular graph divided into sectors, where each sector's area is

proportional to the quantity it represents. Donut charts are similar but have a

hole in the center.

• Considerations: While popular, they can be less effective for comparing many

categories or showing small differences between slices. Bar charts are often

preferred for more precise comparisons

Scatter Plots

What they show: The relationship or correlation between two numerical

variables.

• When to use them:

• To identify patterns, trends, or correlations between two continuous variables (e.g., the

relationship between advertising spend and sales, study hours and exam scores).

• To detect outliers or unusual data points.

• Key features: Individual data points plotted on a two-dimensional graph, with

one variable on the x-axis and the other on the y-axis

Histograms

What they show: The distribution of a single numerical variable. They

illustrate the frequency of data points within specified ranges (bins).

• When to use them:

• To understand the shape, spread, and central tendency of a dataset (e.g., the distribution

of ages in a population, the frequency of different income brackets).

• To identify common values, gaps, or outliers in a distribution.

• Key features: Bars that represent the frequency of data points falling into a

continuous range (bin), with no gaps between bars (unless a bin is empty)

Box Plots (Box-and-Whisker Plots)

What they show: The distribution of numerical data and provide a summary

of its five-number summary: minimum, first quartile (Q1), median (Q2), third

quartile (Q3), and maximum. They also highlight outliers.

• When to use them:

• To compare the distribution of a continuous variable across different categories or

groups (e.g., comparing salary distributions across different departments, or test scores

between different schools).

• To quickly identify potential outliers.

• Key features: A box representing the interquartile range (IQR), a line within

the box for the median, and "whiskers" extending to the minimum and

maximum non-outlier values.

Heat Maps

What they show: The magnitude of a phenomenon as a color in two

dimensions. They are excellent for visualizing correlation matrices, complex

relationships, or geographic data.

• When to use them:

• To display intensity or density of values across categories (e.g., website click patterns,

correlation between multiple variables).

• For geographic data, where color intensity represents a value over a region.

• Key features: A grid where cells are colored based on the value they

represent, often with a color scale