Introduction to Data Science - Complete Lecture Notes

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/74

flashcard set

Earn XP

Description and Tags

Vocabulary practice flashcards covering foundational concepts, lifecycles, preprocessing techniques, descriptive statistics, regression modeling, and evaluation metrics from Introduction to Data Science.

75 Terms

1
New cards

Data Science

The domain of study that deals with vast volumes of data using modern tools and techniques to find unseen patterns, derive meaningful information, and make business decisions using complex machine learning algorithms.

2
New cards

Data Science Lifecycle

A five-stage framework consisting of Capture, Maintain, Process, Analyze, and Communicate, each designated with specific data-handling tasks.

3
New cards

Capture (Lifecycle Stage)

The initial stage of the data science lifecycle involving gathering raw structured and unstructured data through data acquisition, data entry, signal reception, and data extraction.

4
New cards

Maintain (Lifecycle Stage)

The lifecycle stage that covers taking raw data and putting it into a usable format through data warehousing, data cleansing, data staging, data processing, and data architecture.

5
New cards

Process (Lifecycle Stage)

The lifecycle stage where data scientists examine prepared data for patterns, ranges, and biases through data mining, clustering/classification, data modeling, and data summarization to determine usefulness in predictive analysis.

6
New cards

Analyze (Lifecycle Stage)

The core stage of the data science lifecycle that involves performing analyses such as exploratory/confirmatory analysis, predictive analysis, regression, text mining, and qualitative analysis.

7
New cards

Communicate (Lifecycle Stage)

The final stage of the data science lifecycle in which analysts prepare findings into readable formats such as charts, graphs, and reports for business intelligence and decision-making.

8
New cards

The Future of Data Analysis (1962)

The landmark article by American mathematician John W. Tukey that first articulated the data science dream and foresaw the emergence of the field nearly two decades before the first personal computers.

9
New cards

International Association for Statistical Computing (IASC)

An association established in 1977 with the mission to link traditional statistical methodology, modern computer technology, and domain expert knowledge to convert data into information and knowledge.

10
New cards

Data Analyst

A professional responsible for visualization, munging, database querying, and data processing, who relies heavily on optimization and technologies like SQL, R, SAS, and Python.

11
New cards

Data Engineer

A professional who builds and tests scalable Big Data ecosystems, updates existing systems, and maintains data efficiency using technologies such as Hive, NoSQL, R, Ruby, Java, C++, and Matlab.

12
New cards

Database Administrator

A professional responsible for the proper functioning of enterprise databases, granting or revoking user access, managing database software, data security, and backup and recovery.

13
New cards

Machine Learning Engineer

A specialized engineer who designs, builds, tests, and deploys machine learning systems and pipelines, performs A/B testing, and implements algorithms such as classification and clustering.

14
New cards

Data Scientist

A professional who understands business challenges and provides solutions by performing predictive analysis, integrating unstructured data, and extracting actionable insights using tools like R, MatLab, SQL, and Python.

15
New cards

Data Architect

A professional who creates blueprints for data management so databases can be easily integrated, centralized, and secured, requiring expertise in data warehousing, data modelling, ETL, Hive, Pig, and Spark.

16
New cards

Statistician

A professional with a sound understanding of statistical theories and data organization who extracts insights from data clusters and creates new methodologies using SQL, data mining, and machine learning.

17
New cards

Business Analyst

A role focused on understanding organizational business, separating high-value data from low-value data, and improving business processes using business intelligence, data modeling, and visualization tools.

18
New cards

Stages in a Data Science Project

The five iterative phases of executing a project: defining a problem, data processing, modelling, evaluation, and deployment to production.

<p>The five iterative phases of executing a project: defining a problem, data processing, modelling, evaluation, and deployment to production.</p>
19
New cards

Data Security

The process of protecting corporate data and preventing data loss from unauthorized access, ransomware, corruption, or destruction, while ensuring data is available to authorized users.

20
New cards

Data Privacy

The distinction between data that can be shared with third parties (non-private) and data that cannot be shared (private), primarily focusing on confidentiality via access control and data protection.

21
New cards

Insider Threats

Security threats posed by employees within an organization, divided into non-malicious insiders (accidental/negligent), malicious insiders (deliberate harm for personal gain), and compromised insiders.

22
New cards

Ransomware

Malicious software that infects corporate devices and encrypts data, rendering it unusable until a ransom is demanded to release the decryption key.

23
New cards

SQL Injection (SQLi)

A security exploit used by attackers to gain illicit access to databases by inserting malicious code into seemingly innocent database queries.

24
New cards

Data Masking

A security solution that creates a synthetic, functional version of organizational data for testing and training without exposing real sensitive records.

25
New cards

Data Encryption

The conversion of human-readable plaintext into unreadable ciphertext that can only be read or processed after using a specific decryption key.

26
New cards

Primary Data Collection

The collection of original, first-hand data directly from primary sources that has never been used previously, collected via interviews, observations, surveys, focus groups, or oral histories.

27
New cards

Secondary Data Collection

The gathering of pre-existing data that has already been collected and compiled by others, accessible through the Internet, government archives, and libraries.

28
New cards

Data Preprocessing

Any processing performed on raw data to transform it into an understandable and clean format for machine learning, data mining, and analytics pipelines.

29
New cards

Dummy Variable

A numerical variable that takes the value 0 or 1 to indicate the absence or presence of a categorical feature expected to shift an outcome.

30
New cards

Feature Scaling

A preprocessing technique used to limit the range of variables so that they can be compared and processed on common numerical grounds.

31
New cards

Data Integration

A preprocessing approach formally defined as the triple ⟨G,S,M⟩\langle G, S, M \rangle (global schema, source schema, mapping) that combines data from heterogeneous sources into a coherent data store.

32
New cards

Tight Coupling

A data integration approach where data from different sources is physically extracted, transformed, and loaded (ETL) into a single consolidated data warehouse.

33
New cards
<p>Loose Coupling</p>

Loose Coupling

A data integration architecture where a mediated schema and wrapper interface send translated user queries directly to source databases while data remains stored in the original sources.

34
New cards

Min-Max Normalization

A linear transformation method mapping attribute values ViV_i to a new range [new_minA,new_maxA][\text{new\_min}_A, \text{new\_max}_A] using the formula: Vi−minAmaxA−minA(new_maxA−new_minA)+new_minA\frac{V_i - \text{min}_A}{\text{max}_A - \text{min}_A}(\text{new\_max}_A - \text{new\_min}_A) + \text{new\_min}_A.

35
New cards

Z-Score Normalization

A normalization technique that standardizes an attribute value ViV_i based on the attribute mean Aˉ\bar{A} and standard deviation σA\sigma_A using the formula: Vi−AˉσA\frac{V_i - \bar{A}}{\sigma_A}.

36
New cards

Decimal Scaling

A normalization method that shifts the decimal point of attribute values by dividing by 10j10^j, where jj is the smallest integer such that the maximum absolute value is strictly less than 1.

37
New cards

Data Discretization

The process of converting continuous attribute values into a finite set of discrete intervals or categorical labels to simplify data mining and analysis.

38
New cards

Dimensionality Reduction

A data reduction process that eliminates weakly important, redundant, or irrelevant attributes from a dataset using techniques such as Wavelet Transforms, PCA, and Attribute Subset Selection.

39
New cards

Numerosity Reduction

A technique that reduces original data volume by substituting it with alternative data representations, divided into parametric methods (regression, log-linear models) and non-parametric methods (histograms, clustering, sampling).

40
New cards
<p>Data Compression</p>

Data Compression

Encoding or modifying data structure to consume less storage space, classified into lossless compression (exact data reconstruction) and lossy compression (approximated data restoration).

41
New cards

Descriptive Statistics

Statistical procedures that describe, show, and summarize the basic empirical features and measurements of a dataset sample without drawing inferences or predictions.

42
New cards

Measures of Central Tendency

Metrics that estimate the central position or average of a dataset, comprising the mean, median, and mode.

43
New cards

Variance

A measure of the dispersion or spread of a dataset, calculated as the average of the squared deviations from the mean: Variance=∑(xi−mean)2N\text{Variance} = \frac{\sum (x_i - \text{mean})^2}{N}.

44
New cards

Standard Deviation

The average amount of variability in a dataset that reflects how far individual observations lie from the mean, calculated as the square root of variance (σ=Variance\sigma = \sqrt{\text{Variance}}).

45
New cards

Skewness

A statistical measure of the asymmetry of a probability distribution relative to a symmetrical Gaussian/normal distribution, represented by the third standardized moment.

46
New cards

Positive Skewness

A distribution where extreme values are larger, causing the tail to extend to the right side, characterized by the relationship: Mean>Median>Mode\text{Mean} > \text{Median} > \text{Mode}.

47
New cards

Negative Skewness

A distribution where extreme values are smaller, causing the tail to extend to the left side, characterized by the relationship: Mean<Median<Mode\text{Mean} < \text{Median} < \text{Mode}.

48
New cards

Pearson's First Coefficient of Skewness

A skewness formula calculated by subtracting the mode from the mean and dividing the difference by the standard deviation: Mean−ModeStandard Deviation\frac{\text{Mean} - \text{Mode}}{\text{Standard Deviation}}.

49
New cards

Pearson's Second Coefficient of Skewness

A skewness formula calculated by multiplying the difference between the mean and median by 3 and dividing by the standard deviation: 3(Mean−Median)Standard Deviation\frac{3(\text{Mean} - \text{Median})}{\text{Standard Deviation}}.

50
New cards

Kurtosis

A metric that characterizes the shape and tail heaviness of a probability distribution relative to a normal distribution, based on the fourth standardized central moment.

51
New cards

Excess Kurtosis

The difference between a distribution's kurtosis and the kurtosis of a normal distribution (33), calculated as: Excess Kurtosis=Kurt−3\text{Excess Kurtosis} = \text{Kurt} - 3.

52
New cards

Leptokurtic Distribution

A heavy-tailed distribution with positive excess kurtosis (kurtosis>3\text{kurtosis} > 3), possessing a distinct sharp peak, thick tails, and a high likelihood of outliers.

53
New cards

Platykurtic Distribution

A short-tailed distribution with negative excess kurtosis (kurtosis<3\text{kurtosis} < 3), characterized by a flatter, broader curve with fewer extreme values compared to a normal distribution.

54
New cards

Mesokurtic Distribution

A distribution with a kurtosis of 3 and excess kurtosis of 0 (3−3=03 - 3 = 0), identical in peakedness and tail heaviness to a normal distribution.

55
New cards
<p>Box Plot</p>

Box Plot

A visual display summarizing data via the Five Number Summary: Minimum Value, Lower Quartile (25th Percentile), Median, Upper Quartile (75th Percentile), and Maximum Value.

56
New cards

Interquartile Range (IQR)

The difference between the upper quartile (75th percentile) and the lower quartile (25th percentile), representing the spread of the middle 50% of the dataset: IQR=Upper Quartile−Lower Quartile\text{IQR} = \text{Upper Quartile} - \text{Lower Quartile}.

57
New cards

Pivot Table

A data summarization tool that reorganizes and aggregates information from large 2-dimensional tables, converting 'long data' into 'wide data' using aggregation functions such as sum, mean, or count.

58
New cards

Heatmap

A 2-dimensional visual representation of data matrices where numeric values are rendered as varying colors using hue, saturation, or luminance.

59
New cards

Correlation Coefficient

A standardized statistical measure ranging between −1-1 and +1+1 that indicates the direction and strength of the linear relationship between two variables.

60
New cards

ANOVA (Analysis of Variance)

A statistical hypothesis testing method used to compare means across three or more groups by calculating the FF ratio: Variation between groupsVariation within groups\frac{\text{Variation between groups}}{\text{Variation within groups}}.

61
New cards

Simple Linear Regression

A predictive modeling technique that establishes a straight-line relationship between a single independent variable and a single dependent variable using the equation: Y=mX+cY = mX + c.

62
New cards

Multiple Linear Regression

A regression model that establishes a linear relationship between one continuous target variable and two or more predictor variables using the equation: Y=b0+b1x1+b2x2+⋯+bnxnY = b_0 + b_1 x_1 + b_2 x_2 + \dots + b_n x_n.

63
New cards

Residual

The vertical distance or error between an actual observed data point and the value predicted by a regression line (y−y^y - \hat{y}).

64
New cards

Residual Plot

A graph displaying residuals on the vertical y-axis and the independent variable on the x-axis, used to assess whether model errors are random, independent, and normally distributed.

65
New cards

Polynomial Regression

A form of regression analysis modeling a non-linear relationship between independent variable xx and the conditional mean of dependent variable yy as an nthn\text{th} degree polynomial: E(y∣x)E(y|x).

66
New cards

Mean Squared Error (MSE)

An in-sample evaluation metric that measures regression fit by calculating the average of squared differences between actual and predicted values: MSE=1n∑(actual−forecast)2\text{MSE} = \frac{1}{n} \sum (\text{actual} - \text{forecast})^2.

67
New cards

R-squared (R2R^2)

A metric of correlation and goodness-of-fit indicating the proportion of variance in the dependent variable explained by a regression model relative to baseline variation.

68
New cards
<p>Confusion Matrix</p>

Confusion Matrix

A tabular layout that summarizes the classification performance of a model on test data with known true values, categorized into True Positives, True Negatives, False Positives, and False Negatives.

69
New cards

Type I Error

A statistical hypothesis testing error equivalent to a False Positive (FP) that occurs when a researcher mistakenly rejects a null hypothesis that is actually true.

70
New cards

Type II Error

A statistical hypothesis testing error equivalent to a False Negative (FN) that occurs when a researcher fails to reject (accepts) a null hypothesis that is false in reality.

71
New cards
<p>K-Fold Cross Validation</p>

K-Fold Cross Validation

A validation technique where the training dataset is partitioned into kk equal folds, iteratively training the model on k−1k-1 folds and evaluating it on the remaining holdout fold.

72
New cards

Overfitting

A condition where a machine learning model learns training data details and noise so extensively that it exhibits high variance and low bias, resulting in poor generalization on new test data.

73
New cards

Underfitting

A situation where a machine learning model fails to learn the underlying structure of the data due to high bias and low variance, performing poorly on both training and test data.

74
New cards

Ridge Regression

A regularized regression technique that mitigates multicollinearity by applying an L2L2 regularization penalty to the cost function: min⁡(∣∣Y−Xθ∣∣2+λ∣∣θ∣∣2)\min(||Y - X\theta||^2 + \lambda||\theta||^2).

75
New cards

Grid Search

A brute-force hyperparameter tuning technique that builds and evaluates models across every combination of specified hyperparameters using cross-validation to select the optimal model.