Big Data

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/62

encourage image

There's no tags or description

Looks like no tags are added yet.

63 Terms

1
New cards
Incompleteness error
Data is missing; fix by deleting the record or imputing a value
2
New cards
Invalidity error
Data falls outside a meaningful range; fix by checking against other sources
3
New cards
Inaccuracy error
Data is not a true measure of the real value; fix by validating against reliable sources
4
New cards
Inconsistency error
Data conflicts with other data in the same record; fix by using clearly defined data codes
5
New cards
Non-uniformity error
Data is not in an identical format; fix by converting to a common format
6
New cards
Duplication error
The same observation appears more than once; fix by removing duplicates
7
New cards
Exam clue: blank or missing value
Incompleteness error
8
New cards
Exam clue: value outside a possible range (e.g. negative age)
Invalidity error
9
New cards
Exam clue: wrong value that looks plausible
Inaccuracy error
10
New cards
Exam clue: contradicting entries in one record (e.g. city and country don't match)
Inconsistency error
11
New cards
Exam clue: mixed formats (e.g. dates, currencies, units)
Non-uniformity error
12
New cards
Exam clue: the same row appears twice
Duplication error
13
New cards
Data cleansing
Fixing or removing errors in raw data before it is used in a model
14
New cards
Data wrangling (preparation)
Transforming and structuring cleansed data so it is ready for the model
15
New cards
Data wrangling: extraction
Creating a new variable from an existing one (e.g. pulling the year from a date)
16
New cards
Data wrangling: aggregation
Combining related data into one variable (e.g. summing sales across stores)
17
New cards
Data wrangling: filtration
Removing rows that are not needed
18
New cards
Data wrangling: selection
Removing columns (features) that are not needed
19
New cards
Data wrangling: conversion
Changing the data type of a variable (e.g. text to number)
20
New cards
Outlier
An observation far away from the rest of the data
21
New cards
Trimming (truncation)
Removing the extreme values from the dataset
22
New cards
Winsorization
Replacing extreme values with a set percentile value instead of deleting them
23
New cards
Scaling
Putting features on a comparable scale so that large-valued features don't dominate
24
New cards
Normalization (min-max scaling)
Rescales values to a range of 0 to 1; sensitive to outliers
25
New cards
Standardization
Rescales values to a mean of 0 and standard deviation of 1; works best with roughly normal data
26
New cards
Standardization vs normalization (outliers)
Standardization is less sensitive to outliers than normalization
27
New cards
Structured data
Organized in rows and columns (tables, spreadsheets)
28
New cards
Unstructured data
Has no set format (text, images, audio, video)
29
New cards
Text cleansing
Removing HTML tags, punctuation, numbers and extra white spaces from raw text
30
New cards
Text normalization
Making text consistent by lowercasing, removing stop words, and applying stemming or lemmatization
31
New cards
Tokenization
Splitting text into individual pieces called tokens (usually words)
32
New cards
Stop words
Very common words (e.g. "the", "is") that add little meaning and are removed
33
New cards
Stemming
Reducing a word to its base form by removing endings; rule-based and may produce non-words
34
New cards
Lemmatization
Reducing a word to its dictionary root (lemma); more accurate but more computationally expensive
35
New cards
Stemming vs lemmatization
Stemming is faster and cruder; lemmatization is slower and more accurate
36
New cards
Bag-of-words (BOW)
A collection of distinct tokens from the text; word order is ignored
37
New cards
Document term matrix (DTM)
A table with documents as rows and tokens as columns, with word counts in the cells
38
New cards
N-gram
A group of n consecutive words used as one token; helps keep word order and context
39
New cards
Why use n-grams
A bag-of-words ignores word order, but n-grams preserve some of it
40
New cards
Exploratory data analysis (EDA)
Examining the data with summary statistics and charts to find patterns and problems before modeling
41
New cards
EDA charts for structured data
Histograms, box plots, scatter plots, and summary statistics
42
New cards
Feature selection
Choosing the most useful features and removing the rest to reduce noise and overfitting
43
New cards
Feature selection methods (text)
Term frequency, chi-square test, and mutual information
44
New cards
Feature engineering
Creating new, more useful features from existing ones (e.g. combining, binning, one-hot encoding)
45
New cards
One-hot encoding
Converting a categorical variable into binary (0/1) columns, one per category
46
New cards
Class imbalance
One class has far more observations than another; fix by oversampling the minority or undersampling the majority
47
New cards
Training, validation and test sets
Train the model, tune it on the validation set, then judge final performance on the test set
48
New cards
Cross-validation
Repeatedly splitting the data into training and validation parts to get a more reliable performance estimate
49
New cards
Overfitting
Model fits the training data very well but performs poorly on new data
50
New cards
Underfitting
Model is too simple and performs poorly on both training and new data
51
New cards
Confusion matrix
A table of true positives, false positives, false negatives and true negatives
52
New cards
Precision
True positives ÷ (true positives + false positives)
53
New cards
Recall
True positives ÷ (true positives + false negatives)
54
New cards
Accuracy
(True positives + true negatives) ÷ all observations
55
New cards
F1 score
The harmonic mean of precision and recall: 2 × precision × recall ÷ (precision + recall)
56
New cards
Precision vs recall (which to use)
Precision when false positives are costly; recall when false negatives are costly
57
New cards
ROC curve
Plots the true positive rate against the false positive rate at different thresholds
58
New cards
AUC (area under the curve)
Higher is better; 0.5 is no better than random guessing and 1.0 is perfect
59
New cards
RMSE
Root mean squared error; the standard performance measure for models predicting continuous values
60
New cards
Hyperparameter
A setting chosen before training that is not learned from the data
61
New cards
Tuning
Adjusting hyperparameters to improve performance and balance bias and variance
62
New cards
Grid search
Trying many combinations of hyperparameter values and picking the best-performing one
63
New cards
Fitting curve
A plot of training and validation error against model complexity, used to spot overfitting