Descriptive Analytics I: Nature of Data, Big Data and Statistical Modeling (copy)

A set of 55 practice flashcards covering data taxonomy, data preprocessing tasks, Big Data principles, Hadoop/Spark architectures, and statistical modeling.

Data

A collection of facts usually obtained as the result of experiences, observations, or experiments, serving as the lowest level of abstraction from which information and knowledge are derived.

Structured Data

Data targeted primarily for computers to process, characterized by predefined formats and models, divided into numeric and nominal/categorical types.

Unstructured Data

Data primarily targeted for humans to process and digest, lacking a formal predefined schema or structure (such as natural language text documents, images, audio, and video).

Semi-Structured Data

A hybrid form of data that does not conform to a traditional relational database schema but contains internal markers, tags, or hierarchies to separate semantic elements (such as XML, HTML, JSON, and log files).

Data Taxonomy

A structural classification that organizes analytics data into structured data (subcategorized into categorical [nominal, ordinal] and numerical [interval, ratio]) and unstructured/semistructured data (textual, multimedia, and web/log).

Data Granularity

A critical quality metric for analytics-ready data that describes the scale or level of detail at which data units are represented and aggregated.

Data Currency (Data Timeliness)

A metric evaluating how up-to-date and temporally appropriate data is for its intended analytical modeling and decision-making purpose.

Data Preprocessing

An iterative art-and-science process with multiple feedbacks and redo cycles that cleans, transforms, consolidates, and reduces dirty, misaligned, and complex raw real-world data into analytics-ready data.

Data Consolidation

The data preprocessing phase encompassing the collection, access, selection, filtering, integration, and unification of disparate data into a single coherent structure using SQL queries, agents, or Web services.

Data Cleaning

The preprocessing task of handling missing values, reducing noise, and eliminating erroneous or duplicate records to enhance data quality.

Data Imputation

The data cleaning method of replacing missing observations with statistically reasonable values, such as the mean, median, mode, minimum, or maximum.

Outlier Smoothing

A technique used to identify anomalies using standard deviations or cluster analysis, followed by removing them or smoothing them using binning, regression, or simple averages.

Data Transformation

The phase of data preprocessing that modifies variables via normalization, discretization or aggregation, and constructing newly derived attributes.

Normalization

A data transformation procedure that scales and compresses the range of values in each numerical variable to a standard interval, such as $[0, 1]$ or $[-1, +1]$.

Discretization

The transformation technique of converting continuous numeric variables into discrete categories using range- or frequency-based binning methods.

Data Reduction

The preprocessing step aimed at simplifying datasets by reducing the number of variables (attributes), reducing the number of records (cases/samples), or balancing skewed class distributions.

Dimensionality Reduction

A data reduction process that limits the number of variables under consideration using methods such as Principal Component Analysis (PCA), Independent Component Analysis (ICA), chi-square tests, correlation analysis, or decision tree induction.

Class Imbalance Problem

A modeling challenge where one target outcome class heavily outnumbers other classes (e.g., $80\%$ 'No' vs. $20\%$ 'Yes'), which requires rebalancing through oversampling or undersampling.

Ensemble Models

Advanced machine learning architectures—such as Bagging (Random Forest), Boosting (Boosted Trees), and Information Fusion (Weighted Average)—that combine predictions from multiple individual models to improve accuracy.

Big Data

Datasets characterized by massive volume, high velocity, and extreme variety that exceed the capture, storage, management, and processing capabilities of traditional analytical platforms.

Volume (in Big Data)

The physical magnitude and storage scale of data generated by organizations, scaling from terabytes to petabytes, exabytes, and beyond.

Velocity (in Big Data)

The rapid speed at which data is produced, captured, and processed in motion, often requiring real-time continuous streaming analysis.

Variety (in Big Data)

The structural diversity of data types arriving at an enterprise, spanning structured tables, semi-structured files, and unstructured media formats.

Veracity (in Big Data)

The accuracy, reliability, authenticity, availability, accountability, and overall trustworthiness of data coming from disparate sources.

Variability (in Big Data)

The shifting rates, peaks, and inconsistent flows of data over time, which can complicate continuous real-time processing.

Value (in Big Data)

The economic, strategic, and analytical business worth realized only when massive, fast, and multi-structured data is combined with advanced analytical methods.

Petabyte ($\text{PB}$)

A unit of digital storage capacity equal to $10^{15}$ bytes, representing high-velocity data scales such as the Large Hadron Collider generating $1\,\text{PB/sec}$.

Brontobyte ($\text{BB}$)

A non-official SI data scale metric representing digital information storage equal to $10^{27}$ bytes.

Gegobyte ($\text{GeB}$)

An unofficial digital storage unit magnitude representing information equal to $10^{30}$ bytes.

Schema-on-Demand

A flexible storage approach that applies structural schemas to data upon read/query execution rather than enforcing a rigid schema during initial data ingestion.

In-Memory Analytics

A high-performance Big Data enabler that stores and manipulates an entire active dataset directly inside Random Access Memory ($\text{RAM}$) to bypass slower disk read/write latencies.

In-Database Analytics

The architecture of embedding analytical algorithms and queries directly within the database management system where data resides, avoiding heavy data transport overhead.

Massively Parallel Processing (MPP)

A computing paradigm that utilizes coordinated networks of multiple independent processors and machines executing computations concurrently in parallel.

Hadoop

An open-source Apache framework originally created by Doug Cutting at Yahoo! that coordinates clusters of commodity hardware to store and process massive unstructured and semi-structured datasets.

Hadoop Distributed File System (HDFS)

The distributed file storage component of Hadoop that breaks large multi-structured data files into blocks, replicates them across slave nodes, and provides fault tolerance.

Name Node

The primary master facilitator in an HDFS architecture that manages directory namespaces, file block locations, and overall cluster metadata.

Job Tracker

A core management component in Hadoop that accepts computational jobs from clients and schedules tasks across worker nodes in the cluster.

MapReduce

A parallel programming paradigm popularized by Google that partitions tasks by distributing data processing across clusters via mapping (filtering/sorting) and reducing (aggregating) functions.

Hive

An open-source data warehouse software infrastructure built on top of Apache Hadoop that provides an SQL-like query interface for querying large datasets stored in HDFS.

NoSQL

Non-relational database management systems ('Not Only SQL') that provide flexible schemas (document, key-value, graph, column) for high-scale, real-time, unstructured data operations.

Apache Spark

An open-source cluster-computing framework developed in 2009 by Apache that uses in-memory processing to execute iterative algorithms, streaming data, and machine learning orders of magnitude faster than disk-based MapReduce.

Stream Analytics

The real-time analytical process of evaluating data-in-motion as it continuously arrives from smart sensors, logs, or meters before it loses immediate operational relevance.

Data for Good

Collaborative, volunteer-driven, and crowdsourced community initiatives that utilize publicly available data and analytics tools to address social and environmental challenges.

Descriptive Statistics

A collection of mathematical techniques utilized to organize, characterize, summarize, and describe the properties of a dataset as it currently exists.

Inferential Statistics

A branch of statistics that draws inferences, tests hypotheses, and makes predictions regarding an entire population based on sample data.

Arithmetic Mean

A central tendency metric representing the sum of all observed values divided by the number of observations: $\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i$.

Median

The measure of central tendency identifying the middle numerical value in an ordered dataset, splitting the upper $50\%$ of data from the lower $50\%$.

Mode

The central tendency statistic defined as the single most frequently occurring observation within a given dataset.

Sample Variance ($s^2$)

A measure of dispersion calculating the mean squared deviation of sample values around the arithmetic mean: $s^2 = \frac{1}{n-1}\sum_{i=1}^{n} (x_i - \bar{x})^2$.

Mean Absolute Deviation (MAD)

A measure of dispersion that computes the arithmetic average of the absolute distances between individual data points and their mean: $\text{MAD} = \frac{1}{n}\sum_{i=1}^{n} |x_i - \bar{x}|$.

Box-and-Whiskers Plot

A graphical summary of data dispersion displaying the minimum, lower quartile ($Q_1$), median, mean, upper quartile ($Q_3$), maximum, and outlier thresholds.

Skewness

A distributional shape metric that measures asymmetry around the mean, calculated as: $S = \frac{1}{n}\sum_{i=1}^{n} \left(\frac{x_i - \bar{x}}{s}\right)^3$.

Kurtosis

A descriptive shape metric evaluating the peakedness or tail heaviness of a probability distribution curve, calculated as: $K = \frac{1}{n}\sum_{i=1}^{n} \left(\frac{x_i - \bar{x}}{s}\right)^4 - 3$.

Ordinary Least Squares (OLS)

A mathematical optimization procedure used in linear regression that estimates model coefficients ($\beta_0, \beta_1$) by minimizing the sum of squared error residuals between observed and predicted values.

Logistic Regression

A supervised classification technique developed in the 1940s that models the probability of a binary categorical outcome via the sigmoid logistic equation: $f(y) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}$.