DS 266 Big Data Management Flashcards

Vocabulary flashcards covering key Big Data concepts, dimensions, database types, technologies (Hadoop, Spark, MongoDB), and query operators from the lecture series.

Big Data

Information or data sets so large and complex that they cannot be processed or analyzed using traditional database management tools or applications.

Volume

The dimension of Big Data representing data at rest, ranging from terabytes to exabytes of existing data.

Velocity

The dimension of Big Data representing data in motion and the speed at which streaming data flows and must be processed.

Variety

The dimension of Big Data representing data in many forms, including structured, semi-structured, unstructured, text, and multimedia data.

Veracity

The dimension of Big Data representing data in doubt, characterized by uncertainty due to incompleteness, inconsistency, and latency.

DRIP Effect

Data Rich Information Poor effect, describing how 87.5% of global data has not been really developed or used.

Data Rawness

The property of keeping the rawest data possible to maximize the ability to obtain new insights rather than overwriting or deleting information.

Data Immutability

The principle that data is not modified or deleted, but rather new data is continuously added.

Eternally True Data

The concept that a piece of data, once true, must always be true, typically achieved in practice using timestamps.

Unstructured Data

Free-form text, digital photos, videos, and audio that cannot be accommodated by traditional relational database technology.

Semi-structured Data

Typewritten data that has been processed to some extent and can be electronically stored in flat files, HTML/XML markup, or Word files.

Structured Data

Categorized data stored in files according to a particular format description that adheres to Structured Query Language (SQL).

DBMS

Database Management System; a specialized structure or format that allows computers to store, manage, and retrieve data quickly.

MapReduce

An algorithm designed by Google for efficiently executing batch-mode functions across large amounts of data using Map and Reduce operations.

Master Node

A distribution node that stores all metadata, access rights, mapping, and locations of files and blocks in a cluster.

Slave Nodes

Nodes in a distributed cluster architecture where actual data is physically stored.

Hadoop

A software framework derived from MapReduce and BigTable that runs applications on large clusters of commodity hardware.

Impala

A massively-parallel query execution engine developed specifically to run on Hadoop.

HBase

A distributed, column-oriented database component built on Hadoop.

Hive

A component that acts as a distributed data warehouse on Hadoop.

Spark

An in-memory (RAM-based) unified processing platform that accelerates processing for batching, streaming, machine learning, and interactive analysis.

MLlib

A distributed machine learning framework built on top of Spark that runs up to ten times faster than disk-based Hadoop.

AsterixDB

A scalable big data management system designed for querying semi-structured data sets.

MongoDB

A document-oriented database that stores structured and unstructured data in JSON or CSV files without using tables, rows, or columns.

Document (MongoDB)

An ordered set of keys with associated values that serves as the basic unit of data in MongoDB, replacing relational database rows.

Collection (MongoDB)

A group of documents in MongoDB, serving as the analog to a table in relational databases.

id Key

A special 12-byte key unique within a MongoDB collection assigned to identify each document.

$lt

A MongoDB comparison operator representing less than.

$gt

A MongoDB comparison operator representing greater than.

$in

A MongoDB operator used to match documents where a key's value equals any value in a specified array.

$all

A MongoDB operator used to query arrays for documents that contain all specified search elements regardless of order.

$size

A MongoDB operator used to query for arrays of a specific size or length.

Mutable Variable (var)

A variable declared in Scala using the var keyword that can be reassigned or modified after initialization.

Immutable Variable (val)

A variable declared in Scala using the val keyword that cannot be reassigned once initialized.