DS/CMPSC 410 Programming Models for Big Data Midterm Exam Study Guide

These flashcards cover key vocabulary and concepts from the DS/CMPSC 410 Programming Models for Big Data course, aiding in the review of important topics for the midterm exam.

Big Data

Large and complex datasets that traditional data processing software cannot manage efficiently.

MapReduce

A programming model for processing large data sets with a distributed algorithm on a cluster.

Cluster

A set of connected computers that work together and can be viewed as a single system.

RDD (Resilient Distributed Dataset)

A fundamental data structure of Spark, representing an immutable distributed collection of objects.

Lazy Evaluation

An optimization technique in Spark that delays execution until the result is actually needed.

SparkSession

An entry point to programming Spark with the DataFrame and Dataset API.

Key-Value Pair

A fundamental data representation method in distributed computing where each key is associated with a value.

JOIN operation

A method to combine records from two or more tables in a database based on related columns.

Pandas DataFrame

A 2-dimensional, size-mutable, potentially heterogeneous tabular data structure with labeled axes in Python.

Hyperparameter Tuning

The process of optimizing the parameters that govern the learning process of a machine learning model.

Execution Log

A file that records the output of an execution or job, including error messages and performance metrics.

Streaming Data

Data that is continuously generated by different sources and needs to be processed in real time.

Plan-ahead

A strategy in MapReduce that anticipates and addresses potential bottlenecks in processing tasks.

SparkContext

The entry point for Spark functionality that allows the application to access Spark's cluster resources.

Error Message Generation

A notification produced when an error occurs in code execution, indicating where the issue happened.