Final Exam Study Guide

Core Concepts and Definitions

• Big Data Characteristics: Understand the 3 V's: Volume, Variety, and Velocity. Big data is not just about volume, but also the diversity of data types and the speed of data generation.

• Datafication: Recognize how aspects of life are being turned into quantifiable data through IoT devices, wearables, and platforms like LinkedIn.

• Data Units: Be familiar with the different units of data storage, from bits to yottabytes, and the distinction between bits (b) and bytes (B).

• Data Analytics vs. Data Science: Understand the differences in scope, complexity, time orientation, data types, and technical depth between data analytics and data science. Data analytics solves defined problems using established methods, while data science explores open-ended questions and develops new methods.

• Flavors of Analytics: Know the different types of analytics: descriptive, diagnostic, predictive, and prescriptive.

Data Storage and File Formats

• Row-based vs. Column-based Formats: Understand the differences in how data is stored and the implications for analytical tasks. Row-based formats store all data for one record together, while column-based formats store all values for a single column together.

• Use Cases for Row and Column Storage: Row-based formats are better when you need to access all the data of a single record; column-based formats are ideal for column-specific tasks, like calculating total sales or aggregating data by date.

• File Format Characteristics: Consider compression, splittability, and schema evolution when choosing file formats.

• Specific File Formats: Understand the characteristics and use cases for JSON, Parquet, and Avro. Parquet is a column-based format good for read-heavy operations, while JSON is a row-based format that is easy to split and is good for writes.

Data Processing and Analysis

• MapReduce: Understand the basic principles of MapReduce, including the mapping, shuffling, and reducing phases.

• Data Partitioning: Understand how data is divided into chunks and distributed across multiple machines for parallel processing.

• Hadoop: Understand that Hadoop is an open-source implementation of MapReduce and it uses the Hadoop Distributed File System (HDFS).

• Apache Spark: Know the key components of Spark, including Spark Core, Spark SQL, Spark Streaming, MLlib, and GraphX. Understand that Spark uses functional programming, is in-memory based and has lazy execution.

• RDDs and DataFrames: Understand the difference between RDDs and DataFrames in Spark; RDDs are a lower-level abstraction while DataFrames are higher-level and better for structured data. Know that RDDs are list-like. Know how to create RDDs with parallelize().

• Spark Transformations and Actions: Understand the difference between transformations (lazy operations) and actions (return data) in Spark. Know that transformations include operations like map, filter, groupByKey, reduceByKey, join, sortByKey, cartesian, distinct, and union. Know that actions include operations like collect(), count(), sum(), mean(), and stats(). Understand the concepts of narrow and wide transformations.

• Spark Stages: Understand that Spark creates stages as part of its job execution plan. Know that stages are created when data needs to be distributed, and that reduceByKey and collect usually create a new stage.

• Data Sampling: Understand the importance of sampling when working with big data, and that you can do random subsampling, or more complex methods like cluster-based or grid-based sampling.

• Approximated Nearest Neighbors: Understand that in practice, we can do well with the approximate closest neighbors

Text Analysis and Natural Language Processing

• Text Representation: Understand techniques to convert text into numerical formats, including Bag of Words, TF-IDF, and word embeddings.

• Word Frequency Distributions: Know that words in natural language follow Zipf's law.

• TF-IDF: Understand how TF-IDF (Term Frequency-Inverse Document Frequency) is calculated and used to represent documents as vectors.

• Document Similarity: Understand how to measure document similarity using cosine similarity on TF-IDF vectors.

• Tokenization: Know that tokenization involves breaking text into subwords, words, or characters.

• Word Embeddings: Know that word embeddings like Word2Vec capture semantic relationships between words and that embeddings can also be made for sentences and documents. Understand that these embeddings are high-dimensional vectors.

• Cosine Similarity: Understand that cosine similarity is often used to measure distances between embeddings because it focuses on the angle rather than magnitude.

• Transformers: Know the basic architecture of transformers, and how they use an encoder and a decoder. Understand that transformers use multi-head attention.

• Subword Tokenization: Know that subword tokenization can handle previously unseen words by breaking them into known subwords.

• Fine-Tuning: Understand that pre-trained models can be fine-tuned for specific tasks.

• LangChain: Understand that LangChain is used to connect multiple AI tools and models into a cohesive pipeline. Know that it uses Pydantic for data validation.

• Word Piece Algorithm: Understand that the Word Piece Algorithm breaks down words into subword units.

Clustering

• Clustering Concepts: Understand that clustering is an unsupervised learning technique to find natural groupings in data. Understand that clusters are often subjective, and are defined by having high similarity within and low similarity across regions.

• Distance and Similarity Measures: Know that distance or similarity measures are needed in clustering.

• Clustering Algorithms: Be familiar with k-means clustering and hierarchical clustering and their characteristics.

• Evaluating Clusters: Understand the basic idea of evaluating cluster results using cohesion.

• k-Nearest Neighbors: Understand the k-Nearest Neighbors algorithm and that it can be used for classification, regression and clustering. Know that increasing k increases the number of neighbors considered.

Additional Concepts

• Data Quality and Validity: Be aware of the importance of ensuring data quality and validity.

• Actionable Insights: Understand the need to balance insight value against implementation costs.

• Interpretability: Know the importance of making complex patterns understandable.

• Benchmarking: Understand that benchmarking is used to measure the performance of a system.

• Profiling: Know that profiling helps identify inefficient code sections that use excessive resources.

• PyArrow: Understand that PyArrow is used for memory-efficient, columnar data representation. It is faster for in-memory operations but has less built-in data manipulation functionality compared to Pandas. Understand that Parquet is for on-disk storage and Arrow is primarily for in-memory processing.

• Matplotlib: Understand that Matplotlib is used for creating plots.