BDA

INTRODUCTION TO BIG DATA

1.1 INTRODUCTION TO BIG DATA

Big Data represents a significant technological trend that organizations are increasingly discussing. The primary challenge organizations face is to extract maximum value from their existing data while also predicting what data should be collected in the future. The ability to derive meaningful insights from past data is a crucial focus in executive meetings across various companies.
The surge of data, commonly referred to as "Big Data", has transformed data management from a traditional task into a necessary reality across numerous organizations.
The overarching goal shared among organizations and experts is to optimize data utilization, though the methodologies and starting points differ across entities. As organizations experiment with and design big data solutions, they uncover various related opportunities.
It is important to note that no single solution or vendor can claim comprehensive expertise in Big Data, resulting in a vast landscape encompassing various architectures, technologies, and players within this domain.
The defining characteristics of Big Data are encapsulated by the three Vs: Velocity, Volume, and Variety.

1.2 BIG DATA CHARACTERISTICS

1. Volume

Volume refers to the exponential growth in data storage capabilities. Data can now exist in multiple formats beyond text, including videos, music files, and large images, especially prevalent on social media platforms. Organizations often manage storage systems that can reach Terabytes (TB) and Petabytes (PB) of data.
As databases expand, the applications and architectures supporting this data collection must be frequently reassessed. Sometimes, reevaluating the same data from different perspectives reveals new insights, thereby further compounding the data explosion inherent to Big Data.

2. Velocity

The growth of data, particularly through social media platforms, has dramatically altered the perception of data recency. While traditional media outlets still convey information based on outdated timelines, today's communications are often real-time. Users demand constant updates and discard outdated messages, leading to a near-instantaneous data flow characterized by high velocity.

3. Variety

Variety signifies the presence of data in various formats. Data can be stored in databases, Excel files, CSVs, and may even appear in non-traditional formats like videos and SMS messages. The challenge here is to organize this varied data meaningfully, which becomes increasingly difficult when data formats differ significantly from one another.

1.3 TYPES OF BIG DATA

1. APACHE HADOOP

Apache Hadoop stands as a core component within the realm of Big Data technologies. It enables the processing of large quantities of structured or unstructured data cost-effectively.
Hadoop is an open-source project comprised of software libraries and frameworks that facilitate data management across multiple servers, ranging from one server to thousands.
Hadoop incorporates robust error detection mechanisms that function at the application layer rather than relying solely on hardware.
As of December 2012, Apache released version 1.0.0 of Hadoop, with installation guides found in the Apache Hadoop documentation. Importantly, Hadoop consists of not just one, but several supporting technologies.

2. MAPREDUCE

MapReduce was conceptualized by Google to facilitate the creation of vast web search indexes. This framework allows developers to create applications that can process both structured and unstructured data through a partitioning method.
MapReduce divides tasks into smaller components to run across multiple nodes, thereby enhancing its ability to process large data sets efficiently and with fault tolerance. More resources on MapReduce and tutorials can be found in the Apache documentation.

3. HDFS (Hadoop Distributed File System)

The Hadoop Distributed File System (HDFS) is a Java-based system designed for the storage of data across extensive clusters of distributed servers. HDFS allows for both structured and unstructured data storage without stringent regulatory demands on data organization. Developers are responsible for ensuring data meaningfulness through their code. HDFS also guarantees high fault tolerance and can utilize lower-cost hardware setups.
The HDFS capabilities, along with installation guidelines, are detailed in the Apache HDFS documentation.

4. HIVE

Initially created by Facebook and now available as an open-source solution, Hive serves as an intermediary between SQL queries and Hadoop.
It essentially acts as a data warehouse which enables ad-hoc querying, data summarization, and analysis on substantial data sets within Hadoop-compatible file systems.
Hive employs a query language called HiveQL, analogous to SQL, that automates the transformation of SQL-like queries into MapReduce jobs.
In January 2013, Apache released Hive 0.10.0, with further information found in the Apache Hive documentation.

5. PIG

Pig was developed by Yahoo and is currently open-source. Like Hive, it also facilitates data queries over Hadoop clusters, but utilizes a script-based approach rather than a SQL-like front-end.
Apache Pig provides a high-level programming platform for data processing and analytics through user-defined functions. January 2013 saw the release of Pig 0.10.1, compatible with Hadoop 0.10.1 or later versions, with additional resources available in the Apache Pig Getting Started documentation.

1.4 TRADITIONAL VS BIG DATA BUSINESS APPROACH

1. OVERVIEW OF DATABASES

Organizations have historically utilized row and table-based relational databases to perform online transactions. However, when faced with the influx of unstructured data, there arises a necessity for schema-less databases that lack hard schema constraints.
Numerous databases fit this category, catering to the storage of unstructured, semi-structured, and fully structured data, with specific emphasis on simplifying data migration.
An example of a prominent schema-less database is MongoDB, renowned for its flexibility in handling vast amounts of data without necessitating intricate relationships between data elements.
"NoSQL (Not Only SQL) describes a category of databases that do not rely solely on schema-based structures for data processing."

2. STRUCTURED VS UNSTRUCTURED APPROACH

The traditional business approach relies on structured data organized into distinct tables and columns. It does not effectively accommodate the fluidity of live data inflows that define the Big Data landscape, leading to an iterative analytical approach within this new paradigm.
Big Data analytics thrive on processing unstructured data, devoid of specific patterns and often not confined to traditional organization in rows and columns.

1.5 BIG DATA ARCHITECTURE

1.1 HADOOP OVERVIEW

  1. Hadoop serves as an open-source framework that promotes the processing of extensive data sets in distributed computing environments.

  2. The Hadoop framework includes essential components: MapReduce and HDFS, along with various other related projects such as Apache Hive, HBase, and Zookeeper.

  3. This Apache Hadoop system is free to access and heavily reliant on Java, offering capabilities to manage Big Data across expansive clusters of commodity hardware, handling thousands of TBs of data across numerous nodes.

  4. A distinct advantage of the Hadoop framework is its reliability and high availability, ensuring consistent operations even during node breakdowns.

1.2 USES OF HADOOP

Implementing Hadoop presents numerous advantages:

  1. Robust and Scalable: New nodes can be integrated flexibly as business needs evolve.

  2. Cost-Effective: Hadoop operates efficiently on normal server hardware without the need for specialized systems.

  3. Adaptable and Inclusive: Designed to accommodate both structured and unstructured data.

  4. High Availability and Fault Tolerance: Automated transitions occur for failed nodes, ensuring continuity in service.

1.3 CORE HADOOP COMPONENTS

The Hadoop architecture is built around two primary components that facilitate essential tasks:

  1. Hadoop MapReduce splits large data problems into smaller, manageable chunks distributed across numerous commodity servers. Each server processes locally, gathering results to a central server for assembly, making it an efficient tool for large data processing.

  2. Hadoop Distributed File System (HDFS), acts as a virtual file system uniquely splintering files into smaller pieces that are replicated across multiple servers (e.g., typically three) for added fault tolerance and high availability.

1.4 NAMES OF HADOOP NODES AND DATA MANIPULATION

  1. NameNode: Recognized as the core of the Hadoop system, it oversees the file system namespace while managing metadata information about data blocks and their locations. While it does store critical metadata (such as namespaces and edit logs), it is important to note that it does not maintain actual data. The NameNode is crucial as it oversees the mapping of data blocks to DataNodes.

  2. Secondary NameNode: This node is utilized to periodically merge the namespace image and edit log from the NameNode, creating backups in case of crashes.

  3. DataNode: Responsible for actual data storage and retrieval, frequently reporting block information back to the NameNode.

  4. JobTracker and TaskTracker: JobTracker manages the scheduling of client jobs while distributing tasks among TaskTrackers located on DataNodes, ensuring that all jobs are efficiently scheduled and monitored.

2.1 SCHEMAS IN HADOOP

Hadoop’s coordination allows flexibility with data structures and enables handling of various types of data, from structured to unstructured. With applications like Pig and Hive, teams can work with datasets without focusing on rigid schemas, keeping analysis adaptive and dynamic.

2.2 HADOOP DAEMONS

  1. NameNode: This master node maintains the file system's namespace, stores metadata, and manages block locations.

  2. DataNode: This slave node stores actual data in blocks and executes read/write operations based on client requests.

  3. Secondary NameNode: This node periodically creates checkpoints by merging the metadata, ensuring efficient recovery in the event of a NameNode crash.

  4. JobTracker: A master process in the MapReduce framework responsible for resource management and scheduling of tasks across the cluster.

  5. TaskTracker: Responsible for executing tasks assigned by the JobTracker, these slave processes operate on the DataNodes.

3.1 FUNCTIONALITIES OF HADOOP COMPONENTS

As previously mentioned, each component plays a crucial role in managing and processing Big Data, from initial storage to real-time data processing and analysis.

2.4 LIMITATIONS OF HADOOP

Despite its many benefits, Hadoop presents several limitations:

  • It was designed for specific Big Data applications and may not suit all types of tasks.

  • Random read requirements give Hadoop limited scalability during processes necessitating fast access to small data segments.

  • Hadoop lacks caching mechanisms to further enhance data accessibility during repeated reads 🢶 this shortfall pertains particularly to non-real-time batch processing datasets.

The information captured provides a comprehensive overview of Big Data and Hadoop architecture, along with its operational components and function. This structure serves as a definitive study guide for understanding the core concepts and technologies of Big Data.

1.1 INTRODUCTION TO BIG DATA

Big Data is a fancy way of saying a massive amount of information. Organizations today have a big goal: to take all the information they've collected and use it to figure out what people might want in the future.

  • Simplified Idea: Imagine if you kept a record of every snack you ever ate. After a year, you could look at that data to predict which snacks you will crave next Tuesday. Businesses do the same thing on a much larger scale.

1.2 BIG DATA CHARACTERISTICS (THE THREE VS)
1. Volume (The Amount)

Volume is how much data we are dealing with. We now have more than just text; we have videos, music, and giant photos.

  • Example: Think of your phone. A few years ago, you might have had 1616 GB of space. Now, companies deal with Petabytes (1,0001,000 Terabytes) of data—that's enough to hold millions of high-definition movies.

2. Velocity (The Speed)

Velocity is how fast information is coming in. In the old days, you waited for the morning newspaper. Today, information is live.

  • Example: When something happens at a football game, it shows up on social media in seconds. That's high velocity—data moving almost instantly.

3. Variety (The Format)

Variety means data comes in many different shapes and sizes. It isn't just organized lists.

  • Example: Think of your computer files. You have Word documents (text), MP3s (audio), JPEGs (photos), and YouTube links (video). Organizing all these different types into one system is a challenge.

1.3 TYPES OF BIG DATA TOOLS
1. APACHE HADOOP

Hadoop is a special software system that helps process huge amounts of data using many different computers at once. It’s open-source, which means it’s free for anyone to use and improve.

2. MAPREDUCE

This is a tool that breaks a big job into tiny pieces.

  • Example: If you had to count every word in a library, it would take forever. MapReduce would give one shelf to each of your friends (Mapping), and then you would add all their totals together at the end (Reducing).

3. HDFS (Hadoop Distributed File System)

This is the storage part of Hadoop. It takes one big file, cuts it into pieces, and stores those pieces on different computers.

  • Simplified Idea: It’s like a group project where everyone holds one page of the report so if one person loses their page, the others still have copies.

4. HIVE

Hive allows people to talk to Hadoop using a language that looks like standard database language (SQL). It’s like a translator that turns simple commands into complex Hadoop tasks.

5. PIG

Pig is another tool for analyzing data, but instead of using SQL-like commands, it uses scripts. It’s Great for step-by-step data processing.

1.4 TRADITIONAL VS BIG DATA BUSINESS APPROACH
1. Regular Databases

Traditional databases are like a neat filing cabinet where everything has a specific folder (Row) and a specific label (Column).

2. Big Data Databases (NoSQL)

Big Data databases, like MongoDB, are more like a giant toy box. You can just throw things in without having to organize them into strict folders first. This is called a "schema-less" approach.

1.5 BIG DATA ARCHITECTURE (HOW IT'S BUILT)
1.1 Hadoop Roles

Hadoop uses a "Master and Slave" system to get work done:

  1. NameNode (The Manager): This node is the boss. It doesn't hold the actual data, but it knows where everything is stored. If you want a file, you ask the NameNode where to find it.

  2. DataNode (The Workers): These nodes do the heavy lifting. They store the actual blocks of data and do what the NameNode tells them to do.

  3. Secondary NameNode (The Assistant): This node keeps a backup of the NameNode's records just in case the boss's computer crashes.

  4. JobTracker and TaskTracker: Think of the JobTracker as a project manager who assigns homework, and the TaskTracker as the student who actually finishes the assignment.

2.4 LIMITATIONS OF HADOOP

Even though Hadoop is powerful, it has some weak spots:

  • It isn't good for "Quick Searches." If you need to find one tiny piece of information instantly, Hadoop can be slow because it was built to look at everything at once in big batches.

  • It doesn't have "Short Term Memory" (Caching). It has to re-read everything from the hard drive every time, which can slow things down.

1.1 INTRODUCTION TO BIG DATA

Big Data is a fancy way of saying a massive amount of information. Organizations today have a big goal: to take all the information they've collected and use it to figure out what people might want in the future.

  • Simplified Idea: Imagine if you kept a record of every snack you ever ate. After a year, you could look at that data to predict which snacks you will crave next Tuesday. Businesses do the same thing on a much larger scale.

1.2 BIG DATA CHARACTERISTICS (THE FIVE VS)

While many people start with three main features, there are actually five key characteristics that define Big Data:

1. Volume (The Amount)

Volume is how much data we are dealing with. We now have more than just text; we have videos, music, and giant photos.

  • Example: Think of your phone. A few years ago, you might have had 1616 GB of space. Now, companies deal with Petabytes (1,0001,000 Terabytes) of data—that's enough to hold millions of high-definition movies.

2. Velocity (The Speed)

Velocity is how fast information is coming in. In the old days, you waited for the morning newspaper. Today, information is live.

  • Example: When something happens at a football game, it shows up on social media in seconds. That's high velocity—data moving almost instantly.

3. Variety (The Format)

Variety means data comes in many different shapes and sizes. It isn't just organized lists.

  • Example: Think of your computer files. You have Word documents (text), MP3s (audio), JPEGs (photos), and YouTube links (video). Organizing all these different types into one system is a challenge.

4. Veracity (The Truth)

Veracity is about the quality or accuracy of the data. Sometimes data is messy, incomplete, or even incorrect.

  • Example: Imagine you are reading reviews for a video game. If half of the reviews are fake or bots, that data has low veracity. Businesses need to filter out the "junk" data to find the truth.

5. Value (The Worth)

Value is the most important V. It refers to the usefulness of the data. There is no point in having data if it doesn't help you solve a problem.

  • Example: A grocery store collects data on what people buy. The value is found when they use that data to send you a coupon for your favorite cereal, making you more likely to shop there again.

1.3 TYPES OF BIG DATA TOOLS
1. APACHE HADOOP

Hadoop is a special software system that helps process huge amounts of data using many different computers at once. It’s open-source, which means it’s free for anyone to use and improve.

2. MAPREDUCE

This is a tool that breaks a big job into tiny pieces.

  • Example: If you had to count every word in a library, it would take forever. MapReduce would give one shelf to each of your friends (Mapping), and then you would add all their totals together at the end (Reducing).

3. HDFS (Hadoop Distributed File System)

This is the storage part of Hadoop. It takes one big file, cuts it into pieces, and stores those pieces on different computers.

  • Simplified Idea: It’s like a group project where everyone holds one page of the report so if one person loses their page, the others still have copies.

4. HIVE

Hive allows people to talk to Hadoop using a language that looks like standard database language (SQL). It’s like a translator that turns simple commands into complex Hadoop tasks.

5. PIG

Pig is another tool for analyzing data, but instead of using SQL-like commands, it uses scripts. It’s Great for step-by-step data processing.

1.4 TRADITIONAL VS BIG DATA BUSINESS APPROACH
1. Regular Databases

Traditional databases are like a neat filing cabinet where everything has a specific folder (Row) and a specific label (Column).

2. Big Data Databases (NoSQL)

Big Data databases, like MongoDB, are more like a giant toy box. You can just throw things in without having to organize them into strict folders first. This is called a "schema-less" approach.

1.5 BIG DATA ARCHITECTURE (HOW IT'S BUILT)
1.1 Hadoop Roles

Hadoop uses a "Master and Slave" system to get work done:

  1. NameNode (The Manager): This node is the boss. It doesn't hold the actual data, but it knows where everything is stored. If you want a file, you ask the NameNode where to find it.

  2. DataNode (The Workers): These nodes do the heavy lifting. They store the actual blocks of data and do what the NameNode tells them to do.

  3. Secondary NameNode (The Assistant): This node keeps a backup of the NameNode's records just in case the boss's computer crashes.

  4. JobTracker and TaskTracker: Think of the JobTracker as a project manager who assigns homework, and the TaskTracker as the student who actually finishes the assignment.

2.4 LIMITATIONS OF HADOOP

Even though Hadoop is powerful, it has some weak spots:

  • It isn't good for "Quick Searches." If you need to find one tiny piece of information instantly, Hadoop can be slow because it was built to look at everything at once in big batches.

  • It doesn't have "Short Term Memory" (Caching). It has to re-read everything from the hard drive every time, which