data engineering unit 1

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/32

flashcard set

Earn XP

Description and Tags

unit 1 hadoop and all

33 Terms

1
New cards

Big data?

Big Data is a collection of huge amounts of data generated from various sources such as social media, websites, smartphones, sensors, and online transactions.

This data may be structured, semi-structured, or unstructured.

Traditional databases may not be sufficient to handle such large and complex data efficiently. Therefore, advanced technologies such as Hadoop, Spark, and NoSQL databases are used to store, process, and analyze Big Data.

2
New cards

Types of big data?

1. Structured Data

  • Data that is organized in a fixed format, such as rows and columns.

  • It can be easily stored and managed using relational databases.

  • Example: Student records, banking transactions, and employee databases.

2. Semi-Structured Data

  • Data that does not follow a fixed tabular structure but contains tags, keys, or other markers for organization.

  • Example: XML files, JSON files, and emails.

3. Unstructured Data

  • Data that does not have a predefined format or structure.

  • It is more difficult to store, search, and analyze using traditional databases.

  • Example: Images, videos, audio recordings, and social media posts.


3
New cards

Features of big data? (5 v’s)

1. Volume (Amount of Data)

Volume refers to the huge amount of data generated and collected from various sources.

  • Big Data involves large datasets measured in terabytes (TB), petabytes (PB), or even exabytes (EB).

  • Traditional databases may face difficulties in storing and processing such massive datasets.

  • Distributed storage systems like Hadoop HDFS are used to manage large volumes of data.

Example: YouTube stores enormous amounts of video data uploaded by users worldwide.

2. Velocity (Speed of Data)

Velocity refers to the speed at which data is generated, collected, and processed.

  • Data is continuously generated at high speed from various digital sources.

  • Many applications require real-time or near-real-time data processing.

  • High velocity requires efficient technologies for fast data handling and analysis.

Example: Stock market applications process continuously changing stock prices.

3. Variety (Different Types of Data)

Variety refers to the different formats and types of data generated from multiple sources.

Big Data includes three major types:

  • Structured Data: Organized in rows and columns, such as relational database tables.

  • Semi-Structured Data: Contains organizational markers but does not follow a fixed tabular format, such as JSON and XML.

  • Unstructured Data: Does not have a predefined structure, such as images, videos, and audio files.

Example: Social media platforms process text messages, images, videos, and user information.

4. Veracity (Accuracy and Reliability of Data)

Veracity refers to the accuracy, quality, consistency, and trustworthiness of data.

  • Big Data may contain incorrect, incomplete, duplicated, or inconsistent information.

  • Poor-quality data can produce inaccurate analytical results.

  • Data cleaning and validation techniques are used to improve data reliability.

Example: Fake product reviews on e-commerce websites can affect the accuracy of customer sentiment analysis.

5. Value (Usefulness of Data)

Value refers to the ability to extract meaningful information and useful insights from large datasets.

  • Simply collecting large amounts of data is not sufficient.

  • Data must be analyzed to obtain useful information.

  • Valuable insights help organizations improve decision-making, reduce costs, and increase efficiency.

Example: Amazon analyzes customer browsing and purchase history to recommend relevant products.

4
New cards

Dawn of the Big Data Era

The Dawn of the Big Data Era refers to the beginning of a new period in which massive amounts of data started being generated from various digital sources, leading to the development of advanced technologies for storing, processing, managing, and analyzing large and complex datasets.

5
New cards

Evolution of the Big Data Era


1. Traditional Data Era

  • Data was generated in small amounts.

  • Traditional databases and file systems were used for storage.

  • Data processing was mainly performed on individual computers.

2. Internet Era

  • The growth of the internet increased the amount of digital data.

  • Websites, emails, and online transactions became major sources of data.

3. Social Media and Mobile Era

  • Social media platforms and smartphones started generating massive amounts of data.

  • Data included text, images, videos, and user activities.

4. Big Data Era

  • Traditional systems became insufficient for handling massive datasets.

  • Technologies such as Hadoop, NoSQL, and distributed computing were introduced for efficient data management.


6
New cards

Factors Responsible for the Emergence of Big Data


1. Growth of the Internet: The increasing use of websites, search engines, and online services generates enormous amounts of digital data.

2. Social Media: Platforms such as Instagram, Facebook, and YouTube generate huge amounts of data through posts, comments, images, and videos.

3. Smartphones: Mobile applications continuously generate data related to user activities, locations, and transactions.

4. Internet of Things (IoT): Connected devices and sensors generate real-time data from various environments.

5. Digital Transactions: Online banking, e-commerce, and digital payment systems generate large amounts of transaction data.

6. Cloud Computing: Cloud platforms provide scalable storage and computing resources for managing massive datasets.

7
New cards

Importance of the Big Data Era?

  • Better Decision-Making: Helps organizations make accurate decisions based on data analysis.

  • Improved Customer Experience: Enables businesses to understand customer needs and provide personalized services.

  • Cost Reduction: Improves resource utilization and reduces operational costs.

  • Real-Time Analysis: Allows organizations to analyze data quickly and respond to changing conditions.

  • Innovation: Supports the development of artificial intelligence, machine learning, and predictive analytics.


8
New cards

Big Data Value

Big Data Value refers to the useful information, meaningful insights, and benefits obtained by analyzing large and complex datasets. It helps organizations improve decision-making, increase efficiency, reduce costs, and achieve business objectives.

9
New cards

How Big Data Creates Value


Data Collection

↓

Data Storage

↓

Data Processing

↓

Data Analysis

↓

Decision-Making

10
New cards

Importance of Big Data Value

1. Better Decision-Making

Big Data helps organizations make accurate and informed decisions by analyzing historical and real-time data.

2. Improved Customer Experience

Organizations can understand customer preferences and provide personalized products and services.

3. Cost Reduction

Big Data helps identify unnecessary expenses, optimize resources, and reduce operational costs.

4. Increased Business Efficiency

By analyzing business processes, organizations can improve productivity and overall performance.

5. Risk Management

Big Data helps organizations identify potential risks, detect fraud, and prevent financial losses.

6. Predictive Analysis

Organizations can analyze historical data to predict future trends, customer demands, and market conditions.

7. Competitive Advantage

Big Data enables organizations to identify new opportunities and develop better strategies than their competitors.

11
New cards

The Development of Big Data

The Development of Big Data refers to the evolution of data storage, processing, and analysis technologies from traditional database systems to advanced distributed computing systems capable of handling massive, complex, and rapidly growing datasets.

12
New cards

Stages in the Development of Big Data

1. Traditional File-Based Systems (Before 1970s)

  • Data was stored in simple files and physical storage systems.

  • Data processing was mainly performed manually or using basic computer programs.

  • These systems were suitable for small amounts of data.

2. Relational Database Era (1970s–1990s)

  • Relational Database Management Systems (RDBMS) were developed to store structured data.

  • SQL was introduced for efficient data retrieval and management.

  • Examples: Oracle, MySQL, and other relational database systems.

3. Internet and Web Era (1990s–2000s)

  • The growth of the internet led to a rapid increase in digital data.

  • Websites, emails, and online transactions became major sources of data.

  • Traditional databases started facing challenges in handling large-scale data.

4. Hadoop and Distributed Computing Era (2000s)

  • Google introduced the concepts of distributed file storage and MapReduce processing.

  • Apache Hadoop was developed to store and process large datasets across multiple computers.

  • Distributed computing made large-scale data processing more efficient and scalable.

5. Modern Big Data Era (2010s–Present)

  • Technologies such as Apache Spark, NoSQL databases, and cloud computing became widely used.

  • Real-time analytics, artificial intelligence, and machine learning became important applications of Big Data.

  • Organizations started using Big Data for predictive analysis and intelligent decision-making.


13
New cards

Challenges of Big Data

Challenges of Big Data refer to the difficulties faced by organizations in collecting, storing, managing, processing, analyzing, and securing massive amounts of data generated from various sources.

14
New cards

Major Challenges of Big Data

1. Data Storage

  • Big Data involves enormous amounts of data that require large storage capacity.

  • Traditional storage systems may not efficiently handle such massive datasets.

  • Distributed storage technologies such as Hadoop HDFS and cloud storage help overcome this challenge.



2. Data Processing

  • Processing large datasets requires significant computing power and time.

  • Traditional processing systems may become slow when handling massive amounts of data.

  • Technologies such as Hadoop MapReduce and Apache Spark support distributed and parallel processing.



3. Data Quality

  • Big Data may contain incomplete, incorrect, duplicate, or inconsistent information.

  • Poor-quality data can lead to inaccurate analysis and incorrect decisions.

  • Data cleaning, validation, and preprocessing techniques are required to maintain data quality.



4. Data Scalability

  • The amount of data continuously increases over time.

  • Systems must be capable of handling growing datasets without significant performance reduction.

  • Distributed computing and cloud-based solutions provide scalable infrastructure.



5. High Cost and Resource Requirements

  • Managing Big Data requires computing infrastructure, storage resources, software, and skilled professionals.

  • Infrastructure maintenance and data management can increase operational costs.

  • Cloud computing and open-source technologies can help reduce expenses.


15
New cards

Big Data Analytics

Big Data Analytics is the process of examining, processing, and analyzing large and complex datasets to discover hidden patterns, correlations, trends, and useful information that help organizations make better decisions.

16
New cards

Types of Big Data Analytics


1. Descriptive Analytics (What Happened?)

  • Descriptive Analytics analyzes historical data to understand past events and activities.

  • It summarizes data using reports, charts, and dashboards.

  • Example: An e-commerce company analyzes its monthly sales reports to understand previous sales performance.



2. Diagnostic Analytics (Why Did It Happen?)

  • Diagnostic Analytics examines data to identify the reasons behind past events.

  • It uses techniques such as data mining, correlation analysis, and drill-down analysis.

  • Example: A company analyzes customer feedback to identify why its product sales decreased.



3. Predictive Analytics (What Will Happen?)

  • Predictive Analytics uses historical data, statistical models, and machine learning techniques to predict future outcomes.

  • It helps organizations identify possible future trends and risks.

  • Example: A bank analyzes previous transactions to predict potential fraudulent activities.



4. Prescriptive Analytics (What Should Be Done?)

  • Prescriptive Analytics recommends suitable actions based on data analysis and predicted outcomes.

  • It uses optimization techniques, mathematical models, and machine learning.

  • Example: A delivery company determines the most efficient routes to reduce transportation costs.


17
New cards

Process of Big Data Analytics

Data Collection

↓

Data Storage

↓

Data Cleaning and Processing

↓

Data Analysis

↓

Data Visualization

↓

Decision-Making

18
New cards

Apache Hadoop

Apache Hadoop is a framework used to store and process huge amounts of data that cannot be efficiently handled by a single computer.

Instead of storing and processing all the data on one machine, Hadoop divides the data into smaller blocks and distributes them across multiple computers called nodes.



Hadoop is mainly based on two concepts:

  • Distributed Storage: Data is stored across multiple computers using HDFS.

  • Parallel Processing: Data is processed simultaneously on multiple computers using MapReduce.


19
New cards

Main Components of Apache Hadoop

1. Hadoop Distributed File System (HDFS)

  • HDFS is the storage system of Hadoop.

  • It divides large files into blocks and distributes them across multiple nodes.

  • It maintains multiple copies of data blocks to provide fault tolerance.

  • It mainly consists of NameNode and DataNodes.



2. MapReduce

  • MapReduce is a programming model used for parallel processing of large datasets.

  • It divides processing into two main phases:

    • Map: Processes input data and generates intermediate key-value pairs.

    • Reduce: Combines and processes intermediate results to produce the final output.

  • It improves the efficiency of large-scale data processing.



3. YARN (Yet Another Resource Negotiator)

  • YARN is responsible for resource management and job scheduling in Hadoop.

  • It allocates computing resources to different applications.

  • It helps multiple applications run efficiently on a Hadoop cluster.



4. Hadoop Common

  • Hadoop Common contains the common libraries and utilities required by other Hadoop modules.

  • It provides essential services and tools for the Hadoop framework.

  • It supports communication and coordination among Hadoop components.


20
New cards

archi


<p></p>
21
New cards

Analyzing Data with Unix Tools.

Analyzing Data with Unix Tools refers to the process of examining, filtering, sorting, transforming, and summarizing data using Unix command-line utilities to extract meaningful information from datasets.

Unix provides various tools such as grep, awk, sed, sort, uniq, and wc for efficient data processing and analysis.

22
New cards

Common Unix Tools Used for Data Analysis

1. grep: Used to search for specific words or patterns in a file.

2. awk: Used to extract, process, and analyze text data.

3. sed: Used to search, replace, and modify text in files.

4. sort: Used to arrange data in alphabetical or numerical order.

5. uniq: Used to identify or remove consecutive duplicate lines.

6. wc: Used to count lines, words, and bytes in a file.

7. cut: Used to extract specific fields or columns from a file.

8. head: Used to display the first few lines of a file.

9. tail: Used to display the last few lines of a file.

23
New cards

Analyzing Data with Hadoop

Analyzing Data with Hadoop refers to the process of storing, processing, and analyzing large datasets using the Hadoop framework, which provides distributed storage through HDFS and parallel data processing through MapReduce.

24
New cards

Steps Involved in Analyzing Data with Hadoop

1. Data Collection

Data is collected from various sources such as websites, social media, databases, sensors, and log files.


2. Data Storage in HDFS

The collected data is divided into blocks and distributed across multiple DataNodes using the Hadoop Distributed File System (HDFS).


3. Data Processing Using MapReduce

MapReduce processes large datasets in parallel through two major phases:

  • Map Phase: Processes input data and generates intermediate key-value pairs.

  • Reduce Phase: Combines and summarizes intermediate data to generate final results.


4. Resource Management Using YARN

YARN allocates computing resources and schedules tasks across different nodes in the Hadoop cluster.



5. Result Storage

The final processed results are stored in HDFS or transferred to other systems for further analysis.



6. Data Analysis and Decision-Making

The processed results are examined to identify patterns, trends, and useful information for decision-making.


25
New cards

Hadoop Streaming

Normally, Hadoop MapReduce programs are written in Java.

However, Hadoop Streaming allows developers to write MapReduce programs in other programming languages, such as Python.

It uses standard input (stdin) to receive data and standard output (stdout) to produce results.

This makes Hadoop MapReduce programming easier and more flexible without requiring developers to write Java code.

26
New cards

Hadoop Ecosystem

The Hadoop Ecosystem is a collection of open-source tools and frameworks that work together with Apache Hadoop to store, process, manage, analyze, and transfer large amounts of data in a distributed computing environment.

27
New cards

Components of Hadoop Ecosystem

1. HDFS (Hadoop Distributed File System)

  • HDFS is the primary storage system of Hadoop.

  • It stores large datasets across multiple computers in a distributed environment.

  • It provides fault tolerance through data replication.



2. MapReduce

  • MapReduce is a programming model used for parallel processing of large datasets.

  • It processes data using two major phases: Map and Reduce.

  • It distributes processing tasks across multiple nodes.



3. YARN (Yet Another Resource Negotiator)

  • YARN is responsible for resource management and job scheduling.

  • It allocates computing resources to different applications.

  • It improves resource utilization in a Hadoop cluster.



4. Hive

  • Apache Hive is a data warehousing tool used to query and analyze large datasets.

  • It provides a SQL-like language called HiveQL.

  • It allows users to analyze data without writing complex MapReduce programs.



5. Pig

  • Apache Pig is a high-level data processing platform used for analyzing large datasets.

  • It uses a scripting language called Pig Latin.

  • It simplifies complex data transformations and processing tasks.



7. Sqoop

  • Apache Sqoop is used to transfer bulk data between Hadoop and relational databases.

  • It supports importing and exporting data.

  • It connects databases such as MySQL and Oracle with Hadoop.


28
New cards

IBM Big Data Strategy

IBM Big Data Strategy is an approach developed by IBM (International Business Machines) to collect, store, manage, process, and analyze large volumes of structured, semi-structured, and unstructured data using advanced technologies and analytical tools.

Its main objective is to convert large amounts of raw data into meaningful information for better decision-making and business growth.

29
New cards

Main Components of IBM Big Data Strategy

1. Hadoop-Based Data Processing (InfoSphere BigInsights)

  • Provides a Hadoop-based platform for storing and processing large datasets.

  • Supports distributed storage and parallel processing.

  • Helps analyze structured and unstructured data.



2. Stream Computing (InfoSphere Streams)

  • Processes continuously generated data in real time.

  • Analyzes data from sources such as sensors, financial transactions, and social media.

  • Helps organizations respond quickly to changing conditions.



3. Data Warehousing

  • Stores and manages structured business data for reporting and analysis.

  • Supports complex queries and business intelligence applications.

  • Helps organizations analyze historical data and identify trends.



4. Information Integration and Governance

  • Integrates data from multiple sources into a consistent format.

  • Helps maintain data quality, security, and reliability.

  • Supports effective management of organizational data.



5. Advanced Analytics

  • Uses statistical analysis, data mining, and machine learning to extract useful insights.

  • Helps identify patterns and predict future outcomes.

  • Supports better business planning and decision-making.


30
New cards

Introduction to InfoSphere BigInsights

IBM InfoSphere BigInsights is a Hadoop-based Big Data analytics platform developed by IBM that provides tools for storing, processing, managing, and analyzing large volumes of structured, semi-structured, and unstructured data.

It extends Apache Hadoop by providing additional capabilities for data analysis, management, and visualization.

31
New cards

. Features of IBM InfoSphere BigInsights

1. Hadoop Integration: Uses Apache Hadoop for distributed storage and processing of large datasets.


2. Scalability: Supports increasing data volumes by adding more computing resources.


3. Data Analysis: Provides tools for analyzing structured and unstructured data.


4. Data Visualization: Supports the presentation of analyzed data in an understandable format.


5. Data Management: Provides capabilities for managing, accessing, and processing large datasets.


6. Advanced Analytics: Supports analytical operations such as text analysis and pattern identification.

32
New cards

IBM BigSheets

BigSheets is a tool that makes Big Data analysis easier by providing an interface similar to a spreadsheet.


Users can analyze large datasets without writing complex Hadoop or MapReduce programs.


It allows users to organize data, apply filters, perform calculations, and visualize results.


BigSheets is particularly useful for business analysts who want to analyze data without extensive programming knowledge.

33
New cards

Working of IBM BigSheets

Data Collection

↓

Data Storage in Hadoop HDFS

↓

Data Access Through BigSheets

↓

Data Filtering and Transformation

↓

Data Analysis

↓

Data Visualization

↓

Useful Insights