ISIT312 - Big Data Management

What is a Computer Cluster?

• Collection of computers (nodes) that are connected through high speed local area networks that work together to simulte a single more powerful computer system

• Each node is controlled by its own OS and performs a different version of the same tasks

• It can range from a simple two-node system to a super computer with a cluster architecture

What are Computer Clusters used for?

• Used to speed up computing through sharding (shared nothing) partitioning of data and parallelisation of data processting on the cluster’s nodes

• They provide high availability through automatic replacement of a failed node with a replica node

Advantages:

• Faster processing speed,

• Large storage capacity

• Better data integrity

• Greater reliability

• Wider availability of resources

What is a Cluster Computing?

• Process of sharing the comptuation tasks among multiple computers included in a computer cluster

• Good paradigm for processing large scale science, engineering and commercial applications

• Requires specialsed algorithms like load balancing, resource sharing and resource scheduling for optimisation and data processing

• Simplest congfiuration: master node and slave nodes

Advantages

• Cost efficiency

• Processing speed

• Expandability

• High availability of resources

What is Big Data?

• It is so big that it cannot be stored on the persistent storage devices attached to a single computer system

• May also mean an infine amount of data

e.g.

• Clickstream data

• Call centre data

• Email and messaging

• Sensor data

• Unstructured *

• Geographic *

• Satellite *

• Image *

• Temporal *

What are the 3 main sources of Big Data?

• Business process data: databases, transactions and business operations

• Enterprise & human data: websites, emails, social media and videos

• Machine data: sensors, logs, recordings, images, satellite data and videos

What are the V’s the characterise Big Data?

Main 3:

• Volume: rows and columns

• Variety: complexity of data types and structures

• Velocity: speed of new data creation and growth

Additional ones:

• Veracity: ability to represent and process uncertain and imprecise data

• Value

• Viability: benefits we can get from data analysis

• Vagueness: meaning of found data can be unclear

• Validity

• Vane

What are the strengths of traditional data architectures?

• Centralised governance of data repositories

• Inquires are performed regularly

• Optimisation

• Security and access control

• Fault-tolerance and back up

What are the challenges for traditional data architectures?

• Unstrcuted and semi-structued data

• Large amounts of data flowing into organisations

• Non-traditional NoSQL databases

• Increasing cost of storing and analysing the large amounts of data

• Increasing use of data analytics, which needs significant storage and processing capabilities

What is Hadoop and its components and features?

• This is a project that develops open-source software for reliable, scalable and dsitributed computing

Components

• Different data-processing frameworks

• YARN (OS)

• HDFS (Disttributed File System)

Features

• Capability to handle large data sets

• File size from Gbs to Tbs

• Store millions of files

• High fault tolerance

• Supports data replication, streaming access to data, batch processing and interactive, iterative and stream processing

• Implements write-once-read-many access model

• Runs on commodity hardware

• Inexpensive

• Deployed on premises or in the cloud

What is the master-slave architecture of Hadoop?

• Set of master node servers: this is where the daemons support key frameworks

• Set of worker nodes: host storage (HDFS) and computing (YARN) work

• One or more edge serves: acesses the Hadoop cluster to launch applications

• One or more relational databases for storing metadata repositories

• Dedicated servers for special framework

What is the pseudo-distributed mode of Hadoop?

• All HDFS and YARN daemons run on a single node

• Simulates full cluster

• Easy for beginner’s

• Easy for testing and debug

This is the lab work’s setting

What is a Database and why is it used in Big Data?

• A collection of databases managed by one running database server

How it works:

• A very large database in a cluster is partitioned (sharded) over a number of smaller databases each located on a separate node of a cluster

• Requires replication and sharding

Improves

• Performance

• Availability

• Scalability

What is Kubernetes (K8) in Big Data?

• A orchestration container or microserve platform that manages any containerised application (computing, networking and storage)

• Consists of a single maser node and multiple worker nodes

Benefits

• Horizontal calling

• Automated rollouts and rollbacks

• Service discovery and load balancing

• Self healing

• Batch execution

• Automatic binpacking