TOPIC 3

Topic 3: Big Data Platforms and Storage

5V's of Big Data

1. Volume – Scale of Data
  • Definition: The term ‘Big Data’ refers to a significantly large amount of data, where the size plays a critical role in determining its value.
  • Importance: Whether data is classified as Big Data depends on its volume.
  • Example: In 2016, global mobile traffic was estimated at 6.2 Exabytes (6.2 billion GB) per month. Projections for 2020 anticipate approximately 40,000 Exabytes of data.
2. Velocity – Speed of Data
  • Definition: Velocity indicates the speed at which data is accumulated.
  • Data Sources: Data flows continuously from sources such as machines, networks, social media, and mobile phones, indicating a massive influx.
  • Potential: The speed of data generation and processing is crucial for meeting demands. Sampling data can mitigate challenges associated with high velocity.
  • Example: Google processes over 3.5 billion searches per day; Facebook experiences an approximate 22% annual user growth.
3. Variety – Diversity of Data
  • Definition: Variety refers to the different types of data, including structured, semi-structured, and unstructured data, which come from diverse sources.
  • Types of Data:
    • Structured Data: Organized data with defined length and format.
    • Semi-Structured Data: Data that does not conform to a formal structure, such as log files.
    • Unstructured Data: Unorganized data that does not fit neatly into traditional databases, including texts, images, and videos.
4. Veracity – Trustworthiness of Data
  • Definition: Veracity relates to the trust and quality of data, acknowledging the inconsistencies and uncertainties inherent in large datasets.
  • Challenges: High volume data can create confusion, while smaller datasets may provide incomplete information.
5. Variability – Inconsistency of Data
  • Definition: Variability denotes the changes in data flow rates, formats, or meanings over time.
  • Examples:
    • Taste shifts in a favorite ice cream can be interpreted as variability.
    • A tweet containing “bad” may carry different meanings depending on context.
6. Value – Usefulness of Data
  • Definition: Value reflects the usefulness of data; raw data without transformation adds no value to businesses.
  • Significance: Emphasizes that the conversion of data into meaningful information is crucial, thereby positioning Value as the most significant of the 6 V's.

The Need for Distributed Storage

  • Volume necessitates distributed storage due to the sheer amount of data.
  • Velocity requires fast read/write operations to manage data inflow efficiently.
  • Variety calls for flexible schemas to accommodate different data types.

Distributed Storage Systems

  • Overview: Distributed storage systems are essential for handling large data volumes effectively and reliably.
  • Advantages:
    • Enhanced management of vast data quantities.
    • Improved safety against potential data loss due to failures in one part of the system.

Definition of Distributed Storage System

  • Description: A distributed storage system is an infrastructure designed for data storage and management across multiple interconnected nodes or servers, as opposed to a conventional centralized storage system.
  • Characteristics:
    • Architecture: Data is either replicated or partitioned across multiple nodes.
    • Resilience: System resilience against failures is enhanced due to decentralization, eliminating single points of failure.

Types of Distributed Storage Systems

1. Block Repository
  • Definition: A block repository stores data in fixed-sized blocks ranging from kilobytes to several megabytes.
  • Features:
    • Each block is treated as a separate entity with its own address.
    • Commonly used in cloud computing and virtual infrastructures requiring raw storage access.
    • Suited for high-performance applications such as databases needing efficient data access.
2. File Repository
  • Definition: A distributed file system or file repository manages files among multiple nodes.
  • Features:
    • Organizes files similarly to traditional file systems with directories and subdirectories.
    • Supports safe collaboration via metadata management and access control.
  • Examples: Hadoop Distributed File System (HDFS), Google File System (GFS), Lustre.
3. Object Repository
  • Definition: An object repository is designed to store objects, which consist of data and associated metadata.
  • Features:
    • Each object can include unstructured data like documents and images.
    • Supports versioning, replication, and lifecycle management.

Quick Comparison Table: Types of Distributed Storage Systems

FeatureBlock StorageFile StorageObject Storage
StructureBlocksFiles & foldersObjects (flat)
MetadataNoYesRich metadata
SpeedVery fastModerateModerate
ScalabilityLimitedModerateVery high
AccessLow-levelPath-basedAPI-based
Best forDatabasesShared filesBig data, cloud

What is Metadata?

  • Simple Definition: Metadata refers to "data about data" and serves to describe the content rather than containing it.
In Storage Systems Context
  1. Block Storage: No metadata, only block numbers; the system lacks understanding of data content.
    • Example: For a file named "research paper.pdf", the metadata is just the file name itself.
  2. File Storage: Basic metadata includes filename, size, and permissions.
  3. Object Storage: Rich metadata allows for custom tags and provides comprehensive file identification and management capabilities.
    • Example: Metadata for a file may include:
      • Size: 2 MB
      • Created date: 10 March 2026
      • Author: Neha Gautam
      • File type: PDF

Architectures of Distributed Storage Systems

1. Replication-based Architecture
  • Definition: Data is replicated across multiple nodes to ensure fault tolerance.
  • Types:
    • Synchronous Replication: Requires acknowledgment from all nodes before completing the write operation, ensuring data consistency but potentially introducing latency.
    • Asynchronous Replication: Acknowledges write operation immediately after it’s written to the primary node, allowing lower latency but risking inconsistency if the primary fails before copies update.
2. Sharding Architecture
  • Definition: Partitioning data into smaller segments called shards across different nodes enhances scalability and load distribution.
  • Mechanisms: Includes routing for identifying shards from which data is retrieved and coordination for maintaining data consistency during migrations and rebalancing.
3. Distributed File System (DFS)
  • Overview: A DFS permits multiple users to access and manage files seamlessly across several machines.
  • Features:
    • Provides a unified view of file storage across various servers.
    • Enables parallel access to improve performance.
    • Examples: HDFS, GFS.
4. Object Storage Architecture
  • Definition: Stores data as objects integrated with metadata and a unique identifier in a flat hierarchy.
  • Applications: Ideal for unstructured data and scalable solutions.
  • Examples: OpenStack Swift, Azure Blob Storage, Amazon S3.

Scalability in System Design

  • Definition: Scalability is the capacity of a system to manage increased workloads without compromising performance.
  • Benefits:
    • Supports more users and traffic;
    • Maintains stable response times during workload increases;
    • Ensures reliability during high demand.

Reliability in System Design

  • Definition: The ability of a system to function correctly under varying conditions.
  • Key Considerations:
    • Data replication for fault tolerance;
    • Fault tolerance strategies to recover from crashes;
    • Consistency guarantees across distributed nodes;
    • Efficient failure detection and recovery mechanisms.

Limitations of DFS

  • Limitations:
    • Good for storage, but not conducive for fast queries or flexible data models.
    • Not suitable for real-time applications.
  • Challenge: Highlights the necessity for NoSQL databases.

NoSQL Databases

  • Definition: NoSQL databases are designed for managing large volumes of unstructured and semi-structured data, offering flexible schemas and high scalability.
Features of NoSQL Databases
  1. Dynamic Schema: Accommodate flexible data structures without necessitating schema alterations.
    • Importance: Reduces development time by avoiding costly migrations when data structures change.
    • Example: MongoDB allows different user data structures without schema change.
  2. Performance: Optimization for high-speed operations enhances real-time analytics capabilities.
    • Example: Redis for caching, enabling rapid data retrieval.
  3. Flexibility: Supports various data types and structures for adaptable data management.
    • Example: Nested structures in MongoDB.
  4. Distributed and High Availability: Automatic replication and fault tolerance across nodes ensure zero downtime.
    • Example: Apache Cassandra maintains data availability even during server crashes.
  5. Horizontal Scalability: Allows addition of nodes to manage increased load without physical server upgrades.
    • Example: Apache Cassandra for scaling clusters under heightened traffic.
Conclusion
  • The efficient management of data through advanced storage solutions, including distributed storage systems and NoSQL databases, addresses the evolving challenges of big data, ensuring scalability, reliability, and value extraction from complex datasets.