Definition: The term ‘Big Data’ refers to a significantly large amount of data, where the size plays a critical role in determining its value.
Importance: Whether data is classified as Big Data depends on its volume.
Example: In 2016, global mobile traffic was estimated at 6.2 Exabytes (6.2 billion GB) per month. Projections for 2020 anticipate approximately 40,000 Exabytes of data.
2. Velocity – Speed of Data
Definition: Velocity indicates the speed at which data is accumulated.
Data Sources: Data flows continuously from sources such as machines, networks, social media, and mobile phones, indicating a massive influx.
Potential: The speed of data generation and processing is crucial for meeting demands. Sampling data can mitigate challenges associated with high velocity.
Example: Google processes over 3.5 billion searches per day; Facebook experiences an approximate 22% annual user growth.
3. Variety – Diversity of Data
Definition: Variety refers to the different types of data, including structured, semi-structured, and unstructured data, which come from diverse sources.
Types of Data:
Structured Data: Organized data with defined length and format.
Semi-Structured Data: Data that does not conform to a formal structure, such as log files.
Unstructured Data: Unorganized data that does not fit neatly into traditional databases, including texts, images, and videos.
4. Veracity – Trustworthiness of Data
Definition: Veracity relates to the trust and quality of data, acknowledging the inconsistencies and uncertainties inherent in large datasets.
Challenges: High volume data can create confusion, while smaller datasets may provide incomplete information.
5. Variability – Inconsistency of Data
Definition: Variability denotes the changes in data flow rates, formats, or meanings over time.
Examples:
Taste shifts in a favorite ice cream can be interpreted as variability.
A tweet containing “bad” may carry different meanings depending on context.
6. Value – Usefulness of Data
Definition: Value reflects the usefulness of data; raw data without transformation adds no value to businesses.
Significance: Emphasizes that the conversion of data into meaningful information is crucial, thereby positioning Value as the most significant of the 6 V's.
The Need for Distributed Storage
Volume necessitates distributed storage due to the sheer amount of data.
Velocity requires fast read/write operations to manage data inflow efficiently.
Variety calls for flexible schemas to accommodate different data types.
Distributed Storage Systems
Overview: Distributed storage systems are essential for handling large data volumes effectively and reliably.
Advantages:
Enhanced management of vast data quantities.
Improved safety against potential data loss due to failures in one part of the system.
Definition of Distributed Storage System
Description: A distributed storage system is an infrastructure designed for data storage and management across multiple interconnected nodes or servers, as opposed to a conventional centralized storage system.
Characteristics:
Architecture: Data is either replicated or partitioned across multiple nodes.
Resilience: System resilience against failures is enhanced due to decentralization, eliminating single points of failure.
Types of Distributed Storage Systems
1. Block Repository
Definition: A block repository stores data in fixed-sized blocks ranging from kilobytes to several megabytes.
Features:
Each block is treated as a separate entity with its own address.
Commonly used in cloud computing and virtual infrastructures requiring raw storage access.
Suited for high-performance applications such as databases needing efficient data access.
2. File Repository
Definition: A distributed file system or file repository manages files among multiple nodes.
Features:
Organizes files similarly to traditional file systems with directories and subdirectories.
Supports safe collaboration via metadata management and access control.
Examples: Hadoop Distributed File System (HDFS), Google File System (GFS), Lustre.
3. Object Repository
Definition: An object repository is designed to store objects, which consist of data and associated metadata.
Features:
Each object can include unstructured data like documents and images.
Supports versioning, replication, and lifecycle management.
Quick Comparison Table: Types of Distributed Storage Systems
Feature
Block Storage
File Storage
Object Storage
Structure
Blocks
Files & folders
Objects (flat)
Metadata
No
Yes
Rich metadata
Speed
Very fast
Moderate
Moderate
Scalability
Limited
Moderate
Very high
Access
Low-level
Path-based
API-based
Best for
Databases
Shared files
Big data, cloud
What is Metadata?
Simple Definition: Metadata refers to "data about data" and serves to describe the content rather than containing it.
In Storage Systems Context
Block Storage: No metadata, only block numbers; the system lacks understanding of data content.
Example: For a file named "research paper.pdf", the metadata is just the file name itself.
File Storage: Basic metadata includes filename, size, and permissions.
Object Storage: Rich metadata allows for custom tags and provides comprehensive file identification and management capabilities.
Example: Metadata for a file may include:
Size: 2 MB
Created date: 10 March 2026
Author: Neha Gautam
File type: PDF
Architectures of Distributed Storage Systems
1. Replication-based Architecture
Definition: Data is replicated across multiple nodes to ensure fault tolerance.
Types:
Synchronous Replication: Requires acknowledgment from all nodes before completing the write operation, ensuring data consistency but potentially introducing latency.
Asynchronous Replication: Acknowledges write operation immediately after it’s written to the primary node, allowing lower latency but risking inconsistency if the primary fails before copies update.
2. Sharding Architecture
Definition: Partitioning data into smaller segments called shards across different nodes enhances scalability and load distribution.
Mechanisms: Includes routing for identifying shards from which data is retrieved and coordination for maintaining data consistency during migrations and rebalancing.
3. Distributed File System (DFS)
Overview: A DFS permits multiple users to access and manage files seamlessly across several machines.
Features:
Provides a unified view of file storage across various servers.
Enables parallel access to improve performance.
Examples: HDFS, GFS.
4. Object Storage Architecture
Definition: Stores data as objects integrated with metadata and a unique identifier in a flat hierarchy.
Applications: Ideal for unstructured data and scalable solutions.
Definition: Scalability is the capacity of a system to manage increased workloads without compromising performance.
Benefits:
Supports more users and traffic;
Maintains stable response times during workload increases;
Ensures reliability during high demand.
Reliability in System Design
Definition: The ability of a system to function correctly under varying conditions.
Key Considerations:
Data replication for fault tolerance;
Fault tolerance strategies to recover from crashes;
Consistency guarantees across distributed nodes;
Efficient failure detection and recovery mechanisms.
Limitations of DFS
Limitations:
Good for storage, but not conducive for fast queries or flexible data models.
Not suitable for real-time applications.
Challenge: Highlights the necessity for NoSQL databases.
NoSQL Databases
Definition: NoSQL databases are designed for managing large volumes of unstructured and semi-structured data, offering flexible schemas and high scalability.
Features of NoSQL Databases
Dynamic Schema: Accommodate flexible data structures without necessitating schema alterations.
Importance: Reduces development time by avoiding costly migrations when data structures change.
Example: MongoDB allows different user data structures without schema change.
Performance: Optimization for high-speed operations enhances real-time analytics capabilities.
Example: Redis for caching, enabling rapid data retrieval.
Flexibility: Supports various data types and structures for adaptable data management.
Example: Nested structures in MongoDB.
Distributed and High Availability: Automatic replication and fault tolerance across nodes ensure zero downtime.
Example: Apache Cassandra maintains data availability even during server crashes.
Horizontal Scalability: Allows addition of nodes to manage increased load without physical server upgrades.
Example: Apache Cassandra for scaling clusters under heightened traffic.
Conclusion
The efficient management of data through advanced storage solutions, including distributed storage systems and NoSQL databases, addresses the evolving challenges of big data, ensuring scalability, reliability, and value extraction from complex datasets.