Data Analytics

Characteristics of big data:

1.      Volume: the sheer amount of data being generated. Traditional practices cannot handle this huge amount of data.

2.      Velocity: defines the speed at which data is generated and needs to be processed.

3.      Variety: a wide variety of data formats that need to be processed.

6 challenges with big data:

1.      Emerging growth from unstructured data and NoSQL databases.

2.      Data storage.

3.      Admin may try to mine data without permission.

4.      The sheer size of big data makes regular data audits impossible.

5.      Data is at risk unless the environment is updated frequently.

6.      IT security must be proficient in cleaning, deleting malware, and restoring system operation.

4 Big Data Security Tools:

1.      Authentication: the process of verifying the identity of a user, device, or entity attempting to access a system or resource. It ensures that the person or system is who they claim to be. Examples: Passwords, Biometrics, Multi-Factor Authentication (MFA): A combination of two or more authentication methods

2.      Authorisation: the process of determining what an authenticated user or system is allowed to do. It defines the permissions and access levels assigned to users or entities. Examples Access Control Lists (ACLs), Role-Based Access Control (RBAC), Policy-Based Access Control (PBAC):

3.      Centralised administration and audit refer to the management and monitoring of security policies, configurations, and logs from a single, central point within an organization. This approach ensures consistency and simplifies the management of security practices. Examples Centralized Management Console, Audit Logs, and Compliance Reporting.

4.      Data at rest/in motion encryption: the process of converting data into a coded form to prevent unauthorized access. It can be applied to data at rest and data in motion. Examples File-Level Encryption, Disk Encryption, Database Encryption, SSL: Protocols for encrypting data sent over the internet, Virtual Private Networks that create secure tunnels for data transmission, Email Encryption.

5 big data security Technologies:

1.     Encryption: the process of converting data into coded form to prevent unauthorized users in data at rest and in motion.

2.     centralized key management: the process of managing cryptographic keys from a central location within an organization.

-         ensures that keys are securely generated, stored, and distributed while maintaining strict access control and audit capabilities.

3.     user access control refers to the mechanisms and policies used to manage and restrict the access rights of users to various resources within an information system.

4.     Intrusion detection and prevention are critical components of an organization’s security infrastructure, designed to detect, analyze, and respond to unauthorized activities within a network or system.

5.     Physical security: involves protecting an organization's personnel, equipment, and data from physical actions and events that could cause serious loss or damage. This includes protection against natural disasters, unauthorized access, theft, and vandalism—examples of CCTV Cameras, Motion Detectors, and Security Patrols.

Principal Component Analysis:

-         A Major step in big data is pre-processing,

-         It is the pre-processing of cleaning and reducing data so that it can be analyzed.

-         reduce the dimensionality and size of datasets by altering a huge set of variables into a smaller set of variables that still contains most of the information contained in the larger original set (reduce a Huge set of variables of data into a small set of variables).

-         NB: One primary method.

Steps of Principal Component Analysis:

1.     Standardize the range of continuous initial variables.

2.     Compute the covariance matrix to find correlations.

3.     Identify the principal components using the covariance matrix.

4.     Create a feature vector to determine which principal components to retain

5.     Recast data along the axes of the principal components

Standardization:

-         standardize the range of a variety of continuous variables so that each of them contributes equally to the analysis.

-         conversion may be done using the following formula:

Z(standard value)=(value-mean)/standard deviation

Covariance Matrix Computation stage:

-         determine if there are any relationships between variables of the input set.

CNN (Convolutional Neural Networks):

-         specialized type of deep learning algorithm and architecture designed for tasks that require object recognition such as image classification, detection, and segmentation.

Components of CNN:

1.      Convolutional Layers: to automatically detect and learn features from input data.

2.      Rectified Linear Unit (known as RELU): Introduces non-linearity by setting all negative values to zero.

3.      Pooling Layers: used to reduce the spatial dimensions of the feature maps generated by convolutional layers (This reduces computational load).

4.     Fully Connected Layer: Interprets the features extracted by previous layers and makes the final prediction.

5.     Output Layer: Produces the final classification or regression output.

Advantages of CNN:

-         Automatic feature extraction.

-         Can extract features without pre-processing the image.

-         Versatile

Hadoop:

-         open-source framework developed by the Apache Software Foundation.

-         allows for the distributed storage and processing of large datasets across clusters of computers.

advantages in storage processing:

-        It can handle petabytes and exabytes of data, making it suitable for large-scale data storage needs.

-        As an open-source project, Hadoop eliminates the need for costly proprietary software licenses.

-        Hadoop Distributed File System (HDFS) automatically replicates data across multiple nodes, ensuring that data is not lost in case of node failures.

-        Hadoop can store and process structured, semi-structured, and unstructured data, making it versatile for various use cases.

-        Hadoop can handle large volumes of data quickly and efficiently.

Disadvantages of Hadoop:

-         Hadoop requires a significant amount of technical knowledge to set up, configure, and manage.

-         Managing a Hadoop cluster can be complex and time-consuming, requiring skilled administrators to handle tasks like resource allocation, node management, and troubleshooting.

-         The NameNode is a single point of failure in HDFS.

-         Integrating Hadoop with existing enterprise systems and databases can be challenging and may require additional connectors or custom development.

-         Migrating existing data into Hadoop can be a complex process, involving data transformation, cleansing, and loading procedures.