Chapter 8_Big Data Analytics

Big Data Analytics

• it is the use of advanced analytic techniques against very large, diverse big data sets that include structured, semi-structured and unstructured data, from different sources, and in different sizes from terabytes to zettabytes.

• you can ultimately fuel better and faster decision-making, modelling and predicting of future outcomes and enhanced business intelligence.

• refers to collecting, processing, cleaning, and analyzing large datasets to help organizations operationalize their big data.

Big Data

• It can be defined as data sets whose size or type is beyond the ability of traditional relational databases to capture, manage and process the data with low latency.

• refers to the massive volume of structured, semi-structured, and unstructured data that inundates a business on a day-to-day basis.

• It comes from a variety of sources, including social media, sensors, devices, transactions, and more.

• is characterized by the volume, velocity, and variety of data generated, which traditional data processing methods struggle to handle efficiently.

high volume,

high velocity,

high variety

Characteristics of big data (3)

Data analytics

• is the process of examining data sets to uncover hidden patterns, correlations, trends, and other useful information.

• It involves various techniques and tools to turn raw data into actionable insights

leverage their data,

increasing efficiency,

reduce costs

Why is big data analytics important?

• Big data analytics is important because it helps companies _____ to identify opportunities for improvement and optimization.

• Across different business segments, _____ leads to overall more intelligent operations, higher profits, and satisfied customers.

• Big data analytics helps companies _____ and develop better, customer-centric products and services.

Gain Insights,

Improve Decision-Making,

Enhance Customer Experience,

Optimize Operations,

Predict Trends and Outcomes

Applications of Big Data and Data Analytics (5)

Gain Insights

By analyzing large datasets, organizations can uncover valuable insights that can inform decision-making, improve processes, and drive innovation

Improve Decision-Making

Data analytics provides decision-makers with the information they need to make informed decisions, reducing uncertainty and improving the likelihood of success.

Enhance Customer Experience

By analyzing customer data, organizations can better understand customer needs, preferences, and behaviors, allowing them to tailor products, services, and marketing strategies to meet those needs effectively.

Optimize Operations

Big data and analytics can help organizations optimize various aspects of their operations, such as supply chain management, resource allocation, and risk management, leading to increased efficiency and cost savings.

Predict Trends and Outcomes

Predictive analytics enables organizations to forecast future trends and outcomes based on historical data, allowing them to anticipate market changes, identify opportunities, and mitigate risks.

Collect Data,

Process Data,

Clean Data,

Analyze Data

How big data analytics works (3)

Collect Data

• Data collection looks different for every organization.

  • With today's technology, organizations can gather both structured and unstructured data from a variety of sources from cloud storage to mobile applications to in-store IoT sensors and beyond.

• Some data will be stored in data warehouses where business intelligence tools and solutions can access it easily.

  • Raw or unstructured data that is too diverse or complex for a warehouse may be assigned metadata and stored in a data lake.

Process Data

• Once data is collected and stored, it must be organized properly to get accurate results on analytical queries, especially when it's large and unstructured.

  • Available data is growing exponentially, making data processing a challenge for organizations.

• One processing option is batch processing, which looks at large data blocks over time. Batch processing is useful when there is a longer turnaround time between collecting and analyzing data.

  • Stream processing looks at small batches of data at once, shortening the delay time between collection and analysis for quicker decision-making. Stream processing is more complex and often more expensive.

Clean Data

• Data big or small requires scrubbing to improve data quality and get stronger results; all data must be formatted correctly, and any duplicative or irrelevant data must be eliminated or accounted for.

• Dirty data can obscure and mislead, creating flawed insights.

Analyze Data

Getting big data into a usable state takes time. Once it's ready, advanced analytics processes can turn big data into big insights.

Data mining,

Predictive analytics,

Deep learning

Big data analysis methods (3)

Data mining

sorts through large datasets to identify patterns and relationships by identifying anomalies and creating data clusters.

Deep learning

imitates human learning patterns by using artificial intelligence and machine learning to layer algorithms and find patterns in the most complex and abstract data.

descriptive analytics,

diagnostic analytics,

predictive analytics,

prescriptive analytics

Data analytics approaches/types of big data analytics (4)

Descriptive analytics

• what happened

• refers to data that can be easily read and interpreted.

• This data helps create reports and visualize information that can detail company profits and sales.

• focuses on summarizing historical data in a way that's easy to read, interpret, and act upon.

• It helps organizations understand what has happened by creating clear reports and visualizations.

Diagnostics analytics

• why it happened

• helps companies understand why a problem occurred.

• Big data technologies and tools allow users to mine and recover data that helps dissect an issue and prevent it from happening in the future.

Predictive analytics

• what will happen

• uses an organization's historical data to make predictions about the future, identifying upcoming risks and opportunities.

• looks at past and present data to make predictions.

• With artificial intelligence (AI), machine learning, and data mining, users can analyze data to predict market trends.

Prescriptive analytics

• what should be done about it

• provides a solution to a problem, relying on AI and machine learning to gather data and use it for risk management.

Faster, better decision making,

Cost reduction and operational efficiency,

Improved data-driven go to market

Benefits of Big Data Analytics (3)

Faster, better decision making

• Businesses can access a large volume of data and analyze a large variety sources of data to gain new insights and take action.

• Get started small and scale to handle data from historical records and in real-time.

Cost reduction and operational efficiency

• Flexible data processing and storage tools can help organizations save costs in storing and analyzing large amounts of data.

• Discover patterns and insights that help you identify do business more efficiently.

Improved data-driven go to market

• Analyzing data from sensors, devices, video, logs, transactional applications, web and social media empowers an organization to be data-driven.

• Gauge customer needs and potential risks and create new products and services

Hadoop,

Spark,

Data integration software,

Stream analytics tools,

Distributed storage,

Predictive analytics hardware and software,

Data mining tools,

NoSQL databases,

Data warehouses

Big Data Analytics tools (9)

Hadoop

• An open-source framework that stores and processes big data sets.

• is able to handle and analyze structured and unstructured data.

Spark

An open-source cluster computing framework used for real-time processing and analyzing data.

Data integration software

Programs that allow big data to be streamlined across different platforms, such as MongoDB, Apache, Hadoop, and Amazon EMR.

Stream analytics tools

Systems that filter, aggregate, and analyze data that might be stored in different platforms and formats, such as Kafka.

Distributed storage

Databases that can split data across multiple servers and have the capability to identify lost or corrupt data, such as Cassandra.

Predictive analytics hardware and software

Systems that process large amounts of complex data, using machine learning algorithms to predict future outcomes, such as fraud detection, marketing, and risk assessments.

Data mining tools

Programs that allow users to search within structured and unstructured big data.

NoSQL databases

Non-relational data management systems ideal for dealing with raw and unstructured data.

Data warehouses

Storage for large amounts of data collected from many different sources, typically using predefined schemas.

Making big data accessible,

Maintaining quality data.

Keeping data secure,

Finding the right tools and platforms

Big challenges of big data

Making big data accessible

Collecting and processing data becomes more difficult as the amount of data grows. Organizations must make data easy and convenient for data owners of all skill levels to use

Maintaining quality data

With so much data to maintain, organizations are spending more time than ever before scrubbing for duplicates, errors, absences, conflicts, and inconsistencies.

Keeping data secure

As the amount of data grows, so do privacy and security concerns. Organizations will need to strive for compliance and put tight data processes in place before they take advantage of big data.

Finding the right tools and platforms

• New technologies for processing and analyzing big data are developed all the time.

• Organizations must find the right technology to work within their established ecosystems and address their needs.

• Often, the right solution is also a flexible solution that can accommodate future infrastructure changes.