ch.1 what are data and data science?

Data science

The field of study that investigates how to collect, manage, and analyze all types of data in order to retrieve meaningful information

Data

Any pieces of evidence or observations that can be analyzed to provide some insights

How was data science handled in the past?

It was spread across multiple subjects and it was thought that different people should do a different aspect of data science

What subjects was data science spread around?

Including, but not limited to statistics, math, computer science, and social science

Who was thought to be qualified to handle data collection?

So-called domain experts, like psychologists for psych data and economists for economic data, as they had the full context of the data

Who was thought to be qualified to handle data management?

Computer scientists and engineers as they knew how to store and process data in computing systems

Who was thought to be qualified to handle data analysis?

Statisticians and mathematicians as they knew how to derive meaningful insights from the data

How did tech advancements affect data science?

They brought about the proliferation of data, muddying the boundaries between different specialties

Nowadays, in what areas are all data scientists expected to have some expertise?

Data analysis, data collection, and data management

How do smartphones affect data science?

They contain and log a tremendous amount of data; a smartphone app is more efficient at collecting health data than a health professional, from a data collection perspective

Data science cycle

Problem definition, data collection, preparation, analysis, and reporting

Which parts of the process are the most time and effort consuming?

Data collection and preparation

How long will data collection and cleaning take?

Takes up half the process, half the time

How much of the process do data analysis and communication take up?

1/3 each

What is needed to gain clear objectives for the goal and scope of the project?

A precise definition of the problem statement

Data collection

The systematic process of gathering information on variables of interest

Who often collects data?

Domain experts

Why is data often collected?

To find answers to predefined problems

When it is not collected purposefully, why is data collected?

It is a byproduct of our daily activities and is kept because it may be useful later

In what state is data often found?

Non-optimal form for analysis

Data preparation or data processing

Data is processed so it can be analyzed

Data analysis

Once the data is prepared, it must be analyzed to discover meaningful insights

What methods to data analysts often choose?

They start with simple methods and move on to more advanced ones if they want to investigate further

Data reporting

Involves the presentation of data in a way that will convey information learned from analysis

What do domain experts have to do with data reporting?

It's important to present the data in a way that domain experts can understand

Data visualization

A graphical way of presenting and reporting data to point out the hidden patterns, trends, and hidden insights

What does data visualization involve the use of?

Visual elements such as charts, graphs, and maps to present data in a way that is easy to comprehend and analyze

What is the goal of data visualization?

To communicate information effectively and facilitate better decision making

What was data management like in the early days?

There wasn’t so much data and it was possible to store all data locally and process it locally as well

Why were cloud-based management systems created?

To help manage the vast amounts of data that later became available

Where are cloud-based management systems kept?

on a designated server outside of a local computer

What happened as data processing became more time consuming?

Cloud-based management systems evolved to handle some automatic data processing

Data warehousing

Systems that store and manage large volumes of data from various sources in a central location

What benefit does data warehousing give?

Efficient retrieval and analysis for business intelligence and decision-making

What do enterprises do in order to warehouse data?

They subscribe to cloud warehouse services

What are some cloud warehouse services?

Amazon RedShift(runs on the Amazon WebServices) and Google BigQuery(runs on Google Cloud)

What does data warehousing replace?

Companies don’t buy physical storage or configure data management/processing systems on their own

What kind of data can be generated within a business?

Data based on customers, products, and sales

Business analytics

Investigate business data to launch new products and to maximize their business revenue and profit

Big data

Unstructured data from millions of customers

How is detecting and managing fraudulent transactions nowadays?

With automated machine learning algorithms

Walmart example for business analytics

They used date science to figure out which products were most popular during hurricane season and stocked extra units

Amazon example for business analytics

They figured out in which regions items were most popular so they could warehouse those items there and reduce delivery times

Internet of things(IoT)

The network of electronics that communicate with each other through the internet

Why is data science important to IoT?

Most objects will perform tasks based on the data that another object will gather

AI irrigation system

There is a control tower in Idaho that analyzes how much water plants use, which allows farmers to conserve more water

Why is the AI irrigation system so useful to the state of Idaho?

Idaho gets a lot of their GDP from agriculture, so this tech is a boon to the state

Why are smart cities more efficient?

They used a lot of automated services that analyze various aspects like energy usage and the fullness of garbage cans

How has data science benefitted education?

Various online apps have allowed the tracking of individual students’ progress which allows for customized teaching strategies to better teach each student