ch.1 what are data and data science?

Data science

The field of study that investigates how to collect, manage, and analyze all types of data in order to retrieve meaningful information

Data

Any pieces of evidence or observations that can be analyzed to provide some insights

How was data science handled in the past?

It was spread across multiple subjects and it was thought that different people should do a different aspect of data science

What subjects was data science spread around?

Including, but not limited to statistics, math, computer science, and social science

Who was thought to be qualified to handle data collection?

So-called domain experts, like psychologists for psych data and economists for economic data, as they had the full context of the data

Who was thought to be qualified to handle data management?

Computer scientists and engineers as they knew how to store and process data in computing systems

Who was thought to be qualified to handle data analysis?

Statisticians and mathematicians as they knew how to derive meaningful insights from the data

How did tech advancements affect data science?

They brought about the proliferation of data, muddying the boundaries between different specialties

Nowadays, in what areas are all data scientists expected to have some expertise?

Data analysis, data collection, and data management

How do smartphones affect data science?

They contain and log a tremendous amount of data; a smartphone app is more efficient at collecting health data than a health professional, from a data collection perspective

Data science cycle

Problem definition, data collection, preparation, analysis, and reporting

Which parts of the process are the most time and effort consuming?

Data collection and preparation

How long will data collection and cleaning take?

Takes up half the process, half the time

How much of the process do data analysis and communication take up?

1/3 each

What is needed to gain clear objectives for the goal and scope of the project?

A precise definition of the problem statement

Data collection

The systematic process of gathering information on variables of interest

Who often collects data?

Domain experts

Why is data often collected?

To find answers to predefined problems

When it is not collected purposefully, why is data collected?

It is a byproduct of our daily activities and is kept because it may be useful later

In what state is data often found?

Non-optimal form for analysis

Data preparation or data processing

Data is processed so it can be analyzed

Data analysis

Once the data is prepared, it must be analyzed to discover meaningful insights

What methods to data analysts often choose?

They start with simple methods and move on to more advanced ones if they want to investigate further

Data reporting

Involves the presentation of data in a way that will convey information learned from analysis

What do domain experts have to do with data reporting?

It's important to present the data in a way that domain experts can understand

Data visualization

A graphical way of presenting and reporting data to point out the hidden patterns, trends, and hidden insights

What does data visualization involve the use of?

Visual elements such as charts, graphs, and maps to present data in a way that is easy to comprehend and analyze

What is the goal of data visualization?

To communicate information effectively and facilitate better decision making

What was data management like in the early days?

There wasn’t so much data and it was possible to store all data locally and process it locally as well

Why were cloud-based management systems created?

To help manage the vast amounts of data that later became available

Where are cloud-based management systems kept?

on a designated server outside of a local computer

What happened as data processing became more time consuming?

Cloud-based management systems evolved to handle some automatic data processing

Data warehousing

Systems that store and manage large volumes of data from various sources in a central location

What benefit does data warehousing give?

Efficient retrieval and analysis for business intelligence and decision-making

What do enterprises do in order to warehouse data?

They subscribe to cloud warehouse services

What are some cloud warehouse services?

Amazon RedShift(runs on the Amazon WebServices) and Google BigQuery(runs on Google Cloud)

What does data warehousing replace?

Companies don’t buy physical storage or configure data management/processing systems on their own

What kind of data can be generated within a business?

Data based on customers, products, and sales

Business analytics

Investigate business data to launch new products and to maximize their business revenue and profit

Big data

Unstructured data from millions of customers

How is detecting and managing fraudulent transactions nowadays?

With automated machine learning algorithms

Walmart example for business analytics

They used date science to figure out which products were most popular during hurricane season and stocked extra units

Amazon example for business analytics

They figured out in which regions items were most popular so they could warehouse those items there and reduce delivery times

Internet of things(IoT)

The network of electronics that communicate with each other through the internet

Why is data science important to IoT?

Most objects will perform tasks based on the data that another object will gather

AI irrigation system

There is a control tower in Idaho that analyzes how much water plants use, which allows farmers to conserve more water

Why is the AI irrigation system so useful to the state of Idaho?

Idaho gets a lot of their GDP from agriculture, so this tech is a boon to the state

Why are smart cities more efficient?

They used a lot of automated services that analyze various aspects like energy usage and the fullness of garbage cans

How has data science benefitted education?

Various online apps have allowed the tracking of individual students’ progress which allows for customized teaching strategies to better teach each student

Predictive analytics

Uses data science to analyze the history of something and make predictions about the future

What did doctors have to do before predictive analytics?

Rely on their experience and on a person's self-reported symptoms

What piece of technology also helps make diagnoses?

Health-oriented smart devices

The Precision Medicine Initiative

A long-term research project run by the NIH, it's goal is to analyze various factors to determine how they affect a person's health

Sports analytics

The analysis of sports data

Sabermetrics

A statistical approach used to identify critical yet traditionally overlooked metrics such as on-base percentage and slugging percentage

How was sabermetrics dramatized?

In the movie Moneyball

Recommendation systems

Data-driven personalized suggestions that satisfy consumers

Netflix Research

A recommendation system that considers subscribers’ watch histories, satisfaction with the content, and interaction records

What is Netflix Research's goal?

To make perfect personalized recommendations despite some challenges, including the fact that viewers themselves often don’t know what they want to see

How can machine learning help you in data science?

It can help better understand data

How can AI help you in data science?

It can make automated, data-driven decisions on a task

What are some ethics and privacy issues that have come up in data science?

Who gets to collect our personal data and what do they get to use it for?

What are the effects of bias in data?

The machine learning algorithms and AI will adopt that bias

Besides numbers, what else can be considered data?

Anything that we can use to compile information

Quantitative data

Deals with quantities and amounts and is usually analyzed using statistical methods

Examples of quantitative data

Numerical measurements like height, weight, temperature, heart rate, and sales figures

Qualitative data

Non-numerical data that generally describe subjective attributes or characteristics and are analyzed using methods such as thematic analysis or content analysis

Examples of qualitative data

Descriptions, observations, interviews, and open-ended survey responses that address unquantifiable details

What are the two primary types of quantitative data?

Numerical and categorical

Numerical data

Represented in numbers that indicate measurable quantities; it may be followed by symbols to indicate units

What are the subtypes of numerical data?

Continuous and discrete

Continuous data

Measurable values along an unbroken scale. Can you infinitely break it down into decimals? Answer should be yes

Discrete data

Contable values(whole integers), no fractions or decimals

Note on continuous and discrete data

Certain continuous data can be discrete if certain restrictions are placed

Example of continuous data also being discrete

Recording the amount of students in each class; the possible numbers are technically infinite because it could have 1 student or more; but the school may limit the class size to 200 which would make it discrete

Categorical data

Represented in different forms such as words, symbols, and even numbers

Where does categorical data come from?

A finite set of values and the value doesn’t necessarily indicate a measurable quantity

What smaller categories can categorical data be divided into?

Nominal and ordinal data

Nominal data

The set of possible values includes an ordering notion

Dataset

A collection of observations or data entities organized for analysis and interpretation

How can many datasets be represented?

As a table where each row indicates a unique data entity and each column defines the structure of the entities

What are other names for entities?

Items, entries, or instances

Attributes

Also known as features or variables or characteristics

Structured dataset

Although the actual values of the attributes are different across entities, all entities have values for the same attributes

Unstructured dataset

One that lacks a predefined or organized data model

In what format are most structured datasets found?

Tables

In what format are most Unstructured datasets found?

Text, video, images, audio recordings, or other content where the data doesn’t fit neatly into rows

Of which type of dataset is there more of, according to opinion?

Unstructured

Examples of Unstructured datasets

Amazon reviews, Twitter posts, images on instagram, and videos on TikTok

What must be done to unstructured datasets so they can be analyzed?

They must become structured

What are the most commonly used formats for structured datasets?

Comma separated values(CSV), JavaScript Object Notation(JSON), and Extensible Markup Language(XML)

What is the most intuitive way of encoding a tabular dataset?

CSV

In what format is most of the data we collect from the web(websites, mobile apps)?

XML or JSON

Which data storage format is best for complex datasets?

XML because of it's hierarchy-friendly nature

What programs can you use to open XML, CSV, and JSON files?

Since they all store data as plain text, you can open them eith typical text editors like Notepad, Visual Studio Code, Sublime Text, or VI Editor

CSV pros

Simple

CSV cons

Difficult to add metadata, difficult to parse if there are special characters, flat structure

CSV typical use

Tabular datasets