Data Science Exam 2 Study Guide

How does data get dirty?

when data is inaccurate, incomplete, inconsistent, or outdated. Happens because of human error or glitches.

Consequences of dirty data

Bad data creates mistrust and kills data driven decision making.

Explain the impact of bad data on future decision-making processes

Creates a vicious, negative feedback loop because managers don’t trust new data/

How to fix bad data

make sure newly produced data will be correct

The agency problem

the data creator is often NOT the data consumer (sometimes leads to dirty data)

How to help an organization's future data quality/fix agency problem

Establishing better communication between the data creator and users of the data!

How do you clean data?

1. Remove duplicate or irrelevant observations 2. Fix structural errors 3. Filter unwanted outliers 4. Handle missing data 5. Validate and QA

How should you deal with outliers?

Filter out unwanted outliers

How do you deal with ambiguous data (i.e., when you don’t know what the label means)?

Verify the meaning using documentation or ask the data source for clarification (guess never the correct answer)

How should you organize to ensure data quality?

Use a consistent, standardized format, document changes, and regularly validate and check data for errors, duplicates, and missing values

What is a scorecard?

measures performance against several goals (KPIs). Rows of X and ✓

What is a scorecard used for?

Can be used as a neutral comparison between items or to demonstrate that one item is superior to others

What is the difference in definition and purpose between a scorecard and a dashboard?

Dashboards contain different data visualizations related to the same topic. They are often interactive, and allow users to explore data further. (can contain scorecards)

Indicators

variables that measure a business relevant value

KPI

measures if an aspect of business is performing at an adequate level. SMART criteria A does NOT MEAN accurate

The tyranny of success

the difference between success and failure, for instance, in non-profits may not be clearly defined

Overreliance of KPIs

can cause people to lose the customer experience.

What can you do when there is the tyranny of success?

be sure to check our KPIs against the smart criteria and use multiple KPIs to measure success as a combination of multiple dimensions

What is a dashboard?

multiple data visualizations related to the same phenomenon or topic

When to use a relational database

to store large amounts of data

Relational database benefits

integrity (easier to maintain data integrity), flexibility (create diff cuts into the data), and efficiency (faster to retrieve info)

What is ETL and why is it necessary?

Extract, transform and load, Combines data from multiple data sources into a single, consistent data source with a common field

Purpose of ETL

improves data quality, combines data from different sources and prepares the data for analysis

What is the purpose of Hadoop

stores data in smaller chunks across a network on different computers (nodes).

What is the purpose of MapReduce

processes the pieces of data in parallel in different nodes and combines the results together.

How do you resolve conflicts in data when combining data(i.e., PA versus Penna. versus Pennsylvania)?

transform into a consistent value

When should you not bother resolving conflicts or even fixing your data?

when it’s not relevant

How can you use Excel’s MATCH() function to find invalid values?

checks if a value can be found in a list

Match function

=MATCH(value2check, space2check, matchtype) [0 = returns where the match can be found… not necessarily the row #]

What is the output of this function in Excel “=MATCH(A2,$C$2:$C$10,0)” if the zip code 19123 is in cell A2 and in your excel sheet on row 1,13 & 89?

N/A (only searches on rows C2 through C10!

IIF(AVG([WeightCars])>AVG([WeightTrucks]),"Cars","Trucks")

Returns “Trucks”. (If the avg weight of cars is greater than the avg weight of trucks (false). If the condition was true, then it would return “cars”, because it is false, it returns “trucks”)

Filtering

dropping observations from a list of data

Sorting

changing the order of which items are presented in a sequence