1/22
DATA WRANGLING
DATA WRANGLING
also called data cleaning, data remediation, or data mungingārefers to a variety of processes designed to transform raw data into more readily used formats.
6 STEPS IN DATA WRANGLING
discovering
structuring
cleaning
enriching
validating
publishing
DISCOVERY
the process of familiarizing yourself with data so you can conceptualize how you might use it.
STRUCTURING
the process of taking raw data and transforming it to be more readily leveraged. The form your data takes will depend on the analytical model you use to interpret it..
CLEANING
the process of removing inherent errors in data that might distort your analysis or render it less valuable.
Cleaning can come in different forms, including deleting empty cells or rows, removing outliers, and standardizing inputs.Ā
ENRICHING
Once you understand your existing data and have transformed it into a more usable state, you must determine whether you have all the data necessary for the project at hand.Ā
VALIDATING
the process of verifying that your data is both consistent and of a high enough quality.
PUBLISHING
This involves making it available to others within your organization for analysis.Ā
PROPERTIES OF MEASUREMENTS
identity
magnitude
equal intervals
a minimum of zero
IDENTITY
Identity refers to each value having a unique meaning.
MAGNITUDE
Magnitude means that the values have an ordered relationship to one another, so there is a specific order to the variables.
EQUAL INTERVALS
Equal intervals mean that data points along the scale are equal, so the difference between data points one and two will be the same as the difference between data points five and six.
A MINIMUM VALUE OF ZERO
A minimum value of zero means the scale has a true zero point. Degrees, for example, can fall below zero and still have meaning. But if you weigh nothing, you donāt exist.
4 SCALES OF MEASUREMENTS
nominal scale
ordinal scale
interval scale
ratio scale
NOMINAL SCALE
used for naming variables in no particular order.
ex: eye color
ORDINAL SCALE
used for variable ranked order, but the difference is not determined.
ex: #1 happy, #2 neutral, #3 unhappy
INTERVAL SCALE
used for numerical variables with known equal intervals of same distance.
ex: time
RATIO SCALE
used for variables on a scale that have measurable intervals.
ex: weight
DESCRIPTIVE STATISTIC
present data in an easy-to-understand and presentable form, such as a table or graph. Without description, the data would be in its raw form with no explanation.
FREQUENCY COUNTS
Describe the number of times a variable exists in a data set. For example, the number of people with blue eyes or the number of people with a driverās license in the sample can be counted by frequency.Ā
INFERENTIAL STATISTIC
To develop a hypothesis from the data set. It would be impossible to get data from an entire population, so data scientists can use inferential statistics to extrapolate their results.
DATA VISUALIZATION
Describes the techniques used to create a graphic representation of a data sample by encoding it with visual pieces of information.
EXAMPLES:
Bar Chart
Line Chart
Scatter Plot
Pie Chart
Funnel Chart
Histograms
QUANTITATIVE MESSAGE
describe the relationships of the data. Depending on the sample, there areĀ different waysĀ to communicate quantitative data.
EXAMPLES:
Nominal Comparison
Time Series
Ranking
Part to WholeĀ
Deviation
Frequency Distribution
Correlation