ISOM 475 Final

Tufts definition of data viz

An effective way to describe, explore, and summarize a set of numbers
Analyzing and communicating statistical information (Tufte)

Purposes of Data Visualization (Explore vs Explain)

Data scientist to data: visualization to explore
Confirm and analyze

Data scientist to reader/viewer
Visualization to explain
Inform and persuade

Visual analytics:

Visual analytics: the process of critical thinking enabled by interactive data visualizations

5 traits of good data visualizations

Show the data variation, not the design variation
Make large sets of data coherent
Communicate insights and complex ideas with clarity, precision, and efficiency
Use simple but effective designs
Tell the truth about data

Good Visual Perception cues

Understanding without training (intuitive)
Cross culture validity
Hard wired

How visual perception works

Neurons extract features involuntarily →

Assembly of information into the significance (patterns) and object identification →

Visual working memory (3 chunks) with active conscious attention

Gestalt Principals

Principal of proximity: perceive objects close together as a group

Principal of similarity: group objects together that look similar

Principal of enclosure: objects appear to belong together when they are enclosed within a boundary
We naturally perceive open, incomplete, and unusual forms as closed, whole, and regular

Principal of continuity : We perceive objects as belonging together, as a part of a single whole, if they are aligned with one another or appear to form a continuation of one another
Principal of connection: The perception that objects that are connected as part of the same group

Pre-attentive attributes

processing occurs below the level of consciousness at a high speed and is tuned to detect specific visual attributes

Form
Color
position

Pre-attentive attribute: Form

Length, width, shape, orientation, size, enclosure

Pre-attentive attribute: Color

Hue, Value, Intensity

Pre-attentive attribute: Position

Relative and absolute and motion

Spatial positions can show relationships and group membership
When animation is used, aim for motion in the range of 0.5 to 4 degrees/second of a visual angle (Rule G6.30)

Rules of thumb for how people perceive charts

We don't go in order
We see first what stands out
We see only a few things at once
We seek meaning and make connections
Correlation does not imply causation
We rely on conventions and metaphors

Common components of a chart/graph

Scatterplot

2 continuous variables

Set symbols at maximum distinction, and make them differ by both color and shape
Maximize the distance of the color wheel with categorical variables

Line Graph

Two continuous variables with one being time

Use for trends over time

Pie Chart

1 continuous and 1 categorical

Bar Chart

1 continuous and 1 categorical

Glyphs/marks: symbols meant to represent a numerical attribute or entire, representing through the use of our building blacks of form, color, position, and motion

Use small, closed shapes to represent data entities

Use the color, shape, and size of those shapes to represent attributes of the entity

Can be used to show time series
Stacked bars
Diverging bars

Histogram

One continuous variable
can be used with multiple categories

Distributions

Stem and leaf plot
Histogram
Boxplot
Violin Plot

Cloropleth

Maps with color
use of color to represent the distribution of standardized variables values using pre-defined boundaries (e.g. countries)
Loses gradient subtly - the phenomenon of interest may respect boundaries
Explains via familiar units

Heat Map

use of color to represent quantitative value imposed upon an area/location (vs. leveraging predefined boundaries ‘agnostic; to the mapped values)

Isopleth

a quantitative value used to impose isolines (constant value)

Cartogram

Use of size of predefined units (states) to represent the distribution of variable values

topology(unit relationships) maintained

Explains via familiar units

Shape Map

Length of diameter vs area of a circle

Tree Map

use of color and form (area) to represent two quantitative values

May also use (relative) position to represent nested/hierarchical data

Precision is not important

Boxes represent entities

Better for large data sets where the smallest category is still relatively significant

Principals of Graphical evidence (Tufte)

1. Graphical excellence is the well-designed presentation of interesting data – a matter of substance, statistics, and design.
2. Graphical excellence consists of complex ideas communicated with clarity, precision, and efficiency.
3. Graphical excellence is that which gives to the viewer the greatest number of ideas in the shortest time with the least ink in the smallest space.
4. Graphical excellence is nearly always multivariate.
5. Graphical excellence requires telling the truth about the data.

Purpose of EDA

EDA summarizes the main characteristics of a data set, typically w visual methods

Used primarily for seeing what the data can tell us beyond and prior to formal modeling (“model free evidence”)

Common measures of continuous/numeric variables

1. estimates of location
2. estimates of spread
3. estimates of shape

Estimates of location

Mean
- Most commonly used for average
- All values weighted equally

Median
- Median splits the data into the upper half and lower half
- Not sensitive to extreme values
- 50th percentile

Mode
- an estimate for discrete and categorical data
Can be bi-modal or multi-modal

Outliers and trimmed mean

Percentiles tell us the value at which that percentage of observations falls below

Interquartile Range = difference between the 75th and 25th percentiles (middle 50% of data values)

IQR is often used to find outliers
Q1-1.5 IQR or above Q3 + 1.5IQR

Deciles- percentiles in 10% increment

Outliers are not inherently invalid and erroneous
Must determine if outliers are bad

Estimates of spread

Absolute deviation
variance
standard deviation

Absolute Deviations

Mean absolute deviation: mean of the absolute deviation from the mean for all observations

Median absolute deviation: median of the absolute deviation from the median for all observations

Variance

Average of the squared deviations from the mean in a database

Standard Deviation

- Expressed in the same units as the data
- The Square root of the variance
- S = √Variance
- Standard deviation in normally distributed data

Estimates of Shape

Skewness

Histogram template

Bar height: represents frequency or probability of bar range

Bar width: represents a range on the value axis, uniform across all bars

Empirical (cumulative) Distribution Functions

Kernel density plots
density plots

Kernel density plots

Based on the declared “bandwidth: the kernel function is used to estimate the density of each band. Then all density estimates are added together

Density Plot

Probability scale: Proportion of given value in population

Curve Height: Estimated probability of corresponding value occurring

Area: Total area under curve equals one

Value axis: value displayed for a single, continuous variable

Tukeys 5 Number summary

Location and spread:

Minimum observation
Lower quartile
Median
Upper quartile
Maximum observation

Quartile-Quartile Plot (Q-Q)

Compare continuous variable to a theoretical distribution (normal Distributions)

- Plots the quartiles of your variable against the quantiles of a theoretical distribution
- If the points follow the 45-degree line, the quantiles match and distributions are similar
- Can be used to compare two distributions

Numeric-Numeric Comparisons

Scatterplots give insights for numeric-numeric comparisons
“Casual” relationships (linear and non-linear)
Associations
Outliers and groups of outliers
Clusters
Gaps

How to show conditional relationships

Small multiples of scatterplots help us see if relationships are conditional

Scatterplot template

Similar to time series designs but X axis is a second continuous variable that is not time

Y-axis: usually a single, dependent variable
X-axis: values displayed for single variable
Cannot be time

Correlation

Linear correlation coefficient (persons correlation_ measures the direction (positive or negative) and the degree (strong or weak) of the linear relationship between two sets of quantitative values

All values fall between -1 and +1
- 0 = no linear correlation
- +1 = perfect positive linear relationship
- -1 = perfect negative linear relationship

The greater the magnitude of the value, the stronger the relationship

T-stastistic

T- statistic: is the correlation coefficient statistically significant

R Squared

Square of persons correlation coefficient
If trend is negative sloping, add a - sign

Reading correlation

The value of correlation indicates the strength of the linear relationship

The value of the slope does not indicate the strength of the relationship, but tells you the response to a one-unit increase in the predictor

Correlation coefficients do not apply directly to non-linear relationships

Plots for accessing correlation

Rugs (strip plots of histograms) add distributional information to scatterplots
Ex. NFL combine

Contour plots help highlight relationships when the dataset is large and dense

Numerical-categorical Comparisons Purpure

With one numeric and one categorical variable, its typically about comparing distributions

Strip Plots/ Dot plots

simplest way to show numeric by categorical data

Violin plots

display distribution shape

Butterfly charts and Histograms

Comparing numeric variables across two categories

Small Multiples

numeric-categorical comparison

Dot plots

Effective way to compare single numeric values of summary stats across categories

What to do with two categorical variables

simple bar plots of multiple categorical variables show where the data observations occur

- Mosaic plots: an alternative way to show occurrence in categorical data combinations. They provide quick information about where the data lies, but making comparisons is harder

- Simple frequency table:
especially when color is used to highlight differences

Purpose of EDA

Summarize the main characteristics of a data set using visualization

Suggest hypotheses about the causes of observed phenomena

Assess assumptions on which statistical inference will be based

Support the selection of appropriate statistical tools and techniques

Provide a basis for further data collection through surveys or experiments

Nominal Comparisons

nominal/categorical comparisons: displays a set of discrete quantitative values so they can be easily read and compared
- Box and Wisker plot: comparing distributions across categories
- Cleeveland dot plot

Time Series

A series of quantitative values that show how something changed over time

Useful when ur quantitative message includes: change, rise, increase, fluctuate, grow, decline, decrease, or trend

Time is always the horizontal access and vertical is the quantitative scale

Vertical bars: emphasize individual values or compare categories of values, rather than the overall pattern
Effective when emphasizing individual values

not good for overall trend?

Stacked bars:

Discrete points in time

Useful if there are subcategories and the sum of the subcategories is meaningful for your quantitative message

points only graph

values where collected at irregular intervals of time of there is little short term up and down fluctuation in values

Not good for overall trends

Lines (or points with lines)

pattern of change through time

The line implies continuous data, and the line between dots may not longer be appropriate depending on the context

When data is collected irregularly, a line is not good as it may indicate there is a trend that isnt really there

Step chart

implies steady change from point A to B.

Works with population or stock prices

gwim_step()
Ggplot2

Violin plot/vertical box plot

how distribution changes over time

Linear trend line fits data with

the best line using least square regression

LOESS curves

reforms multiple local regressions that place higher weighting on closer points

No priori model
No simple regression function to describe the trend
Large a makes a smoother model
Overfitting may be an issue

Rankings Basics

Use one axis for categorical items and use the other axis for a quantitative scale

Bars are most often preferred

Except when the scale dosent begin at zero- then use points

Sorting is key in effectively communicating
Highlight highest values - sort descending
Bump chart:
Comparing rankings over time

Part to whole

how individual values that make a whole of something compare to each other and the whole

Rate, percent, percent of total, share, accounts for X percent

No pie graphs- use single stacked bar charts

Proportions by category

bar chart
stacked bar

Hierarchical proportions

Tree maps

Proportions over time

Stacked bar chart

Deviation designs:

show how one or more sets of quantitative values differ from a reference set of values

plus/minus, variance, difference, relative to
Deviation with time series

Purpose of Interaction

Give control to the user
Guide the user through your story
Handle too much data or too many variables
Allow for data exploration and new question
Can be more fun and engaging

Definition of visual information seeking mantra

Overview, zoom and filter, then details on demand

5 most common interaction

sorting
filter
highlighting
details on demand
zoom/pan

Filter

- Reduces the data viewed to a subset
- Filters data based on any info in the connected data, not just currently displayed
- Give a visual to remind reader that the filter is present
- Allows multiple graphs to be linked such that filters apply across graphs
- Allow data to be filtered rapidly using controls such as:
- Sliders, check, boxes, view actions
- Allows user to directly select and remove items from a graph
- Provide ability for complex logic with multiple features

Highlighting

Highlights a subset of data by selecting from lists of categorical items

Should highlight subset but have whole graph still visible

Brushing: when a user selects items in the visualization with an input device (mouse) which can then be highlighted or used as a filter

Details on demand

Provides an easy means to view details related to an item in a visualization when needed

Provides an easy means make details disappear from view when they are no longer needed

Common approaches:
Tooltips
Linked cross tabs

Zoom/Pan

Take a closer look at a specific section of a graph
Provide the means to directly select and zoom in 1 click (+zoom out)

Directed interaction

question, view., insight

Exploratory Interaction

view, question, view, insight

Musts of interactive graphs

assume nobody will see tooltip
very few people click on anything
focus on the needs of the audience, not the aesthetics

Fews definition of Dashboards

A visual display of the most important information needed to achieve one or more objectives, consolidated and arranged on a single screen so the information can be monitored at a glance

Dashboards are:

Visual
Display the information needed to achieve specific objectives
Fits on one computer screen
Monitor information at a glance
Present information using small, concise, direct and clear display media
CustomizedD

Dashboards are not

A display that is primarily used for data exploration and analysis
A portal
A scorecard
A report that people use to look up specific facts

Why use a dashboard:

Maintaining situation awareness

Performance monitoring process
- Update high-level situation awareness
- Identify and focus on particular items that need attention
- Update awareness of this item in greater detail
- Determine whether action is required
- If action is required, access additional information that is needed, if any, to determine an appropriate response
- Respond

Common Dashboard mistakes

Exceeding the boundaries of a single screen
Supplying inadequate context for the data
Displaying excessive detail
Expressing measures indirectly
Choosing inappropriate media of display
Introducing meaningless variety
Providing different graphs for variety rather than true medium
Using poorly designed media
Encoding quantitative data inaccurately
Arranging the data poorly
Ineffective highlighting of whats important
Clustering the screen with useless decoration
Misusing or overusing color
Designing an unappealing visual display

Dashboard Design Principles

See the big picture
Focus in on the specific items of information that need attention
Quickly drill into additional information that is needed to take action

Fundamental Dashboard Usage requirements

Dashboard design best practices

Organize information to support meaning and use

Maintain consistency to enable quick and accurate interpretation

Put supplementary information within reach

Make the experience aesthetically pleasing
Choose meaningful colors
Subdued over bright
Off-white background
High resolution images
Legible font
Alight content and follow good layout principals

Expose lower-level conditions

Prevent excessive alerts

Keep viewers in the loop

Accommodate real-time monitoring (when needed)

Ideal charts for dashboards

Bullter graphs
Bar Bullet
Dot plots
Line graphs
Slope charts
Sparklines (see image)
Box-plots
Spatial maps
Heat maps
Treemaps

Analogous colors

colors next to each-other on color wheel

Complimentary colors

opposite from eachother on color wheel

Split complementary

colors that are adjacent to the opposite on color wheel

Triad color pallet

3 colors in a triangle pattern on color wheel

Tetratic color palette

form a rectangle on color wheel; 2 sets on complementary patterns
One main color, 3 accent colors

Color Identifiers

RGB: 0-163
Hex code: #00FFA3; 00 = red value, FF = green value, A3= Blue value
HSL: Hue (degree), saturation (%), lightness (%)
CMYK: 0-1; cyan, magenta, yellow, black

color variable relationships

Sequential pallets for ordinal values or unidirectional continuous values

Diverging pallets for bidirectional categories or continuous variables with positive and negative values

circular / categorical pallets for nominal categories

Font Sizing

1 pica = 12 points
1 inch = 72 points
12 points = 16 pixels
1 pica= 16 pixels

Kerning (font spacing)

adjusting the spacing between characters