Data Science and Statistics: Key Concepts and Tools for Analysis

Data science

Using statistics, programming, data, and real-world/disciplinary expertise—plus ethical reasoning and communication—to make good decisions with data (at scale, for organizations, in context).

Data

Not only numbers, not only facts, and not strictly objective truth—data are generated by humans and/or human systems and tools, so they carry human choices.

Data & identity

Data both produce identity and are shaped by it—e.g., targeted advertising based on demographics, medical data used in diagnosis and treatment.

Quantified self movement

People tracking data about themselves (steps, sleep, heart rate, etc.) to measure and understand their own lives.

Surveillance capitalism

Shoshana Zuboff's term: companies claim human experience as free raw material, translate it into behavioral data, and turn it into prediction products that are sold.

Behavioral surplus

In Zuboff's framing, the behavioral data beyond what is needed to improve a service; it is fed into machine intelligence to make prediction products.

Behavioral futures markets

Zuboff's term for the marketplace where prediction products about what people will do next are traded.

"Data is the new oil"

Clive Humby's metaphor: data, like crude oil, is valuable but must be refined (processed and analyzed) to be useful; treats data as a commodity.

Research (inference/explanation) question

Asks what is true and why. Goal: learn about the world and use evidence to support a defensible conclusion. Example: Does YouTube create echo chambers?

Prediction question

Asks what is likely to happen. Success = accurate predictions on new observations; less about why, more about how accurately. Example: predict the ideology of a YouTube video.

Good data scientist's three statements

"Here is what we found. Here is how strongly the evidence supports it. Here is what we cannot conclude."

Open science

Showing and sharing your work (data, code, methods) so others can check and build on it; promoted by the Center for Open Science.

Reinhart-Rogoff Excel error

"Growth in a Time of Debt" (Reinhart & Rogoff) was used to support austerity; grad student Thomas Herndon found spreadsheet errors (e.g., a bad AVERAGE range). Shows why transparent, reproducible code matters.

Florence Nightingale

Used careful data collection, analysis, and visualization (her rose/"coxcomb" diagram of causes of mortality in the Crimean War) to show most soldier deaths were from preventable disease.

Choropleth map

A map where predefined areas (countries, states, counties) are shaded or colored to show a value. Early example: Charles Dupin's 1826 map of France.

Isopleth map

A map that uses continuous lines (contours) connecting points of equal value, regardless of political boundaries.

Descriptive statistics

Summaries that describe data you have—e.g., measures of central tendency, dispersion, and variability; absolute and relative values.

Inferential statistics

Using data to draw conclusions or make predictions beyond the data, based on hypothesis testing—e.g., regression analyses.

Central tendency

The "center" or typical value of a set of data; main measures are the mean and the median.

Mean

The arithmetic average: add all values and divide by how many there are. Sensitive to extreme values (outliers). Example: mean(c(1, 2, 3)) # 2

Median

The middle value when data are sorted. Resistant to outliers—"Median to the rescue!" in the Bill Gates bar example. Example: median(x)

Outlier (Bill Gates in a bar)

An extreme value that distorts the mean. Ten $35k workers + one $1bn earner → mean ≈ $91 million, but the median is still $35k.

Dispersion / variability

How spread out the values in a data set are (e.g., range, standard deviation). Example: sd(x)

"Lies, damned lies, and statistics"

Phrase popularized by Mark Twain; a warning that numbers can be used to mislead, so we must understand what statistics mean (and don't).

Exploratory data analysis (EDA)

Investigating data to find patterns and questions—"EDA is a state of mind" (Wickham). Tools: visualization, transformation, modeling.

Variation

The tendency of values of a single variable to change from measurement to measurement. EDA question: what type of variation occurs within my variables?

Covariation

The tendency of the values of two or more variables to vary together in a related way. EDA question: what type of covariation occurs between my variables?

Correlation (positive / negative)

A relationship where two variables move together (positive) or in opposite directions (negative); can be linear or non-linear. Penguin question: body mass vs. flipper length.

R

An open-source programming language designed for statistical computing and data visualization; the engine that runs your code and performs calculations.

RStudio

The application (made by Posit) that provides a user-friendly interface to write, edit, and run R code. R = engine, RStudio = dashboard; you need both.

IDE (Integrated Development Environment)

Software that combines a programmer's tools in one place: code editor (syntax highlighting), debugger, compiler/interpreter, and autocomplete. RStudio is an IDE.

Posit

The company (formerly RStudio, PBC) that makes RStudio and supports open-source data science tools like the tidyverse.

CRAN

The Comprehensive R Archive Network—the official site to download R and R packages.

Source pane

RStudio pane (top-left) where you write and edit scripts / .Rmd files—your code editor.

Console

RStudio pane where R code actually runs; good for one-time commands like install.packages(). The Source pane is a notepad; the Console is the engine.

Environment pane

RStudio pane showing the objects (data, values, variables) currently stored in your session; also holds History.

Output pane (Files/Plots/Packages/Help/Viewer)

RStudio pane that shows plots, files, installed packages, and help pages.

Markdown

A lightweight markup language that adds formatting to plain text with symbols (e.g., # Heading, **bold**). Created by John Gruber (2004). Example: # Heading one; **this text is bold**

WYSIWYG

"What you see is what you get" editors (like Word) where formatting shows instantly—unlike Markdown, where you add syntax.

R Markdown (.Rmd)

A file type that combines text (explanations), code (analysis), and output (tables, plots) in one place; results update automatically and steps are recorded.

Code chunk

A block inside an .Rmd file where R code goes; starts with ```{r} and ends with ```. Run it with the green play arrow. Example: ```{r}; 2 + 2; ```

YAML header

Settings block at the top of an .Rmd file (between ---) with title, author, date, and output format. Example: title: "Homework 1"; output: html_document

Knitting

Rendering an .Rmd file into a finished document (HTML, PDF, Word) that includes your text, code, and output. Submit both the .Rmd and the PDF.

Working directory

The folder R reads files from and saves files to by default. Setting it correctly is part of good file storage hygiene. Example: setwd("~/DS1000"); getwd()

Good file storage hygiene

Organizing folders and file names clearly—"we are programming for people, not the computer."

Comment (#)

Text after # that R ignores; used to annotate/explain code so others (and future you) can read it. "Annotating your code is top-tier behavior." Example: # Access the first item; fruits[1]

Package

A bundle of pre-written R functions (plus data/documentation) that researchers share; must be installed before use. ggplot2 is a package.

Library

Where installed packages are stored on your computer; library() loads a package into your current session.

install.packages()

Downloads a package to your machine. Type in the Console, once. Example: install.packages("tidyverse")

library()

Loads an installed package so you can use its functions. Put it at the top of your script—every time. Example: library(tidyverse)

tidyverse

An opinionated collection of R packages for data science (ggplot2, dplyr, tidyr, readr, tibble, purrr, stringr, forcats) sharing a design philosophy and grammar.

Hadley Wickham

Chief Scientist at Posit who leads the tidyverse team; co-author of R for Data Science.

Base R

The functions that come with R by default, without extra packages; can look different from tidyverse code, which aims to make common operations simpler.

? (help)

Typing ? before a function or object name opens its help page. Example: ?mean

Keyboard shortcuts

Alt + - (Option + - on Mac) types the assignment operator

Object

A named container that stores a saved result (a value, vector, data frame, etc.). Think of a labeled box: the label is the name, the contents are the value. Example: x

Assignment operator (

Assigns the value on the right to the name on the left; read as "gets." Creates or updates an object. Example: age

Variable (in R code)

A named object that holds a value, e.g., age, city, is_active.

Naming conventions

Use descriptive names in lowercase with underscores; avoid spaces, special characters, or starting with a number (2nd, ^mean, _day cause errors). Example: total_sales # good

snake_case

Naming style using lowercase words joined by underscores (my_var, scores_plus5); recommended for R variable names.

Case-sensitive

R treats uppercase and lowercase as different: number and Number are two separate objects. Example: number

Arithmetic operators

+ addition, - subtraction, * multiplication, / division, ^ exponent, %% modulus (remainder), %/% integer division. Example: 17 %% 5 # 2; 17 %/% 5 # 3; 2^6 # 64

Modulus (%%)

Returns the remainder after division. Example: 10 %% 3 # 1

Integer division (%/%)

Divides and keeps only the whole-number part. Example: 10 %/% 3 # 3

Data types

Kinds of values: numeric (5, 2.5), character/string ("hello", in quotes), logical (TRUE/FALSE).

Character (string)

Text data; must be in quotation marks. Example: city

Logical value

TRUE or FALSE (all caps, no quotes). Example: is_active

Vector

The most fundamental data structure in R: an ordered collection of elements of the same data type. Example: my_vector

c()

The combine/concatenate function; glues values into a vector. Example: fruits

Vector indexing ([ ])

Access an item by its position in square brackets. R starts counting at 1. Example: fruits[1] # "banana"

Element-wise (vectorized) operation

One operation applied to every element of a vector at once. Example: scores

Sequence (: and seq())

Creates a sequence of numbers. Example: 1:5 # 1 2 3 4 5; seq(1, 10)

Function

A self-contained block of code that performs a specific task: takes inputs (arguments), processes them, and often returns output. Format: function_name(arguments).

Argument

The information passed to a function; goes inside the parentheses after the function name. Example: mean(x, na.rm = TRUE)

length()

Returns the number of elements in a vector. Example: length(c(1, 2, 3)) # 3

sum()

Adds together all numeric elements. Example: sum(c(10, 20, 30)) # 60

sqrt()

Computes the square root. Example: sqrt(16) # 4

print()

Displays an object's value. (Typing the object's name alone also inspects it.) Example: print(x)

NA

"Not available"—R's marker for a missing value.

NAs are contagious

Most math/stat functions (mean, sum, sd, median) return NA if even one value is missing. Example: mean(c(1, NA, 3)) # NA

na.rm = TRUE

Argument that removes missing values during a calculation (and documents how you handled them). Example: mean(c(1, NA, 3), na.rm = TRUE) # 2

Boolean expression

An expression that evaluates to either TRUE or FALSE. Example: 10 > 9 # TRUE

Relational (comparison) operators

== equal to, != not equal to, > greater than, < less than, >= greater than or equal to,

== vs =

== tests whether two values are equal (returns TRUE/FALSE); = (or

AND (&)

Returns TRUE only if both conditions are TRUE. Example: scores > 70 & scores < 95

OR (|)

Returns TRUE if at least one condition is TRUE. (Pipe key is above Enter.) Example: scores < 65 | scores > 95

NOT (!)

Negates (flips) a logical value. Example: !(scores >= 80)

Vector comparison

Comparing a vector to a value checks every element and returns a logical vector. Example: scores > 80; # FALSE TRUE TRUE FALSE TRUE

Data frame (df)

A table of data in rows and columns, built from vectors; columns can be different data types. Variables are columns, observations are rows. Example: students

Observation

A single row in a data frame (e.g., one penguin, one student).

Variable (in a data frame)

A single column in a data frame—one measured characteristic (e.g., flipper_length_mm).

Tibble

The tidyverse's modern data frame, with easier viewing and other improvements over original R data frames.

$ operator

Accesses/extracts one column (or other object) from a data frame. Example: students$hours

head()

Shows the first rows of a data frame. Example: head(students)

str()

Shows a data frame's structure: number of observations/variables and each column's data type. Example: str(students)

nrow() / ncol()

Return the number of rows / columns in a data frame. Example: nrow(students); ncol(students)

dplyr

The tidyverse package of "verbs" (functions) for manipulating data frames.

filter()

Picks observations (rows) by their values based on a condition. Example: students %>% filter(hours > 5)

arrange()

Reorders rows (sorts), ascending by default—like sorting in Excel. Example: students %>% arrange(hours)

desc() / arrange(-x)

Sorts in descending order inside arrange(). Example: students %>% arrange(desc(hours))