Data Science Final Exam Review Flashcards

Final Exam Logistics and Information

• Format: The final exam consists of $50$ multiple-choice questions on paper.

• Wording/Style: Questions are worded and formatted similarly to the quizzes given throughout the term.

• Scope: The exam is cumulative over the whole term but places emphasis on topics from the second half.

• Duration: Students have $2$ hours to complete the exam.

• Permitted Materials:

  • A $3 \times 5$ notecard is allowed.

  • Scientific or graphing calculators are permitted, though the exam will involve minimal calculations.

• Schedule and Location: Tuesday $12:30$ – $2:30\,p$ in Tykeson $32$ (the usual classroom).

• End of Term Notes:

  • An announcement regarding final grades will be sent out after the final exam.

  • Attendance at the Friday lab is still mandatory, though it is reserved for Project $2$ work time.

  • Lab $08$ is due on the evening of the review session; no resubmissions are allowed.

  • Project $02$ is due Friday; no resubmissions are allowed.

  • Office Hours: Kamy's office hours are by appointment only this week; there are no office hours during finals week.

Special Python Expressions and Arithmetic

• Modulo Operator (%): Returns the remainder from division.

  • Example: Running $7 \% 5$ in Python returns $2$. This is because $7/5 = 1$ with a remainder of $2$.

• Exponentiation Operator (**): Returns the first number raised to the power of the second number.

• The Caret (^) Symbol: In Python, ^ represents a different type of expression (bitwise XOR) which is not covered in this class. It is not used for exponentiation; therefore, $3**2$ is not the same as $3^2$ in the context of typical Python math operations.

Data Types: Numbers and Strings

• Numbers: Python has two real number types:

  • Integers (int): Whole numbers of any size (e.g., $1$, $3$, $5$, $7$). An int never has a decimal point.

  • Floats (float): Numbers with an optional fractional part or decimal point (e.g., $1.0$, $3.2$, $5.70000000001$). A float always possesses a decimal point.

• Limitations of Floats:

  • Size: They have a limited size, although the limit is extremely large.

  • Precision: They have limited precision of $15-16$ decimal places.

  • Arithmetic Inaccuracy: After performing arithmetic, the final few decimal places of a float value can be incorrect.

• Scientific Notation: Floats might be printed using scientific notation (e.g., using e for powers of $10$).

• Strings: A series of characters surrounded by quotation marks, either single (' ') or double (" ").

  • Examples: 'Hello, world!', "eight", or '10'.

Data Types: Arrays, Ranges, and Lists

• Arrays:

  • An array contains a sequence of values.

  • All elements within an array should share the same data type.

  • Arithmetic is applied to each element individually (element-wise).

  • Adding two arrays together adds their corresponding elements, provided they are of the same length.

• Ranges: A range is an array of consecutive numbers created using np.arange().

  • np.arange(end): Creates an array of increasing integers from $0$ up to, but not including, end.

  • np.arange(start, end): Creates an array of increasing integers from start up to, but not including, end.

  • np.arange(start, end, step): Creates an array of increasing steps from start up to, but not including, end.

  • Rule: The range always includes the start value but excludes the end value.

• Lists:

  • Create lists using square brackets [ ] or the function list().

  • Unlike arrays, lists can contain mixed data types.

  • In this course, lists are primarily used for table operations such as grouping by multiple columns (e.g., survey.group(['Year','Extraversion'], np.average)) or adding new rows to tables.

• Booleans (Bools):

  • Two possible values: True or False. Both must be capitalized.

  • Used in functions (e.g., tbl.sort("col", descending = True)) and as the output of comparison statements.

Table Structure and Basic Operations

• Definition: A Table is a sequence of labeled columns.

• Structure:

  • Rows: Each row represents one individual.

  • Columns: Represent attributes of individuals. Data within a single column represents one particular attribute across all individuals.

  • Labels: Column labels are strings.

  • Types: Columns are stored as arrays, and all columns in a table must be the same length.

• Primary Table Operations:

  • t.select(label): Constructs a new table containing only the specified columns.

  • t.drop(label): Constructs a new table where specified columns are omitted.

  • t.sort(label): Constructs a new table with rows sorted by the specified column.

  • t.where(label, condition): Constructs a new table containing only rows that match the given condition.

• Extraction Differences:

  • t.select(): Pulls a column out as a Table.

  • t.column(): Pulls a column out as an array of its values, meaning array-specific arithmetic rules now apply.

Advanced Table Manipulation and Creating Tables

• Creating Tables:

  • Table.read_table(filename): Reads data from a spreadsheet/file.

  • Table(): Creates an empty table.

  • t = Table().with_column("col name", array): Creates a new table starting with one column.

  • t.with_column("col2 name", array2): Adds a new column to an existing table.

  • t.with_columns("col1", array1, "col2", array2): Adds multiple columns at once.

• Row Manipulation:

  • t.sort(column, descending=True): Sorts rows in decreasing order.

  • t.group(column): Groups by the values of a categorical variable.

  • t.take(row_numbers): Keeps specific numbered rows (indices start at $0$).

  • t.where(column, value): Keeps rows where the value equals a specific value; equivalent to t.where(column, are.equal_to(value)).

• Apply Method:

  • t.apply(function_name, 'column_label'): Creates an array by calling a specific function on every element in the input column.

• Split-Apply-Combine:

  • Process used by .group(). For example, t.group('Name', np.mean):

    1. Split: Data is split into temporary sub-tables based on unique values in the 'Name' column.

    2. Apply: The mean (np.mean) is applied to the other columns in each sub-table.

    3. Combine: The results are gathered into a single summary table.

• Pivot Tables:

  • Converts data from "long form" to "wide form."

  • Table.pivot(columns_variable, rows_variable, values=None, collect=None).

  • columns_variable: Forms the column labels of the grid.

  • rows_variable: Forms the row labels of the grid.

  • If values and collect are omitted, the table defaults to a count of occurrences.

• Joining Tables:

  • .join() merges tables based on a common column.

  • Rows with values in the common column that do not appear in both tables are dropped from the result.

Visualizations

• Numerical Variables:

  • Scatter Plot (t.scatter(x, y)): Best for showing associations or relationships between two quantitative variables.

  • Line Plot (t.plot(x, y)): Best for trends of a numerical variable over sequential values (like time or distance). Use when there is exactly one $y$ value for each $x$ value.

• Categorical Variables:

  • Distribution: Describes the frequencies of different values in a column.

  • Bar Chart (t.barh(category, y)): Displays categorical distributions. The length of the bar represents the count. Good for comparing numerical data across categorical groups.

• Histograms (t.hist(label, bins=array, unit=string)):

  • Displays the distribution of a numerical variable using bins.

  • Height represents density (crowdedness): $Height = \frac{\% \text{ in bin}}{width \text{ of bin}}$.

  • Units of height: percent per unit on the horizontal axis.

  • Area represents percent (number of individuals): $Area = Height \times width \text{ of bin} = \% \text{ in bin}$.

• Decision Matrix:

  • Categorical? -> Bar Chart.

  • Numerical and Sequential? -> Line Plot.

  • Numerical and Non-sequential? -> Scatter Plot.

Control Statements and User-Defined Functions

• Functions: Defined using def statement_name(parameters): followed by an indented body and a return expression.

• Control Statements: Keywords if and for control the sequence of computation.

  • if: Defines behavior based on conditions (if, elif, else).

  • for: Iterates repeatedly through a block of code for each element in a sequence.

Probability

• Notation: $P(A)$ denotes the probability of event $A$ occurring.

• Calculation: $P(A) = \frac{\text{Event A}}{\text{All possible outcomes}}$.

• Bounds: Probability is bounded between $0$ and $1$ (or $0\%$ and $100\%$).

• Addition Rule (OR): If event $A$ can happen in exactly one of two mutually exclusive ways, then $P(A) = P(\text{way 1}) + P(\text{way 2})$. The answer is greater than or equal to the individual chances.

• Multiplication Rule (AND): The chance that both $A$ and $B$ happen in that order is $P(A) \times P(B \text{ given } A)$. The answer is less than or equal to the individual chances.

Statistics and Sampling

• Sampling Types:

  • Systematic Sample: Sampling every $n^{th}$ subject from an ordered list.

  • Deterministic Sample: Scheme involving no chance; a set procedure is followed.

  • Random Sample: Individuals are drawn with a chance equal to their proportion in the population. Every group’s selection probability must be known beforehand.

• Distributions:

  • Empirical Distribution: Based on observed values or repetitions of an experiment (includes simulations).

  • Probability Distribution: Theoretical; describes all possible values and their mathematical probabilities (e.g., rolling dice).

• Terminology:

  • Parameter: A number associated with the entire population (the "true" value).

  • Statistic: A number calculated from a sample, used to estimate a parameter.

  • Confounding Variables: Hidden external factors influencing both independent and dependent variables, distorting cause-and-effect relationships.

  • Observational Study: Passively measuring variables as they naturally occur; reveals association, not causation.

  • Experiment: Reseacher applies treatment to establish cause-and-effect, often via random assignment.

Hypothesis Testing and P-Values

• Model: A set of assumptions about data, often involving randomness.

• Hypotheses:

  • Null Hypothesis ($H_0$): A well-defined probability model; differences between observed and simulated data are due to random chance.

  • Alternative Hypothesis ($H_a$): A different view where observed data are inconsistent with the null model.

• P-Value Definition: The probability of obtaining a result as extreme as, or more extreme than, the observed result, assuming the null hypothesis is true.

• Directional vs. Absolute Differences:

  • One-Sided: Direction stated in advance (e.g., "greater than" or "less than").

  • Two-Sided: Testing if there is any difference at all (not equal to).

• Cutoffs (Alpha):

  • P < 0.05: Statistically significant (reject the null).

  • $P \leq 0.01$: Highly statistically significant.

Bootstrapping and Confidence Intervals

• Bootstrapping: Estimating a population parameter by resampling from the original sample at random, with replacement, using the same sample size.

  • Principle: Bootstrap-world sampling approximates real-world sampling if the original sample is large enough.

• Confidence Intervals (CI): The middle $x\%$ of the distribution of estimates.

  • Example: A $95\%$ CI is found between the $2.5^{th}$ percentile and the $97.5^{th}$ percentile of bootstrap estimates.

  • Interpretation: "We are $95\%$ confident that the true population parameter is within our CI."

Central Limit Theorem and Distributions

• Central Limit Theorem (CLT): The probability distribution of the sum or mean of a sufficiently large random sample drawn with replacement will be roughly normal, regardless of the population's distribution.

• Normal Distribution Properties:

  • Average is at the center.

  • SD (Standard Deviation) is the distance from the average to the inflection points.

  • $68\%$ of observations are within $1\,SD$ of the mean.

  • $95\%$ of observations are within $2\,SD$ of the mean.

  • $\approx 99\%$ (specifically $99.7\%\text{ in standard texts}$) are within $3\,SD$.

• Z-Scores: Measure how many standard deviations an observation is from the mean. Used to identify outliers and standardize comparisons.