Data C8 Midterm 1

[PATTERN] Trace array arithmetic

Work element-by-element and obey Python order of operations. Example: make_array(1,2,3) * 2 + 1 -> array([3,5,7]).

[PATTERN] Boolean arrays in arithmetic

In numeric arithmetic, True acts like 1 and False like 0. Example: np.arange(1,7,2) + make_array(True,False,True) -> array([2,3,6]).

[PATTERN] Table shape after select + where

select changes columns; where filters rows. Example: tbl.select('x').where('x', are.above(2)) has 1 column and only rows where x > 2.

[PATTERN] Count rows meeting a condition

Use tbl.where('col', predicate).num_rows. Example: drinks.where('Rating', are.above(4)).num_rows.

[PATTERN] Filter on two conditions

Chain where calls. Example: drinks.where('Rating', are.above(4)).where('Price', are.below(7)).num_rows.

[PATTERN] Find the row with the largest value

Sort descending, then take/select the first item. Example: teams.sort('Titles', descending=True).column('Team').item(0).

[PATTERN] Top n rows

Sort descending and take the first n row indices. Example: tbl.sort('Crashes', descending=True).take(np.arange(5)).

[PATTERN] Add a derived numeric column

Compute an array from existing columns, then with_column. Example: tbl.with_column('Rate', tbl.column('Count') / tbl.column('Total')).

[PATTERN] Transform every value in a column

Define a function, use apply, then add result with with_column. Example: clean = tbl.apply(clean_name, 'State'); tbl = tbl.with_column('Clean State', clean).

[PATTERN] Count each category

Use group with no aggregation function. Example: tbl.group('Genre') returns one row per genre plus a count column.

[PATTERN] Average within categories

Use group(category, np.mean), usually after selecting relevant columns. Example: players.select('Position','Age').group('Position', np.mean).

[PATTERN] Cross-classify two categorical variables

Use pivot. Example: players.pivot('Position', 'Nation') gives positions as columns, nations as rows, and counts in cells.

[PATTERN] Combine two tables using a shared key

Use join. Same key label: left.join('Team', right). Different labels: left.join('Team', right, 'Nation').

[PATTERN] Choose a plot for one categorical variable

Use a bar chart of category counts. Example: teams.group('Region').barh('Region').

[PATTERN] Choose a plot for one numerical variable

Use a histogram. Example: tbl.hist('Age').

[PATTERN] Compare numerical distributions across categories

Use overlaid histograms. Example: tbl.hist('Time', group='City').

[PATTERN] Relationship between two numerical variables

Use a scatter plot. Example: tbl.scatter('Minutes', 'Price').

[PATTERN] Histogram area principle

Area, not height, represents percent in a bin. Percent in bin = bar height × bin width; with unequal bins, tallest bar need not contain the most data.

[PATTERN] Repeat a simulation many times

Start with make_array(), loop over np.arange(reps), and np.append each simulated statistic/result.

[FUNCTION] make_array(…)

Creates a NumPy-style array used throughout Data C8. Example: make_array('A','B','C').

[FUNCTION] np.arange(start, stop, step)

Creates evenly spaced values from start up to but not including stop. Example: np.arange(1,7,2) -> array([1,3,5]).

[FUNCTION] np.append(arr, value)

Returns a new array with value(s) appended; it does not mutate arr. Example: a = np.append(a, 5).

[FUNCTION] np.count_nonzero(arr)

Counts nonzero/True entries. Example: np.count_nonzero(make_array(True,False,True)) -> 2.

[FUNCTION] np.diff(arr)

Returns adjacent differences: next - previous. Example: np.diff(make_array(10,13,12)) -> array([3,-1]).

[FUNCTION] tbl.column('x')

Returns column x as an array. Example: ages = players.column('Age').

[FUNCTION] tbl.select(…)

Returns a Table with only selected columns. Example: players.select('Name','Age').

[FUNCTION] tbl.where('x', predicate)

Filters rows based on a column condition. Example: tbl.where('Age', are.above(30)).

[FUNCTION] are.above(x) / are.below(x) / are.equal_to(x)

Predicates for where. Example: tbl.where('Price', are.below(7)).

[FUNCTION] tbl.take(indices)

Returns rows at given integer positions. Example: tbl.take(np.arange(5)) returns first 5 rows.

[FUNCTION] tbl.sort('x', descending=False)

Returns rows sorted by a column; descending=True gives largest first. Example: teams.sort('Titles', descending=True).

[FUNCTION] tbl.group('x')

One row per unique x; default aggregation is count. Example: movies.group('Genre').

[FUNCTION] tbl.group('x', np.mean)

Groups by x and aggregates other suitable columns with mean. Example: players.select('Position','Age').group('Position', np.mean).

[FUNCTION] tbl.pivot(col1, col2)

Cross-classifies two categorical columns; col1 values become columns and col2 values become rows; cells default to counts.

[FUNCTION] tbl.join(key, other)

Combines rows from two tables whose key values match. Example: teams.join('Team', rankings).

[FUNCTION] tbl.apply(function, 'x')

Applies a function to each value in a column and returns an array. Example: tbl.apply(len, 'Name').

[FUNCTION] tbl.with_column('name', values)

Returns a table with one added/replaced column. Example: tbl.with_column('Double', 2 * tbl.column('x')).

[FUNCTION] tbl.with_columns(…)

Adds multiple columns. Example: Table().with_columns('x', make_array(1,2), 'y', make_array(3,4)).

[FUNCTION] tbl.hist('x')

Plots the distribution of a numerical variable. Histogram bar area corresponds to percent of observations in the bin.

[FUNCTION] tbl.barh('category')

Makes a horizontal bar chart, usually from a table with one row per category and a count/value column.

[FUNCTION] tbl.scatter('x','y')

Plots two numerical variables to inspect association. Example: shops.scatter('Minutes','Price').

[FUNCTION] tbl.num_rows

Number of rows in a table. Example: tbl.where('Late Night', True).num_rows.

[FUNCTION] arr.item(i)

Returns the item at zero-based index i. Example: teams.column('Team').item(0).

[FUNCTION] np.sum(arr) / sum(arr)

Adds numeric values. Common Data C8 use: sum(abs(p-q))/2 for TVD.