Data Science Quiz 2

Broadcasting

used when we are dealing with 2 array objects that are different sizes

Broadcasting Rules

1. If 2 objects differ in dimension size, the object with fewer dimensions is padded with 1s on its leading (left) side

2. If the shapes do not match at all, the array with shape equal to is stretched to match the other shape

3. If in any dimension the sizes disagree and neither is equal to 1, an error is raised

Masking / Boolean Logic

-index L to R using positive numbers

-index R to L using negative numbers

-index every nth using ::

Fancy Indexing

passing an array or list of indices to access, select, or modify multiple array elements at once

-so you can grab scattered elements in a random order

What Type(s) of Sorting does NumPy Support?

BOGO and Selection

Pearson’s R Correlation

-R = 0 means no correlation

-R > 0 means positive correlation

-R < 0 means negative correlation

p-Value

probability of the null hypothesis, must be low

Null Hypothesis

there is no significant linear relationship

Spearman Correlation

ranking correlation metric

-monotonic increase or decrease

-if high there is a strong positive correlation

What is Monotonic?

linear, exponential, log, factorial

What is NOT Monotonic?

cubic, quadratic, quintic

Models

a simplified representation of reality created to serve a purpose

Mapping

going from feature information to prediction

Induction

refers to the creation of models from data

-uses info in the data to generalize to cases by making general rules (even if the rules do not always hold)

Induction Example

training a loan-prediction model on past data

Deduction

process of applying the existing models to use-cases (more deployment focused)

Deduction Example

using a trained model for future applications

Segmentation

data that is used to train the model, known as “training data”

Splitting for Classification

divides dataset into separate subsets for training, tuning, and evaluating a model to ensure it performs well on new data

Entropy

measures the purity of categorical data features and labels (log base 2)

-good for a single probability distribution

Information Gain

the higher the information gain, the better

-good for 2 different probability distributions

-perfect feature eliminates all uncertainty and disorder in prediction

Classification x Decision Tree Construction

best purest data classification with smallest tree possible

Benefits are:

1. grounded in information theory through IG and Entropy

2. Interpretable

Steps to constructing a decision tree?

using a divide and conquer approach:

1. calculate information gain for each possible feature,

2. create a decision node for this feature,

3. look at the new decision tree branches,

4. and descend the one with the highest entropy

Decision Tree Weakness

1. missing data

2. overfitting sensitivity

3. a lot of features

4. trees are computationally expensive

5. less capacity to take advantage of hardware

6. regression

Pandas

framework for more advanced data manipulation

-built on top of NumPy and relies on its data structures and C-based typing

-good for detailed indexing

pd.DataFrame

2D tabular data structure in Python that organizes data into rows and columns, more advanced than pd.Series

-numpy: ndarray

-python: nested list

Index Object

immutable array

pd.Index = ordered set

Series

provides a mapping from a collection of keys to a collection of values

pd. Series

numpy: array

python: dictionary

Indices In Each Package

Python: non-existent or limited and mutable

NumPy: optional yet looked over

Pandas: everything and immutable

Indexers

.loc - allows indexing and slicing that always references the explicit index (1 = a)

.iloc - always references python-style index (0 = a, 1 = b)

.ix - deprecated

Implicit Slicing of data[0:2]

numbers, exlusive so stops before index 2, matches standard python lists/arrays

Explicit Slicing of data[“a”:”c”]

strings, inclusive so include index “c”, custom dictionary-like slicing behavior

Conditional Indexing ( > )

filter data by selecting only the elements that exceed a specific value

Universal Functions

efficient for vectorization

-used for linear algebra datatypes (np.aray/ndarray, pd.series. pd.dataframe)

Index Alignment

using series/dataframes with different indices can cause NaNs easily because Pandas performs union on the indices and fills in the gaps with NaN values

Components of Floating Point Format with NaN Values

sign bit, exponent, and mantissa/fraction

64-bit FP

IEEE 754 double, 11 exp bits, 52 frac bits

-NaN: exp = 7ff & Frac ≠ 0

32-bit FP

IEEE 754 Single, 8 exp bits, 23 frac bits

NaN: exp = ff & Frac ≠ 0

16-bit FP

IEEE 754 Half, 5 exp bits, 10 frac bits

NaN: Exp = 1f & Frac ≠ 0

8-bit FP

minifloat, exp is a variable [1,6], 7 frac bits of number of exp bits

NaN: exp is all 1s & Frac is ≠ 0

How does NaNs impact aggregate results?

fully takes over

Extra Functions

shape(), reshape(), %timeit

Broadcasting Objects

arrays, vectors, matrices, tensors

Sorting Facts

BOGO Sort is slowest

Selection sort: swap = : + np.argmin()

BOGO sort: np.randomshuffle()

How do we compute p-values?

using t-statistics (N is number of (x,y) pairs)

Subscript

individual entry

Hat

mean value