7 - Data Cleaning and Preparation

dropna

Filters out missing data from a Series or DataFrame, removing rows or columns with NA values based on specified thresholds.

fillna

Fills missing values with a specified value or using interpolation methods like 'ffill' (forward fill) or 'bfill' (backward fill).

isnull

Returns a boolean array indicating which values are missing/NA in a Series or DataFrame.

notnull

Returns the negation of `isnull`, indicating which values are not missing/NA.

drop_duplicates

Use the `drop_duplicates` method, which returns a DataFrame with duplicate rows removed.

duplicated

A boolean Series indicating whether each row is a duplicate of a previous row.

replace

Use the `replace` method, which substitutes occurrences of one value or pattern with another.

rename

Renames axis labels (index or columns) in a DataFrame, either in-place or returning a new DataFrame.

pd.cut

Bins continuous data into intervals based on specified bin edges or quantiles.

pd.qcut

Bins data into equal-sized buckets based on sample quantiles.

detect outliers

Use boolean indexing with conditions (e.g., `np.abs(data) > 3`) or statistical methods like standard deviation.

get_dummies

Converts categorical variables into dummy/indicator variables (one-hot encoding).

split

Use the `split` method, often combined with `strip` to trim whitespace.

str.contains

Checks if each string in a Series contains a specified pattern or substring, returning a boolean Series.

str.extract

Use the `str.extract` or `str.findall` methods with a regex pattern containing groups.

str.replace

Replaces occurrences of a pattern or substring in each string of a Series.

str.cat

Use the `str.cat` method with an optional delimiter.

str.upper

Converts all characters in each string of a Series to uppercase.

str.startswith

Use the `str.startswith` method.

str.len

The length of each string in a Series.