UNIT 3

Data Handling with Pandas

Unit 3 Concepts

This unit covers data handling using the Pandas library in Python, focusing on efficiently processing large datasets. Topics include data wrangling, reshaping, and merging techniques.

Contents Overview

  1. Data Handling:

    • Challenges of handling large datasets

    • General techniques for managing large data volumes

    • Programming tips for large datasets (Slides 4-7)

  2. Data Wrangling:

    • Cleaning, transforming, and merging data (Slides 16-27)

  3. Reshaping:

    • Combining and merging datasets

    • Merging on index

    • Concatenation

    • Combining with overlap

    • Reshaping data (Slides 30-37)

Important Terminologies

  1. GPU (Graphics Processing Unit):

    • Specialized processor for parallel processing, accelerating tasks like machine learning, data science, and simulations.

  2. range() Function:

    • Built-in Python function generating a sequence of numbers for loops and iterative operations.

    • Syntax: range(start, stop, step)

    • Example:
      python for i in range(5): print(i) # Output: 0, 1, 2, 3, 4

  3. Axis in NumPy:

    • axis = 0: Refers to the vertical axis (rows).

    • axis = 1: Refers to the horizontal axis (columns).

    • Example:
      python import numpy as np data = np.array([[1, 2], [3, 4]]) print(np.sum(data, axis=0)) # Output: [4, 6] print(np.sum(data, axis=1)) # Output: [3, 7]

I. Data Handling

  • Involves processing large data volumes efficiently.

  • Avoid loading everything into memory to prevent system crashes or performance slowdown.

  • Use chunking or optimized techniques.

  • Pandas provides tools for memory-efficient data loading and processing.

  • Example:
    python import pandas as pd chunk_size = 1000 for chunk in pd.read_csv('large_dataset.csv', chunksize=chunk_size): print(chunk.head())

Challenges in Handling Large Data

  • Infinite looping algorithms

  • Out-of-memory errors

  • Speed issues

General Techniques for Handling Large Data Volumes

  • Choose the right algorithms

  • Choose the right data structures

  • Choose the right tools

General Programming Tips for Large Datasets

  • Feed the CPU compressed data:

    • Avoid CPU starvation by feeding compressed data instead of inflated raw data.

  • Make use of the GPU:

    • If computations are parallelizable, GPUs offer higher throughput than CPUs.

  • Use multiple threads:

    • Parallelize computations on the CPU using Python threads.

Introduction to Pandas

  • Powerful Python library for data manipulation, analysis, and cleaning.

  • Provides high-performance data structures like DataFrame and Series for structured data.

Key Features:

  • Data Structures:

    • Series: One-dimensional labeled array (like a column in a spreadsheet).

    • DataFrame: Two-dimensional labeled data structure (like a table with rows and columns).

  • Data Manipulation:

    • Filter, sort, and reshape data.

    • Handle missing data effectively with dropna() and fillna().

Series

  • A Pandas Series is like a column in a table.

  • It is a one-dimensional array holding data of any type.
    python import pandas as pd a = [1, 7, 2] myvar = pd.Series(a) print(myvar) # Output: # 0 1 # 1 7 # 2 2 # dtype: int64

  • Use Cases:

    1. Tracking daily closing stock prices.

    2. Monitoring a sensor's temperature readings over time.

    3. Tracking website visitors per hour.

    4. Expenses tracking.

    5. Monitoring error counts in server logs.

DataFrame

  • A DataFrame is a two-dimensional, size-mutable, labeled data structure in Pandas.

  • Similar to a table in a relational database or an Excel spreadsheet, with rows and columns.

  • Each column in a DataFrame can be of a different data type.

  • Use Cases:

    1. E-commerce: Store product details like names, prices, ratings, and quantities for analysis.

    2. Healthcare: Patient data analysis, medical test records, and treatment monitoring.

DataFrame - Example

data = {'Timestamp': ['2024-12-18 10:00', '2024-12-18 10:05'],
        'Error_Code': [500, 404],
        'Description': ['Server Error', 'Page Not Found']}
df = pd.DataFrame(data)
print(df)
# Output:
#           Timestamp  Error_Code     Description
# 0  2024-12-18 10:00         500    Server Error
# 1  2024-12-18 10:05         404  Page Not Found

Python Libraries

  • Pandas: For structured data operations like importing CSV files, creating DataFrames, and data preparation.

  • NumPy: Mathematical library with a powerful N-dimensional array object, linear algebra, Fourier transform, etc.

  • Matplotlib: For data visualization.

  • SciPy: Library with linear algebra modules.

Pandas - Exploring Data Using Series

  • Pandas Series is a one-dimensional labeled array capable of holding data of any type.

  • Basic operations:

    • Creating a Series

    • Accessing elements of a Series

Creating a Series

  • From array: Import NumPy and use the array function.

  • From Lists: Create a list and then create a Series from it.

Accessing Elements of Series

  • With Position: Use the index operator [] to access an element by integer index.

  • Using Label (index): Set values by index label to access elements.

Exploring Data Using DataFrames

  • A DataFrame is a two-dimensional data structure with data aligned in rows and columns.

  • Store any number of datasets and perform operations like arithmetic operations, column/row selection, and addition.

Creating a DataFrame

  1. Empty DataFrame: Created by calling the DataFrame constructor.

  2. Using List: Create a DataFrame using a single list or a list of lists.

  3. From dict of ndarray/lists: All ndarrays must be of the same length. If an index is passed, its length should match the array length.

  4. From lists using dictionary: Transform a dictionary of lists into a DataFrame using pandas.DataFrame.

Viewing Data in a DataFrame

  • Pandas Dataframe/Series.head(): Displays the first n rows.

  • Pandas Dataframe/Series.tail(): Displays the last n rows.

  • Pandas DataFrame describe() Method: Generates descriptive statistics of DataFrame columns.

Index Objects

  • The Index allows quick data searches, efficient slicing, and keeps data properly aligned.

  • Used to label rows of a DataFrame or elements in a Series.

  • Indexes are immutable.

The Index Class

  • Basic object for storing all index types in Pandas.

Key Features:

  • Immutable: Cannot be modified once created.

  • Alignment: Ensures data from different DataFrames or Series can be combined correctly.

  • Slicing: Allows fast slicing and retrieval of data based on labels.

Syntax:
class pandas.Index(data=None, dtype=None, copy=False, name=None, tupleize_cols=True)

  • data: Data for the index (array-like structure or another index object).

  • dtype: Data type for the index values.

  • copy: Boolean to create a copy of the input data.

  • name: Label for the index.

  • tupleize_cols: Boolean to create MultiIndex if possible.

Types of Indexes in Pandas

  • NumericIndex: Contains numerical values (default index).

  • CategoricalIndex: Deals with duplicate labels; memory-efficient for large numbers of duplicates.

  • MultiIndex: Represents multiple levels or layers in the index (hierarchical).

  • IntervalIndex: Represents intervals (ranges) in your data.

  • DatetimeIndex: Represents date and time values for time-series data.

  • TimedeltaIndex: Represents duration between two dates or times.

  • PeriodIndex: Represents regular periods in time (quarters, months, years).

Reindex

  • Used to change row or column labels of a DataFrame to match a new set of indices.

  • Essential for aligning data with a specific structure or dealing with missing data.

  • Missing labels default to NAN values.

Syntax:
DataFrame.reindex(labels=None, index=None, columns=None, axis=None, method=None, copy=True, fill_value=NaN)

  • labels: New labels/indexes.

  • index/columns: New row/column labels.

  • fill_value: Value to use for filling missing entries (default is NAN).

  • method: Method for filling holes (ffill, bfill, etc.).

Types of Reindexing

  1. Reindexing Rows: Reindex a single or multiple rows using reindex(). Missing values are assigned NaN.

  2. Reindexing Columns: Reindex a single column using reindex(). Missing values are assigned NaN. Align columns using the axis keyword.

  3. Using ffill and bfill: Handle missing values after reindexing using forward fill and backward fill methods.

Drop Entry - Delete Rows/Columns

  • Use pandas.drop() to delete rows or columns from a DataFrame.

Syntax:
DataFrame.drop(labels=None, axis=0, index=None, columns=None, level=None, inplace=False, errors='raise')

  • labels: String or list of strings referring to row or column name.

  • axis: 0 for Rows, 1 for Columns.

  • index/columns: Alternative to axis; cannot be used together.

  • level: Specifies level in case of a multi-level index.

  • inplace: Makes changes in the original DataFrame if True.

  • errors: Ignores errors if any value from the list doesn’t exist and drops the rest when errors = 'ignore'.

Data Alignment

  • Use dataframe.style.set_properties() function to align columns to the left in a Pandas DataFrame.

Syntax:
Styler.set_properties(subset=None, **kwargs)

  • subset: A valid slice for data to limit the style application.

  • **kwargs: Dictionary of property, value pairs to be set for each cell.

Index Hierarchy

  • Rows and columns both have indexes (rows are called index, columns are column names).

  • Hierarchical Indexes (Multi-indexing): Setting more than one column name as the index.

Data Acquisition Process

  1. Identifying Data Sources: Determine sources from which data will be collected (physical devices, digital platforms, databases).

  2. Data Collection: Collect data using automatic or manual systems (capturing real-time data, scraping data from websites).

  3. Data Preprocessing: Clean the data, remove duplicates, and transform it into a structured format.

  4. Data Storage: Store data in a database, data warehouse, or cloud storage.

  5. Data Validation: Check for inconsistencies, missing values, or errors.

  6. Data Integration: Combine data from different sources and ensure consistency.

  7. Data Analysis: Use statistical methods or machine learning algorithms to extract insights.

Gather Information from Different Sources

  • Pandas can read data from multiple sources, including CSV files, Excel spreadsheets, SQL databases, and web APIs.

  1. CSV Files:

    • Use read_csv().
      python import pandas as pd df = pd.read_csv('titanic.csv') df.head()

  2. Excel Files:

    • Use read_excel().
      python df = pd.read_excel("Financial Sample.xlsx", engine="openpyxl") df.head()

    • Install openpyxl library: pip install openpyxl

  3. SQL Databases:

    • Use read_sql().
      python import sqlalchemy from sqlalchemy import create_engine, text sakila = 'mysql+pymysql://root:admin12345@localhost/sakila' engine = create_engine(sakila) connection = engine.connect() exct = connection.execute(text("SELECT * FROM film")) cols = exct.keys() data = pd.DataFrame(exct.fetchall(), columns=cols) data.head()

Key Steps for Using Web APIs in Pandas:

  1. Import Required Libraries: requests or httpx for HTTP requests, Pandas for data handling.

  2. Make an API Call: Use requests to send a GET or POST request.

  3. Parse API Response: Convert JSON or XML data into a usable format.

  4. Load Data into a DataFrame: Use pd.DataFrame() or pd.json.normalize().

import requests
import pandas as pd

# Example API URL
api_url = 'https://jsonplaceholder.typicode.com/posts'

# Make a GET request
response = requests.get(api_url)

# Check if the request was successful
if response.status_code == 200:
    data = response.json()
    df = pd.DataFrame(data)
    print(df.head())
else:
    print(f"Failed to fetch data: {response.status_code}")

# API returning CSV
csv_url = "https://example.com/data.csv"

# Load directly into a pandas DataFrame
df = pd.read_csv(csv_url)
print(df.head())

Open Data Sources

  • Publicly available datasets that can be directly accessed and imported into Pandas for data analysis.

Popular Open Data Sources:

  1. Kaggle: Datasets across various domains.

  2. UCI Machine Learning Repository: Datasets for machine learning and data analysis.

  3. Google Dataset Search: Aggregates datasets from different open sources.

  4. World Bank Open Data: Global development data.

  5. US Government Open Data (data.gov): Public datasets across sectors.

  6. GitHub Repositories: Open datasets hosted on GitHub.

Web Scraping

  • Extracting data from webpages and loading it into a DataFrame.

  • Combine Pandas with tools like BeautifulSoup or requests to retrieve and parse content.

How to Perform Web Scraping in Pandas:

  1. Tools Required:

    • Pandas, Requests, BeautifulSoup.

  2. Basic Workflow:

    • Fetch HTML content.

    • Parse the webpage.

    • Convert and structure data into a DataFrame.

import pandas as pd

# URL containing the table
url = "https://en.wikipedia.org/wiki/List_of_countries_by_population_(United_Nations)"

# Read all tables from the webpage
tables = pd.read_html(url)

# Display the first table
df = tables[0]
print(df.head())

II. Data Wrangling

  • Transforming and mapping data from raw form into a valuable format for analysis.

  • Key Steps:

    • Cleaning: Handling missing values, removing duplicates, correcting inconsistencies, and dealing with outliers.

    • Transforming: Converting data from one format to another, scaling or normalizing data, creating new features and aggregating data.

    • Merging: Combining data from multiple sources into a single dataset.

Libraries:

  • Pandas: Data structures and functions for data manipulation and analysis.

  • NumPy: Supports numerical operations.

import pandas as pd
import numpy as np

data = {'A': [1, 2, np.nan, 4], 'B': [5, np.nan, 7, 8], 'C': [9, 10, 11, np.nan]}
df = pd.DataFrame(data)

# Imputation with mean
df_fillna_mean = df.fillna(df.mean())
print("Fill NA with Mean:\n", df_fillna_mean)

# Imputation with forward fill
df_fillna_ffill = df.ffill()
print("\nFill NA with Forward Fill:\n", df_fillna_ffill)

# Remove rows with NA
df_dropna = df.dropna()
print("\nRemove rows with NA:\n", df_dropna)
Duplicate and Inconsistent Data

  • Duplicate Data: Identifying and removing duplicate rows.
    python data = {'A': [1, 2, 2, 4], 'B': [5, 6, 6, 8]} df = pd.DataFrame(data) df_drop_duplicates = df.drop_duplicates() print("Remove Duplicates:\n", df_drop_duplicates)

  • Inconsistent Data: Converting dates to a consistent format.

import pandas as pd

data = {'Date': ['2024-01-01', '01/01/2024', '20240101'], 'Value': [10, 20, 30]}
df = pd.DataFrame(data)

# Convert to datetime format
df['Date'] = pd.to_datetime(df['Date'], format='mixed', errors='coerce')
df['Date_formatted'] = df['Date'].dt.strftime('%d-%m-%Y')
print("Consistent Date Format:\n", df)
Transforming

  1. Data Type Conversion: Modifying data into a suitable format.
    ```python
    import pandas as pd

    data = {'Value': ['1', '2', '3','Hello']}
    df = pd.DataFrame(data)

Convert to numeric

df['Value'] = pd.to_numeric(df['Value'], errors = 'coerce')
print("Convert to Numeric:\n", df.dtypes)
print(df)
```

  1. Scaling and Normalization: Rescaling data to a fixed range or normalizing the distribution.

    • Normalization: Rescales data to a fixed range (usually 0 to 1) X<em>scaled=X−X</em>minX<em>max−X</em>minX<em>{scaled} = \frac{X - X</em>{min}}{X<em>{max} - X</em>{min}}. Compresses data into a defined range. Used when data follows, a non-Gaussian distribution. Example: Neural Networks, k-Nearest Neighbors, clustering.

    • Standardization: Centers data around 0 with a standard deviation of 1. Represented by: Xstandardized=X−μσX_{standardized} = \frac{X - \mu}{\sigma}. Normalizes the distribution of the data. Used when the data is normally distributed or required by the algorithm. Example: Logistic Regression, PCA, SVM.
      ```python
      from sklearn.preprocessing import MinMaxScaler, StandardScaler

data = {'Value': [1, 5, 10, 15]}
df = pd.DataFrame(data)

scaler = MinMaxScaler()
df['Valuescaled'] = scaler.fittransform(df[['Value']])
print("Min-Max Scaling:\n", df)

scaler = StandardScaler()
df['Valuestandardized'] = scaler.fittransform(df[['Value']])
print("\nStandardization:\n", df)
```

  1. Creating New Features: Deriving new features from existing data.

    data = {'Name': ['John Doe', 'Jane Smith'], 'Age': [25, 30]}
df = pd.DataFrame(data)

# Split name into first and last name
df[['First Name', 'Last Name']] = df['Name'].str.split(' ', expand=True)
print("New Features:\n", df)
Merging

  • Combining data from multiple sources.

  • Different Types of Merges:

    • Inner Join: Only includes rows with matching keys in both DataFrames.

    • Outer Join: Includes all rows from both DataFrames, with NaN where there are no matches.

    • Left Join: Includes all rows from the left DataFrame and matching rows from the right DataFrame.

    • Right Join: Includes all rows from the right DataFrame and matching rows from the left DataFrame.

data1 = {'ID': [1, 2, 3], 'Value1': [10, 20, 30]}
df1 = pd.DataFrame(data1)
print('Data Frame 1\n',df1)

data2 = {'ID': [2, 3, 4], 'Value2': [25, 35, 45]}
df2 = pd.DataFrame(data2)
print('Data Frame 2\n',df2)

# Inner Join
df_inner = pd.merge(df1, df2, on='ID', how='inner')
print("Inner Join:\n", df_inner)

# Outer Join
df_outer = pd.merge(df1, df2, on='ID', how='outer')
print("\nOuter Join:\n", df_outer)

# Left Join
df_left = pd.merge(df1, df2, on='ID', how='left')
print("\nLeft Join:\n", df_left)

III. Reshape

  • Changing the dimensions of a data structure without changing its underlying data.

  • Various Reshape Operations:

    • Combining and Merging Datasets

    • Merging on Index

    • Concatenate

    • Combining with overlap

Combining and Merging Datasets
import pandas as pd

df1 = pd.DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'a', 'b'], 'data1': range(7)})
df2 = pd.DataFrame({'key': ['a', 'b', 'd'], 'data2': range(3)})

# Inner merge (default)
merged_inner = pd.merge(df1, df2, on='key')  # Only matching keys
print("Inner Merge:\n", merged_inner)

# Left merge
merged_left = pd.merge(df1, df2, on='key', how='left')  # All of df1, matching in df2
print("\nLeft Merge:\n", merged_left)

# Right merge
merged_right = pd.merge(df1, df2, on='key', how='right')  # All of df2, matching in df1
print("\nRight Merge:\n", merged_right)

# Outer merge
merged_outer = pd.merge(df1, df2, on='key', how='outer')  # All rows from both
print("\nOuter Merge:\n", merged_outer)
Merging on Index
left1 = pd.DataFrame({'key': ['a', 'b', 'a', 'a', 'b', 'c'], 'value': range(6)})
print('Data Frame 1\n',left1)

right1 = pd.DataFrame({'group_val': [3.5, 7]}, index=['a', 'b'])
print("Data Frame 2\n",right1)

merged_index = pd.merge(left1, right1, left_on='key', right_index=True)
print("\nMerge on Index:\n", merged_index)
Concatenate - using concat()
import numpy as np
import pandas as pd

s1 = pd.Series([0, 1], index=['a', 'b'])
print('Series 1\n',s1)

s2 = pd.Series([2, 3, 4], index=['c', 'd', 'e'])
print('Series 2\n',s2)

s3 = pd.Series([5, 6], index=['f', 'g'])
print('Series 3\n',s3)

concatenated = pd.concat([s1, s2, s3])
print("\nConcatenate Series:\n", concatenated)

df1 = pd.DataFrame(np.arange(6).reshape(3, 2), index=['a', 'b', 'c'], columns=['one', 'two'])
df2 = pd.DataFrame(5 + np.arange(4).reshape(2, 2), index=['a', 'c'], columns=['three', 'four'])

concatenated_df = pd.concat([df1, df2], axis=1, sort=False)

print("\nConcatenate DataFrames:\n", concatenated_df)
Combining with Overlap (combine_first)
import numpy as np
import pandas as pd

df1 = pd.DataFrame({'a': [1., np.nan, 5., np.nan],
                    'b': [np.nan, 2., np.nan, 6.],
                    'c': range(2, 6)})
print('Data Frame 1\n',df1)

df2 = pd.DataFrame({'a': [5., 4., np.nan, 3., 7.],
                    'b': [np.nan, 3., 4., 6., 8.]})
print('Data Frame 2\n',df2)

combined = df1.combine_first(df2)
print("\nCombine First:\n", combined)
Reshape : Stacking and Unstacking (STACK, UNSTACK)
import pandas as pd

data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada', 'Nevada'],
        'year': [2000, 2001, 2002, 2001, 2002, 2003],
        'pop': [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]}
frame = pd.DataFrame(data)
print('Data Frame:\n',frame)

frame_stacked = frame.set_index(['state', 'year']).stack()
print("\nStacked:\n", frame_stacked)

frame_unstacked = frame_stacked.unstack()
print("\nUnstacked:\n", frame_unstacked)

Outliers and Noise Anomalies:

  • Outliers are data points that deviate significantly from the majority of the dataset.

  • Noise can arise due to errors, variability, or external influences.

String Manipulation Functions

  1. Changing Case

    • lower(): Converts a string to lowercase.

    • upper(): Converts a string to uppercase.

    • title(): Converts the first character of each word to uppercase.

    • capitalize(): Capitalizes the first character of the string.

    • swapcase(): Swaps the case of each character.

  2. Stripping and Padding

    • strip(): Removes leading and trailing spaces.

    • lstrip(): Removes leading spaces

    • rstrip(): Removes trailing spaces.

    • zfill(width): Pads a numeric string with zeros to the left.

    • center(width): Centers the string with spaces.

    • ljust(width): Left-aligns the string.

    • rjust(width): Right-aligns the string.

  3. Finding and Replacing

    • find(substring): Returns the index of the first occurrence of the substring.

    • rfind(substring): Returns the index of the last occurrence of the substring.

    • replace(old, new): Replaces occurrences of old with new.

  4. Splitting and Joining

    • split(separator): Splits the string into a list based on the separator.

    • rsplit(separator): Splits the string into a list from the right.

    • splitlines(): Splits a string into a list at newline characters.

    • join(iterable): Joins elements of an iterable into a string.

  5. Checking String Properties

    • isalpha(): Checks if all characters are alphabets.

    • isdigit(): Checks if all characters are digits.

    • isalnum(): Checks if all characters are alphanumeric.

    • isspace(): Checks if the string contains only whitespace.

    • startswith(substring): Checks if the string starts with a specific substring.

    • endswith(substring): Checks if the string ends with a specific substring.

Summarizing

  • Summarizing involves aggregating the data to provide a concise representation.

  • Statistical measures: Mean, median, standard deviation, minimum, maximum, etc., for numerical data.

  • Frequency tables: For categorical data to show the distribution of values.

  • Data visualization: Histograms or box plots to understand data trends and anomalies

Binning

  • Binning involves grouping continuous data into discrete intervals or bins.

  • Example: Grouping ages into bins like 0–18 (child), 19–35 (youth), 36–60 (adult), and 60+ (senior).

Classing

  • Classing organizes data into meaningful categories or classes, often for categorical variables.

  • This may involve:

    • Converting numerical data into categorical classes.

    • Grouping similar categories together (e.g., combining "Manager" and "Team Lead" into "Leadership").

Data Standardization

  • Data standardization is a technique to rescale data so that it has a mean of 0 and a standard deviation of 1.

Groupby

import pandas as pd
import numpy as np
df1=pd.read_csv('datasets/stackdatasetexample.csv')
print(df1)
#print(df1.groupby(["state"])[['name']].count())
j=df1['state'].value_counts()
print(j)

  • Data Aggregation: Process where raw data is gathered and expressed in a summary form for statistical analysis.

    • Time aggregation: All data points for a single resource over a specified time period.

    • Spatial aggregation: All data points for a group of resources over a specified geographical area.

  • Summary Statistics: Used to communicate the largest amount of information as simply as possible.

    • Mean: Arithmetic mean - sum of values of a data set divided by number of values: (1+2+2+3+4+7+9)/7(1+2+2+3+4+7+9)/7

    • Median: Middle value separating the greater and lesser halves of a data set

    • Mode: Most frequent value in a data set

Hierarchical Clustering

  • Produces a set of nested clusters organized as a hierarchical tree

  • Can be visualized as a dendrogram – A tree-like diagram that records the sequences of merges or splits

  • Strengths of Hierarchical Clustering

    • No assumptions on the number of clusters. Any desired number of clusters can be obtained by 'cutting' the dendrogram at the proper level

    • Hierarchical clusterings may correspond to meaningful taxonomies. Example in biological sciences (e.g., phylogeny reconstruction, etc), web (e.g., product catalogs) etc.

Hierarchical Clustering Algorithms

  • Two main types of hierarchical clustering

    • Agglomerative: Start with the points as individual clusters. At each step, merge the closest pair of clusters until only one cluster (or k clusters) left

    • Divisive: Start with one, all-inclusive cluster. At each step, split a cluster until each cluster contains a point (or there are k clusters)

  • Traditional hierarchical algorithms use a similarity or distance matrix - Merge or split one cluster at a time

Complexity of hierarchical clustering

  • Distance matrix is used for deciding which clusters to merge/split At least quadratic in the number of data points

  • Not usable for large datasets

Agglomerative clustering algorithm

  • Most popular hierarchical clustering technique