Pandas Data Handling Notes

Overview of Pandas

  • Pandas is a powerful package for data analysis and manipulation.
  • It simplifies the process of creating, manipulating, and working with data.
  • Key structures provided by Pandas include Series and DataFrames.
  • Series is a one-dimensional labeled array.
  • DataFrame is a two-dimensional labeled data structure, similar to an SQL table or spreadsheet.

Advantages of Pandas

  • Easy handling of missing data.
  • Efficient slicing of data.
  • Flexible merging, concatenating, and reshaping of data.

Data Structures in Pandas

  1. Series: A one-dimensional array-like structure containing homogeneous data.
  • Consists of:
    • Data part: Actual data values in an array.
    • Index: Associated array of labels.
  • Properties:
    • Mutable data, immutable size.
    • Row labels referred to as Index.
  1. DataFrame: A two-dimensional table of data, useful for holding heterogeneous data, often used in data manipulation and analysis.
  • Contains:
    • Row index (axis=0)
    • Column index (axis=1)
  • Mutable size & data.

Creating a Series

  • Syntax: pd.Series(data, index=idx (optional))
  • Examples:
  1. Series from an array:
    python import pandas as pd import numpy as np arr = np.array([10, 15, 18, 22]) s = pd.Series(arr)
  2. Series with custom index:
    python arr = np.array(['a', 'b', 'c', 'd']) s = pd.Series(arr, index=['first', 'second', 'third', 'fourth'])

Series Operations

  • Mathematical operations available:
  • Multiply series by a number.
  • Compute square of elements.
  • Filter elements based on conditions (e.g., greater than 2).

Accessing Data in Series

  • Use methods like head() and tail() to retrieve data.
  • head(n): Returns the first n rows.
  • tail(n): Returns the last n rows.
  • Indexing via:
  • loc: Access using labels, e.g., series_name.loc[0:2].
  • iloc: Access using integer-location, e.g., series_name.iloc[0:2].
  • Basic Slicing: series_name[start:end:step].

DataFrame Creation

  • Methods to create DataFrames:
  1. From Series
  2. From lists
  3. From dictionaries
  4. From numpy 2D arrays
  • Example of creating DataFrame from dictionaries of Series:
  empdata = {'Doj': [...], 'empid': [...], 'ename': [...]} 
  df = pd.DataFrame(empdata)

DataFrame Operations

  1. Selecting Columns:
  • Access via df['column_name'] or df.column_name.
  • Access multiple columns using df[[col1, col2]].
  1. Adding/Renaming Columns:
  • df['new_col'] = values
  • Renaming existing columns: df.columns = ['new_name1', 'new_name2'].
  1. Deleting Columns:
  • Methods: del, drop(), pop().
  1. DataFrame Access: Using loc and iloc for row and column access based on index or label.

CSV Handling

  • Importing CSV files using pd.read_csv(file_path).
  • Exporting DataFrames to CSV using df.to_csv('file_path').