6 -Data Loading, Storage And File Format

read_csv

It loads delimited data from a file, URL, or file-like object, using a comma as the default delimiter.

read_table

Use read_table (e.g., pd.read_table('file.txt', sep=' ')).

header parameter in read_csv

It specifies the row number to use as column names (default is 0). Use header=None if there is no header row.

skiprows parameter

Use the skiprows parameter (e.g., pd.read_csv('file.csv', skiprows=[0, 2, 3])).

na_values parameter in read_csv

It specifies a list of strings to treat as missing values (e.g., na_values=['NULL', 'NA']).

nrows parameter

Use the nrows parameter (e.g., pd.read_csv('file.csv', nrows=5)).

chunksize parameter in read_csv

It reads the file in chunks of the specified size, useful for large datasets (e.g., chunker = pd.read_csv('file.csv', chunksize=1000)).

to_csv method

Use the to_csv method (e.g., data.to_csv('output.csv')).

index=False in to_csv

Set index=False in to_csv (e.g., data.to_csv('output.csv', index=False)).

read_json

It reads data from a JSON string or file into a DataFrame (e.g., pd.read_json('data.json')).

to_json method

Use the to_json method (e.g., data.to_json()).

read_html

It parses all tables in an HTML file or URL into a list of DataFrames (e.g., pd.read_html('page.html')).

read_excel

Use read_excel (e.g., pd.read_excel('file.xlsx', sheet_name='Sheet1')).

to_excel method

Use the to_excel method (e.g., data.to_excel('output.xlsx', sheet_name='Sheet1')).

read_sql

It reads the results of a SQL query into a DataFrame (e.g., pd.read_sql('SELECT * FROM table', con)).

read_sql with a connection object

Use read_sql with a connection object (e.g., pd.read_sql('SELECT * FROM test', sqlite3.connect('mydata.sqlite'))).

HDFStore

It provides a dict-like interface for storing DataFrames in HDF5 format (e.g., store = pd.HDFStore('mydata.h5')).

read_hdf function

Use the read_hdf function (e.g., pd.read_hdf('mydata.h5', 'key')).

to_pickle

It serializes a DataFrame to disk in pickle format (e.g., data.to_pickle('data.pkl')).

read_pickle

Use read_pickle (e.g., pd.read_pickle('data.pkl')).

sep parameter in read_csv

It specifies the delimiter to use (e.g., sep=',' for CSV, sep='\s+' for whitespace).

names parameter in read_csv

Use the names parameter (e.g., pd.read_csv('file.csv', header=None, names=['col1', 'col2'])).

index_col parameter in read_csv

It specifies the column(s) to use as the DataFrame's index (e.g., index_col='message').

handle missing values in read_csv

Use the na_values parameter to specify strings to treat as missing (e.g., na_values=['NA', 'NULL']).

parse_dates parameter in read_csv

It attempts to parse specified columns as datetime objects (e.g., parse_dates=['date']).