1/7
Looks like no tags are added yet.
a data processing pipeline
Collect → Extract → Transform → Load → Analyze
This is related to ETL (Extract, Transform, Load).
Collect: Gather raw data from sources such as databases, websites, or sensors.
Extract: Retrieve the required data from those sources.
Transform: Clean, organize, and convert the data into the required format.
Load: Store the prepared data in a database or data warehouse.
Analyze: Examine the data to discover patterns, trends, and useful insights.
collect raw data and extract raw data (explained)
Data Collection (Gathering the raw sources):At this stage, you are simply gathering entire documents and files from different places and dumping them into a central folder. You now have a massive, unorganized pile of raw data.
Data Extraction (Pulling out specific details):retrieve specific data that you need for further processing
Collect: You go out to different places (website, CRM, Twitter) and bring all the raw files into one central folder.
Extract: You open up the files sitting inside that central folder and pull out only the specific details you need (like star ratings or return reasons) while leaving the rest behind.
The central folder is the storage spot; the files inside it are the actual source you process during extraction!
IOT
It refers to physical objects connected to the internet that can collect and exchange data using sensors and other technologies.
How does IoT relate to data science?
Imagine a farm with 100 soil sensors. Every sensor sends temperature and moisture readings.
The sensors collect data.
The data is sent through a network.
The data can be stored in a database or data warehouse.
Data scientists analyze it to identify patterns or predict when crops need water.


Database vs. Data Warehouse
Database: Stores and manages data for daily operations.
Data warehouse: Collects data from one or more sources, organizes it, and stores it for analysis.
Data transfer: Can happen in batches, such as overnight, or continuously.
Data sources: Databases, files, APIs, IoT devices, and other systems.
For example, a supermarket's data warehouse could combine:
Sales from its operational database.
Employee information from an HR system.
Supplier prices from Excel files.
Temperature readings from refrigerators.
Sources of Data Collection
Social networks: Facebook, Instagram, TikTok — posts, comments, likes.
IoT devices: Sensors, smartwatches, connected cars — temperature, heart rate, location.
Websites: Web pages, online searches, user clicks, browsing activity.
Databases: Student records, customer information, sales transactions.
Data → Processing → Information → Interpretation → Knowledge
Data = Raw facts.
Information = Processed data with meaning.
Knowledge = Understanding gained from information and discovered patterns
.Example: Smart agriculture 🌱
Data: A sensor records a soil moisture level of 15%.
Information: The soil moisture level is below the desired level for the plant.
Knowledge: The farmer understands that the plants may need irrigation when soil moisture falls below a certain threshold.

How does a data warehouse relate to a database's day-to-day transactions?
A: It receives copies of the transactions afterward, combines them with data from other sources, cleans and organizes them (ETL), and stores them long term for analysis.
OLTP vs OLAP
OLTP:Online Transaction Processing:Used by operational databases to handle daily transactions, such as orders and payments.
OLAP:Online Analytical Processing:Used for analyzing large amounts of data, often stored in a data warehouse.
Example: a pizza restaurant 🍕
OLTP: Records each customer's order and payment in the operational database.
OLAP: Analyzes sales data in a data warehouse to determine which pizzas sell best and on which days.