general data knowledge

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/7

encourage image

There's no tags or description

Looks like no tags are added yet.

8 Terms

1
New cards

a data processing pipeline

Collect → Extract → Transform → Load → Analyze

This is related to ETL (Extract, Transform, Load).

  1. Collect: Gather raw data from sources such as databases, websites, or sensors.

  2. Extract: Retrieve the required data from those sources.

  3. Transform: Clean, organize, and convert the data into the required format.

  4. Load: Store the prepared data in a database or data warehouse.

  5. Analyze: Examine the data to discover patterns, trends, and useful insights.


2
New cards

collect raw data and extract raw data (explained)

Data Collection (Gathering the raw sources):At this stage, you are simply gathering entire documents and files from different places and dumping them into a central folder. You now have a massive, unorganized pile of raw data.

Data Extraction (Pulling out specific details):retrieve specific data that you need for further processing

  1. Collect: You go out to different places (website, CRM, Twitter) and bring all the raw files into one central folder.

  2. Extract: You open up the files sitting inside that central folder and pull out only the specific details you need (like star ratings or return reasons) while leaving the rest behind.

The central folder is the storage spot; the files inside it are the actual source you process during extraction!

3
New cards

IOT

It refers to physical objects connected to the internet that can collect and exchange data using sensors and other technologies.

How does IoT relate to data science?

Imagine a farm with 100 soil sensors. Every sensor sends temperature and moisture readings.

The sensors collect data.

The data is sent through a network.

The data can be stored in a database or data warehouse.

Data scientists analyze it to identify patterns or predict when crops need water.

<p>It refers to physical objects connected to the internet that can collect and exchange data using sensors and other technologies.</p><p><strong>How does IoT relate to data science?</strong></p><p>Imagine a farm with 100 soil sensors. Every sensor sends temperature and moisture readings.</p><p>The sensors <strong>collect data</strong>.</p><p>The data is sent through a network.</p><p>The data can be stored in a database or data warehouse.</p><p>Data scientists analyze it to identify patterns or predict when crops need water.</p>
4
New cards
<p>Database vs. Data Warehouse</p>

Database vs. Data Warehouse


Database: Stores and manages data for daily operations.

Data warehouse: Collects data from one or more sources, organizes it, and stores it for analysis.

Data transfer: Can happen in batches, such as overnight, or continuously.

Data sources: Databases, files, APIs, IoT devices, and other systems.

For example, a supermarket's data warehouse could combine:

Sales from its operational database.

Employee information from an HR system.

Supplier prices from Excel files.

Temperature readings from refrigerators.

5
New cards

Sources of Data Collection

  • Social networks: Facebook, Instagram, TikTok — posts, comments, likes.

  • IoT devices: Sensors, smartwatches, connected cars — temperature, heart rate, location.

  • Websites: Web pages, online searches, user clicks, browsing activity.

  • Databases: Student records, customer information, sales transactions.


6
New cards

Data → Processing → Information → Interpretation → Knowledge


  • Data = Raw facts.

  • Information = Processed data with meaning.

  • Knowledge = Understanding gained from information and discovered patterns

  • .Example: Smart agriculture 🌱

    • Data: A sensor records a soil moisture level of 15%.

    • Information: The soil moisture level is below the desired level for the plant.

    • Knowledge: The farmer understands that the plants may need irrigation when soil moisture falls below a certain threshold.


7
New cards
<p>How does a data warehouse relate to a database's day-to-day transactions?</p>

How does a data warehouse relate to a database's day-to-day transactions?

A: It receives copies of the transactions afterward, combines them with data from other sources, cleans and organizes them (ETL), and stores them long term for analysis.

8
New cards

OLTP vs OLAP

OLTP:Online Transaction Processing:Used by operational databases to handle daily transactions, such as orders and payments.

OLAP:Online Analytical Processing:Used for analyzing large amounts of data, often stored in a data warehouse.

Example: a pizza restaurant 🍕

OLTP: Records each customer's order and payment in the operational database.

OLAP: Analyzes sales data in a data warehouse to determine which pizzas sell best and on which days.