2.1.4 - Data Sources.

Used for storing structured data crucial for mission-critical applications such as transaction processing systems, Human Resources (HR) management, and workflow automation. Examples include enterprise resource planning (ERP) systems and customer relationship management (CRM) systems.
Prominent database management systems (DBMS) include SQL Server, Oracle, MySQL, and IBM DB2, which ensure data integrity, consistency, and transactional atomicity (ACID properties).
They serve as a primary source for in-depth business analytics, enabling insights into performance metrics like sales by region, inventory levels, and CRM-based demand projections by leveraging well-defined schemas and relationships between tables.

Flat file: A simple plain-text file where each line represents a single record. Data within a record is typically separated by a delimiter, with Comma Separated Values (CSV) being the most common format. Each flat file usually maps directly to a single table structure with columns defined by the order or header row.
Spreadsheets (Excel, Google Sheets, etc.): Offer tabular data organization with the capability for multiple worksheets within a single file. They support advanced features like formulas for calculations, charting capabilities, and custom formatting for data visualization and presentation.
Widely utilized for the distribution and exchange of purchased or public datasets, including demographic information, Point-of-Sale (POS) data from retailers, and various environmental or weather datasets, often serving as an accessible format for non-technical users.

XML (Extensible Markup Language): Employs a tag-based markup structure, similar to HTML but designed for data description rather than display. It supports highly hierarchical and nested data structures, which are richer and more flexible than the flat, tabular format of CSVs.
Frequently used for representing semi-structured data, where the schema defining the data structure is flexible or often embedded within the data itself. Common applications include online survey responses, detailed bank statements, and configuration files where data elements have clear relationships but schema rigidity is not required.

APIs (Application Programming Interfaces) and Web Services: Provide a mechanism for on-demand data retrieval over a network, typically using HTTP/HTTPS requests. They return data in various formats including plain text, XML, JSON (JavaScript Object Notation), HTML, or multimedia files.
Typical uses:
- Social media APIs: Enable access to user posts, interactions, and profile data, which is invaluable for sentiment analysis and opinion mining to understand public perception.
- Stock-market APIs: Offer real-time stock quotes, historical price data, and trading volumes, crucial for financial analytics, algorithmic trading, and market trend prediction.
- Lookup/validation APIs: Used for data cleansing and enrichment, such as converting a ZIP code to a city and state, validating addresses, or retrieving company information, enhancing data accuracy.
Also serve as a common interface for direct database access, whether the database is internal to an organization or exposed externally for partners or public consumption.

Automated extraction: A process of automatically extracting data from unstructured web pages, often referred to as screen scraping. This involves programmatically parsing HTML content to pull specific pieces of information.
Data capture: Can capture a wide variety of public data, including plain text content, contact information, images, product details (e.g., prices, descriptions), and reviews. Use cases include competitive price comparison, lead generation for sales, forum analysis for community insights, and the creation of large datasets for machine learning model training.
Tools: Popular libraries and frameworks include Beautiful Soup (for parsing HTML/XML), Scrapy (a fast high-level web crawling and scraping framework), Pandas (for data manipulation post-scraping), and Selenium (for automating browser interactions for dynamic web content).

Continuous flows: Represent constant streams of data that are time-stamped and often geo-tagged, originating from diverse sources. These include data from IoT (Internet of Things) sensors (e.g., temperature, pressure), GPS devices (location data), application logs (events, errors), social media feeds (live posts, trends), and financial market tickers (stock prices, trades).
Use-cases:
- Demand prediction: Analyzing real-time sales or interaction data to forecast consumer demand.
- Threat detection: Monitoring network traffic or system logs for unusual patterns indicative of security breaches.
- Sentiment analysis: Real-time analysis of social media streams to gauge public mood or reaction to events.
- Web click monitoring: Tracking user behavior on websites for personalization or fraud detection.
- Real-time flight rebooking: Adjusting travel plans dynamically based on immediate conditions like delays or cancellations.
Processing frameworks: Specialized platforms like Apache Kafka (for distributed streaming platforms), Spark Streaming (for scalable fault-tolerant stream processing), and Storm (for real-time distributed computation) are used to ingest, process, and analyze these high-velocity data streams.

Really Simple Syndication (RSS): A web feed format used to publish frequently updated content, such as blog entries, news headlines, and podcast episodes, in a standardized, machine-readable format.
A feed reader or aggregator software automatically converts the raw RSS XML text into a user-friendly, consumable update stream, presenting new content as soon as it's published. This allows users or downstream applications to subscribe to and receive updates without manually visiting each website.