Ingeniería de Datos - Primer Parcial

Key Stages and Components of Data Engineering

The fundamental components of data engineering are organized into several stages of the data life cycle, distinct repositories, and processing patterns. These concepts form the foundation for managing information from its origin to its final utility in analytical systems.

Generation is the initial stage of the data life cycle where data originates from source systems. These sources include applications, sensors, logs, and APIs. Once generated, data must be moved through Ingestion, the process by which data is transferred from source systems into the analytical ecosystem. Ingestion serves as the critical bridge between operational systems and analytical environments.

Transformation is the stage where raw data is converted into a usable format through cleaning, integration, standardization, and the application of business rules. This stage is responsible for defining the meaning and consistency of the data. Finally, Serving is the stage where processed data is made available to end-users, analysts, or machine learning systems for value generation.

In terms of repositories, a Data Lake is a storage system that allows for the collection of large volumes of data in its original format. It does not impose a rigid schema at the time of storage, often referred to as "schema-on-read." In contrast, a Data Warehouse is a structured repository optimized for analytical queries and the analysis of historical data.

Processing patterns are categorized by how and when data is handled. Batch processing handles data in defined blocks or lots at specific time intervals. Streaming processing involves handling data in real-time or near-real-time as it is generated, allowing for low-latency applications. Within these flows, two primary patterns exist for loading and transforming data: ETL (Extract, Transform, Load), the traditional pattern where data is transformed before being loaded into the destination system, and ELT (Extract, Load, Transform), where data is loaded into storage first and transformed subsequently.

Data Lifecycle and Architectural Principles

The engineering of data involves several core principles regarding the lifecycle and architecture of systems. A critical aspect of the data life cycle is the understanding that errors during the generation stage can propagate throughout the entire pipeline. While the data engineer does not have direct control over the source systems where data is generated, they must manage the ingestion and subsequent stages. Furthermore, storage is not merely about saving data; it has significant impacts on the total cost of the system. Security and governance are not isolated steps but rather cross-cutting aspects that must be addressed in every stage of the life cycle.

Architectural design for data must be dynamic rather than static, as requirements evolve over time. High-quality architectures favor distributed systems, which allow for the scaling of processing and storage. A key design goal is low coupling (decoupling) between components, which enables parts of the system to evolve independently without breaking others. High coupling, conversely, makes maintenance and evolution difficult. Every architectural decision involves evaluating trade-axes, such as the compromise between cost and performance.

Modern approaches like microservices allow for the construction of flexible and scalable systems. Additionally, FinOps is an increasingly important methodology focusing on the optimization of the use and cost of cloud resources, ensuring that the infrastructure remains efficient. Security must be integrated from the beginning of the design process and is never an optional or final-step consideration.

Data Warehousing Case Study and Implementation

A practical application of data warehousing involves an e-commerce scenario designed to analyze sales. The source data comes from an OLTP (Online Transactional Processing) system recording individual sales (product, user, date, quantity, and subtotal), a product catalog (name, category), and a user system (ID, name, city). The goal is to support high-level analyses such as total sales by category and year, monthly/yearly comparisons, and sales by city and product.

In designing a data cube for this scenario, the primary measure is the total sales ($total\_ventas$). The dimensions for analysis include Products (stratified by category), Time (stratified by months), and Users (stratified by city). The granularity of the data in this instance is set to monthly intervals. The resulting star schema maps dimensions for time, product, and user to a central fact table.

The primary transformation required for the Data Warehouse is the aggregation of data. Instead of storing granular transactional records, only the data that provides historical value for business analysis is kept. This transformation typically occurs during the processing phase of the ETL pipeline to ensure that the warehouse provides satisfactory returns for business queries.

Analytical queries for this system are written in SQL. To find the total sales by category and year, the query structure is:

$SELECT\, p.categoria,\, t.anio,\, SUM(h.total)\, AS\, total\_venta$$FROM\, hechos\_ventas\, h$$JOIN\, dim\_producto\, p\, ON\, h.id\_producto = p.id\_producto$$JOIN\, dim\_tiempo\, t\, ON\, h.id\_tiempo = t.id\_tiempo$$GROUP\, BY\, p.categoria,\, t.anio$

To retrieve total sales per user for the year 2024, the query used is:

$SELECT\, u.nombre,\, SUM(h.total)\, AS\, total\_ventas$$FROM\, hechos\_ventas\, h$$JOIN\, dim\_usuario\, u\, ON\, h.id\_usuario = u.id\_usuario$$JOIN\, dim\_tiempo\, t\, ON\, h.id\_tiempo = t.id\_tiempo$$WHERE\, t.anio = 2024$$GROUP\, BY\, u.nombre$

Data Lake Architectural Framework

A Data Lake architecture is organized into specific layers to manage the flow and quality of data. In an e-commerce context involving tabular sales records, JSON navigation logs (clicks, views, searches), and product catalogs, the layers function as follows:

The Raw Layer stores data in its original, unprocessed format exactly as it arrives from the source. Its purpose is to hold data in its purest form while it awaits cleaning and analysis. The Processed Layer contains data that has undergone initial cleaning and formatting but may still require further refinement. The Curated Layer contains high-quality, fully processed, and clean data ready for consumption by business analysts and tools.

Data placement within these layers corresponds to the processing level of the information. For example, JSON navigation logs without modifications are stored in the Raw layer. Data where types have been corrected and the structure is made consistent (such as sales records with corrected data types) are stored in the Curated layer. Aggregated data by user, such as the total number of events, is typically found in the Processed or Curated layers depending on its finality.

In an ELT process, raw JSON logs like ${\text{"usuario": "u1", "event": "click", "valor": "10", "fecha": "2024-01-01"}}$ must be transformed into a structured table schema ($usuario,\, evento,\, fecha,\, valor$). Key transformations include data type casting (e.g., converting the string "10" to a numeric value and the "fecha" string to a date type) to ensure schema consistency and enable mathematical operations in the destination.

Queries on the Data Lake target aggregated tables such as $eventos\_usuario(usuario,\, total\_eventos,\, total\_valor)$. To find the total number of events across all users:

$SELECT\, SUM(total\_eventos)\, AS\, total\_eventos\, FROM\, eventos\_usuario$

To find the accumulated total value across all users:

$SELECT\, SUM(total\_valor)\, AS\, total\_valor\, FROM\, eventos\_usuario$

Real-Time Processing with Kappa Architecture

The Kappa Architecture is designed to process data in real-time using a single stream processing engine. The core of this architecture is the event log, which stores a sequence of historical data. This log is crucial because it prioritizes continuous analysis and allows for the reprocessing of data to reconstruct historical results when necessary.

In a Kappa system, data is processed via Streaming. This offers a significant advantage over the Lambda architecture by simplifying the technical stack; instead of maintaining separate batch and speed layers, Kappa relies on the streaming layer for both real-time and historical processing. This facilitates historical analysis without the complexity of merging two different processing paradigms.

Data in the Kappa architecture is characterized by being centric to event processing and focused on historical logs. To build a materialized view such as $eventos\_usuario$, the system determines the basic aspects of the values most important to the user and accumulates them as events arrive. For example, a materialized view named $eventos\_minuto(minuto,\, total\_eventos,\, total\_valor)$ tracks counts and sums within one-minute windows.

To query this architecture for the total events registered per minute, one would use:

$SELECT\, minuto,\, total\_eventos\, FROM\, eventos\_minuto$

To identify the minute with the highest accumulated value, the following query is applied:

$SELECT\, minuto,\, total\_valor$$FROM\, eventos\_minuto$$ORDER\, BY\, total\_valor\, DESC$$LIMIT\, 1$