SIA Module 3

Structured Data Management

A process of acquiring data from numerous sources and merging it to construct a single "unified" view.

Structured Data Management

Modern data teams manage dispersed systems where formats are often incompatible. Integration remedies "Data Silos" to discover hidden business values.

Data Silos

Integration remedies ______ to discover hidden business values.

Data Migration

One-way data transfer in a discrete session. Neutrally handles enormous volumes of records in parallel.

Broadcast

Transactional, near real-time transmission from one source to many destinations. Moves only modified data.

Bi-Directional Sync

Combines two datasets from separate systems to function as one while existing independently. Maintains consistency across platforms.

Correlation

A specialized sync performed only on the intersection of datasets. Filters out irrelevant data to ensure efficiency.

Aggregation

Merging data from multiple systems into a single target system. It provides a consolidated view without duplicating storage.

Aggregation

Ideal for ETL (Extract, Transform, Load) solutions and Business Intelligence (BI) analysis.

Point - Point Integration

Direct Integration: Custom links established between individual systems.

Point - Point Integration

Tight Coupling: Changes in one system necessitate updates in others.

Point - Point Integration

Custom Interfaces: Built from scratch for specific requirements.

Point - Point Integration

Simplicity: Uncomplicated due to reliability on middleware instead of direct connections.

Point - Point Integration

Challenge: When there are an increasing number of integrations that need to be managed, it can become challenging to administer and maintain a large number of direct connections.

Hub-and-Spoke Integration

Centralized Hub: All systems connect to a focal point (Hub) instead of each other.

Hub-and-Spoke Integration

Indirect Communication: They send and receive data to and from the hub which reroutes to the destination system.

Hub-and-Spoke Integration

Decoupling: Systems evolve independently; the Hub handles routing and translation.

Hub-and-Spoke Integration

Scalability: Adding a new system only requires one connection to the Hub.

Hub-and-Spoke Integration

Benefit: The central hub makes it easier to manage and coordinate the exchange of data between systems, which results in improved control, monitoring, and maintenance of integrations.

Hub-and-Spoke Integration

Challenge: Because it relies so heavily on a centralized hub, hub-andspoke integration might result in an increased amount of latency as well as a single point of failure.

Publish-Subscribe Integration

Message Broker:Acts as the primary point of coordination for all communication. It takes in messages and distributes them to subscribers.

Publish-Subscribe Integration

Topic or Channels: Messages are publised to particular channels predetermined by the message broker. Subscribers can select topics.

Publish-Subscribe Integration

Asynchronous Communication: Messages can be published at any moment and subscribers can get them asynchronously.

Publish-Subscribe Integration

Scalability: High-level scabililty can be observed due to the fact that several subscribers can simultaneously consume messages.

Publish-Subscribe Integration

Benefit: It includes event-driven communication in real time, scability, and flexible coupling between components. In its capacity as a scalable and fault-tolerant middleman, the message broker ensures the timely and accurate delivery of messages while simultaneously enabling a high rate of message flow.

Publish-Subscribe Integration

However, it is vital to properly establish the themes and message architecture in order to guarantee that messages are classified and structured in the appropriate manner. It also necessitates taking measures to ensure the availability of the message broker and its resistance to failure.

Request-Response Integration

It enables synchronous communication between different types of systems.

Request-Response Integration

One system will make a request to another system, and the first system will wait for the second system to respond with the required information.

Request-Response Integration

Synchronous Communication: The requesting system stops processing and waits for a response from the target system.

Request-Response Integration

Direct Point-to-Point Communication: During the exchange, it will set up a session that will keep a temporary link open.

Request-Response Integration

Blocking Operation: The system that sent the request either stops processing or waits for the answer until it comes or the timeout period ends, whichever comes first.

Request-Response Integration

Response Data: The response includes the requested data or the result of the action.

Request-Response Integration

Usage: Mostly used in API interactions, web services, and client-server applications, where instant feedback and responses are needed.

Request-Response Integration

Challenge: The fact that this patterns is synchronous could cause performance problems, especially of the target system has high amount of latency or a long processing time.

Request-Response Integration

In order to keep the exchange reliable and on time, timeout settings and error handling should be given careful thought.

File-Based Integration

The transfer of data through the use of files is an integral part of this file-based integration architecture, which is designed for the integration of computer systems.

File Exchange

Process of exchanging information between computer systems consists of both writing and reading files stored in shared places like directories or network drives.

File-Based Integration

Asynchronous Communication: Systems do not interact with one another directly, instead they rely on the presence of files in the shared location.

File-Based Integration

File Formats System must conform to specified files formats and organizational patterns in order to keep their capacity to communicate with one another intact.

File-Based Integration

Data Transformation: The data on both systems are modified to comply to the format that is required for the files. Methods such as data mapping, data validation, and data enrichment.

File-Based Integration

Usage: Utilized in situations in which it is not essential for two systems to speak direcly in real time and transfer massive amount of data. Also widely used in situations that require batch processing with older systems.

File-Based Integration

Challenge: Most notable drawbacks are an additional complexity in administration of file exchange like data consistency and the demand for error-handling processes.

Real-Time Integration

It is a paradigm of system integration that lays an emphasis on the instanteneous movement of data and communication between various computerized settings.

Real-Time Integration

Immediate Data Exchange Integration in real-time ensures that the data is transferred and modified nearly real-time

Real-Time Integration

Event-Driven Communication The process of real-time data exchanges is often set off by events or by other triggers.

Real-Time Integration

Continuous Data Flow The stream of information is both continuous and uninterrupted.

Real-Time Integration

High Throughput and Low Latency Designed to meet the needs for a high data flow and a low latency. Great focus on efficient and speedy data transfer.

Real-Time Integration

In situations when instantaneous data exchange and synchronization are required,_______ is an absolute must. It helps systems run with the most recent and correct data, which enables speedier decisionmaking and delivers better experience for users.

Real-Time Integration

Give careful consideration on scalability, event processing, and data consistency so that real-time integration can be implemented successfully. It is necessary for the integration platform to be capable of handling high message flow, of enabling event-driven communication, and of ensuring data integrity and consistency across systems.

Hybrid Integration Approaches

Approaches that combine traditional and modern methods of integration are the most effective way to meet the challenging integration requirements of today’s businesses.

Hybrid Integration Approaches

Using a variety of patterns in order to construct an integration solution that is comprehensive while retaining its flexibility.

P2P and Message Queue (MQ) Integration

__ is used for immediate request-and-response while __ is used for asynchronous communication.

Hub-and-Spoke and Pub/Sub Integration

Enables the systems to communicate either directly or indirectly, depending on the requirements for integration.

File-Based and Real-Time Integration

____ connectivity enables bulk data interchange, while _____ enables instant data synchronization. Example of this is a data warehouse.

SOA and Microservices Integration

This method was developed by Microsoft, which makes use of the modularity and autonomous deployment capabilities of miscroservices and service composition and orchestration skills of SOA for larger-scale integration situations.

Data Migration

Broadcast

Bi-Directional Sync

Correlation

Aggregation

Five Primary Patterns of Data Integration

Point-to-Point Integration

Hub and Spoke

Published and Subscribed

Request Response

File Based

Real Time

Hybrid

System Integration Patterrns