1/73
Looks like no tags are added yet.
system integration
the practice of connecting independent applications, data stores and services so that they behave as one coordinated whole
fields, types, required values, encoding, customer records, events, files
what moves in an integrated system?
HTTPS/REST, SOAP, gRPC, message queue, SFTP file drop, direct database links
how does data move in an integrated system?
real time requests or responses, event-driven publishes, or scheduled batch actions
when does data in an integrated system move?
data level, application/API level, business process level, presentation/UI level
four levels of integration
data level
this level of integration involves moving and reconciling records between different systems; records are copied between systems so all of them has what it needs locally
ETL, replication, change data capture, file transfer
examples of data level integration?
application/API level integration
this level of integration involves one system calling another system’s functions; instead of copying files, one system asks another for it, or asks it to do something
REST, SOAP, gRPC, GraphQL, SDKs, webhooks
examples of application/API level integration?
business process level integration
this level of integration involves orchestrating a workflow that spans various different systems
ESB orchestration, BPM engines, workflow/saga patterns
examples of business process level integration?
presentation/UI level integration
this level of integration involves unifying only at the glass, not underneath; the systems stay separate and only the user’s screen is unified
portals, dashboards, embedded widgets, mashups
examples of presentation/UI level integration?
coupling
how much a change in one system forces a change in another
payload
actual data carried in one message or call
idempotency
sending the same request twice has the same effect as sending it once
contract
the published interface that has been agreed upon, involving endpoints, fields, types, errors, etc.
synchronous vs asynchronous
sync: the caller waits for the answer
async: the caller hands off the message and continues
latency vs throughput
how long one message takes to send versus how many messages per second the pipe sustains
ERP, CRM, HRIS
three most common organizational use cases?
ERP
this organizational use case owns money and materials
enterprise resource planning
meaning of ERP?
general ledger, inventory, purchase orders, invoices, production
what kind of data is stored in an ERP?
CRM
this organizational use case owns the customer
customer relationship management
meaning of CRM?
leads, opportunities, contacts, quotes, support cases, campaigns
what kind of data is stored in a CRM?
HRIS
this organizational use case owns the employee
human resource information system
meaning of HRIS?
personal records, positions, org chart, leave, payroll inputs
what kind of data is stored in an HRIS?
e-commerce, warehouse, business intelligence, learning, ticketing
other kinds of organizational use cases?
ERP
a single suite of modules sharing one database, covering the operational and financial core of the business
CRM
the system of record for everyone outside of the organization such as prospects and customers
identity verification, duplicate records, consent and data privacy, two-way sync
problems with CRMs?
item/price/stock lists, invoice and payment status, financial extracts to data warehouses, purchase orders to suppliers
what are expected outputs from an ERP?
HRIS
system of record for everyone inside the organization
identity/active directory, ERP finance, access control, project/billing systems, LMS
what are some systems that depend on the HRIS?
process
this type of data transaction is sequential, completing one action triggers another and so on
fan-out
this type of data transaction is non-sequential, one main event occurs which triggers other events that occur simultaneously and independently of each other
point-to-point, middleware, service-oriented
three integration architectures?
point-to-point
this type of integration architecture is direct, lightly-coupled hardcoded connections between pairs of specific applications; better for simple localized scenarios that require immediate data transfer
middleware
this type of integration architecture is a centralized hub infrastructure routing, translating and managing all system messages
service-oriented
this type of integration architecture is a decoupled, reusable microservices that communicate over standard network protocol; provides high agility when combining existing services and is cloud-ready
rapidly creates an unmanageable spaghetti code
complexity growth of point-to-point integration architecture?
siloed → point to point → middleware → SOA/micro
architectural evolution?
data mapping
a specification linking source fields to target fields, written before implementation
prevents data loss, maintains integrity, foundation for transformation, shared documentation
4 main functions of data mapping?
csv, json, xml
standard exchange formats for data mapping?
comma separated values
what does csv mean?
csv
strengths include: compact, human-readable, bulk operations
limitations include: flat only, no data types
extensible markup language
what does xml mean?
xml
key ideas include hierarchical self-describing tags, unique names and validated structures and types
but it is verbose and heavier to process
javascript object notation
what does json mean?
json
supports strings, numbers, boolean and null
lightweight and readable
but has no built in data type or comments
json
standard for RESTful APIs
structural and value
two categories of transformation?
structure
this category of transformation involves changing the format of the data rather than the data itself
value
this category of transformation involves changing the content of the individual fields to a standardized format
extract, transform, load
ETL pipeline?
ETL pipeline
the process of collecting data from different sources and transforming it into a consistent and usable format
extract
this process of the ETL pipeline involves retrieving data from one or more sources
transform
this process of the ETL pipeline involves cleaning, validating, combining and modifying the data
load
this process of the ETL pipeline involves storing the processed data into the destination system
data cleaning
this type of ETL transformation involves removing invalid or incorrect records
data validation
this type of ETL transformation involves checking whether values are acceptable
standardization
this type of ETL transformation involves making formats consistent
data mapping
this type of ETL transformation involves matching fields from different systems
data integration
this type of ETL transformation involves combining different data sets
aggregation
this type of ETL transformation involves calculating summaries
filtering
this type of ETL transformation involves removing unnecessary records
data warehouses, relational databases, cloud databases, data lakes, analytical platforms
common destination systems for ETL loading
batch processing, incremental processing, real-time processing
3 types of ETL processing?
batch processing
this type of ETL processing is when data is collected and processed in groups at scheduled intervals
incremental processing
this type of ETL processing is when the pipeline only retrieves new or modified records since the previous run
real time processing
this type of ETL processing is when data is processed continuously or with very low latency as events arrive