data engineering

ephemeral volume

a storage volume that only exists for the lifetime of the process, container, or instance that created it.

• EKS’s most efficient approach for achieving the lowest latency

aws glue

designed to handle complex tasks that often take over 15 min

aws batch

cannot directly consume data from a streaming source

AWS Glue DataBrew

A visual preparation tool that enables users to clean and normalize any data without writing any code; can be configured to identify and mask PII.

AWS Glue Data Catalog

Primarily serves as a persistent metadata store; resource policies allow you to manage access at a granular level.

AWS Glue Interactive Sessions

Allows you to interactively test, develop, and debug ETL scripts before deploying them as AWS Glue jobs.

AWS Glue Studio

A visual interface for AWS Glue that lets you create ETL jobs by dragging and dropping sources, transforms, and targets, which automatically generates the script.

Enhanced Fan-Out

A feature in Kinesis Data Streams that provides each consumer with its own 2MB per second of dedicated bandwidth per shard, which can help reduce latency. and helps handle increased volume

HTTP/2 data retrieval API

Can help reduce latency, making data delivery from producers to consumers faster.

Partition Projection in Athena

Calculates partition values on the fly using rules you define, dramatically speeding up query planning for highly partitioned tables.

Amazon Redshift Spectrum

Lets you run SQL queries directly against data stored in Amazon S3, without copying or loading that data into Redshift first.

offline

Only supports batch feature access.

online

Provides real-time access to features.

Amazon SageMaker Lineage Tracking

Offers a way to create and store information about each step in an ML workflow, important for establishing model governance and audit standards which are essential for ensuring the accuracy, completeness, and trustworthiness of the data used in ML decisions

.

Cannot extract from Redshift directly to Glacier

Redshift data sharing

Provides read-only access to data across multiple Amazon Redshift clusters, allowing you to read live data without manually moving or copying data.

AWS Glue Data Quality

Measures the quality of your data and uses a rules-based approach to validate data before or after transformations.

Purge operation

Used to permanently delete files from Amazon S3.

current_schema()

A leader node-only function in Redshift.

Materialized views

In Redshift, provides a way to handle complex queries on large datasets.

Amazon Athena User Defined Functions

Allows customers to write custom scalar functions and invoke them in SQL queries.

LOCK command

Restricts access to the database table.