Ch. 3 Streaming Data Collection

What are the different ways you can upload data into S3?

console, S3 API, AWS cli

Kinesis Streams

collects data from data produces & stores the data as shards

Shards

container for streaming data

Shard composition train analogy

train id = partition key, train car = sequence number, passengers = data blob

Retention period of shards?

24 hrs to 7 days

Where are Kinesis Streams shards sent after collection?

sent to data consumers

Data consumers?

data processing tools like EC2, Lambda, Kinesis Data Analytics, EMR w/ Apache Spark

Kinesis Producer Library (KPL)

library that allows you to write to Kinesis Stream

Kinesis Client Library (KCL)

library integrated with KPL for consumer apps to consume & process data from Kinesis Stream

Kinesis APIs

the same function as KPL & KCL, but used for lower level API orientations & manual configuration

What is the difference in processing speed of KPL vs Kinesis API?

KPL is slower compared to Kinesis API because it does things for you behind the scenes

What is the language difference of KPL vs Kinesis API?

KPL is Java only & Kinesis API is whatever languages are supported

What is the difference in automation of KPL vs Kinesis API?

KPL has some automated API calls while Kinesis API is all manual

When should you use Kinesis API?

when you need your data stream in milliseconds

When should you use Kinesis Data Streams consumer wise?

when the data needs to be directly processed by the consumers, instead of to a data store

When should you use Kinesis Data Streams storage wise?

when storing data is optional or data retention is important

Kinesis Data Streams Real World Uses

process & evaluate logs immediately, real time data analytics

Kinesis Firehose

easily stream data to final data store

When should you use Kinesis Firehose collection wise?

when you want to collect streaming data & send to final destination like S3

When should you use Kinesis Firehose processing wise?

when processing data is optional or data retention is not important

Kinesis Firehose Real World Uses

stream & store data from devices, create ETL jobs on streaming data

Kinesis Video Stream

stream videos into AWS

What can you with Kinesis Video Stream?

build real time video procession apps, stream images & audio real time

When should you use Kinesis Video Stream processing wise?

when processing real time streaming video, audio, image, radar

When should you use Kinesis Video Stream batch wise?

batch process & store streaming video, feed streaming data into other AWS services

Kinesis Data Analytics

continuously read & process streaming data in real time via SQL queries to process incoming data & produce output data

When should you use Kinesis Data Analytics SQL wise?

run SQL queries on streaming data & output results into S3

When should you use Kinesis Data Analytics creation-wise?

create metrics, dashboards, monitoring, notifications, alarms, or apps that provide data insight

Kinesis Data Analytics Real World Uses

real time analytics, stream ETL jobs