Week 11: Fast Data Streaming in the Cloud

Introduction to Streaming Systems

  • Introduction

    • Why do we need real-time stream processing systems? What are the fundamental goals of streaming analytics?

    • How do batch and streaming architectures differ, and what are their ideal use cases?

    • What are the typical components of a streaming data pipeline (e.g., producers, brokers, stream processors, sinks)?

    • What challenges arise in stream processing, especially event vs. processing time and exactly-once semantics?

  • Big Data Pipelines

    • What motivated the shift from batch-only systems to real-time architectures in big data pipelines?

    • What are the key elements of Apache Storm's processing model, including topologies, spouts, and bolts?

    • How do hybrid systems like Lambda Architecture combine batch and real-time processing layers to ensure accuracy and responsiveness?

  • Lambda and Kappa Architecture

    • What are the core components of the Lambda Architecture, and how does it manage failures and state consistency?

    • How does the Kappa Architecture differ from Lambda in terms of simplicity and reliance on a single streaming path?

Apache Flink and AWS Kinesis

  • Apache Flink

    • What is the role of Amazon Data Firehose in building serverless ETL pipelines for real-time data delivery?

    • How does Firehose handle buffering, transformations, and delivery to destinations like S3, Redshift, or OpenSearch?

    • What are the benefits and use cases of inline data transformations using AWS Lambda in Firehose?

    • What are the key features of Amazon Managed Service for Apache Flink, and how does it support real-time analytics?

    • How can Flink be configured for streaming jobs using SQL or APIs, and what are the best practices for deployment and monitoring?

  • Exactly Once Semantics

    • What problems did earlier streaming frameworks face with duplicate processing and partial updates?

    • How does Flink use checkpoint barriers and operator state to achieve exactly-once processing?

    • What is the purpose of two-phase commit for external sinks in Flink, and how does it ensure consistency?

    • How does Flink unify real-time correctness with recovery strategies, eliminating the need for Lambda/Kappa reconciliation?

  • Amazon Data Firehose and Managed Flink

    • What are the core features of Apache Flink’s architecture?

    • How do the DataStream, Table, and SQL APIs support a wide range of streaming use cases in Flink?

Spark Streaming

  • Spark Streaming

    • What is Spark Streaming’s discretized stream processing model, and how does it differ from traditional record-at-a-time systems?

    • How does Spark Streaming use micro-batches (DStreams) to achieve fault-tolerant, near-real-time processing?

    • What is the role of RDDs in Spark Streaming’s computation model?

    • What types of data sources can Spark Streaming integrate with out-of-the-box (e.g., Kafka, HDFS, TCP sockets)?

    • What advantages and limitations does Spark Streaming have compared to real-time engines like Flink or Storm?

  • Streaming Ecosystem, Druid

    • What are the key stages of a streaming data pipeline (ingestion to distributed queues) and real-time processing?

    • How does Apache NiFi support visual flow design, data transfer, and site-to-site movement across clusters?

    • How does Kafka serve as a distributed, durable, and high-throughput pub-sub system for decoupling producers and consumers?

    • How do systems like Storm and Spark handle processing, and how does the choice impact latency, throughput, and state management?

    • What role does Druid play in enabling fast OLAP queries, and how is it optimized using columnar storage, bitmap indexes, and real-time ingestion?