Flashcard 1

Q: What are the primary source types for data acquisition in big data systems? 

A: Publishing big data in batches, microbatches, or streaming real-time data.

Flashcard 2

Q: How is velocity defined in the context of data acquisition? 

A: The speed at which data is generated and how frequently it is produced.

Flashcard 3

Q: What characterizes high velocity data? 

A: Real-time or streaming data generated and processed continuously.

Flashcard 4

Q: What are the two main ingestion mechanisms for data? 

A: Push and pull mechanisms, driven by the data consumer.

Flashcard 5

Q: What is Kafka? 

A: A high-throughput distributed messaging system used for building real-time data pipelines and streaming applications.

Flashcard 6

Q: In Kafka, what is a broker? 

A: A server that manages topics, handles persistence, partitions, and replicates data.

Flashcard 7

Q: What is a topic in Kafka? 

A: A stream of messages of a particular type, similar to tables in databases.

Flashcard 8

Q: How does Kafka store messages? 

A: On disk using partitioned commit logs.

Flashcard 9

Q: What roles do producers and consumers play in Kafka? 

A: Producers publish messages to topics, while consumers subscribe to topics and process messages.

Flashcard 10

Q: What is a partition in Kafka? 

A: A division of a Kafka topic that allows messages to be consumed in parallel, maintaining an ordered and immutable sequence of messages.

Flashcard 11

Q: How does Kafka achieve parallel consumption of messages? 

A: By dividing topics into multiple partitions and allowing multiple consumers to read from different partitions simultaneously.

Flashcard 12

Q: What is the role of a server leader in Kafka? 

A: It handles read and write operations for a partition.

Flashcard 13

Q: What are replicas in Kafka? 

A: Followers that replicate the data of the leader to ensure fault tolerance.

Flashcard 14

Q: Describe Kafka's publish-subscribe messaging framework. 

A: Producers publish messages to topics, and consumers subscribe to those topics to receive messages.

Flashcard 15

Q: What is an offset in Kafka? 

A: A unique sequence ID assigned to each message within a partition.

Flashcard 16

Q: What is a consumer group in Kafka? 

A: A group of consumers that work together to consume messages from one or more topics, ensuring each message is processed by only one consumer in the group.

Flashcard 17

Q: How does Kafka handle log storage? 

A: Messages are stored in append-only, ordered, and immutable logs.

Flashcard 18

Q: What is log compaction in Kafka? 

A: A process to clean out obsolete records by retaining only the latest message for each key within a log segment.

Flashcard 19

Q: What are log segments in Kafka? 

A: Portions of a topic partition's log stored as directories of segment files.

Flashcard 20

Q: What determines when log segments are deleted in Kafka? 

A: Reaching size or time limits as defined by the delete policy.

Flashcard 21

Q: What is Amazon Kinesis? 

A: A managed service for ingesting, processing, and analyzing real-time streaming data on AWS.

Flashcard 22

Q: What are Kinesis Data Streams? 

A: Services that allow ingestion and processing of streaming data in real-time.

Flashcard 23

Q: What are Firehose Delivery Streams in Kinesis? 

A: Services that collect, transform, and load ETL streaming data into destinations like S3, Redshift, and Splunk.

Flashcard 24

Q: What does Kinesis Analytics do? 

A: Runs continuous SQL queries on streaming data from Kinesis Data Streams and Firehose Delivery Streams.

Flashcard 25

Q: What are Kinesis Video Streams used for? 

A: Streaming live video from devices to the AWS cloud for real-time video processing and batch-oriented analytics.

Flashcard 26

Q: What is AWS IoT? 

A: A service for collecting and managing data from Internet of Things (IoT) devices.

Flashcard 27

Q: What is the Device Gateway in AWS IoT? 

A: Enables IoT devices to securely communicate with AWS IoT.

Flashcard 28

Q: What is the Device Registry in AWS IoT? 

A: Maintains resources and information associated with each IoT device.

Flashcard 29

Q: What is a Device Shadow in AWS IoT? 

A: Maintains the state of a device as a JSON document, allowing applications to interact with devices even when they are offline.

Flashcard 30

Q: What does the Rules Engine in AWS IoT do? 

A: Defines rules for processing incoming messages from devices.

Flashcard 31

Q: What is Apache Flume? 

A: A distributed system for collecting, aggregating, and moving large amounts of data from various sources to a centralized data store.

Flashcard 32

Q: What is a checkpoint file in Flume? 

A: Keeps track of the last committed transactions, acting as a snapshot for data reliability.

Flashcard 33

Q: What are the main components of Flume Architecture? 

A: Source, Channel, Sink, and Agent.

Flashcard 34

Q: What is a Source in Flume? 

A: The component that receives or polls data from external sources.

Flashcard 35

Q: What is a Channel in Flume? 

A: Transmits data from the source to the sink.

Flashcard 36

Q: What is a Sink in Flume? 

A: Drains data from the channel to the final data store.

Flashcard 37

Q: What is an Agent in Flume? 

A: A collection of sources, channels, and sinks that moves data from external sources to destinations.

Flashcard 38

Q: What is an Event in Flume? 

A: A unit of data flow, consisting of a payload and optional attributes.

Flashcard 39

Q: Name the types of Flume Channels. 

A: Memory channel, File channel, JDBC channel, and Spillable Memory channel.

Flashcard 40

Q: What is a Memory Channel in Flume?

A: Stores events in memory for fast access.

Flashcard 41

Q: What is a File Channel in Flume? 

A: Stores events in files on the local filesystem for durability.

Flashcard 42

Q: What is a JDBC Channel in Flume?

A: Stores events in an embedded Derby database for durable storage.

Flashcard 43

Q: What is a Spillable Memory Channel in Flume? 

A: Stores events in an in-memory queue and spills to disk when the queue is full.

Flashcard 44

Q: What is Apache Sqoop? 

A: A tool for importing data from relational databases into Hadoop Distributed File System (HDFS), Hive, or HBase, and exporting data back to RDBMS.

Flashcard 45

Q: How does Sqoop import data? 

A: By launching multiple map tasks to transfer data as delimited text files, binary Avro files, or Hadoop sequence files.

Flashcard 46

Q: What are Hadoop Sequence Files? 

A: A binary file format specific to Hadoop for storing sequences of key-value pairs.

Flashcard 47

Q: What is Apache Avro? 

A: A serialization framework that provides rich data structures, a compact binary data format, and container files for data storage and processing.

Flashcard 48

Q: What is RabbitMQ? 

A: A messaging queue that implements the Advanced Message Queuing Protocol (AMQP) for exchanging messages between systems.

Flashcard 49

Q: What is the Advanced Message Queuing Protocol (AMQP)? 

A: A protocol that defines the exchange of messages between systems, specifying roles like producers, consumers, and brokers.

Flashcard 50

Q: In RabbitMQ, what are producers and consumers? 

A: Producers publish messages to exchanges, and consumers receive messages from queues based on bindings and routing rules.

Flashcard 51

Q: What is ZeroMQ? 

A: A high-performance messaging library that provides tools to build custom messaging systems without requiring a message broker.

Flashcard 52

Q: What messaging patterns does ZeroMQ support? 

A: Request-Reply, Publish-Subscribe, Push-Pull, and Exclusive Pair.

Flashcard 53

Q: What is RestMQ? 

A: A message queue based on a simple JSON-based protocol using HTTP as the transport, organized as REST resources.

Flashcard 54

Q: How do producers interact with RestMQ? 

A: By making HTTP POST requests with data payloads to publish messages to queues.

Flashcard 55

Q: What is Amazon SQS? 

A: A scalable and reliable hosted queue service that stores messages for distributed applications.

Flashcard 56

Q: What are the two types of queues in Amazon SQS? 

A: Standard queues and FIFO (First-In-First-Out) queues.

Flashcard 57

Q: What are the characteristics of Standard Queues in Amazon SQS? 

A:

• Guarantees message delivery but not order.

• Supports unlimited transactions per second.

• Operates on an at-least-once delivery model, occasionally delivering duplicate messages.

Flashcard 58

Q: What are the characteristics of FIFO Queues in Amazon SQS? 

A:

• Ensures messages are received in the exact order they were sent.

• Supports up to 3,000 messages per second with batching or 300 messages per second without batching.

• Provides exactly-once processing.

Flashcard 59

Q: What are Connectors in the context of messaging systems? 

A: Interfaces that allow data to be published to and consumed from messaging queues, often exposing REST web services or other protocols.

Flashcard 60

Q: How does a REST-based Connector work? 

A: Producers publish data using HTTP POST requests with data payloads, and the connector processes the requests and stores the data to the sink.

Flashcard 61

Q: What is a WebSocket-based Connector? 

A: A connector that uses full-duplex communication, allowing continuous data exchange without setting up new connections for each message.

Flashcard 62

Q: What is an MQTT-based Connector? 

A: A lightweight, publish-subscribe messaging protocol designed for constrained devices, suitable for IoT applications.

Flashcard 63

Q: In MQTT-based systems, what are the main entities? 

A: Publisher, Broker/Server, and Subscriber.

Flashcard 64

Q: What is the role of a Publisher in MQTT? 

A: Publishes data to topics managed by the broker.

Flashcard 65

Q: What does the Broker/Server do in MQTT? 

A: Manages topics and forwards published data to subscribed subscribers.

Flashcard 66

Q: What is the role of a Subscriber in MQTT? 

A: Receives data from topics to which it has subscribed.