File Types

What type of storage format is Apache Parquet, and how does it organize data on disk?

A columnar storage format — it stores all values for a given column together (rather than row-by-row), which enables highly efficient compression and lets query engines read only the columns they need.

You're storing a large dataset in S3 that will primarily be queried with Amazon Athena or Redshift Spectrum, often pulling just a few columns out of many. Which file format minimizes both storage cost and query scan time for this use case?

Parquet (or ORC) — columnar formats let the query engine skip unneeded columns entirely.

What type of storage format is Apache Avro, and how does it organize data on disk?

A row-based storage format — each record's fields are stored together sequentially, and the schema is embedded in the file itself (written in JSON) alongside the binary data.

Avro is frequently the preferred format in which two scenarios: streaming ingestion pipelines (e.g., Kafka) and situations where the schema needs to change over time without breaking older consumers?

Because Avro stores its schema with the data and has strong built-in support for schema evolution (adding/removing/renaming fields safely), making it ideal for write-heavy streaming systems and evolving data contracts.

What is "schema evolution," and which common big-data file format is specifically known for handling it well?

The ability to change a dataset's schema (add, remove, or modify fields) over time while remaining compatible with data written under older schema versions — Avro is the format most known for handling this gracefully.

What type of storage format is Apache ORC (Optimized Row Columnar), and which big-data ecosystem is it most associated with?

A columnar storage format similar in purpose to Parquet, but originally developed for and most tightly integrated with the Apache Hive ecosystem; it includes built-in lightweight indexes and heavy compression.

If Parquet and ORC are both columnar formats, what is the main practical reason a team might choose one over the other?

Ecosystem fit — ORC is more tightly optimized for Hive-based workloads, while Parquet has broader, more universal support across Spark, Athena, Redshift Spectrum, and most modern AWS analytics/ML tooling, making Parquet the more common default choice.

Between a row-based format and a columnar format, which is generally better suited for write-heavy, append-only ingestion (like streaming events), and which is better for analytical read queries over large datasets?

Row-based formats (like Avro) are better for write-heavy ingestion since a full record is written contiguously; columnar formats (like Parquet/ORC) are better for analytical reads since queries can skip irrelevant columns.

What is a key limitation of plain CSV files as a data format for large-scale ML data storage and processing?

CSV has no embedded schema or data types (everything is text), no native compression-friendly columnar layout, and no support for nested/complex data structures — making it inefficient and error-prone at scale compared to Parquet, Avro, or ORC.

You're prototyping a small ML pipeline and need a format that's human-readable, easy to debug, and directly loadable into pandas without extra tooling. Which format is the pragmatic (if not most efficient) choice?

CSV.

What is JSON Lines (JSONL), and how does it differ from a single standard JSON file for data processing purposes?

A format where each line of the file is a separate, independent, valid JSON object — unlike a single JSON file (which is often one large array or object), JSON Lines can be read, written, and processed one record at a time, making it splittable and streaming-friendly.

You need to store semi-structured or nested records (e.g., variable-length arrays of attributes per item) in a way that's still splittable across distributed processing jobs. Which format fits better than a single monolithic JSON file?

JSON Lines (JSONL).

What is RecordIO-protobuf, and in what AWS ML context does it specifically appear?

A binary record format (using Protocol Buffers) that many SageMaker built-in algorithms accept as an efficient, high-throughput training input format, particularly for large numeric datasets.

You're training a SageMaker built-in algorithm (e.g., Linear Learner or Factorization Machines) on a very large numeric dataset and want the most efficient built-in-supported input format to minimize I/O overhead during training. Which format should you consider besides CSV?

RecordIO-protobuf.

Which two SageMaker training input modes let you stream data directly from S3 without downloading the entire dataset to the training instance's local disk first, and which format do they pair well with?

Pipe mode and FastFile mode — both are well suited to streaming large record-oriented files like RecordIO-protobuf (or Parquet/CSV) without requiring a full local copy first.

Between Parquet and Avro, which format would compress better for a dataset with many repeated categorical values in a given column, and why?

Parquet — because columnar formats group identical/similar values together within a column, compression algorithms exploit that repetition far more effectively than in a row-based layout like Avro.

A data engineering team is deciding between Avro and Parquet for a pipeline where data arrives continuously from Kafka and is later loaded into a data lake for analytics. What's the typical pattern for using both formats together?

Ingest and store the raw streaming data in Avro (row-based, schema-evolution friendly, good for continuous writes), then convert/compact it into Parquet for downstream analytical querying (columnar, better read performance).

Why do compression and "splittability" matter together when choosing a file format for datasets processed by distributed engines like Spark or EMR?

A file format needs to support splitting compressed data into independently readable chunks so that distributed workers can process different parts of a large file in parallel; formats like Parquet and ORC support this, while some compression codecs on plain text files (like standard gzip on CSV) do not.

What does it mean for a file format to have an embedded or "self-describing" schema, and which formats among Avro, Parquet, and CSV have this property?

It means the file itself stores metadata describing its structure and data types, so a reader doesn't need external documentation to interpret it correctly — Avro and Parquet both embed schema information; plain CSV does not.