Storage Systems, File Systems, and Binary File Operations

Homework Assignments and Grading Impact:
- Homework assignments are regularly scheduled with deadlines on Fridays.
- Assignments must be taken seriously, as future examination questions directly cover concepts introduced in homework tasks.
- Key objectives of homework assignments:
- Reinforce theoretical and operational concepts presented during lectures, podcasts, and classroom discussions.
- Extend student knowledge beyond basic lecture material.
- Provide hands-on experience with low-level data structures and file operations.
- Serve as primary material for exam review.
Course Schedule Sequence:
- Storage Systems: Primary subject of study beginning this week.
- File Systems: Topic immediately following storage systems.
- Homework 2:
- Combines concepts from both Storage Systems and File Systems due to their inherent structural relationship.
- Scheduled for official release by this Friday.
- Bridges low-level binary operations with relational database file storage mechanics.

Binary File Operations in Applications:
- Databases read and write data in binary mode ( $\text{byte-by-byte}$ ) rather than plain text mode.
- Binary mode eliminates text formatting overhead and enables direct manipulation of underlying physical data bytes.
MySQL InnoDB Tablespace Architecture:
- MySQL utilizes the InnoDB storage engine to write data to disk using tablespace files.
- Tablespace files carry the .ibd file extension (e.g., users.ibd associated with a users table or person table).
- The .ibd file stores two distinct categories of information:
- Data: Actual records and attribute values contained within the database tables.
- Metadata: Structural definitions, headers, schema rules, and index pointers.
- MySQL interacts with .ibd tablespace files purely in binary mode for both reading and writing.
- Homework 2 establishes hands-on binary file handling techniques to prepare for studying the physical data model in MySQL without requiring prior relational database experience.

Core Storage Question:
- Performance in storage systems is evaluated by analyzing how rapidly a given workload (read or write operations) can be completed.
Logical vs. Physical Block Architecture:
- Logical Block:
- An abstraction provided by the file system layer.
- When creating or extending a file, the file system allocates logical blocks to hold data.
- Physical Block:
- The physical unit of storage managed directly by the underlying storage system hardware.
- A single logical block defined by a file system may map to multiple physical blocks on hardware.
Sizing Assumptions and Data Transfer Rules:
- Physical Block Size: Standardized at $4\,\text{kB}$ ( $4\text{ kilobytes}$ ).
- Logical Block Size: Can be configured to $4\,\text{kB}$ , $8\,\text{kB}$ , or other arbitrary sizes depending on system architecture.
- For homework assignments and analysis, physical block size and logical block size are assumed to be identical ( $4\,\text{kB}$ ).
- Fundamental Unit of Transfer:
- The physical block ( $4\,\text{kB}$ ) is the absolute unit of data transfer in storage systems.
- Storage hardware cannot read or write individual bytes or isolated partial data blocks; any read or write operation transfers an entire $4\,\text{kB}$ physical block.

Hard Disk Drives (HDD):
- Though less commonly deployed today compared to modern flash memory, HDDs provide an intuitive physical model to contrast sequential and random operations.
- Data is laid out radially on concentric physical tracks (analogous to lanes on a running track).
Solid State Drives (SSD):
- Modern non-volatile storage standard present in most client and enterprise hardware.
- Stores data in hardware units termed pages (and blocks of pages).

Workload Definitions:
- Sequential Workload:
- Involves accessing physical blocks that are contiguous (located directly adjacent to one another on disk or flash memory).
- Example: Reading physical block $1$ , block $2$ , block $3$ , block $4$ , and block $5$ consecutively on the same row or track.
- Random Workload:
- Involves accessing physical blocks that are non-contiguous and distributed arbitrarily across different regions of the media.
- Example: Reading $5$ total blocks where block $1$ is located at the start of the storage media, block $2$ is located at the end, block $3$ is located in the middle, and subsequent blocks are scattered elsewhere.
Performance Dynamics:
- Sequential workloads complete significantly faster than random workloads.
- Sequential access minimizes physical seek overhead on HDDs and optimizes controller caching and internal page access on SSDs.

Python File Open Modes:
- Mode 'w': Opens a file for writing. Overwrites existing contents completely if the file exists, or creates a new file if it does not.
- Mode 'b': Opens a file in binary mode ('wb' for binary writing, 'rb' for binary reading).
Stream Positioning and Seeking:
- When a file object is initially opened, its internal pointer points directly to position $0$ (the start of the file).
- Iterating over lines or reading standard streams processes data sequentially starting from the beginning.
- f.seek(offset) Method:
- Sets and modifies the active stream position indicator of the file object.
- Required when executing random workloads over binary files.
- Allows the application to reposition the stream pointer directly to an arbitrary byte/block location (e.g., seeking to the second block) prior to executing a read or write operation.

Room Conditions:
- Classroom temperature and air conditioning functionality evaluated in the back of the room.
- Classroom access doors left open to improve ventilation due to the absence of openable windows.