POSIX File Systems, File Descriptors, and Redirection

Assignment Policies and Course Logistics

  • Late Submission Policy:

    • Submissions made after the official deadline receive an immediate penalty of 10%10\%. Exactly 1010 points/percent are subtracted from the score.

    • Penalty severity is independent of how late the submission is past the deadline: being 22 seconds late results in the exact same 10%10\% penalty as being 77 hours late.

    • Canvas timestamp serves as the sole arbiter for submission time.

    • The penalty resets every 2424\text{ hours} at midnight (12 ⁣: ⁣00 AM12\!:\!00\text{ AM}). For example, if an assignment is due at 11 ⁣: ⁣59 PM11\!:\!59\text{ PM} on October 8:

    • Submitting at 12 ⁣: ⁣00 AM12\!:\!00\text{ AM} on October 8/9 incurs a 10%10\% penalty (1 day late).

    • Submitting 2424\text{ hours} and 11\text{ minute} late incurs a 20%20\% penalty (2 days late).

    • Operational Strategy for Late Work: If working up to the last minute and the 11 ⁣: ⁣59 PM11\!:\!59\text{ PM} deadline passes, go to bed and complete the work the following day. Once the deadline passes, the 10%10\% penalty has already applied, so there is no benefit to staying up until 5 ⁣: ⁣00 AM5\!:\!00\text{ AM}.

    • Early Submission Recommendation: Plan to submit assignments 1 to 2 days early to insulate against technical issues, Canvas system hiccups, or web browser crashes right before the deadline.

    • Academic / Travel Exceptions: For legitimate academic commitments or out-of-town travel, contact the instructor directly to arrange an adjusted personal due date.

  • Partner Project Submission Requirements:

    • Only one partner should submit the final assignment to Canvas to avoid unnecessary system complications.

    • Partners must explicitly communicate and confirm who will handle the submission.

    • Both partner members must be listed explicitly by full name and NetID within the review file.

    • An author's file must be submitted containing only the partners' NetIDs and absolutely nothing else.

  • Testing and Memcheck Guidelines:

    • Memcheck serves as a memory stress test intended to run on correct programs; it does not guarantee correctness under invalid inputs or broken processes.

    • Additional dedicated testing files must be created to verify library behavior against malformed or erroneous inputs.

    • Code should be checked for minimal-effort implementations (e.g., hardcoded values or arbitrary size buffers like an 8-byte array).

    • Functions such as gettimeofday must be called twice within timing tests rather than continuously loaded inside loops.

Overview of Streams and Default File Descriptors

  • Definition of Streams:

    • A stream represents an ordered sequence of bytes subjected to input/output operations.

    • Core stream operations consist of reading bytes from the stream or writing bytes into the stream.

    • Stream capabilities vary:

    • Read-only streams.

    • Write-only streams.

    • Read/Write streams.

    • Additional operations include setting the file position offset (seeking) or utilizing multi-channel/multi-stream architectures.

  • Default Standard POSIX Streams:

    • Every newly launched process is automatically provided with three standard streams by default:

    • Stream 00 (stdin): Standard Input. A read-only stream used to ingest data into the process.

    • Stream 11 (stdout): Standard Output. A write-only stream used for normal program output.

    • Stream 22 (stderr): Standard Error. A write-only stream used specifically for diagnostics and error reporting.

File Operations with POSIX open

  • The open System Call Interface:

    • Used to open existing files or create new files, returning an integer file descriptor.

    • Signature conceptually: int open(const char *pathname, int flags, ... /* mode_t mode */);

    • pathname: A string representing an absolute path, relative path, or a standalone file name (which acts as a relative path in the current working directory).

    • flags: An integer formed by bitwise ORing (|) access modes and creation/status flags.

  • Required Access Mode Flags (Must include exactly one):

    • O_RDONLY: Open the file for reading only.

    • O_WRONLY: Open the file for writing only.

    • O_RDWR: Open the file for both reading and writing.

  • Optional Bitwise Flags:

    • O_CREAT: Create the file if it does not already exist. Mandates that the third mode parameter be supplied to open.

    • O_TRUNC: If the file exists and is opened for writing (O_WRONLY or O_RDWR), truncate its length to 00 bytes, discarding existing contents.

    • O_APPEND: Forces all write operations to append data to the end of the file, regardless of current file position.

    • O_EXCL: Used alongside O_CREAT. Ensures exclusive creation; if the target file already exists, open fails.

    • O_CLOEXEC: Sets the close-on-exec flag. Closes the file descriptor automatically when an exec family call executes.

    • O_NONBLOCK: Opens the file in non-blocking mode. Read/write operations return immediately with an error rather than blocking if data is unavailable.

  • Default Write Behavior Without Flags:

    • Opening an existing file with O_WRONLY without specifying O_TRUNC or O_APPEND begins writing at offset 00.

    • Bytes written overwrite existing bytes sequentially. If a file is 100100\text{ bytes} long and 5050\text{ bytes} are written, the first 5050\text{ bytes} are overwritten, while the remaining 5050\text{ bytes} remain unchanged.

    • Standard pattern to completely replace file contents upon writing:

    • open("filename", O_WRONLY | O_CREAT | O_TRUNC, mode)

  • Variadic Nature and Safety Warning for open:

    • C lacks native function overloading or standard optional parameters.

    • open utilizes C variadic parameter lists (... ellipses signature).

    • The underlying implementation inspects the provided flags argument to determine whether to read the 3rd argument (mode).

    • Safety Hazard: If O_CREAT is passed without providing the 3rd mode argument, C cannot validate parameter counts at runtime. open reads whatever garbage data exists in memory where the 3rd argument would reside, leading to unpredictable file permissions.

  • Mode Argument (File Permissions):

    • The mode parameter defines file permissions assigned when O_CREAT creates a new file.

    • Specified using binary/octal representation or POSIX permission macros.

    • Example octal values:

    • 0600: Read and write permissions for the owner; no permissions for group/others.

    • 0644: Read and write permissions for the owner; read-only access for group and others.

File Descriptors, Error Handling, and Lifecycle

  • File Descriptor Return Values:

    • On successful execution, open returns a non-negative integer representing the file descriptor (fd).

    • On failure, open returns −1-1.

    • Always validate file descriptors immediately after opening:

    • If fd < 0 (or fd == -1), handle the failure and log an error.

  • Error Numbers (errno):

    • Defined in <errno.h>.

    • Global variable (thread-local in modern systems) updated by system calls and C standard functions upon failure.

    • Specific error code examples:

    • EACCES: Permission denied (lacking permission to open or create file).

    • EEXIST: File already exists (triggered when using O_CREAT | O_EXCL).

    • Historical note: Error names and legacy POSIX variables use short truncated names due to early C compiler character limits on variable identifiers.

  • Error Reporting Utilities:

    • perror(const char *s) (defined in <stdio.h>):

    • Prints the string s, followed by a colon and space :, and then the system error message string corresponding to the current value of errno directly to stderr.

    • Typical usage: perror("filename.txt") outputs filename.txt: No such file or directory.

    • strerror(int errnum) (defined in <string.h>):

    • Accepts an error number (e.g., errno) and returns a direct pointer to the static descriptive error string.

    • Useful for custom string formatting, logging, or non-printing processing.

    • Error reporting functions do not automatically terminate the execution of a program.

  • Interfacing C Standard I/O (FILE*) and POSIX Descriptors:

    • C high-level fopen streams wrap low-level POSIX file descriptors.

    • fileno(FILE *stream): Extracts the underlying POSIX file descriptor integer from an active FILE* object.

    • fdopen(int fd, const char *mode): Constructs a high-level FILE* stream object associated with an existing, open POSIX file descriptor fd.

  • Closing File Descriptors (close):

    • close(int fd) disconnects the integer file descriptor from the open file description.

    • Returns 00 on success or −1-1 on failure.

    • A failure in close indicates a program bug (e.g., attempting to close an invalid descriptor or double-closing a descriptor).

    • Deleting a underlying file on disk while open does not fail close; future reads/writes fail, but the file descriptor must still be explicitly closed.

    • Unclosed descriptors are automatically reclaimed by the OS kernel upon process termination.

    • Leaving descriptors open leaks system resources and risks hitting system limits on maximum concurrent open files.

  • Shared File Offsets:

    • An open file description maintains a single shared offset pointer.

    • Sequential read or write calls advance this single offset pointer.

    • To read back data immediately after writing it to a file using descriptors, lseek must be called to move the file offset position backward.

Unix Standard I/O Redirection and Pipelines

  • Unix Modular Design Philosophy:

    • Built around small, single-purpose utilities that can be combined dynamically via I/O streams.

    • Standard input (0), output (1), and error (2) historically connected directly to physical hardware (teletype machines, glass terminals) and now map to pseudo-terminals (/dev/tty), GUI terminal emulators, or SSH sessions.

  • Shell Redirection Syntax:

    • > (Output Redirection): Equivalent to 1>. Redirects standard output (fd 1) to a designated file.

    • Example: program > out.txt creates/opens out.txt for writing and sets stdout to point to it.

    • < (Input Redirection): Equivalent to 0<. Redirects standard input (fd 0) from a file.

    • >> (Append Redirection): Redirects stdout to append to the end of a file.

    • 2> (Standard Error Redirection): Redirects standard error (fd 2) to a file.

    • Example: program 2> errors.log captures error diagnostics separately from regular program output.

    • 2>&1: Redirects stderr (fd 2) to the current destination of stdout (fd 1). Order of arguments matters during evaluation by the shell.

    • Arbitrary descriptor redirection: Any integer prefix can precede redirection syntax (e.g., 4< input_file or 5> output_file).

  • Unix Pipelines (|):

    • Connects the standard output (fd 1) of a producing process directly to the standard input (fd 0) of a consuming process concurrently.

    • Data passes between processes in memory buffers without needing intermediate disk files.

    • Standard error (fd 2) remains directed to the terminal, allowing error messages to display directly to the user without contaminating data passing through the pipe.

  • Example Text Processing Pipeline Case Study:

    • Command pipeline:     cat declaration.txt | tr -c 'a-zA-Z' ' ' | sort | uniq -c | sort -n | more

    • Execution breakdown step-by-step:

    1. cat declaration.txt: Reads text content and writes it to stdout (fd 1).

    2. tr -c 'a-zA-Z' ' ': Complement (-c) option matches any non-alphabetic byte and translates it into a newline (), placing each word on its own line.

    3. sort: Lexicographically sorts all incoming words. Note: sort must buffer and consume its entire standard input before producing any standard output.

    4. uniq -c: Filters adjacent duplicate lines and prefixes each unique line with its occurrence count.

    5. sort -n: Performs a numeric sort in ascending order based on the frequency counts, listing the most frequently used words (e.g., "the") at the end.

    6. more: Pager utility allowing user-controlled scrolling of the final output stream.

File Abstraction in Unix System Design

  • The "Everything is a File" Paradigm:

    • The Unix OS exposes hardware devices, system metrics, and IPC channels as entries in the file system tree using identical system call interfaces (open, read, write, close).

  • Special System Directories and Files:

    • /dev/tty: Represents the controlling terminal process for the current session. Opening /dev/tty allows explicit direct read/write interaction with the physical terminal screen/keyboard, even if standard streams (fd 0, fd 1, fd 2) have been redirected to files or pipes.

    • /proc (Proc Filesystem): A virtual file system generated entirely in-memory by the kernel.

    • Contains subdirectories named after every running Process ID (/proc/[pid]/).

    • Allows inspection of runtime process state, including open file descriptor tables, memory maps, and environmental variables without accessing physical disk storage.