Chapter 7: Shell Environment, Scripting, and Command Redirection

Command Input, Output, and File Descriptors

• The command shell serves as the primary user interface and command interpreter within Linux operating systems, managing interactive user input, parsing program requests, and manipulating standard data streams through metacharacters.

• Command execution streams are classified into three distinct abstract channels designated by numeric identifiers known as file descriptors:

  • Standard Input (stdin): Identified by file descriptor $0$. It accepts information processed by a command during execution (default symbol <).

  • Standard Output (stdout): Identified by file descriptor $1$. It delivers the normal, non-error information generated by the command during execution (default symbol >).

  • Standard Error (stderr / stderror): Identified by file descriptor $2$. It outputs informational errors, diagnostic notifications, and exception messages generated by the command (default symbol >).

• File descriptors provide explicit redirection capabilities away from the terminal screen directly into filesystem files:

  • command 1>file or command >file: Directs stdout to a file, overwriting the destination file contents if the file already exists.

  • command 2>file: Directs stderr to a specified file, overwriting previous contents.

  • command 1>fileA 2>fileB or command >fileA 2>fileB: Splits outputs, directing stdout to fileA and stderr to fileB simultaneously.

  • command 1>file 2>&1 or command >file 2>&1: Redirects stdout to a file and subsequently points descriptor $2$ to descriptor $1$, merging output and error into the same file.

  • command 1>&2 2>file or command >&2 2>file: Directs stdout to stderr and sends stderr to a target file.

  • command &>file: Shorthand notation sending both stdout and stderr to the same destination file.

  • command 1>>file or command >>file: Appends stdout to an existing file rather than overwriting it.

  • command 2>>file: Appends stderr to an existing file.

  • command 0<file or command <file: Feeds the contents of an existing file into the command via stdin.

  • command <<delimiter: Here Document (here-doc); directs multiline input into stdin until a designated terminal token or delimiter is encountered.

Command Pipelines and Text Filtering Utilities

• Piping Metacharacter (|):

  • Connects the standard output (stdout) of a preceding command directly into the standard input (stdin) of the subsequent command.

  • Utilized extensively to chain utilities together and format, reduce, or manipulate large volumes of text before display.

  • Multiple pipes may be connected in series to route information across consecutive commands:

• Pipe and Redirection Combinations:

  • Redirection and piping may be combined along a single command line provided input redirection occurs at the initiation of the pipe chain and output redirection occurs at the termination of the pipe chain.

• Filter Utilities:

  • Any command capable of accepting data from stdin and returning modified results to stdout functions as a filter.

  • sort: Orders input lines alphanumerically.

  • sort -r: Sorts input lines in reverse alphanumeric order.

  • wc: Counts file metrics.

  • wc -l: Computes the line count.

  • wc -w: Computes the word count.

  • wc -c: Computes the character count.

  • pr: Formats files for printing, generating top-of-page headers containing the current date and page numbering.

  • pr -d: Formats printing output with double line-spacing.

  • uniq: Removes duplicate contiguous lines from sorted text streams.

  • cut: Extracts and displays designated columns or fields of delimited character data.

  • paste: Merges corresponding lines from two separate input streams or files side by side.

  • tr: Replaces or translates specific individual characters within a stream received through stdin.

  • tee filename: Duplicates data streams by directing stdin simultaneously to stdout and a destination file. Overwrites the file unless the -a (append) flag is declared.

  • xargs command: Converts text lines passed via stdin into delimited command-line arguments and passes them to the designated executable command.

  • split: Fractures large files into smaller segments to optimize downstream pipeline processing.

  • grep: Searches lines matching a basic regular expression pattern.

  • egrep: Searches lines matching an extended regular expression pattern.

  • nl: Generates line numbers for every line of input text.

  • awk: Pattern-scanning and processing language that treats individual lines as structured records in a database, executing user-defined pattern-action instructions.

  • sed: Stream editor used to search, replace, manipulate, or delete specific stream contents based on regular expressions.

Shell Variables: Environment, User-Defined, and Special Variables

• Variables represent named memory blocks designated to store dynamic configuration data accessible by the operating system, shell environment, and child processes.

• Environment Variables:

  • Variables initialized globally by the system to configure default operating parameters.

  • Inspection Commands:

  • set: Lists all active shell variables (environment and user-defined) alongside current values.

  • echo $VARIABLE</code>: Outputs the evaluated value assigned to a specified variable name (prefixed by the <code>$ metacharacter).

  • history: Displays previously executed commands retained in memory.

  • Immutable Variables: System variables HOME and PWD must not be manually modified.

  • Core Environment Variables Reference:

  • BASH: Contains the absolute filesystem path to the active BASH binary.

  • BASH_VERSION: Contains the version number of the active BASH shell.

  • DISPLAY: Directs graphical X Windows output to a designated host, screen, or device.

  • EDITOR: Identifies the system default text editor invoked by shell programs (typically vi or nano).

  • ENV: Absolute path identifying the run-time configuration file of the BASH shell.

  • EUID: Numeric Effective User ID of the currently logged-in user.

  • HISTFILE: Filename storing historical shell commands across sessions (typically ~/.bash_history).

  • HISTFILESIZE: Total command lines retained within HISTFILE upon logout for use in subsequent sessions (typically $1000$).

  • HISTSIZE: Total command lines stored dynamically in active RAM during an active login session (typically $1000$).

  • HOME: Absolute path of the current user's personal home directory.

  • HOSTNAME: Network hostname identifying the local machine.

  • LOGNAME: System username of the current user established at login.

  • MAIL: Path location of the incoming system mail spool file.

  • OSTYPE: Operating system name identifier.

  • PATH: Colon-delimited directory search path scanned sequentially by the shell to locate executable binaries when a command is entered without an absolute or relative path containing /.

  • PS1: Defines the primary shell prompt display format.

  • PWD: Absolute path corresponding to the present working directory.

  • RANDOM: Generates a pseudorandom integer value each time the variable is referenced.

  • SHELL: Path to the user's primary login shell program.

  • TERM: Terminal emulation setting (typically linux or xterm on contemporary Linux distributions, and console on legacy configurations).

  • TERMCAP: Terminal emulation configuration path pointing to a system database (/etc/termcap).

• User-Defined Variables:

  • Custom variables generated by shell users via assignment: VARIABLE_NAME=value.

  • Syntax Identification Rules:

  • Can contain alphanumeric characters (0–9, A–Z, a–z), hyphens (-), or underscores (_).

  • Must never begin with a numeric character.

  • Standard convention requires capitalizing identifiers for readability and distinction from system binaries.

  • Variable Scope and Subshells:

  • Subshells are child shell environments spawned by the current parent shell to execute external scripts and tasks.

  • User-defined variables are local to the current shell instance and are completely inaccessible to child subshells by default.

  • export VARIABLE: Promotes a local variable to an environment variable inherited by all subsequent subshells and child processes.

  • env: Lists all exported variables currently available within the active shell session.

  • printenv: Displays exported variables and active functions residing in current memory.

  • unset VARIABLE: Clears a defined variable or function completely from memory.

• Specialized Variables and Aliases:

  • Specialized shell variables are omitted from standard set and env outputs.

  • umask: Controls default octal file permission masks and is managed exclusively using the umask command.

  • alias: Configures shortcut command aliases, including multi-command sequences delimited by semicolons (;).

  • unalias name: Deletes an alias configuration from memory.

Shell Environment Startup Files

• Session Lifecycle:

  • Variables stored in volatile system memory are instantly lost upon terminal session exit.

  • Persistent configurations require placing initialization directives inside environment files executed automatically at shell creation.

• Standard BASH Environment Startup Files:

  • /etc/profile: Primary system-wide startup script executed for every user upon initial login.

  • /etc/profile.d/*: Directory housing modular application-specific initialization scripts sourced by /etc/profile.

  • /etc/bashrc: System-wide startup configuration file executed whenever a non-login interactive BASH shell or subshell starts.

  • ~/.bashrc: User-specific startup file executed for non-login subshells; typically used to declare custom aliases and personal variables.

  • ~/.bash_profile: Primary user-specific configuration script parsed upon interactive login.

  • ~/.bash_login: Secondary fallback user login script executed if ~/.bash_profile is absent.

  • ~/.profile: General fallback user login script executed if neither ~/.bash_profile nor ~/.bash_login can be located.

Shell Script Creation, Execution, and Terminal Formatting

• Script Characteristics:

  • Plaintext executable files housing sequential shell commands, variable references, and decision constructs parsed line-by-line by an interpreter.

• Hashpling (Shebang):

  • The initial header line of a shell script: #!/bin/bash.

  • Defines the absolute interpreter path invoked by the kernel to parse and run the script contents.

• Execution Permissions and Methods:

  • Starting through a subshell: bash scriptfile (requires read permission on scriptfile).

  • Executing as a standalone binary: chmod +x scriptfile followed by ./scriptfile (requires both read and execute permissions).

• Escape Sequences (Echo Formatting):

  • Used alongside the echo command to format and align text output:

  • \0???: Inserts an ASCII character mapped to a three-digit octal number (???).

  • \x??: Inserts an ASCII character mapped to a two-digit hexadecimal number (??).

  • \\: Prints a literal backslash.

  • \a: Emits an audible ASCII alert bell.

  • \b: Performs a backspace.

  • \c: Prevents/suppresses the standard trailing newline.

  • \f: Emits a form feed.

  • : Advances the cursor to start a new line.

  • \r: Performs a carriage return.

  • \t: Inserts a horizontal tab stop.

  • \v: Inserts a vertical tab stop.

• Standard Input Reading (read):

  • Interactively captures input data from stdin and stores it directly into a designated variable identifier.

  • read VARIABLE: Reads standard terminal entry into VARIABLE.

  • read -p "Prompt text" VARIABLE: Prints a formatted string prompt and captures subsequent input into VARIABLE.

  • Values are accessed downstream via $VARIABLE.

Decision Constructs and Conditional Evaluation

• Purpose:

  • Modifies linear script execution flow depending on command success exit codes or responses evaluated against logical test statements.

• Test Expressions (test or [ ... ]):

  • String Comparisons:

  • [ A = B ]: Evaluates to true if String A equals String B.

  • [ A != B ]: Evaluates to true if String A is not equal to String B.

  • Integer Comparisons:

  • [ A -eq B ]: Evaluates to true if integer A equals integer B.

  • [ A -ne B ]: Evaluates to true if integer A is not equal to integer B.

  • [ A -lt B ]: Evaluates to true if integer A is strictly less than integer B.

  • [ A -le B ]: Evaluates to true if integer A is less than or equal to integer B.

  • [ A -gt B ]: Evaluates to true if integer A is strictly greater than integer B.

  • [ A -ge B ]: Evaluates to true if integer A is greater than or equal to integer B.

  • File System Condition Tests:

  • [ -r A ]: Evaluates to true if path A exists and is readable.

  • [ -w A ]: Evaluates to true if path A exists and is writable.

  • [ -x A ]: Evaluates to true if path A exists and is executable.

  • [ -f A ]: Evaluates to true if path A exists and is a regular file.

  • [ -d A ]: Evaluates to true if path A exists and is a directory.

  • Logical Operators within Test Statements:

  • [ A = B -o C = D ]: Logical OR (-o); returns true if statement $A=B$ is true OR statement $C=D$ is true.

  • [ A = B -a C = D ]: Logical AND (-a); returns true if statement $A=B$ is true AND statement $C=D$ is true.

  • [ ! A = B ]: Logical NOT (!); inverts truth value, returning true if String A is NOT equal to String B.

• The if Construct:

  • Syntax structure:

  • if some condition; then

  • do these commands

  • elif some condition; then

  • do these commands

  • else

  • do these commands

  • fi

  • Syntax rules:

  • The elif and else branches are optional.

  • A construct may contain an unlimited number of elif clauses.

  • The command execution blocks may comprise multiple commands placed one per line, with or without indentation.

  • The construct must terminate with the closing token fi.

  • Condition checks accept either standard test bracket statements or executable command success codes.

• The case Construct:

  • Compares variable contents sequentially against text patterns or regular matching tokens.

  • Syntax structure:

  • case $variable in

  • pattern1 ) do this ;;

  • pattern2 ) do this ;;

  • pattern3 ) do this ;;

  • esac

  • Syntax rules:

  • The pipe | metacharacter allows multiple discrete input patterns to execute the same target commands.

  • The asterisk * serves as the default catch-all wildcard pattern matching unintended or unhandled inputs.

  • Command bodies can contain multiple commands per line or across multiple indented/unindented lines.

  • Every pattern command block must be closed by double semicolons ;;.

  • The entire construct must terminate with esac.

  • Subsequent script logic resumes following esac.

• Short-Circuit Logical Operators (&& and ||):

  • Evaluates single-step conditional operations inline:

  • command1 && command2: Logical AND. command2 runs if and only if command1 returns a successful exit status code ($0$).

  • command1 || command2: Logical OR. command2 runs if and only if command1 returns an unsuccessful, non-zero exit status code.

Loop Constructs and Repetitive Execution

• Repetition Logic:

  • Loop constructs automate iterative task execution until an evaluation condition changes or target data lists are exhausted.

  • Counter Variables: Variables updated systematically within a loop to prevent infinite execution cycles.

• The for Loop:

  • Iterates across a predefined set of strings, objects, or command outputs.

  • String iteration syntax:

  • for variable in string1 string2 string3; do

  • do these commands

  • done

  • Sequence-based iteration syntax using command substitution:

  • for variable inseq n; do

  • do these commands

  • done

• The while Loop:

  • Executes commands repeatedly as long as the evaluation test statement evaluates to true.

  • Syntax structure:

  • while this returns true; do

  • do these commands

  • done

  • Integrates a counter variable modified in each iteration to eventually trigger loop termination.

• The until Loop:

  • Executes commands repeatedly as long as the evaluation test statement remains false, terminating when the expression returns true.

  • Syntax structure:

  • until this returns true; do

  • do these commands

  • done

• Helper Loop Utilities:

  • seq "start [inc] end": Generates sequential numeric lists starting at start, incrementing by inc (optional), and ending at end.

  • expr "math operation": Calculates integer mathematical evaluations in shell operations.

Advanced Shell Scripting: Expansions, Positional Parameters, and Functions

• Shell Expansion Variables:

  • Command Substitution: Executes a command and captures its standard output directly into a statement.

  • Backtick syntax: `command`

  • Parentheses syntax: $(command)</code></p></li><li><p>Arithmetic Expansion: Evaluates mathematical operations natively via <code>$((math operation)).

  • Variable Expansion Braces:

  • ${VAR:=value}</code>: Assigns <code>value</code> to variable <code>VAR</code> if unset or null.</p></li><li><p><code>${!pattern}: Variable substitution matching an expanded pattern.

• Array Variables:

  • Stores multiple elements mapped to numbered index positions.

  • In the BASH shell, array element numbering starts at index $0$.

• Positional Parameters:

  • Captures command-line arguments passed into a script or function during invocation ($1$ to $n$):

  • Evaluated through positional indicators $1, $2, $3, …, $9, ${10}</code>, …, <code>${n}.

• Built-in System Variables:

  • #: Total count of positional parameters passed to the script or function.

  • *: All positional parameter values represented as a single contiguous string.

  • $@</code>: All positional parameter values represented as individual quoted strings.</p></li><li><p><code>$?: Numeric exit status code of the last completed foreground command ($0$ indicates complete success; non-zero indicates an execution error).

  • ``: Process ID (PID) number of the currently running shell.\n * `!`: Process ID (PID) number of the most recently initiated background process.\n * `-`: Listing of active operational flags enabled within the current shell instance.\n\n* Shell Functions and Libraries:\n * Shell Function: A named block of instructions that accepts localized positional parameters and executes repetitive commands on demand throughout a parent script.\n * Function Library: A script file containing modular function definitions exclusively.\n * Loading Libraries: Libraries are read into active memory using the `source` or dot (`.`) command (e.g., `. /path/to/library`).\n\n# Git Version Control Architecture\n\n* Purpose:\n * A systematic change-tracking engine recording file modifications, version iterations, and historical baselines for shell scripts and program source code.\n\n* Git Characteristics:\n * The primary open-source distributed version control system across Linux, macOS, and Windows environments.\n\n* Commits:\n * Permanent historical point-in-time snapshots of tracked repository files.\n\n* Repositories (Repos):\n * A local or remote database directory where Git archives project files, tracking configurations, and associated commit trees.\n\n# The Z Shell (Zsh)\n\n* Architectural Role:\n * Z shell operates as a backward-compatible superset of the BASH shell, providing advanced terminal control, scripting, and shell customization.\n\n* Core Differences from BASH:\n * Hashpling Interpreter: Zsh scripts declare `#!/bin/zsh` as their initial shebang.\n * Array Indexing: Array values in Zsh are indexed starting from 1$(differing from the BASH convention of starting at$0).\n * Environment Initialization Files: Uses distinct environment initialization paths, managing the primary configuration through `/etc/bashrc` and `~/.zshrc`.\n\n* Advanced Redirection and Piping:\n * Multi-Stream Redirection: Simultaneously accepts standard input from multiple files, and routes standard output and standard error across multiple files.\n * Direct In-Pipe Redirection: Redirects standard output and standard error directly inside pipe streams without requiring the `tee` utility.\n * Interactive Input Viewing: Utilizes standard input redirection alone directly on the command line to display file text.\n\n* Shell Options and Automation:\n * Features built-in parameters for completion, globbing, command history, and emulation.\n * `autocd`: Changes the active working directory to a specified subdirectory path without typing the `cd` command.\n * `correct`: Analyzes terminal commands and suggests interactive spelling corrections for mistyped commands.\n\n* Module Architecture:\n * Modifies built-in capabilities dynamically via runtime modules.\n * `zmodload`: Displays all currently loaded modules when invoked without arguments.\n\n* Function Libraries:\n * Located in directory trees defined in the `$fpath` search variable.\n * Autoloading: Declaring `autoload -U mylibrary && mylibrary` within `~/.zshrc` automatically imports and executes library functions.\n * `compinit`: Core function library extending standard tab-completion to encompass detailed command options, switches, and positional arguments.\n\n* Plugins and Themes:\n * Extensible framework supporting over 250 community plugins providing workflow enhancements.\n * Configurable interface supporting over 150$$ terminal display themes modifying terminal prompts and indicators.

Exam Coding Questions (Final Study Guide)

• Redirection

  • Overwrite a file with stdout: ls > out.txt (same as 1>out.txt)

  • Append to a file: echo "log" >> file.txt

  • Redirect stdin from a file: command < file

  • Errors only: grep "error" file.txt 2> err.txt

  • stdout and stderr to the same file (any of these): ls /etc/host &> output.txt, >output.txt 2>&1, 2>output.txt >&2

  • File descriptors: stdin = 0, stdout = 1, stderr = 2.

• Piping (sends stdout of one command to the next command's stdin)

  • Page through output: ls -l | less

  • Count lines: ps aux | wc -l

  • Filter with grep: dmesg | grep "error"

  • Unique sorted list: cat names.txt | sort | uniq

• awk

  • Print lines containing manager: awk '/manager/ {print}' employee.txt

  • Print the 1st and 3rd fields of those lines: awk '/manager/ {print $1 $3}' employee.txt (use {print $1, $3} to put a space between them)

• grep

  • Simple search: grep "error" logfile; recursive: grep -r "TODO" .; extended regex: grep -E "[a-z]+@[a-z]+\.[a-z]+" emails.txt

Questions & Discussion

• Knowledge Check 7-1:

  • Question: The sed and awk commands are filter commands commonly used to format data within a pipe.

  • a. True

  • b. False

  • Answer: a. True

  • Detailed Explanation: Both sed and awk function as stream filters that accept text from stdin, process data based on instructions, and format output stream information inside pipes.

• Knowledge Check 7-2:

  • Question: Before a user-defined variable can be used by processes that run in subshells, that variable must be _____.

  • a. imported

  • b. validated by running the env command

  • c. exported

  • d. redirected to the shell

  • Answer: c. exported

  • Detailed Explanation: By default, local user-defined variables remain private to the parent shell. Running the export command copies the variable definition into the environment space inherited by child subshells.

• Knowledge Check 7-3:

  • Question: A for construct is a loop construct that processes a specified list of objects. As a result, it is executed as long as there are remaining objects to process.

  • a. True

  • b. False

  • Answer: a. True

  • Detailed Explanation: The for loop executes its defined commands iteratively across each finite token provided in an object list, concluding once the list is exhausted.

• Self-Assessment Questions:

  • Question 1: Why would you want to create a shell script?

  • Question 2: Have you ever created a shell script on a Linux system? If so, what commands were executed in the script?