Shell Scripting Notes

Introduction to Shell Scripting

Shell scripting allows users to automate the execution of UNIX commands in a single file (a "script") instead of typing them sequentially. This script can be run in the bash shell, which is a command-line interpreter for interacting with the operating system. While shell scripts can be quite powerful, they are not a complete substitute for a full programming language such as C, Python, or R. Nevertheless, they are essential for automating tasks such as backups, file format conversions, and file handling (adding, deleting, moving).

Your First Shell Script

To start creating a shell script, certain conventions and syntax rules should be followed:

  1. Names for variables should typically be in UPPERCASE.

  2. Complex variable names can use snakecase (e.g., VARNAME).

  3. When assigning values to variables, do not include spaces around the "=" (e.g., MY_VAR=value is correct).

Example of a Basic Shell Script

Create a file called boilerplate.sh and begin with:

#!/bin/bash
# Author: Your Name
your.login@imperial.ac.uk
# Script: boilerplate.sh
# Desc: simple boilerplate for shell scripts
# Arguments: none
# Date: Oct 2019
echo -e "\nThis is a shell script! \n"

The initial line is called a shebang, which tells the system to run this script using the bash interpreter. Comments can be added with the # symbol.

Special Characters

In shell scripting, some characters are considered "special" and need to be escaped. For example, the semicolon ; should be escaped using a backslash \ in certain contexts. Familiarity with which characters need escaping is crucial when writing scripts.

Running Shell Scripts

There are two primary methods to run a shell script:

  1. Using the bash interpreter directly:

   bash myscript.sh

  1. Making the script executable:

   chmod +x myscript.sh
   ./myscript.sh

To make a script accessible across the system, place it in the ~/bin directory and ensure that this location is included in the $PATH variable. You can check your existing $PATH utilizing the command:

echo $PATH

Variables in Shell Scripts

Shell scripts extensively work with variables, which can be categorized into:

Special Variables

  1. Environmental Variables: These are system-wide variables such as $PATH that are inherited by newly spawned processes.

  2. Special Internal Variables: Exist for the current execution environment of the script and are not available afterward unless exported.

Examples of Special Variables:

  • $0: The filename of the current script.

  • $n ($1…$9): Positional parameters passed to the script.

  • #: Number of arguments supplied to the script.

  • $@: All arguments passed to the script.

Assigned Variables

These are defined by the user and can take values through:

  1. Explicit declaration: MY_VAR=myvalue

  2. Reading user input with read MY_VAR

  3. Command substitution: MY_VAR=$(command)
    Example:

MY_VAR=$(ls | wc -l)

A Useful Shell-Scripting Example

To illustrate variable usage, save a script named variables.sh with the following content:

#!/bin/sh
# Special variables  
echo "This script was called with $# parameters"  
echo "The script's name is $0"  
echo "The arguments are $@"

Additionally, here’s a demonstration on transforming files:
Convert CSV to TSV or vice versa using tr command.

Example Shell Script to Convert Format
#!/bin/sh
# Author: Your Name
# Script: tabtocsv.sh
# Description: Substitute tabs in a file with commas
# Arguments: 1 -> tab delimited file
# Date: Oct 2019

echo "Creating a comma delimited version of $1 ..."
cat $1 | tr -s "\t" "," >> $1.csv
echo "Done!"

More Examples

Counting Lines in a File

Save this as CountLines.sh:

#!/bin/bash
NumLines=$(wc -l < $1)
echo "The file $1 has $NumLines lines"
Merging Two Files
#!/bin/bash
cat $1 > $3
cat $2 >> $3
echo "Merged File is"
cat $3

Practicals

When working on practicals:

  • Ensure scripts are well organized in directories like CMEECourseWork/week1/code.

  • Include a README file outlining the contents and functionality of each script.

  • Verify that scripts work on different UNIX/Linux systems using relative paths.

In summary, shell scripting is a powerful tool for automating tasks on UNIX systems. By following conventions, understanding the use of variables, and creating useful scripts, users can significantly enhance productivity.

Glossary of Shell Scripting Terms and Concepts

This glossary defines key terms, commands, and special variables used in shell scripting, as discussed in the notes.

  • Shell Scripting

    • Definition: The process of automating UNIX commands by writing them in a single file (a "script") instead of typing them sequentially. It's used for tasks like backups, file conversions, and handling.

    • Example: Creating a .sh file to perform a series of file operations.

  • Shebang (#!/bin/bash)

    • Definition: The initial line in a shell script that tells the system which interpreter to use for executing the script.

    • Example: #!/bin/bash specifies that the script should be run using the bash interpreter.

  • Comments (#)

    • Definition: Lines in a script that are ignored by the interpreter and used for documentation or to disable parts of the code. They begin with a hash symbol.

    • Example: # Author: Your Name

  • echo

    • Definition: A command used to display lines of text or the values of variables to standard output.

    • Example: echo -e "\nThis is a shell script! \n" prints a message with newlines.

  • chmod +x

    • Definition: A command used to change file permissions, specifically to make a script executable. The +x option adds execute permission.

    • Example: chmod +x myscript.sh makes myscript.sh executable.

  • Running a Script with bash

    • Definition: Directly invoking the bash interpreter to execute a script, useful when the script isn't executable or a different interpreter is preferred.

    • Example: bash myscript.sh

  • $PATH Environmental Variable

    • Definition: A system-wide environmental variable that contains a colon-separated list of directories where the shell looks for executable commands.

    • Example: echo $PATH displays the current search path for executables.

  • Environmental Variables

    • Definition: System-wide variables inherited by newly spawned processes, containing information about the system and user environment.

    • Example: $PATH is an environmental variable.

  • Special Internal Variables

    • Definition: Variables that exist specifically for the current execution environment of a script and provide information about the script's invocation and arguments.

  • $0

    • Definition: A special internal variable that holds the filename of the current script.

    • Example: echo "The script's name is $0" prints the name of the script being executed.

  • $n ($1…$9)

    • Definition: Special internal variables representing positional parameters (arguments) passed to the script, where $1 is the first argument, $2 the second, and so on.

    • Example: In tabtocsv.sh, $1 refers to the first argument supplied (the tab-delimited file).

  • $#

    • Definition: A special internal variable that holds the total number of arguments supplied to the script.

    • Example: echo "This script was called with $# parameters" prints how many arguments were given.

  • $@

    • Definition: A special internal variable that expands to all arguments passed to the script, treated as separate strings.

    • Example: echo "The arguments are $@" prints all arguments.

  • Assigned Variables (VAR=value)

    • Definition: Variables explicitly defined by the user within a script. Values are assigned without spaces around the equals sign.

    • Example: MY_VAR=myvalue assigns the string myvalue to MY_VAR.

  • read

    • Definition: A command used to read a line of input from standard input and assign it to a variable.

    • Example: read MY_VAR waits for user input and stores it in MY_VAR.

  • Command Substitution ($(command))

    • Definition: A mechanism where the output of a command is captured and used as the value for a variable or as part of another command.

    • Example: MY_VAR=$(ls | wc -l) assigns the number of files/directories in the current directory to MY_VAR.

  • cat

    • Definition: A command used to concatenate files and print their content to standard output.

    • Example: cat $1 | tr -s "\t" "," reads content from $1.

  • tr

    • Definition: A command used to translate or delete characters. The -s option squeezes repeating characters.

    • Example: tr -s "\t" "," replaces all tab characters (\t) with commas (,) and squeezes multiple consecutive delimiters into one.

  • wc -l

    • Definition: A command (wc for word count with the -l option) used to count the number of lines in a file or input stream.

    • Example: NumLines=$(wc -l < $1) counts lines in the file specified by $1.

  • Redirection (>, >>)

    • Definition: Operators used to change where a command's output or input goes.

      • >: Overwrites a file with the command's output.

      • >>: Appends the command's output to the end of a file.

    • Example: cat $1 >> $1.csv appends the content of $1 to $1.csv.

  • Pipe (|)

    • Definition: An operator used to direct the standard output of one command to be the standard input of another command, linking them together.

    • Example: cat $1 | tr -s "\t" "," sends the output of cat $1 as input to tr.