Studied by 80 peopleShell
Provides a command interface to an operating system
A shell operates in a loop
Waits for input
parses input into separate commands
executes each command
What is the CLI Input
string (ASCII or Unicode)
What is the command in CLI
"first" token is program name
What is CLI arguments
any other tokens (after the first) • flags are "decorated" in some way: Unix-style with '-' • anything not a flag is usually considered to be a filename
Interactive
calling ls, cat, grep, etc. from the terminal prompt, e.g. ls -al
Shells scripts
short programs containing shell / command line command and executed by an interpreter
Standard means of specifying data (that does not require human interaction)
• command line arguments • filenames • environment variables (much more uncommon, and can have side effects due to their global nature)
Some standard for input/output + file handling
• streaming using stdin and stdout • filesystem standards • known file and filesystem structure
Testable program results
exit status byte: • '0' is success • nonzero is different types of failure
Commands (first token)
what are we doing
Arguments
what we are operating on • Piped input, files names, etc. • Maybe implied or explicit
Arguments / flags / options
Additional information controlling the execution of the command
Return status to indicate success of failure
possibly with some options for different error types
ls on *nix
lists directory contents
Use: ls [,OPTION/]... [,FILE/]...
• It may accept 0 or more file names (arguments, if we want to thing of them that way) • It may accept 0 or more options
$ls - displays "non-hidden" files in the current directory
Defaults: • FILE is the current directory • Ignores all entities starting with .
which are assumed to be hidden • This includes files prefixed with . • The link to the current directory: . • The link to the parent directory: ..
ls common flags
E.g. two flags: • -a to display all files and ignore nothing • -l to display the long listing format and include file size, creation date, permission, etc. • ls -a -l, or just ls -al
ls other uses
• ls file1 file2 - displays the names of the files if they exist • ls -al file1 file2 - displays the details of the files if they exist • ls dirName - displays the contents of the directory if the path is valid and the user has theread permission for it
ls Exit status
• 0 if OK, • 1 if minor problems (e.g., cannot access subdirectory), • 2 if serious trouble (e.g., cannot access command-line argument).
As you know, the *nix systems contain the manuals for their commands and functions
Accessed using the man (manual) command
sh - the Bourne Shell: the shell on the first popular Unix
• bash - the Bourne Again Shell - adds the command-line extensions of tcsh to sh, as well as someprogramming niceties • zsh - the Z-Shell: named after the author's teaching assistant at the time. Now the default login shell onMacOSX. Asserts backward compatibility with bash
csh - the "C" Shell attempt to look more like a C program in its loops, etc.
tcsh - added in lots of good things for command line use that later shells take for granted: [TAB]completion, history, rollback, history editing
We can also execute shell commands as a complete program
• Written in the language if the shell • Saved in a script file
you can run bash by...
We can run with bash ./hello.sh
The first line of a shell script usually begins with the combination of #! is commonly known by
shebang
Bash variables Assign to variables with no spaces around the '=' sign
• i=123 • ANIMAL=cat • PROBLEM=This is a test # does not work • DATA="This is a test" # must assign only one token
Access the value of the variable using the $ operator
echo $ANIMAL
Encase the variable name in curly brackets {} for clarity
• echo an${i}mal - inserts value of variable i instead of the variable
• echo an$imal - attempts to access variable imal, fails, and replaced it with an empty string
Replacing a variable with its value is referred to as
expansion
Placing characters in double quotes allows variable expansion and preserves spaces
echo "I like my ${ANIMAL}" produces: I like my cat
Using single quotes prevents variable expansion
echo 'I like my ${ANIMAL}' produces: I like my ${ANIMAL}
Inside the shell script, the command line parameters can be accessed by the positionparameters $0, $1, $2, etc.
• The first 10 command line parameters can be accessed as $0 to $9.
• Other parameters must be referenced as ${10} to ${n} for n≥10
• You can see the "value of" operator $ in action again
$0
is the filename of the script
$1 and up
are the command-line parameters
Some useful shorthands
• $# - the number of command-line parameters (0 if none)
• $* - all command-line parameters, i.e. all parameters starting from $1
Branching takes two forms
if-then-fi
if-then-else-fi Different control statements in Bash have adifferent keyword to indicate where they end• For if, it is fi
This is a built-in command Test Syntax
test expression or [ expression ] • The test command evaluates an expression • Returns a condition code indicating that the expression is either true (0) or false (not 0)
Expression contains one or more criteria
• Logical AND operator to separate two criteria: -a • Logical OR operator to separate two criteria: -o • Negate any criterion: !• Group criteria with parentheses • Separate each element with a SPACE
integer comparison
Operator Description -gt Greater than -ge Greater than or equal to -eq Equal to -ne Not euqal to -le Less than or equal to -lt Less than
The expression $?
returns the exit code of the last command
Bash Loops
While loop while [ condition ] do commands done
For loop for VARIABLE in string1 string2 .... stringN do commands done
To increment in bash do
i=$(( $i + 1 ))
An expression in double parentheses (( expression ))
performs arithmetic
if (( something equal to zero ))
evaluates to 1, which is false
if (( something not equal to zero ))
evaluates to 0, which is true
let varName=expression
also performs arithmetic
There are various options to control where input/output are directed
• | pipe stdout of one program to the stdin of another • < redirect stdin to come from a file • > redirect stdout be saved into a newly created file • >> redirect stdout be appended onto an existing file
&2
redirect stdout onto the numbered file descriptor (here 2)
2>&1
collect stderr and stdout together, e.g.,:ls -l > output.txt 2>&1
$find / -name fileName >out.txt
• Search for the specified filename starting from the root directory /, i.e. on the entire drive • Redirect stdout into the file out.txt • equivalent to $find / -name fileName 1>out.txt
$find / -name fileName >out.txt 2>/dev/null
• Search for there specified filename starting from the root directory /, i.e. on the entire drive • Redirect stdout into the file out.txt • Redirect stderr to /dev/null • Anything saved to the file /dev/null is erased • This allows us to suppress all the "Permission denied" the errors
Blocks of script can be created using group commands, subshells or functions
• {...} a group command is constructed by putting scroll braces around the block of code. • commands within group terminated by newline or semicolon
• (...) a subshell is constructed using parentheses creates new execution environment • variable assignments and builtins do not affect the parent
• Braces must be separated from the list of commands by space or metacharacter • Return status is that of the list of commands
Any expressions in back ticks (`) is calculated, and the standard output produced issubstituted into the line
• echo There are wc jabber.txt | cut -c7-9 words • echo "There are "wc jabber.txt | cut -c7-9" words" • echo "There are wc jabber.txt | cut -c7-9 words" • echo 'There are wc jabber.txt | cut -c7-10 words'
• The first three examples produce the same output: back-ticks are substituted beforecomputing double quoted strings • The last one just prints the string - the back-ticked expression is never evaluated
Commonly used to set a variable
today=dateecho "Today is $today"
echo "Text files: ls *.txt"
We can redirect stuff into a loop
while [ someCondition ] do #do things here done <fileName
Special values
• $$ the process ID of the shell (but not a subshell)
• $? the exit status of the most recently completed (and waited upon) command
• $# number of arguments passed to this shell ($1 and up, since $0 is the name of the script)
• "$@" list of all of the arguments, where arguments that have spaces are preserved as one value
• $@ list of all of the arguments, where arguments that have spaces are further split into words
• "$*" similar "$@" but use $IFS (input field separator) as the delimiter instead of a space between words• result is a single token
Defining functions in bash
Functions info in bash
Special parameters $1 through $9 and $@, $* now refer to the function arguments
Variables from parent scope are accessible within the function
When a sub-process (through fork(2)) is created, environment variables are copied into the new environment
BVAR and CVAR are environment variables
Other common environmental variables
• $USER - the login name of the current user • $HOME - the home directory of the current user • $PATH - the list of directories to search for programs • $CWD - the current working directory • $EDITOR - name of (and optionally full path to) your favourite editor (e.g.,"vi", "/usr/bin/vim", /usr/bin/atom) • $LANG - name of your language setting character set if you are working solely in English, thisis typically "en_US.UTF-8"
the value of the shell symbol ~ also expands
~ is the home directory of the current user (i.e. whoever is running the shell)
~someUser is the home directory of someUser
Steps in creating a shell script
• place the code in a file • ensure the shebang is set up correctly • made the file executable: chmod a+x
Each shell has its own "resource" file that will be in your $HOME directory.
• For bash it is .bashrc (note the leading dot) • For csh it is .cshrc • These files are run in your interactive environment once before giving you the first prompt, soany commands or settings put here will take effect for all of your sessions.
.bash_profile
The shell that gets created when you log in, is called a "login shell" and it is controlled bythe .bash_profile file
.bashrc
Later interactive shells created within your login shell will inherit environment settings, but arecontrolled by the .bashrc file
.profile
is the profile of the sh (Bourne) shell
• IIRC, it was automatically created, since the default shell is sh, not bash
.bash_history
contains a history of commands you have entered in the terminal
• On Docker, it might be reset every time you start /stop the container - check
.bash_logout
is optional - contains the commands that execute when the shell terminates • e.g the you log out
filter
A filter is a command that processes stdin and writes on stdout • these allow us to build command pipelines • they usually take an optional file list
sort
order data in various ways, potentially removing duplicates
grep
look for things within files. • Comes from "go to regular expression and print." • Available with "basic" and "extended" regular expression support via grep and egrep
tee
copy the contents that were piped into it two destinations: • a file • to standard output • It's a "T splitter" - the input gets sent to two destinations
echo
copy the command line arguments to standard output
Useful filters: selecting and combining content
• Separate by lines: • head print the first N lines of each file (default 10) • tail print the last N lines of each file (default 10); also print the last all but N lines (i.e.; trim aheader) • cat concatenate the contents of all files together
• Separate by columns: • cut by delimited column or character position • paste combine input as columns of data, concatenating lines together • join perform an "equality join" on the specifier files, combining lines
cmp
compare two files to see if they are the same
diff
print the differences between two files
patch
apply a set of editor commands to update a file based on differences
sed: stream editor
• edit a stream programmatically! • anchors - provide ranges for command action: • line number : absolute coordinates, beginning at 1 • /pattern/: match text• if none given, whole file is implied • editing commands:• s: search and replace • a, i, c: add, insert or change data • d: delete lines • p: print line after edit (used with -n, which suppresses default output)
big-endian
system stores the mostsignificant byte of a word at the smallest memory address and the least significant byte at the largest
little-endian
system stores the least significant byte of a word at the smallest memory address and the most significant byte at the largest
Intel 32-bit and 64-bit architectures
are little-endian
Older Motorola 68* series and PowerPC architectures
big-endian
More commonly used to densely load bytes with flags or small integer values
• e.g. the status value in pid_t wait(int *status); • It is a 32-bit int, where different bits have different meaning: • 24 bits are used to store several pieces of information about how the process terminated • 8 bits store the returned status - i.e. the value from the child's return or exit() statement
| :
bitwise OR - for each bit in the output number, the result bit is 1 if any of the bits are 1 inthe input
0100 0101 | 1100 0000 = 1100 0101 0100 0101 | 0000 1111 = 0100 1111
& :
bitwise AND - for each bit in the output number, the result bit is 1 only if both bits are 1 inthe input
0100 0101 & 1100 0000 = 0100 0000 0100 0101 & 0000 1111 = 0000 0101
^ :
bitwise XOR - for each bit in the output number, the result bit 1 if the two bits in the input are different
0100 0101 ^ 1100 0000 = 1000 0101 0100 0101 ^ 0000 1111 = 0100 1010
~ :
bitwise NOT - inverts every bit of the inout number ~ 0100 0101 = 1011 1010 ~ 0000 1111 = 1111 0000
<< :
left shift - shifts the bits of the first operand to the left by n bits, where n is the second operand • 0100 0101 << 2 = 0001 0100 • 1100 0000 << 2 = 0000 0000
:
right shift - shifts the bits of the first operand to the right by n bits, where n is the secondoperand • 0100 0101 >> 2 = 0001 0001 • 1100 0000 >> 2 = 0011 0000
header file defined flags combined with bitwise OR
reading and writing: 00000002
• create and truncate to zero bytes if file exists: 00000100, 00001000
• do not follow symlinks (no filesystem level redirection): 00400000
• close this file descriptor if this process calls exec: 02000000
Colour pixel data is often stored in 32-bit ints - RGBA format
• RRRRRRRR GGGGGGGG BBBBBBBB AAAAAAAA • 8 bits for each colour channel • 8 bits for Alpha, or transparency
If we don't need the alpha, we can pack more colour information into a 32-bit int:R10G10B10
• RRRRRRRRRR GGGGGGGGGG BBBBBBBBBB UU • Where we have 10 bits per colour and 2 unused bits
Vararg functions must be declared with at least one argument, followed by 3 dots
int func(type arg1, ...) // n = last specified parameter
Inside the function, parameters can be accessed by the C library macros
• void va_start(va_list ap, last); // start param list from lastnamed parameter
• type va_arg(va_list ap, type); // type = next parameter type
• va_end(va_list ap); // clear parameter list
Based on the known type of at least one initial argument
va_arg lets you pull off the nextargument as an iterator based on the type of the argument
Using vararg functions So how does printf() do this?
• It has a wide mix of types! the first argument to printf() provides the list of all of the remainingargument types!
• provided an earlier argument indicates the type of a later one, this works - because the C stackis ordered left to right
• modern linters have special handling for printf() to try and make sure that they types match —but this won't be true for other vararg functions
strconcat prototype
• Needs to describe how the argument list starts: char * strconcat(const char *, ...);
• We also need to specify the end of the argument list
• In out function, we pass a NULL pointer as the last argument
• Please note that the term "list" is a bit of a misnomer - the varargs are actually a contiguousblock of memory with no intrinsic terminator
• We need to pass the type to va_arg, so it knows how to advance its next element pointer
• The implementation details are compiler-specific, however
Inner workings of vararg functions
• Each function has its own stack frame
• In order to call a function, the parent sets aside memory for the parameters for the child call
• Each function sets aside memory for its own local variables
• Parameter and local variables can be found relative to the frame pointer (FP)
• The previous frame pointer and program counter are stored as part of the frame
Varargs vs unspecified number of parameters
• C allows the following syntax:type funcName()
• In this case, funcName() has an unspecified number of arguments
• The compiler will not check the number or type of arguments
• However, the arguments of arguments if not variable - it's fixed
• This is not equivalent to a variable number of arguments
• This is old, pre-ANSI style of writing C
• It prevents the compiler from doing proper type/signature checking and can lead to undefinedbehaviour
• I would recommend again it
Note
Flashcard