CIS3050 Midterm 2

Shell

Provides a command interface to an operating system

A shell operates in a loop

- Waits for input
- parses input into separate commands
- executes each command

What is the CLI Input

string (ASCII or Unicode)

What is the command in CLI

"first" token is program name

What is CLI arguments

any other tokens (after the first)
• flags are "decorated" in some way: Unix-style with '-'
• anything not a flag is usually considered to be a filename

Interactive

calling ls, cat, grep, etc. from the terminal prompt, e.g. ls -al

Shells scripts

short programs containing shell / command line command and executed by an interpreter

Standard means of specifying data (that does not require human interaction)

• command line arguments
• filenames
• environment variables (much more uncommon, and can have side effects due to their global nature)

Some standard for input/output + file handling

• streaming using stdin and stdout
• filesystem standards
• known file and filesystem structure

Testable program results

exit status byte:
• '0' is success
• nonzero is different types of failure

Commands (first token)

what are we doing

Arguments

what we are operating on
• Piped input, files names, etc.
• Maybe implied or explicit

Arguments / flags / options

Additional information controlling the execution of the command

Return status to indicate success of failure

possibly with some options for different error types

ls on *nix

lists directory contents

Use: ls [,OPTION/]... [,FILE/]...

• It may accept 0 or more file names (arguments, if we want to thing of them that way)
• It may accept 0 or more options

$ls - displays "non-hidden" files in the current directory

Defaults:
• FILE is the current directory
• Ignores all entities starting with .
- which are assumed to be hidden
• This includes files prefixed with .
• The link to the current directory: .
• The link to the parent directory: ..

ls common flags

E.g. two flags:
• -a to display all files and ignore nothing
• -l to display the long listing format and include file size, creation date, permission, etc.
• ls -a -l, or just ls -al

ls other uses

• ls file1 file2 - displays the names of the files if they exist
• ls -al file1 file2 - displays the details of the files if they exist
• ls dirName - displays the contents of the directory if the path is valid and the user has theread permission for it

ls Exit status

• 0 if OK,
• 1 if minor problems (e.g., cannot access subdirectory),
• 2 if serious trouble (e.g., cannot access command-line argument).

As you know, the *nix systems contain the manuals for their commands and functions

Accessed using the man (manual) command

sh - the Bourne Shell: the shell on the first popular Unix

• bash - the Bourne Again Shell - adds the command-line extensions of tcsh to sh, as well as someprogramming niceties
• zsh - the Z-Shell: named after the author's teaching assistant at the time. Now the default login shell onMacOSX. Asserts backward compatibility with bash

csh - the "C" Shell attempt to look more like a C program in its loops, etc.

tcsh - added in lots of good things for command line use that later shells take for granted: [TAB]completion, history, rollback, history editing

We can also execute shell commands as a complete program

• Written in the language if the shell
• Saved in a script file

you can run bash by...

We can run with bash ./hello.sh

The first line of a shell script usually begins with the combination of #! is commonly known by

shebang

Bash variables Assign to variables with no spaces around the '=' sign

• i=123
• ANIMAL=cat
• PROBLEM=This is a test # does not work
• DATA="This is a test" # must assign only one token

Access the value of the variable using the $ operator

echo $ANIMAL

Encase the variable name in curly brackets {} for clarity

• echo an${i}mal - inserts value of variable i instead of the variable

• echo an$imal - attempts to access variable imal, fails, and replaced it with an empty string

Replacing a variable with its value is referred to as

expansion

Placing characters in double quotes allows variable expansion and preserves spaces

echo "I like my ${ANIMAL}"
produces: I like my cat

Using single quotes prevents variable expansion

echo 'I like my ${ANIMAL}'
produces: I like my ${ANIMAL}

Inside the shell script, the command line parameters can be accessed by the positionparameters $0, $1, $2, etc.

• The first 10 command line parameters can be accessed as $0 to $9.

• Other parameters must be referenced as ${10} to ${n} for n≥10

• You can see the "value of" operator $ in action again

$0

is the filename of the script

$1 and up

are the command-line parameters

Some useful shorthands

• $# - the number of command-line parameters (0 if none)

• $* - all command-line parameters, i.e. all parameters starting from $1

Branching takes two forms

- if-then-fi
- if-then-else-fi
Different control statements in Bash have adifferent keyword to indicate where they end• For if, it is fi

This is a built-in command Test Syntax

test expression or [ expression ]
• The test command evaluates an expression
• Returns a condition code indicating that the expression is either true (0) or false (not 0)

Expression contains one or more criteria

• Logical AND operator to separate two criteria: -a
• Logical OR operator to separate two criteria: -o
• Negate any criterion: !• Group criteria with parentheses
• Separate each element with a SPACE

integer comparison

Operator Description
-gt Greater than
-ge Greater than or equal to
-eq Equal to
-ne Not euqal to
-le Less than or equal to
-lt Less than

The expression $?

returns the exit code of the last command

Bash Loops

While loop
while [ condition ] do
commands
done

For loop
for VARIABLE in string1 string2 .... stringN do
commands
done

To increment in bash do

i=$(( $i + 1 ))

An expression in double parentheses (( expression ))

performs arithmetic

if (( something equal to zero ))

evaluates to 1, which is false

if (( something not equal to zero ))

evaluates to 0, which is true

let varName=expression

also performs arithmetic

There are various options to control where input/output are directed

• | pipe stdout of one program to the stdin of another
• < redirect stdin to come from a file
• > redirect stdout be saved into a newly created file
• >> redirect stdout be appended onto an existing file

>&2

redirect stdout onto the numbered file descriptor (here 2)

2>&1

collect stderr and stdout together, e.g.,:ls -l > output.txt 2>&1

$find / -name fileName >out.txt

• Search for the specified filename starting from the root directory /, i.e. on the entire drive
• Redirect stdout into the file out.txt
• equivalent to $find / -name fileName 1>out.txt

$find / -name fileName >out.txt 2>/dev/null

• Search for there specified filename starting from the root directory /, i.e. on the entire drive
• Redirect stdout into the file out.txt
• Redirect stderr to /dev/null
• Anything saved to the file /dev/null is erased
• This allows us to suppress all the "Permission denied" the errors

Blocks of script can be created using group commands, subshells or functions

• {...} a group command is constructed by putting scroll braces around the block of code.
• commands within group terminated by newline or semicolon

• (...) a subshell is constructed using parentheses creates new execution environment
• variable assignments and builtins do not affect the parent

• Braces must be separated from the list of commands by space or metacharacter
• Return status is that of the list of commands

Any expressions in back ticks (`) is calculated, and the standard output produced issubstituted into the line

• echo There are `wc jabber.txt | cut -c7-9` words
• echo "There are "`wc jabber.txt | cut -c7-9`" words"
• echo "There are `wc jabber.txt | cut -c7-9` words"
• echo 'There are `wc jabber.txt | cut -c7-10` words'

• The first three examples produce the same output: back-ticks are substituted beforecomputing double quoted strings
• The last one just prints the string - the back-ticked expression is never evaluated

Commonly used to set a variable

today=`date`echo "Today is $today"

echo "Text files: `ls *.txt`"

We can redirect stuff into a loop

while [ someCondition ] do
#do things here
done

Special values

• $$ the process ID of the shell (but not a subshell)

• $? the exit status of the most recently completed (and waited upon) command

• $# number of arguments passed to this shell ($1 and up, since $0 is the name of the script)

• "$@" list of all of the arguments, where arguments that have spaces are preserved as one value

• $@ list of all of the arguments, where arguments that have spaces are further split into words

• "$*" similar "$@" but use $IFS (input field separator) as the delimiter instead of a space between words• result is a single token

Defining functions in bash

function
{# function body}
• or
()
{# function body}

Functions info in bash

Special parameters $1 through $9 and $@, $* now refer to the function arguments

Variables from parent scope are accessible within the function

When a sub-process (through fork(2)) is created, environment variables are copied into the new environment

BVAR and CVAR are environment variables

Other common environmental variables

• $USER - the login name of the current user
• $HOME - the home directory of the current user
• $PATH - the list of directories to search for programs
• $CWD - the current working directory
• $EDITOR - name of (and optionally full path to) your favourite editor (e.g.,"vi", "/usr/bin/vim", /usr/bin/atom)
• $LANG - name of your language setting character set if you are working solely in English, thisis typically "en_US.UTF-8"

the value of the shell symbol ~ also expands

~ is the home directory of the current user (i.e. whoever is running the shell)

~someUser is the home directory of someUser

Steps in creating a shell script

• place the code in a file
• ensure the shebang is set up correctly
• made the file executable: chmod a+x

Each shell has its own "resource" file that will be in your $HOME directory.

• For bash it is .bashrc (note the leading dot)
• For csh it is .cshrc
• These files are run in your interactive environment once before giving you the first prompt, soany commands or settings put here will take effect for all of your sessions.

.bash_profile

The shell that gets created when you log in, is called a "login shell" and it is controlled bythe .bash_profile file

.bashrc

Later interactive shells created within your login shell will inherit environment settings, but arecontrolled by the .bashrc file

.profile

is the profile of the sh (Bourne) shell

• IIRC, it was automatically created, since the default shell is sh, not bash

.bash_history

contains a history of commands you have entered in the terminal

• On Docker, it might be reset every time you start /stop the container - check

.bash_logout

is optional - contains the commands that execute when the shell terminates
• e.g the you log out

filter

A filter is a command that processes stdin and writes on stdout
• these allow us to build command pipelines
• they usually take an optional file list

sort

order data in various ways, potentially removing duplicates

grep

look for things within files.
• Comes from "go to regular expression and print."
• Available with "basic" and "extended" regular expression support via grep and egrep

tee

copy the contents that were piped into it two destinations:
• a file
• to standard output
• It's a "T splitter" - the input gets sent to two destinations

echo

copy the command line arguments to standard output

Useful filters: selecting and combining content

• Separate by lines:
• head print the first N lines of each file (default 10)
• tail print the last N lines of each file (default 10); also print the last all but N lines (i.e.; trim aheader)
• cat concatenate the contents of all files together

• Separate by columns:
• cut by delimited column or character position
• paste combine input as columns of data, concatenating lines together
• join perform an "equality join" on the specifier files, combining lines

cmp

compare two files to see if they are the same

diff

print the differences between two files

patch

apply a set of editor commands to update a file based on differences

sed: stream editor

• edit a stream programmatically!
• anchors - provide ranges for command action:
• line number : absolute coordinates, beginning at 1
• /pattern/: match text• if none given, whole file is implied
• editing commands:• s: search and replace
• a, i, c: add, insert or change data
• d: delete lines
• p: print line after edit (used with -n, which suppresses default output)

big-endian

system stores the mostsignificant byte of a word at the smallest memory address and the least significant byte at the largest

little-endian

system stores the least significant byte of a word at the smallest memory address and the most significant byte at the largest

Intel 32-bit and 64-bit architectures

are little-endian

Older Motorola 68* series and PowerPC architectures

big-endian

More commonly used to densely load bytes with flags or small integer values

• e.g. the status value in pid_t wait(int *status);
• It is a 32-bit int, where different bits have different meaning:
• 24 bits are used to store several pieces of information about how the process terminated
• 8 bits store the returned status - i.e. the value from the child's return or exit() statement

| :

bitwise OR - for each bit in the output number, the result bit is 1 if any of the bits are 1 inthe input

0100 0101 | 1100 0000 = 1100 0101
0100 0101 | 0000 1111 = 0100 1111

& :

bitwise AND - for each bit in the output number, the result bit is 1 only if both bits are 1 inthe input

0100 0101 & 1100 0000 = 0100 0000
0100 0101 & 0000 1111 = 0000 0101

^ :

bitwise XOR - for each bit in the output number, the result bit 1 if the two bits in the input are different

0100 0101 ^ 1100 0000 = 1000 0101
0100 0101 ^ 0000 1111 = 0100 1010

~ :

bitwise NOT - inverts every bit of the inout number
~ 0100 0101 = 1011 1010
~ 0000 1111 = 1111 0000

<< :

left shift - shifts the bits of the first operand to the left by n bits, where n is the second operand
• 0100 0101 << 2 = 0001 0100
• 1100 0000 << 2 = 0000 0000

>> :

right shift - shifts the bits of the first operand to the right by n bits, where n is the secondoperand
• 0100 0101 >> 2 = 0001 0001
• 1100 0000 >> 2 = 0011 0000

header file defined flags combined with bitwise OR

reading and writing: 00000002

• create and truncate to zero bytes if file exists: 00000100, 00001000

• do not follow symlinks (no filesystem level redirection): 00400000

• close this file descriptor if this process calls exec: 02000000

Colour pixel data is often stored in 32-bit ints - RGBA format

• RRRRRRRR GGGGGGGG BBBBBBBB AAAAAAAA
• 8 bits for each colour channel
• 8 bits for Alpha, or transparency

If we don't need the alpha, we can pack more colour information into a 32-bit int:R10G10B10

• RRRRRRRRRR GGGGGGGGGG BBBBBBBBBB UU
• Where we have 10 bits per colour and 2 unused bits

Vararg functions must be declared with at least one argument, followed by 3 dots

int func(type arg1, ...) // n = last specified parameter

Inside the function, parameters can be accessed by the C library macros

• void va_start(va_list ap, last); // start param list from lastnamed parameter

• type va_arg(va_list ap, type); // type = next parameter type

• va_end(va_list ap); // clear parameter list

Based on the known type of at least one initial argument

va_arg lets you pull off the nextargument as an iterator based on the type of the argument

Using vararg functions So how does printf() do this?

• It has a wide mix of types! the first argument to printf() provides the list of all of the remainingargument types!

• provided an earlier argument indicates the type of a later one, this works - because the C stackis ordered left to right

• modern linters have special handling for printf() to try and make sure that they types match —but this won't be true for other vararg functions

strconcat prototype

• Needs to describe how the argument list starts:
char * strconcat(const char *, ...);

• We also need to specify the end of the argument list

• In out function, we pass a NULL pointer as the last argument

• Please note that the term "list" is a bit of a misnomer - the varargs are actually a contiguousblock of memory with no intrinsic terminator

• We need to pass the type to va_arg, so it knows how to advance its next element pointer

• The implementation details are compiler-specific, however

Inner workings of vararg functions

• Each function has its own stack frame

• In order to call a function, the parent sets aside memory for the parameters for the child call

• Each function sets aside memory for its own local variables

• Parameter and local variables can be found relative to the frame pointer (FP)

• The previous frame pointer and program counter are stored as part of the frame

Varargs vs unspecified number of parameters

• C allows the following syntax:type funcName()

• In this case, funcName() has an unspecified number of arguments

• The compiler will not check the number or type of arguments

• However, the arguments of arguments if not variable - it's fixed

• This is not equivalent to a variable number of arguments

• This is old, pre-ANSI style of writing C

• It prevents the compiler from doing proper type/signature checking and can lead to undefinedbehaviour

• I would recommend again it