Software testing

Software Testing

inductive inference to validate software

software executed on a sample of inputs - test set or test suite

evaluated for conformance to requirements

failure

runtime deviation from required behavior

defect, fault, or bug

a flaw in a program that can cause it to fail

error

human action that results in a defect or fault

correctness

property that program absolutely satisfied functional requirements

reliabilty

numerical measure of extent to which behavior of deployed software conforms to requirements (frequency of failures, mean time between failures)

Synthetic testing

alpha testing - tester or tool generates test data

operational/field testing

software tested in field (or with inputs captured in field)

prerequisite for measuring reliabilty

simulation testing

when field testing isn’t feasible

approx to field testing

requires probabilistic simulation model

goals of testing

reliability assessment

reliability improvement

reliability assessment

important for release/deployment decisions

often done based on inhouse testing, should be field

reliability improvement

defect identification and repair

limitation of testing

cannot generally determine program correctness or incorrectness

cannot reveal all defects

Halting problem

given a program and input, will program terminate?

Alternatives to testing

inspection

static program analysis

proofs

reliability estimation

static program analysis

effective for finding violations of implicit programming rules

proof of correctness

automatic theorem proving

finite state model checking  - checks properties weaker than correctness, intensive 

testing costs

analysis of specification and program

test data creation/generation

program execution

evaluation of program behavior

all manual effort

test oracle

means of determining if test outcome is correct

humans check test output or determine expected results manually

partial correctness checks can often be automated

drivers and stubs

simple test scaffolding used to isolate the component under test

stub

lightweight called collaborator that the unit under test depends on

returns minimal behavior so the test can run without real dependency

driver

lightweight caller that supplies inputs and collects/asserts outputs

testing phases

unit testing, integration and subsystem testing, system testing, field testing, acceptance testing, regression testing

unit testing

routine, module, or class tested

drivers and stubs often needed

integration and subsystem testing

units integrated into subsystem

interactions between units tested

subsystem tested as a whole

drivers and stubs often needed

System testing

subsystems integrated

entire system tested

field testing

system used in the field by ordinary users

users report problems

acceptance testing

tested by customer

basis for accepting software

regression testing

retesting after maintenance

Alpha testing

performed by internal testers

beta testing

performed by customers or end users

dogfooding

in house beta testing on own employees

canary releases

route small percentages of real users to new version initially

dark launching

deploy a feature but hide it for internal testing

blue green deployment

blue- live

green - new

deploy to green and run tests, then switch to green, issues go to blue

when to test principle

test each component as soon as it becomes possible

why test as soon as possible

dev has motivation and understanding

cost of repair is least - less dependencies

Synthetic testing goal

create and run artificial test cases likely to reveal defects

sources of info used for syntehtic testing

srs

code

knowledge of programming errors

knowledge of app domain

testing criteria

conditions that must be satisfied by test set _each req has a test, statement coverage)

Types of synthetic testing

functional, structural, boundary and special values, interaction testing, model based testing, fault based testing, equivalence partition and subdomain testing, random testing, hybrid techniques

Functional testing

Exercises each specific functional requirement at least once. Any requirement could be wrong

Subdomains

subset of all possible inputs

Equivalence partition and subdomain testing

techniques for economizing on test effort

divide input domain into finite number of subdomains

disjoint subdomains can be viewed as equivalence classes (partition of input domain)

category partition testing

decompose functional specification into functional units that can be tested independently

determine general categories of input params and environmental variables

partition each category into separate choices

determine constraints among choices of different categories

write the test specification

use a generator to produce test frames from the test specification

from test frame, create test case

structural testing

covers each program element of a certain type

statements or blocks

conditional branches

any element may be defective

percentage coverage achieved by test set is one measure of testing completenes

Control flow graph

depicts potential control flow between program statements or instructions

each simple statement and decision point is represented by a vertex

each potential branch is represented by a directed edge

Modified condition/decision coverage

ensures each condition in a decision statement independently affects the decision’s outcome

each condition is tested twice, with results true and false, while keeping the values of all the other conditions in the decision the same

satisfy MC/DC

each possible outcome of each decision occurs

each possible outcome of each simple condition in a decision occurs

each entry and exit point is invoked

each simple condition in a decision is shown to independently affect the outcome of the decision

criticism of MC/DC

difficult and expensive

not directly related to safety reqs

boundary and special vals testing

testing with these

used in conjunction with other tests

loops - 0,1,small,large

load testing

stress testing

interaction testing

involves testing interactions between vars, objects, events, statements, components, or features

reveals faults coverage doesn’t

feature interaction occurs when the integration of two features can modify the behavior of one or both features

problem - number of possible interactions grows rapidly with number of elements involved

combinatorial interaction testing

black box testing technique that samples input params and configures and combines them systematically

main goal is to ensure that software can handle different combinations of parameter values and configuration options

seeks to cover a broad range of input combinations using a min number of test cases

testing all t-way interactions

doesn’t need to make sense

Data flow and dependence testing

find evidence of potential causal interactions between program elements

graph thing

Mock object

dummy object that mimics real object’s behavior

simpler than object

when to use mock

real object DNE

has nondeterministic behavior

difficult to set up

hard to trigger

slow

is a ui

short commings of mocks

hides integration problems

adds clutter

may be fragile

model-based testing

any type of testing guided by model

state machine

data flow

UML activity

models abstract things

Fault based testing

reveals a specific type of programming fault

Mutation testing

small changes are injected into the program

tests are run

mutant is killed if it produces different output than original

qaulity = percentage killed

Does not contain fault

test kills mutant, fails on mutant program, and passes on original

contains a fault that mutation partly fixes

kills mutant, fails on original, and passes on mutant

critism of mutant

large number of mutants

no effective way to automatically generate inputs that kill mutants

random (fuzz) testing

random tests

inexpensive and unbiased test generation

reveals many defects

fails good coverage

run for days

check for crashes