411 Quiz (Slides 11-17)

Software Testing

Process of evaluating software to assess quality and find defects by executing it against expected requirements

Verification

Checking if the product is built correctly — "Am I building the product right?"

Validation

Checking if the correct product is built — "Am I building the right product?"

Defect

A flaw in any representation of software including requirements or code

Fault

A defect with the potential to cause a failure

Error

When an event activates a fault in a program

Failure

The manifested inability of a program to perform its required functions

Test Case

A pair of used to verify behavior

Test Oracle

A mechanism that verifies correctness of program outputs by comparing expected vs actual results

Exhaustive Testing

Testing all possible input combinations — practically impossible for most systems

Unit Testing

Testing individual program units like methods or procedures in isolation

Integration Testing

Testing modules assembled together to verify they work as a subsystem

System Testing

Testing the full system including functionality and load and performance

Acceptance Testing

Testing whether the system meets customer expectations (UAT and BAT)

White-Box Testing

Structural testing that examines source code focusing on control flow and data flow

Black-Box Testing

Functional testing applied at external interface level without knowledge of internal structure

Control Flow Graph (CFG)

A graphical representation of a program's control structure showing nodes and edges

Procedure Node

A node in a CFG with out-degree of 1 (assignment statements)

Decision Node

A node in a CFG with out-degree greater than 1 (conditional statements)

Program Path

A sequence of statements from entry to exit of a program

Feasible Path

A path for which there exists an input that causes it to execute

Infeasible Path

A path for which no input exists to execute it

Statement Coverage

Selecting paths so every statement is executed at least once

Branch Coverage

Selecting paths so every edge/branch in the CFG is traversed at least once

Simple Path Coverage

Every simple path (no repeated edges) is executed once

All Path Coverage

Every possible program path is executed at least once — requires infinite test cases with loops

Visit-Each-Loop Coverage

Testing each loop by skipping it entirely and by making one pass through it

Cyclomatic Complexity

McCabe's metric: C = edges minus nodes plus 2 and gives max number of linearly independent paths

Linearly Independent Paths

Paths where each has at least one edge not in any of the other paths

Boundary Value Testing

Testing at edges of input ranges using min and min+ and nom and max- and max values

Boundary Value Analysis Formula

n input variables yield 4n+1 test cases

Robustness Testing

Extends BVA by adding values slightly beyond min and max (min- and max+)

Worst-Case Testing

Rejects single fault assumption and takes Cartesian product of boundary values — 5^n test cases

Robust Worst-Case Testing

Combines robustness and worst-case — 7^n test cases

Equivalence Class Partitioning

Dividing input domain into subdomains where each class has similar behavior for testing

Decision Table Testing

Tabular format mapping combinations of conditions to actions for logic-based test derivation

All-Pairs Testing

Generating test cases that cover every pair of parameter values and finds 70–90% of faults

Exploratory Testing

Concurrent test design and execution guided by tester intuition using tours and time-boxes

Landmark Tour

Execute most important features in different orders

FedEx Tour

Focus on data — create and change and delete and verify it's saved and transported correctly

Anti-Social Tour

Intentionally provide invalid or unexpected inputs

Obsessive Compulsive Tour

Repeat steps many times especially data manipulation and error messages

After-Hours Tour

Test background tasks that run when the user isn't interacting

TOGOF Tour

Test One Get One Free — run multiple instances of the app simultaneously

Museum Tour

Test legacy code that hasn't been changed in a long time

Rained-Out Tour

Cancel every long-running operation to verify cancel functionality

Guidebook Tour

Use product documentation as a testing guide and verify accuracy

Supporting Actor Tour

Vary the specifics of test cases slightly to find new bugs

Basic System Tests

Verify the system can be installed and configured and brought to operational state

Functionality Tests

Comprehensive testing across full range of requirements

Robustness System Tests

Determine how well the system recovers from input errors and failures

Interoperability Tests

Verify the system can work with third-party products

Performance Tests

Measure throughput and response time and resource utilization under various conditions

Scalability Tests

Determine the system's scaling limits for users and geography and resources

Stress Tests

Push system beyond designed capacity to find limitations and failure behavior

Load and Stability Tests

Verify system remains stable for long periods under full load

Reliability Tests

Measure the system's ability to operate for a long time without failure

MTTF (Mean Time To Failure)

Average time between failures: sum of failure intervals divided by number of failures

MTTR (Mean Time To Repair)

Average time to repair a failure after it occurs

MTBF (Mean Time Between Failures)

MTTF plus MTTR

Regression Testing

Re-running existing test cases to verify no new defects were introduced by changes

Documentation Tests

Verifying user guides and help for accuracy and usability at read and hands-on and functional levels

STRIDE

Microsoft security threat model: Spoofing and Tampering and Repudiation and Information disclosure and Denial of service and Elevation of privilege

Spoofing

Illegally using another user's authentication information

Tampering

Malicious modification of data

Repudiation

User denies performing an action without proof to the contrary

Non-Repudiation

Ability of a system to prove that a user performed an action

Information Disclosure

Exposure of information to unauthorized individuals

Elevation of Privilege

Unprivileged user gains privileged access to compromise the system

Data Flow Testing

Testing based on how data values flow from definition to use through a program

Static Data Flow Testing

Analyzing source code without executing it to find data flow anomalies

Dynamic Data Flow Testing

Actual program execution to test data value paths

Data Flow Anomaly Type 1 (dd)

Variable defined and then defined again without being used

Data Flow Anomaly Type 2 (ur)

Variable referenced without being defined first

Data Flow Anomaly Type 3 (du)

Variable defined but never referenced

Beta Testing

Pre-release testing by external users in real-world conditions before public launch

Corrective Maintenance

Isolating and fixing defects in deployed software

Adaptive Maintenance

Modifying software to work with changes in its environment like new hardware or OS

Perfective Maintenance

Adding new features or enhancing existing functions

Defect Removal Efficiency (DRE)

Defects found in testing divided by (defects found in testing plus defects found after release)

Spoilage Metric

(Sum of defects times their PhAge) divided by total defects — measures how late defects are found

Defect Priority

How urgently a defect needs to be fixed (Critical and High and Medium and Low)

Defect Severity

How much the defect impacts the product's operation (Critical and High and Medium and Low)

Test Effort Estimation

Estimating number of test cases and execution time and can use function points^1.2

Function Points

Weighted sum of inputs and outputs and master files and inquiries used to estimate project size

Non-Functional Requirement (NFR)

A requirement about how the system behaves (quality attributes) rather than what it does

Quality Attribute Scenario

Six-part specification: source and stimulus and environment and artifact and response and response measure

Security (QA)

System's ability to protect data from unauthorized access while allowing authorized access

Confidentiality

Data and services are only available to those authorized to access them

Integrity (Security)

Data and services are protected from unauthorized modification

Availability (QA)

System is there and ready to carry out its task when needed and minimizes service outage

Modifiability

About the cost and risk of making changes to the system

Interoperability (QA)

Degree to which systems can usefully exchange meaningful information

Performance (QA)

Software system's ability to meet timing requirements for events and responses

Testability

Ease with which software can demonstrate its faults through testing

Usability

How easy it is for the user to accomplish a desired task and the support the system provides

Energy Efficiency

Managing software's energy consumption especially for mobile and IoT and cloud systems

Architectural Tactics

Primitive design techniques architects use to achieve a quality attribute response

Software Architecture

The set of structures needed to reason about the system — elements and relations and their properties

Module Structure

Architectural structure organizing code into modules with assigned responsibilities