Software Testing - 10

What is Testing in this context?

Testing is the process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs) and to evaluate the features of the software item.

Why is Software Testing important?

High-level goal: Ensure software behaves as intended

• Software developers are humans

  • We all make mistakes

  • These mistakes can lead to faults/bugs in software

• Testing helps us to identify and fix bugs!

  • Preventing a system from failure

  • Includes identification of security critical bugs (i.e., vulnerabilities)

Mistake

A human action that produces an incorrect result - synonym: Error

Fault

An imperfection or deficiency in a work product where it does not meet its requirements or specifications - Synonyms: Defect, Bug

Failure

An event in which a component or system does not perform a required function within specified limits.

What is the errors flow diagram?

What are the four levels of testing?

What is Unit Testing?

• Tests functionality of basic building blocks (e.g, single functions)

• Focuses on components in isolation

• May include stubs or drivers to execute test

What is integration testing?

• Tests functionality of building blocks when combined together

• Focuses on interaction between components

• Usually groups (unit tested) modules together following fixed strategies:

  • E.g, top-down, bottom-up, hybrid

What is system testing?

• Tests functionality of fully integrated system

• Focuses on interaction between all components and the correctness of system as a whole

• May be guided by requirement specifications

What is acceptance testing?

• Tests whether system fulfills acceptance criteria (e.g., requirement/specification)

• May involve (end-)users of the system

• Includes well-known testing types:

  • Alpha testing, beta testing

What is a test type?

A group of test activities based on specific test objectives aimed at specific characteristics of a component or system

• Focus on specific objectives and characteristics

  • While test levels focus on which parts of the system are tested

  • However: each level can also be seen as a test type

What are some example test types?

Regression tests, smoke tests, conformance tests, security tests

What are regression tests?

• Tests to ensure that introduced changes do not cause new defects

  • These defects would be “regressions”

• Most common method: re-run all tests after change

  • If not possible: selection of specific regression tests or prioritization of specific test cases

• Does not assess correctness of new parts of the software!

What are smoke tests?

• Non-exhaustive tests to assess main/critical functionality of the software

  • Usually early on in testing/development process

• Used to determine if further testing should proceed

  • If smoke test fails, additional tests make likely no sense!

• Examples:

  • Does the software start?

  • Can a user provide input?

What are conformance tests?

• Tests to ensure that software behaves according to their respective standard/specification

  • Test cases itself may be standardized as well

  • May be part of certification process

  • Especially relevant for safety-critical systems

• Example applications:

  • Compilers

  • Cellular Basebands

What are security tests?

• Specialized tests to find security critical defects in the target software

  • May be carried out by 3rd-parties, rather than development team

• Can focus on different types of security related bugs, for instance:

  • Logic bugs (e.g., authentication bypass, API misuse, etc.)

  • Configuration errors (e.g., use of weak ciphers, default credentials, etc)

  • Memory corruptions (e.g., stack-based buffer overflows, use-after-free, etc)

• Examples:

  • Source code audits, fuzz testing, symbolic execution

Why do we do Test Automation?

Automating test execution and result analysis is highly beneficial - Most languages provide test automation frameworks

CI/CD

Continuous integration and deployment

Commonly used in modern software engineering

- Core ideas:

• Merge changes frequently into a single mainline

• Before merge, ensure functionality via automated tests

Examples are Github actions and gitlab runners

What is The “Box Model”?

Testing can be classified based on the information provided to the test/tester:

• black-box

• grey-box

• white-box

What does the black-box mean?

- No information about the internal structure

- Tester observes input and output

What does the grey-box mean?

- Partial information about the internal structure

- Tester can create informed tests

What does the white-box mean?

- Full information about the internal structure

- Tester can leverage source code

What is static testing?

- Also called “off-line” testing

- Testing without execution

- Reasons about the code

- May create false-positives

What is dynamic testing?

- Also called “on-line” testing

- Testing by executing the target

- Observes the code’s behavior

- May not cover full program

What is test coverage?

Metrics to understand how complete given tests are

• Provides insights which parts of a software are tested

What types of test coverage are there?

• Requirement coverage (which of the requirements are tested?)

• Compatibility coverage (which hardware/software targets is the target tested with?)

• Code coverage (which parts of the code are tested?)

What is code coverage?

Useful metric for security testing

More code coverage == more parts of the system tested

What are the types of code coverage?

• Function coverage

  • Which parts of the software are executed?

• Line coverage

  • Which parts of the source code are executed?

• Block coverage

  • Which parts of the machine code are executed?

• Edge coverage

  • Which parts of the control flow graph are executed?

What are some limitations of code coverage?

• Writing complete tests is not trivial!

• Edge coverage most complete:

  • Function coverage may miss large parts of functionality

  • Line coverage assumes functionality isolated in individual lines

  • Block coverage does not track all transition

• Code coverage may not give insights about data-dependent faults: - E.g., consider off-by-one errors in loops or the arguments to a strcpy call

What is Static Code Analysis?

Most common scenario: source code analysis - However, techniques for static binary code analysis exist as well!

Scales well to full programs - But may report many false positives

What is an example of static code analysis?

• Example: Detecting vulnerable calls to APIs

  • Could be as simple as $ grep strcpy

  • Or using advanced tooling such as CodeQL [1]

What is Taint Analysis?

• Testing techniques which can be implemented both dynamically or statically

• Aims to identify flow of user input throughout the program

  • And whether it can has security implications

• Core idea:

  • Tagging user input and tracking (“tainting”) its use through the program

  • Check whether tainted values ever reach vulnerable functions (“sinks”) without prior sanitization

What is Fuzzing?

• Executing the target program repeatedly with (pseudo-)random input

  • Mutate existing or generate new input for each run

• Monitor program behaviour for crashes

  • Crashes demonstrate shortcomings (failures) of the target software

• Fuzzing has been shown very effective for finding security vulnerabilities

  • We will have a deep dive next lecture

• Scales well to full programs

  • But may miss substantial parts of the target

What is Symbolic Execution?

• Instead of testing/executing the target with concrete inputs “execute” it with symbolic values

  • Goal: “Exploring” all states of a program

• On each branch, collect constraints on inputs

  • If test case to reach a certain state is required, invoke a solver

What are the issues with symbolic execution?

- Scalability issues:

• State/Path explosion

• Constraint solving

How does symbolic execution work on this code?