Lab4 Entry Test Software Engineering

Unit testing

Testing level where

• individual code units (functions or classes or a single class)

• Return values of a single class can be checked.

• are tested in isolation.

Integration testing

Testing level that

• examines the interaction

• between the distinct components of the system.

System testing

Testing level that

• verifies the operation of the entire system.

Acceptance testing

Test performed

• by the end user or client

• to ensure the software meets business needs and requirements.

Regression testing

Re-running existing tests after a change ( bug-fix )

• to check that working parts still work.

Performance testing

Non-functional test type that

• checks speed

• checks response time

• checks behaviour under load.

Non-functional testing

Testing qualities such as

• performance, usability or security

• rather than what the system does.

Black-box testing

Test design technique:

• test cases are derived

• from the specification

• without looking at the code.

White box testing

It is a test design technique

Test cases are derived from

• the internal structure of the code;

• measured with coverage.

Model-based testing

Test design technique:

• Test cases are generated from a model of the system's behaviour

• (e.g. a state machine).

Test suite

The collection of test cases that are executed together.

SUT

System Under Test:

• the system which is being tested.

Test oracle

A method or principle that

• helps the tester decide whether the test was successful.

Test case

A set of defined

• input values

• execution conditions and expected results

• describing a single test for a specific goal.

Test verdict

The result of running a test:

• pass, fail, error or inconclusive.

Tester

The person who designs, runs and evaluates the tests.

Mocking

Replacing a real dependency of the unit under test

with a fake object that imitates it,

so the unit can be tested in isolation.

Test driven development (TDD)

An approach where the tests are written

• before the code

• the code is then written to

make the tests pass.

Test coverage

The proportion of testable items

• requirements

• statements

• decisions

that the tests have exercised.

Statement

A single instruction

• assigning a value to a variable: int x = 10;

Block

A sequence of statements executed together,

foo = bar / 2;

++i;

Condition

A single boolean expression inside a decision

e.g. bar > 0

Decision

The boolean expression that controls the flow

e.g. if (foo < 10 && bar > 0);

It may contain several conditions and may have more than two outcomes.

e.g. switch

Branch

One possible path out of

• a decision.

if (array[i] > max) {

max = array[i]; //

}

Statement coverage

Executed statements / total statements

(the number of tests is NOT in the formula).

Decision coverage

• Decision outcomes taken / total decision outcomes;

• JaCoCo calls it branch coverage.

100% statement coverage vs decision coverage

100% dec. coverage -→ 100% stat. coverage.

100% stat. coverage -/→ 100% dec. coverage.

100% statement coverage and branches

It guarantees coverage of

• ALL NON-EMPTY branches,

• but not of EMPTY branches.

Code coverage and requirements

It helps find

• untested code

It cannot find

• missing requirements.

because unimplemented requirements have no code.

Outcomes of an if (with or without else)

2 outcomes:

• true and false

Outcomes of a switch

One outcome per case

plus the default.

Test verdict when the test runs and every assert evaluates to true

Pass

Test verdict when an assert evaluates to false

Fail

Test verdict:

• the test cannot run properly

• (e.g. an unexpected exception or a broken test environment)

Error

Test verdict when it cannot be decided whether the test passed or failed

Inconclusive

Testing level: checking how several components of the system work together

Integration testing

Testing level: verifying the operation of the entire system

System testing

Testing level:

• the end user or client checks that the software meets the business needs

Acceptance testing

Test verdict vs. test oracle

One is the result of the test:

• pass or fail.

Other is the method

• used to decide the test.

Test suite

It is a collection of test cases that are executed together.

Where white-box vs. black-box test cases come from?

White-box testing derives test cases from

the internal structure of the code.

Black-box testing derives test cases from:

from the specification.

Coverage of a whole test suite

Take the UNION of everything covered by all tests divided by the total in the code;

• do not add the tests' fractions together.

Structural element:

i < array.size()

Condition:

• a logical expression without logical operators (and, or…).

Structural element:

if (array[i] > max)

{ max = array[i] } //

Branch:

• one possible outcome of a decision

• (the code entered after the if).

Is black-box testing only possible for integration testing?

Black-box test design is

• NOT limited to integration testing.

Which testing strives to cover

• every possible execution path?

Structure-based (white-box) testing, not black-box testing.

Is black-box testing just jargon for manually trying out the code?

• It is a real test design technique:

• Test cases are derived from the specification.

What is characteristic of black-box testing?

It is a specification-based technique.

Equivalence partitioning: how should invalid partitions be covered first?

In equivalence partitioning,

• invalid partitions should

• first be covered in separate tests, one at a time.

Equivalence partitioning: what happens if a test contains several invalid values?

If a test contains several invalid values,

they can mask each other's effect.

Why are invalid partitions tested one at a time?

Several invalid values in one test

• can mask (cancel out) each other's effect.

Should ,nvalid partitions first be covered in separate tests?

Invalid partitions should first be covered in separate tests,

• one at a time;

• because several invalid values can mask each other's effect.

When should verification and validation activities start?

As early as possible: early verification and validation activities

• reduces cost.

Do we test the software in agile development?

It includes

• testing.

Can specification-based and structure-based test design techniques be differentiated?

• Specification-based ( black-box testing )

• Structure-based test design techniques ( white-box testing )

can be differentiated.

What do we do during static analysis?

We examine the product without executing it (e.g. review)

• It is not about static variables.

Boundary value analysis

It examines

• the boundaries of data partitions (equivalence classes);

• it is NOT for increasing statement coverage.

How are equivalence classes defined?

Equivalence classes are

• selected using heuristics;

• they are not always defined automatically, algorithmically.

Use cases and test cases (true statement)

Test cases can be derived from use cases.

100% statement coverage and errors (true statement)

With 100% statement coverage

• you can NOT be sure that your code is error-free.

Is testing always a manual process? (true statement)

Testing is not always a manual process; tests can be automated.

100% statement coverage and system behaviours (true statement)

100% statement coverage does NOT mean that we have tested all possible behaviors of our system.

Typos and testing (true statement)

Testing can help to detect errors in software caused by typos.

Testing and correctness (true statement)

Testing can reveal errors,

• but it can NOT be used to verify (prove) the correctness of the software.

How many test inputs do we choose from each equivalence class?

One: each equivalence class is represented by one test input.

Is black box testing specification-based or structure-based?

Black box testing is a specification-based technique.

What is SUT the abbreviation of?

System Under Test

Do we check the main path (happy path)?

• A typical use-case test set has 1 test for the main path (happy path).

Tests from use cases: do we also test alternative paths that deviate from the main path?

• There are separate tests for each alternative path.

Do we test what happens when the prerequisites are not met?

We DO test generally what happens when the prerequisites are not met.

Tests from use cases: at what level are they typically?

At a higher level: system testing, acceptance testing, etc.

Which statement is FALSE for tests derived from use cases?

'We do not usually check the so-called happy path.' (The happy path is always tested.)

Main difference between specification-based and structure-based testing

It is where the test cases come from:

• the specification or the code's structure,

• not the testing framework.

Code coverage measurement

Code coverage measurement can be useful

• for identifying code sections that have not been tested yet.

Essence of equivalence classes

Inputs are split into classes expected

• to cover the same faults

• each class is represented by one test input

• not one class is represented for every test.

Does specification-based testing replace structure-based testing?

Specification-based testing does not replace structure-based testing;

• the two complement each other.