Effective Software Testing

What is Planning Poker?

Planning poker is a popular agile estimation technique. In a planning poker session, developers estimate the effort required to build a specific feature of the backlog. After the team discusses the feature, each developer gives an estimate: a number ranging from one to any number the team defines.

The developer with the smallest estimate and the developer with the highest estimate explain their points of view to the other members of the team. After more discussion, the planning poker repeats until the team members agree about how much effort the feature will take.

What is Test Driven Development (TDD)?

TDD basically means writing the tests before the implementation.

What are „good weather" tests?

Good weather tests test the valid behaviour of a function. That means, we want to find out, whether executing the function with valid input will generate the correct output.

What is meant by „bugs love boundaries"?

Most bugs happen on the boundary of cases. If a function is supposed to throw an exception when it recieves a list with one element as input, but return a correct result when a list with at least two elements is given as input, developers should make tests that test the behaviour of the function on this boundary.

What is example-based testing?

Example-based tests pick one specific input out of many possible inputs (e.g. specifically the input 2).

What is property-based testing?

Property-based tests assert general properties with automatically generated inputs.

What is domain testing?

In domain testing, the input is divided in to different domains. Each domain is expected to behave in the same way, which allows us to only write one test per domain.

What is structural testing (code coverage)?

Structural testing makes sure that every line of code is covered by at least one test.

What is integration testing?

Integration testing exercises the integration between a component in the system and an external component (e.g. a database).

What is system testing?

During system testing, the entire system is run, meaning frontend, backend, databases, etc., in order to see whether the system as a whole runs.

What is the testing pyramid?

What is the pesticide paradox?

The pesticide paradox describes the fact that using only one testing technique will not find every possible bug. Instead, developers should use multiple testing techniques.

What is the difference between validation and verification?

Validation: Make sure that we are designing the product such that it meets the design specifications.

Verification: Make sure that we are making the right product such that the user‘s needs are met.

What are on and off points?

In boundary testing, an on point is an input which is located on the boundary, while the off point is the point located closest to the boundary belonging to the partition the on point is not located on (e.g. if a program prints “hiphip” when an int is smaller than 10, and prints “hooray” when an int is greater or equal to 10, the on point would be 10, and the off point would be 9).

What are in and out points?

In boundary testing, in points are inputs that return an output from one partition (in the on and off point example, in points would be 10, 15, 23, etc.), and out points are inputs that return an output from the other partition (i.e. 9, 6, 4, etc.).

What is specification testing?

In specification testing, tests are designed according to a system’s specifications (user/system requirements, etc).

What is the seven-steps approach to derive systematic test cases based on a specification?

What are the steps in structural testing?

What is line coverage?

A developer who aims to achieve line coverage wants at least one test case that covers the line under test. It does not matter if that line contains a complex if statement full of conditions. If a test touches that line in any way, the developer can count the line as covered.

What is branch coverage?

Branch coverage takes into consideration the fact that branching instructions (ifs, fors, whiles, and so on) make the program behave in different ways, depending how the instruction is evaluated. For a simple if(a && b) statement, having a test case T1 that makes the if statement true and another test case T2 that makes the statement false is enough to consider the branch covered.

What is condition + branch coverage?

Condition + branch coverage considers not only possible branches but also each condition of each branch statement. E.g. the statement if (a && b) tests would need to cover the cases where each single statement evaluates to true or false and then also check whether the whole condition is true or false.

What is path coverage?

A developer aiming for path coverage covers all the possible paths of execution of the program. While ideally this is the strongest criterion, it is often impossible or too expensive to achieve. In a single program with three conditions, where each condition could be independently evaluated to true or false, we would have 2³ = 8 paths to cover. This would also apply to loops, so testers aiming for path coverage would have to write tests where the loop is executing once, twice, etc.

What is Modified condition/Decision coverage (MC/DC)?

The MC/DC criterion looks at combinations of conditions, like path coverage. However, instead of testing all possible combinations, we identify the important ones.

How many test cases are needed for 100% MC/DC coverage if conditions only have binary outcomes?

N + 1, where N is the number of conditions in the decision (e.g. if (A && (B || C)) would need 3 + 1 = 4 test cases).

What are coverage criteria subsumption relations?

What is mutation testing?

In mutiation testing, we purposefully insert a bug in a working program to see whether our test suite will find the bug or miss it. The buggy versions of the program are called mutants. If a test suite breaks (meaning, the tests successfully identified a bug), the mutant is “killed”. If a test suite doesn’t break (the bug is not found), the mutant survives. If a test suite manages to kill every possible mutant, the test suite has 100% mutation coverage.

What are pre- and post-conditions?

Pre-conditions are conditions that have to hold before a function is executed (e.g. the input parameter for a function has to be a positive value). Post-conditions are conditions that have to hold at the end of a program (e.g. the returned value has to be a positive number).

What are invariants?

Invariants are conditions that should hold before and after the execution of a function.

What happens if we change the pre- and post-conditions to be weaker and thus accept more values?

What happens if we change the pre- and post-conditions to be stronger and thus accept less values?

-Weaker pre-condition: The system works as expected, since existing dependencies don’t make use of the new values.

-Weaker post-condition: The system might break, since existing dependencies don’t expect some values to be returned.

-Stronger pre-condition: The system might break, since existing dependencies might be trying to pass values that were accepted in the old pre-condition but aren’t accepted anymore.

-Stronger post-condition: The system works as expected, since existing dependencies already handle the returned values correctly.

What is the difference between input validation and code contracts?

Input validation ensures that bad or invalid data that may come from users does not infiltrate our systems (e.g. if the user inputs a string instead of an int, an error message is shown to the user).

Code contracts ensure that communication between classes happens without a problem. We do not expect problems to occur — the data is already validated. However, if a violation occurs, the program halts, since something unexpected happened. The application also returns an error message to the user.

What are reasons why one would want to test classes without their dependencies and mock them instead?

Exercising the class under test together with its concrete dependencies might be too slow, too hard, or too much work.

What are advantages of using mock objects?

• We have more control (if a method is supposed to throw an error, we just throw it without needing to set up the error)

• Simulations are faster

• When used as a design technique, mocks enable developers to reflect on how classes should interact with each other, what their contracts should be, and the conceptual boundaries

What types of simulated objects exist?

-Dummy objects: These are passed to the class under test but never used (e.g. when a class Customer depends on other classes like Address, we can just create a dummy Address object and pass it to Customer).

-Fake objects: These have real working implementations of the class they simulate. However, they usually do the same task in a much simpler way. Imagine a fake database class that uses an array list instead of a real database.

-Stubs: These provide hard-coded answers to the calls performed during the test. Unlike fake objects, stubs do not have a working implementation. If the code calls a stubbed method getAllInvoices, the stub will return a hard-coded list of invoices.

-Mocks: These are similar to stubs, however, mocks go beyond that. They save all the interactions and allow you to make assertions afterward. For example, maybe we only want the getAllInvoices method to be called once. If the method is called twice by the class under test, this is a bug, and the test should fail.

-Spies: As the name suggests, spies spy on a dependency. They wrap themselves around the real object and observe its behavior. Strictly speaking, we are not simulating the object but rather recording all the interactions with the underlying object we are spying on.

What are mocking frameworks?

Mock frameworks offer a simple API, enabling developers to set up stubs and define expectations in mock objects with just a few lines of code. Mockito is the most popular framework in Java.

What is dependency injection?

With dependency injection, classes don’t instantiate their dependencies by themselves but instead receive them in the constructor. It allows us to inject mocks and also makes the production code more flexible.

What is the argument captor in Mockito?

The argument captor allows us to capture arguments we passed in mocks created with Mockito and assert that these arguments are in the correct form.

What is a disadvantage of mocks?

When should dependencies be mocked?

-Dependencies that are too slow: When functions are dealing with databases, webservices, etc., it is usually a good idea to mock these.

-Dependencies that communicate with external infrastructure: Communication with external infrastructure is usually very slow, which makes these cases fitting for mocks.

-Cases that are hard to simulate: A common example of this would be testing when a function is supposed to throw an exception.

When should dependencies not be mocked?

-Entities: Entities are classes that represent business concepts. If we have a ShoppingCart class, we also need to create instances of the Product class. Since we can easily create new instances, it would be more work to mock them.

-Native libraries and utility methods: Why should we mock ArrayList, or a call to String.format?

-Things that are simple enough: If you feel like a class is too simple to be mocked, it probably is.

How can we mock or stub dependencies like LocalDate?

There are two ways:

• Mockito allows developers to mock static methods

• Another solution is to encapsulate all the date and time logic into a class. In other words, we create a class called Clock, and this class handles these operations (e.g. instead of LocalDate.now(), we call Clock.now()) This allows us to stub the Clock class.

What is fuzzing?

Fuzzing is a software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program.

What types of fuzzers exist?

Generation-based: Fuzzers that generate random inputs to crash programs.

Mutation-based: Fuzzers that take existing "seeds" (valid inputs) and mutate them by deleting, inserting, or flipping random characters to create new test cases.

Grammar-based: Fuzzers that generate inputs based on a defined grammar or specification, used for structured data formats like HTML; XML; JSON; or phone numbers.

Blackbox fuzzers: A type of fuzzer that performs random mutations without feedback from the program's internal structure; often having a very low probability of reaching specific exceptions.

Greybox fuzzers: A type of fuzzer that leverages coverage information to guide the generation of inputs; aiming to explore more code paths and find bugs more efficiently.

Whitebox fuzzers: A type of fuzzer that has full visibility into the program's code and execution path (implied by the taxonomy alongside blackbox and greybox fuzzers).

AI-powered fuzzers: An advanced fuzzing topic that involves using Artificial Intelligence (AI) techniques such as training Neural Networks to predict coverage from inputs; learning grammars; setting up fuzzing harnesses; and learning effective mutations.

What is domain code and infrastructure code?

The domain is where the core of the system lies: that is, where all the business rules, logic, entities, services, and similar elements reside. Entities like Invoice and services such as ChristmasDiscount are examples of domain classes. Infrastructure relates to all code that handles an external dependency: for example, pieces of code that handle database queries (in this case, the database is an external dependency) or web service calls or file reads and writes. In our previous examples, all of our data access objects (DAOs) are part of the infrastructure code.

What is the most important advice when considering testability of a system?

Separate infrastructure code from domain code. In practice, when domain code and infrastructure code are mixed, the system becomes harder to test.

When is a class fully controllable and fully observable?

A class is fully controllable, if you can easily control what the class under test does.

A class is fully observable, if you can see what is going on with the class under test and inspect its outcome.

How can we improve the observability of classes?

We can provide getter methods, ensure dependencies can be injected by tests, etc. If, for example, we have a ShoppingCart class with a function markAsReadyForDelivery(), instead of checking whether this function was executed within our test case, we can provide a function isReadyForDelivery() in the ShoppingCart class, which returns whether the ShoppingCart instance is ready for delivery or not.

What are cohesive and non-cohesive classes?

Cohesive classes have a single responsibility with all of its functionality focused on fulfilling its responsibility. Cohesive classes usually need fewer test cases and are easier to test.

Non-cohesive classes have multiple responsibilities and thus have more test cases and are harder to test.

What is coupling?

Coupling describes how much one class is coupled to other classes. This could harm the evolution of the software, since changes in one class may propagate to other classes in ways that are not clear.

What are Singletons and what do they mean for testing?

Singletons are a design pattern, where only one instance of a class ever exists. If the program requests an instance of a singleton, the same one is always returned. This means, that we have to write test code which resets the singleton every time we test something. It is thus better to avoid this design pattern.

What is the red-green-refactor cycle?

1. We wrote a (unit) test for the next piece of functionality we wanted to implement. The test failed.Reflecting on our first TDD experience

2. We implemented the functionality. The test passed.

3. We refactored our production and test code.

When is it a good idea to use TDD, and when should we not use TDD?

TDD is good, when working on a complex problem, where we have little expertise and we need to learn on the fly.

On the other hand, when we need to solve a problem where we already have a lot of expertise and basically know the solution already, we can implement it directly.

What is the classicist school of TDD (Detroit school of TDD, inside-out TDD)?

Classicists go from the inside (entities and business rules) to the outside (interface with the user). They also try to avoid mocks and instead test the whole system at once, or completely isolate parts of the system.

What is the London school of TDD (outside-in TDD, mockist TDD)?

They prefers to start from the outside (such as the UI or the controller that handles the web service) and then slowly work toward the units that will handle the functionality.

What are principles of maintainable test code?

• Tests should be fast

• Tests should be cohesive, independent, and isolated

• Tests should have a reason to exist (not just to increase code coverage)

• Tests should be repeatable and not flaky

• Tests should have strong assertions

• Tests should break if the behaviour changes

• Tests should have a single and clear reason to fail

• Tests should be easy to write

• Tests should be easy to read

• Tests should be easy to change and evolve

What can we change if we have a slow test?

• Use mocks or stubs to replace slower components that are part of the test

• Redesign production code so that slower pieces of code can be tested separately from fast pieces of code

• Move slower tests to a different test suite that we can run less often

Note that slow test cannot always be avoided (e.g. SQL).

What are fat tests (eager tests)?

Fat tests exercise multiple functionalities and are often complex in terms of implementation. This makes understanding the tests at a glance and maintaining them difficult. Simpler tests, which exercise only one functionality, are much preferred.

What are reasons for tests to become flaky?

• If a test depends on external or shared resources (e.g. a database is not available at the moment the test is executed)

• Due to improper timeouts (e.g. if a test has to wait for a response from a web service, and it takes longer than usual)

• Because of a hidden interaction between different test methods (test A somehow influences test B, causing it to fail)

What are code smells and test smells?

The term code smell indicates symptoms that may indicate deeper problems in the system’s source code. Some well-known examples are Long Method, Long Class, and God Class. Research shows that code smells hinder comprehensibility and maintainability of software systems.

Test smells are the same, but for test code.

What are well known test smells?

Excessive Duplication: Much like duplicated code in production, duplicated test code should be avoided.

Unclear assertions: The test smell emerges, if it is hard to understand an assertion and the reason for its failure.

Bad handling of complex or external resources: A common smell is to be optimistic about the external resource. E.g. if we assume that a database is readily available when executing our tests. If it isn’t available, the tests fail without a clear reason. The test itself needs to set up the external resource in order to avoid this.

Fixtures that are too general: A fixture is the set of input values used to exercise the component under test.

Sensitive assertions: Assertions should not be oversensitive to internal changes.

What are five suggested guidelines by Tuya, Suárez-Cabal, and De La Riva for SQL testing?

What is the Liskov Substitution Principle (LSP)?

According to the Liskov Substitution Principle, a superclass should be substitutable by one of its subclasses without a change in program behaviour.

What is model checking?

Model checking is an automatic technique to perform exhaustive testing of finite-state models.

What does a linear-time model checking problem look like?

What is linear temporal logic (LTL) and what is it used for in the example of the microwave oven modeled as a finite state automaton (FSA)?

X: Next-operator (door implies next door, meaning the door is pushed again immediately in the next step)

◇: Finally-operator (door implies finally door, meaning the door is pushed again at some point after it was first pushed)

U: Until-operator (There is a series of cook events until the power is turned off)

☐: Always/Globally-operator (The power button is always not pressed (never))

What are the three algorithms used in model checking?

What are states?

A state captures the shared and local states of a concurrent program.

What is a complete state/transition model?

The complete state/transition model for the concurrent counter example explicitly shows all possible interleavings.

Why are software random number generators (RNGs) technically called pseudorandom?

Because the output of a deterministic program cannot truly be random.

What is a key tension in the requirements for an RNG that makes it subtle to test?

The output should be unpredictable from a user's perspective, but it is completely predictable from a programmatic perspective if the seed is known.

Why is it impossible to test if a single output value from an RNG is correct?

Because random values should be unpredictable and jiggle around

Why should tests for RNGs occasionally fail?

If a test never fails, it demonstrates a predictable attribute of the random number sequence, which is contrary to randomness.

What is a Uniform Random Number Generator (RNG)?

An RNG that produces values from a given interval (e.g., (0, 1)) with all values equally likely.

What is the Mersenne Twister?

A uniform RNG algorithm that has been extensively studied by experts and and is recommended for its good theoretical properties and empirical performance.

What is the grand strategy for generating random numbers?

1. Generate random values uniformly distributed in the interval (0, 1).

2. Transform those values into whatever other distribution you need.

What is a Nonuniform Random Number Generator?

An RNG that generates numbers following specific distributions, such as an exponential distribution.

How is the Central Limit Theorem applied to testing nonuniform RNGs?

It states that if enough values are averaged from any distribution, the average will approximate a normal distribution, allowing statistical tests on the average.

What is the Bucket Test (Chi-Square Test) for RNGs?

It involves dividing the range of values into buckets and checking if the observed number of samples in each bucket matches the expected number based on the distribution.

What is a shortcoming of the Bucket Test?

It might not detect issues with the distribution of samples *within* the buckets.

What is the Kolmogorov-Smirnov (K-S) Test for RNGs?

A fine-grained test that directly compares the empirical distribution function of the samples to the theoretical distribution function to find maximum differences.