Software Engineering Testing Notes

Software Engineering Testing: Mutation Basics

Learning from Mistakes

• The key idea is to learn from earlier mistakes to prevent them from happening again.
• A key technique involves simulating earlier mistakes to see whether the resulting defects are found.
• This is known as fault-based testing or mutation testing.

Seeding Defects

• Defects are seeded into the program, generating a "mutant," which is a mutation of the original program.
• The test suite is run to see whether it detects the defects (referred to as "killing the mutants").
• A mutant that is not killed indicates a weakness of the test suite.
• It's possible for a mutant to be 100% equivalent to the original program.

Estimating #Defects

• Mutation testing helps estimate how many defects remain in the software.

Fish Tag Analogy

• A method for estimating the number of defects remaining in software is provided, using a fish-tagging analogy:

  • 1,000 fish are caught and tagged.
  • Later, a sample of 300 fish is caught, and it's found that 50 of them are tagged.
  • An estimate of the untagged fish population is calculated as follows:

  \frac{1000}{300} = \frac{50}{\text{untagged fish population}}

Applying to Programs

• We seed 1,000 mutations into the program.

• After running tests, we find that 50 mutants are not killed out of a sample of 300.

• The estimated remaining defects are:

  \frac{1000}{300} = \frac{50}{\text{remaining defects}}

Assumptions

• Mutations are representatives of earlier mistakes, according to the "competent programmer hypothesis."
• Failures often result from a combination of minor mistakes.
• However, there may be logical errors that cross-cut the program.
• It's important to note that these hypotheses are not proven.

Mutation Operators for Java

1. ABS –– Absolute Value Insertion:

• abs() returns the absolute value of the expression, while negAbs() returns the negative of the absolute value.
• failOnZero() tests whether the value of the expression is zero. If it is, the mutant is killed; otherwise, execution continues and the value of the expression is returned.
• This operator forces the tester to cause each numeric expression to have the value 0, a negative value, and a positive value.
• For example, the statement x = 3 * a; is mutated to create the following three statements:
  • x = 3 * abs(a);
  • x = 3 * -abs(a);
  • x = 3 * failOnZero(a);
• Each arithmetic expression (and subexpression) is modified by the functions abs(), negAbs(), and failOnZero().

2. AOR –– Arithmetic Operator Replacement:

• leftOp returns the left operand (the right is ignored), rightOp returns the right operand, and mod computes the remainder when the left operand is divided by the right.
• For example, the statement x = a + b; is mutated to create the following seven statements:
  • x = a - b;
  • x = a * b;
  • x = a / b;
  • x = a % b;
  • x = a;
  • x = b;
• Each occurrence of one of the arithmetic operators +, －, *, /, and % is replaced by each of the other operators. In addition, each is replaced by the special mutation operators leftOp, and rightOp.

4. COR –– Conditional Operator Replacement:

• leftOp returns the left operand (the right is ignored) and rightOp returns the right operand.
• falseOp always returns false, and trueOp always returns true.
• For example, the statement if (a && b) is mutated to create the following eight statements:
  • if (a || b)
  • if (a & b)
  • if (a | b)
  • if (a ^ b)
  • if (false)
  • if (true)
  • if (a)
  • if (b)
• Each occurrence of one of the logical operators (&&, ||, &, |, ^) is replaced by each of the other operators; in addition, each is replaced by falseOp, trueOp, leftOp, and rightOp.