Software Testing according to lan Sommerville - Tenth Edition Study Guide
Fundamental Principles of Software Testing
Definition and Purpose: Software testing involves executing a program using artificial data to demonstrate that the software performs as intended and to discover defects before the program is put into operational use.
Integrated Test Facility (ITF): Testing involves checking the results of a test run for errors, anomalies, or information regarding the program’s non-functional attributes.
Limitations of Testing: Testing can only reveal the presence of errors, never their absence. It is impossible to prove a program is 100% error-free through testing.
V&V Framework: Testing is a component of the broader Verification and Validation (V&V) process, which also encompasses static validation techniques.
Program Testing Goals
Requirement Demonstration: The first goal is to prove to both the developer and the customer that the software meets its specified requirements.
Custom Software: There must be at least one test case for every single requirement listed in the requirements document.
Generic Software Products: Testing must cover all system features and combinations of these features that will be included in the product release.
Defect Discovery: The second goal is to identify situations where the software behavior is incorrect, undesirable, or fails to conform to its specification.
Defect Testing: This is specifically concerned with rooting out undesirable behaviors such as system crashes, unwanted interactions with other systems, incorrect computations, and data corruption.
Validation and Defect Testing
Validation Testing: The goal is to show the system performs correctly. Test cases reflect the system’s expected use; a successful test is one where the system operates as intended.
Defect Testing: The goal is to expose defects. Test cases are designed to be deliberately obscure and do not necessarily reflect normal usage; a successful test is one that makes the system perform incorrectly, thereby exposing a fault.
Verification vs. Validation (V&V)
Verification: Defined by the question, ‘Are we building the product right?’ This ensures the software conforms to its technical specification.
Validation: Defined by the question, ‘Are we building the right product?’ This ensures the software does what the user actually requires.
Confidence Levels: The aim of V&V is to establish confidence that the system is ‘fit for purpose.’ This depends on:
Software Purpose: How critical the software is to the organization.
User Expectations: Users may have different standards for different types of software.
Marketing Environment: Pressure to get a product to market early may outweigh the need for exhaustive defect discovery.
Software Inspections vs. Software Testing
Software Inspections (Static Verification): These involve humans examining source representations (requirements, UML designs, database schemas, code) to discover anomalies and defects without executing the system.
Software Testing (Dynamic Verification): This involves exercising and observing product behavior by executing the system with test data.
Advantages of Inspections:
Error Masking: In testing, one error can hide another. In inspections, which are static, you do not need to worry about error interactions.
Incomplete Systems: Inspections can be performed on partial systems without the need for specialized test harnesses required for execution.
Quality Attributes: Inspections can look for broader issues like standards compliance, portability, and maintainability.
Relationship: Inspections and testing are complementary. Inspections check conformance with specifications but cannot check performance, usability, or the user’s real requirements (which requires dynamic execution).
Development Testing
Definition: All testing activities carried out by the team developing the system.
Stages of Development Testing:
Unit Testing: Testing individual program units (functions, methods, or object classes) in isolation. The focus is on the functionality of objects or methods.
Component Testing: Integrating several individual units to create composite components. The focus is on testing the component interfaces.
System Testing: Integrating components to create a full system or version. The focus is on component interactions and emergent behavior, ensuring components are compatible and transfer data correctly across interfaces.
Unit Testing Details
Object Class Testing: Complete coverage involves:
Testing all operations associated with an object.
Setting and interrogating all object attributes.
Exercising the object in all possible states.
Inheritance Challenge: Testing is harder because the information to be tested is not localized within a single class.
Automated Testing: Should be used whenever possible using frameworks like JUnit. Components include:
Setup: Initializing the system with inputs and expected outputs.
Call: Executing the object or method.
Assertion: Comparing the result with the expected result.
Choosing Test Cases:
Type 1: Reflects normal operation to show the component works as expected.
Type 2: Based on experience where common problems arise, using abnormal inputs to ensure the component doesn't crash.
Testing Strategies and Guidelines
Partition Testing: Identifying groups of inputs (equivalence partitions/domains) that have common characteristics and are processed the same way. Tests should be chosen from each partition.
Guideline-based Testing: Using experience to choose tests.
Sequence Guidelines: Test with sequences of size one; use different sizes; test first, middle, and last elements; test with zero-length sequences.
General Guidelines: Force all error messages; cause buffer overflows; repeat the same input numerous times; force invalid outputs; force computations to be too large or too small.
Test-Driven Development (TDD)
Process: An approach where testing and code development are interleaved. Tests are written before the code.
Cycle:
Identify new functionality (normally small).
Write an automated test for that functionality.
Run the test (it fails because the code doesn't exist).
Implement the functionality and refactor.
Run all tests. If they pass, move to the next increment.
Benefits:
Code Coverage: Every segment of code has at least one associated test.
Regression Testing: A suite of tests is built incrementally and can be run regularly.
Simplified Debugging: If a test fails, the error is likely in the code just written.
Documentation: Tests serve as documentation for what the code should do.
Regression Testing: Re-running previous tests to ensure that changes have not ‘broken’ existing functionality. Automated testing makes this inexpensive and straightforward.
Release Testing
Definition: Testing a version of the system intended for use outside the development team. It is a form of system testing but is usually performed by a separate team.
Goal: To convince the supplier that the system is good enough for use (validation testing).
Methodology: Usually black-box testing, derived solely from the specification.
Requirements-Based Testing: Validating specific requirements.
Example (Mentcare System): A requirement states if a patient has an allergy, prescribing that medication must trigger a warning.
Test Cases for Mentcare Requirement:
Test a patient with no allergies; ensure no warning.
Test a patient with one known allergy; ensure warning is issued.
Test a patient with multiple allergies; prescribe separately and check warnings.
Prescribe two drugs the patient is allergic to; check for two warnings.
Overrule a warning and check that the system requires a reason.
Scenario Testing and Performance Testing
Scenario Testing: Inventing a typical usage scenario to derive test cases.
George/Jim Mentcare Scenario: A nurse (George) uses the system to download patient records (Jim) to a laptop using a key phrase, checks side effects (sleeplessness) against a drugs database, updates the record, and later uploads it to the clinic database.
Features Tested by Scenario: Authentication, Upload/Download, Scheduling, Encryption/Decryption, Record retrieval/modification, Side-effect database links, Call prompting.
Performance Testing: Testing emergent properties by steadily increasing load until performance is unacceptable.
Stress Testing: A type of performance testing where the system is deliberately overloaded to observe its failure behavior.
User Testing
Definition: A stage where users or customers provide input and advice. Essential because environmental factors (reliability, usability) cannot be fully replicated in a lab.
Types of User Testing:
Alpha Testing: Users work with the development team at the developer’s site.
Beta Testing: A release is made available to users to experiment with in their own environment.
Acceptance Testing: Customers test the system to decide if it is ready for deployment. Primarily for custom systems.
Acceptance Testing Process Stages:
Define acceptance criteria.
Plan acceptance testing.
Derive acceptance tests.
Run acceptance tests.
Negotiate test results.
Reject or accept system.
Agile and Acceptance Testing: The customer is part of the development team; tests are defined by the user and run automatically. There is no separate process, though finding a ‘typical’ embedded user can be difficult.