Software Testing and V&V Study Guide

Concepts of Verification and Validation (V&V)

• Definition of V&V: V&V is a system engineering discipline that utilizes a rigorous methodology for evaluating and assessing the correctness and quality of software throughout the entire software life cycle.

• V&V vs. Quality Assurance (QA): While V&V and QA are not identical, they are complementary processes. V&V specifically focuses on:

  • Ensuring that project requirements are being met.

  • Ensuring the project is focused on the correct objectives.

  • Managing project risks.

• The Two Pillars of V&V:

  • Verification: Addresses the question, "Are we building the product right?" It ensures the software conforms to its specific technical specification.

  • Validation: Addresses the question, "Are we building the right product?" It ensures the software performs the functions that the user actually requires.

The V&V Process and Planning

• Applicability: The V&V process applies to every stage of the selected software development model.

• Objectives of V&V:

  • To discover defects within deliverables and software artifacts.

  • To assess the usability and usefulness of the system before it becomes operational.

  • To establish confidence that the software is fit for its intended use.

  • To assess the degree of stakeholder acceptance.

• Planning Guidelines:

  • Planning must start early in the development process.

  • A balance must be maintained between static and dynamic V&V approaches.

  • Formal standards and procedures for V&V should be established.

  • Inspection checklists must be created.

  • A Software Test Plan must be defined to establish standards for the testing process and allocate resources/time estimates.

Software Inspection (Static V&V)

• Definition: Software Inspection is a static technique concerned with analyzing the representation of the system (documents, diagrams, source code) to discover problems without executing the system. It may be supplemented by tool-based analysis.

• Effectiveness and Advantages:

  • Inspections are proven to discover more defects than testing, though they do not guarantee the software will be error-free.

  • A single inspection can reveal multiple different defects, whereas in testing, one error might mask another, requiring multiple executions for discovery.

  • Reviewers can reuse domain and programming knowledge to identify common error types.

  • Inspections check conformance with specifications (verification) but cannot confirm conformance with real customer requirements (validation).

  • Inspection and testing are complementary, not opposing, techniques.

• Program Inspection Details:

  • A formalized approach to document reviews intended explicitly for defect detection, not correction.

  • Commonly targeted defects include logical errors, code anomalies (e.g., uninitialized variables), and non-compliance with standards.

The Inspection Process and Roles

• The Six Stages of Inspection:

  1. Planning: Defining the scope and schedule.

  2. Overview: Presenting the system overview to the inspection team.

  3. Individual Preparation: Team members review code and documents in advance.

  4. Inspection Meeting: Discovered errors are noted during the session.

  5. Rework: Modifications are made by the author to repair discovered errors.

  6. Follow-up: Verifying the rework; re-inspection may be required.

• Specific Inspection Roles:

  • Author/Owner: The programmer or designer responsible for the product and for fixing discovered defects.

  • Inspector: Identifies errors, omissions, inconsistencies, and broader issues.

  • Reader: Presents the code or document during the meeting.

  • Scribe: Records the results and findings of the meeting.

  • Chairman/Moderator: Facilitates the meeting, manages the process, and reports results.

  • Chief Moderator: Responsible for overall process improvements, updating checklists, and developing standards.

Inspection Metrics and Checklists

• Inspection Checklist Examples:

  • Data Faults: Initialization of variables, naming of constants, array bounds (lower bound $0$ vs. $1$, upper bound $size$ vs. $size - 1$), and string delimiters.

  • Control Faults: Condition correctness in conditional statements, loop termination, statement bracketing, and case statement coverage.

  • Input/Output Faults: Utilization of input variables and assignment of values to output variables.

  • Interface Faults: Parameter count, type matching, parameter order, and shared memory structure consistency.

  • Storage Management Faults: Reassignment of links in structures, correct space allocation, and explicit de-allocation.

  • Exception Management Faults: Accounting for all possible error conditions.

• Inspection Rates and Costs:

  • Overview Rate: $500\,statements/hour$.

  • Individual Preparation Rate: $125\,source\,statements/hour$.

  • Inspection Rate: $90-125\,statements/hour$.

  • Economic Impact: Inspection is expensive; inspecting $500\,lines$ requires approximately $40\,man/hours$ of effort, costing about $£2800$ at UK rates.

Static Analysis (Automated)

• Static vs. Dynamic: Static analysis occurs without program execution. It can be manual (inspections) or automated (Automated Static Analysis - ASA).

• Benefits: ASA allows for analysis across all code paths throughout development. Dynamic analysis (testing) is limited by the line coverage of the specific test suite.

• Language Considerations: ASA is highly valuable for languages with weak typing (like $C$) where compilers miss errors. It is less cost-effective for languages with strong type checking (like $Java$).

• Stages of Static Analysis:

  • Control Flow Analysis: Checks for multiple exit/entry points in loops and unreachable code.

  • Data Use Analysis: Detects uninitialized variables, redundant writes, and unused declarations.

  • Interface Analysis: Checks consistency of routine declarations and their usage.

  • Information Flow Analysis: Highlights dependencies of output variables for further review.

  • Path Analysis: Identifies all executable paths through the program.

• Formal Methods: These involve mathematical specifications and detailed mathematical analysis to prove a program conforms to its specification. They represent the ultimate static verification technique.

Software Testing (Dynamic V&V)

• Definition: The process of analyzing a software item to detect differences between existing and required conditions (bugs) and to evaluate features.

• Views of Test Objects:

  • Black Box (Closed Box): Focuses on the functionality of the test object.

  • White Box (Clear Box): Focuses on the internal structure of the test object.

  • Static White-box: Methodical review of design or code without execution.

• Development Testing Types: Includes activities by the development team, often using programmer/tester pairs. The six types are unit, integration, function/system, acceptance, regression, and beta. White-box techniques are used for three of these.

Core Testing Levels and Strategies

• Unit Testing:

  • Testing of individual units or groups of related units. A unit is a component that cannot be further subdivided.

  • Important because it is harder to find the cause of failures once code is integrated.

  • Approximately $65\%$ of all bugs can be caught in unit testing.

  • Owner: The software engineer who wrote the code.

  • Timing: The Unit Test Plan (UTP) is prepared at the end of the design phase.

  • Methods: Includes code review, walkthroughs, and inspections.

• Integration Testing: Testing where software and/or hardware components are combined to evaluate their interaction.

• Regression Testing: Selective retesting to verify that modifications (bug fixes or updates) have not caused unintended side effects and that the system still complies with requirements. White-box test cases are often reused here.

Software Test Plan Components

1. Testing Process: Description of major phases (sub-systems, interfaces).

2. Requirements Traceability: Ensuring every requirement is individually tested.

3. Tested Items: Specification of products to be tested.

4. Testing Schedule: Resource allocation and timeline.

5. Test Recording Procedures: Systematic recording of results to allow for auditing.

6. Hardware and Software Requirements: Required tools and estimated hardware utilization.

7. Constraints: Anticipated issues like staff shortages.

8. System Tests: Definition of specific test cases derived from the requirements specification.

Designing and Identifying Test Cases

• Test Case Composition: Consists of a set of inputs, execution preconditions, and expected outcomes.

• Test Data: The specific inputs devised to test the system.

• Types of Test Cases:

  • Positive: Designed to show the program succeeds with valid input.

  • Negative: Designed to test robustness by providing invalid inputs/conditions to ensure appropriate error codes are generated.

• Identification Techniques:

  • Boundary Value Analysis (BVA): Tests values on, just above, and just below boundaries.

    • Example: For range $0$ to $100$, test values are $-1$, $0$, $1$, $99$, $100$, and $101$.

  • Equivalence Partitioning: Dividing inputs into classes where any member of the class is representative.

    • Example: For range $1$ to $999$, valid class is $1 \le items \le 999$, invalid classes are items < 1 and items > 999. Values should not be boundary values.

  • Logic Coverage: Aiming for specific coverage criteria: every statement, every branch, or every path executed at least once.

  • Random Generation: Automated generation of test inputs.

System Testing and Performance

• Functional Testing: Tests the implementation of business needs.

• Performance Testing (Non-functional):

  • Load Testing: Testing with many concurrent users.

  • Endurance Testing: Long-duration testing for reliability.

  • Stress Testing: Finding the breaking point/maximum number of users.

  • Spike Testing: Sudden, short-duration stress application.

The V-Model

• Hierarchy of Testing Levels:

  1. Component Test (Unit): Checks features in the Component Design.

  2. Interface Test: Checks communication between components according to System Design.

  3. System Test: Evaluates the system as a whole ($one\,giant\,component$) against the System Specification.

  4. Acceptance Test: Validates the system against User Requirements.

  5. Release Test: Checks if the system works in the existing business/technical environment.

Defect Testing and Debugging

• Defect Testing: The goal is to discover defects and confirm their presence by causing anomalous behavior. A "successful" test is one that finds a bug.

• Debugging: A distinct process from testing. It involves locating and repairing errors by formulating and testing hypotheses about program behavior.

• Debugging Process Cycle:

  1. Locate error (using test results and specification).

  2. Design error repair.

  3. Repair error.

  4. Retest program using test cases.