Software Quality Assurance (SQA) Notes
Software Quality Assurance (SQA)
Software, Software Quality, and Software Quality Assurance
Software Definition: Combination of computer programs (code), procedures, documentation, and data.
All four components are needed to assure quality during development and maintenance.
Software Quality (SQ)
Definition: Degree of conformance to:
Specific functional requirements.
Specified software quality standards.
Good Software Engineering Practices (GSEP).
Software Quality Assurance (SQA)
Definition: Systematic, planned actions to provide confidence that software development or maintenance:
Conforms to functional and technical requirements.
Meets managerial requirements (schedules, budget).
Software Errors, Faults, and Failures
Software Errors:
Incorrect code sections due to mistakes (grammatical, logical, etc.).
Made by systems analysts, programmers, or other team members.
Software Faults:
Software errors that cause incorrect functioning during specific application.
Typically found after software release.
Software Failures:
Occur when software faults are "activated" by a user.
Root cause: a software error.
Causes of Software Errors
Faulty requirements definition.
Client-developer communication failures.
Deliberate deviations from requirements.
Logical design errors.
Coding errors.
Non-compliance with documentation and coding instructions.
Shortcomings of the testing process.
Procedure errors.
Documentation errors.
Objectives of Software Quality Assurance Activities
Conformance to Requirements:
Assuring adherence to functional and technical requirements with acceptable confidence levels.
Conformance to Managerial Constraints:
Assuring adherence to scheduling and budget requirements with acceptable confidence levels.
Improvement and Efficiency:
Initiating and managing activities to improve software development and SQA efficiency.
Relationship between Software Quality Assurance and Software Engineering
Software Engineering:
Application of a systematic, disciplined, quantifiable approach to software development, operation, and maintenance.
Characteristics: Systematic, disciplined, and quantitative.
Cooperation between software engineers and SQA teams is vital for efficient and economic development while assuring quality.
Software Testing Techniques
Testing Objectives (Glen Myers)
Error Detection: Testing is the process of executing a program with the intent of finding errors.
Effective Test Cases: A good test case has a high probability of finding an as-yet undiscovered error.
Successful Test: A successful test uncovers an as-yet-undiscovered error.
Testing Principles (Davis)
Traceability: All tests should be traceable to customer requirements.
Most severe defects are those causing failure to meet requirements.
Early Planning: Tests planned long before testing begins.
Test planning can begin as soon as the requirements model is complete.
Detailed test cases can begin when the design model is solidified.
Incremental Testing: Testing begins "in the small" and progresses toward testing "in the large."
Initial tests focus on individual components, progressing to integrated clusters and the entire system.
Exhaustive Testing Impossibility: Exhaustive testing is not possible due to the large number of path permutations.
Adequate coverage of program logic and exercising all conditions in component-level design is possible.
Independent Testing: Testing should be conducted by an independent third party for maximum effectiveness.
The software engineer who created the system is not the best person to conduct all tests.
Test Case Design
Designing tests can be as challenging as the initial product design.
Methods provide a systematic approach to testing, ensuring completeness and error discovery.
Two main testing approaches:
Black-Box Testing: Tests based on the specified function of the product.
Verifies each function is fully operational and searches for errors in each function.
White-Box Testing: Tests based on the internal workings of the product.
Ensures internal operations are performed according to specifications and all internal components are adequately exercised.
Black-Box Testing
Conducted at the software interface.
Demonstrates software functions are operational.
Verifies proper input acceptance and correct output production.
Maintains integrity of external information (e.g., database).
Examines fundamental system aspects without regard for internal logical structure.
White-Box Testing
Predicated on close examination of procedural detail.
Tests logical paths through software by exercising specific conditions and/or loops.
Examines the "status of the program" at various points to determine if actual status corresponds to the expected or asserted status.
All logical paths are defined, test cases developed, and results evaluated.
Exhaustive Testing Limitations
The number of possible logical paths can be very large, even for small programs.
Example: A 100-line C program with two nested loops (1 to 20 iterations each) and four if-then-else constructs has approximately possible paths.
Assuming a magic test processor develops, executes, and evaluates a test case in one millisecond, it would take 3170 years to test this program exhaustively.
Exhaustive testing is impossible for large software systems.
White-Box Testing - Practical Application
A limited number of important logical paths can be selected and exercised.
Important data structures can be probed for validity.
Combining black- and white-box testing validates the software interface and ensures internal workings are correct.
WHITE-BOX TESTING
White-box testing, also called glass-box testing, is a test case design method that uses the control structure of the procedural design to derive test cases.
Using white-box testing methods, the software engineer can derive test cases that:
Guarantee that all independent paths within a module have been exercised at least once.
Exercise all logical decisions on their true and false sides.
Execute all loops at their boundaries and within their operational bounds, and
Exercise internal data structures to ensure their validity.
Loop Testing
Loops are fundamental to most algorithms implemented in software.
Loop testing is a white-box technique focusing on the validity of loop constructs.
Loop Classes:
Simple loops.
Concatenated loops.
Nested loops.
Unstructured loops.
*Why do loop testing ?It helps to identify loops initialization problems.
It helps to identify infinite loops.
Simple Loops Testing:
n = maximum number of allowable passes through the loop.
Skip the loop entirely.
Only one pass through the loop.
Two passes through the loop.
m passes through the loop where m < n.
n _1, n, n + 1 passes through the loop.
Nested Loops Testing:
Extending simple loop tests to nested loops increases the number of tests geometrically.
An approach to reduce the number of tests:
Start at the innermost loop, setting other loops to minimum values.
Conduct simple loop tests for the innermost loop while holding outer loops at their minimum iteration parameter values. Add tests for out-of-range/excluded values.
Work outward, testing the next loop, keeping outer loops at minimum values, and other nested loops to "typical" values.
Continue until all loops have been tested.
Concatenated Loops Testing:
If loops are independent, use the approach defined for simple loops.
If loops are not independent (e.g., loop counter of loop 1 is the initial value for loop 2), use the nested loops approach.
Unstructured Loops:
Should be redesigned to use structured programming constructs whenever possible.
Software Testing Strategies
Testing is planned in advance and conducted systematically.
A software testing template should be defined for the software process.
Generic characteristics of testing strategies:
Testing begins at the component level and works outward.
Different testing techniques are appropriate at different points.
Testing is conducted by the developer and (for large projects) an independent test group.
Testing and debugging are different activities, but debugging must be accommodated.
Verification and Validation (V&V)
Software testing is part of V&V.
Verification: Ensures software correctly implements a specific function.
"Are we building the product right?"
Validation: Ensures the software built is traceable to customer requirements.
"Are we building the right product?"
V&V encompasses many Software Quality Assurance (SQA) activities.
A Software Testing Strategy
The software engineering process viewed as a spiral.
System engineering defines the role of software.
Software requirements analysis establishes:
Information domain.
Function.
Behavior.
Performance.
Constraints.
Validation criteria.
Moving inward on the spiral leads to design and coding.
Software development spirals inward, decreasing abstraction on each turn.
A testing strategy is also viewed in the context of the spiral.
Unit Testing: Begins at the vortex, focusing on each software component/module.
Integration Testing: Moves outward, focusing on design and software architecture construction.
Validation Testing: Validates requirements against the constructed software.
System Testing: Tests the software and other system elements as a whole.
Unit Testing
Focuses verification effort on the smallest unit of software design.
Uses component-level design as a guide to test important control paths for errors.
White-box oriented and conducted in parallel for multiple components.
Unit Test Considerations
Testing unit interfaces to ensure proper information flow.
Examining local data structures to ensure integrity during algorithm execution.
Testing boundary conditions to ensure proper operation at limits.
Exercising all independent paths (basis paths) through the control structure.
Testing all error handling paths.
Basis path and loop testing are effective techniques for uncovering path errors.
Unit Test Procedures
Normally an adjunct to the coding step.
After code development, review, and verification, unit test case design begins.
Requires driver and/or stub software.
Driver: "Main program" that accepts test case data, passes it to the component, and prints results.
Stubs: Replace subordinate modules, use their interfaces, may do minimal data manipulation, print verification of entry, and return control.
Drivers and stubs are overhead (not delivered with the final product).
Unit testing is simplified with high cohesion components.
When a component addresses only one function, the number of test cases is reduced, and errors are more predictable.
Integration Testing
Addresses the problem of "putting the modules together" after unit testing.
Problems:
Data loss across interfaces.
Adverse effects of one module on another.
Subfunctions not producing the desired major function when combined.
Unacceptable magnification of imprecision.
Problems with global data structures.
Integration testing is a systematic technique for constructing program structure while uncovering interfacing errors.
Objective: build a program structure dictated by design.
Nonincremental Integration (Big Bang Approach):
All components are combined in advance.
The entire program is tested as a whole.
Results in chaos, difficult error isolation, and endless error correction loops.
Incremental Integration:
Construct and test the program in small increments.
Errors are easier to isolate and correct.
Top-Down Integration
An incremental approach to program structure construction.
Modules are integrated downward through the control hierarchy, starting with the main control module.
Integration can be depth-first or breadth-first.
Depth-First: Integrates all components on a major control path.
Breadth-First: Integrates all components directly subordinate at each level.
Integration Process:
Use main control module as a test driver, with stubs for subordinate components.
Replace stubs one at a time with actual components (depth- or breadth-first).
Conduct tests as each component is integrated.
Replace another stub with a real component.
Conduct regression testing to ensure that new errors have not been introduced.
Continue from step 2 until the entire program structure is built.
Bottom-Up Integration
Begins construction and testing with atomic modules (lowest levels).
Components are integrated from the bottom up.
Processing for subordinate components is always available, eliminating the need for stubs.
Implementation Steps:
Combine low-level components into clusters (builds) that perform a specific sub-function.
Write a driver (control program for testing) to coordinate test case input and output.
Test the cluster.
Remove drivers and combine clusters moving upward in the program structure.
As integration moves upward, the need for separate test drivers lessens.
*If the top two levels of program structure are integrated top down, the number of drivers can be reduced substantially and integration of clusters is greatly simplified.
Comments on Integration Testing
Advantages of one strategy (top-down or bottom-up) tend to result in disadvantages for the other.
Top-Down Disadvantage: Need for stubs and associated testing difficulties.
Bottom-Up Disadvantage: The program as an entity does not exist until the last module is added.
Selection depends on software characteristics and project schedule.
A combined approach (sandwich testing) using top-down for upper levels and bottom-up for subordinate levels may be best.
SOFTWARE QUALITY ASSURANCE
*Goal of quality assurance is to provide management with the data necessary to be informed about product quality, thereby gaining insight and confidence that product.
Quality Assurance:
A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements.
A set of activities designed to evaluate the process by which the products are developed or manufactured. Contrast with: quality control.
SQA Activities
Involve software engineers and an SQA group.
SQA group responsibilities:
Planning.
Oversight.
Record keeping.
Analysis.
Reporting.
If the data provided through quality assurance identify problems, it is management’s responsibility to address the problems and apply the necessary resources to resolve quality issues. *SQA activities are performed by an independent SQA group that:
Prepares an SQA plan for a project.
Developed during project planning and reviewed by all parties.
Identifies:
Evaluations to be performed.
Audits and reviews to be performed.
Applicable standards.
Error reporting and tracking procedures.
Documents to be produced by the SQA group.
Amount of feedback provided to the team.
Participates in the development of the project’s software process description.
Reviews the process for compliance with organizational policy, software standards, and other standards.
Reviews software engineering activities to verify compliance with the defined software process.
Identifies, documents, and tracks deviations from the process and verifies that corrections have been made.
Audits designated software work products to verify compliance with the software process.
Reviews selected work products; identifies, documents, and tracks deviations; verifies corrections; and reports results to the project manager.
Ensures that deviations are documented and handled according to procedure.
Deviations may occur in the project plan, process description, standards, or work products.
Records noncompliance and reports to senior management.
Noncompliance items are tracked until resolved.
Software Reviews
Act as a "filter" for the software engineering process.
Uncover errors and defects during software development.
"Purify" analysis, design, and coding activities.
Include informal meetings and formal presentations.
FORMAL TECHNICAL REVIEWS (FTR)
A software quality assurance activity by software engineers and others.
Objectives:
Uncover errors in function, logic, or implementation.
Verify that software meets its requirements.
Ensure software is represented according to standards.
Achieve uniform software development.
Improve project manageability.
INFORMAL TECHNICAL REVIEWS (ITR)
*ITRs allows more casual approach in which one or more reviewers help the software engineer improve his work products. Typically, these meeting do not follow any particular plan and no formal minutes are recorded. pair programming is example on the review oriented aspects.
Software errors, Software fault, Software failure
*The goal of software quality control is to remove quality problems in the software.
These problems are referred to by various names— errors, or defects, bugs, faults.
an error (a quality problem found before the software is released to end users) and
a defect or fault (a quality problem found only after the software has been released to end users).
As software engineers, we want to find and correct as many errors as possible before the customer and/or end user encounter them. We want to avoid defects—because defects (justifiably) make software people look bad.
Defect Amplification and Removal
A defect amplification model illustrates error generation and detection during development steps (preliminary design, detail design, and coding).
*Errors passed through from previous steps are amplified (amplification factor, x) by current work. The box subdivisions represent each of these characteristics and the percent of efficiency for detecting errors, a function of the thoroughness of the review.The model includes:
Software development steps.
Error generation during each step.
Review effectiveness in uncovering errors.
Error amplification by current work.
*First Figure below illustrates a hypothetical example of defect amplification for a software development process in which no reviews are conducted. Referring to the figure, each test step is assumed to uncover and correct 50 percent of all incoming errors without introducing any new errors. Ten preliminary design defects are amplified to 94 errors before testing commences. Twelve latent errors are released to the field.
*Second Figure considers the same conditions except that design and code reviews are conducted as part of each development step. In this case, ten initial preliminary design errors are amplified to 24 errors before testing commences and only three latent errors exist.
Defect Amplification Example:
Without reviews, 10 preliminary design defects amplify to 94 errors before testing, releasing 12 latent errors.
With design and code reviews, 10 initial defects amplify to 24 errors, releasing only 3 latent errors.
*To conduct reviews, you must expend time and effort, and your development organization must spend money. However, the results of the preceding example leave little doubt that you can pay now or pay much more later.
Review Metrics
Technical reviews are part of good software engineering practice.
Total Review Effort:
where:
= Preparation effort (person-hours).
= Assessment effort (person-hours).
= Rework effort (person-hours).
Total Errors Discovered:
where:
= Minor errors found.
= Major errors found.
Error Density:
where:
WPS = Work product size (e.g., UML models, document pages, lines of code).
*Requirements model is reviewed to uncover errors, inconsistencies, and omissions, it would be possible to compute the error density in a number of different ways. The requirements model contains 18 UML diagrams as part of 32 overall pages of descriptive materials. The review uncovers 18 minor errors and 4 major errors.
1.2 errors per UML diagram or 0.68 errors per requirements model page
If reviews are conducted for a number of different types of work products (e.g., requirements model, design model, code, test cases), the percentage of errors uncovered for each review can be computed against the total number of errors found for all reviews.
*In addition, the error density for each work product can be computed.
*Once data are collected for many reviews conducted across many projects, average values for error density enable you to estimate the number of errors to be found in a new (as yet unreviewed document). For example, if the average error density for a requirements model is 0.6 errors per page, and a new requirement model is 32 pages long, a rough estimate suggests that your software team will find about 19 or 20 errors during the review of the document. If you find only 6 errors, you’ve done an extremely good job in developing the requirements model or your review approach was not thorough enough.Once testing has been conducted, it is possible to collect additional error data, including the effort required to find and correct errors uncovered during testing and the error density of the software. The costs associated with finding and correcting an error during testing can be compared to those for reviews.
THE SQA PLAN
Provides a road map for instituting software quality assurance. The plan serves as a template for SQA activities that are instituted for each software project.
A standard for SQA plans has been published by the IEEE . The standard recommends a structure that identifies:
(1) the purpose and scope of the plan,
(2) a description of all software engineering work products (e.g., models, documents, source code) that fall within the purview of SQA,
(3) all applicable standards and practices that are applied during the software process,
(4) SQA actions and tasks (including reviews and audits) and their placement throughout the software process,
(5) the tools and methods that support SQA actions and tasks,
(6) software configuration management procedures,
(7) methods for assembling, safeguarding, and maintaining all SQA- related records, and
(8) organizational roles and responsibilities relative to product quality.