Software Development Management: Software Quality Assurance and Testing Methodologies

Fundamentals of Quality and Quality Management

General Definitions of Quality:
- Oxford Dictionary Definition: Quality is defined as "the standard of something when it is compared to other things like it; how good or bad something is."
- Quality.org Definition of Quality Management: Quality management "is about making organisations perform for their stakeholders – from improving products, services, systems and processes to ensuring that the whole organisation is fit and effective."
- Core Purpose of Managing Quality: Constantly pursuing excellence by ensuring that organizational output is fit for purpose, remains fit for purpose over time, and continuously improves.
Software Quality Definition:
- Ian Sommerville Definition: Software quality is "concerned with ensuring that the required level of quality is achieved in a software product."
- Quality encompasses both functional suitability (conformance to functional requirements) and non-functional characteristics.
- Subjective Software Quality: Based primarily on non-functional characteristics reflecting real-world user experience. If software is excessively slow or unreliable, end-users cannot accomplish their objectives, regardless of functional completeness.
The High Cost of Poor Quality:
- Global Financial Impact: In 2017, poor software quality was estimated to have cost companies approximately $\$1.7\text{ trillion}$ , an amount equivalent to the total Gross Domestic Product (GDP) of Canada in that same year.
- Safety and Healthcare Critical Systems Failure: In 2016, a critical defect in software utilized by the National Health Service (NHS) in the UK miscalculated the heart attack risk levels for $300{,}000$ patients. As a consequence, patients classified as low risk were erroneously prescribed medications carrying severe side effects, whereas high-risk individuals were misinformed that they were at low risk.
Evaluating Software "Fitness for Purpose":
- Has the software system been properly and thoroughly tested?
- Is the software sufficiently dependable to be safely put into production use?
- Is the operational performance acceptable for normal day-to-day use?
- Is the user interface and interaction design usable?
- Is the underlying codebase well-structured, clean, and easily understandable?
- Have organizational programming standards and documentation guidelines been strictly observed throughout development?

ISO/IEC 25010 Software Product Quality Model

ISO 25010 Software Product Quality Model Breakdown

Overview of ISO/IEC 25010:
- Provides a standardized, hierarchical framework for evaluating software product quality through eight primary quality characteristics, each subdivided into specific sub-characteristics.
The Eight Quality Characteristics and Sub-Characteristics:
- Functional Suitability:
  - Appropriateness: The degree to which functions facilitate specified tasks and objectives.
  - Accuracy: The provision of correct or agreed-upon results with the needed precision.
  - Compliance: Adherence to standards, conventions, or regulations in laws and similar prescriptions relating to functional suitability.
- Reliability:
  - Availability: Operational readiness and accessibility when required for use.
  - Fault tolerance: Ability of the system to operate as intended despite the presence of hardware or software faults.
  - Recoverability: Capability to re-establish a specified level of performance and recover affected data in the event of a failure.
  - Compliance: Conformance to standards and regulations regarding reliability.
- Performance Efficiency:
  - Time-behaviour: Response times, processing times, and throughput rates when performing functions under stated conditions.
  - Resource-utilisation: Amounts and types of resources consumed (e.g., CPU, memory, storage, bandwidth) when performing functions.
  - Compliance: Adherence to performance efficiency standards.
- Operability (Usability):
  - Appropriateness-recognisability: Whether users can recognize whether the software is appropriate for their needs.
  - Learnability: How easily users can learn to use the system.
  - Ease-of-use: Operational control and ease of handling.
  - Helpfulness: System assistance and guidance provided during execution.
  - Attractiveness: Aesthetic satisfaction derived by the user.
  - Technical-accessibility: Usability for people with the widest range of capabilities (accessibility standards).
  - Compliance: Conformance to usability and accessibility regulations.
- Security:
  - Confidentiality: Ensuring that data is accessible only to those authorized to have access.
  - Integrity: Preventing unauthorized modification or deletion of computer programs or data.
  - Non-repudiation: Proving that an action or event took place so it cannot be denied later.
  - Accountability: Tracing actions of an entity uniquely to that entity.
  - Authenticity: Verifying the identity of a subject or resource.
  - Compliance: Adherence to security laws and industry policies.
- Compatibility:
  - Replaceability: Ability of a software component to replace another specified software product for the same purpose in the same environment.
  - Coexistence: Ability to share a common hardware/software environment without adverse impact on any component.
  - Interoperability: Degree to which two or more systems or components can exchange information and use the information that has been exchanged.
  - Compliance: Conformance to compatibility standards.
- Maintainability:
  - Modularity: The degree to which a system is composed of discrete components such that changes to one component have minimal impact on others.
  - Reusability: Asset usability in more than one computer program or software system.
  - Analyzability: Ease with which the impact of an intended change can be evaluated, or defects diagnosed.
  - Changeability: Capability of software to enable a specified modification to be implemented.
  - Modification stability: Resistance to unexpected secondary defects following code changes.
  - Testability: Ease with which modified or unchanged software components can be validated.
  - Compliance: Conformance to maintainability standards.
- Transferability (Portability):
  - Portability: Ease of transferring a system from one hardware, software, or operational environment to another.
  - Adaptability: Capability to be adapted for different specified environments without applying actions other than those provided for this purpose.
  - Installability: Ease with which software can be successfully installed and uninstalled in a specified environment.
  - Compliance: Conformance to transferability rules and specifications.
Quality Attribute Conflicts and Trade-Offs:
- A software system cannot be concurrently optimized for every single quality attribute.
- Conflict Example: Optimizing for extreme fault tolerance or robustness often involves redundant processing, assertion validation, and defensive checking, which causes a measurable reduction in performance efficiency.
- The Quality Plan: Project quality plans must establish an unambiguous prioritization of the most essential quality attributes for the specific domain, establishing a systematic quality assessment process to verify that target qualities are present.

Software Quality Management and Quality Assurance

Software Development Process vs Quality Management Process Timeline

Three Principal Concerns of Quality Management:
- Organizational Level: Establishing standard organizational processes, policies, and operational standards aimed at systematically producing high-quality deliverables.
- Project Execution Level: Implementing project-specific quality procedures and verifying that team members execute them rigorously.
- Project Planning Level: Formulating a dedicated project quality plan detailing quality goals, assessment gates, compliance criteria, and standards for deliverables.
Quality Management Activities & Operational Independence:
- Validates that development workflows run efficiently and deliverables (e.g., $D_1, D_2, D_3, D_4, D_5$ ) adhere to organizational standards.
- Generates quality review reports across development milestones based on predefined standards and procedures.
- Team Independence Requirement: The quality management and assurance team must remain organizationally independent of the core development team. This organizational boundary ensures an objective, unbiased assessment of software quality, preventing quality ratings from being skewed by development budget crises or deadline pressures.

Software Quality Assurance Elements and Framework

The Structure of Software Quality Assurance (SQA):
- Organizational Enablers: Process definition, quality audits, formal reviews, tool identification and selection, and continuous training activities.
- Core Technical Pillars:
  - Software Testing: Execution of software to validate functionality and detect bugs.
  - Quality Control: Operational techniques and inspections that ensure quality requirements are fulfilled.
  - Software Configuration Management (SCM): Tracking, controlling, and managing changes across the software baseline.
- Underpinning Criteria: Governed by institutional standards, standardized procedures, coding and naming conventions, and exact system specifications.

Anatomy of Bugs: Faults, Errors, and Failures

Formal Definitions and the Bug Chain:
- Fault: A static defect embedded in the software source code (the fundamental cause of a defect).
- Error: An incorrect internal execution state of the program resulting from the execution of a fault (an erroneous program state).
- Failure: An external, observable deviation in software behavior relative to specified requirements or user expectations (propagation of the erroneous state to output channels).
Root Causes of Software Defects:
- Programming Errors: Accidental implementation slips introduced directly into source code, such as classic off-by-one boundary errors ( $\text{off-by-1}$ ).
- Understanding Errors: Incomplete or incorrect comprehension of system domain requirements. For instance, designing a file creation date processing utility without accounting for diverse international date formats.
Worked Case Study: The numZero Function:
- Consider the following Java method designed to count occurrences of zero within an array: java public static int numZero (int[] arr) { // Effects: If arr is null throw NullPointerException // else return the number of occurrences of 0 in arr int count = 0; // L1 for (int i = 1; i < arr.length; i++) // L2 if (arr[i] == 0) // L3 count++; // L4 return count; // L5 }         
- Defect Identification:
  - Fault: The for-loop counter variable is initialized to index $1$ (int i = 1) rather than index $0$ (int i = 0). As a result, array element arr[0] is omitted from traversal.
  - Correction: Alter the initialization statement on line L2 to int i = 0;.
State Space Representation and Tracing:
- Program execution states can be represented using the formal vector notation: $\langle \text{var}_1 = v_1, \dots, \text{var}_n = v_n, \text{PC} = \text{program counter} \rangle$
- Execution Trace of Buggy Code with Input arr = {0, 4, 6}:
  - $\text{State } 1: \langle \text{arr}=\{0, 4, 6\}, \text{PC}=[\text{int count}=0\text{ (L1)}] \rangle$
  - $\text{State } 2: \langle \text{arr}=\{0, 4, 6\}, \text{count}=0, \text{PC}=[i=1\text{ (L2)}] \rangle$
  - $\text{State } 3: \langle \text{arr}=\{0, 4, 6\}, \text{count}=0, i=1, \text{PC}=[i < \text{arr.length}\text{ (L2)}] \rangle$
  - $\dots$
  - $\text{Final State}: \langle \text{arr}=\{0, 4, 6\}, \text{count}=0, \text{PC}=[\text{return count; (L5)}] \rangle$
- Execution Trace of Correct Code (i = 0) with Input arr = {0, 4, 6}:
  - $\text{State } 1: \langle \text{arr}=\{0, 4, 6\}, \text{PC}=[\text{int count}=0\text{ (L1)}] \rangle$
  - $\text{State } 2: \langle \text{arr}=\{0, 4, 6\}, \text{count}=0, \text{PC}=[i=0\text{ (L2)}] \rangle$
  - $\text{State } 3: \langle \text{arr}=\{0, 4, 6\}, \text{count}=0, i=0, \text{PC}=[i < \text{arr.length}\text{ (L2)}] \rangle$
- The Error State: Defined as the exact point where an execution state sequence diverges from the state sequence of the correct implementation. Divergence begins at Step 2 where $i=1$ instead of $i=0$ .
Test Case Execution Analysis and Failure Conditions:
- Test Case 1: Input arr = [4, 6, 0], Expected output = $1$ .
  - Fault Present: Yes (i = 1).
  - Error Manifested: Yes ( $i$ is $1$ instead of $0$ on the initial check; arr[0] is skipped).
  - Failure Observed: None. The skipped element is $4$ , which is non-zero. The loop inspects arr[1] ( $6$ ) and arr[2] ( $0$ ), increments count to $1$ , and outputs $1$ . Expected output equals actual output, meaning the error state does not propagate to output.
- Test Case 2: Input arr = [0, 4, 6], Expected output = $1$ .
  - Fault Present: Yes (i = 1).
  - Error Manifested: Yes ( $i$ is $1$ ; arr[0] containing $0$ is skipped).
  - Failure Observed: Yes. The skipped element is zero. The loop inspects arr[1] ( $4$ ) and arr[2] ( $6$ ). Variable count remains $0$ . At line L5, the function returns $0$ , violating the specification.
- Failure Rule: A failure occurs if and only if: $\text{arr.length} > 0 \quad \text{and} \quad \text{arr}[0] = 0$

Software Testing Principles and Limitations

Comparison of Real System Execution Against Ideal System Specifications

Definition and Function of Testing:
- The systematic process of checking the quality, functionality, usability, and performance of a software artifact prior to commercial or production release.
- Functionally acts as a comparison engine: identical test input data is provided to both the real system implementation and an ideal system specification/oracle, and actual outputs are compared against expected outputs to identify discrepancies.
Modern Architectural Challenges:
- Systems have evolved into distributed, highly asynchronous, and dynamic architectures.
- Systems must maintain high reliability across environments.
- Contemporary platforms interact with enormous arrays of web and cloud microservices.
- Modern enterprise systems routinely comprise millions of lines of code.
Inherent Testing Limitations and Trade-Offs:
- Resource and Cost Overhead: According to the National Institute of Standards and Technology (NIST), testing consumes over $30\text{--}40\%$ of standard software project resources. In mission-critical and safety-critical domains, verification consumes up to $60\%$ of total development effort.
- Incompleteness of Test Coverage: Exhaustive testing is mathematically and practically impossible; test suites cannot cover the entire input and execution space of a non-trivial project.
- Applicability Constraints: Not all theoretical testing methodologies can be applied across every tier of an enterprise architecture.
- Resource and Time Constraints:
  - Constrained by severe delivery deadlines, budget ceilings, and contractual scopes.
  - Forces teams into critical decisions regarding what to test versus what to deliberately leave untested.
- Emerging Acceleration Paradigms: Cloud-based distributed test execution frameworks and Large Language Models (LLMs) are increasingly leveraged to generate test cases and manage testing overhead.

Testing Perspectives: People, Scope, and Change

Core Dimensions of Testing Strategies:
- Testers: Identifies the role executing the tests (developers, QA specialists, external end-users, domain specialists).
- Coverage: Identifies which system features, branches, or paths are tested, and which areas remain omitted.
- Potential Problems: The underlying operational risks or vulnerability vectors targeted by the test.
- Activities: The technical mechanics and procedures used to carry out verification.
- Evaluation: The criteria and test oracles utilized to determine pass/fail status.
People-Focused Testing Classifications:
- User Testing: Direct testing by prospective end-users to evaluate usability and workflow alignment.
- Alpha Testing: Internal system acceptance testing conducted within the development organization, but outside the direct implementation team.
- Beta Testing: Releasing a late-stage pre-release version to a representative sample of external users in real-world operating environments.
- Bug Bashes: Intensive collaborative internal sessions where diverse personnel across an organization (programmers, product managers, technical writers, marketers) halt standard tasks to test the software concurrently and locate defects.
- Subject-Matter Expert Testing: Verification conducted by domain specialists (e.g., accountants for tax software, medical practitioners for healthcare systems) to validate domain logic.
- Paired Testing: Two team members working together simultaneously at a single workstation (or virtual workspace)—one executing tests while the other records observations, analyzes edge cases, and inspects results.
- Dogfooding ("Eat Your Own Dogfood") / A/B Testing: Internal day-to-day operational use of pre-release software by the organization that produces it; closely related to A/B testing where alternative variants are run in parallel to evaluate comparative outcomes.

Change-Driven Testing with Test-Impact and Test-Gap Analysis

Change-Driven Testing Framework:
- Transitions software from an existing baseline (Version A) to a modified codebase (Version B).
- Test-Impact Analysis (TIA): Analyzes source code alterations to identify and select existing tests in the suite that execute the modified components (impacted tests), isolating tests that now fail due to regressions.
- Test-Gap Analysis (TGA): Identifies modified or newly added blocks of code in Version B that possess no corresponding test cases (missing tests), pinpointing coverage blind spots requiring new test creation.

Structural and Functional Testing Methodologies

Application Perspectives: Black-Box vs. White-Box Testing:
- Black-Box Testing (Functional / Specification-Based): Tests the system from an external perspective without access to or knowledge of internal source code structures; behavior is validated purely against external functional specifications.
- White-Box Testing (Structural / Implementation-Based): Tests software with full access to source code and architectural implementations, designing test inputs based on units, methods, branches, condition paths, and statements.
Comprehensive Comparison of Black-Box and White-Box Testing:
- Specification vs. Implementation Dependency:
  - Black-box: Tests depend exclusively on external requirements and specifications.
  - White-box: Tests depend concurrently on internal code implementation details and specifications.
- Test Suite Reusability Following Code Modification:
  - Black-box: Tests remain valid and reusable when internal source code is refactored, updated, or bug-fixed, provided the external functional specification does not change.
  - White-box: Tests are frequently invalidated whenever internal implementation details change, often requiring complete test restructuring.
- Lifecycle Phase of Test Development:
  - Black-box: Test cases can be written early in the development lifecycle before any code is produced, as they require only functional specifications.
  - White-box: Test suites can only be constructed after source code is drafted and compiled, requiring both executable units and specifications.
- Defect Detection Capabilities:
  - Black-box: Cannot guarantee that all execution paths in the code have been executed; can miss errors of commission (unintended or malicious functions implemented in code that are absent from specifications).
  - White-box: Cannot verify that source code satisfies the entire requirements scope; can miss errors of omission (required features completely missing from the implementation).
- Tooling and Metric Automation:
  - Black-box: Few automated tools provide direct functional coverage measurement based strictly on black-box specifications.
  - White-box: Extensive commercial and open-source tooling provides automated code coverage measurement (statement coverage, branch coverage, path coverage, mutation coverage).
Targeted System Testing Types and Core Objectives:
- Functional Testing:
  - Goal: Validate functional operations against specifications across the entire integrated system.
  - Primary Objectives: Uncover as many implementation defects as possible and provide tangible empirical evidence that the application functions as intended.
- User Testing:
  - Goal: Ensure that the product is useful, accessible, and intuitive for actual end-users.
  - Primary Objectives: Validate that system capabilities support real-world user workflows and verify that users intuitively understand how to locate and operate features.
- Performance and Load Testing:
  - Goal: Verify processing throughput and operational speed under realistic and peak transaction volumes.
  - Primary Objectives: Measure processing latency and response times to ensure user-acceptable thresholds; demonstrate that the system scales gracefully without crashes or data corruption under peak load.
- Security Testing:
  - Goal: Maintain system integrity and safeguard digital assets.
  - Primary Objectives: Assess defenses protecting confidential user data from unauthorized exfiltration, malicious modification, denial of service, and physical or logical corruption.

Interactive Exercises and Module Resources

Interactive Brainstorming Prompts:
- Quality Attributes Exploration: Identification and analysis of non-functional software attributes beyond the ISO 25010 standard model.
- Testing Perspectives Mentimeter Activity: Interactive classroom polling on testing paradigms hosted at menti.com utilizing access code 8526 9875 (URL: https://www.menti.com/alyh8xu4rxh8).
- Mid-Module Feedback Submission: Collection of student feedback for course management via Microsoft Forms (https://forms.office.com/e/W8C8QH0EwG).
Academic References and Literature Foundations:
- Ian Sommerville: Engineering Software Products (2020, Pearson).
- Ian Sommerville: Software Engineering (2011, Pearson Education Inc.).
- Sebastian Goericke: The Future of Software Quality Assurance (2020, Springer Nature).
- Amro Al-Said Ahmad, Peter Brereton, and Peter Andras: A systematic mapping study of empirical studies on software cloud testing methods (2017 IEEE QRS-C).
- Ralf Bierig, Stephen Brown, Edgar Galván, and Joseph Timoney: Essentials of Software Testing (2021, Cambridge University Press).