Software Engineering: Software Testing

Course & Module Information

• Course Code: MCA CS1T03
• Course Title: Software Engineering
• Chapter: Software Testing
• Instructor: Praveen Kumar, Assistant Professor, Department of MCA, Patna Women’s College

Software Testing Objectives

• Primary Learning Objectives:
  • How to define Software Testing Principles.
  • Understanding the various types of Software Tests.
  • Understanding Test Planning.
  • Mastery of Test Execution and Reporting.
  • Understanding Real-Time Testing.
• Post-Chapter Learning Outcomes:
  • Ability to articulate core Software Testing Principles.
  • Ability to categorize and explain types of software tests.
  • Understanding the workflow of test planning, test execution, and result reporting.
  • Ability to apply testing techniques in real-time scenarios.

Fundamental Concepts of Software Testing

• Definition of Software Testing:
  • Testing is the process of evaluating a system or its component(s) with the intent to find whether it satisfies specified requirements or not.
  • It involves executing a system in order to identify any gaps, errors, or missing requirements contrary to actual requirements.
• ANSI/IEEE 1059 Standard Definition:
  • According to the ANSI/IEEE 1059 standard, testing is defined as a process of analyzing a software item to detect differences between existing and required conditions (defects, errors, or bugs) and to evaluate the features of the software item.

Testing Roles and Stakeholders

• Contextual Responsibility:
  • The allocation of testing responsibilities depends on the software development process and associated project stakeholders.
• Corporate Practice:
  • Large IT companies maintain dedicated evaluation teams responsible for testing developed software against requirements.
  • Software developers perform testing known as Unit Testing.
• Key Personnel Involved in System Testing:
  • Software Tester
  • Software Developer
  • Project Lead / Manager
  • End User
• Industry Designations:
  • Roles vary across organizations based on domain experience and knowledge, commonly including designations such as:
  • Software Tester
  • Software Quality Assurance (SQA) Engineer
  • QA Analyst

Software Testing Lifecycle Timing

When to Start Testing

• Cost & Time Reduction: Starting testing early reduces the overall cost and time needed for rework, ensuring the delivery of an error-free software product to the client.
• SDLC Integration: Testing can begin as early as the Requirements Gathering phase and continue through software deployment across the Software Development Life Cycle (SDLC).
• Dependency on Development Models:
  • Waterfall Model: Formal testing is conducted primarily during the dedicated testing phase.
  • Incremental Model: Testing is performed at the end of every iteration/increment, followed by testing of the entire application at the project's conclusion.
• Testing Activities Across SDLC Phases:
  • Requirements Gathering Phase: Requirements analysis and verification count as testing.
  • Design Phase: Reviewing system design to make design improvements is categorized as testing.
  • Coding Phase: Developer verification upon completing code modules is categorized as testing.

When to Stop Testing

• Continuous Nature: Testing is an ongoing process; it is impossible to claim that a software application is 100% tested.
• Stopping Criteria:
  • Arrival of testing deadlines.
  • Completion of planned test case execution.
  • Reaching defined functional and code coverage thresholds.
  • Bug rate dropping below a designated threshold with zero unresolved high-priority bugs.
  • Management decision to halt testing.

Verification vs Validation

• Core Differences:
  • Concern Addressed:
  • Verification: Addresses the question: "Are you building it right?"
  • Validation: Addresses the question: "Are you building the right thing?"
  • Functionality Scope:
  • Verification: Ensures the software system meets all functionality.
  • Validation: Ensures that functionalities meet intended behavior.
  • Execution Sequence & Scope:
  • Verification: Occurs first; includes checking documentation, code, design, etc.
  • Validation: Occurs after verification; involves checking the overall final product.
  • Personnel Involved:
  • Verification: Conducted by developers.
  • Validation: Conducted by testers.
  • Activity Type:
  • Verification: Consists of static activities (reviews, walkthroughs, inspections).
  • Validation: Consists of dynamic activities (executing software against requirement specs).
  • Process Nature:
  • Verification: An objective process requiring no subjective decisions.
  • Validation: A subjective process involving subjective evaluations of system performance.

Primary Types of Testing

Manual Testing

• Definition: Testing software manually without using automated tools or executable scripts.
• Role Play: The tester assumes the persona of an end-user to detect unexpected behavior or bugs.
• Stages: Spans across Unit Testing, Integration Testing, System Testing, and User Acceptance Testing (UAT).
• Artifacts: Testers utilize test plans, test cases, or test scenarios to ensure absolute testing coverage.
• Exploratory Testing: Includes exploratory testing where testers actively explore the application to locate unscripted errors.

Automation Testing

• Definition: Also known as Test Automation. Involves writing test scripts and utilizing specialized software tools to evaluate the product.
• Purpose: Automates manual procedures to execute test scenarios rapidly and repeatedly.
• Applications: Primarily used for Regression Testing as well as Load, Performance, and Stress Testing.
• Benefits: Expands test coverage, improves testing accuracy, saves significant time, and lowers costs compared to manual testing.
• What to Automate:
  • Transactional user interaction areas (e.g., login forms, user registration forms).
  • System modules subject to high simultaneous user concurrency.
  • Graphical User Interface (GUI) elements, database connectivity, and field input validations.
• When to Automate:
  • Large-scale and mission-critical projects.
  • Projects requiring frequent re-testing of identical system areas.
  • Applications with stable, non-changing requirement sets.
  • Performance and load testing involving hundreds or thousands of virtual users.
  • Software that has stabilized under manual testing.
  • When appropriate time and resources are allocated.
• How to Automate (Step-by-Step Process):
  • Identify application areas suitable for automation.
  • Select the appropriate test automation tool.
  • Write automated test scripts using scripting languages (such as VB Scripting).
  • Develop modular test suites.
  • Execute automated test scripts.
  • Generate performance and execution result reports.
  • Identify potential bugs, defects, or performance bottlenecks.

Software Testing Tools

• HP Quick Test Professional (QTP)
• Selenium
• IBM Rational Functional Tester
• SilkTest
• TestComplete
• Testing Anywhere
• WinRunner
• LoadRunner
• Visual Studio Test Professional
• WATIR

Software Testing Methods

Black-Box Testing

• Definition: Testing performed without any knowledge of the internal logic, architecture, or code structure of the application.
• Synonyms: Closed-box testing, data-driven testing, functional testing.
• Mechanism: Testers interact exclusively with the User Interface by feeding inputs and analyzing output responses without knowing how or where data is processed internally.
• Advantages:
  • Highly effective and efficient for large code bases.
  • Source code access is not required.
  • Creates a clear boundary between user perspective and developer perspective.
  • Allows large groups of moderately skilled testers to execute tests without knowledge of programming languages, implementation details, or operating systems.
• Disadvantages:
  • Limited coverage since only a restricted subset of test scenarios is executed.
  • Inefficient testing due to limited tester domain knowledge.
  • Blind coverage (inability to target high-risk, error-prone code segments).
  • Test cases can be difficult to design accurately.

White-Box Testing

• Definition: Detailed investigation of internal logic, control structures, and code statements.
• Synonyms: Glass testing, open-box testing, clear-box testing, structural testing, code-based testing.
• Mechanism: Requires full visibility and understanding of source code to analyze which specific code unit/chunk is malfunctioning.
• Advantages:
  • Knowledge of source code makes identifying appropriate test data straightforward.
  • Directly assists in code optimization.
  • Uncovers hidden defects by revealing redundant lines of code.
  • Achieves maximum test coverage during scenario design.
• Disadvantages:
  • Significantly increases cost due to the requirement for highly skilled software engineers/testers.
  • Exhaustive path coverage is impossible, leaving many code branches untested.
  • High maintenance overhead, requiring specialized tools like code analyzers and debuggers.

Grey-Box Testing

• Definition: Testing technique performed with partial/limited knowledge of internal application workings.
• Synonyms: Translucent testing.
• Mechanism: Relies on high-level architecture documents, design specs, database diagrams, and data flow diagrams without direct reliance on full source code.
• Advantages:
  • Combines the benefits of both black-box and white-box techniques.
  • Relies on interface definitions and functional specifications rather than source code.
  • Enables creation of targeted test scenarios for communication protocols and data type handling.
  • Maintains testing perspective focused on the end-user rather than the software designer.
• Disadvantages:
  • Code execution path coverage remains limited without complete source code access.
  • High risk of redundant test cases if developers have already executed equivalent unit tests.
  • Exhaustive testing of all input streams is unrealistic due to time constraints, leaving code paths unexamined.

Comprehensive Method Comparison Matrix

Software Testing Levels

Functional Testing

• Definition: A type of black-box testing based on functional specifications. Inputs are provided to the system and outputs are verified against intended functionality.
• Scope: Conducted on a complete, integrated system to assess compliance with requirements.
• Five Steps of Functional Testing:
  1. Step I: Determination of the functionality the application is intended to perform.
  2. Step II: Creation of test data based on application specifications.
  3. Step III: Determination of expected output based on test data and specifications.
  4. Step IV: Writing test scenarios and executing test cases.
  5. Step V: Comparison of actual outcomes against expected results.

Unit Testing

• Execution: Performed by developers prior to passing builds to the QA team.
• Scope: Tests isolated units of source code in assigned development modules.
• Test Data: Uses developer-specific test data, distinct from QA team test data.
• Goal: Isolate each software module to prove that individual parts are correct in functionality and requirements.
• Limitations:
  • Cannot catch every bug because evaluating every possible code path is impossible.
  • Constrained by finite test data and developer scenarios.
  • Testing must eventually stop to allow code units to merge with the broader application.

Integration Testing

• Definition: Testing combined parts/modules of an application to verify correct interaction and data flow.
• Integration Methods:
  • Bottom-Up Integration: Begins with unit testing, followed by testing progressively higher-level combinations of units called modules or builds.
  • Top-Down Integration: The highest-level modules are tested first, followed by progressively testing lower-level modules.
• Standard Approach: In enterprise environments, bottom-up testing is performed first, followed by top-down testing, concluding with real-world simulated scenario testing.

System Testing

• Definition: Rigorous testing of the completely integrated software application to verify adherence to Quality Standards.
• Executers: Performed by a dedicated, specialized testing team.
• Importance:
  • Serves as the first SDLC phase where the entire system is evaluated as a whole.
  • Verifies functional and technical specifications.
  • Conducted in environment configurations mirroring live production environments.
  • Validates both business requirements and software architecture.

Regression Testing

• Definition: Re-testing modified software to ensure bug fixes or code updates have not broken existing functionality or violated business rules.
• Importance:
  • Minimizes testing gaps when deploying modified applications.
  • Confirms new code modifications do not create unintended side effects in un-edited modules.
  • Mitigates operational risks.
  • Enhances overall test coverage without extending delivery timelines.
  • Accelerates product time-to-market.

Acceptance Testing

• Definition: Essential testing conducted by Quality Assurance teams and clients to verify whether the product satisfies operational and business requirements.
• Execution: Uses pre-scripted end-to-end user scenarios and test cases.
• Purpose: Uncovers major logical flaws, structural errors, and system crash bugs beyond basic typo or alignment checks.
• Significance: Determines production readiness and satisfies legal and contractual acceptance criteria.

Alpha Testing

• Definition: First-stage internal acceptance testing executed jointly by internal Developer and QA teams.
• Composition: Combines Unit, Integration, and System testing activities.
• Focus Areas:
  • Spelling mistakes and typographic errors.
  • Broken links and broken navigation paths.
  • Unclear or cloudy operational instructions.
  • Latency and execution speed when run on minimum hardware specifications.

Beta Testing

• Definition: Pre-release testing where a sample audience of actual end-users tests the software application.
• Distribution: Often distributed via web platforms to collect real-world feedback and provide a preview of upcoming releases.
• Focus Areas:
  • Installation and real-world execution testing by target users.
  • Uncovering lingering typos, application flow confusion, and platform-specific crashes.
  • Collecting structured feedback so development teams can patch issues prior to public release.
  • Increasing overall user satisfaction and product launch quality.

Non-Functional Testing

• Definition: Testing non-functional quality characteristics such as performance, security, usability, and system portability.

Performance Testing

• Primary Focus: Designed to locate performance bottlenecks rather than functional software defects.
• Causes of Poor Performance:
  • Network delay and latency.
  • Excessive client-side processing.
  • Database transaction delays.
  • Improper server load balancing.
  • Slow data rendering.
• Core Dimensions:
  • Speed: Response times, page rendering, and data retrieval rates.
  • Capacity: System volume thresholds.
  • Stability: Operational consistency under workload.
  • Scalability: System capability to expand capacity under increased load.
• Sub-types: Divided into Load Testing and Stress Testing.

Load Testing

• Definition: Evaluates software performance under normal and peak workload conditions by applying maximum operational load.
• Objective: Identifies maximum system capacity and monitors behavior under load spikes.
• Tooling: Executed via automated tools using Virtual Users (VUsers).
  • Common Load Testing Tools: LoadRunner, AppLoader, IBM Rational Performance Tester, Apache JMeter, Silk Performer, Visual Studio Load Test.
• Flexibility: VUser counts can be dynamically increased or decreased concurrently or incrementally.

Stress Testing

• Definition: Evaluates software resilience under abnormal conditions or load constraints exceeding design capacity.
• Objective: Applies high stress while depleting infrastructure resources to locate the system's exact breaking point.
• Simulated Stress Scenarios:
  • Randomly shutting down or restarting hardware network ports.
  • Abruptly toggling database connections on and off.
  • Spawning resource-heavy background processes to consume CPU, RAM, and server bandwidth.

Usability Testing

• Definition: A black-box testing approach observing end-users during operation to identify UI friction and operational issues.
• Nielsen’s 5 Usability Factors:
  • Efficiency of use
  • Learn-ability
  • Memory-ability (Memorability)
  • Errors / Safety
  • Satisfaction
• Bevan & Macleod View: Defines usability as a measurable interaction outcome satisfying user goals efficiently through proper resource usage.
• Molich (2000) 5 Goals: A user-friendly system must be:
  1. Easy to learn
  2. Easy to remember
  3. Efficient to use
  4. Satisfactory to use
  5. Easy to understand
• Usability Quality Standards: ISO-9126, ISO-9241-11, ISO-13407, IEEE std.610.12.

UI vs Usability Testing

• UI Testing: Evaluates graphical user interface components against specifications, verifying properties like colors, alignment, fonts, and element sizing.
• Usability Testing: Evaluates overall user-friendliness and ease of operational workflow.
• Relationship: UI testing represents a subset of Usability testing.

Security Testing

• Definition: Evaluates software for security gaps, logic vulnerabilities, and architectural flaws.
• 6 Pillars of Security:
  • Confidentiality
  • Integrity
  • Authentication
  • Availability
  • Authorization
  • Non-repudiation
• Specific Security Aspects Tested:
  • Protection against known and unknown security vulnerabilities.
  • Data storage and transport security.
  • Adherence to regulatory compliance standards.
  • Input checking, data validation, and buffer overflows.
  • Protection against SQL Injection and General Injection flaws.
  • Robustness of Session Management.
  • Vulnerability to Cross-Site Scripting (XSS).
  • Vulnerability to Directory Traversal attacks.

Portability Testing

• Definition: Evaluates the reusability and operational movement of software across target deployment environments.
• Portability Strategies:
  • Transferring installed application builds directly across machines.
  • Building cross-platform executables (.exe) to run across operating systems, browsers, and hardware.
• Relationship: Portability testing functions as a specialized subset of System Testing.
• Pre-conditions for Portability Testing:
  • Software must be architected and coded to portability standards.
  • Unit testing of modules must be complete.
  • Integration testing must be complete.
  • Target multi-platform test environments must be configured.

Testing Documentation

• Purpose: Serves to estimate effort, track test coverage, and maintain requirement traceability.
• Key Artifacts: Test Plan, Test Scenario, Test Case, Traceability Matrix.

Test Plan

• Author: Typically written by the Quality Assurance Lead.
• Definition: Documents overall strategy, resource allocation, target test environments, testing limitations, and schedule schedules.
• Key Sections Included:
  • Introduction document.
  • Testing assumptions.
  • List of included test cases.
  • List of software features to be tested.
  • Strategic testing approach.
  • List of test deliverables.
  • Resource allocation details.
  • Risk analysis and contingency plans.
  • Schedule of milestones and task deliverables.

Test Scenario

• Definition: A one-line statement defining what specific business flow or functionality will be tested.
• Scope: Ensures complete end-to-end operational path coverage. A module can contain anywhere from one to hundreds of scenarios based on complexity.
• Difference from Test Case: A test scenario contains multiple sequential operational steps, whereas a test case represents a single step verification. Scenarios string dependent test steps together, where step output feeds into subsequent steps.

Test Case

• Definition: A set of conditions, inputs, and step-by-step procedures used to evaluate pass/fail criteria.
• Varieties: Functional, negative, error, logical, physical, and UI test cases.
• Core Components Included in Every Test Case:
  • Test Case ID
  • Product Module
  • Product Version
  • Revision History
  • Purpose
  • Assumptions
  • Pre-conditions
  • Execution Steps
  • Expected Outcome
  • Actual Outcome
  • Post-conditions
• Test Suites: A grouped collection of multiple test cases written for a software application.

Questions & Exercise

Objective Type Questions (Section A)

1. Verification is:

   • A. Checking that we are building the right system
   • B. Checking that we are building the system right
   • C. Performed by an independent test team
   • D. Making sure that it is what the user really wants
   • Answer: B. Checking that we are building the system right

2. Before launching a software which testing is to be done in-house?

   • A. Beta
   • B. Gamma
   • C. Alpha
   • D. None of the above
   • Answer: C. Alpha

3. Which testing phase tests individual software modules combined together as a group?

   • A. Module testing
   • B. Integration testing
   • C. White Box testing
   • D. Software testing
   • Answer: B. Integration testing

4. The main focus of acceptance testing is:

   • A. finding faults in the system
   • B. ensuring that the system is acceptable to all users
   • C. testing the system with other systems
   • D. testing for a business perspective
   • E. testing by an independent test team
   • Answer: B. ensuring that the system is acceptable to all users

Short Answer Questions (Section B)

1. Difference between white box testing and black box testing?

   • White-box testing involves investigating internal source code logic with full visibility of internal code, executed primarily by developers using structural/code-based methods. Black-box testing evaluates application functionality strictly through the user interface without requiring code access or internal knowledge.

2. What is integration testing discuss with example?

   • Integration testing evaluates combined software modules to verify data interaction and functional integration. Methods include Bottom-Up (testing low-level units moving upward to high-level builds) and Top-Down (testing high-level modules first down to lower units).

3. Explain briefly nonfunctional testing?

   • Non-functional testing measures system quality attributes rather than discrete functional features. It includes evaluating performance bottlenecks, load limits, stress breaking points, usability, security vulnerabilities, and platform portability.

Long Answer Questions (Section C)

1. What are different type of test methods discuss with support of example?

   • Software testing methods are categorized into Black-Box, White-Box, and Grey-Box testing. Black-box checks UI inputs/outputs; White-box analyzes branch logic and code path optimization; Grey-box utilizes high-level data flow diagrams and database schemas to test system boundary conditions.

2. Discuss functional testing with example?

   • Functional testing is a black-box testing level evaluating system behaviors against functional specification requirements across five phases: determining functionality, generating test data, identifying expected output, writing/executing test cases, and comparing actual versus expected outcomes.