Testing in SCRUM
Introduction to Testing in SCRUM
Overview
Agile software development methods aim to:
Reduce development time.
Improve software quality.
Increase customer satisfaction.
This lecture describes the agile development process from a testing and QA perspective, explaining how agile testing works and where traditional testing techniques remain necessary.
Intended Learning Outcomes
By the end of this lesson, you should be able to:
Understand the differences between testing in the waterfall method and Agile methodology.
Identify Agile quality assurance tools.
Analyze various testing approaches.
Compare different testing methods.
List of Subtopics
Testing in waterfall method vs Agile Testing
Agile Quality Assurance vs Traditional Quality Assurance
Traditional Quality Assurance Tools
Agile Quality Assurance Tools
Test Planning in Scrum
Unit testing
The Test First approach
Unit Testing Frameworks
Unit Test Management
Integration Testing
Designing Integration Test Cases
Dependencies and Interfaces
Integration Levels
Continuous Integration
Implementing CI
Integration Test Management
System Testing and Testing Nonstop
Manual System Testing
Automated System Testing
Using Test First for System Testing
Non-functional Testing
Automated Acceptance Testing
System Test Management
Testing in Waterfall Method vs Agile Testing
Traditional Test Management
In traditional process models, projects are divided into distinct phases to achieve predefined milestones.
In the waterfall model, testing is a separate phase.
In Agile, testing is performed alongside development.
Agile Testing
Agile testing adheres to the Agile Manifesto principles, applying agile methodology to software testing.
In Scrum, quality is integrated continuously by a cross-functional Scrum team throughout each sprint.
This reduces the need for significant late testing to track quality.
An Agile tester performs continuous, parallel testing at each sprint.
Critical Factors in Agile Testing
Test Automation: Automating test cases allows for timely feedback, crucial for reliable applications and clean code.
Exploratory Testing: Manual testing is still needed since not every case can be immediately automated. Exploratory Testing involves testing without extensive preparation, allowing for short-term testing of new features.
Test Expertise within the Team: Testing tasks are treated like any other sprint activity, with each developer contributing to testing. External testers may be incorporated but must test as part of the agile team; independent testing is not allowed.
Multiple Teams: If multiple Scrum teams are working on a single project, features must integrate seamlessly. System-wide testing is essential, and test managers and full-time testers should exchange views, similar to a Scrum of Scrums.
Agile Quality Assurance vs Traditional Quality Assurance
Traditional Quality Assurance Tools
Traditional software development QA measures can be constructive or analytical.
A constructive measure example: a document created according to a template.
Analytical measures examples: testing, checking, and reviewing.
A QA plan is drafted by the QA manager.
A separate QA group focuses exclusively on QA tasks to identify defects before delivery, which is a key advantage of traditional project organization.
Agile Quality Assurance Tools
The entire team is responsible for the product and its quality.
There is no dedicated QA group.
Each team member uses their specific skills, not limited to particular jobs.
QA in an agile team is based on "Inspection and Adaptation".
Inspection: Scrum artifacts are frequently inspected to ensure progress towards sprint goals.
Adaptation: Adjustments are made quickly to minimize deviations from expected limits.
Decisions on when and how checks are performed are made during sprint meetings like sprint planning, daily scrum, review, and retrospective.
QA issues are discussed in daily scrums to identify deficits early.
Advantages of Agile QA Measures
Every development artifact is tested.
Checking program codes, diagrams, and design documents through task boards and backlogs.
Corrective measures are taken as early as possible.
If an automated test case fails, the responsible programmer fixes the code immediately or provides a solution in the current sprint. If the problem persists, creating a solution is added to the product backlog.
Constructive QA tools.
Techniques like clean code, test automation, and test-first are used within the development process. Team members use their own standards to increase product quality.
Process improvements are made from the bottom up.
Immediate actions are taken based on the team's needs.
Test Planning in Scrum
Testing activities are planned and controlled within a Sprint, similar to other activities (Product backlog, Sprint backlog, Task board).
Key considerations when planning Scrum-based tests:
Definition of Ready (DOR): A checklist ensures User Story quality. From a tester’s perspective, a User Story is not sufficiently refined if it is unclear how to draft a test case (i.e., when the test should pass or fail).
Definition of Done (DoD): A checklist used to determine if a feature is ready for the Sprint Review. It includes test types, test coverage, and criteria to ensure product quality and user satisfaction.
Unit Testing
Unit Testing
All developer-applied tests on their code are known as "developer testing" or "unit testing."
Traditional approaches have separate integration and system testing teams, but in Agile, the team handles all tasks.
Unit testing is performed on individual components, making defect rectification easier.
For object-oriented languages, classes and their methods are elements; for non-object-oriented languages, functions and modules are individual components.
Test First Approach
Traditional software development follows a "Program first, then test" approach.
Test First is an effective method used in Agile development.
It involves determining necessary tests to show software fulfills new specifications before any code changes are made.
Tests are designed and automated beforehand and will initially fail because the product code does not yet exist.
The developer writes the product code only after completing the automated tests.
Coding is complete when all tests pass.
The developer addresses the product code until all test cases pass if any test fails.
This approach is called "Test-driven development," where tests drive the developer.
The development cycle repeats: "write test, run test, change code”.
Strict use of the Test First methodology significantly enhances the effectiveness of unit testing as a quality management tool.
Scrum does not require Test First but can work with traditional unit testing.
However, a Scrum team can benefit greatly from the accelerated feedback loop provided by Test First techniques.
Each developer receives extra feedback for every programming task.
While normal unit tests written post-coding check for damage to existing functionality, Test First enables checking of new functionality.
Implementing the Test First Approach
Implementing tests for non-existent objects can be challenging.
Test First requires developers to think more abstractly than conventional unit testing.
When switching to Test First, pair programming is particularly helpful.
Code reviews at the team level are also important to enhance the Test First approach.
Unit Testing Frameworks
Automating unit tests is not difficult.
Example Scenario:
Consider a class 'abc' that requires testing. The test class is 'abcTest'. Each test case is implemented using a unique method.
Use meaningful names for test methods to reflect their content.
Each test case should check only one aspect of the test object.
Each test case has four sections: 'setup', 'test procedure', 'check', and 'teardown'.
In the 'check' section, compare actual output with expected output.
If actual and expected outputs match, the test 'passes'; otherwise, it 'fails'.
These methods can be run individually or as a batch using the framework's 'run' method.
Leveraging Unit Test Frameworks
Developers can include additional functionality using pre-coded unit test frameworks.
Frameworks simplify the structure of the test code.
These frameworks are freely available online for most popular languages.
Examples: JUnit for Java, CppUnit for C++, NUnit for .Net
These frameworks are based on the SUnit framework.
Handling Component Dependencies
Objects on which unit tests are run are components of larger systems and may have dependencies on other components.
Therefore, all dependencies must be installed in the environment to test an object but some dependent components may not be available at the testing time since they are to be implemented in coming Sprints.
As a solution for the unavailability of some components at the time of testing, we can replace dependencies with placeholders, called ‘test doubles’.
Following are different types of placeholders:
Stub: Replaces a depended-on component with one that has an identical interface that produces specific reactions (returns certain values according to a predefined pattern).
Spy: A stub with the additional technique of recording calls and data handed over by the test object. Recorded data can be used for debugging.
Mock: A stub that analyzes calls and data received from the test object and returns results to the object.
Fake: A simplified implementation to replace a depended-on component that is required for testing. It does not influence the test results.
Dummy: An empty object or null pointer that replaces a data object. A dummy is not interpreted and therefore forms no part of the test data.
Importance in Agile Development
Placeholders play an important role in Agile development compared to traditionally managed projects.
As components are developed incrementally, some components may not yet exist.
Place holders are essential to prevent blocking feedback due to missing test components.
Test First automatically runs existing unit tests each time the code is modified.
Therefore, minimizing test runtime is important.
Creating placeholders involves an effort and this needs to be considered when planning a Sprint.
Unit Test Management
The following aspects need to be agreed upon in test management:
Unit test framework: The Scrum master ensures the whole team writes similarly structured tests. Tests need to be stored centrally with an agreed structure. It is advised to keep the test code separately but parallel to the program code. The team must agree on a naming convention for test classes and test methods.
Coverage measurement: The Scrum master ensures test coverage is measured reliably, possibly by integrating a coverage measurement tool. Measurements such as class coverage, method coverage within a class, and line coverage need to be taken.
Static code analysis: In addition to static code analysis, dynamic unit tests need to be implemented. Developers can use different tools based on the programming language they use.
Test code reviews: Regular test code reviews should be carried out to check test design, boundary values, effectiveness, etc. If deficiencies are found, the team should start appropriate training. A review also helps in identifying which test cases still need to be written to increase coverage.
Extending Unit Tests for Non-Functionalities
After several Sprints of successful quality automated tests, the team can extend unit tests to check non-functionalities such as performance and security.
However, these tests do not guarantee the non-functional aspects of the final product but help identify weaknesses early.
Integration Testing
Integration testing ensures that independently designed components of a system perform together as expected.
Integration tests identify potential defects in interactions between individual components and user interfaces.
For example, an integration test can be performed to test if classes A and B are working well together. If the interaction is between methods b2 and a3, the test should ensure if b2 is called correctly and if it returns the correct values for a3.
Class A
function a1() {...}
...
function a3() {
...
res = B->b2(x,y);
…
}
function an(){...}
Class B
function b1() {...}
function b2(a,b) {
…
}
function bm(){...}
Designing Integration Test Cases
The steps to derive integration test cases:
Analyze interaction: Identify and list inter-component interactions using the architecture diagram.
Equivalence partitioning: Identify the parameters or messages (inputs) that are exchanged in the identified interactions.
Partition these parameters into sets of equal size.
Determine which of them cause the behavior of the affected component.
Establish test case inputs: Use the equivalences determined in step 2 to identify the calls or calling sequences in a component’s API that are required to call the other component.
Define an expected behavior: Define the expected behaviors of both observed components for each test case.
Test robustness of asynchronous communication: Asynchronous calls from one component to another require additional testing to check and fix timing, throughput, capacity, and performance defects. This step is only necessary when the components run parallel and are connected asynchronously.
Dependencies and Interfaces
Explicit (Direct) Dependency
One component calls another directly via its API.
This dependency type can clearly be observed from the code.
Identified by the compiler or static code analysis.
Integration tests for this component type need to cover all relevant variations on method calls or the data transfer between components in case of asynchronous communication.
Implicit (Indirect) Dependency
Multiple components share a single recourse such as a global variable, a file or a database.
These dependencies are caused by the changes in the behavior of one component due to the content of a shared resource altering as a result of a write by another component.
The code itself only reveals that each component accesses the same resource but not how or when.
Integration tests should either cover all read/write sequences that affect the shared resource and/or the relevant contents of the resource.
Integration Levels
Components to be integrated may be of different levels of granularity depending on the level of abstraction on which the integration takes place.
Class Integration:
Lowest level of integration.
In object-oriented programming, smaller units are encapsulated within a class and are tested as part of a unit test.
Types of Class Integration:
Vertical Integration
Classes are integrated in order of their inheritance hierarchies.
For example, if class B inherits its attributes from class A, then if class A is altered, the calls from B would fail or behave differently. Therefore, an integration test of the interaction between A and B has to be repeated after each change to A.
If methods in B can overwrite methods in A, then changes in B should also be followed by tests.
When there is only one runtime object that runs the inherited code in its method, the test will follow the inheritance hierarchy and can be viewed as a unit test.
Horizontal Integration
Describes an interaction between two classes with no inherited attributes and with at least one object of each existing at runtime.
One class uses the other.
The interaction between the two classes requires integration testing.
Compound Classes and Objects
One class contains the other in the form of a data structure.
Class A accesses B, using B’s methods and/ or variables.
Class B can be seen as part of A.
Even though unit testing A tests B indirectly, B’s behavior cannot be observed.
Dependency injections can be used in testing, where the dependent object is not embedded as a variable of A but is instead given as a parameter, allowing B’s behavior to be observed from outside.
Subsystem Integration:
Once paired classes have been tested and work as expected, they can be compounded together to create a new component.
Repeated integration of such classes creates a cluster of classes referred to as a package or subsystem.
The interface used by the cluster varies based on the programming language whereas some languages allow specifying an interface for the package/ subsystem, while others use the some of the APIs within the cluster.
System Integration:
Once all subsystems have been tested, they are integrated to form a complete system.
Includes testing if the system uses the intended interfaces for communicating, often as part of system testing.
Testing at this level ensures that the customer receives a defect-free software and eliminates the costs that could be incurred should the product be revealed to be faulty once deployed/ installed on the customers’ devices.
Continuous Integration
Continuous integration (CI) is a step that facilitates incremental integration.
Incremental integration is when every piece of code is installed in the integration environment and integrated as soon as it is finished.
CI is fully automated with the following steps:
Steps in the CI Process
The team maintains a version-managed code in a central repository with frequent updates.
Automatic code integration in the CI server includes the sequential following of the steps below:
Code is compiled and the warnings and errors are logged.
Logging the results of the static code analysis in terms of quality metrics and coding guidelines.
The code is deployed and installed in the test environment once 2.1 and 2.2 are successful.
Initialization
Unit testing and logging the results to the CI server
Integration testing and logging the results to the CI server
System testing and logging the results to the CI server
CI server displays all results on a dashboard in real-time.
Implementing CI
Preparing the CI environment should be done before the first sprint as it may include a complete reorganization of the team. The process may take up to several weeks.
The steps in setting up a CI environment are as follows:
Check the current state of the configuration management system
What’s the CM tool used?
How widely is it used?
How does the team use the tool?
How long does it take to compile a build?
Questions such as the above should be answered and addressed appropriately.
Explore available CI server software with the team
After steps one and two to set up the CI environment the next steps to implement are:
Selecting CM and CI tools
A CI server has to reliably identify changes in code within the CM system.Installing CM and CI tools
Installation of the selected tools on a separate computer.Migrate CM
Importing the old code repository to the new CM.CI scripting
The scripts received with the software are adapted to the project environment.
At this point, the system is similar to a conventional CM system as it does not include automated testing yet.
Therefore, appropriate automated test packages are gradually adapted and installed
The following outline few points to strategize the degree of automation required:
Every build should automatically be unit tested.
Integration tests should be added.
If the team deems it necessary, the unit and integration test environments can be separated.
System tests should be automated and included in the CI system.
Integration Test Management
Ensures that sufficient numbers of tests are written and run.
Integration tests should reflect the planned architecture of the system.
Integration tests should be designed using the Test First principle.
Results of the integration tests can be used to update the system’s architecture in terms of architecture diagrams or automated Test First test cases.
Integration strategy is dictated by the Story Map.
Individual sprint integration strategy is dictated by the Sprint Backlog.
Due to continuous integration, sprint planning determines the integration test sequence in advance.
Integration testing effort should be taken into consideration in Sprint Planning.
The effort required depends on the number of dependencies between the components and not on the number of components itself.
Additional integration-related code analyses should be carried out when possible.
Integration tests should be sorted into batches.
The speed of the CI process should continuously be optimized.
System Testing and Testing Nonstop
System Testing
Scrum produces shippable products at the end of every sprint that requires a user interface and the capacity of interoperating with the customer’s existing systems.
System tests check that the product works from the user’s perspective with their own interfaces.
These elements are not covered by unit or integration testing and test cases trigger a data flow that passes through the entire system.
System tests are derived directly from the requirements and the acceptance criteria in the Product Backlog or use cases.
System Test Environments
System test environments are more complex than other test environments due to the replication of the production environment as accurately as possible by representing the external factors the system will work with in the user’s environment.
The environment would have as many real-world components (Hardware, software, networks, other systems as possible) as possible resulting in a large number of ways that these components can be configured in.
The system testing effort depends on the number of different test environments that need to be checked and not the number of test cases itself.
Manual System Testing
Exploratory Testing
These tests begin by only defining the objectives of the test (e.g.: a feature or a user story to be tested) and the test will primarily focus on this.
The structure and the individual steps are not predefined and are decided depending on the tester’s observations during the test.
Components that behave normally will be lightly tested or skipped altogether, focusing on components that display unusual behavior to find the cause.
These tests are ideal for checking new features quickly.
The quality of these tests depends heavily on the tester’s competence therefore, are difficult to be reproduced.
Session-Based Testing
The tester briefly describes the objectives and the strategy for the test in 2-3 lines.
Test setup, design and execution, defect localization and defect reporting are all limited to a maximum of 90 minutes.
The objectives, procedures, coverage, tested elements, defects discovered, etc. are noted in a session sheet.
Session sheets based on keywords facilitate long-term test automation.
Session-based testing overcomes the disadvantages of exploratory testing such as the inability to reproduce the tests, by electronically logging the session sheets.
Provides insight into the current state of the product and how users interact with it.
Acceptance Testing
An approval test selected by the customer.
Helps determine if the system is acceptable.
Identifies if the system does what was intended with the correct functionality.
Did we build the right system?
A sprint’s acceptance tests will focus primarily on newly added features.
Acceptance tests alone are insufficient to identify possible unwanted side effects of new changes to a system.
Automated System Testing
System testing is difficult to automate due to the following reasons:
Complex environments
Requires dedicated GUI tools as the main testing interface is the GUI and as GUI tests require reactions from other components, they’re relatively much slower.
A clearly defined and reproducible original state of the system configuration is required to base the test scripts on
It’s difficult to automatically compare the expected behavior with actual behavior.
There are many interfaces involved and therefore additional tests and tools are required.
Lack of team members who are experienced in automating system tests.
Types of Automated System Testing
Record/ Playback Testing
A record/ playback tools are used to record all user commands (e.g.: manual keyboard and mouse commands) given during the test as a script.
Running the saved script reproduces the test.
Drawbacks:
It is recommended that new test cases are written during each Sprint for new functions rather than updating existing tests because the GUI changes significantly in each Sprint.
Does not cater to alternative GUIs or GUI layouts.
Keyword-Driven Testing
Uses domain-specific vocabulary to describe the test procedure instead of general navigation commands.
The test cases represent what the system should be capable of doing but not how it works.
These are easily understood by technical and non-technical members.
A change in implementation (such as an interface change) will not affect the test case’s validity.
Drawbacks:
The team has to agree on a stable vocabulary and assign an interpreter to convert the commands.
Requires a sequence control mechanism and an adapter to connect the mechanism to the test objects.
Each test object needs to be a unique ID that will remain the same throughout the Sprints.
Time and resource-consuming to set up a keyword-based system test automation.
Behavior-Driven Testing (BDT)
Uses behavior-driven techniques and natural language.
The tester can use central keywords to define BDT scripts.
A feature is tested using various scenarios.
A scenario corresponds to a test case and is divided into sections based on the Given-When-Then principle.
BDT uses the test object’s API as the test automation interface.
Using Test First for System Testing
The Test First principle cannot be applied when GUI test tools are being used as they require direct access to the system’s user interface where they already have to be in existence.
But, if test cases are keyword-based or BDT based then, because these types of tests are independent of the system’s technical implementations and interfaces, can be created before the test object actually exists and thereby allow Test First based system test automation.
Additional Notes
System Test Repository: A centralized repository enables collecting and managing a team’s standardized set of notations and keywords for BDT or Keyword-Based test case creation.
Pair Programming: Pair Programming enables the following in terms of implementing Test First for system testing:
A programmer and a tester should together design the test cases, where the programmer will ensure that the correct feature is being focused on and the tester will ensure that the test cases represent the user’s point of view.
A pair of testers produce better results when drafting and maintaining the standard set of keywords.
Teams of testers and programmers together write the relevant programs in GUI test tool’s scripting language and perform xUnit programming to enable the implementation of keywords on all tiers of the test sequence controller.
Non-Functional Testing
Checks how well and at which level of quality the system performs its intended tasks.
The system is tested for efficiency, compatibility, usability, reliability, security, maintainability and portability.
Given below are types of non-functional tests:
Load Testing
Checks system behavior with increasing load (e.g.: parallel user, database transactions).
Performance Testing
Measures the processing speed and response time for specific use cases.
Volume/ Mass/ Stress Testing
Checks system behavior for varying amounts of data and when overloaded
Security Testing
Observes the system against unauthorized system or data access.
Reliability and Stability Testing
Testing the system under continuous use.
Compatibility and Data Conversion Testing
Observes the level of interoperability of the system with other systems and processes.
Robustness Testing
Tests error handling and recovery with misuse, programming faults, hardware failure, etc.
Configuration Testing
Observes the system against unauthorized system or data access.
Usability Testing
Determines if the system is easy to use and learn.
Documentation Checks
Checks if the system behavior matches the documentation.
Changeability and Maintainability Testing
Checks how up-to-date and accurate the documentation, system structure, etc. are.
Challenges:
Load, performance, volume and other, similar tests are, extensive and long-term and therefore, would slow down the CI process.
Robustness, hardware failure, recovery and other, similar tests are difficult to automate as they require manual intervention during the environment setup.
Usability, documentation and code maintainability tests are run manually with extensive reviews.
Non-Functional requirements should be addressed early on in the project.
If feedback from Non-Functional testing is not available on a timely manner, comprehensive refactoring of the system may be required later on.
Solutions for addressing the issue are given below:
Designing, automation and performance of tests to check non-functional attributes of features during the Sprint.
Performing robustness, hardware failure and other, similar tests as exploratory tests.
Performing user-friendliness, documentation, code maintainability and other, similar tests continuously from an early stage using pair programming.
Conducting regular reviews of the test results.
Automated Acceptance Testing
When designing a set of acceptance tests (acceptance test suite), the Product Owner can use already automated unit or integration tests that cater to a specific approval test requirement.
Product Owner may opt for automated system testing instead of testing manually.
When selecting the tests he should focus on the acceptance criteria that need to be covered.
And should only perform manual testing for the criteria that are not covered by the existing automated tests.
System Test Management
Address system testing tasks (e.g.: writing test cases, setting up the environment and automation) in the Sprint Planning and allocate effort for those tasks.
Individual tests and test results should be reviewed periodically.
Test cases should be updated and replaced continuously.
All team members should be aware of the test results.
A dedicated test manager can be elected to examine the quality of tests, interpret results, defining the team’s testing tasks, etc.
Daily bugs should be analyzed prior to the daily standup so bugs that require dedicated tasks can be decided quickly.
If there are manual tests to be performed, the test manager should which are to be run as regression tests and how frequently.
If there’s a large number of manual tests, they should be selected based on risk assessment.
Manual tests should be monitored, and pair programming, predefined tests and checklists will facilitate that.
The test framework should be built step by step including only the components that are needed immediately.