Chapter 1: Fundamentals of Testing (Part 2)

repeated

When the developer fixes the defect, test activities have to be ____________

re-testing

_________ is done by the tester to confirm that the defect in the work product was fixed.

regression testing

Testers also conduct _______________ to check if the modification made by the developer or introduced/uncovered any other issues in the same work product and related work products

repeating test activities as a result of debugging

- Schedules need to be identified to accommodate retests (Rework).

- Metrics such as Defect Rates, Defect Density,, Mean Time Between Failure, Defect Turnaround times can be used to track information and aid teams to improve work or to augment/support manual testing with test automation.

Developers

Component Testing and Component Integration Testing is generally done by the ____________.

Unit Testers

If a requirement for test independence exists, Unit Testing and Component Integration testing may be done by testers with specialized skills and are called _____________.

Sufficient Debugging

______________ is to be done before testing can begin - The work product should be in a sufficiently stable state for testing to commence or progress from the previous testing checkpoint.

debugging

Though there is some testing involved during ____________ the software, it may not verify or validate the validate the code against the requirement. This is why testing is further required.

Developers

_____________ are one of the key stakeholder for test teams. They need clear, concise defect reports and any information from the testers that helps them reproduce and analyze the defect (so they can quickly isolate and fix the cause.)

Developers

Testers rely on the _____________ to be informed about which version of the work product and when the defect fix will be available so that they can plan their retest and regression testing effort.

Collaboration between Testers and Developers

• Communication between the testers and developers should be neutral and fact focused.

• Testers and developers should collaborate with each other to increase the level of quality in a work product and the system.

In Agile development, as an example, Testers may parallelly conduct testing while the code is being created by the developer. These Unit testers

• in Agile Development, pair with the developers to create and test the components while they are being created. (Pair Testing)

• In development methodologies which employ Continuous Integration, testers could be involved with designing and writing component tests and acceptance test scenarios and translate them to automated test scripts and to enable Continuous Testing for Continuous Delivery (DevOps)

Collaboration between Testers and Developers

Effective debugging before testing allows developers to fix any issues or defects in the software, making it more stable and reliable for testing. By fixing errors early in the development process, developers can reduce the time and effort required for testing and ensure that the software is of higher quality.

TESTING

- Testing starts with known conditions, uses predefined procedures and has predictable outcomes.

- Testing can be done by testers, developers and any stakeholder throughout all stages of development and the entire life cycle of the product.

- Testing can and should be planned, designed and scheduled to demonstrate project goals.

- The objects of testing can be any work product that is being tested, such as documents and code.

- Test Cases can be created in advance for the desired/planned Test Coverage and measured against those criteria during Test Execution.

- Testing is a demonstration of error or apparent correctness.

- Testing proves a program's failure.

- Testers use varied tools for varied purposes during testing.

- Testing can be done without detailed knowledge of code or design.

- Testing can be done by independent test teams outside the organization.

- Test execution and design can be automated.

DEBUGGING

- Debugging starts from possibly unknown initial conditions and the end cannot be predicted except statistically.

- Software debugging is done by programmers themselves; they modify the program (design, coding) and eliminate errors mainly during the development stage.

- Duration of debugging or the procedure cannot be constrained.

- The object of debugging is Code.

- Debugging cannot be planned for or done in advance.

- Debugging is a deductive process.

- Debugging is the programmer's correction (Justification).

- The main tools that developers use for debugging are Debuggers.

- Debugging is impossible without detailed design knowledge.

- Debugging must be done by developers within the development organization.

- Automation of Debugging may not be possible since reliable tools to use Code to fix Code are still a work in progress.

PROCESS

TESTING IS NOT AN ACTIVITY - IT IS A ____________.

before and after

Testing involves activities both ___________ executing tests

(Source: ISTQB Certified Tester Foundation Level Syllabus, 1.1. What is Testing?)

"Software testing is a way to assess the quality of the software and to reduce the risk of software failure in operation."

(Source: ISTQB Certified Tester Foundation Level Syllabus, 1.1. What is Testing?)

• Testing is an integral part of the software development process. •Testing systematically explores a work product, a component or a system with the main aim of finding and reporting defects in the test object by designing and executing test cases against the work products allowing for their timely correction and the prevention of failures.

• Testing evaluates the quality characteristics of the work products and provides test metrics/reports to other stakeholders so they can take informed decisions.

• Debugging is essential before testing begins because it helps to ensure that the software is as error-free as possible, reducing the risk of inaccurate test results and potential issues for the end-users.

test objectives

When clear ___________ are identified in alignment with project goals and effective testing is performed according to a well defined test strategy, testing can significantly contribute to project success

How much Testing to do?

- Find all faults by testing everything?

- To test everything is rarely possible

- The time required makes it impractical

- The resource commitment required makes it impractical

How much testing is enough?

- How do we know when to stop?

- If we stop testing too soon we risk errors in the live system

- If we carry on testing we can delay the product launch

- This may lose revenue and damage the corporate image

TRUE

TRUE or FALSE

Exhaustive Testing is IMPOSSIBLE !!

Prioritization

____________ in software testing involves assessing and ranking test activities, test cases, or defects based on certain factors to optimize testing efforts and resources.

Some factors for prioritization

- RISK: The likelihood and potential consequences of failure in development or production.(example: cost of failure, historical information,)

- BUSINESS VALUE: The potential impact of a defect in functionality that affects business goals, revenue, reputation, or customer satisfaction. Example: High-priority items are those that directly affect critical business functions, financial transactions, or compliance with legal/regulatory requirements.

- FUNCTIONAL IMPORTANCE: The significance of the functionality being tested in terms of meeting user requirements and system functionality. Examples: Critical functionalities or core features essential to system operation are prioritized higher than non-essential or supplementary features.

- TECHNICAL CLARITY: Technical criticality indicates aspects of the technical infrastructure that are critical for functionality, performance, reliability, security, or maintainability of the system, Example: test cases that address technical constraints or dependencies to ensure compatibility and functionality across different platforms or configurations.

Some factors for prioritization

- COMPLEXITY: In software development, complexity can refer to the intricacy of software code, algorithms, or data structures. Complex codebases are often characterized by a large number of functions, classes, or modules, intricate control flows, nested conditionals, or convoluted logic, which can make the code difficult to understand, test, debug, or maintain. This also applies to complex hardware and networking configurations.

- REGRESSION IMPACT: Consider areas that change regularly and how often modifications, updates, or changes occur in a particular system, software, or project. These changes may break or change existing functionality. Impact analysis and change management are vital to safely and successfully implement changes with minimum disruption caused by side effects of these changes (Regression)

- VISIBILITY OF FAILURE: Visibility of failure refers to how noticeable or impactful a failure or defect would be if it occurred in the software system. Example: Defects that may cause damage to reputation or user experience.

- TIME AND BUDGET: The availability of time and resources allocated for testing within project schedules and deadlines. includes deadlines, milestones, and project schedules. Resources encompass the people, tools, equipment, materials, and budget required to complete project tasks.

higher priority tasks

PRIORITY AND DEPENDENCY

typically scheduled earlier to ensure they are completed on time

lower priority tasks

PRIORITY AND DEPENDENCY

may be scheduled later on deprioritized if time constraints are tight

test dependencies

testers also need ____________ when defining the order of test execution

test dependencies

refer to the scenario where the execution or outcome of one test is reliant on the execution or outcome or outcome of another test. Dependencies can arise due to share data, configuration states, or sequentila processing needs, test environment availability etc.

test dependencies

can significantly affect test prioritization by influencing the order in which test need to be executed to ensure accurate and meaningful results

Example:

- Sequential dependency: test that have dependencies must be executed in a specific order to ensure they produce valid results

- Dependency on defect resolution: if there are any unresolved defects that prevent the execution of some tests

Testing, as a part of software development, plays a crucial role in establishing confidence in work products and contributes to overall success of various processes, products, and projects.

- Testing provides a cost-effective means of detecting defects.

These defects can then be removed (by debugging - a non-testing activity), so testing indirectly contributes to higher quality test objects.

- Testing provides a means of directly evaluating the quality of a test object at various stages in the SDLC. These measures are used as part of a larger project management activity, contributing to decisions to move to the next stage of the SDLC, such as the release decision.

- Testing provides users with indirect representation on the development project.Testers ensure that their understanding of users' needs are considered throughout the development lifecycle.

The alternative is to involve a representative set of users as part of the development project, which is not usually possible due to the high costs and lack of availability of suitable users.

- Testing may also be required to meet contractual or legal requirements, or to comply with regulatory standards.

cost savings

TESTING'S CONTRIBUTION TO SUCCESS

Detecting and fixing issues during the testing phase is generally more cost-effective than addressing them later in the development or production cycle.

Improper testing or insufficient attention paid to testing during development may result in a significant increase in cost of defects and costs associated with failures (examples)

INTERNAL FAILURE

• Significant expense associated with correcting an error or a defect found later in software development VS testing done at the earliest possible opportunity to prevent defects or failures

• Cost of Scrap, rework, Product Recall, Warranty Claims, system downtime for defect fixing (loss in revenue)

EXTERNAL FAILURE

•Loss of quality and customer dissatisfaction in software systems which in turn can impact revenue

• Significant financial damages when a system fails during operational use (Example: software used in Banks).

• Worst case scenario is that software defects may also cause loss of life (Example: Aeronautical Software).

demonstration of quality

TESTING'S CONTRIBUTION TO SUCCESS

Testing provides information about the level of quality of a test object by appraising the quality of a test object through various test levels so that the stakeholders can take informed decisions (such as the release decision.)

TESTING PROVIDES FEEDBACK REQUIRED BY STAKEHOLDERS

DECISION SUPPORT SYSTEM FOR STAKEHOLDERS

- DEVELOPERS

- PROJECT MANAGERS

- PROJECT SPONSORS

- END USERS

- REGULATORY AUTHORITIES

STAKEHOLDERS' EXPECTATIONS

the criteria by which a stakeholder will consider the project a success.

Challenge: a myriad of questions.

100.0%

ICH

representation of end users and ensuring customer satisfaction

TESTING'S CONTRIBUTION TO SUCCESS

Testers bring a unique perspective as end users to the development lifecycle.

They provide feedback to the development team by conducting tests for realistic scenarios from an End User's perspective expected from live use ensuring that the software meets user needs, is intuitive to use, and provide a positive user experience ensure that their understanding of users' needs are considered throughout the development lifecycle and hence ensuring the success of delivery.

Testers view the system through the perspective of an end user - a Tester aims to understand the expectations of an end user, seeks to understand and clarify how the expectation is going to be satisfied by the system and reports on how well the system can satisfy the users.

testing demonstrates adherence contractual/legal requirments

TESTING'S CONTRIBUTION TO SUCCESS

Testing may also be required to meet contractual or legal requirements, or to comply with regulatory standards.

• Testing teams can analyze work products to identify and determine whether they meet industry standards or best practices

• When contractual or regulatory requirements exist, testing providing insights into how well they are meeting the established criteria.

TESTERS ARE FACT FINDERS AND FACT PRESENTERS

INC.

• To present the right kind of information, appropriate metrics need to be identified and tracked throughout the software development process.

• Metrics help ensure that testing activities align with the overall project goals and objectives, validating that the testing process is contributing to project success.

• Metrics also play a crucial role in assessing, improving, and ensuring the effectiveness of the testing process itself against the test objectives.

Testing, as a part of software development, plays a crucial role in establishing confidence in work products and contributes to overall success of products, and projects.

• Having testers involved in requirements reviews or user story refinement could detect defects in these work products. The identification and removal of requirements defects reduces the risk of incorrect or untestable features being developed.

• Having testers work closely with system designers while the system is being designed can increase each party's understanding of the design and how to test it. This increased understanding can reduce the risk of fundamental design defects and enable tests to be identified at an early stage.

• Having testers work closely with developers while the code is under development can increase each party's understanding of the code and how to test it. This increased understanding can reduce the risk of defects within the code and the tests.

• Having testers verify and validate the software prior to release can detect failures that might otherwise have been missed, and support the process of removing the defects that caused the failures (i.e., debugging). This increases the likelihood that the software meets stakeholder needs and satisfies requirements.

In addition to these examples, the achievement of defined test objectives contribute to overall software development and maintenance success.

testers are fact finders and fact presenters

• During testing, Testers find and report defects in work products before they become failures.

• When defects are fixed, the likelihood of failure is reduced.

• Testing provides information about software quality (or the lack of it) through metrics.

• Depending on when testing started and the metrics chosen, testing can provide consistent information to various stakeholders about aspects such as quality of work products and amount of work 'Done'.

By providing timely feedback, which is crucial for identifying and rectifying issues early in the development process Testing can not only ensure that potential problems are addressed promptly but also enhances the overall quality and reliability of the product.

quality

the degree to which a work product satisfies stated and implied needs of its stakeholders

quality characteristics

a category of quality attributes that bears on work product quality

synonyms: quality attribute, software quality characteristics, software product characteristic

product-based quality

a view of quality measured by the degree that well-defined quality characteristics are met. With these characteristics are not met, the product is deemed 'low' quality or 'unfit'

software quality characteristics

- accuracy

- reliability

- efficiency

- security

- maintainability

quality risk

A product risk related to a quality characteristic. (A risk associated with quality characteristics absent or not fulfilled by a product)

• Product Quality Risks should be identified as early as possible in the SDLC

Measures should be taken to prevent or mitigate these Risks throughout the development process.

• Testing only demonstrates how much Quality has been achieved.

reliability

QUALITY RISK IDENTIFICATION EXAMPLE:

a system or functionality first has to be correct (Correctness/Accuracy). ________ is a degree to which a system/functionality consistently delivers the correct outcome each time it is used.

"Data in the reports should always be correct" - implies correct dataretrieved with each transaction and no data corruption.

QUESTIONS: How many times are these reports generated by an employee in a day? How many of these employees generating reports exist in the organization?

efficiency

QUALITY RISK IDENTIFICATION EXAMPLE:

Degree to which the system/functionality is reliable AND fulfils expectation with less time/resources.

TRUE

TRUE or FALSE

while the phrase Quality Assurance is generally used for Testing, Quality Assurance (QA) and Testing are NOT the same

quality management

Testing and Quality Assurance (QA) are under a large umbrella of _______________ in an organization

quality management

Quality Assurance (QA) and Quality Control are a part of _________________

quality

is of utmost importance in any product. It is an aim to assure that product delivers as expected

quality

may be achieved by implementing and following three standard and established procedures, namely: Quality Assurance, Quality Control, and Testing

quality management

the act of overseeing different activities and tasks within an organization to ensure that products and services offered, as well as the means used to provide them, are consistent. It helps to achieve and maintain a desired level of quality within the organization

QUALITY ASSURANCE: PROCESS FOCUSED - PREVENTATIVE

focuses on the implementation and improvement of processes. It works on the basis that if a good process is followed correctly, then it will generate a good product.

QA applies to both the development and testing processes, and is the responsibility of everyone on a project.

• typically focused on adherence to proper processes, in order to provide confidence that the appropriate levels of quality will be achieved.

• When processes are carried out properly, the work products created by those processes are generally of higher quality, which contributes to defect prevention.

• In addition, the use of root cause analysis to detect and remove the causes of defects, along with the proper application of the findings of retrospective meetings to improve processes, are important for effective quality

assurance.

For example, in DevOps, the role of QA in Continuous Delivery (CD) pipeline is to ensure Continuous Testing happens. QA monitors the effectiveness of this process and identifies opportunities for Continuous Improvement of the processes involved (CI-CD)

quality assurance

activities focused on providing confidence that quality requirements will be fulfilled

- all those planned actions for (product/ service) development process used to fulfill the requirements for quality

- activities: implementing and directing a quality policy, quality objectives, quality planning, quality control, quality assurance, and quality improvement for an organization

quality control

activities designed to evaluate the quality of a component or system

-overseeing the product development process to ensure that QA procedures and standards are being followed

- involves various activities, including test activities

QUALITY CONTROL: PRODUCT FOCUSED - CORRECTIVE

focuses on activities supporting the achievement of appropriate levels of quality.

• involves various activities, including test activities, that support the achievement of appropriate levels of quality.

hecking and prod

• Testing is a major form of quality control, while others include formal methods (model checking and proof of correctness), simulation and prototyping.

• Test activities are part of the overall software development or maintenance process.

ding and proof

• Since quality assurance is concerned with the proper execution of the entire process, quality assurance supports proper testing.

TESTING'S CONTRIBUTION TO QUALITY

Example for QC,

Testers working as a part of technical teams in DevOps are involved early in the product development process (as part of continuous collaboration within the team) starting at the requirements phase (where they can flesh out any ambiguity in requirements) by ensuring the whole team has clarity on how the feature will be developed, tested, deployed, and monitored post-launch.

CONTINUOUS TESTING becomes part of everyone's responsibility.

'QUALITY' IS EVERYONE'S RESPONSIBILITY.

test results

used by QA and QC. In QC they are used to fix defects, while in QA they provide feedback on how well the development and test processess are performing

TESTING'S CONTRIBUTION TO QUALITY

Quality Characteristics to be measured by testing

1. functional suitability

2. reliability

3. efficiency

4. usability

5. security

6. compatibility

7. maintainability

8. portability

correctness

___________ of a software system (or any work product) is the first step towards achieving quality. This is achieved by identifying defects during testing and subsequently correcting them

root cause analysis

the use of ____________ to detect and remove the causes of defects, along with the proper application of the findings of retrospective meetings to improve processes, are important for effective quality assurance

error

a human action that produces an incorrect result (ISTQB Glossary)

A person can make an error (mistake), which can lead to the introduction of a defect (fault or bug) in the software code or in some other related work product. An error that leads to the introduction of a defect in one work product can trigger an error that leads to the introduction of a defetc in a related work product.

Causes for Errors

• Time pressure

• Human fallibility

• Inexperienced or insufficiently skilled project participants

Miscommunication between project participants, including miscommunication about requirement and design

• Complexity of the code, design, architecture, the underlying problem to be solved, and/or the technologies used

a srnten nu stete

• Misunderstandings about intra-system and inter-system interfaces, especially when such intra system and inter- system interactions are large in number

• New, unfamiliar technologies

In addition to failures caused due to defects in the code, failures can also be caused by environmental conditions. For example, radiation, electromagnetic fields, and pollution can cause defects in firmware or influence the execution of software by changing hardware conditions.

defect

An imperfection or deficiency in a work product where it does not meet its requirements or specifications (ISTQB Glossary) - (also).. an internal fault of the system.

A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition.

• A defect, if encountered during execution, may cause a failure of the component

or system (Internal or External Failures)

• Testing uncovers defects that may otherwise would have been missed so that they can be fixed.

• Undetected defects may cause failures.

When testing systematically explores software products throughout the SDLC to expose defects in those work products and they are corrected, it increases the quality of software.

failure

An event in which a component or system does not perform a required function within it's specified limits.

"A deviation of the component or system from its expected delivery, service or result." (ISTQB Glossary)

Inevitably, ALL Failures are caused by Faults.

• Failures are caused by Defects

• Not all defects cause failures.

DRA

failure

"A deviation of the component or system from its expected delivery, service or result"

Example: poor client satisfaction because of usability testing

failure

can also be caused by environmental conditions and inadequate attention to software quality during its development (and testing).

Example: performance degradation, security vulnerabilities

failure

may also be caused by an improperly configured tests during testing or the software environment at operation.

It is important for testers to verify their testware before test execution begins

• Unrealistic or unarticulated project goals • Inaccurate estimates of needed resources

• Badly defined system requirements

• Poor reporting of the project's status

• Unmanaged risks

Poor communication among customers, developers, and users

• Use of immature technology

• Inability to handle the project's complexity

• Sloppy development practices

Poor project management

Stakeholder politics

• Commercial pressures

Failures may also be caused by:

root cause

a source of a defect such that if it is removed, the occurrence of the defect type is decreased

root cause

a fundamental reason for the occurrence of a problem (e.g., a situation that leads to to an error)

root cause analysis

an analysis technique at identifying the root cause of defects.

By directing corrective measures at root causes, it is hoped that the likelihood of defect recurrence will be minimized

root cause analysis

defects can be analyzed to identify their root causes, so as to reduce or prevent the occurrence of similar defects in the future

root cause analysis

can help organizations find and address problems earlier through continuous learning and process improvements in the Software Development Life Cycle (SDLC).

Improved processes mean finding Defects and fixing them earlier in the development cycle that can help enterprises create better-quality software faster and in a more cost-effective way.

This also promotes customer satisfaction and protects the company's reputation.

• Improved Software Quality

Accelerated time to market

• Improved User Experience

• Reduced Cost

root cause analysis key benefits in software development include:

TESTING AS A RISK MITIGATION ACTIVITY TO PREVENT FAILURES OR UNDESIRABLE CONSEQUENCES AND TO IMPROVE QUALITY.

focused on identifying defects or potential issues in a system before it becomes operational or a work product before it is used for the next stage of implementation.

• Testing can uncover vulnerabilities, assess the reliability of the software, and ensure that it meets the specified requirements.

Testing can minimize the risk of encountering failures, improving the overall quality and performance of the System Under Test.

• By identifying and resolving potential problems early in the development process, testing can contribute to

a more robust and dependable solution, ultimately reducing the likelihood of failure or negative impact on users or the business.