Читать книгу: «Software Testing Foundations»

About the Authors

Andreas Spillner is emeritus professor of computer science at the University of Applied Sciences Bremen. During the 1990s and early 2000s he spent 10 years as spokesman for the TAV (Test, Analysis, and Verification) group at the Gesellschaft für Informatik (German Computer Science Society) that he also helped to found. He is a founder member of the German Testing Board and was made an honorary member in 2009. He was made a fellow of the Gesellschaft für Informatik in 2007. His software specialty areas are technology, quality assurance, and testing.

Tilo Linz is co-founder and a board member of imbus AG, a leading software testing solution provider. He has been deeply involved in software testing and quality assurance for more than 25 years. As a founding member and chairman of the German Testing Board and a founding member of the International Software Testing Qualifications Board, he has played a major role in shaping and advancing education and training in this specialist area both nationally and internationally. Tilo is the author of Testing in Scrum (published by Rocky Nook), which covers testing in agile projects based on the foundations presented in this book.

Andreas Spillner · Tilo Linz

Software Testing Foundations

A Study Guide for the Certified Tester Exam

■ Foundation Level

■ ISTQB® Compliant

5th, revised and updated Edition

Andreas Spillner · andreas.spillner@hs-bremen.de

Tilo Linz · tilo.linz@imbus.de

Editor: Dr. Michael Barabas / Christa Preisendanz

Translation and Copyediting: Jeremy Cloot

Layout and Type: Josef Hegele

Production Editor: Stefanie Weidner

Cover Design: Helmut Kraus, www.exclam.de

Printing and Binding: mediaprint solutions GmbH, 33100 Paderborn, and Lightning Source®, Ingram Content Group.

Bibliographic information published by the Deutsche Nationalbibliothek (DNB)

The Deutsche Nationalbibliothek lists this publication in the Deutsche

Nationalbibliografie; detailed bibliographic data can be found on the Internet at http://dnb.dnb.de.

ISBN dpunkt.verlag:

ISBN Rocky Nook:

5th, revised and updated edition 2021 Copyright © 2021 dpunkt.verlag GmbH

Wieblinger Weg 17

69123 Heidelberg

Title of the German Original: Basiswissen Softwaretest

Aus- und Weiterbildung zum Certified Tester – Foundation Level nach ISTQB®-Standard

6., überarbeitete und aktualisierte Auflage 2019

ISBN 978-3-86490-583-4

Distributed in the UK and Europe by Publishers Group UK and dpunkt.verlag GmbH.

Distributed in the U.S. and all other territories by Ingram Publisher Services and Rocky Nook, Inc.

Many of the designations in this book used by manufacturers and sellers to distinguish their products are claimed as trademarks of their respective companies. Where those designations appear in this book, and dpunkt.verlag was aware of a trademark claim, the designations have been printed in caps or initial caps. They are used in editorial fashion only and for the benefit of such companies, they are not intended to convey endorsement or other affiliation with this book. No part of the material protected by this copyright notice may be reproduced or utilized in any form, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission of the copyright owner. While reasonable care has been exercised in the preparation of this book, the publisher and author assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

This book is printed on acid-free paper.

Printed in Germany and in the United States.

5 4 3 2 1 0

Preface to the 5th Edition

Bestseller

The first edition of the book was published in German at the end of 2002. Since then, Basiswissen Softwaretest has been the best-selling book on software testing in the German-speaking world.

This 5th edition in English has been comprehensively revised and updated. It is based on the latest (6^th) edition of the German-language book and the current 2018 ISTQB® Certified Tester – Foundation Level syllabus.

The Certified Tester training scheme

The Certified Tester qualification scheme is extremely successful and is widely recognized and accepted within the IT industry. It has become the de facto global standard for software testing and quality assurance education. By the end of 2020 there were over 955,000 exams taken and more than 721,000 certifications issued in 129 countries around the world [URL: ISTQB]. Many IT employment ads for beginners and experienced workers reflect this, and certified training is often an obligatory requirement. The Certified Tester scheme is also part of the curriculum at many universities and technical colleges.

Grass-roots knowledge required in the IT world

In spite of this rapid development, there is a lot of the grass-roots knowledge in the field of computer science that doesn’t change very much over the years. We take the Foundations part of our book title seriously and don’t discuss topics that have yet to be proven in everyday practice. Specialist topics such as web app or embedded system testing are not part of these foundations.

What’s new?

This 5th edition of Software Testing Foundations has been comprehensively revised and extended, and its content brought completely up to date.

Side notes are not part of the official syllabus

The latest revision of the ISTQB® syllabus has seen some test techniques shifted to higher training levels, so these are no longer part of the Foundations syllabus. However, we have kept the corresponding sections in the book and have highlighted them as side notes. If you are using the book exclusively for exam preparation you can simply skip the side note sections.

New test techniques included

Many readers have told us that they use the book for reference in their everyday work scenarios. This is why we have included a number of additional test techniques that do not appear in the Foundations syllabus. These include techniques such as pair-wise testing that weren’t covered in previous editions.

The case study that illustrates the implementation of the test techniques has been adapted and comprehensively updated.

We have revised the lists of standards to reflect the changes made by the introduction of ISO 29119, and all the URLs referenced in the text have been updated too.

Online resources

Any future changes to the syllabus and the glossary that affect the book text can be found on our website [URL: Softwaretest Knowledge], where you will also find exercises that relate to the individual chapters in the book. Any necessary corrections or additions to the book text are also made available at the website.

Thanks

For a book like this, success is rarely down to the authors alone, and we would like to thank all our colleagues at the German Testing Board and the International Software Testing Qualifications Board, without whom the Certified Tester program would never have achieved the global success that it enjoys. Many thanks also to Hans Schaefer, our co-author of the previous four editions of the book, for his constructive cooperation.

We would further like to thank our readers for their many comments and reviews, which have encouraged us during our work and motivated us to keep getting better. Heartfelt thanks also go to our editor Christa Preisendanz and the entire team at dpunkt.verlag for years of successful cooperation.

We wish all our readers success in the practical implementation of the testing approaches described in the book and—if you are using the book to prepare for the Certified Tester Foundation Level exam—we wish you every success in answering the exam questions.

Andreas Spillner and Tilo Linz

May 2021

Foreword by Yaron Tsubery

The software systems industry continues to grow rapidly and, especially over the last two decades, exponentially. Market requirements and a growing appetite for exciting new challenges have fuelled the development of new software technologies. These new opportunities affect almost everyone on our planet and reach us primarily via the internet and, subsequently, via smart devices and technologies.

The need for software that is easy to create and maintain has caused many key industries—such as health, automotive, defense, and finance— to open up and become visible to the world via applications and/or web interfaces. Alongside these traditional domains, new types of services (such as social media and e-commerce) have appeared and thrived on the global market. The rapid growth and enormous demands involved in introducing new software-based products that greatly impact our lifestyles and our wellbeing require new and faster ways of producing software solutions.

This situation has created a market in which multiple companies compete for market share with extremely similar products. Such competition is beneficial to consumers (i.e., software users) and, as a result, software-based products have started to become commoditized. Software manufacturers have begun to think more economically, generating increased revenues using fewer resources (i.e., doing more with less). Continual introduction of new products into our daily lives has given rise to the “agile” design and production ethos—driving a cultural change in the tradition software development life cycle, as well as pushing forward the necessity of more and early automatic tests (e.g. as driven by the DevOps movement)—that is increasingly commonplace in today’s software industry, while the business leaders behind software-based products have understood that the world is becoming smaller and that competition is getting fiercer all the time. An increasingly short time to market is essential not only for generating revenue, but also simply to survive in today’s market. Successful and innovative companies understand that they need to put the customer first if they want to maintain product quality, generate brand loyalty, and increase their market share. In other words, the software industry has understood the importance of the customer to the overall product life cycle.

We in the software testing business have always known the importance of quality to the customer, because part of our job is to represent the customer’s point of view. The challenges we face have grown with the complexity of software products, and we sometimes still find ourselves having to justify the necessity for software testing, even if it has become a largely standard practice within the software industry. Recently, the rise of software-based artificial intelligence (AI)—such as software enhancement in robots and autonomous devices—has created a whole new set of challenges.

Software testing is an extremely important factor in the industry. Alongside controlling costs and quality, the main issue is customer focus. Preserving a healthy balance between cost and quality is an essential customer requirement, making it critical to have well-trained and highly professional people assigned to quality and software testing roles. Recruiting skilled professionals is the key to success. The primary factors we look for when recruiting are related to a person’s knowledge and skills. We look at the degree to which a person is aligned with the software testing profession, and with the required technology and industry domain (such as web, mobile, medical devices, finance, automotive, and so on). We also have to ask ourselves whether a person is suited to work in the product domain itself (for example, when candidates come from competitors). Communications and soft skills that fit in with the team/group/company are important too. In the case of industry newcomers, we have to consider how much potential a person has. This book teaches the fundamentals of software testing and provides a solid basis for enhancing your knowledge and experience through constant learning from external sources, your own personal experience, and from others.

When reading an educational book, I expect it to be sequentially structured and easy to understand. This book is based on the Certified Tester Foundation Level (CTFL) syllabus, which is part of the ISTQB® (International Software Testing Qualifications Board) education program. The ISTQB® has created a well-organized and systematic training program that is designed to teach and qualify software testers in a variety of roles and domains. One of the primary objectives of the ISTQB® program is to create professional and internationally accepted terminology based on knowledge and experience. The chapters in the book are designed to take you on that journey and provide you with the established and cutting-edge fundamentals necessary to becoming a successful tester. They combine comprehensive theory with detailed practical examples and side notes that will enhance and broaden your view of software systems and how to test them. This book provides a great way to learn more about software testing for anyone who is studying the subject, thinking about joining the software testing profession, or for newcomers to the field.

For those who already have a role in software testing, the practical examples provided (based on a case study and corresponding side notes) are sure to help you learn. They provide a great basis for comparison with and application to your own real-world projects. This book contains a wealth of great ideas that will help you to build and improve your own software testing skills. The new, revised edition is based on the latest (2018) ISTQB® CTFL, which has been updated to cover agile processes and experience gained from changes that have taken place within the industry over the last few years. It also includes references to the other syllabi and professional content upon which it is based, and an updated version of the case study introduced in earlier editions. The case study is based on a multilayer solution that includes both specific and general technical aspects of software system architecture. The case study in this edition is based on a new-generation version of the system detailed in previous editions, thus enabling you to learn from a practical, project-based viewpoint.

The world is changing fast every day. Some of the technologies that we use today will become obsolete within a few years and the products we build will probably become obsolete even sooner. Software is an integral and essential part of virtually all the technology that surrounds us. Along with growth and expansion in the artificial intelligence (AI) arena and other new technologies that have yet to be introduced, this continual change offers new and exciting opportunities for the software testing profession. We are sure to find ourselves tuning our knowledge and experience in various ways, and we may even find ourselves teaching and coaching not only humans but also machines and systems that test products for us.

The fundamental knowledge, grass-roots experience, and practical examples provided by this book will prepare you for the ever-changing world and will shape your knowledge to enable you to test better and, in the future, perhaps pass on your knowledge to others.

I wish you satisfying and fruitful reading.

Yaron Tsubery

Former ISTQB® President

President ITCB®

1 Introduction

Software is everywhere! Nowadays there are virtually no devices, machines, or systems that are not partially or entirely controlled by software. Important functionality in cars—such as engine or gear control—have long been software-based, and these are now being complemented by increasingly smart software-based driver assist systems, anti-lock brake systems, parking aids, lane departure systems and, perhaps most importantly, autonomous driving systems. Software and software quality therefore not only govern how large parts of our lives function, they are also increasingly important factors in our everyday safety and wellbeing.

Equally, the smooth running of countless companies today relies largely on the reliability of the software systems that control major processes or individual activities. Software therefore determines future competitiveness. For example, the speed at which an insurance company can introduce a new product, or even just a new tariff, depends on the speed at which the corresponding IT systems can be adapted or expanded.

High dependency on reliable software

Quality has therefore become a crucial factor for the success of products and companies in the fields of both technical and commercial software.

Most companies have recognized their dependence on software, whether relying on the functionality of existing systems or the introduction of new and better ones. Companies therefore constantly invest in their own development skills and improved system quality. One way to achieve these objectives is to introduce systematic software evaluation and testing procedures. Some companies already have comprehensive and strict testing procedures in place, but many projects still suffer from a lack of basic knowledge regarding the capacity and usefulness of software testing procedures.

Grass-roots knowledge of structured evaluation and testing

This book aims to provide the basic knowledge necessary to set up structured, systematic software evaluation and testing techniques that will help you improve overall software quality.

This book does not presume previous knowledge of software quality assurance. It is designed for reference but can also be used for self-study. The text includes a single, continuous case study that provides explanations and practical solutions for each of the topics covered.

This book is aimed at all software testers in all types of companies who want to develop a solid foundation for their work. It is also for programmers and developers who have taken over (or are about to take over) existing test scenarios, and it is also aimed at project managers who are responsible for budgeting and overall procedural improvement. Additionally, it offers support for career changers in IT-related fields and people involved in application approval, implementation, and development.

Especially in IT, lifelong learning is essential, and software testing courses are offered by a broad range of companies and individuals. Universities, too, are increasingly offering testing courses, and this book is aimed at teachers and students alike.

Certification program for software testers

The ISTQB® Certified Tester program is today seen as the worldwide standard for software testing and quality assurance training. The ISTQB® (International Software Testing Qualifications Board) [URL: ISTQB] coordinates qualification activities in individual countries and ensures the global consistency and comparability of the syllabi and exam papers. National Testing Boards are responsible for publishing and maintaining local content as well as the organization and supervision of exams. They also approve courses and offer accreditation for training providers. Testing Boards therefore guarantee that courses are of a consistently high standard and that participants end up with an internationally recognized certificate. Members of the Testing Boards include training providers, testing experts from industrial and consulting firms, and university lecturers. They also include representatives from trade associations.

Three-stage training scheme

The Certified Tester training scheme is made up of units with three levels of qualification. For more details, see the ISTQB® [URL: ISTQB] website. The basics of software testing are described in the Foundation Level syllabus. You can then move on to take the Advanced Level exam, which offers a deeper understanding of evaluation and testing skills. The Expert Level certificate is aimed at experienced software testing professionals, and consists of a set of modules that cover various advanced topics (see also section 6.1.2). In addition, there are syllabi for agile software development (foundation and advanced level) as well as special topics from the testing area (for example, Security Tester, Model-Based Tester, Automotive Software Tester).

This book covers the contents of the Foundation Level syllabus. You can use the book for self-study or in conjunction with an approved course.

Chapter overview

The topics covered in this book and the basic content of the Foundation Certificate course are as follows:

Software testing basics

Chapter 2 discusses the basics of software testing. Alongside the concepts of when to test, the objectives to aim for, and the required testing thoroughness, it also addresses the basic concepts of testing processes. We also talk about the psychological difficulties that can arise when you are looking for errors in your own work.

Lifecycle testing

Chapter 3 introduces common development lifecycle models (sequential, iterative, incremental, agile) and explains the role that testing plays in each. The various test types and test levels are explained, and we investigate the difference between functional and non-functional testing. We also look at regression testing.

Static testing

Static testing (i.e., tests during which the test object is not executed) are introduced in Chapter 4. Reviews and static tests are used successfully by many organizations, and we go into detail on the various approaches you can take.

Dynamic testing

Chapter 5 addresses testing in a stricter sense and discusses “black-box” and “white-box” dynamic testing techniques. Various test techniques and methods are explained in detail for both. We wrap up this chapter by looking at when it makes sense to augment common testing techniques using experience-based or intuitive testing techniques.

Test management

Chapter 6 discusses the organizational skills and tasks that you need to consider when managing test processes. We also look at the requirements for defect and configuration management, and wind up with a look at the economics of testing.

Test tools

Testing software without the use of dedicated tools is time-consuming and extremely costly. Chapter 7 introduces various types of testing tools and discusses how to choose and implement the right tools for the job you are doing.

Most of the processes described in this book are illustrated using a case study based on the following scenario:

Case Study: Virtual-ShowRoom VSR-II

A car manufacturer has been running an electronic sales system called VirtualShowRoom (VSR) for over a decade. The system runs at all the company’s dealers worldwide:

■ Customers can configure their own vehicle (model, color, extras, and so on) on a computer, either alone or assisted by a salesperson. The system displays the available options and immediately calculates the corresponding price. This functionality is performed by the DreamCar module.

■ Once the customer has selected a configuration, he can then select optimal financing using the EasyFinance module, order the vehicle using the JustInTime module, and select appropriate insurance using the NoRisk module. The FactBook module manages all customer and contract data.

The manufacturer’s sales and marketing department has decided to update the system and has defined the following objectives:

■ VSR is a traditional client-server system. The new VSR-II system is to be web-based and needs to be accessible via a browser window on any type of device (desktop, tablet, or smartphone).

■ The DreamCar, EasyFinance, FactBook, JustInTime, and NoRisk modules will be ported to the new technology base and, during the process, will be expanded to varying degrees.

■ The new ConnectedCar module is to be integrated into the system. This module collects and manages status data for all vehicles sold, and communicates data relating to scheduled maintenance and repairs to the driver as well as to the dealership and/or service partner. It also provides the driver with various additional bookable services, such as a helpdesk and emergency services. Vehicle software can be updated and activated “over the air”.

■ Each of the five existing modules will be ported and developed by a dedicated team. An additional team will develop the new ConnectedCar module. The project employs a total of 60 developers and other specialists from internal company departments as well as a number of external software companies.

■ The teams will work using the Scrum principles of agile development. This agile approach requires each module to be tested during each iteration. The system is to be delivered incrementally.

■ In order to avoid complex repeat data comparisons between the old and new systems, VSR-II will only go live once it is able to duplicate the functionality provided by the original VSR system.

Within the scope of the project and the agile approach, most project participants will be confronted or entrusted with test tasks to varying degrees. This book provides the basic knowledge of the test techniques and processes required to perform these tasks. Figure 1-1 shows an overview of the planned VSR-II system.

Fig. 1-1 VSR-II overview

Certified Tester syllabus and exam

The appendices at the end of the book include references to the syllabus and Certified Tester exam, a glossary, and a bibliography. Sections of the text that go beyond the scope of the syllabus are marked as side notes.

The book’s website

The book’s website [URL: Softwaretest Knowledge] includes sample exam questions relating to each chapter, updates and addenda to the text, and references to other books by authors whose work supports the Certified Tester training scheme.

Web-based Training System vsr.testbench.com

We have put a free implementation of VSR-II as a test object online for training purposes¹. It reproduces the VSR-II examples included in the book on a realistic, executable system, so you can “test” live to find the software bugs hidden in VSR-II by applying the test techniques presented in the book. It takes just a few mouse clicks to get started:

1. Open your browser and load vsr.testbench.com

2. Create your personal VSR-II training workspace

3. Log into your VSR-II workspace and start

Fig. 1-2 VSR-II Training System Login-Screen

Also included in your registration for a VSR-II training workspace is a free basic license for the test management system TestBench CS, which includes the VSR-II test specification as a demo project and several of the VSR-II test cases presented in the book.

You can use TestBench CS not only for learning and training, but also for efficient testing of your own “real” software. A description of all features can be found at [URL: TestBench CS].

Many thanks to our colleagues at imbus Academy, imbus JumpStart and imbus TestBench CS Development Team for this awesome implementation of the VSR-II Case Study as a web-based training system.

2 Software Testing Basics

This introductory chapter explains the basic concepts of software testing that are applied in the chapters that follow. Important concepts included in the syllabus are illustrated throughout the book using our practical VSR-II case study. The seven fundamental principles of software testing are introduced, and the bulk of the chapter is dedicated to explaining the details of the testing process and the various activities it involves. To conclude, we will discuss the psychological issues involved in testing, and how to avoid or work around them.

2.1 Concepts and Motivations

Quality requirements

Industrially manufactured products are usually spot-checked to make sure they fulfill the planned requirements and perform the required task. Different products have varying quality requirements and, if the final product is flawed or faulty, the production process or the design has to be modified to remedy this.

Software is intangible

What is generally true for industrial production processes is also true for the development of software. However, checking parts of the product or the finished product can be tricky because the product itself isn’t actually tangible, making “hands-on” testing impossible. Visual checks are limited and can only be performed by careful scrutiny of the development documentation.

Faulty software is a serious problem

Software that is unreliable or that simply doesn’t perform the required task can be highly problematic. Bad software costs time and money and can ruin a company’s reputation. It can even endanger human life—for example, when the “autopilot” software in a partially autonomous vehicle reacts erroneously or too late.

Testing helps to assess software quality

It is therefore extremely important to check the quality of a software product to minimize the risk of failures or crashes. Testing monitors software quality and reduces risk by revealing faults at the development stage. Software testing is therefore an essential but also highly complex task.

Case Study: The risks of using faulty software

Every release of the VSR-II system has to be suitably tested before it is delivered and rolled out. This aims to identify and remedy faults before they can do any damage. For example, if the system executes an order in a faulty way, this can cause serious financial problems for the customer, the dealer and the manufacturer, as well as damaging the manufacturer’s image. Undiscovered faults like this increase the risk involved in running the software.

Testing involves taking a spot-check approach

Testing is often understood as spot-check execution¹ of the software in question (the test object) on a computer. The test object is fed with test data covering various test cases and is then executed. The evaluation that follows checks whether the test object fulfills its planned requirements.²

Testing involves more than just executing tests on a computer

However, testing involves much more than just performing a series of test cases. The test process involves a range of separate activities, and performing tests and checking the results are just two of these. Other testing activities include test planning, test analysis, and the design and implementation of test cases. Additional activities include writing reports on test progress and results, and risk analysis. Test activities are organized differently depending on the stage in a product’s lifecycle. Test activities and documentation are often contractually regulated between the customer and the supplier, or are based on the company’s own internal guidelines. Detailed descriptions of the individual activities involved in software testing are included in sections 2.3 and 6.3.

Static and dynamic testing

Alongside the dynamic tests that are performed on a computer (see Chapter 5), documents such as requirement specifications, user stories, and source code also need to be tested as early as possible in the development process. These are known as static tests (see Chapter 4). The sooner faults in the documentation are discovered and remedied, the better it is for the future development process, as you will no longer be working with flawed source material.

Verification and validation

Testing isn’t just about checking that a system fulfills its requirements, user stories, or other specifications; it is also about ensuring that the product fulfills the wishes and expectations of its users in a real-world environment. In other words, checking whether it is possible to use the system as intended and making sure it fulfills its planned purpose. Testing therefore also involves validation (see Principle #7 in section 2.1.6—“Absence-of-errors is a fallacy”).

No large system is fault-free

There is currently no such thing as a fault-free software system, and this situation is unlikely to change for systems above a given degree of complexity or those with a large number of lines of code. Many faults are caused by a failure to identify or test for exceptions during code development—things like failing to account for leap years, or not considering constraints when it comes to timing or resource allocation. It is therefore common—and sometimes unavoidable—that software systems go live, even though faults still occur for certain combinations of input data. However, other systems work perfectly day in day out in all manner of industries.

Freedom from faults cannot be achieved through testing

With the exception of very small programs, even if every test you perform returns zero defects, you cannot be sure that additional tests won’t reveal previously undiscovered faults. It is impossible to prove complete freedom from faults by testing.

2.1.1 Defect and Fault Terminology

The test basis as a starting point for testing

A situation can only be classed as faulty if you define in advance what exactly is supposed to happen in that situation. In order to make such a definition, you need to know the requirements made of the (sub)system you are testing as well as other additional information. In this context, we talk about the test basis against which tests are performed and that serves as the basis for deciding whether a specific function is faulty.

What counts as a defect?

A defect is therefore defined as a failure to fulfill a predefined requirement, or a discrepancy between the actual behavior (at run time or while testing) and the expected behavior (as defined in the specifications, the requirements, or the user stories). In other words, when does the system’s behavior fail to conform to its actual requirements?

Unlike physical systems, software systems don’t fail due to age or wear. Every defect that occurs is present from the moment the software is coded, but only becomes apparent when the system is running.

Faults cause failures

System failures result from faults and only become apparent to the tester or the user during testing or at run-time. For example, when the system produces erroneous output or crashes.

We need to distinguish between the effects of a fault and its causes. A system failure is caused by a fault in the software, and the resulting condition is considered to be a defect. The word “bug” is also used to describe defects that result from coding errors, such as an incorrectly programmed or forgotten instruction in the code.

Defect masking

It is possible that a fault can be offset by one or more other faults in other parts of the program. Under these circumstances, the fault in question only becomes apparent when the others have been remedied. In other words, correcting a fault in one place can lead to unexpected side effects in others.

Not all faults cause system failures, and some failures occur never, once, or constantly for all users. Some failures occur a long way from where they are caused.

A fault is always the result of an error or a mistake made by a person— for example, due to a programming error at the development stage.

People make errors

Errors occur for various reasons. Some typical (root) causes are:

■ All humans make errors!

■ Time pressure is often present in software projects and is a regular source of errors.

■ The complexity of the task at hand, the system architecture, the system design, or the source code.

■ Misunderstandings between participants in the project—often in the form of differing interpretations of the requirements or other documents.

■ Misunderstandings relating to system interaction via internal and external interfaces. Large systems often have a huge number of interfaces.

■ The complexity of the technology in use, or of new technologies previously unknown to project participants that are introduced during the project.

■ Project participants are not sufficiently experienced or do not have appropriate training.

A human error causes a fault in part of the code, which then causes some kind of visible system failure that, ideally, is revealed during testing (see figure 2-1: Debugging, see below). Static tests (see Chapter 4) can directly detect faults in the source code.

System failures can also be caused by environmental issues such as radiation and magnetism, or by physical pollution that causes hardware and firmware failures. We will not be addressing these types of failures here.

Fig. 2-1 The relationships between, errors, faults, and failures

False positive and false negative results

Not every unexpected test result equates to a failure. Often, a test will indicate a failure even though the underlying fault (or its cause) isn’t part of the test object. Such a result is known as a “false positive”. The opposite effect can also occur—i.e., a failure doesn’t occur even though testing should reveal its presence. This type of result is known as a “false negative”. You have to bear both of these situations in mind when evaluating your test results. Your result can also be a “correct positive” (failure revealed by testing) or a “correct negative” (expected behavior confirmed by testing). For more detail on these situations, see section 6.4.1.

Learning from your mistakes

If faults and the errors or mistakes that cause them are revealed by testing it is worth taking a closer look at the causes in order to learn how to avoid making the same (or similar) errors or mistakes in future. The knowledge you gain this way can help you optimize your processes and reduce or prevent the occurrence of additional faults.

Case Study: Vague requirements as a cause of software faults

Customers can use the VSR EasyFinance module to calculate various vehicle- financing options. The interest rate the system uses is stored in a table, although the purchase of vehicles involved in promotions and special offers can be subject to differing interest rates.

VSR-II is to include the following additional requirement:

REQ: If the customer agrees to and passes an online credit check, the EasyFinance module applies an interest rate from a special bonus interest rate table.

The author of this requirement unfortunately forgot to clarify that a reduction in the interest rate is not permissible for vehicles sold as part of a special offer. This resulted in this special case not being tested in the first release. In turn, this meant that customers were offered low interest rates online but were charged higher rates when billed, resulting in complaints.

2.1.2 Testing Terminology

Testing is not debugging

In order to remedy a software fault it has to be located. To start with, we only know the effect of the fault, but not its location within the code. The process of finding and correcting faults is called debugging and is the responsibility of the developer. Debugging is often confused with testing, although these are two distinct and very different tasks. While debugging pinpoints software faults, testing is used to reveal the effect a fault causes (see figure 2-1).

Confirmation testing

Correcting a fault improves the quality of the product (assuming the correction doesn’t cause additional, new faults). Tests used to check that a fault has been successfully remedied are called confirmation tests. Testers are often responsible for confirmation testing, whereas developers are more likely to be responsible for component testing (and debugging). However, these roles can change in an agile development environment or for other software lifecycle models.

Unfortunately, in real-world situations fault correction often leads to the creation of new faults that are only revealed when completely new input scenarios are used. Such unpredictable side effects make testing trickier. Once a fault has been corrected you need to repeat your previous tests to make sure the targeted failure has been remedied, and you also need to write new tests that check for unwanted side effects of the correction process.

Objectives of testing

Static and dynamic tests are designed to achieve various objectives:

■ A qualitative evaluation of work products related to the requirements, the specifications, user stories, program design, and code

■ Prove that all specific requirements have been completely implemented and that the test object functions as expected for the users and other stakeholders

■ Provide information that enables stakeholders to make a solid estimate of the test object’s quality and thus generate confidence in the quality provided³

■ The level of quality-related risk can be reduced through identification and correction of software failures. The system will then contain fewer undiscovered faults.

■ Analysis of the program and its documentation in order to avoid unwanted faults, and to document and remedy known ones

■ Analyze and execute the program in order to reproduce known failures

■ Receive information about the test object in order to decide whether the component in question can be committed for integration with other components

■ Demonstrate that the test object adheres and/or conforms to the necessary contractual, legal and regulatory requirements and standards

Objectives depend on context

Test objectives can vary depending on the context. Furthermore, they can vary according to the development model you use (agile or otherwise) and the level of test you are performing—i.e., component, integration, system, or acceptance tests (see section 3.4).

When you are testing a component, your main objective should be to reveal as many failures as possible and to identify (i.e., debug) and remedy the underlying faults as soon as possible. Another primary objective can be to select tests that achieve the maximum possible level of code coverage (see section 2.3.1).

One objective of acceptance testing is to confirm that the system works and can be used as planned, and thus fulfills all of its functional and non-functional requirements. Another is to provide information that enables stakeholders to evaluate risk and make an informed decision about whether (or not) to go live.

Side Note: Scheme for naming different types of testing

The various names used for different types of tests can be confusing. To understand the naming of tests it is useful to differentiate between the following naming categories:

1. Test objective

The naming of a test type is based on the test objective (for example, a “load test”).

2. Test method/technique

A test is named according to the method or technique used to specify and/or perform the test (i.e., “state transition testing”, as described in section 5.1.3)

3. Test object

A test is named according to the type of object to be tested (for example, “GUI test“ or “database test“)

4. Test level

A test is named according to the corresponding level of the development model being used (for example, a “system test“)

5. Test person

A test is named after the person or group who perform the test (for example, “developer test“, “user test“)

6. Test scope

A test is named according to its scope (for example, a “partial regression test“)

As you can see, not all of these terms define a distinct type of test. Instead, the different names highlight different aspects of a test that are important or in focus in a particular context or with regard to a particular testing objective.

2.1.3 Test Artifacts and the Relationships Between Them

The previous sections have already described some types of test artifacts. The following sections provide an overview of the types of artifacts necessary to perform dynamic testing.

Test basis

The test basis is the cornerstone of the testing process. As previously noted, the test basis comprises all documents that help us to decide whether a failure has occurred during testing. In other words, the test basis defines the expected behavior of the test object. Common sense and specialist knowledge can also be seen as part of the test basis and can be used to reach a decision. In most cases a requirements document, a specification, or a user story is available, which serves as a test basis.

Test cases and test runs

The test basis is used to define test cases, and a test run takes place when the test object is fed with appropriate test data and executed on a computer. The results of the test run are checked and the team decides whether a failure has occurred—i.e., whether there is a discrepancy between the test object’s expected and actual behaviors. Usually, certain preconditions have to be met in order to run a test case—for example, the corresponding database has to be available and filled with suitable data.

Test conditions

An individual test cannot be used to test the entire test basis, so it has to focus on a specific aspect. Test conditions are therefore extrapolated from the test basis in order to pursue specific test objectives (see above). A test condition can be checked using one or more tests and can be a function, a transaction, a quality attribute, or a structural element of a component or system. Examples of test conditions in our case study VSR-II system are vehicle configuration permutations (see section 5.1.5), the look and feel of the user interface, or the system’s response time.

Test item

By the same token, a test object can rarely be tested as a complete object in its own right. Usually, we need to identify separate items that are then tested using individual test cases. For example, the test item for VSR-II’s price calculation test condition is the calculate_price() method (see section 5.1.1). The corresponding test cases are specified using appropriate testing techniques (see Chapter 5).

Test suites and test execution schedules

It makes little sense to perform test cases individually. Test cases are usually combined in test suites that are executed in a test cycle. The timing of test cycles is defined in the test execution schedule.

Test scripts

Test suites are automated using scripts that contain the test sequence and all of the actions required to create the necessary preconditions for testing, and to clean up once testing is completed. If you execute tests manually, the same information has to be made available for the manual tester.

Test logs

Test runs are logged and recorded in a test summary report.

Test plan

For every test object, you need to create a test plan that defines everything you need to conduct your tests (see section 6.2.1). This includes your choice of test objects and testing techniques, the definition of the test objectives and reporting scheme, and the coordination of all test-related activities.

Figure 2-2 shows the relationships between the various artifacts involved. Defining the individual activities involved in the testing process (see section 2.3) helps to clarify when each artifact is created.

Fig. 2-2 The relationships between test artifacts

2.1.4 Testing Effort

Testing effort depends on the project (environment)

Testing takes up a large portion of the development effort, even if only a part of all conceivable tests—or, more precisely, all conceivable test cases— can be considered. It is difficult to say just how much effort you should spend testing, as this depends very much on the nature of the project at hand.⁴

The importance of testing—and thus the amount of effort required for testing—is often made clear by the ratio of testers to developers. In practice, the following ratios can be found: from one tester for every ten developers to three testers per developer. Testing effort and budget vary massively in real-world situations.

Case Study: Testing effort and vehicle variants

VSR-II enables potential customers to configure their own vehicle on a computer screen. The extras available for specific models and the possible combinations of options and preconfigured models are subject to a complex set of rules. The old VSR System allowed customers to select combinations that were not actually deliverable. As a consequence of the VSR-II QA/Test planning requirement Functional suitability/DreamCar = high (see below) customers should no longer be able to select non-deliverable combinations.

The product owner responsible for the DreamCar module wants to know how much testing effort will be required to test this aspect of the module as comprehensively as possible. To do this, he makes an estimate of the maximum number of vehicle configuration options available. The results are as follows:

There are 10 vehicle models, each with 5 different engine options; 10 types of wheel rims with summer or winter tires; 10 colors, each with matt, glossy, or pearl effect options; and 5 different entertainment systems. These options result in 10×5×10×2×10×3×5=150.000 different variants, so testing one variant every second would take a total of 1.7 days.

A further 50 extras (each of which is selectable or not) produce a total of 150.000×2⁵⁰ = 168.884.986.026.393.600.000 variations.

The product owner intuitively knows that he doesn’t have to test for every possible combination, but rather for the rules that define which combinations of options are not deliverable. Nevertheless, possible software faults create the risk that the DreamCar module wrongly classifies some configurations as deliverable (or permissible combinations as non-deliverable).

How much testing effort is required here and how much can it effectively cost? The product owner decides to ask the QA/testing lead for advice. One possible solution to the issue is to use pairwise testing (see the side note in section 5.1.5).

Side Note: When is increased testing effort justifiable?

Is a high testing effort affordable and justifiable? Jerry Weinberg’s response to this question is: “Compared with what?” [DeMarco 93]. This response points out the risks of using a faulty software system. Risk is calculated from the likelihood of a certain situation arising and the expected costs when it does. Potential faults that are not discovered during testing can later generate significant costs.

Example: The cost of failure

In March 2016, a concatenation of software faults destroyed the space telescope Hitomi, which was built at a cost of several hundred million dollars. The satellite’s software wrongly assumed that it was rotating too slowly and tried to compensate using countermeasures. The signals from the redundant control systems were then wrongly interpreted and the speed of rotation increased continuously until the centrifugal force became too much and Hitomi disintegrated (from [URL: Error Costs]).

In 2018 and 2019 two Boeing 737 MAX 8 airplanes crashed due to design flaws in the airplane’s MCAS flight control software [URL: MAX-groundings]. Here too, the software—misdirected by incorrect sensor information—generated fatal countermeasures.

Testing effort has to remain in reasonable proportion to the results testing can achieve. “Testing makes economic sense as long as the cost of finding and remedying faults is lower than the costs produced by the corresponding failure occurring when the system is live.”⁵ [Pol 00]. Reasonable testing effort therefore always depends on the degree of risk involved in failure and an evaluation of the danger this incurs. The price of the destroyed space telescope Hitomi could have paid for an awful lot of testing.

Case Study: Risks and losses when failures occur

The DreamCar module constantly updates and displays the price of the current configuration. Registered customers with validated ID can order a vehicle online.

Once a customer clicks the Order button and enters their PIN, the vehicle is ordered and the purchase committed. Once the statutory revocation deadline has passed, the chosen configuration is automatically passed on to the production management system that initiates the build process.

Because the online purchase process is binding, if the system calculates and displays an incorrect price the customer has the right to insist on the paying that price. This means that wrongly calculated prices could lead to the manufacturer selling thousands of cars at prices that are too low. Depending on the degree of miscalculation, this could lead to millions of dollars in losses. Having each purchase order checked manually is not an option, as the whole point of the VSR-II system is that vehicles can be ordered completely automatically online.

Defining test thoroughness and scope depending on risk factors

Systems or system parts with a high risk have to be tested more extensively than those that do not cause major damage in case of failure.⁶ Risk assessment has to be carried out for the individual system parts or even for individual failure modes. If there is a high risk of a system or subsystem malfunctioning, the test requires more effort than for less critical (sub) systems. These procedures are defined through international standards for the production of safety-critical systems. For example, the [RTC-DO 178B] Airborne Systems and Equipment Certification standard prescribes complex testing procedures for aviation systems.

Although there are no material risks involved, a computer game that saves scores incorrectly can be costly for its manufacturer, as such faults affect the public acceptance of a game and its parent company’s other products, and can lead to lost sales and damage to the company’s reputation.

2.1.5 Applying Testing Skills Early Ensures Success

Testing is an important factor in any success story

In software projects, it is never too early to begin preparing your tests. The following examples illustrate the benefits of involving testers with appropriate test knowledge in individual activities within the software development life cycle:

Close cooperation between developers and testers throughout the development process

■ If testers are involved in checking the requirements (for example, using reviews, detailed in section 4.2) or refining user stories, they can use their specialist knowledge to find and remedy ambiguities and faults very early in the work product. The identification and correction of flawed requirements reduces the risk of producing inappropriate or non-testable functionality.

■ Close cooperation between testers and systems designers at the design stage helps all those involved to better understand the system’s design and the corresponding tests. This increased awareness can reduce the risk of producing fundamental construction faults and makes it easier to design appropriate tests—for example, to test interfaces during integration testing (see section 3.4.2).

■ Developers and testers who work together at the coding stage have a better understanding of the code itself and the tests it requires. This reduces the risk of producing faulty code and of designing inappropriate tests (see false negatives in section 6.4.1).

■ If testers verify and validate software before release, they are sure to identify and remedy additional faults that would otherwise remain undiscovered. This increases the probability that the product fulfills its requirements and satisfies all of its stakeholders.

In addition to these examples, achieving the previously defined test objectives will also aid successful software development and maintenance.

2.1.6 The Basic Principles of Testing

The previous sections addressed software testing, whereas the following section summarize the basics of testing in general. These are guidelines that have developed over decades of testing experience.

1. Principle #1:

Testing shows the presence of defects, not their absence

Testing establishes the presence of defects and reveals the faults that cause them. Depending on the effort made and the thoroughness of the tests involved, testing reduces the probability of leaving undiscovered faults in the test object. However, testing does not enable us to prove that a test object contains no faults. Even if an object doesn’t fail during testing, this is no proof of freedom from faults or overall correctness.

2. Principle #2:

Exhaustive testing is impossible

With the exception of extremely simple or trivial test objects, it is impossible to design and perform a complete test suite that covers all possible combinations of input data and their preconditions. Tests are always samples, and the effort allocated to them depends on the risks they cover and the priority assigned to them.

3. Principle #3:

Early testing saves time and money

Dynamic and static testing activities should be defined and begun as early as possible in the system’s lifecycle. The term “shift left” implies early testing. Early testing reveals faults at an early stage of the development process. In a software context, this helps to avoid (or at least reduce) the increasingly costly repair of faults later in the development lifecycle.

4. Principle #4:

Defects cluster together

Generally speaking, defects are not evenly distributed throughout a system. Most defects can usually be found in a small number of modules, and this (estimated or observed) clustering effect can be used to help analyze risk. Testing effort can then be concentrated on the most relevant parts of the system (see also principle #2 above).

5. Principle #5:

Beware the pesticide paradox

Over time, tests become less effective the same way insects develop resistance to pesticides. If you repeat tests on an unchanged system, they won’t reveal any new failures. In order for your tests to remain effective you need to check your test cases regularly and, if necessary, modify them or add new ones. This ensures that you test previously unchecked components and previously untested input data, thus revealing any failures that these produces. The pesticide paradox can have a positive effect too. For example, if an automated regression test reveals a low number of failures, this may not be the result of high software quality but rather due to the ineffectiveness of the (possibly outdated) test cases in use.

6. Principle #6:

Testing is context-dependent

Tests need to be adapted to the proposed purpose and the surrounding environment of the system in question. No two systems can be effectively tested the same way. Testing thoroughness, exit criteria, and other parameters need to be defined uniquely according to the system’s working environment. An embedded system requires different tests than, for example, an e-commerce system. Testing in an agile project will be very different from that in a project based on a sequential life-cycle model.

7. Principle #7:

Absence-of-errors is a fallacy

Even if you test all requirements comprehensively and correct all the faults you find, it is still possible to develop a system that is difficult to use, that doesn’t fulfill the user’s expectations, or that is simply of poor quality compared with other, similar systems (or earlier versions of the same system). Prototyping and early involvement of a system’s users are preventive measures used to avoid this problem.

Side Note
2.2 Software Quality

Software testing serves to identify and remedy failures and increase software quality. Test cases should be chosen to mirror the subsequent real-world use that the system is designed for. The quality that testing verifies should then equate to the quality of the user experience.

However, software quality is about more than just correcting the faults found during testing.

2.2.1 Software Quality according to ISO 25010

ISO 25010: Quality in Use and Product Quality models

According to the ISO 25010 standard [ISO 25010], software quality can be classified in two major ways⁷:

■ The Quality in Use Model, and

■ The Product Quality Model

The quality in use model comprises the following five characteristics:

■ Effectiveness

■ Efficiency

■ Satisfaction

■ Freedom from risk

■ Context coverage

The product quality model comprises eight characteristics:

■ Functional suitability

■ Performance efficiency

■ Compatibility

■ Usability

■ Reliability

■ Security

■ Maintainability

■ Portability

The product quality model has the most similarities with the previous ISO 9126 standard. Details of the Data Quality Model can be found in the ISO 25012 standard [ISO 25012].

In order to effectively judge the quality of a software system, all of these characteristics and quality criteria need to be considered during testing. The level of quality that each characteristic of the test object is intended to fulfill has to be defined in advance in the quality requirements. The fulfillment of these requirements and their criteria then has to be checked using appropriate tests.

Forty (sub-)characteristics

ISO 25010 breaks down the 13 quality characteristics listed above into a total of 40 further sub-characteristics. It is beyond the scope of this text to go into detail on all 40 sub-characteristics of the quality in use and product quality models. More details are available online at [ISO 25010]. Some of the more important characteristics are summarized below:

Functional suitability/functionality

The functional suitability (or, more simply, functionality) of the product quality model covers all the characteristics involved in describing the planned functionality of the system.

A quality characteristic is divided into three sub-characteristics:

■ Functional completeness

Does the function set cover all specified tasks and user objectives?

■ Functional correctness

Does the product/system deliver correct results with the required degree of accuracy?

■ Functional appropriateness

To what degree do the available functions fulfill the required tasks and specified objectives?

Appropriate tests can be used to check whether specified and implicit requirements are mirrored in the available functionality, thus answering the questions posed above.

Functionality is usually described in terms of specified input/output behavior and/or a specific system reaction to specified input. Tests are designed to demonstrate that each required functionality has been implemented in such a way that the specified input/output behavior or system behavior is complied with.

Reliability

The reliability aspect of the product quality model describes a system’s ability to perform at a specific level under specified circumstances for a specified period of time.

This quality characteristic has four sub-characteristics:

■ Maturity

To what degree does a system, product, or component provide the required reliability under normal operating conditions?

■ Availability

Is the system, product, or component always ready for use, and how easily is it available when it is required?

■ Fault tolerance

How well does the system, product, or component function in spite of the presence of known hard- or software faults?

■ Recoverability

How long does it take to recover specific data and normal service following a system or product failure or crash?

Satisfaction

The satisfaction aspect of the quality in use model addresses the degree to which user needs are fulfilled when the product or system is used under specified circumstances.

This quality characteristic has four sub-characteristics:

■ Usefulness

How happy is the user with the perceived fulfillment of pragmatic objectives, including the results and consequences of using the system?

■ Trust

How certain is the user (or other stakeholder) that the product or system will behave as intended?

■ Pleasure

How much pleasure does the user experience when using the system to fulfill his/her personal requirements?

■ Comfort

How comfortable does the user find the system—also in terms of physical well- being?

Most of the characteristics of the quality in use model have a strong personal element and can only be objectively viewed and precisely evaluated under exceptional circumstances. In order to test this quality characteristic you will need to refer to multiple users (or user groups) in order to obtain usable test results.

A software system cannot fulfill all quality characteristics to the same degree. Fulfilling one characteristic often means not fulfilling another. A highly efficient software system is not easily portable, as its developers will have designed it to utilize specific attributes of the platform it runs on.

Prioritizing quality characteristics

It is therefore necessary to prioritize these characteristics. The resulting priorities will also act as a guideline for the testing thoroughness for each characteristic.

Case Study: Applying ISO 25010 to VSR-II

The VSR-II testing/QA lead suggests using the product quality model described in ISO 25010 to the project steering committee. The committee agrees and asks the testing/QA lead to prepare a concept paper on how to apply the standard in the context of the VSR-II project. The core of the draft is a matrix that illustrates the relevance of each quality attribute to each product component and which interpretations to apply. The initial draft of the matrix looks like this:

Table 2-1 Classifying quality characteristics

These risk classifications are to be interpreted relative to one another and are justified by the testing/QA lead for each quality characteristic, e.g.:

■ Functional suitability/all modules

Every module serves large numbers of users and processes a lot of data, so functional failures have the potential to produce considerable costs. The requirement is therefore classified as “high” for all modules.

■ Compatibility/ConnectedCar

There are no requirements, as this module is to be built from scratch.

■ Usability/FactBook

There are no requirements, as this is a back-end module and the API already exists.

■ Portability/DreamCar

This characteristic is classified as “low” because the framework in use covers it without the application of additional measures.

Which checks and tests are required will be established during QA/test planning for each module. This top-level classification can be used to establish basic parameters—for example, automated continuous integration tests (see section 3.2) are required for a “high” attribute, a single round of acceptance testing is sufficient for “mid” attributes, while a written design guideline on how the team approaches an issue is sufficient for “low” attributes. The QA/test lead is sure to have to go through a number of assessment rounds with the teams to arrive at an agreement on these high-level rules.

2.2.2 Quality Management and Quality Assurance

QM

Quality management (QM) covers all organizational activities and measures that serve to control quality. Quality management is usually responsible for defining quality policies, quality objectives, quality planning, quality assurance, and quality improvement. This makes quality management a core management activity. Quality management is mandatory in industries such as aviation, automotive, and healthcare.

ISO 9000

The ISO 9000 family of quality management standards [ISO 9000] is widely used, and the ISO/IEC 90003 Software Engineering standard [ISO 90003] stipulates how the ISO 9001 [ISO 9001] general guidelines are applied to computer software.

QA

Quality assurance (QA) usually concentrates on measures which aim to ensure that specified procedures are applied and defined processes are adhered to. It is assumed, if a company sticks to its predefined processes, that it will fulfill the required quality characteristics and therefore achieve the specified quality levels. Under these circumstances, the results will usually show increased quality, which in turn helps to avoid failures in the work products and the corresponding documentation.

QA and testing

The term “quality assurance” is often used when referring to testing processes or even as a synonym for testing. QA and testing processes are closely related but are definitely not the same. QA generates confidence in the fulfillment of quality requirements, which can be achieved by testing. Effective QA also involves analyzing the causes of all kinds of defects, and serves to identify (test) and remedy (debug) them. The results are discussed in meetings called “retrospectives” and can be used to improve the processes involved. Testing therefore serves to demonstrate that the required quality levels are achieved.

Testing activity is part of the overall software development and maintenance process and, because QA is about making sure such processes are implemented and executed correctly, QA also supports effective testing. The following section describes the testing process itself in more detail.

2.3 The Testing Process

Development models

Chapter 3 introduces different types of software development lifecycle models (also referred to more simply as “development models”). These are designed to aid structuring, planning, and management of new or continuing software projects. In order to perform well-structured tests, you will usually need more than just a description of the activities that make up the development model. In addition to positioning testing within the overall development process, you will also need a detailed dedicated testing schedule. In other words, the content of the development task called “testing” needs to be broken down into smaller, more manageable steps.

Testing comprises a sequence of individual activities

There are many widely used and proven test activities, and a test process will be made up of these kinds of activities. You need to put together a suitable test process according to the specified (or inherited) project situation. The specific test activities you choose, and how (and when) you implement them will depend on a number of factors, and will generally be based on a company or project-specific testing strategy (see section 6.2). If you ignore certain test activities you will increase the likelihood of the test process failing to reach its objectives.

Fig. 2-3 The testing process

The main activities

A test process⁸ will generally comprise the following activities (see figure 2-3):

■ Test planning

■ Test monitoring and control

■ Test analysis

■ Test design

■ Test implementation

■ Test execution

■ Test completion

Each of these activities comprises multiple individual tasks that produce their own output and vary in nature according to the project at hand.

Fig. 2-4 The test process showing time overlap

Iterative testing

Even if the individual tasks involved in the test process are defined in a logical sequence, in practice they can (and may) overlap, and are sometimes performed concurrently (see figure 2-4). Even if you intend to perform your test activities in a predefined sequence, a sequential development model (for example, the “waterfall” model) can cause overlap, combination, concurrency, or cancellation of individual activities or parts of activities.

Adapting these activities to fit the system and project context (see below) is usually necessary, regardless of which development model you are using.

Fig. 2-5 An iterative test process

Software is often developed in small iterative steps—for example, agile development methods use continuous build and test iterations. The corresponding test activities therefore need to take place continuously and iteratively (see figure 2-5).

The following sections provide an overview of the individual test steps and their output. Test management is responsible for the monitoring and execution of most of these activities, which are described in detail in Chapter 6 (see sections 6.2 and 6.3).

2.3.1 Test Planning

Structured handling of a task as extensive as testing will not work without a plan. Test planning begins directly at the start of a software project. Like with any plan, it is necessary to review your testing plan regularly and update or adapt it to fit changing situations and project parameters. Test planning is therefore a repeat activity that is carried out and, if necessary, adjusted throughout the entire product lifecycle.

The test plan: planning the content

The main task when planning your testing is the creation of a test plan based on your chosen testing strategy. This test plan defines the test objects, the quality characteristics and the testing objectives, as well as the testing activities you plan to use to verify them. The test plan thus describes your testing technique, the required resources, and the time required to perform the corresponding test activities.

Coverage criteria

The point at which you have performed sufficient testing is determined by the planned coverage and is also part of the test plan. Such criteria are often referred to as “completion criteria” or “exit criteria” or, in the case of agile projects, “definition of done”. If coverage criteria are defined for each test level or type, you can evaluate objectively whether the tests you have performed can be seen as sufficient. Coverage criteria can also be used to monitor and control testing, and they also verify when you have reached your testing objectives.

The test plan also contains information about the test basis, which serves as the cornerstone for all your testing considerations. The test plan also needs to include information regarding traceability between the test basis and the results of your test activities. For example, this can help you to determine which changes to the test basis modify which testing activities, thus enabling you to adapt or augment them.

The test plan also defines which tests are to be performed at which test level (see section 3.4). It often makes sense to draft a separate test plan for each test level, and you can use a master test plan to aggregate these into one.

Test planning: scheduling

The test schedule contains a list of all activities, tasks and/or events involved in the test process. This list includes the planned start and end times for every activity. Interdependencies between activities are noted in the test schedule.

The plan also defines deadlines for key activities to support its practical implementation during the project. The test schedule can be part of the test plan.

2.3.2 Test Monitoring and Control

Ensuring traceability

Monitoring and control involve constant observation of the current testing activities compared with the planned activities, reporting any discrepancies, and the execution of the activities required to achieve the planned objectives under the changed circumstances. The update of the plan must also be based on the changed situation.

Are the exit criteria fulfilled?

Test monitoring and control activities are based on the exit criteria for each activity or task. The evaluation of whether the exit criteria for a test at a particular test level have been fulfilled can include:

■ Achievement of the degree of coverage defined in the test plan according to the available test results and logs. If the predefined criteria are fulfilled, the activity can be terminated.

■ The required component or system quality is determined based on test results and logs. If the required quality has been achieved, the test activity can be concluded.

■ If risk evaluation is part of the test plan and you need to prove that you have sufficient risk coverage, this can also be determined using the test results and logs.

Perform additional tests or take a risk?

If the required exit criteria have not been fulfilled by the tests you have performed, you need to design and execute additional tests. If this isn’t possible for any reason, you need to clarify the situation and evaluate the ensuing risk.

Progress and completion reports

Stakeholders expect to receive regular test progress reports on current testing progress compared with the overall plan. Alongside any deviation from the original plan, these reports should also contain information regarding any prematurely terminated tests (see above) or non-fulfillment of the planned exit criteria. Test summary reports are to be provided when project milestones are reached.

All test reports should contain details relevant to their recipients and include a progress report as well as test results. Reports should also answer or preempt management questions, such as the (expected) end time, planned vs. actual use of resources, and the amount of testing effort involved.

Progress monitoring can be based on the reports made by team members or on figures and analysis provided by automated tools.

2.3.3 Test Analysis

“What” do we need to test?

Test analysis involves determining what exactly needs to be tested. For this purpose, the test basis is examined to see whether the documents to be used are sufficiently detailed and contain testable features in order to derive test conditions. The degree to which the test conditions need to be checked is determined by measurably defined coverage criteria.

The following documents and data can be used to analyze the test basis and the planned test level:

Analyzing the test basis

■ Requirements that specify the planned functional and non-functional system or component behavior. The requirements specifications include technical requirements, functional requirements, system requirements, user stories⁹, epics¹⁰, use cases, and similar work products or documentation. For example, if a requirement doesn’t specify the predicted result and/or system behavior precisely enough, then test cases cannot be simply derived. A reworking is necessary.

Analyzing documentation

■ Design or implementation data that gives rise to specific component or system structures. System or software architecture documents, design specifications, call graphs, model diagrams (for example, UML¹¹ or entity relationship diagrams¹²), interface specifications, and similar materials can be used for the test basis. For example, you will need to analyze how easily interfaces can be addressed (interface openness) and how easily the test object can be divided into smaller sub-units in order to test these separately. You need to consider these aspects at the development stage, and the test object needs to be designed and coded accordingly.

Check the test object

■ You need to investigate the individual components or the system itself, including the code base, database metadata, database queries, and other interfaces. For example, you need to check that the code is well structured and easy to understand, and that the required code coverage (see section 5.1) is easy to achieve and verify.

Consider risk analysis

■ Risk analysis reports that cover functional, non-functional, and structural aspects of the system or its components need to be investigated too. If potential software failures create serious risks, testing needs to be correspondingly thorough. Testing can be performed less formally for software that is not mission-critical.

Potential documentation errors

The cornerstone of the entire testing process is the test basis. If the test basis contains defects, you cannot formulate “correct” test conditions and you won’t be able to draft “proper” test cases. You therefore need to analyze the test basis for defects too. Check whether it contains ambiguities, or whether there are gaps or omissions in the descriptions of functions. You need to check the documentation for inconsistencies, imprecision, contradictions, and for repeat and/or redundant passages. Any defects or discrepancies you find should be corrected immediately.

The discovery and removal of defects from the test basis is extremely important, especially if the documentation hasn’t been reviewed (see section 4.3). Development methodologies such as behavior-driven development (BDD, [URL: BDD]) and acceptance test-driven development (ATDD, [URL: ATDD]) use acceptance criteria and user stories to create test conditions and test cases before coding begins. This approach makes it simpler to identify and remedy defects at a much earlier stage in the development process.

Prioritizing tests

Once the test conditions have been identified and defined, you need to prioritize them. This ensures that the most important and high-stakes test conditions are tested first. In real-world situations, time restrictions often make it impossible to perform all the planned tests.

Traceability is important

At the planning stage, you need to ensure that there is unambiguous bi-directional traceability between the test basis and the results of your testing activities (see above). This traceability has to precisely define which test condition checks which requirement and vice versa.

This is the only way to ensure that you can later determine how thoroughly which requirements need to be tested and—depending on the test conditions—using which test cases.

Choosing test techniques

It is useful to consider which test technique to use (i.e., black-box, white-box, experience-based, see Chapter 5) at the analysis stage. Each technique has its own system for reducing the likelihood of overlooking test conditions and helping to define these precisely. For example, using an equivalence partition (or “equivalence class”) test (see section 5.1.1) ensures that the entire input domain is used for the creation of test cases. This prevents you from forgetting or overlooking negative input in your requirements definitions. You can then define your test conditions to cover negative input data.

If you use an experience-based testing technique (see section 5.3), you can use the test conditions defined during analysis as objectives for your test charter. A test charter is a kind of “assignments” that, alongside traditional test objectives, provides potential ideas for additional tests. If the test objectives are traceable back to the test basis, you can evaluate the achieved coverage (for example, for your requirements), even when you are using an experience-based technique.

2.3.4 Test Design

How to test and which test cases to use

When designing tests you determine how you are going to test. At the design stage, test conditions are used to create test cases (or sequences of test cases). Here, you will usually use one of the test techniques detailed in Chapter 5.

Test cases can be specified on two “levels”: abstract and concrete (see Case Study: Abstract and concrete test cases below).

Abstract and concrete test cases

An abstract (or “high-level”) test case doesn’t include specific input values or expected results, and is described using logical operators. The advantage of such cases is that they can be used during multiple test cycles and with varying data but can still adequately document the scope of each case. A concrete (or “low-level”) test case uses specific input data and expected result values.

When you begin to design a test, you can use abstract as well as concrete test cases. Because only concrete test cases can be executed on a computer, abstract test cases have to be fleshed out with real input and output values. In order to utilize the advantages of abstract test cases (see above), you can derive abstract test cases from concrete cases too.

Test cases involve more than just test data

Preconditions have to be defined for every test case. A test also requires clear constraints that must be adhered to. Furthermore, you need to establish in advance what results or behavior you expect the test to produce. In addition to output data, results include any changes to global (persistent) data and states, as well as any other reactions to the execution of the test case.

The test oracle

You need to use adequate sources to predict test results. In this context, people talk about the “test oracle” that has to be “consulted” regarding the expected results. For example, specifications can serve as an oracle. There are two basic approaches:

■ The tester derives the expected output value from the input value based on the test object’s requirements and specifications as defined in the test basis.

■ If the inverse of a function exists, you can execute the inverse and compare its output with the input value for your original test case. This technique can, for example, be used when testing encryption and decryption algorithms

The following example illustrates the difference between abstract and concrete test cases.

Case Study: Abstract and concrete test cases

A dealership can give its salespeople the option of discounts to apply to the price of a vehicle. For prices below $15,000 there is no discount. For prices up to $20,000, a discount of 5% is appropriate. If the price is below $25,000, a 7% discount is possible. If the price is above $25,000, a discount of 8.5% is to be applied.

The above text enables us to derive the following relationships between price and discount:

The text itself obviously offers potential for interpretation¹³. In other words, the text can be misunderstood, whereas the mathematical formulae derived from it are unambiguous.

Based on the formulae, we can define the following test cases (see table 2-2):

Table 2-2 Abstract test cases

In order to execute these tests, the abstract cases have to be converted to concrete cases—i.e., we have to apply specific input values (see table 2-3). Exceptional conditions and boundary cases are not covered here.

Table 2-3 Concrete test cases

The values shown here serve only to illustrate the difference between abstract and concrete test cases. We didn’t use a specific test technique to design these tests, and the cases shown aren’t meant to test the discount component exhaustively. For example, there is no test case that covers false input (such as a negative price). You will find more detail on the systematic creation of test cases using a specific test technique in Chapter 5.

In addition to specifying abstract and concrete test cases, test design also includes prioritizing your test cases and providing appropriate testing infrastructure:

Priorities and traceability

■ Test analysis has already prioritized the test conditions. These same priorities can be used and fine-tuned for your test cases (or sets of test cases). This way, you can assign different priorities to the individual tests within a set that are designed to verify a single test condition. High- priority tests are executed first.

The same principle applies to the traceability of test conditions, which can be broken down to cover individual test cases or sets of cases.

Testing infrastructure and environment

■ The required testing infrastructure has to be evaluated and implemented. Test infrastructure consists of all the organizational elements required for testing. These include the test environment, testing tools, and appropriately equipped workstations. A test environment is required in order to run the test object on a computer and verify the specified test cases (see below). This environment comprises hardware, any necessary simulation equipment and software tools, and other supporting materials. In order to avoid delays while testing, the test infrastructure should be up and running (and tested) before testing begins.

Following on from the test analysis, the test design stage can reveal further defects in the test basis. Likewise, test conditions that were defined at the analysis stage can be fine-tuned at the design stage.

2.3.5 Test Implementation

Is everything ready to begin testing?

The task of test implementation is the final preparation of all necessary activities so that the test cases can be executed in the next step. Test design and test implementation activities are often combined.

Firm up the testing infrastructure

One of main tasks during implementation is the creation and integration of all testware, with special attention being paid to the details of the testing infrastructure. The test framework¹⁴ has to be programmed and installed in the test environment. Checking the environment is extremely important, as faults in the testing environment can cause system failures. In order to ensure that there are no delays or obstacles to testing, you also need to check that all additional tools (such as service virtualization, simulators, and other infrastructure elements) are in place and working.

Firm up your test cases

To ensure that your concrete test cases can be utilized without further changes or amendments, all test data have to be correctly transferred to the test environment. Abstract test cases have to be fed with specific test data.

As well as firming up your test cases, you need to define the test procedure itself—i.e., the planned tests have to be put into a logical sequence. This takes place according to the priorities you have defined (see section 6.2).

Test suite, test procedure, test script

To keep a test cycle effective and to retain a logical structure, test cases are grouped into test suites. A test suite consists of multiple tests grouped according to the planned sequence of execution. The sequence has to planned so that the postconditions of a test case serve as the preconditions for the following test case. This simplifies the test procedure, as it obviates the need for dedicated pre- and postconditions for every test case. A test suite also includes the cleanup activities required once execution is complete.

Automating test procedures using scripts saves a lot of time compared with manual testing.

The most efficient way to plan the execution of test cases, test suites, and test procedures is defined in a test execution schedule (see section 6.3.1).

The traceability of your test cases to the requirements and/or test conditions needs to be checked and, if necessary, updated. Test suites and procedures also have to be taken into account when checking traceability.

2.3.6 Test Execution

Completeness check

It makes sense to begin by checking that all the components you want to test and the corresponding testware are available. This involves installing the test object in the test environment and checking that it can be started and run. If this check reveals no obstacles, testing can begin.

Our Tip

Check the test object’s main functions

■ Test execution should begin by checking the test object’s main function (see the “smoke test” section in the Other Techniques side note in section 5.1.6). If this reveals failures or deviations from the expected results, you need to remedy the corresponding defect(s) before continuing with testing.

Tests without logs are worthless

The test execution process—whether manual or automated according to the test execution schedule—has to be precisely and completely logged. You need to log each test result (i.e., pass, fail, blocked¹⁵) so that they are comprehensible to people who are not directly involved with the test process (for example, the customer). Logs also serve to verify that the overall testing strategy (see section 6.2) has been performed as planned. Logs should show which parts were tested, when, by whom, how intensively and with what results. If a planned test case or test sequence is left out for any reason, this needs to be logged too.

The importance of clarity and reproducibility

Alongside the test object (or test item), a whole raft of documents and other information are involved in test execution. These include: the test framework, input data, logs, and so on. The data and other information that relate to a test case or test run have to be managed so that the test can easily be repeated later using the same data and constraints. The IDs and/or version numbers of the tools used should be noted and recorded by configuration management (see section 6.5).

Is there really a failure?

If there is a difference between the expected and actual result, evaluating the test logs will help you to decide whether this really is due to a failure. If you do discover a failure, make sure you document it before you begin to look for its causes. It may be necessary to specify and execute supplementary test cases. You should also report the defect. See section 6.4.1 for more on test logging, section 6.3.4 for more on the defect report, and section 6.4 for a general discussion of defect management.

Retesting

Once you have corrected a fault, you need to check that the corresponding failure has also been resolved and that no new failures have been caused by the correction process. You may need to specify new test cases to verify the modified code.

Our Tip

Check whether a defect is actually due to the test object itself

■ You will need to check carefully whether a failure really is due to the test object. There is nothing worse for a tester’s reputation than a reported failure whose cause is actually a flawed test case. At the same time, you have to take care not to be over-wary of such cases, and you mustn’t be scared to report potential failures, even if you are not sure of their cause. Both these situations are bad for the project.

Individual fault correction is not practical

■ Ideally, you will correct faults individually and retest to make sure that your corrections don’t unintentionally influence each other. However, this approach is simply not practical in the real world. If a test is executed by an independent tester rather than the developer, individual corrections won’t be possible. Reporting every fault to the developer and waiting for these to be remedied before retesting is not a justifiable effort. The usual approach is to correct bunches of faults and then set up a new version of the software for testing.

In addition to logging the differences between the expected and actual results, you need to evaluate coverage and—if necessary—log the run time of the tests. There are specialized tools available for these kinds of tasks (see Chapter 7).

Traceability is important here too

Bi-directional traceability is an important part of all the testing activities we have looked at so far. Here too, you need to check traceability and, if necessary, update the system so that the relationships between the test basis and the test conditions, cases, test runs, and results are up to date. Once a test sequence is completed, you can use its inbuilt traceability to evaluate whether every item in the test basis has been covered by a corresponding test process.

Test objectives fulfilled?

This way, you can check which requirements have passed planned and executed tests and which requirements couldn’t be verified due to failed tests or failures that were registered during testing. You may find that some requirements have not yet been verified because the corresponding tests have not yet been performed. Information of this kind enables you to verify whether the planned coverage criteria have been fulfilled and therefore whether the test can be viewed as successfully completed.

Effective coverage criteria and traceability enable you to document your test results so that they are easily comprehensible for all the project’s stakeholders.

Other common exit criteria are discussed in section 6.3.1.

2.3.7 Test Completion

The right time for test completion

Test completion is the final activity in the testing process and involves collating all the data collected by the completed test activities in order to evaluate the test experience and consolidate the testware and its associated materials. The correct moment for test completion varies according to the development model you use. It can be:

■ When the system goes live

■ The end (or discontinuation) of the test project

■ The completion of an agile project iteration (for example, as part of a retrospective or a review meeting)

■ The completion of testing activities for a specific test level

■ The completion of test activities for a maintenance release

At completion time, you also need to make sure that all planned activities have been completed and that the defect reports are complete. Open or unresolved failures (i.e., unresolved deviations from an existing requirement) remain open and are carried over to the next iteration or release. In agile environments, such unresolved cases are classed as new product backlog items for inclusion in the next iteration.

Change requests and modified requirements that stem from the evaluation of test results are handled similarly.

Test summary report

The test summary report (see section 6.3.4) aggregates all your testing activities and results, and includes an overall evaluation of the tests you have performed compared with your predefined exit criteria. The summary report is distributed to all stakeholders.

Archiving testware

Software systems are generally utilized over a long period of time, during which failures will turn up that weren’t discovered during testing. During its lifetime, a system will also be subject to change requests from its users (or customers). Both of these situations mean that the system has to be reprogrammed and the modified code has to be tested anew. A large portion of the testing effort involved in this kind of maintenance can be avoided if the testware you originally used (test cases, logs, infrastructure, tools, and so on) are still available and can be handed over to the maintenance department. This means that the existing testware only have to be adapted rather than set up from scratch when it comes to performing system maintenance. Testware can also be profitably adapted for use in similar projects. For some industries, the law requires proof of testing, and this can only be provided if all the testware are properly archived.

Our Tip

Create an image or use a Container

■ Conserving testware after testing can be extremely laborious, so testers often capture an image of the test environment or use the Docker freeware to create a so-called Container—an easily transportable and reusable file that contains all the related resources and that can be installed and run as an independent test environment.

Learning from experience

The experience you gather when testing can be analyzed and used in future projects. Deviations between your plans and the actual activities you perform are just as interesting as looking for their causes. You should use your findings to unleash your potential for improvement and make changes to the activities you undertake for future iterations, releases, and projects. These kinds of changes help the overall test process to mature.

2.3.8 Traceability

Traceability between the test basis and test results

When we talk about testing activities, we often mention the importance of traceability. The following sections summarize the traceability between the test basis and test results and detail other advantages of effective traceability.

Traceability is essential to effective test monitoring and control that spans the entire test process. It establishes the relationships between each item in the test basis and the various test results. Traceability helps you to assess the degree of coverage you achieve—for example, checking whether all requirements have been covered by at least one test case. Such traceability data helps you control the overall progress of the test process.

Over and above checking coverage, effective traceability also supports the following aspects:

■ Traceability helps to assess the impact of changes by analyzing which changes to the requirements affect which test conditions, test runs, test cases, and test items

Higher, abstract level

■ Traceability enables you to elevate your results to a “higher” abstract level, thus making the testing process more easily comprehensible to non-specialists. Considering only the executed test cases doesn’t provide information on how thorough or broad-based the testing process was. It is only when you include traceability information that the output becomes intelligible to all relevant groups of stakeholders. It also helps to fulfill abstract IT governance criteria.

■ The same argument is valid for test progress and test summary reports—i.e., effective traceability makes them easier to understand. The status of each item in the test basis should be included in reports. One of the following statements can be made for each individual requirement:

● Tests were passed

● Tests failed, failures occurred

● Planned tests have not yet been executed

■ Traceability also enables us to make technical aspects of testing comprehensible to all stakeholders. Using company objectives as a yardstick, it enables them to evaluate product quality, procedural capacity, and project progress.

Tool-based support

In order to best utilize the advantages of traceability, some companies build dedicated management systems that organize test results in a way that automatically delivers traceability data. There are various test management tools available that inherently implement traceability.

2.3.9 The Influence of Context on the Test Process

The context of the testing process

The following are some of the factors that influence the testing process within an organization (see also section 6.2.5):

■ The choice and usage of test activities depends on the software development lifecycle model you use. Most agile models don’t include detailed directions for testing and its associated activities.

■ Depending on the chosen and implemented system architecture the system may be divided into subsystems. This affects test levels (component, integration, system, and acceptance testing—see section 3.4) and the corresponding technique used to derive or select test cases (see Test Techniques in Chapter 5).

■ The test type also influences the testing process. A test type is a group of test activities designed to test a component or a system for interrelated quality characteristics. A test type is often focused on a single quality characteristic—for example, load testing (see section 3.5).

■ In the case of significant product or project risk, the test process needs to be designed to address the maximum number of risks through test cases, and thus minimize them (see section 6.2.4).

■ The context in which a software system is used also has an effect on the testing process. For example, software used exclusively for company-internal vacation planning will be subject to less thorough testing than a custom system built to control a client’s industrial facility.

■ Company guidelines and best practices can also have an effect on the testing process and the rules that govern it. The advantage here is that the factors influencing the process do not have to be discussed and redefined for each project. Required internal and external standards have the same effect.

Operational limitations

Operational limitations also affect the structure and implementation of test processes. Such limitations include:

■ Budget and other resources (for example, specialist staff) that are usually in short supply for testing purposes

■ Extended deadlines that leave less time than planned for testing

■ The complexity of the system to be tested. Complex systems are not easy to test! The test process must be designed to reflect the complexity of the system.

■ Contractual and regulatory requirements. These can have a direct effect on the scope of the testing process, and therefore on that of the individual activities involved.

2.4 The Effects of Human Psychology on Testing

Errare humanum est

Everyone makes mistakes, but nobody likes to admit it! The main objective of software testing is to reveal deviations from the specifications and/or the customer’s requirements. All failures found must be communicated to the developers. The following sections address how to deal with the psychological issues that can result.

Software development is often seen as a constructive activity while testing the product and checking the documentation are more often seen as destructive. This view often results in the people involved in a project having very different attitudes to their work. However, this assumption is not at all justifiable as, according to [Myers 12, p. 13, Software testing principle #10]: “Testing is an extremely creative and intellectually challenging task.” Furthermore, testing plays a significant role in the success of a project and the quality of the resulting product (see section 2.1).

Disclosing errors

Failures found must be communicated to the author of the faulty document or source code. Management, too, needs to know about which (or at least how many) defects have been revealed by testing. The way this communicated can be beneficial to the relationships among analysts, product owners, designers, developers, and testers, but can also have a negative effect on these important lines of communication. Admitting (or proving) errors is not an easy thing to do and requires a great deal of sensitivity.

Discovering failures can be construed as criticism of a product or its author, and can occur as a result of static or dynamic tests. Examples of such situations are:

■ A review meeting in which a requirements document is discussed

■ A meeting at which user stories are fleshed out and fine-tuned

■ During dynamic test execution

Always take confirmation bias into account

“Confirmation bias”¹⁶ is a psychological term that describes the tendency to search for, interpret, favor, and recall information in a way that confirms or strengthens your prior personal beliefs or hypotheses. This tendency plays a significant role in the communication of software faults. Software developers generally assume that their code is fault-free, and confirmation bias makes it difficult for them to admit that their work might be flawed. Other cognitive preconceptions, too, can make it tricky for those involved to understand or accept the results of software tests.

Recognized defects are a positive thing

Another thoroughly human trait is the tendency to “shoot the messenger” who brings bad news. Test results are often seen as bad news, even though the opposite is often true: recognized defects can be remedied and the quality of the test object improved. Software failures that are out in the open are actually a positive thing!

Soft skills required

In order to avoid (or at least reduce) the potential for this kind of conflict, testers and test managers need well-developed soft skills. Only then can everyone involved effectively discuss faults, failures, test results, test progress, and risk. Mutual respect always creates a positive working environment and engenders good relationships between all the people involved in the project.

Examples of positive communication:

Examples of positive communication

■ Arguments or simply “getting loud” are never good for cooperation in the workplace. To make sure work progresses amicably, it often helps to remind everyone of their mutual objectives and the high product quality they are aiming for.

■ As already mentioned, you can never repeat enough that discovering a defect and rectifying it is a positive thing. Other positive aspects are:

● Error recognition can help developers to improve their work and therefore their results too. A lot of developers are not aware that test results can help them hone their own skills.

● Discovered and rectified faults are best framed to management as a means to save time and money, and as a way to reduce the risk of poor product quality.

Keep documentation neutral and factual

■ Documentation style also plays a role in communication. Test results and other findings need to be documented neutrally and should stick to the facts. If a document is flawed, don’t criticize its author. Always write objective, fact-based defect reports and review findings.

■ Always consider the other person’s point of view. This makes it easier to understand why someone perhaps reacts negatively to something you wrote or said.

■ Misunderstandings in general and talking at cross-purposes never engender constructive communication. If you find yourself in such a situation (or expect one to arise), always ask how what you said was understood and confirm what was actually said. This works in both directions.

Section 2.1.2 lists typical testing objectives. Clearly defined objectives also have a distinct psychological effect. People often like to align their behavior to clearly defined objectives. As well as testing objectives, you can orient yourself toward team objectives, management objectives, or other stakeholders’ objectives. The important thing is that testers pursue and stick to these objectives as impartially as possible.

2.4.1 How Testers and Developers Think

Different mindsets

Because they pursue very different objectives, developers and testers usually think quite differently. Developers design and produce a product, while testers verify and validate the product with a view to discovering faults and failures. It is possible to improve product quality by combining these two different mindsets.

Assumptions and preferred decision-making techniques are reflected in the way most people think. Alongside curiosity, a tester’s mindset will usually include a healthy level of professional pessimism combined with a critical view of the test object. Interest in detail and the will to communicate positively with everyone involved are other characteristics that are useful in testers. Testers develop their soft skills through years of hands-on experience.

A developer’s main interest is in designing and implementing a solution to a problem, so he won’t be keen on considering which aspects of the solution are flawed or buggy. Although some aspects of a developer’s mindset are similar to those of a tester, the confirmation bias described above always makes it more difficult for developers to recognize errors in their own work (see also section 6.1.1).

Side Note

Can a developer change his mindset and test his own software? There is no simple answer to this question, and someone who both develops and tests a product requires a rare degree of critical distance to their own work. After all, who likes to investigate and confirm their own mistakes? Generally speaking, developers prefer to find as few faults as possible in their own code.

Developer tests

The greatest weakness of “developer tests” (see section 3.4.1) is that every developer who tests his own code will always view his work with too much optimism. It is highly likely that he will overlook meaningful test cases or, because he is more interested in programming than testing, he won’t test thoroughly enough.

Blind to your own mistakes

If a developer misunderstands the assignment and designs a program with a fundamental flaw, an appropriate test case simply won’t occur to him and he won’t find the flaw. One way to reduce this common “blindness to your own mistakes” is to work in pairs and get each developer to test the other’s work (see section 6.1, model #1).

On the other hand, inside knowledge of your own test object can be an advantage, as you don’t need to spend time getting to know the code. It is up to management to decide when the advantage of knowing your own work outweighs the disadvantages caused by blindness to your own mistakes. This decision is based on the importance of the test object and the level of risk involved if it fails.

In order to design meaningful test cases, a tester has to learn about the test object, which takes time. On the other hand, a tester has specialist skills that a developer would have to learn in order to test effectively—a learning process for which there is usually no time to spare.

Developers require testing skills, testers require development skills.

It is a great aid to cooperation and understanding between developers and testers if both acquire some knowledge to the other’s skills. Developers should always know some testing basics, and testers should always have some development experience.

2.5 Summary

■ Testing terminology is only loosely defined and similar terms are often used to mean different things—a cause of frequent misunderstandings. This is why consistent use of terminology is an important part of the Certified Tester course. The glossary at the end of this book provides an overview of all the most important terms.

■ Testing takes up a large proportion of a project’s development resources. Precisely how much testing effort is required depends on the type of project at hand.

■ The testing process—starting with planning and preparation steps—needs to be begun as early as possible in order to generate the maximum testing benefits within the project.

■ Always follow the seven basic testing principles.

Side Note

■ Testing is an important part of the quality assurance complex in the context of software development. Appropriate quality models and characteristics are defined by the international ISO 25010 standard [ISO 25010].

■ It is important to recognize and observe the connections and the boundaries between testing, quality assurance, and quality management.

■ The testing process comprises test planning, monitoring and control, analysis, design, implementation, execution, and completion. These activities can overlap and can be performed sequentially or in parallel. The overall testing process has to be adapted to fit the project at hand.

■ Bidirectional traceability between the results of the individual testing activities ensures that you can make meaningful statements about the results of the testing process, and to make a reasonable estimate of how much effort will be involved in making changes to the system. Traceability is also critical to effective test monitoring and control.

■ All of the many factors that influence testing within an organization have to be considered.

■ People make mistakes but don’t usually like to admit it! This is why psychological issues play a significant role in the overall testing process.

■ The mindsets of testers and developers are very different, but both can benefit by learning from one another.

3 Testing Throughout the Software Development Lifecycle

This chapter offers a brief introduction to common lifecycle models used in software development projects, and explains the role testing plays in each. It discusses the differences between various test levels and test types, and explains where and how these are applied within the development process.

Most software development projects are planned and executed along the lines of a software development lifecycle model that is chosen in advance. Such models are also referred to as software development process models or, more concisely, development models.

Such a model divides a project into separate sections, phases, or iterations and arranges the resulting tasks and activities in a corresponding logical order (see [URL: SW-Dev-Process]). Additionally, the model usually describes the roles that each task is assigned to and which of the project’s participants is responsible for each task. The development methods to be used in the individual phases are often described in detail too.

Every development model has its own concepts regarding testing, and these can vary widely in meaning and scope. The following sections detail popular development models from a tester’s point of view.

Types of lifecycle models

The two basic types of development model in use today are sequential and iterative/incremental. The following sections include discussion of both types.

3.1 Sequential Development Models

As the name suggests, a sequential development model arranges the activities involved in the development process in a linear fashion. The assumption here is that development of the product and its feature set is finished when all the phases of the development model have been completed. This model does not envisage overlaps between phases or product iterations. The planned delivery date for projects run this way can lie months—or even years—in the future.

3.1.1 The Waterfall Model

An early model was the so-called “waterfall model” [Royce 70]. It is impressively simple and, in the past, enjoyed a high degree of popularity. Each development phase can only begin once the previous phase has been completed, hence the model’s name¹. However, the model can produce feedback loops between neighboring phases that require changes to be made in a previous phase. Figure 3-1 shows the phases incorporated in Royce’s original model:

Fig. 3-1 The waterfall model according to Royce

The major shortcoming of this model is that it bundles testing as a single activity at the end of the project. Testing only takes place once all other development activities have been completed, and is thus seen as a kind of “final check” akin to inspecting goods that leave a factory. In this case, testing is not seen as an activity that takes place throughout the development process.

3.1.2 The V-Model

The V-model is an extension of the waterfall model (see [Boehm 79], [ISO/IEC 12207]). The advent of this model made a huge and lasting difference to the way testing is viewed within the development process. Every tester and every developer should learn the V-model and learn about how it integrates the testing process. Even if a project is based on a different development model, the principles illustrated here can still be applied.

The basic idea is that development and testing are corresponding activities of equal value. In the diagram, they are illustrated by the two branches of the “V”²:

Fig. 3-2 The V-model

The left-hand branch represents the steps that are required to design and develop the system with increasing detail up to the point at which it is actually coded.²

The constructional activities in the left-hand branch correspond to the activities outlined in the waterfall model:

■ Definition of requirements

This is where the customer and end-user requirements are collected, specified, and approved. The purpose and proposed features of the system are now stipulated.

■ Functional design

The requirements are mapped to specific features and dialog flows.

■ Technical design

The functional design is mapped to a technical design that includes definition of the required interfaces to the outside world, and divides the system into easily manageable components that can be developed independently (i.e., the system architecture is drafted).

■ Component specification

The task, behavior, internal construction, and interfaces to other components are defined for each component.

■ Programming

Each specified component is programmed (i.e., implemented as a module, unit, class etc.) using a specific programming language.

Because it is easiest to identify defects at the level of abstraction on which they occur, each of the steps in the left-hand branch is given a corresponding testing step in the right-hand branch. The right-hand branch therefore represents an integration and testing flow during which system components are successively put together (i.e., integrated) to build increasingly large subsystems, that are then tested to ensure that they fulfill their proposed functions. The integration and testing process ends with acceptance testing for the complete system.

■ Component tests ensure that each individual component fulfills its specified requirements.

■ Integration tests ensure that groups of components interact as specified by the technical design.

■ System tests ensure that the system as a whole functions according to its specified requirements.

■ The acceptance test checks that the system as a whole adheres to the contractually agreed customer and end-user criteria.

These test steps represent a lot more than just a chronological order. Each test level checks the product (and/or specific work products) at a different level of abstraction and follows different testing objectives.

This is why the various test levels involve different testing techniques, different testing tools, and specialized personnel. Section 3.4 presents more details regarding each of these test levels.

In general, each test level can include verification and validation tests:

Did we build the system right?

■ Verification involves checking that the test object fulfills its specifications completely and correctly. In other words, the test object (i.e., the output of the corresponding development phase) is checked to see whether it was “correctly” developed according to its specifications (the input for the corresponding phase).

Did we build the right system?

■ Validation involves checking that the test object is actually usable within its intended context. In other words, the tester checks whether the test object actually solves the problem assigned to it and whether it is suited to its intended use.

Practically speaking, every test includes both aspects, although the validation share increases with each level of abstraction. Component tests are largely focused on verification, whereas an acceptance test is mainly about validation.

The V-model’s hallmarks

To summarize, the most important characteristics of the V-model are:

■ Development and test activities take place separately (indicated by the left and right-hand branches) but are equally important to the success of the project.

■ The model’s “V” shape helps to visualize the verification/validation aspects of testing.

■ It differentiates between collaborative test levels, whereby each level is testing against its corresponding development level.

The principle of early testing

The V-model creates the impression that testing begins late in the development process, following implementation. This is wrong! The test levels in the right-hand branch of the model represent the distinct phases of test execution. Test preparation (planning, analysis, and design) must begin within the corresponding development step in the left-hand branch.

3.2 Iterative and Incremental Development Models

Iterative development

The basic idea behind iterative development is that the development team can use the experience they gain from previous development stages along with real-world and customer feedback from earlier system versions to improve the product in future iterations. Such improvements can take the form of fault corrections or the alteration, extension or addition of specific features. The primary objective of all these scenarios is to improve the product step by step in order to meet customer expectations increasingly accurately.

Incremental development

The idea behind incremental development is to develop a product in preplanned stages, with each completed stage offering a more full-featured version (increment) of the product. Increments can vary greatly in size— for example from changing a simple web page layout to adding a complete new module with additional functionality. The primary objective of incremental development is to minimize time to market—i.e., to release a simple product version (or a simple version of a feature) to provide the customer as quickly as possible with a working version of the product or feature. Further enhancements will then be offered continually depending on the customer’s responses and wishes.

Iterative-incremental development

In practice, the borders between these two methodologies are blurred and they are often referred to together as iterative-incremental development. A defining characteristic of both is that each product release enables you to receive regular, early feedback from the customer and/or end-user. This reduces the risk of developing a system that doesn’t meet the customer’s expectations.

Examples of combined iterative-incremental models are: the spiral model [Boehm 86], Rapid Application Development (RAD) [Martin 91], Rational Unified Process (RUP) [Kruchten 03], and Evolutionary Development [Gilb 05].

Agile software development

All forms of agile software development are iterative-incremental development models. The best-known agile models are: Extreme Programming (XP) [Beck 04], Kanban [URL: Kanban], and Scrum [Beedle 02], [URL: Scrum Guide]. In recent years, Scrum has become the most popular of these and is extremely widespread.

Fig. 3-3 Scrum-based agile development

Testing to the rhythm of the iterations

The pace at which new increments/releases are created varies from model to model. While non-agile iterative-incremental projects tend to foresee releases at intervals of six months to a year, or sometimes even longer, agile models in contrast attempt to reduce the release cycle to a quarterly, monthly, or even weekly rhythm.

Here, testing has to be adapted to fit such short release cycles. For example, this means that every component requires re-usable test cases that can be easily and instantly repeated for each new increment. If this condition is not met, you risk reducing system reliability from increment to increment.

Each increment also requires new test cases that cover any additional functionality, which means the number of test cases you need to maintain and execute (on each release) increases over time. The shorter the release cycle, it remains critical but becomes more difficult for all test cases to be satisfactorily executed within the allotted release timeframe. Therefore test automation is an important tool when adapting your testing to agile development.

Continuous Integration and Continuous Deployment

Once you have set up a reliable automated test environment that executes your test cases with sufficient speed, you can use it for every new build. When a component is modified, it is integrated into the previous complete build, followed by a fresh automated test run³. Any failures that appear should be fixed in the short term. This way, the project always has a fully integrated and tested system running within its test environment. This approach is called “Continuous Integration” (CI).

This approach can be augmented using “Continuous Deployment” (CD): If the test run (during CI) is fault-free, the tested system is automatically copied to the production environment and installed there and thus deployed in a ready-to-run state⁴.

Continuous Delivery = Continuous Testing

Combining CI and CD results in a process called “Continuous Delivery”. These techniques can only be successfully applied if you have a largely automated testing environment at your disposal which enables you to perform “continuous testing”.

Continuous testing and other critical agile testing techniques are explained in detail in [Crispin 08] and [Linz 14].

3.3 Software Development in Project and Product Contexts

The requirements for planning and traceability of development and testing vary according to the context. Likewise, the appropriateness of a particular lifecycle model for the development of a specific product also depends on the contexts within which it is developed and used. The following project- and product-based factors play a role in deciding which model to use:

■ The company’s business priorities, project objectives, and risk profile. For example, if time-to-market is a primary requirement.

■ The type of product being developed. A small (perhaps department-internal) system has a less demanding development process than a large project designed for multi-year use by a huge customer base, such as our VSR-II case study project. Such large products are often developed using multiple models.

■ The market conditions and technical environment in which the product is used. For example, a product family developed for use in the Internet of Things (IoT) can consist of multiple types of objects (devices, services, platforms, and so on), each of which is developed using a specific and suitable lifecycle model. Because IoT objects are used for long periods of time in large numbers, it makes sense if their operational usage (distribution, updates, decommissioning, and so on) is mirrored in specific phases or catalogs of tasks within the lifecycle model. This makes developing new versions of such a system particularly challenging.

■ Identified product risks. For example, the safety aspects involved in designing and implementing a vehicle braking system.

■ Organizational and cultural aspects. For example, the difficulties generated by communication within international teams can make iterative or agile development more difficult.

Case Study: Mixing development models in the VSR-II project

One of the objectives of the VSR-II project is to make it “as agile as possible”, so the DreamCar module and all the browser-based front-end components and subsystems are developed in an agile Scrum environment. However, because they are safety-critical, the ConnectedCar components are to be developed using the traditional V-model.

Prototyping [URL: Prototyping] is also an option early on in a project and, once the experimental phase is complete, you can switch to an incremental approach for the rest of the project.

Tailoring

A development model can and should be adapted and customized for use within a specific project. This adaptation process is called “tailoring”.

Tailoring can involve combining test levels or certain testing activities and organizing them especially to suit the project at hand. For example, when integrating off-the-shelf commercial software into a larger system, interoperability tests at the integration testing stage (for example, when integrating with existing infrastructure or systems) can be performed by the customer rather than the supplier, as can acceptance testing (functional and non-functional operational and customer acceptance tests). For more detail, see section 3.4 and 3.5.

The tailored development model then comprises a view of the required activities, timescales, and objectives that is binding for all project participants. Any detailed planning (schedules, staffing, and infrastructure allocation) can then utilize and build upon the tailored development model.

Attributes of good testing

Regardless of which lifecycle model you choose, your tailoring should support good and effective testing. Your testing approach should include the following attributes:

■ Testing and its associated activities are included as early as possible in the lifecycle—for example, drafting test cases and setting up the test environment (see the principle of early testing above).

■ For every development activity, a corresponding test activity is planned and executed.

■ Test activities are planned and managed specifically to suit the objectives of the test level they belong to.

■ Test analysis and test design begin within the corresponding development phase.

■ As soon as work products (requirements, user stories, design documents, code etc.) exist, testers take part in discussions that refine them. Testers should participate early and continuously in this refinement process.

3.4 Testing Levels

A software system is usually composed of a number of subsystems, which in turn are made up of multiple components often referred to as units or modules. The resulting system structure is also called the systems “software architecture” or “architecture”. Designing an architecture that perfectly supports the system’s requirements is a critical part of the software development process.

During testing, a system has to be examined and tested on each level of its architecture, from the most elementary component right up to the complete, integrated system. The test activities that relate to a particular level of the architecture are known as a testing “level”, and each testing level is a single instance of the test process.

The following sections detail the differences between the various test levels with regard to their different test objects, test objectives, testing techniques, and responsibilities/roles.

3.4.1 Component Testing

Terminology

Component testing involves systematically checking the lowest-level components in a system’s architecture. Depending on the programming language used to create them, these components have various names, such as “units”, “modules” or (in the case of object-oriented programming) “classes”. The corresponding tests are therefore called “module tests”, “unit tests”, or “class tests”.

Components and component testing

Regardless of which programming language is the used, the resulting software building blocks are the “components” and the corresponding tests are called “component tests”.

The test basis

The component-specific requirements and the component’s design (i.e., its specifications) are to be consulted to form the test basis. In order to design white-box tests or to evaluate code coverage, you must analyze the component’s source code and use it as an additional test basis. However, to judge whether a component reacts correctly to a test case, you have to refer to the design or requirements documentation.

Test objects

As detailed above, modules, units, or classes are typical test objects. However, things like shell scripts, database scripts, data conversion and migration procedures, database content, and configuration files can all be test objects too.

A component test verifies a component’s internal functionality

A component test typically tests only a single component in isolation from the rest of the system. This isolation serves to exclude external influences during testing: If a test reveals a failure, it is then obviously attributable to the component you are testing. It also simplifies design and automation of the test cases, due to their narrowly focused scope.

A component can itself consist of multiple building blocks. The important aspect is that the component test has to check only the internal functionality of the component in question, not its interaction with components external to it. The latter is the subject of integration testing. Component test objects generally arrive “fresh from the programmers hard disk”, making this level of testing very closely allied to development work. Component testers therefore require adequate programming skills to do their job properly.

The following example illustrates the point:

Case Study: Testing the calculate_price class

According to its specifications, the VSR-II DreamCar module calculates a vehicle’s price as follows:

We start with the list price (baseprice) minus the dealer discount (discount). Special edition markup (specialprice) and the price of any additional extras (extraprice) are then added.

If three or more extras not included with the special edition are added (extras), these extras receive a 10% discount. For five extras or more, the discount increases to 15%.

The dealer discount is subtracted from the list price, while the accessory discount is only applied to the extras. The two discounts cannot be applied together.

The resulting price is calculated using the following C++ method⁵:

The test environment

In order to test this calculation, the tester uses the corresponding class interface by calling the calculate_price() method and providing it with appropriate test data. The tester then records the component’s reaction to this call—i.e., the value returned by the method call is read and logged.

This piece of code is buggy: the code for calculating the discount for ≥ 5 can never be reached. This coding error serves as an example to explain the white-box analysis detailed in Chapter 5.

To do this you need a “test driver”. A test driver is a separate program that makes the required interface call and logs the test object’s reaction (see also Chapter 5).

For the calculate_price() test object, a simple test driver could look like this:

The test driver in our example is very simple and could, for example, be extended to log the test data and the results with a timestamp, or to input the test data from an external data table.

Developer tests

To write a test driver you need programming skills. You also have to study and understand the test object’s code (or at least, that of its interface) in order to program a test driver that correctly calls the test object. In other words, you have to master the programming language involved and you need access to appropriate programming tools. This is why component testing is often performed by the component’s developers themselves. Such a test is then often referred to as a “developer test”, even though “component testing” is what is actually meant. The disadvantages of developers testing their own code are discussed in section 2.4.

Testing vs. debugging

Component tests are often confused with debugging. However, debugging involves eliminating defects, while testing involves systematically checking the system for failures (see section 2.1.2).

Our Tip

Use Component test frameworks

■ Using component test frameworks (see [URL: xUnit]) significantly reduces the effort involved in programming test drivers, and creates a consistent component test architecture throughout the project. Using standardized test drivers also makes it easier for other members of the team who aren’t familiar with the individual components or the test environment to perform component tests⁶. These kinds of test drivers can be controlled via a command-line interface and provide mechanisms for handling test data, and for logging and evaluating test results. Because all test data and logs are identically structured, it is possible to evaluate the results across multiple (or all) tested components.

Component test objectives

The component testing level is characterized not only by the type of test objects and the test environment, but also by very specific testing objectives.

Testing functionality

The most important task of a component test is checking that the test object fully and correctly implements the functionality defined in its specifications (such tests are also known as “function tests” or “functional tests”). In this case, functionality equates to the test object’s input/output behavior. In order to check the completeness and correctness of the implementation, the component is subjected to a series of test cases, with each covering a specific combination of input and output data.

Case Study: Testing VSR-II’s price calculations

This kind of testing input/output data combinations is illustrated nicely by the test cases in the example shown above. Each test case inputs a specific price combined with a specific number of extras. The test case then checks whether the test object calculates the correct total price.

For example, test case #2 checks the “discount for five or more extras”. When test case #2 is executed, the test object outputs an incorrect total price. Test case #2 produces a failure, indicating that the test object does not fulfill its specified requirements for this input data combination.

Typical failures revealed by component testing are faulty calculations or missing (or badly chosen) program paths (for example, overlooked or wrongly interpreted special cases).

Testing for robustness

At run time, a software component has to interact and swap data with multiple neighboring components, and it cannot be guaranteed that the component won’t be accessed and used wrongly (i.e., contrary to its specification). In such cases, the wrongly addressed component should not simply stop working and crash the system, but should instead react “reasonably” and robustly. Testing for robustness is therefore another important aspect of component testing. The process is very similar to that of an ordinary functional test, but serves the component under test with invalid input data instead of valid data. Such test cases are also referred to as “negative tests” and assume that the component will produce suitable exception handling as output. If adequate exception handling is not built in, the component may produce runtime errors, such as division by zero or null pointer access, that cause the system to crash.

Case Study: Negative tests

For the price calculation example we used previously, a negative test would involve testing with negative input values or a false data type (for example, char instead of int)⁷:

Various interesting things come to light:

■ Because the number of possible “bad” input values is virtually limitless, it is much easier to design “negative tests” than it is to design “positive tests”.

■ The test driver has to be extended in order to evaluate the exception handling produced by the test object.

■ Exception handling within the test object (evaluating ERR_CODE in our example) requires additional functionality. In practice, you will often find that half of the source code (or sometimes more) is designed to deal with exceptions. Robustness comes at a price.

Alongside functionality and robustness, component testing can also be used to check other attributes of a component that influence its quality and that can only be tested (if at all) using a lot of additional effort at higher test levels. Examples are the non-functional attributes “efficiency” and “maintainability”⁸.

Testing for efficiency

The efficiency attribute indicates how economically a component interacts with the available computing resources. This includes aspects such as memory use, processor use, or the time required to execute functions or algorithms. Unlike most other test objectives, the efficiency of a test object can be evaluated precisely using suitable test criteria, such as kilobytes of memory or response times measured in milliseconds. Efficiency testing is rarely performed for all the components in a system. It is usually restricted to components that have certain efficiency requirements defined in the requirements catalog or the component’s specification. For example, if limited hardware resources are available in an embedded system, or for a real-time system that has to guarantee predefined response-time limits.

Testing for maintainability

Maintainability incorporates all of the attributes that influence how easy (or difficult) it is to enhance or extend a program. The critical factor here is the amount of effort that is required for a developer (team) to get a grasp of the existing program and its context. This is just as valid for a developer who needs to modify a system that he programmed years ago as for someone who is taking over code from a colleague.

The main aspects of maintainability that need to be tested are code structure, modularity, code commenting, comprehensibility and up-to-dateness of the documentation, and so on.

Case Study: Code that is difficult to maintain

The sample calculate_price() code contains a number of maintainability issues. For example, there are no code comments at all, and numerical constants have not been declared as such and are instead hard-coded. If, for example, such a constant needs to be modified, it isn’t clear if and where else in the system it needs to be changed, forcing the developer to make huge efforts figuring this out.

Attributes like maintainability cannot of course be checked using dynamic tests (see Chapter 5). Instead, you will need to analyze the system’s specifications and its codebase using static tests and review sessions (see section 4.3). However, because you are checking attributes of individual components, this kind of analysis has to be carried out within the context of component testing.

Testing strategies

As already mentioned, component testing is highly development-oriented. The tester usually has access to the source code, supporting a white-box oriented testing technique in component testing. Here, a tester can design test cases using existing knowledge of a component’s internal structure, methods, and variables (see section 5.2).

White-box tests

The availability of the source code is also an advantage during test execution, as you can use appropriate debugging tools (see section 7.1.4) to observe the behavior of variables during testing and see whether the component functions properly or not. A debugger also enables you to manipulate the internal state of a component, so you can deliberately initiate exceptions when you are testing for robustness.

Case Study: Code as test basis

The calculate_price() code includes the following test-worthy statement:

Additional test cases that fulfill the condition (discount > addon_discount) are simple to derive from the code. But the price calculation specification contains no relevant information, and corresponding functionality is not part of the requirements. A code review can reveal a deficiency like this, enabling you to check whether the code is correct and the specification needs to be changed, or whether the code needs to be modified to fit the specification.

However, in many real-world situations, component tests are “only” performed as black-box tests—in other words, test cases are not based on the component’s inner structure⁹. Software systems often consist of hundreds or thousands of individual building blocks, so analyzing code is only really practical for selected components.

During integration, individual components are increasingly combined into larger units. These integrated units may already be too large to inspect their code thoroughly. Whether component testing is done on the individual components or on larger units (made up of multiple components) is an important decision that has to be made as part of the integration and test planning process.

Test-first

“Test-first” is the state-of-the-art approach to component testing (and, increasingly, on higher testing levels too). The idea is to first design and automate your test cases and to program the code which implements the component as a second step. This approach is strongly iterative: you test your code with the test cases you have already designed, and you then extend and improve your product code in small steps, repeating until the code fulfills your tests. This process is referred to as “test-first programming”, or “test-driven development” (often abbreviated to TDD—see also [URL: TDD], [Linz 14]). If you derive your test cases systematically using well founded test design techniques (see Chapter 5), this approach produces even more benefits—for example, negative tests, too, will be drafted before you begin programming and the team is forced to clarify the intended product behavior for these cases.