Multi-purpose Testing Tool Kit

I would like to reveal my secret toolkit which I use to tackle any test challenge.

These three thinking tools are shaped to work on every Testing task from small to big. Sometimes, using these tools makes the difference between the ordinary and the expert testers’ way of thinking.

What is in my kit?

I keep the following three thought tools at hand:

·         Definition of Quality

·         Risk Formula

·         Awareness to ROI

Definition of Quality

Every action that we do as testers should have connection to the quality of the product. We

learn the product, test it and provide information, in order to improve the quality. Otherwise - what is the point?

To decide whether an activity is connected to the quality of the product - we must first align ourselves on “what is quality”.

My preferred definition for quality is “value to someone” (Jerry Weinberg). Cem Kaner adds the extension “who matters”, which is useful for focusing on the needs of the person that we are testing for. The benefit of such a basic definition is that it can be used to criticize less basic baselines of quality, like “adhering to the official requirements” or quantitative measurements which are used in the organization process.

The Risk Calculation Formula

Our testing is an activity to mitigate risk. We test in order to provide information about the quality of the product, so that defects with the higher risk to the product quality will be fixed.  

A basic formula of calculating a risk is:

Risk = Probability multiple by impact

Using this formula helps us to compare between the different risks that we want to address. We will try to focus our work first on areas which have the highest probability of having issues that will have the highest impact on quality.

[Late note: My coulege, Amit Wertheimer point out in the hebrew testing forum of Tapuz  that it is important to remember that the caculation is based on our limited analisys  and not on accurate certain factors ]

Awareness of ROI

Testing is an economic activity. Testing consumes resources and is done in order to supply the one who send us to test with information that he values. Whether it is issues that he chooses to fix or risk assessment that will allow him to determine the shipping time of the product.

ROI - return on investment, means that something we do is “worth it” and does not cost us more than what we will get from it. Of course, since Testing is an activity of learning and gathering information, we can’t know ahead of time whether we will learn something that is “worth it”. Here, performing Risk analysis will help us to decide if a testing activity has a good probability of resulting in positive ROI or not.

Examples

Let’s look at some examples for using the tools:

A tester gets a new version of the SW and has to decide what to test first. The risk formula will guide him by viewing the risk factors: Looking at the Probability of possible failures, he will determine the areas which have higher probability of having defects - areas that been changed. Areas that tend to be buggy and so on... On the other hand he will look at the impact of possible issues that he can test for: what flows are the most critical for the product users, what type of defects are more costly to the firm. The first areas that he will pick to test will be the ones that will calculate the highest risk - probability to fail, multiplied by the impact the quality. These areas will have the most ROI as he assesses them as the most valuable.

This example can be true also when planning a test strategy for a whole product for a whole team.

Discussing Whether to fix a Defect or not

If you consider quality, risk and ROI, you understand that fixing a bug is not always the right choice.

A smart tester will use the risk formula and will guide the discussion to consider the impact on the quality, the probability that a user will be effected by the defect on the one hand, and on the other hand - the probability that fixing the defect will cause other defects. This way we can estimate the total cost of the fix in terms of risk and decide whether the defect fix has positive ROI.

Sometimes, a proposal to fix a defect is actually a choice between different quality attributes. For example, making the product more secure may reduce the performance. Examining the impact on the “someone who matters”, usually our user, will help us make a choice.

ROI is a key thinking tool to examine investments in infrastructure, tools and automation. A realistic estimation of how much the tool will cost - in terms of development, integration, maintenance and fees versus the benefits we expect to gain, such as time saving, additional test coverage, will help us decide whether to “go for it” or not.

Some Critique of the Above

Essentially, all models are wrong, but some are useful - said George E. P. Box. This is true for the thought models that I just described. Since Testing is a learning activity, and learning is unpredictable, sometimes wandering around may be the right thing to do. Low risk and low ROI activity that we barely connect to the product quality in the first place, may lead us to critical information that has high ROI and high risk which is connected to the product quality.

What is your secret tool? Do you use a different version of similar tools? Let me know.