Multi-purpose Testing Tool Kit

I would like to reveal my secret toolkit which I use to tackle any test challenge.

These three thinking tools are shaped to work on every Testing task from small to big. Sometimes, using these tools makes the difference between the ordinary and the expert testers’ way of thinking.

What is in my kit?

I keep the following three thought tools at hand:

·         Definition of Quality

·         Risk Formula

·         Awareness to ROI

Definition of Quality

Every action that we do as testers should have connection to the quality of the product. We

learn the product, test it and provide information, in order to improve the quality. Otherwise - what is the point?

To decide whether an activity is connected to the quality of the product - we must first align ourselves on “what is quality”.

My preferred definition for quality is “value to someone” (Jerry Weinberg). Cem Kaner adds the extension “who matters”, which is useful for focusing on the needs of the person that we are testing for. The benefit of such a basic definition is that it can be used to criticize less basic baselines of quality, like “adhering to the official requirements” or quantitative measurements which are used in the organization process.

The Risk Calculation Formula

Our testing is an activity to mitigate risk. We test in order to provide information about the quality of the product, so that defects with the higher risk to the product quality will be fixed.  

A basic formula of calculating a risk is:

Risk = Probability multiple by impact

Using this formula helps us to compare between the different risks that we want to address. We will try to focus our work first on areas which have the highest probability of having issues that will have the highest impact on quality.

[Late note: My coulege, Amit Wertheimer point out in the hebrew testing forum of Tapuz  that it is important to remember that the caculation is based on our limited analisys  and not on accurate certain factors ]

Awareness of ROI

Testing is an economic activity. Testing consumes resources and is done in order to supply the one who send us to test with information that he values. Whether it is issues that he chooses to fix or risk assessment that will allow him to determine the shipping time of the product.

ROI - return on investment, means that something we do is “worth it” and does not cost us more than what we will get from it. Of course, since Testing is an activity of learning and gathering information, we can’t know ahead of time whether we will learn something that is “worth it”. Here, performing Risk analysis will help us to decide if a testing activity has a good probability of resulting in positive ROI or not.

Examples

Let’s look at some examples for using the tools:

A tester gets a new version of the SW and has to decide what to test first. The risk formula will guide him by viewing the risk factors: Looking at the Probability of possible failures, he will determine the areas which have higher probability of having defects - areas that been changed. Areas that tend to be buggy and so on... On the other hand he will look at the impact of possible issues that he can test for: what flows are the most critical for the product users, what type of defects are more costly to the firm. The first areas that he will pick to test will be the ones that will calculate the highest risk - probability to fail, multiplied by the impact the quality. These areas will have the most ROI as he assesses them as the most valuable.

This example can be true also when planning a test strategy for a whole product for a whole team.

Discussing Whether to fix a Defect or not

If you consider quality, risk and ROI, you understand that fixing a bug is not always the right choice.

A smart tester will use the risk formula and will guide the discussion to consider the impact on the quality, the probability that a user will be effected by the defect on the one hand, and on the other hand - the probability that fixing the defect will cause other defects. This way we can estimate the total cost of the fix in terms of risk and decide whether the defect fix has positive ROI.

Sometimes, a proposal to fix a defect is actually a choice between different quality attributes. For example, making the product more secure may reduce the performance. Examining the impact on the “someone who matters”, usually our user, will help us make a choice.

ROI is a key thinking tool to examine investments in infrastructure, tools and automation. A realistic estimation of how much the tool will cost - in terms of development, integration, maintenance and fees versus the benefits we expect to gain, such as time saving, additional test coverage, will help us decide whether to “go for it” or not.

Some Critique of the Above

Essentially, all models are wrong, but some are useful - said George E. P. Box. This is true for the thought models that I just described. Since Testing is a learning activity, and learning is unpredictable, sometimes wandering around may be the right thing to do. Low risk and low ROI activity that we barely connect to the product quality in the first place, may lead us to critical information that has high ROI and high risk which is connected to the product quality.

What is your secret tool? Do you use a different version of similar tools? Let me know.

Using the definition of quality as a tool for context awareness

What is quality?
Since testers job is to perform quality assessments, ask questions and provide answers regarding quality, understanding what quality is can help identify dilemmas in our work and put them in context.
My preferred definition for quality is “value to someone” (Jerry Weinberg). Cem Kaner adds the extension “who matters”. Usually I refer to this VIP as a “User”.

Recently, I noticed that using this definition helps identify context and explain the context to others during discussion.
A few examples:

Focusing a discussion on the goal rather than the process.
Process is important, but sometimes process discussions disconnect from the goal. For example, when a bug is opened and there is a discussion about whether it’s a requirement violation or not, providing insight on the User value could help direct the discussion to a productive place. This is also true when a tester spots an issue and is not sure whether it falls under his responsibility to report it – “is there a threat to the value to the users?” is a good litmus test to aid the decision.


Selecting a process
When defining a process, understanding how it’s connected to the value to the user is a good way to examine it. When we define a process it should connect our efforts to the goal rather than disconnect them. A negative example is when mixing between the priority for fixing the bug and the bug Severity. Many times there is correlation between the two, but it will be a good idea to define process that will address the cases when there is a difference between the two (like setting different field for each goal) so the information of the value threat severity will not be replaced by the work plan.

Determine classification of a problem
We face many types of problems. Some related to the quality of the product and some interfere with other aspects of our work, delay our progress or block our testing efforts. When a testability issue is examined in perspective of user value, it can be underrated since our inability to test efficiently is the real issue here, and not the impact of the tested attribute on the end user.
Sometimes there are two types of issues combined together in one problem description. Distinguishing between the value to the user and the other issue, helps provide a clearer explanation.

Overcoming the tunnel effect when setting bug severity
Setting correct bug exposure classification helps the bug life cycle start on the right foot. When a tester tests his area of responsibility and spots a problem, sometimes it’s not easy to relate to the big picture – what will be the impact on the user? Will he be able to recover? Or in other words – what is the threat in terms of value to the user. Answering this question easily directs the bug submitter to specify the correct bug severity.